Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions modules/storage-ephemeral-storage-manage-config-and-eviction.adoc

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confused about how this stacks with modules/storage-ephemeral-storage-manage.adoc.

Ephemeral storage configuration effects pod scheduling and eviction:
In the old module, this section is entirely excluded for MS. In this new module, only the bullets are excluded.

Ephemeral storage configuration effects pod eviction:
This is included for MS in both the original module and this one.

Also echoing Darragh's question about inclusion of the module

I'm also a bit confused about what this gets around. IIUC, you were making this module because you had to get rid of the == headings in the old module, is that right? But this module also adds an == heading anyway.

Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
// Module included in the following assemblies:
//
//* storage/understanding-persistent-storage.adoc
//* storage/understanding-ephemeral-storage.adoc
//* microshift_storage/understanding-ephemeral-storage-microshift.adoc

:_mod-docs-content-type: CONCEPT
[id=storage-ephemeral-storage-manage-config-and-eviction_{context}]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Has this been added to a MiroShift assembly understanding-ephemeral-storage-microshift)?

= Ephemeral storage management configuration effects pod scheduling and eviction

[role="_abstract"]
The settings in the pod spec affect both how the scheduler makes a decision about scheduling pods, and when kubelet evicts pods.

ifndef::microshift[]

* First, the scheduler ensures that the sum of the resource requests of the scheduled containers is less than the capacity of the node. In this case, the pod can be assigned to a node only if the node's available ephemeral storage (allocatable resource) is more than 4GiB.
* Second, at the container level, because the first container sets a resource limit, kubelet eviction manager measures the disk usage of this container and evicts the pod if the storage usage of the container exceeds its limit (4GiB). The kubelet eviction manager also marks the pod for eviction if the total usage exceeds the overall pod storage limit (8GiB).
endif::microshift[]
ifdef::microshift[]
[id=storage-ephemeral-storage-eviction_{context}]
== Ephemeral storage configuration effects pod eviction
At the container level, because the first container sets a resource limit, kubelet eviction manager measures the disk usage of this container and evicts the pod if the storage usage of the container exceeds its limit (4GiB). The kubelet eviction manager also marks the pod for eviction if the total usage exceeds the overall pod storage limit (8GiB).

[NOTE]
====
This policy is strictly for `emptyDir` volumes and is not applied to persistent storage. You can specify the `priorityClass` of pods to exempt the pod from eviction.
====
endif::microshift[]
17 changes: 17 additions & 0 deletions modules/storage-ephemeral-storage-manage-overview.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
// Module included in the following assemblies:
//
//* storage/understanding-persistent-storage.adoc
//* storage/understanding-ephemeral-storage.adoc
//* microshift_storage/understanding-ephemeral-storage-microshift.adoc

:_mod-docs-content-type: CONCEPT
[id=storage-ephemeral-storage-manage-overview_{context}]
= Ephemeral storage management overview

[role="_abstract"]
Cluster administrators can manage ephemeral storage within a project by setting quotas that define the limit ranges and number of requests for ephemeral storage across all pods in a non-terminal state. Developers can also set requests and limits on this compute resource at the pod and container level.

You can manage local ephemeral storage by specifying requests and limits. Each container in a pod can specify the following:

* `spec.containers[].resources.limits.ephemeral-storage`
* `spec.containers[].resources.requests.ephemeral-storage`
61 changes: 61 additions & 0 deletions modules/storage-ephemeral-storage-manage-requests-and-limits.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
// Module included in the following assemblies:
//
//* storage/understanding-persistent-storage.adoc
//* storage/understanding-ephemeral-storage.adoc
//* microshift_storage/understanding-ephemeral-storage-microshift.adoc

:_mod-docs-content-type: CONCEPT

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would type this as ref

[id=storage-ephemeral-storage-manage-requests-and-limits_{context}]
= Ephemeral storage management limits and requests

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not understand the microshift docs that well. Would you be OK to provide a direct link to this module in the MShift docs?


[role="_abstract"]
Limits and requests for ephemeral storage are measured in byte quantities. You can express storage as a plain integer or as a fixed-point number by using one of these suffixes: E, P, T, G, M, k. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki. For example, the following quantities all represent approximately the same value: 128974848, 129e6, 129M, and 123Mi. Pod-level limits aggregate all container limits plus emptyDir volumes, enabling proper scheduling and preventing pods from exhausting node storage.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Limits and requests for ephemeral storage are measured in byte quantities. You can express storage as a plain integer or as a fixed-point number by using one of these suffixes: E, P, T, G, M, k. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki. For example, the following quantities all represent approximately the same value: 128974848, 129e6, 129M, and 123Mi. Pod-level limits aggregate all container limits plus emptyDir volumes, enabling proper scheduling and preventing pods from exhausting node storage.
Limits and requests for ephemeral storage are measured in byte quantities. You can express storage as a plain integer or as a fixed-point number by using one of these suffixes: E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki. For example, the following quantities all represent approximately the same value: 128974848, 129e6, 129M, and 123Mi. Pod-level limits aggregate all container limits plus `emptyDir` volumes, enabling proper scheduling and preventing pods from exhausting node storage.

Unless lowercase k is correct?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: lowercase k - there are a few of these, and it looks like it was that way in existing content. If it's an existing issue probably makes sense to fix as a docs bug broadly.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is also too long. When rewriting:

Avoid granular detail and reorient on user value (why they care about this). The specifics should be in the body, this should focus on why the user is here (think of it as information that you would see in a search blurb and go "aha! that's me, that's my question")


[IMPORTANT]
====
The suffixes for each byte quantity are case-sensitive. Be sure to use the correct case. Use the case-sensitive "M", such as used in "400M", to set the request at 400 megabytes. Use the case-sensitive "400Mi" to request 400 mebibytes. If you specify "400m" of ephemeral storage, the storage request is only 0.4 bytes.
====

The following example configuration file shows a pod with two containers:

* Each container requests 2GiB of local ephemeral storage.
* Each container has a limit of 4GiB of local ephemeral storage.
* At the pod level, kubelet works out an overall pod storage limit by adding up the limits of all the containers in that pod.
** In this case, the total storage usage at the pod level is the sum of the disk usage from all containers plus the pod's `emptyDir` volumes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
** In this case, the total storage usage at the pod level is the sum of the disk usage from all containers plus the pod's `emptyDir` volumes.
** In this case, the total storage usage at the pod level is the sum of the disk usage from all containers plus the `emptyDir` volumes of a pod.

IBM gets hung up on possessives.

** Therefore, the pod has a request of 4GiB of local ephemeral storage, and a limit of 8GiB of local ephemeral storage.

.Example ephemeral storage configuration with quotas and limits
[source, yaml]
----
apiVersion: v1
kind: Pod
metadata:
name: frontend
spec:
containers:
- name: app
image: images.my-company.example/app:v4
resources:
requests:
ephemeral-storage: "2Gi"
limits:
ephemeral-storage: "4Gi"
volumeMounts:
- name: ephemeral
mountPath: "/tmp"
- name: log-aggregator
image: images.my-company.example/log-aggregator:v6
resources:
requests:
ephemeral-storage: "2Gi"
limits:
ephemeral-storage: "4Gi"
volumeMounts:
- name: ephemeral
mountPath: "/tmp"
volumes:
- name: ephemeral
emptyDir: {}
----
* `spec.containers.name.resources.requests.ephemeral-storage`: Container request for local ephemeral storage.
* `spec.containers.name.resources.limits.ephemeral-storage`: Container limit for local ephemeral storage.
35 changes: 17 additions & 18 deletions modules/storage-ephemeral-storage-manage.adoc
Comment thread
lpettyjo marked this conversation as resolved.
Original file line number Diff line number Diff line change
Expand Up @@ -17,26 +17,24 @@ You can manage local ephemeral storage by specifying requests and limits. Each c
* `spec.containers[].resources.requests.ephemeral-storage`

[id=storage-ephemeral-storage-limits-requests-units_{context}]
== Ephemeral storage limits and requests units

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TBH, I am also confused why you were told to get rid of the == headings at all. The mod docs guidelines have them for concepts:

Image

Limits and requests for ephemeral storage are measured in byte quantities. You can express storage as a plain integer or as a fixed-point number using one of these suffixes: E, P, T, G, M, k. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.

For example, the following quantities all represent approximately the same value: 128974848, 129e6, 129M, and 123Mi.

Ephemeral storage limits and requests units::
Limits and requests for ephemeral storage are measured in byte quantities. You can express storage as a plain integer or as a fixed-point number by using one of these suffixes: E, P, T, G, M, k. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki. For example, the following quantities all represent approximately the same value: 128974848, 129e6, 129M, and 123Mi. Pod-level limits aggregate all container limits plus emptyDir volumes, enabling proper scheduling and preventing pods from exhausting node storage.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Limits and requests for ephemeral storage are measured in byte quantities. You can express storage as a plain integer or as a fixed-point number by using one of these suffixes: E, P, T, G, M, k. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki. For example, the following quantities all represent approximately the same value: 128974848, 129e6, 129M, and 123Mi. Pod-level limits aggregate all container limits plus emptyDir volumes, enabling proper scheduling and preventing pods from exhausting node storage.
Limits and requests for ephemeral storage are measured in byte quantities. You can express storage as a plain integer or as a fixed-point number by using one of these suffixes: E, P, T, G, M, k. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki. For example, the following quantities all represent approximately the same value: 128974848, 129e6, 129M, and 123Mi. Pod-level limits aggregate all container limits plus `emptyDir` volumes, enabling proper scheduling and preventing pods from exhausting node storage.

+
[IMPORTANT]
====
The suffixes for each byte quantity are case-sensitive. Be sure to use the correct case. Use the case-sensitive "M", such as used in "400M", to set the request at 400 megabytes. Use the case-sensitive "400Mi" to request 400 mebibytes. If you specify "400m" of ephemeral storage, the storage request is only 0.4 bytes.
====

[id=storage-ephemeral-storage-requests-limits_{context}]
== Ephemeral storage requests and limits example
[id=storage-ephemeral-storage-requests-limits-example_{context}]
Ephemeral storage requests and limits example::
The following example configuration file shows a pod with two containers:

+
* Each container requests 2GiB of local ephemeral storage.
* Each container has a limit of 4GiB of local ephemeral storage.
* At the pod level, kubelet works out an overall pod storage limit by adding up the limits of all the containers in that pod.
** In this case, the total storage usage at the pod level is the sum of the disk usage from all containers plus the pod's `emptyDir` volumes.
** Therefore, the pod has a request of 4GiB of local ephemeral storage, and a limit of 8GiB of local ephemeral storage.

+
.Example ephemeral storage configuration with quotas and limits
[source, yaml]
----
Expand All @@ -50,9 +48,9 @@ spec:
image: images.my-company.example/app:v4
resources:
requests:
ephemeral-storage: "2Gi" <1>
ephemeral-storage: "2Gi"
limits:
ephemeral-storage: "4Gi" <2>
ephemeral-storage: "4Gi"
volumeMounts:
- name: ephemeral
mountPath: "/tmp"
Expand All @@ -70,24 +68,25 @@ spec:
- name: ephemeral
emptyDir: {}
----
<1> Container request for local ephemeral storage.
<2> Container limit for local ephemeral storage.
+
* `spec.containers.resources.requests.ephemeral-storage`: Container request for local ephemeral storage.
* `spec.containers.resources.limits.ephemeral-storage`: Container limit for local ephemeral storage.

ifndef::microshift[]
[id=storage-ephemeral-storage-scheduling-eviction_{context}]
== Ephemeral storage configuration effects pod scheduling and eviction
Ephemeral storage configuration effects pod scheduling and eviction::
The settings in the pod spec affect both how the scheduler makes a decision about scheduling pods and when kubelet evicts pods.

+
* First, the scheduler ensures that the sum of the resource requests of the scheduled containers is less than the capacity of the node. In this case, the pod can be assigned to a node only if the node's available ephemeral storage (allocatable resource) is more than 4GiB.

+
* Second, at the container level, because the first container sets a resource limit, kubelet eviction manager measures the disk usage of this container and evicts the pod if the storage usage of the container exceeds its limit (4GiB). The kubelet eviction manager also marks the pod for eviction if the total usage exceeds the overall pod storage limit (8GiB).
endif::microshift[]

ifdef::microshift[]
[id=storage-ephemeral-storage-eviction_{context}]
== Ephemeral storage configuration effects pod eviction
Ephemeral storage configuration effects pod eviction::
The settings in the pod spec affect when kubelet evicts pods. At the container level, because the first container sets a resource limit, kubelet eviction manager measures the disk usage of this container and evicts the pod if the storage usage of the container exceeds its limit (4GiB). The kubelet eviction manager also marks the pod for eviction if the total usage exceeds the overall pod storage limit (8GiB).

+
[NOTE]
====
This policy is strictly for `emptyDir` volumes and is not applied to persistent storage. You can specify the `priorityClass` of pods to exempt the pod from eviction.
Expand Down
6 changes: 4 additions & 2 deletions modules/storage-ephemeral-storage-monitoring.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,15 @@
= Monitoring ephemeral storage

[role="_abstract"]
To monitor ephemeral storage usage, use the `/bin/df` utility. By using this tool, you can track disk space consumption on the volumes where ephemeral container data resides, specifically `/var/lib/kubelet` and `/var/lib/containers`.
Monitor ephemeral storage usage with the `/bin/df` utility. By using this tool, you can track disk space consumption on the volumes where ephemeral container data resides, specifically `/var/lib/kubelet` and `/var/lib/containers`, preventing pod evictions from storage exhaustion. Regular monitoring helps you identify storage-hungry workloads and adjust requests and limits before kubelet evicts pods.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also too long (~400 char)


When you use the `df` command, the available space for only `/var/lib/kubelet` is shown if `/var/lib/containers` is placed on a separate disk by the cluster administrator.

You can use `/bin/df` as a tool to monitor ephemeral storage usage on the volume where ephemeral container data is located, which is `/var/lib/kubelet` and `/var/lib/containers`. The available space for only `/var/lib/kubelet` is shown when you use the `df` command if `/var/lib/containers` is placed on a separate disk by the cluster administrator.

.Procedure

* To show the human-readable values of used and available space in `/var/lib`, enter the following command:
* To show the human-readable values of used and available space in `/var/lib`, run the following command:
+
[source,terminal]
----
Expand Down
4 changes: 2 additions & 2 deletions modules/storage-ephemeral-storage-overview.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,11 @@
= Overview of ephemeral storage

[role="_abstract"]
To manage local storage for stateless applications, use ephemeral storage. Ephemeral storage is designed for the transient nature of pods and containers. You can use this feature to handle operations that require storage only for the duration of the lifecycle for the workload. Both developers and administrators can use the feature.
To manage local storage for stateless applications, use ephemeral storage. Ephemeral storage is designed for the transient nature of pods and containers. You can use this feature to handle operations that require storage only for the duration of the lifecycle for the workload. Pods use ephemeral local storage for scratch space, caching, and logs. Both developers and administrators can use the feature.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be shorter (50-300 char for short descs). I know that seems nitpicky, but AEM enforces the character limit and will not let you save a topic if the short desc outside that count.


In addition to persistent storage, pods and containers can require ephemeral or transient local storage for their operation. The lifetime of this ephemeral storage does not extend beyond the life of the individual pod, and this ephemeral storage cannot be shared across pods.

Pods use ephemeral local storage for scratch space, caching, and logs. Issues related to the lack of local storage accounting and isolation include the following:
Issues related to the lack of local storage accounting and isolation include the following:

* Pods cannot detect how much local storage is available to them.
* Pods cannot request guaranteed local storage.
Expand Down
8 changes: 2 additions & 6 deletions modules/storage-ephemeral-storage-types.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,6 @@
[role="_abstract"]
To provision ephemeral local storage, you can create the primary partition by using either root or runtime methods. This storage is always made available in the primary partition, providing temporary space for your workloads.

== Root
* *Root*: This partition holds the kubelet root directory, `/var/lib/kubelet/` by default, and `/var/log/` directory. This partition can be shared between user pods, the operating system, and Kubernetes system daemons. This partition can be consumed by pods through `EmptyDir` volumes, container logs, image layers, and container-writable layers. Kubelet manages shared access and isolation of this partition. This partition is ephemeral, and applications cannot expect any performance SLAs, such as disk IOPS, from this partition.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd recommend a description list instead of manually formatting an unordered list with bold and colons


This partition holds the kubelet root directory, `/var/lib/kubelet/` by default, and `/var/log/` directory. This partition can be shared between user pods, the OS, and Kubernetes system daemons. This partition can be consumed by pods through `EmptyDir` volumes, container logs, image layers, and container-writable layers. Kubelet manages shared access and isolation of this partition. This partition is ephemeral, and applications cannot expect any performance SLAs, such as disk IOPS, from this partition.

== Runtime

This is an optional partition that runtimes can use for overlay file systems. {product-title} attempts to identify and provide shared access along with isolation to this partition. Container image layers and writable layers are stored here. If the runtime partition exists, the `root` partition does not hold any image layer or other writable storage.
* *Runtime*: This is an optional partition that runtimes can use for overlay file systems. {product-title} attempts to identify and provide shared access along with isolation to this partition. Container image layers and writable layers are stored here. If the runtime partition exists, the `root` partition does not hold any image layer or other writable storage.
20 changes: 13 additions & 7 deletions storage/understanding-ephemeral-storage.adoc
Original file line number Diff line number Diff line change
@@ -1,25 +1,31 @@
:_mod-docs-content-type: ASSEMBLY
[id="understanding-ephemeral-storage"]
= Understanding ephemeral storage
include::_attributes/common-attributes.adoc[]
:toc:
:toc-title:
:context: understanding-ephemeral-storage

toc::[]
Comment on lines +4 to -7

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not familiar with this construct

:toc:
:toc-title:

I don't follow why it would change from toc::[].

I'm not saying I'm right, just that it made me scratch my head 😂 If you can point me at a resource about this I would appreciate it so that I understand 🤓

include::_attributes/common-attributes.adoc[]

[role="_abstract"]
To manage temporary data that exists only for the duration of a workload, review the properties of ephemeral storage.
Ephemeral storage provides temporary per-pod storage for scratch data, caches, and logs that do not need to persist beyond the pod's lifetime. Understanding the different ephemeral storage types and their resource management helps you choose the right option for stateless workloads while preventing node storage exhaustion.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Ephemeral storage provides temporary per-pod storage for scratch data, caches, and logs that do not need to persist beyond the pod's lifetime. Understanding the different ephemeral storage types and their resource management helps you choose the right option for stateless workloads while preventing node storage exhaustion.
Ephemeral storage provides temporary per-pod storage for scratch data, caches, and logs that do not need to persist beyond the pod's lifetime. By understanding the different ephemeral storage types and their resource management, you can choose the right option for stateless workloads while preventing node storage exhaustion.

Removes anthrophomorphism.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SMEs review might be needed for these abstracts if they are new.


toc::[]

include::modules/storage-ephemeral-storage-overview.adoc[leveloffset=+1]

include::modules/storage-ephemeral-storage-types.adoc[leveloffset=+1]

include::modules/storage-ephemeral-storage-manage.adoc[leveloffset=+1]
include::modules/storage-ephemeral-storage-manage-overview.adoc[leveloffset=+1]

include::modules/storage-ephemeral-storage-manage-requests-and-limits.adoc[leveloffset=+2]

ifndef::openshift-dedicated,openshift-rosa,openshift-rosa-hcp[]
[id="additional-resources_{context}"]
== Additional resources

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the first add resources be of the format?

[role="additional-resources"]
[id="additional-resources
{context}"]
== Additional resources

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[role="additional-resources"]
[id="additional-resources_{context}"]
== Additional resources

Is only for the assembly-level AR section that comes at the very end of the assembly. It is never valid at any other location.

[role="_additional-resources"]
.Additional resources

* xref:../applications/quotas/quotas-setting-per-project.adoc#quotas-setting-per-project_quotas-setting-per-project[Resources managed by quotas]
endif::openshift-dedicated,openshift-rosa,openshift-rosa-hcp[]

include::modules/storage-ephemeral-storage-manage-config-and-eviction.adoc[leveloffset=+2]

include::modules/storage-ephemeral-storage-monitoring.adoc[leveloffset=+1]