-
Notifications
You must be signed in to change notification settings - Fork 1.9k
OSDOCS-16945# CQA work Stor1 - understanding ephemeral storage #106594
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| // Module included in the following assemblies: | ||
| // | ||
| //* storage/understanding-persistent-storage.adoc | ||
| //* storage/understanding-ephemeral-storage.adoc | ||
| //* microshift_storage/understanding-ephemeral-storage-microshift.adoc | ||
|
|
||
| :_mod-docs-content-type: CONCEPT | ||
| [id=storage-ephemeral-storage-manage-config-and-eviction_{context}] | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Has this been added to a MiroShift assembly understanding-ephemeral-storage-microshift)? |
||
| = Ephemeral storage management configuration effects pod scheduling and eviction | ||
|
|
||
| [role="_abstract"] | ||
| The settings in the pod spec affect both how the scheduler makes a decision about scheduling pods, and when kubelet evicts pods. | ||
|
|
||
| ifndef::microshift[] | ||
|
|
||
| * First, the scheduler ensures that the sum of the resource requests of the scheduled containers is less than the capacity of the node. In this case, the pod can be assigned to a node only if the node's available ephemeral storage (allocatable resource) is more than 4GiB. | ||
| * Second, at the container level, because the first container sets a resource limit, kubelet eviction manager measures the disk usage of this container and evicts the pod if the storage usage of the container exceeds its limit (4GiB). The kubelet eviction manager also marks the pod for eviction if the total usage exceeds the overall pod storage limit (8GiB). | ||
| endif::microshift[] | ||
| ifdef::microshift[] | ||
| [id=storage-ephemeral-storage-eviction_{context}] | ||
| == Ephemeral storage configuration effects pod eviction | ||
| At the container level, because the first container sets a resource limit, kubelet eviction manager measures the disk usage of this container and evicts the pod if the storage usage of the container exceeds its limit (4GiB). The kubelet eviction manager also marks the pod for eviction if the total usage exceeds the overall pod storage limit (8GiB). | ||
|
|
||
| [NOTE] | ||
| ==== | ||
| This policy is strictly for `emptyDir` volumes and is not applied to persistent storage. You can specify the `priorityClass` of pods to exempt the pod from eviction. | ||
| ==== | ||
| endif::microshift[] | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| // Module included in the following assemblies: | ||
| // | ||
| //* storage/understanding-persistent-storage.adoc | ||
| //* storage/understanding-ephemeral-storage.adoc | ||
| //* microshift_storage/understanding-ephemeral-storage-microshift.adoc | ||
|
|
||
| :_mod-docs-content-type: CONCEPT | ||
| [id=storage-ephemeral-storage-manage-overview_{context}] | ||
| = Ephemeral storage management overview | ||
|
|
||
| [role="_abstract"] | ||
| Cluster administrators can manage ephemeral storage within a project by setting quotas that define the limit ranges and number of requests for ephemeral storage across all pods in a non-terminal state. Developers can also set requests and limits on this compute resource at the pod and container level. | ||
|
|
||
| You can manage local ephemeral storage by specifying requests and limits. Each container in a pod can specify the following: | ||
|
|
||
| * `spec.containers[].resources.limits.ephemeral-storage` | ||
| * `spec.containers[].resources.requests.ephemeral-storage` |
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,61 @@ | ||||||
| // Module included in the following assemblies: | ||||||
| // | ||||||
| //* storage/understanding-persistent-storage.adoc | ||||||
| //* storage/understanding-ephemeral-storage.adoc | ||||||
| //* microshift_storage/understanding-ephemeral-storage-microshift.adoc | ||||||
|
|
||||||
| :_mod-docs-content-type: CONCEPT | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Would type this as ref |
||||||
| [id=storage-ephemeral-storage-manage-requests-and-limits_{context}] | ||||||
| = Ephemeral storage management limits and requests | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I do not understand the microshift docs that well. Would you be OK to provide a direct link to this module in the MShift docs? |
||||||
|
|
||||||
| [role="_abstract"] | ||||||
| Limits and requests for ephemeral storage are measured in byte quantities. You can express storage as a plain integer or as a fixed-point number by using one of these suffixes: E, P, T, G, M, k. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki. For example, the following quantities all represent approximately the same value: 128974848, 129e6, 129M, and 123Mi. Pod-level limits aggregate all container limits plus emptyDir volumes, enabling proper scheduling and preventing pods from exhausting node storage. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Unless lowercase
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. re: lowercase
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is also too long. When rewriting: Avoid granular detail and reorient on user value (why they care about this). The specifics should be in the body, this should focus on why the user is here (think of it as information that you would see in a search blurb and go "aha! that's me, that's my question") |
||||||
|
|
||||||
| [IMPORTANT] | ||||||
| ==== | ||||||
| The suffixes for each byte quantity are case-sensitive. Be sure to use the correct case. Use the case-sensitive "M", such as used in "400M", to set the request at 400 megabytes. Use the case-sensitive "400Mi" to request 400 mebibytes. If you specify "400m" of ephemeral storage, the storage request is only 0.4 bytes. | ||||||
| ==== | ||||||
|
|
||||||
| The following example configuration file shows a pod with two containers: | ||||||
|
|
||||||
| * Each container requests 2GiB of local ephemeral storage. | ||||||
| * Each container has a limit of 4GiB of local ephemeral storage. | ||||||
| * At the pod level, kubelet works out an overall pod storage limit by adding up the limits of all the containers in that pod. | ||||||
| ** In this case, the total storage usage at the pod level is the sum of the disk usage from all containers plus the pod's `emptyDir` volumes. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
IBM gets hung up on possessives. |
||||||
| ** Therefore, the pod has a request of 4GiB of local ephemeral storage, and a limit of 8GiB of local ephemeral storage. | ||||||
|
|
||||||
| .Example ephemeral storage configuration with quotas and limits | ||||||
| [source, yaml] | ||||||
| ---- | ||||||
| apiVersion: v1 | ||||||
| kind: Pod | ||||||
| metadata: | ||||||
| name: frontend | ||||||
| spec: | ||||||
| containers: | ||||||
| - name: app | ||||||
| image: images.my-company.example/app:v4 | ||||||
| resources: | ||||||
| requests: | ||||||
| ephemeral-storage: "2Gi" | ||||||
| limits: | ||||||
| ephemeral-storage: "4Gi" | ||||||
| volumeMounts: | ||||||
| - name: ephemeral | ||||||
| mountPath: "/tmp" | ||||||
| - name: log-aggregator | ||||||
| image: images.my-company.example/log-aggregator:v6 | ||||||
| resources: | ||||||
| requests: | ||||||
| ephemeral-storage: "2Gi" | ||||||
| limits: | ||||||
| ephemeral-storage: "4Gi" | ||||||
| volumeMounts: | ||||||
| - name: ephemeral | ||||||
| mountPath: "/tmp" | ||||||
| volumes: | ||||||
| - name: ephemeral | ||||||
| emptyDir: {} | ||||||
| ---- | ||||||
| * `spec.containers.name.resources.requests.ephemeral-storage`: Container request for local ephemeral storage. | ||||||
| * `spec.containers.name.resources.limits.ephemeral-storage`: Container limit for local ephemeral storage. | ||||||
|
lpettyjo marked this conversation as resolved.
|
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -17,26 +17,24 @@ You can manage local ephemeral storage by specifying requests and limits. Each c | |||||
| * `spec.containers[].resources.requests.ephemeral-storage` | ||||||
|
|
||||||
| [id=storage-ephemeral-storage-limits-requests-units_{context}] | ||||||
| == Ephemeral storage limits and requests units | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. TBH, I am also confused why you were told to get rid of the
|
||||||
| Limits and requests for ephemeral storage are measured in byte quantities. You can express storage as a plain integer or as a fixed-point number using one of these suffixes: E, P, T, G, M, k. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki. | ||||||
|
|
||||||
| For example, the following quantities all represent approximately the same value: 128974848, 129e6, 129M, and 123Mi. | ||||||
|
|
||||||
| Ephemeral storage limits and requests units:: | ||||||
| Limits and requests for ephemeral storage are measured in byte quantities. You can express storage as a plain integer or as a fixed-point number by using one of these suffixes: E, P, T, G, M, k. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki. For example, the following quantities all represent approximately the same value: 128974848, 129e6, 129M, and 123Mi. Pod-level limits aggregate all container limits plus emptyDir volumes, enabling proper scheduling and preventing pods from exhausting node storage. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| + | ||||||
| [IMPORTANT] | ||||||
| ==== | ||||||
| The suffixes for each byte quantity are case-sensitive. Be sure to use the correct case. Use the case-sensitive "M", such as used in "400M", to set the request at 400 megabytes. Use the case-sensitive "400Mi" to request 400 mebibytes. If you specify "400m" of ephemeral storage, the storage request is only 0.4 bytes. | ||||||
| ==== | ||||||
|
|
||||||
| [id=storage-ephemeral-storage-requests-limits_{context}] | ||||||
| == Ephemeral storage requests and limits example | ||||||
| [id=storage-ephemeral-storage-requests-limits-example_{context}] | ||||||
| Ephemeral storage requests and limits example:: | ||||||
| The following example configuration file shows a pod with two containers: | ||||||
|
|
||||||
| + | ||||||
| * Each container requests 2GiB of local ephemeral storage. | ||||||
| * Each container has a limit of 4GiB of local ephemeral storage. | ||||||
| * At the pod level, kubelet works out an overall pod storage limit by adding up the limits of all the containers in that pod. | ||||||
| ** In this case, the total storage usage at the pod level is the sum of the disk usage from all containers plus the pod's `emptyDir` volumes. | ||||||
| ** Therefore, the pod has a request of 4GiB of local ephemeral storage, and a limit of 8GiB of local ephemeral storage. | ||||||
|
|
||||||
| + | ||||||
| .Example ephemeral storage configuration with quotas and limits | ||||||
| [source, yaml] | ||||||
| ---- | ||||||
|
|
@@ -50,9 +48,9 @@ spec: | |||||
| image: images.my-company.example/app:v4 | ||||||
| resources: | ||||||
| requests: | ||||||
| ephemeral-storage: "2Gi" <1> | ||||||
| ephemeral-storage: "2Gi" | ||||||
| limits: | ||||||
| ephemeral-storage: "4Gi" <2> | ||||||
| ephemeral-storage: "4Gi" | ||||||
| volumeMounts: | ||||||
| - name: ephemeral | ||||||
| mountPath: "/tmp" | ||||||
|
|
@@ -70,24 +68,25 @@ spec: | |||||
| - name: ephemeral | ||||||
| emptyDir: {} | ||||||
| ---- | ||||||
| <1> Container request for local ephemeral storage. | ||||||
| <2> Container limit for local ephemeral storage. | ||||||
| + | ||||||
| * `spec.containers.resources.requests.ephemeral-storage`: Container request for local ephemeral storage. | ||||||
| * `spec.containers.resources.limits.ephemeral-storage`: Container limit for local ephemeral storage. | ||||||
|
|
||||||
| ifndef::microshift[] | ||||||
| [id=storage-ephemeral-storage-scheduling-eviction_{context}] | ||||||
| == Ephemeral storage configuration effects pod scheduling and eviction | ||||||
| Ephemeral storage configuration effects pod scheduling and eviction:: | ||||||
| The settings in the pod spec affect both how the scheduler makes a decision about scheduling pods and when kubelet evicts pods. | ||||||
|
|
||||||
| + | ||||||
| * First, the scheduler ensures that the sum of the resource requests of the scheduled containers is less than the capacity of the node. In this case, the pod can be assigned to a node only if the node's available ephemeral storage (allocatable resource) is more than 4GiB. | ||||||
|
|
||||||
| + | ||||||
| * Second, at the container level, because the first container sets a resource limit, kubelet eviction manager measures the disk usage of this container and evicts the pod if the storage usage of the container exceeds its limit (4GiB). The kubelet eviction manager also marks the pod for eviction if the total usage exceeds the overall pod storage limit (8GiB). | ||||||
| endif::microshift[] | ||||||
|
|
||||||
| ifdef::microshift[] | ||||||
| [id=storage-ephemeral-storage-eviction_{context}] | ||||||
| == Ephemeral storage configuration effects pod eviction | ||||||
| Ephemeral storage configuration effects pod eviction:: | ||||||
| The settings in the pod spec affect when kubelet evicts pods. At the container level, because the first container sets a resource limit, kubelet eviction manager measures the disk usage of this container and evicts the pod if the storage usage of the container exceeds its limit (4GiB). The kubelet eviction manager also marks the pod for eviction if the total usage exceeds the overall pod storage limit (8GiB). | ||||||
|
|
||||||
| + | ||||||
| [NOTE] | ||||||
| ==== | ||||||
| This policy is strictly for `emptyDir` volumes and is not applied to persistent storage. You can specify the `priorityClass` of pods to exempt the pod from eviction. | ||||||
|
|
||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -8,13 +8,15 @@ | |
| = Monitoring ephemeral storage | ||
|
|
||
| [role="_abstract"] | ||
| To monitor ephemeral storage usage, use the `/bin/df` utility. By using this tool, you can track disk space consumption on the volumes where ephemeral container data resides, specifically `/var/lib/kubelet` and `/var/lib/containers`. | ||
| Monitor ephemeral storage usage with the `/bin/df` utility. By using this tool, you can track disk space consumption on the volumes where ephemeral container data resides, specifically `/var/lib/kubelet` and `/var/lib/containers`, preventing pod evictions from storage exhaustion. Regular monitoring helps you identify storage-hungry workloads and adjust requests and limits before kubelet evicts pods. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Also too long (~400 char) |
||
|
|
||
| When you use the `df` command, the available space for only `/var/lib/kubelet` is shown if `/var/lib/containers` is placed on a separate disk by the cluster administrator. | ||
|
|
||
| You can use `/bin/df` as a tool to monitor ephemeral storage usage on the volume where ephemeral container data is located, which is `/var/lib/kubelet` and `/var/lib/containers`. The available space for only `/var/lib/kubelet` is shown when you use the `df` command if `/var/lib/containers` is placed on a separate disk by the cluster administrator. | ||
|
|
||
| .Procedure | ||
|
|
||
| * To show the human-readable values of used and available space in `/var/lib`, enter the following command: | ||
| * To show the human-readable values of used and available space in `/var/lib`, run the following command: | ||
| + | ||
| [source,terminal] | ||
| ---- | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -9,11 +9,11 @@ | |
| = Overview of ephemeral storage | ||
|
|
||
| [role="_abstract"] | ||
| To manage local storage for stateless applications, use ephemeral storage. Ephemeral storage is designed for the transient nature of pods and containers. You can use this feature to handle operations that require storage only for the duration of the lifecycle for the workload. Both developers and administrators can use the feature. | ||
| To manage local storage for stateless applications, use ephemeral storage. Ephemeral storage is designed for the transient nature of pods and containers. You can use this feature to handle operations that require storage only for the duration of the lifecycle for the workload. Pods use ephemeral local storage for scratch space, caching, and logs. Both developers and administrators can use the feature. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This needs to be shorter (50-300 char for short descs). I know that seems nitpicky, but AEM enforces the character limit and will not let you save a topic if the short desc outside that count. |
||
|
|
||
| In addition to persistent storage, pods and containers can require ephemeral or transient local storage for their operation. The lifetime of this ephemeral storage does not extend beyond the life of the individual pod, and this ephemeral storage cannot be shared across pods. | ||
|
|
||
| Pods use ephemeral local storage for scratch space, caching, and logs. Issues related to the lack of local storage accounting and isolation include the following: | ||
| Issues related to the lack of local storage accounting and isolation include the following: | ||
|
|
||
| * Pods cannot detect how much local storage is available to them. | ||
| * Pods cannot request guaranteed local storage. | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -10,10 +10,6 @@ | |
| [role="_abstract"] | ||
| To provision ephemeral local storage, you can create the primary partition by using either root or runtime methods. This storage is always made available in the primary partition, providing temporary space for your workloads. | ||
|
|
||
| == Root | ||
| * *Root*: This partition holds the kubelet root directory, `/var/lib/kubelet/` by default, and `/var/log/` directory. This partition can be shared between user pods, the operating system, and Kubernetes system daemons. This partition can be consumed by pods through `EmptyDir` volumes, container logs, image layers, and container-writable layers. Kubelet manages shared access and isolation of this partition. This partition is ephemeral, and applications cannot expect any performance SLAs, such as disk IOPS, from this partition. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd recommend a description list instead of manually formatting an unordered list with bold and colons |
||
|
|
||
| This partition holds the kubelet root directory, `/var/lib/kubelet/` by default, and `/var/log/` directory. This partition can be shared between user pods, the OS, and Kubernetes system daemons. This partition can be consumed by pods through `EmptyDir` volumes, container logs, image layers, and container-writable layers. Kubelet manages shared access and isolation of this partition. This partition is ephemeral, and applications cannot expect any performance SLAs, such as disk IOPS, from this partition. | ||
|
|
||
| == Runtime | ||
|
|
||
| This is an optional partition that runtimes can use for overlay file systems. {product-title} attempts to identify and provide shared access along with isolation to this partition. Container image layers and writable layers are stored here. If the runtime partition exists, the `root` partition does not hold any image layer or other writable storage. | ||
| * *Runtime*: This is an optional partition that runtimes can use for overlay file systems. {product-title} attempts to identify and provide shared access along with isolation to this partition. Container image layers and writable layers are stored here. If the runtime partition exists, the `root` partition does not hold any image layer or other writable storage. | ||
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -1,25 +1,31 @@ | ||||||
| :_mod-docs-content-type: ASSEMBLY | ||||||
| [id="understanding-ephemeral-storage"] | ||||||
| = Understanding ephemeral storage | ||||||
| include::_attributes/common-attributes.adoc[] | ||||||
| :toc: | ||||||
| :toc-title: | ||||||
| :context: understanding-ephemeral-storage | ||||||
|
|
||||||
| toc::[] | ||||||
|
Comment on lines
+4
to
-7
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not familiar with this construct I don't follow why it would change from I'm not saying I'm right, just that it made me scratch my head 😂 If you can point me at a resource about this I would appreciate it so that I understand 🤓 |
||||||
| include::_attributes/common-attributes.adoc[] | ||||||
|
|
||||||
| [role="_abstract"] | ||||||
| To manage temporary data that exists only for the duration of a workload, review the properties of ephemeral storage. | ||||||
| Ephemeral storage provides temporary per-pod storage for scratch data, caches, and logs that do not need to persist beyond the pod's lifetime. Understanding the different ephemeral storage types and their resource management helps you choose the right option for stateless workloads while preventing node storage exhaustion. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Removes anthrophomorphism.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. SMEs review might be needed for these abstracts if they are new. |
||||||
|
|
||||||
| toc::[] | ||||||
|
|
||||||
| include::modules/storage-ephemeral-storage-overview.adoc[leveloffset=+1] | ||||||
|
|
||||||
| include::modules/storage-ephemeral-storage-types.adoc[leveloffset=+1] | ||||||
|
|
||||||
| include::modules/storage-ephemeral-storage-manage.adoc[leveloffset=+1] | ||||||
| include::modules/storage-ephemeral-storage-manage-overview.adoc[leveloffset=+1] | ||||||
|
|
||||||
| include::modules/storage-ephemeral-storage-manage-requests-and-limits.adoc[leveloffset=+2] | ||||||
|
|
||||||
| ifndef::openshift-dedicated,openshift-rosa,openshift-rosa-hcp[] | ||||||
| [id="additional-resources_{context}"] | ||||||
| == Additional resources | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should the first add resources be of the format? [role="additional-resources"]
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is only for the assembly-level AR section that comes at the very end of the assembly. It is never valid at any other location. |
||||||
| [role="_additional-resources"] | ||||||
| .Additional resources | ||||||
|
|
||||||
| * xref:../applications/quotas/quotas-setting-per-project.adoc#quotas-setting-per-project_quotas-setting-per-project[Resources managed by quotas] | ||||||
| endif::openshift-dedicated,openshift-rosa,openshift-rosa-hcp[] | ||||||
|
|
||||||
| include::modules/storage-ephemeral-storage-manage-config-and-eviction.adoc[leveloffset=+2] | ||||||
|
|
||||||
| include::modules/storage-ephemeral-storage-monitoring.adoc[leveloffset=+1] | ||||||

There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Confused about how this stacks with modules/storage-ephemeral-storage-manage.adoc.
Ephemeral storage configuration effects pod scheduling and eviction:
In the old module, this section is entirely excluded for MS. In this new module, only the bullets are excluded.
Ephemeral storage configuration effects pod eviction:
This is included for MS in both the original module and this one.
Also echoing Darragh's question about inclusion of the module
I'm also a bit confused about what this gets around. IIUC, you were making this module because you had to get rid of the
==headings in the old module, is that right? But this module also adds an==heading anyway.