diff --git a/content/runbooks/node/NodeDiskIOQueueHigh.md b/content/runbooks/node/NodeDiskIOQueueHigh.md new file mode 100644 index 0000000..0badb84 --- /dev/null +++ b/content/runbooks/node/NodeDiskIOQueueHigh.md @@ -0,0 +1,153 @@ +--- +title: Node Disk IO Queue High +weight: 20 +--- + +# NodeDiskIOQueueHigh + +## Meaning + +The average disk I/O queue length (`aqu-sq`, equivalent to the iostat `aqu-sz` +value, or in Prometheus `rate(node_disk_io_time_weighted_seconds_total[1m])`) +on a block device has been above the warning threshold (typically `10`) for a +sustained period (e.g. 30 minutes). A persistently long request queue means +the kernel is issuing I/O faster than the device can service it. + +
+Full context + +`aqu-sq` is the time-weighted average number of I/O requests that were issued +to the device but have not yet completed (in-flight + queued). For most +SATA/SAS disks the hardware queue depth is 32; NVMe devices support up to +1024 per queue. A sustained value of `10` or more on a SATA/SAS disk +typically means the device is approaching saturation and request latency +(`await` / `r_await` / `w_await`) is climbing. + +The Prometheus equivalent of `aqu-sq` from `node_exporter` is: + +``` +rate(node_disk_io_time_weighted_seconds_total{device="",instance=""}[1m]) +``` + +> NOTE: The blank lines above and below the text inside this `
` tag are required to use markdown inside of html tags. + +
+ +## Impact + +Any process that issues I/O against the affected device experiences higher +latency. Common consequences: + +- Slow pod start-up (image pull, volume mount, container fs writes). +- kubelet / container runtime falling behind on writes (logs, cadvisor, + checkpoints) and possibly being marked `NotReady` if PLEG times out. +- etcd `fsync` / WAL latency spikes if etcd lives on this disk – which can + trigger leader elections and control-plane unavailability. +- Application-level timeouts for any workload doing synchronous writes + (databases, message brokers, log shippers). + +If the queue keeps growing, requests will eventually time out at the block +layer (`blk_update_request: I/O error`) and filesystems may remount +read-only. + +## Diagnosis + +Replace `` and `` with the values from the alert labels. + +1. Confirm the symptom and identify the device on the affected node: + + ``` + iostat -xz 2 5 + ``` + + Look for `aqu-sz` (or `aqu-sq`) above the threshold combined with high + `%util` (>90%) and rising `r_await` / `w_await`. + +2. Find the heaviest writers/readers: + + ``` + sudo iotop -oPa + sudo pidstat -d 2 5 + ``` + +3. Check whether the load comes from kernel writeback or a specific + process. Inspect dirty memory: + + ``` + grep -E 'Dirty|Writeback' /proc/meminfo + ``` + +4. From Prometheus, correlate the typical saturation signals over the + alert window: + + ``` + # Queue length (this alert) + rate(node_disk_io_time_weighted_seconds_total{device="",instance=""}[1m]) + + # Utilization (% of time the device was busy) + rate(node_disk_io_time_seconds_total{device="",instance=""}[1m]) + + # Average request latency (seconds per op) + rate(node_disk_read_time_seconds_total{device="",instance=""}[1m]) + / rate(node_disk_reads_completed_total{device="",instance=""}[1m]) + rate(node_disk_write_time_seconds_total{device="",instance=""}[1m]) + / rate(node_disk_writes_completed_total{device="",instance=""}[1m]) + + # IOPS / throughput + rate(node_disk_reads_completed_total{device="",instance=""}[1m]) + rate(node_disk_writes_completed_total{device="",instance=""}[1m]) + rate(node_disk_read_bytes_total{device="",instance=""}[1m]) + rate(node_disk_written_bytes_total{device="",instance=""}[1m]) + ``` + +5. Map the device to the Kubernetes workload: + + ``` + lsblk -f + findmnt /var/lib/kubelet + findmnt /var/lib/etcd + findmnt /var/lib/containerd + crictl ps -a + ``` + + Identify whether the device hosts `/var/lib/etcd`, kubelet pod volumes, + container image storage, or a PV. Then drain / reschedule the noisy + workload if appropriate. + +6. Check the underlying hardware / virtualization layer. Long queues + without a corresponding increase in IOPS often indicate degraded media + (failing SSD, RAID rebuild, noisy neighbour on a shared LUN): + + ``` + sudo smartctl -a /dev/ + sudo dmesg -T | grep -iE '|ata|nvme|i/o error' + ``` + +## Mitigation + +Pick the action(s) appropriate for what diagnosis showed: + +- **Noisy pod** – `kubectl cordon` the node and `kubectl drain` (or delete) + the offending pod so it is rescheduled elsewhere; lower its I/O via + `requests`/`limits` or cgroup tuning. +- **Backup / batch job** – throttle or reschedule it to off-peak hours + (e.g. via `ionice -c 3` or job-level rate limiting). +- **etcd on shared disk** – move etcd's data directory to a dedicated fast + disk; co-locating etcd with general workloads on a saturated spindle is + unsupported. +- **Failing device** – replace the disk; if it is part of a RAID array let + the array rebuild during a quiet window. +- **Tuning** – consider switching the I/O scheduler + (`/sys/block//queue/scheduler`) to `mq-deadline` (or `none` for + NVMe), increasing `nr_requests`, or enabling write caching if disabled. +- **Capacity** – if the workload legitimately needs more IOPS than the + device can provide, scale horizontally or migrate to faster storage + (NVMe / SSD / higher-tier cloud volume). + +After mitigation, verify the queue length returns below the threshold: + +``` +rate(node_disk_io_time_weighted_seconds_total{device="",instance=""}[1m]) < 10 +``` + +[1]: https://github.github.com/gfm/#html-block