From 6b35a6bbb09046f82cdc5f652ded094c384d3272 Mon Sep 17 00:00:00 2001 From: gurpalw <37112979+gurpalw@users.noreply.github.com> Date: Mon, 12 Jan 2026 22:21:38 +0000 Subject: [PATCH 1/2] Update NodeClockNotSynchronising documentation Added detailed diagnosis and mitigation steps for node clock synchronization issues. --- .../node/NodeClockNotSynchronising.md | 67 ++++++++++++++++++- 1 file changed, 64 insertions(+), 3 deletions(-) diff --git a/content/runbooks/node/NodeClockNotSynchronising.md b/content/runbooks/node/NodeClockNotSynchronising.md index 8370926..f699392 100644 --- a/content/runbooks/node/NodeClockNotSynchronising.md +++ b/content/runbooks/node/NodeClockNotSynchronising.md @@ -11,12 +11,73 @@ Clock not synchronising. ## Impact -Time is not automatically synchronizing on the node. This can cause issues with handling TLS as well as problems with other time-sensitive applications. +Time is skewed on the node. This can cause issues with handling TLS as well as problems with other time-sensitive applications. ## Diagnosis -TODO +1. **Confirm the alert condition** + + ```bash + timedatectl + ``` + + Look for: + + * `System clock synchronized: no` + * `NTP service: inactive` + +2. **Check the time synchronisation service** + Most nodes use `chronyd`: + + ```bash + systemctl status chronyd + ``` + + Common failure modes: + + * Service is stopped or failed + * Service was OOM-killed + * No reachable NTP sources + ## Mitigation -See [Node Clok Skew Detected]({{< ref "./NodeClockSkewDetected.md" >}}) for mitigation steps. +1. **Restart the time synchronisation service** + Restarting `chronyd` is safe on production nodes and does not restart workloads. + + ```bash + sudo systemctl restart chronyd + ``` + +2. **Verify the service is running** + + ```bash + systemctl status chronyd + ``` + +3. **Confirm clock synchronisation** + + ```bash + timedatectl + ``` + + Expected: + + * `System clock synchronized: yes` + +4. **If the service was OOM-killed** + Investigate memory pressure on the node: + + ```bash + journalctl -k | grep -i oom + ``` + + Consider: + + * Reviewing workload memory limits + * Adding a small amount of swap as a safety buffer + * Investigating recent deploys or memory spikes + +5. **If the issue persists** + Follow the mitigation steps in + [Node Clock Skew Detected](./NodeClockSkewDetected.md) From 414e82033fd0a7acba5a934c20e62c318381e714 Mon Sep 17 00:00:00 2001 From: gurpalw <37112979+gurpalw@users.noreply.github.com> Date: Mon, 12 Jan 2026 22:23:01 +0000 Subject: [PATCH 2/2] Clarify description of clock synchronization issue --- content/runbooks/node/NodeClockNotSynchronising.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/runbooks/node/NodeClockNotSynchronising.md b/content/runbooks/node/NodeClockNotSynchronising.md index f699392..0718aeb 100644 --- a/content/runbooks/node/NodeClockNotSynchronising.md +++ b/content/runbooks/node/NodeClockNotSynchronising.md @@ -11,7 +11,7 @@ Clock not synchronising. ## Impact -Time is skewed on the node. This can cause issues with handling TLS as well as problems with other time-sensitive applications. +Time is not automatically synchronizing on the node. This can cause issues with handling TLS as well as problems with other time-sensitive applications. ## Diagnosis