Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 63 additions & 2 deletions content/runbooks/node/NodeClockNotSynchronising.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,69 @@ Time is not automatically synchronizing on the node. This can cause issues with

## Diagnosis

TODO
1. **Confirm the alert condition**

```bash
timedatectl
```

Look for:

* `System clock synchronized: no`
* `NTP service: inactive`

2. **Check the time synchronisation service**
Most nodes use `chronyd`:

```bash
systemctl status chronyd
```

Common failure modes:

* Service is stopped or failed
* Service was OOM-killed
* No reachable NTP sources


## Mitigation

See [Node Clok Skew Detected]({{< ref "./NodeClockSkewDetected.md" >}}) for mitigation steps.
1. **Restart the time synchronisation service**
Restarting `chronyd` is safe on production nodes and does not restart workloads.

```bash
sudo systemctl restart chronyd
```

2. **Verify the service is running**

```bash
systemctl status chronyd
```

3. **Confirm clock synchronisation**

```bash
timedatectl
```

Expected:

* `System clock synchronized: yes`

4. **If the service was OOM-killed**
Investigate memory pressure on the node:

```bash
journalctl -k | grep -i oom
```

Consider:

* Reviewing workload memory limits
* Adding a small amount of swap as a safety buffer
* Investigating recent deploys or memory spikes

5. **If the issue persists**
Follow the mitigation steps in
[Node Clock Skew Detected](./NodeClockSkewDetected.md)