Should setup container log retention policy on master & agent node

[yangl900]

Problem

Master node stops working for everything and claim etcd failure or it is not configured correctly.

Root Cause

Looking into logs we found that the master node container log has taken 27 GB and used all disk space in OS disk. The cluster has been running for ~2 months.

Proposed Fix

Setup log retention policy so we never run out of disk space.

[colemickens]

@yangl900 Ah, were the huge files: /var/lib/docker/containers/[CONTAINER ID]/[CONTAINER_ID]-json.log?

We should mitigate this short term with some sort of log rotations, long term presumably we'd want to run fluentd w/ logrotate and push cluster logs to some storage system...

[yangl900]

That's correct. Note that the file is holding by the running container, we had to restart in order to remove the file completely.

@brendandburns @anhowe as FYI, I think we should take this as a high priority fix. we cannot run cluster in production without the rotation.

[brendandburns]

Yes, we should add this ASAP. The 'logrotate' tool is a good simple tool to add.

…

--brendan

________________________________ From: Anders Liu <notifications@github.com> Sent: Thursday, March 2, 2017 12:17:25 PM To: Azure/acs-engine Cc: Brendan Burns; Mention Subject: Re: [Azure/acs-engine] Should setup container log retention policy on master & agent node (#350) That's correct. Note that the file is holding by the running container, we had to restart in order to remove the file completely. @brendandburns<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fbrendandburns&data=02%7C01%7Cbburns%40microsoft.com%7Cb80d9973c7604c62fd3d08d461a92609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636240826491757905&sdata=Oo7uJIb3UkPJjfbSScE5medZvJdJXUL6Kqoie75MaqU%3D&reserved=0> @anhowe<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fanhowe&data=02%7C01%7Cbburns%40microsoft.com%7Cb80d9973c7604c62fd3d08d461a92609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636240826491757905&sdata=IeDtbZrUKOsF6hZ51RVW8Vm1EectkporDpwwJ%2FdpCPE%3D&reserved=0> as FYI, I think we should take this as a high priority fix. we cannot run cluster in production without the rotation. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Facs-engine%2Fissues%2F350%23issuecomment-283767572&data=02%7C01%7Cbburns%40microsoft.com%7Cb80d9973c7604c62fd3d08d461a92609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636240826491767909&sdata=dOYA82angbyOKIIcC1F%2FppFrZUDiwONGQqkFpWFqV8Y%3D&reserved=0>, or mute the thread<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAFfDgqoj3Yvnlvg2oFXWbvsOg4NvJQo2ks5rhyPVgaJpZM4MReEw&data=02%7C01%7Cbburns%40microsoft.com%7Cb80d9973c7604c62fd3d08d461a92609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636240826491767909&sdata=KawgbZUV8C9YcIPEr8iPk6qIfCmAkR38kbYBIsK7ibU%3D&reserved=0>.

[rjtsdl]

I am looking into this problem.

Currently, the reason that docker daemon created multiple /var/lib/docker/containers/[CONTAINER ID]/[CONTAINER_ID]-json.log files, is because, docker daemon is using the default log-driver : json-file.

Now i have two options for the short term purpose:

1. Keep using json-file log-driver for docker, then we can leverage docker's option like max-size to limit the size.

2. Change log-driver to journald, in this way, we can leverage journald's log rotation strategy to manage docker along with other daemons.

I personally prefer option 2. However, i may overlook some issues, such as backward compatibility.

[colemickens]

I like option 2 as well, and I doubt we're going to end up supporting any distros that don't use systemd anyway.

[tekmaven]

This may become a problem on Windows Agents I'm assuming, so that needs to be considered.

[rjtsdl]

PR #373

@anhowe @JiangtianLi , how do you guys think about Windows Agents concern?

[colemickens]

It'd be one thing if if the Windows Service was harvesting the output of kubelet on windows, but instead we're writing out to a text file manually. Does NSSM take care of pumping stdout/err to Windows logs? If so, that might be easiest.

[JiangtianLi]

@rjtsdl @colemickens The issue that is solved here is docker/container logs, not kubelet logs, right? What is kubelet log retention policy on Linux? A quick look at NSSM doc and it has some rotation parameters.

[colemickens]

kubelet runs as a container in Linux, so it's solved by this fix as well. Otherwise it would run as a bare systemd unit, which also already has log rotation. Hence my remarks about WinService logs + NSSM. I just don't know Windows well enough to know what the convention is.

[anhowe]

In Windows, log rotation is handled in the NSSM setup here: https://github.com/Azure/acs-engine/blob/master/parts/kuberneteswindowssetup.ps1#L232-L270

[rjtsdl]

Just merged the PR. I will close the issue here.

For existing clusters, try to do modification showed in PR #373
After that, do

sudo rm -rf /var/lib/cloud/instances
sudo cloud-init -d init
sudo cloud-init -d modules -m config
sudo cloud-init -d modules -m final
sudo systemctl restart kubelet