SwarmMode shows agents as "down"

[SorraTheOrc]

I deployed a Swarm Mode cluster with two agents. When I do docker node ls I get:

ID                           HOSTNAME                     STATUS  AVAILABILITY  MANAGER STATUS
hckh3be59qj45k8dpcrrpibyy *  swarm-master-75773396-0      Ready   Pause         Leader
jhhsx11wymnhugckdsvilcw8p    swarm-public-75773396000001  Ready   Active
rqtoizc7of1hqv1d2tm7430ms    swarm-public-75773396000002  Down    Active
uwofxnqb9056a7iws58qjat5u    swarm-public-75773396000000  Ready   Active

I assume that the third worker was a result of overprovisioning and was removed after it had registered with the cluster.

Is there any way we can clean up after initialization?

/cc @amanohar

[anhowe]

some possible workarounds:

1. determine if there is a timeout for expiring old nodes, and submit a bug to docker if there isn't. They should build for a cloud where VMs are dynamic
2. not great, but potentially adding a 60s sleep before registering
3. we can disable overprovisioning

[SorraTheOrc]

-1 on disabling overprovisioning unless there is some mechanism to ensure the right number of agents is brought up.

I'm assuming #2 would result in a 60s increase in overall deployment time. I'm not keen on this.

Lets explore #1 at an appropriate time (I think we can live with this for a period)

[amanohar]

We shouldn't try to clean up after initialization in this case at least unless we know of way to determine if the node was down due to overprovisioning or if the node was down for another (unexpected) reason.

@rgardler, when you say clean up after initialization you mean that Swarm shouldn't show those down nodes at all?

I am not keen on #2 either as it adds to overall deployment time and is not deterministic. I have seen VMSS take more than 60s sometimes to remove overprovisioned nodes.

[SorraTheOrc]

If I ask for 2 workers and have 2 healthy and 1 unhealthy then it’s a good assumption that it’s overprovisioning. Even if it’s not I don’t care about a VM that I didn’t ask for not being active. I should never see it as “down”. Ross From: Anubhuti Manohar [mailto:notifications@github.com] Sent: Wednesday, January 25, 2017 11:04 AM To: Azure/acs-engine <acs-engine@noreply.github.com> Cc: Ross Gardler <Ross.Gardler@microsoft.com>; Mention <mention@noreply.github.com> Subject: Re: [Azure/acs-engine] SwarmMode shows agents as "down" (#215) We shouldn't try to clean up after initialization in this case at least unless we know of way to determine if the node was down due to overprovisioning or if the node was down for another (unexpected) reason. @rgardler<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Frgardler&data=02%7C01%7Cross.gardler%40microsoft.com%7C50710fe2872f4472a40808d44554fd9c%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636209678712293563&sdata=xujaNT6W3cdG5kNJBfIN8z%2F4B1Cn%2B3RPmHPPFkvUJGs%3D&reserved=0>, when you say clean up after initialization you mean that Swarm shouldn't show those down nodes at all? I am not keen on #2<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Facs-engine%2Fpull%2F2&data=02%7C01%7Cross.gardler%40microsoft.com%7C50710fe2872f4472a40808d44554fd9c%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636209678712293563&sdata=fP9XvDr%2BIWTxNoo291KpR3T3ONsBl0oC1UCtoYZhS3o%3D&reserved=0> either as it adds to overall deployment time and is not deterministic. I have seen VMSS take more than 60s sometimes to remove overprovisioned nodes. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Facs-engine%2Fissues%2F215%23issuecomment-275201102&data=02%7C01%7Cross.gardler%40microsoft.com%7C50710fe2872f4472a40808d44554fd9c%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636209678712293563&sdata=JNqmFuqvz7ooeftgQ5B72WWWb1%2FIv0S7TMRhVxn3T6Q%3D&reserved=0>, or mute the thread<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAPRgAX5O1ZI717ykWCdhtD8T_jHcRuxks5rV5ylgaJpZM4LtRjg&data=02%7C01%7Cross.gardler%40microsoft.com%7C50710fe2872f4472a40808d44554fd9c%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636209678712293563&sdata=ggWSi42Et04QzrzCHOB%2BMY%2B4ckS9hBRIGcEirkLnI2o%3D&reserved=0>.

[amanohar]

Swarm Mode does something similar in case of scale down as well. If you scale down, it still show the node with status as 'Down' and availability as 'Active' which seems incorrect/confusing.

I will leave my cluster running the whole day today and see if it removes the node at some point. I have tried it for 30 mins and it didn't seem to be remove the node. If it doesn't remove the node we can follow up with Docker.

[amanohar]

Update: I created a cluster with 3 agent nodes. I scaled down to 2 and left the cluster running for 5+ hours. Swarm Mode still show the deleted node when doing "docker node ls" with status = down, availability = active.

I will follow up with Docker on this.

[anhowe]

Dropping to P2, since it is not clear that the downed node is causing scheduling issues and more a display issue from the CLI point of view. If this is causing scheduling issues, let's raise back to P1.

[anhowe]

This also affects scale-down. This is not great UI wise, however, this is an upstream issue, and does not affect scheduling of containers.

Please re-open if this causes container scheduling issues.

[artisticcheese]

Can somebody explain to me how scaling up/down and reimaging of the the VM scale sets reflects on swarm nodes. I can see that provisioning script for hybrid Windows container deployment in ACS have all the neccessary plumbing for adding nodes to swarm. But what happens when you reimage for example individual VM scale set machine, or you can scale up and then scale down worker nodes. There is no clean up script provided anywhere. Does it mean that swarm will end up with dead members forever?