Upgrade process for kubernets is not working (acs-engine not ready for production workload)

[jalberto]

Is this a request for help?:
YES & a BUG REPORT

Is this an ISSUE or FEATURE REQUEST? (choose one):
ISSUE

What version of acs-engine?:
0.14.5

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)
kubernetes 1.8.4 to 1.8.10

What happened:
run `upgrade command following #2062 have lots of troubles:

• the process failed 16 times, it finally was successful on 17
• every problem described in upgrade k8s process is broke #2022 continue happening (months after being reported)
• my exisint gcluter use a custom vnet with peering, it need to be delete, then upgrade then recreated
• etcd install script still try to install 2.5.2, I modify it manually to isntall 2.3.7 (latest 2.x release)
• after etcd finally installed, and masters rebooted still not connectivity
• each upgraded with acs-engine is an adventure and not suitable for production workflows
• now etcd is running but yet not connectivity to cluster using kubectl

What you expected to happen:
to work

How to reproduce it (as minimally and precisely as possible):
just try to upgrade an existing cluster

Anything else we need to know:
this is really critical, as my prod cluster is down right now

[jalberto]

maybe related:

• Latest acs-engine deployed k8s cluster not working #2124
• Breaking change to upgrade in v0.12.0 #2062

[jalberto]

Is it possible to run manually cloud-init to recreate what is missing?

[jalberto]

etcd logs:

Mar 30 16:32:06 k8s-master-11577755-0 etcd[48953]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 4430                                          
Mar 30 16:32:06 k8s-master-11577755-0 etcd[48953]: 134af6e5d3e35861 [logterm: 1857, index: 265391380] sent vote request to 2a0965170762fd09 at term 4430      
Mar 30 16:32:06 k8s-master-11577755-0 etcd[48953]: 134af6e5d3e35861 [logterm: 1857, index: 265391380] sent vote request to 91d25258d70ad868 at term 4430      
Mar 30 16:32:07 k8s-master-11577755-0 etcd[48953]: publish error: etcdserver: request timed out                                                               
Mar 30 16:32:08 k8s-master-11577755-0 etcd[48953]: 134af6e5d3e35861 is starting a new election at term 4430                                                   
Mar 30 16:32:08 k8s-master-11577755-0 etcd[48953]: 134af6e5d3e35861 became candidate at term 4431                                                             
Mar 30 16:32:08 k8s-master-11577755-0 etcd[48953]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 4431                                          
Mar 30 16:32:08 k8s-master-11577755-0 etcd[48953]: 134af6e5d3e35861 [logterm: 1857, index: 265391380] sent vote request to 2a0965170762fd09 at term 4431      
Mar 30 16:32:08 k8s-master-11577755-0 etcd[48953]: 134af6e5d3e35861 [logterm: 1857, index: 265391380] sent vote request to 91d25258d70ad868 at term 4431      
Mar 30 16:32:09 k8s-master-11577755-0 etcd[48953]: 134af6e5d3e35861 is starting a new election at term 4431                                                   
Mar 30 16:32:09 k8s-master-11577755-0 etcd[48953]: 134af6e5d3e35861 became candidate at term 4432                                                             
Mar 30 16:32:09 k8s-master-11577755-0 etcd[48953]: 134af6e5d3e35861 received vote from 134af6e5d3e35861 at term 4432                                          
Mar 30 16:32:09 k8s-master-11577755-0 etcd[48953]: 134af6e5d3e35861 [logterm: 1857, index: 265391380] sent vote request to 2a0965170762fd09 at term 4432      
Mar 30 16:32:09 k8s-master-11577755-0 etcd[48953]: 134af6e5d3e35861 [logterm: 1857, index: 265391380] sent vote request to 91d25258d70ad868 at term 4432

kubelet logs:

Mar 30 17:08:49 k8s-master-11577755-0 systemd[1]: Started Kubelet.
Mar 30 17:08:50 k8s-master-11577755-0 docker[5567]: Flag --non-masquerade-cidr has been deprecated, will be removed in a future version
Mar 30 17:08:50 k8s-master-11577755-0 docker[5567]: Flag --keep-terminated-pod-volumes has been deprecated, will be removed in a future version
Mar 30 17:08:50 k8s-master-11577755-0 docker[5567]: Flag --non-masquerade-cidr has been deprecated, will be removed in a future version
Mar 30 17:08:50 k8s-master-11577755-0 docker[5567]: I0330 17:08:50.064943    5595 feature_gate.go:162] feature gates: map[]
Mar 30 17:08:50 k8s-master-11577755-0 docker[5567]: I0330 17:08:50.065489    5595 mount_linux.go:196] Detected OS without systemd
Mar 30 17:08:50 k8s-master-11577755-0 docker[5567]: I0330 17:08:50.065523    5595 client.go:75] Connecting to docker on unix:///var/run/docker.sock
Mar 30 17:08:50 k8s-master-11577755-0 docker[5567]: I0330 17:08:50.065572    5595 client.go:95] Start docker client with request timeout=2m0s
Mar 30 17:08:50 k8s-master-11577755-0 docker[5567]: W0330 17:08:50.066686    5595 cni.go:196] Unable to update cni config: No networks found in /etc/cni/net.d
Mar 30 17:08:50 k8s-master-11577755-0 docker[5567]: Error: failed to run Kubelet: could not init cloud  not init cloud provider "azure": No credentials provided for AAD applicationcredentials provided for AAD appliad units

[jalberto]

so I change any reference to "2.5.2" to "2.3.7" in my _output dir, and try to upgrade to k8s 1.9, and surprise, etcd 2.5.2 still trying to be installed in master

[jackfrancis]

Hi @jalberto thanks for your courage. What's the first thing we can look at?

[jalberto]

@jackfrancis thanks for your time

IMHO:

1. don't mark a VM as "upgraded" is any component is not 100% working (so we can repeat the command until working)
2. find why acs-engine insist in installing 2.5.2 (that version doesn't exist, latest in 2.x series is 2.3.7)
3. figure out how a "reset" can be forced in a node even if it's already upgraded (to mitigate 1)

[jackfrancis]

#1 and #3 sound like longer-term improvements to the upgrade implementation, so thanks for your patience while those are not in place (and the burden remains on the user to reconcile cluster state).

Let's figure out why the 2.5.2 etcd bug is still present, that was fixed a while ago. What does "etcd 2.5.2 still trying to be installed in master" mean exactly?

[jalberto]

when running the upgrade command in a working cluster with etcd 2.3.7 new acs-engine crates the file /opt/azure/containers/setup-etcd.sh with this content:

#!/bin/bash
set -x
source /opt/azure/containers/provision_source.sh
ETCD_VER=v2.5.2
DOWNLOAD_URL=https://acs-mirror.azureedge.net/github-coreos
retrycmd_if_failure 5 5 curl --retry 5 --retry-delay 10 --retry-max-time 30 -L ${DOWNLOAD_URL}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /usr/bin/ --strip-components=1
systemctl daemon-reload
systemctl enable etcd.service
sudo sed -i "1iETCDCTL_ENDPOINTS=https://127.0.0.1:2379" /etc/environment
sudo sed -i "1iETCDCTL_CA_FILE=/etc/kubernetes/certs/ca.crt" /etc/environment
sudo sed -i "1iETCDCTL_KEY_FILE=/etc/kubernetes/certs/etcdclient.key" /etc/environment
sudo sed -i "1iETCDCTL_CERT_FILE=/etc/kubernetes/certs/etcdclient.crt" /etc/environment

clearly the curl command faills to fetch that version, but the script continues running to end so acs-engine it was successful.

@jackfrancis is there a way to run a commadn in master to recreate the initial provisioning commands? (previous remove of /opt/azure/containers/*.complete files)

Why? becasue if manually I go into each amster, and change that value, and re-run setup_etcd.sh successfully, the global state of master is inconsistent (as etcd wassn't ready to continue some key steps in the process)

[jalberto]

BTW point 1 can be achived just exiting of any script if there is any error, at least that will stop the process

[jackfrancis]

There is no way to recreate an original provisioning, no. The only semi-notion of cluster "state" (quotations intentional) lives in the api model on the client side, which as you've discovered is only a fractional representation of the actual cluster, especially w/ respect to the original api model representation vs a newer version of acs-engine.

The ETCD_VER=v2.5.2 derives from the value of etcdVersion, in the kubernetesConfig at the time of template generation. E.g., in your api model:

<etc>
    "kubernetesConfig" {
        "etcdVersion": "3.2.16"
    }
<etc>

[jalberto]

@jackfrancis this can be related: kubernetes/kubernetes#54918
I am using calico and no fiel is being created in etc/cni how can I install manually?

[jalberto]

@jackfrancis I understand it should come from there, but before I run upgrade, I edited my _output/foo/apimodel.json to fix etcd version.

an suggestion? my cluster is down for so many hours now

[jalberto]

@jackfrancis more context:

root@k8s-master-11577755-0:~# docker ps --format '{{.Image}} - {{.Names}}'
gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.9.5 - wizardly_bartik
gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:a31961a719a1d0ade89149a6a8db5181cbef461baa6ef049681c31c0e48d9f1e - k8s_kube-controller-manager_kube-controller-manager-k8s-master-11577755-0_kube-system_beaaf22644028e3842cf0847ccb58d15_1
k8s-gcrio.azureedge.net/kube-addon-manager-amd64@sha256:3519273916ba45cfc9b318448d4629819cb5fbccbb0822cce054dd8c1f68cb60 - k8s_kube-addon-manager_kube-addon-manager-k8s-master-11577755-0_kube-system_61d4fa32deceb6175822bf42fb7410f2_1
gcrio.azureedge.net/google_containers/hyperkube-amd64@sha256:a31961a719a1d0ade89149a6a8db5181cbef461baa6ef049681c31c0e48d9f1e - k8s_kube-scheduler_kube-scheduler-k8s-master-11577755-0_kube-system_27fb8458832c33c3b8754aca44f00158_1
k8s-gcrio.azureedge.net/pause-amd64:3.1 - k8s_POD_kube-controller-manager-k8s-master-11577755-0_kube-system_beaaf22644028e3842cf0847ccb58d15_1
k8s-gcrio.azureedge.net/pause-amd64:3.1 - k8s_POD_kube-apiserver-k8s-master-11577755-0_kube-system_a0596cf3432e0574f040528197bc3441_1
k8s-gcrio.azureedge.net/pause-amd64:3.1 - k8s_POD_kube-addon-manager-k8s-master-11577755-0_kube-system_61d4fa32deceb6175822bf42fb7410f2_1
k8s-gcrio.azureedge.net/pause-amd64:3.1 - k8s_POD_kube-scheduler-k8s-master-11577755-0_kube-system_27fb8458832c33c3b8754aca44f00158_1

I0330 19:08:21.516738   35291 kubelet.go:316] Watching apiserver
E0330 19:08:21.517265   35291 file.go:149] Can't process manifest file "/etc/kubernetes/manifests/audit-policy.yaml": /etc/kubernetes/manifests/audit-policy.yaml: couldn't parse as pod(no kind "Policy" is registered for version "audit.k8s.io/v1beta1"), please check manifest file.
E0330 19:08:21.545031   35291 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.240.255.5:443/api/v1/pods?fieldSelector=spec.nodeName%3Dk8s-master-11577755-0&limit=500&resourceVersion=0: dial tcp 10.240.255.5:443: getsockopt: connection refused
E0330 19:08:21.545778   35291 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:480: Failed to list *v1.Node: Get https://10.240.255.5:443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-master-11577755-0&limit=500&resourceVersion=0: dial tcp 10.240.255.5:443: getsockopt: connection refused
E0330 19:08:21.546344   35291 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:471: Failed to list *v1.Service: Get https://10.240.255.5:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.240.255.5:443: getsockopt: connection refused
W0330 19:08:21.569904   35291 kubelet_network.go:139] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
I0330 19:08:21.569952   35291 kubelet.go:577] Hairpin mode set to "hairpin-veth"
W0330 19:08:21.570207   35291 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
I0330 19:08:21.570229   35291 plugins.go:190] Loaded network plugin "cni"
I0330 19:08:21.570291   35291 client.go:80] Connecting to docker on unix:///var/run/docker.sock
I0330 19:08:21.570305   35291 client.go:109] Start docker client with request timeout=2m0s
W0330 19:08:21.572091   35291 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
W0330 19:08:21.575640   35291 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
I0330 19:08:21.575711   35291 plugins.go:190] Loaded network plugin "cni"
I0330 19:08:21.575838   35291 docker_service.go:232] Docker cri networking managed by cni