We tried to deploy a k8s ACS cluster in a custom corporate vnet. The deployment was always successful, but the result did not work. Neither docker nor kubectl is installed on the machines. After a short glance into the azuredeploy.json file i have the following theory.
There is a number of regular Ubuntu VMs deployed and then on every machine a script is executed, which has among others the following steps:
- apt-get update
- apt-get install -y apt-transport-https ca-certificates
- apt-key adv --keyserver hkp://ha.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
- apt-get update
- apt-get install -y docker-engine
- /usr/bin/curl -sSL --retry 12 --retry-delay 10 https://storage.googleapis.com/kubernetes-release/release/',variables('kubectlVersion'),'/bin/linux/amd64/kubectl > /usr/local/bin/kubectl
Note that #1-5 are part of the standard way to install Docker on Linux.
If access to the public internet is only possible via a proxy server every single command runs in a time-out.
To configure a Ubuntu VM to use a proxy the following lines should be added to the /etc/profile file:
export http_proxy=<PROXY_SERVER>
export https_proxy=<PROXY_SERVER>
export HTTP_PROXY=$http_proxy
export HTTPS_PROXY=$https_proxy
Furthermore often corporate proxy servers do not support the hkp protocol, which is needed in Command 3 to load the docker GPG key. This should be replaced by providing the key as file and executing "apt-key add -f key.gpg"
Probably also some adaptions to the kubernetes configuration are necessary, but I have no experience with this. Adapting the /etc/profile file was sufficient to enable a proxy in standard ACS kubernetes deployment in the public internet. But in a version where we peered a deployment into a custom vnet, this did not suffice. Here the IP adress is a interal 10.a.b.c address and is not reachable from a pod. However, the proxy is reachable from the agents...
At least a short validation of the deployment should be included. Currently even a basic command like "kubeclt version" fails, but the deployment is marked as successful.
Some part of the used azuredeploy.json is attached. If requested I can provide the whole file.
part_azuredeploy.txt
We tried to deploy a k8s ACS cluster in a custom corporate vnet. The deployment was always successful, but the result did not work. Neither docker nor kubectl is installed on the machines. After a short glance into the azuredeploy.json file i have the following theory.
There is a number of regular Ubuntu VMs deployed and then on every machine a script is executed, which has among others the following steps:
Note that #1-5 are part of the standard way to install Docker on Linux.
If access to the public internet is only possible via a proxy server every single command runs in a time-out.
To configure a Ubuntu VM to use a proxy the following lines should be added to the /etc/profile file:
export http_proxy=<PROXY_SERVER>
export https_proxy=<PROXY_SERVER>
export HTTP_PROXY=$http_proxy
export HTTPS_PROXY=$https_proxy
Furthermore often corporate proxy servers do not support the hkp protocol, which is needed in Command 3 to load the docker GPG key. This should be replaced by providing the key as file and executing "apt-key add -f key.gpg"
Probably also some adaptions to the kubernetes configuration are necessary, but I have no experience with this. Adapting the /etc/profile file was sufficient to enable a proxy in standard ACS kubernetes deployment in the public internet. But in a version where we peered a deployment into a custom vnet, this did not suffice. Here the IP adress is a interal 10.a.b.c address and is not reachable from a pod. However, the proxy is reachable from the agents...
At least a short validation of the deployment should be included. Currently even a basic command like "kubeclt version" fails, but the deployment is marked as successful.
Some part of the used azuredeploy.json is attached. If requested I can provide the whole file.
part_azuredeploy.txt