[kubernetes] unable to create cluster with custom vnet

[jpoon]

What happened:

Creating a k8s cluster using an existing vnet, the cluster is unable to create routes in the Azure Route table, and is therefore unable to schedule any pods.

How to reproduce it:

1. Create a custom vnet
2. Configure the template and deploy

When the cluster is up, the nodes report as ready:

gfadmin@k8s-master-35738843-0:~$ kubectl get nodes
NAME                        STATUS                     AGE
k8s-agentpool1-35738843-0   Ready                      16h
k8s-agentpool1-35738843-1   Ready                      16h
k8s-agentpool1-35738843-2   Ready                      16h
k8s-master-35738843-0       Ready,SchedulingDisabled   16h

Wtih NetworkUnavailable message of RouteController failed tocreate a route:

gfadmin@k8s-master-35738843-0:~$ kubectl describe node k8s-master-35738843-0
Name:                   k8s-master-35738843-0
Labels:                 beta.kubernetes.io/arch=amd64
                        beta.kubernetes.io/instance-type=Standard_D2_v2
                        beta.kubernetes.io/os=linux
                        failure-domain.beta.kubernetes.io/region=westus
                        failure-domain.beta.kubernetes.io/zone=0
                        kubernetes.io/hostname=k8s-master-35738843-0
Taints:                 <none>
CreationTimestamp:      Wed, 23 Nov 2016 18:40:52 +0000
Phase:
Conditions:
  Type                  Status  LastHeartbeatTime                       LastTransitionTime                        Reason                          Message
  ----                  ------  -----------------                       ------------------                        ------                          -------
  OutOfDisk             False   Thu, 24 Nov 2016 11:02:41 +0000         Wed, 23 Nov 2016 18:40:52 +0000   KubeletHasSufficientDisk        kubelet has sufficient disk space available
  MemoryPressure        False   Thu, 24 Nov 2016 11:02:41 +0000         Wed, 23 Nov 2016 18:40:52 +0000   KubeletHasSufficientMemory      kubelet has sufficient memory available
  DiskPressure          False   Thu, 24 Nov 2016 11:02:41 +0000         Wed, 23 Nov 2016 18:40:52 +0000   KubeletHasNoDiskPressure        kubelet has no disk pressure
  Ready                 True    Thu, 24 Nov 2016 11:02:41 +0000         Wed, 23 Nov 2016 18:40:52 +0000   KubeletReady                    kubelet is posting ready status
  NetworkUnavailable    True    Thu, 24 Nov 2016 11:02:47 +0000         Thu, 24 Nov 2016 11:02:47 +0000   NoRouteCreated                  RouteController failed tocreate a route

Looking at the kube-controller logs (/var/log/containers):

routecontroller.go:132] Could not create route 5cb8901d-b1ac-11e6-89eb-000d3a32ff9f 10.244.2.0/24 for node k8s-master-35738843-0 after 38.691596ms: network.SubnetsClient#Get: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code=\"ResourceNotFound\" Message=\"The Resource 'Microsoft.Network/virtualNetworks/subscriptions' under resource group 'ACSRG2' was not found.\"\n","stream":"stderr","time":"2016-11-23T18:51:29.914307462Z"}

Notice the error message has an malform resource: Microsoft.Network/virtualNetworks/subscriptions.

Workaround

We've deduced this to the /etc/kubernetes/azure.json expecting unqualified names for both the vnet and subnet. Instead, the fully-qualified names are present:

{
    ...
    "subnetName": "/subscriptions/76aabf62-fa6e-41ac-a2f3-5532b22811b5/resourceGroups/ACSRG2/providers/Microsoft.Network/virtualNetworks/k8s-vnet-test/subnets/k8s-subnet-test",
    "securityGroupName": "...",
    "vnetName": "/subscriptions/76aabf62-fa6e-41ac-a2f3-5532b22811b5/resourceGroups/ACSRG2/providers/Microsoft.Network/virtualNetworks/k8s-vnet-test",
    ...
}

After changing the subnet and vnet to unqualified names and restarting kubelet, we see the routes as being created and things are back to normal.

Much of the credit in debugging this goes to @jamesbak.

[jpoon]

cc @colemickens. I can provide private keys to help debug, but it should be fairly easy to get a repro

[colemickens]

Problem is here: https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/azure/azure_routes.go#L90

[colemickens]

We go straight to the route table that is listed in the config file, but we also check the subnet to see if it's properly configured.

Options:

1. Skip the subnet check
2. Take a special vnetResourceGroup that can override. (Can you reference a vnet in a different sub? If so, also need `vnetSubscrpitionId)
3. Start versioning the config with a nested struct, or two-pass decode, and then start using full identifiers everywhere. This means more util functions to rip the identifiers apart since the SDK APIs address resources by individual, separate strings of the inner identifiers.

I think #1 might possibly be the right thing to do, depending on if we can support multiple subnets of machines with same route table. If we can, then subnetName is sort of meaningless. We really just care about configuring the appropriate route table in the Routes implementation. At some point we have to expect the user set things up correctly.

CC: @brendandburns for any thoughts.

[colemickens]

This presents another question though, where does the route table live. Need to find out if the route table for the subnet in the existing vnet can live in both resource groups, only one, etc. If it can live in the existing-vnet's resource group, then we need to support the full identifier string for the routeTable field anyway...

[mogthesprog]

Just came back here to report the same issue (finally got around to looking at it from #99, sorry for the delay). My two cents would be that full Resource IDs feels like it would be the azure idiomatic way of declaring resources, makes me wonder if point 3 of yours makes more sense here?

[jpoon]

Thanks @colemickens. Would it not be easiest to modify the azure.json that the ACS-engine generates and puts on the master node? (ie. the workaround that I mentioned above)

I think this is what you meant by option 3. As deploying Kubernetes under a custom VNET never worked, there would be no need to start versioning this config......yet.

[colemickens]

@jpoon How would that help anything? The problem is that the code assumes that all resources are in the same resource group specified in the config file.

[jpoon]

In our case, everything is under the same resource group. Are there situations where people would deploy a VNET under a separate resource group?

[colemickens]

I had assumed as much, but I don't actually know that as a solid fact. Is that something you would have data on?

@rgardler or @sauryadas for customers who want to deploy clusters (particularly Kubernetes) into existing vnets... are they generally putting the cluster into the existing resource group, or are the existing vnet and new cluster typically in different resource groups?

[jpoon]

As custom vnets don't work at all, would it be reasonable to do a quick fix to support things in the same RG?

[colemickens]

Due to the fact that the apimodel takes vnetSubnetID as the full identifier string (which I agree with), it means that for this "quick fix" we need two template functions - one to extract the vnet name, and another to extract the subnet name.

So that in the template we can write {{ subnetNameFromId .VnetSubnetID }} (very pseudocode-y).

PRs are very welcome, I'm not going to be able to get to this for a while.

[colemickens]

I have a branch here that might fix this issue for deployments into a single RG. I don't have an easy way of testing it. Is anyone here willing to give it a shot? https://github.com/colemickens/acs-engine/tree/colemickens-pr-fix-custom-vnet

[lmickh]

@colemickens I tested the branch with a config very similar to the custom vnet example. New single RG, new vnet, and so on. Basically just changed the vnet and RG names. The result after applying the templates was the same as before. Both vnetName and subnetName are fully qualified in /etc/kubernetes/azure.json. Not sure why that is the case.

[colemickens]

I think I fixed it. Could I get you to pull, rebuild and try again? Thanks so much.