Extend API Model to offer choice of OS for Agent and Master Nodes

[uday31in]

Is this a request for help?:

No

Is this an ISSUE or FEATURE REQUEST? (choose one):

Feature Request - #1088

What version of acs-engine?:

Current

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)

Kubernetes

Request:

Allows OS selection at Agent/Master Pool level to use acs-engine for OS other than Ubuntu. For Example.

@JackQuincy @anhowe @amanohar

"agentPoolProfiles": [{
				"name": "coreos",
				"count": 1,
				"vmSize": "Standard_DS2_v2",
				"availabilityProfile": "AvailabilitySet",
				"osImageOffer": "CoreOS",
				"osImagePublisher": "CoreOS",
				"osImageSKU": "Stable",
				"osImageVersion": "latest"

			},
			{
				"name": "ubuntu",
				"count": 1,
				"vmSize": "Standard_DS2_v2",
				"availabilityProfile": "AvailabilitySet"

			}

Anything else we need to know:
#1088

[amanohar]

@uday31in, Also please remove: offer/sku/version/publisher from the API. Those properties will be chosen based on is os type (API model doesn't allow customer to choose these for Ubuntu as well).

[uday31in]

@amanohar - This is essential to this request. I am proposing we should let user specify.

For Example, if we just use "OS" as "CoreOS" - then how and where do we specify whether to use
CoreOS:CoreOS:Stable:latest or CoreOS:CoreOS:Beta:latest?

We should let user make the configuration choice. My proposal is, "batteries included but removable" . as per #1088 . We are using UbuntuServer:Canonical:16.04-LTS:16.04.201706191 as default.

[amanohar]

I am not certain about each agent pool to be able choose a different OS type + offer/sku/version combination. This introduces too many variables in cluster making it hard to debug/explodes the test matrix/verifying any errors reported hard. Plus I am not clear about the value add of this use case.

If choosing OS image is indeed essential I propose moving osType and osImage (containing offer/sku/version/publisher fields) to linuxProfile and have the master and all (Linux) agent pools use the same os version. Providing a lot of nobs for configuration is not necessarily a powerful thing in all cases.

[amanohar]

@uday31in, if OS offer, sku, version is indeed a requirement here's my proposal:

"agentPoolProfiles": [{
    "name": "agentpool1
    "count": 1,
    "vmSize": "Standard_DS2_v2",
    "availabilityProfile": "AvailabilitySet",
    "imageReference": {
                "offer": " CoreOS ",
                "publisher": "CoreOS",
                "sku": “Stable",
                "version": “latest”
            }
}]

[anhowe]

Allowing specification of pub/offer/sku/version can cause an explosion of support issues. This is a similar reason why we chose to pull kubernetesHyperkubeSpec into parameters as opposed to the API model.

How about this:

1. pick the successfully working SKU, and have that used by default, and this is the one we tests against to provide confidence of solution.
2. pull pub/offer/sku of each agent pool out into the parameters like we do for kubernetesHyperkubeSpec

#1 ensures a constant pub/offer/sku that works out of the box, and we test against
#2 implicitly communicates, that they are adjusting the batteries, and will need to support this, just like how it does for when with kubernetesHyperkubeSpec

Would that work?

[anhowe]

Further discussion with @uday31in, here is the latest proposal. This uses a new optional "distro" property on the agent pool. This has values 'ubuntu'|'coreos', and defaults to 'ubuntu' if not specified. Also, add validation, that this field only works with osType 'linux'. The parameters file will contain the pub/offer/sku for each agent pool.

The below example, creates 3 agent pools where the first one is coreos, and the second and third are ubuntu.

"agentPoolProfiles": [{
		"name": "agentpool1",
		"count": 1,
		"vmSize": "Standard_DS2_v2",
		"availabilityProfile": "AvailabilitySet",
		"distro": "coreos"
	},
	{
		"name": "agentpool2",
		"count": 1,
		"vmSize": "Standard_DS2_v2",
		"availabilityProfile": "AvailabilitySet",
		"distro": "ubuntu"
	},
	{
		"name": "agentpool3",
		"count": 1,
		"vmSize": "Standard_DS2_v2",
		"availabilityProfile": "AvailabilitySet"
	}]

[amanohar]

Note, @uday31in mentioned that it was a requirement for his customer to be able to deploy agent pools with different OS images. Hence the proposal from my side to unblock him. I also favor not providing the ability of specify image name because of the reasons I mention above.

[uday31in]

Here is the approach I am suggestion as per discussion with @anhowe @amanohar

Based on "distro", each kubernetes agent pool will have four additional "parameters" in generated ARM template with default value specified for Offer, Publisher, Sku and Version.

1. agentpoolname.osImageOffer
2. agentpoolname.osImagePublisher
3. agentpoolname.osImageSKU
4. agentpoolname.osImageVersion

If user want to change the defaults e.g. For default distro type "coreos" from CoreOS:CoreOS:Stable:latest -> CoreOS:CoreOS:Beta:latest they can do so by specifying above values in azuredeploy.parameters.json file.

Acs-engine generated template will only known to work for the defaults used for each distro type only. This way they are not forced to use default distro and versions however user also accepts that by act of overriding default parameters "THEY ARE DOING IT AT THEIR OWN RISK".

Here is resulting template for the above example.

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {

...

"agentpool1osImageOffer": {
      "defaultValue": "CoreOS",
      "metadata": {
        "description": "osImageOffer for Agent Pool 'agentpool1'."
      },
      "type": "string"
    },
    "agentpool1osImagePublisher": {
      "defaultValue": "CoreOS",
      "metadata": {
        "description": "osImagePublisher for Agent Pool 'agentpool1'."
      },
      "type": "string"
    },
    "agentpool1osImageSKU": {
      "defaultValue": "Stable",
      "metadata": {
        "description": "osImageSKU for Agent Pool 'agentpool1'."
      },
      "type": "string"
    },
    "agentpool1osImageVersion": {
      "defaultValue": "latest",
      "metadata": {
        "description": "osImageVersion for Agent Pool 'agentpool1'."
      },
      "type": "string"
    },

...

"agentpool2osImageOffer": {
      "defaultValue": "UbuntuServer",
      "metadata": {
        "description": "osImageOffer for Agent Pool 'agentpool2'."
      },
      "type": "string"
    },
    "agentpool2osImagePublisher": {
      "defaultValue": "Canonical",
      "metadata": {
        "description": "osImagePublisher for Agent Pool 'agentpool2'."
      },
      "type": "string"
    },
    "agentpool2osImageSKU": {
      "defaultValue": "16.04-LTS",
      "metadata": {
        "description": "osImageSKU for Agent Pool 'agentpool2'."
      },
      "type": "string"
    },
    "agentpool2osImageVersion": {
      "defaultValue": "16.04.201706191",
      "metadata": {
        "description": "osImageVersion for Agent Pool 'agentpool2'."
      },
      "type": "string"
    },

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contribution. Note that acs-engine is deprecated--see https://github.com/Azure/aks-engine instead.