Implement Gang-scheduling

[auhlig]

Description

Cortex should support gang-/co-scheduling for Kubernetes workloads.
A job starts only when all pods in the gang and their required resources are available at the same time.

Objectives

• Implement a controller that groups pods into a gang. Consider prior-art on how to define these.
• Track gang readiness and resource feasibility
• Submit a joint scheduling request to Cortex. Consider a single request or using Cortex' reservation feature to allocate resources.
• Bind all pods only after Cortex confirms full gang placement
• Provide minimal metrics/logs
• Documentation

Acceptance Criteria

• Gang-scheduling implemented
• Pods of a gang schedule only when the full gang can be placed
• Basic e2e tests

Dependencies

N/A

Additional Notes

N/A

[Varsius]

Implement a controller that groups pods into a gang. Consider prior-art on how to define these.

JobSets might be interesting for this.
The kube-scheduler recently merged a PR to introduce gang-scheduling.

[Varsius]

Brief explanation of how the default scheduler implements gang-scheduling:

Introduction of a new API object called Workload:

apiVersion: scheduling.k8s.io/v1alpha1
kind: Workload
metadata:
  name: jobset
spec:
  podGroups:
    - name: "job-1"
      policy:
        gang:
          minCount: 100

Pods now include a reference to the respective Workload:

apiVersion: v1
kind: Pod
metadata:
  name: jobset-job-1-abc123
spec:
  ...
  workloadRef:
    name: jobset
    podGroup: job-1
    podGroupReplicaKey: key-2
  ...

Implementation through the following hooks:

• PreEnqueue - used as a barrier to wait for the Workload object and all the necessary pods to be observed by the scheduler before even considering them for actual scheduling
• WaitOnPermit - used as a barrier to wait for the pods to be assigned to the nodes before initiating potential preemptions and their bindings

Source