Scheduling Queue and Event-driven Scheduling Attempts

[Varsius]

Description

Introduce a scheduling queue in Cortex that keeps track of unschedulable workloads. These workloads should be re-evaluated when the cluster state changes (e.g. nodes added, workloads complete), rather than relying only on exponential backoff.

Objectives

• Implement a pending queue for unschedulable pods with ordering by some priority (e.g. submission time)
• Register and handle cluster events to trigger scheduling attempts for pending pods
• Integrate queueing hints to selectively re-try only affected workloads
• Documentation

Acceptance Criteria

• Unschedulable workloads are queued and re-tried when relevant cluster events occur
• Basic e2e tests

Dependencies

N/A

Additional Notes

• Detailed explanation of the default scheduler's queueing system