Fix/kaiwoqueueconfig failed on missing clusterqueue

[salexo]

Bug Fixes: KaiwoQueueConfig Transient FAILED Status on ClusterQueue Deletion

Background

KaiwoQueueConfig was observed entering a FAILED status transiently during normal project lifecycle operations (project creation, deletion, or any external removal of a ClusterQueue that is still present in the spec). The status would recover on the next reconcile, but the window was long enough to be visible and could cause downstream issues.

---

Root Causes

1. syncLocalQueues: IsNotFound on ClusterQueue Get treated as fatal

When syncLocalQueues iterated over the ClusterQueues in the spec, it called r.Get() on each one before managing its LocalQueue objects. If the ClusterQueue was not yet visible in the informer cache (either due to cache lag after syncClusterQueues had just created it in the same reconcile, or because it had been externally deleted while still referenced by the spec), r.Get() returned IsNotFound. This was treated as a hard failure: success = false, causing SyncKueueResources to return an error and the overall status to flip to FAILED.

This is the expected initial state during a reconcile that just created the ClusterQueue — it is inherently transient and should be handled gracefully.

File: internal/controller/kaiwoqueueconfig_controller.go

2. Stale LocalQueue cleanup: IsNotFound on Delete treated as fatal

syncLocalQueues maintains a map of "expected" LocalQueues as it iterates. When a ClusterQueue is skipped (via continue — either because it was not found, or for another reason), its associated LocalQueues are never added to the expected map. The subsequent stale-LocalQueue cleanup loop then considers those LocalQueues as stale and attempts to delete them. However, LocalQueues are owned by their ClusterQueue and are cascade-deleted when the ClusterQueue is removed. The delete therefore returns IsNotFound, which was treated as a hard failure: success = false.

This compounded root cause #1: even after fixing the Get failure, the cleanup loop would independently set success = false for every delete on an already-gone LocalQueue.

File: internal/controller/kaiwoqueueconfig_controller.go

3. All other sync-function delete loops: IsNotFound on Delete treated as fatal

The same defensive gap existed in the delete loops of:

• syncResourceFlavors
• syncClusterQueues (which also had an accidental duplicate r.Delete call — a second delete was being issued in the if condition of the same loop body)
• syncTopologies
• syncWorkloadPriorityClasses

Any of these could be triggered if an object was garbage-collected or externally removed between the List call and the Delete call in the same reconcile.

File: internal/controller/kaiwoqueueconfig_controller.go

4. syncTopologies: empty-named Topology causes Create to fail

CreateDefaultTopology populates spec.topologies using config.DefaultTopologyName from the KaiwoConfig spec. When KaiwoConfig is auto-created with an empty spec (the defaultTopologyName field is omitempty), this produces a Topology entry with metadata.name: "". The Kueue API server rejects the subsequent Create call with a validation error, setting success = false on every reconcile and keeping the KaiwoQueueConfig permanently in FAILED.

A separate fix for the root cause in CreateDefaultTopology (falling back to a constant when the configured name is empty) is tracked for a follow-up PR. This PR adds a defensive guard in syncTopologies that skips any topology entry with an empty name and logs a clear advisory message, preventing the API-server rejection from causing a permanent FAILED state.

File: internal/controller/kaiwoqueueconfig_controller.go

5. Missing Owns(&ClusterQueue{}) watch

The controller's SetupWithManager did not watch ClusterQueue objects owned by KaiwoQueueConfig. This meant that when syncClusterQueues re-created a deleted ClusterQueue, no new reconcile was enqueued. The LocalQueues for that ClusterQueue would therefore only be created on the next reconcile triggered by some other event (e.g. a Node change), which could be arbitrarily delayed.

File: internal/controller/kaiwoqueueconfig_controller.go

6. FAILED status not requeued

When SyncKueueResources returned an error, the Reconcile function set the status to FAILED but returned ctrl.Result{}, nil — meaning the controller would only retry if another event happened to trigger a reconcile. Combined with the missing Owns watch, this could leave the controller stuck in FAILED indefinitely.

File: internal/controller/kaiwoqueueconfig_controller.go

---

Fixes

#	Location	Change
1	syncLocalQueues — ClusterQueue Get	IsNotFound → log at Info level ("ClusterQueue not yet available, deferring LocalQueue sync") and continue; only non-IsNotFound errors set success = false
2	syncLocalQueues — stale LocalQueue delete loop	Added !errors.IsNotFound(err) guard so cascade-deleted LocalQueues do not cause a failure
3a	syncResourceFlavors delete loop	Added !errors.IsNotFound(err) guard
3b	syncClusterQueues delete loop	Removed duplicate r.Delete call; added !errors.IsNotFound(err) guard
3c	syncTopologies delete loop	Added !errors.IsNotFound(err) guard
3d	syncTopologies create loop	Added empty-name guard: skip entries where kueueTopology.Name == "" with an advisory log message
3e	syncWorkloadPriorityClasses delete loop	Added !errors.IsNotFound(err) guard
4	SetupWithManager	Added Owns(&kueuev1beta1.ClusterQueue{}) so ClusterQueue create/delete events trigger a reconcile for the owning KaiwoQueueConfig
5	Reconcile — final return	When queueConfig.Status.Status == FAILED, return ctrl.Result{RequeueAfter: 30 * time.Second} instead of ctrl.Result{}, nil

---

Testing

A new Chainsaw regression test is added at test/chainsaw/tests/standard/kaiwoqueueconfigs/clusterqueue-deletion/. It:

1. Applies a KaiwoQueueConfig spec containing a namespaced ClusterQueue (fizz) and waits for a stable READY state.
2. Runs three delete-and-recover cycles: directly deletes the ClusterQueue, triggers a reconcile, and continuously watches the KaiwoQueueConfig status — asserting that no FAILED transition is ever observed.
3. After the final deletion, asserts that both the ClusterQueue and its LocalQueue are autonomously recreated within 90 s (exercising the Owns watch and RequeueAfter recovery path).

The test was verified to fail against the unfixed controller and pass against the fixed controller.