chore(recipes): check and update runtime component versions across all recipes

[yuanchen8911]

Summary

Audit the pinned versions of every runtime component referenced by AICR recipes and update them to current stable upstream releases where appropriate.

Motivation

Component versions in recipes/registry.yaml, recipes/overlays/*.yaml, and recipes/components/*/values.yaml drift over time as upstream projects ship security fixes, bug fixes, and Kubernetes compatibility updates. Without a periodic refresh, AICR-generated bundles deploy stale versions that may miss CVE fixes or fail against newer Kubernetes versions.

Scope

Check and (where appropriate) update versions for every component declared in recipes/registry.yaml, including (non-exhaustive list — registry is authoritative):

• GPU stack: gpu-operator, nvidia-device-plugin, dcgm-exporter
• Schedulers / CRDs: kai-scheduler, kueue, volcano
• Inference: dynamo-platform, kgateway
• Training: kubeflow-trainer
• Observability: kube-prometheus-stack, prometheus-adapter
• Platform: nfd, cert-manager, node-feature-discovery
• Any other components in recipes/registry.yaml not listed here

Also check overlay-level pins in recipes/overlays/*.yaml and any version constraints embedded in mixins (recipes/mixins/*.yaml).

Definition of done

• For each component, current pinned version is compared against latest stable upstream release.
• Where a newer version is selected, the version pin is updated and the change is justified (CVE fix, K8s compatibility, etc.) in the PR description.
• Where a version is intentionally held back, the reason is documented (in recipes/registry.yaml comments or a short note in docs/contributor/data.md).
• make qualify passes locally on the updated recipes.
• KWOK simulated cluster tests pass (make kwok-test-all).
• At least one full GPU CI run on H100 nvkind passes against the updated bundle.
• Documentation under docs/user/component-catalog.md is regenerated/updated if any displayName, repository, or default chart changes.

Out of scope

• Adding new components (handled in separate issues).
• Upgrading recipes to a different Kubernetes minor (separate compatibility effort).
• Removing components.

Notes

• Compatibility constraint: AICR targets Kubernetes 1.33+. Verify each upgraded component supports this floor.
• Constraints in recipes (spec.constraints[*].name = "K8s.server.version") should be re-evaluated when upgrading.
• For Helm-installed components, confirm the upstream chart version matches the app version expected by nodeScheduling and any value overrides we set.

[yuanchen8911]

Note: when auditing kube-prometheus-stack version, check whether the bundled grafana subchart includes grafana-community/helm-charts#418 (merged 2026-04-26, fixes grafana-community/helm-charts#417).

The PR exposes grafana.progressDeadlineSeconds and grafana.imageRenderer.progressDeadlineSeconds as Helm values. Once available, AICR can tune the Deployment progress deadline on slow-runner overlays (e.g. recipes/overlays/kind.yaml) instead of working around ProgressDeadlineExceeded by disabling Grafana entirely (grafana.enabled: false).

[yuanchen8911]

Component version audit (2026-04-29)

Audit of every component pin in recipes/registry.yaml, recipes/overlays/*.yaml, and recipes/mixins/*.yaml against upstream latest.

Helm components

#	Component	Current pin	Latest upstream	Delta
1	nfd (node-feature-discovery)	0.18.3	v0.18.3	up to date
2	gpu-operator	v25.10.1	v26.3.1 (2026-04-18)	major: 25.10 → 26.3
3	network-operator	25.1.0	v26.1.1 (2026-02)	major: 25.1 → 26.1
4	aws-efa-k8s-device-plugin	v0.5.3	chart v0.5.26 (app v0.5.18)	many minor bumps; verify pin is chart vs app
5	cert-manager	v1.17.2	v1.20.2 (2026-04-11)	3 minors behind (1.17 → 1.20)
6	nodewright-operator (skyhook)	v0.14.0	chart/v0.15.1 (2026-04-14, app v0.15.0)	minor bump
7	nvsentinel	v1.1.0	v1.3.0 (2026-04-20)	2 minors behind
8	nvidia-dra-driver-gpu	25.12.0	v25.12.0 (2026-02-12)	up to date
9	kube-prometheus-stack	82.8.0	84.4.0 (2026-04-29)	2 minors behind — see grafana note above
10	prometheus-adapter	5.3.0	5.3.0	up to date
11	aws-ebs-csi-driver	2.55.0	2.59.0 (2026-04-21)	4 patch bumps behind
12	k8s-ephemeral-storage-metrics	1.19.2	1.19.2	up to date
13	kai-scheduler	v0.13.0	v0.14.1 (2026-04-29)	minor bump
14	dynamo-crds	0.9.1 (registry) / 0.9.0 (overlays)	1.0.2 (2026-04-23)	major: 0.9 → 1.0 + registry/overlay drift
15	dynamo-platform	0.9.1 (registry) / 0.9.0 (overlays)	1.0.2 (2026-04-23)	major: 0.9 → 1.0 + registry/overlay drift
16	kgateway-crds	v2.0.0	v2.2.3 (2026-04-13)	2 minors behind
17	kgateway	v2.0.0	v2.2.3 (2026-04-13)	2 minors behind
18	k8s-nim-operator	3.1.0	v3.1.0 (2026-03-13)	up to date
19	kueue	0.17.0	v0.17.1 (2026-04-16)	patch bump
20	kubeflow-trainer	2.1.0	v2.2.0 (2026-03-20)	minor bump

Manifest-only / kustomize (no chart version to compare)

• gke-nccl-tcpxo — kustomize-based; tracks nccl-tcpxo-installer.yaml + nri-device-injector.yaml directly. Not a Helm pin; would need a separate manifest-version audit.
• nodewright-customizations — in-tree tuning.yaml / tuning-gke.yaml; no upstream version.

Inconsistencies worth fixing alongside the bumps

1. Dynamo drift — recipes/registry.yaml defaults to 0.9.1 for both dynamo-crds and dynamo-platform, but every overlay overrides to 0.9.0. Whichever way we go, registry default and overlays should converge.
2. GPU driver pin — 580.126.20 is set as a values override under gpu-operator, separate from the chart pin. Worth re-checking against the latest GA driver in the same PR as the chart bump.

Suggested batching for follow-on PRs

• Major NVIDIA stack: gpu-operator 25.10 → 26.3, network-operator 25.1 → 26.1, nvsentinel 1.1 → 1.3, kai-scheduler 0.13 → 0.14, skyhook 0.14 → 0.15.1, dynamo 0.9 → 1.0.2 (largest blast radius — H100 CI gate required).
• Cert / observability: cert-manager 1.17 → 1.20, kube-prometheus-stack 82.8 → 84.4 (validate grafana progressDeadlineSeconds exposure per the note above), prometheus-adapter unchanged.
• Cloud providers: aws-ebs-csi-driver 2.55 → 2.59, aws-efa v0.5.3 → v0.5.26.
• Workload runtimes: kueue 0.17.0 → 0.17.1, kubeflow-trainer 2.1 → 2.2, kgateway/kgateway-crds 2.0 → 2.2.3.

[yuanchen8911]

Deep dive: Dynamo drift and gpu-operator driver pins

Following up on the two inconsistencies flagged in the audit — full root-cause analysis and proposed fixes for each.

---

1. Dynamo drift

What the drift is

recipes/registry.yaml declares defaultVersion: "0.9.1" for both dynamo-crds and dynamo-platform, but every Dynamo overlay also declares an explicit version: "0.9.0" on the same components. The overlay's explicit version: overrides the registry default, so every Dynamo bundle generated today actually ships chart 0.9.0. The registry default is effectively dead code for Dynamo.

Affected overlays (6 files, 12 redundant pins):

• recipes/overlays/h100-kind-inference-dynamo.yaml
• recipes/overlays/h100-eks-ubuntu-inference-dynamo.yaml
• recipes/overlays/h100-aks-ubuntu-inference-dynamo.yaml
• recipes/overlays/h100-gke-cos-inference-dynamo.yaml
• recipes/overlays/gb200-eks-ubuntu-inference-dynamo.yaml
• recipes/overlays/gb200-oke-ubuntu-inference-dynamo.yaml

This anti-pattern is specific to Dynamo — the other overlay-pinned components (gpu-operator, kube-prometheus-stack, nvsentinel, kgateway*) either have no registry default (overlay is intentionally authoritative) or the values match.

How it happened — PR history

• feat(recipes): upgrade dynamo-platform to v0.9.0 and disable etcd/nats #257 (merged 2026-03-02) introduced the dynamo overlays with both registry default 0.9.0 and explicit overlay pin 0.9.0. The duplication existed from day one but values matched, so no visible drift.
• fix(recipes): override deprecated gcr.io kube-rbac-proxy image for dynamo #318 (merged 2026-03-09) was a kube-rbac-proxy image override fix that also bumped the registry default 0.9.0 → 0.9.1 — but left every overlay pin untouched. This is where drift was silently introduced.
• chore(recipe): bump dynamo-platform from 0.9.x to 1.0.2 and add Grove chart #459 (still OPEN since 2026-03-31) is an in-flight major bump to 1.0.2. The author updated registry default + 5 of 6 overlay pins but missed gb200-oke-ubuntu-inference-dynamo.yaml — exactly the failure mode the duplicated-pin pattern enables. PR has been awaiting author response since 2026-04-09.

The redundant pin pattern has now caused drift twice (silent miss in #318, partial miss in #459).

Proposed fix

Drop the redundant version: lines from all 6 overlays so they inherit the registry default. This matches the pattern used for kubeflow-trainer in mixins/platform-kubeflow.yaml (no version → inherits from registry).

     - name: dynamo-crds
       type: Helm
       source: https://helm.ngc.nvidia.com/nvidia/ai-dynamo
-      version: "0.9.0"
       valuesFile: components/dynamo-crds/values.yaml

     - name: dynamo-platform
       type: Helm
       source: https://helm.ngc.nvidia.com/nvidia/ai-dynamo
-      version: "0.9.0"
       valuesFile: components/dynamo-platform/values.yaml

Sequencing options vs in-flight #459:

1. Land the dedup cleanup first — drop overlay pins so they inherit 0.9.1. Then chore(recipe): bump dynamo-platform from 0.9.x to 1.0.2 and add Grove chart #459 simplifies dramatically (only edits the registry default). Fewer review surfaces, no chance of missing an overlay. Recommended.
2. Pivot chore(recipe): bump dynamo-platform from 0.9.x to 1.0.2 and add Grove chart #459 to do both — drop redundant pins and bump registry to 1.0.2 in one PR. Requires taking over the branch since the author has been silent for ~3 weeks.
3. Status quo — wait for chore(recipe): bump dynamo-platform from 0.9.x to 1.0.2 and add Grove chart #459 to land, then dedup as follow-up. Slowest, and chore(recipe): bump dynamo-platform from 0.9.x to 1.0.2 and add Grove chart #459 keeps needing manual reconciliation while it sits.

---

2. gpu-operator driver pin

Where the driver is actually pinned

Three locations, all in the values layer (separate from the chart pin):

Location	Pin	Rationale
recipes/components/gpu-operator/values.yaml:59	580.105.08	Global default for every overlay. No comment — purpose unclear from the file.
recipes/overlays/gb200-eks-training.yaml:63	580.126.20	Explicit comment: "NVIDIA's recommended floor for GB200+EFA"
recipes/overlays/gb200-eks-inference.yaml:57	580.126.20	Same GB200 override (no comment)
recipes/overlays/kind.yaml	enabled: false	nvkind uses pre-installed host drivers — keep as is

Per-platform values files (values-gke-cos*.yaml, values-oke*.yaml) don't override driver.version — they inherit the global 580.105.08.

How those pins line up with upstream

Source	driver.version default
gpu-operator chart v25.10.1 (our current pin)	580.105.08
gpu-operator chart v26.3.1 (upstream latest)	580.126.20
NVIDIA R580 release stream (newest first)	580.159.03 → 580.142 → 580.126.20 → … → 580.105.08

So:

• The global default (580.105.08) is not stale — it matches the chart default we're pinned at. It tracks the chart, but redundantly.
• The GB200 override (580.126.20) is a forward-port — it adopted what would later become the v26.3.1 chart default, motivated by the GB200+EFA floor.
• Newer drivers (580.142, 580.159.03, R590/R595) exist but the gpu-operator chart hasn't blessed them yet. Chasing them independently means owning compatibility for off-chart driver images — risk we currently avoid.

Proposed fix

Remove the existing overrides in the same PR that bumps gpu-operator v25.10.1 → v26.3.1. Keep the capability to override for legitimate cases.

After the chart bump:

• The global 580.105.08 pin holds us back to an old driver (chart's new default is 580.126.20).
• The GB200 580.126.20 pin matches the chart default — no longer load-bearing.

In the bump PR:

1. Delete driver.version: 580.105.08 from components/gpu-operator/values.yaml.
2. Delete driver.version: 580.126.20 from gb200-eks-training.yaml (with its rationale comment) and gb200-eks-inference.yaml.
3. Add a top-of-file comment in components/gpu-operator/values.yaml: "Do not pin driver.version unless required by hardware/CVE — track the chart default."

Policy for future overrides (keep the escape hatch)

The override capability is needed for real-world cases:

• New hardware floor before chart catches up — exactly the GB200+EFA case. When the next hardware (B200, GB300, etc.) ships, NVIDIA typically publishes the required driver before the chart's default rolls over.
• Off-cycle driver CVEs — security patch released before the next chart release.
• Platform-specific quirks — a future GKE/AKS/OKE driver build for kernel ABI reasons.

When an override is added, the values block must include a comment with:

• Why the override is needed (hardware floor, CVE, platform quirk)
• When it can be removed (e.g., "remove when gpu-operator chart default ≥ 580.126.20")

The existing GB200 comment is a good template. The global 580.105.08 pin had no comment, so nobody could tell whether it was load-bearing or leftover — that's the failure mode to avoid.

---

Combined effect

Both fixes share a single principle: single source of truth, with documented exceptions. The registry/chart is the default; overrides exist only with a written justification and a removal trigger.

Net result after both fixes:

• Dynamo: 12 redundant overlay pins deleted; one registry default knob.
• gpu-operator driver: 3 redundant value pins deleted; chart's blessed default applies; explicit policy for when overriding is allowed.

[yuanchen8911]

Phased rollout plan — 4 PRs

Splitting the version bump campaign into 4 PRs, one per phase, ordered by blast radius. Each phase is serial — wait for the prior to land and CI to settle before opening the next.

Phase 1 — Minor / patch bumps (single PR)

Low-risk, batch all 10 components together.

• nvsentinel v1.1.0 → v1.3.0
• skyhook (nodewright-operator) v0.14.0 → v0.15.1
• kai-scheduler v0.13.0 → v0.14.1
• cert-manager v1.17.2 → v1.20.2 (3 minors — review changelogs for deprecated APIs)
• kube-prometheus-stack 82.8.0 → 84.4.0 (verify grafana progressDeadlineSeconds exposure per earlier note)
• aws-ebs-csi-driver 2.55.0 → 2.59.0
• aws-efa v0.5.3 → v0.5.26 (verify chart-version vs app-version semantics first)
• kueue 0.17.0 → v0.17.1
• kubeflow-trainer 2.1.0 → v2.2.0
• kgateway / kgateway-crds v2.0.0 → v2.2.3 (two minors — H100 inference run)

Validation: make qualify + KWOK e2e + H100 training and inference CI.

Phase 2 — Dynamo (cleanup + major bump, single PR)

Drops the redundant overlay pins and bumps to 1.0.2 atomically. Supersedes / takes over #459.

• Bump registry defaultVersion for dynamo-crds and dynamo-platform: 0.9.1 → 1.0.2
• Drop redundant version: "0.9.0" lines from all 6 dynamo overlays:
  • recipes/overlays/h100-kind-inference-dynamo.yaml
  • recipes/overlays/h100-eks-ubuntu-inference-dynamo.yaml
  • recipes/overlays/h100-aks-ubuntu-inference-dynamo.yaml
  • recipes/overlays/h100-gke-cos-inference-dynamo.yaml
  • recipes/overlays/gb200-eks-ubuntu-inference-dynamo.yaml
  • recipes/overlays/gb200-oke-ubuntu-inference-dynamo.yaml
• Rewrite recipes/components/dynamo-platform/values.yaml for the 1.0 Helm schema: global.* subchart controls, upgradeCRD: true, remove the kube-rbac-proxy override (fixed upstream)
• Coordinate with Jont828 / supersede chore(recipe): bump dynamo-platform from 0.9.x to 1.0.2 and add Grove chart #459

Validation: H100 inference Dynamo run; KWOK e2e for each Dynamo overlay.

Phase 3 — Network operator major (single PR)

• network-operator 25.1.0 → v26.1.1

Validation: H100 + GB200 RDMA / IB regression on overlays that consume network-operator.

Phase 4 — GPU operator major + driver pin dedup (single PR)

Largest blast radius — every overlay uses gpu-operator. Driver dedup lands atomically with the chart bump because the chart's new default (580.126.20) is what makes the existing pins redundant.

• gpu-operator chart v25.10.1 → v26.3.1
• Delete driver.version: 580.105.08 from recipes/components/gpu-operator/values.yaml
• Delete driver.version: 580.126.20 from recipes/overlays/gb200-eks-training.yaml (with its rationale comment, lines 60–63)
• Delete driver.version: 580.126.20 from recipes/overlays/gb200-eks-inference.yaml
• Add a top-of-file comment in recipes/components/gpu-operator/values.yaml documenting the override policy: "Do not pin driver.version unless required by hardware/CVE — track the chart default. When overriding, include why and a removal trigger."
• Re-check the GPU driver against any newer R580/R590/R595 release blessed by the chart at PR time

Validation: H100 + GB200 full CI gate (training + inference).

Sequencing

Phase 1 (all minor/patch in one PR) ──┐
                                      │
Phase 2 (dynamo dedup + 1.0.2)      ──┼── serial, one major each
                                      │
Phase 3 (network-op 26.1)           ──┤
                                      │
Phase 4 (gpu-operator 26.3 + driver dedup) ──┘

Out of scope for this campaign

• gke-nccl-tcpxo manifest version audit (kustomize-based; tracked separately)
• nodewright-customizations (in-tree manifests, no upstream version)
• Components already up to date: nfd, nvidia-dra-driver-gpu, prometheus-adapter, k8s-ephemeral-storage-metrics, k8s-nim-operator