feat: surface unrecoverable container errors during pod wait#15
feat: surface unrecoverable container errors during pod wait#15flysky22222 wants to merge 1 commit into
Conversation
When a CI job references a non-existent image or one it lacks permission to pull, the pod stays in Pending and waitForPodPhases previously timed out with only a generic phase-status message. GitHub Actions users had no indication of the real cause. Detect unrecoverable container waiting reasons (ImagePullBackOff, ErrImagePull, InvalidImageName, CreateContainerConfigError, CreateContainerError) on both init and regular containers, and fail fast with the container name, reason, and Kubernetes message so the error is visible in the Actions log. Refactor getPodPhase into readPod + parsePodPhase so the pod object can be inspected for container errors, and add describePodFailure to aggregate pod phase, conditions, container statuses and Warning events into a single diagnostic string. describePodWarningEvents provides best-effort retrieval of recent Warning events (requires optional events RBAC permission, degrades gracefully when missing). getUnrecoverableWaitingReasons allows operators to extend the built-in fast-fail whitelist via the ACTIONS_RUNNER_K8S_UNRECOVERABLE_WAITING_REASONS env var. All three failure paths in waitForPodPhases now attach full diagnostics: 1. Non-backoff phase (e.g. Failed) — includes pod details + events 2. Unrecoverable container error (e.g. ImagePullBackOff) — fail-fast 3. Timeout — includes pod details so the user can see WHY the pod never became ready README: document the optional events permission and the new env var. Tests: 19 new test cases covering parsePodPhase, getContainerErrors, waitForPodPhases, getUnrecoverableWaitingReasons, describePodFailure, describePodWarningEvents, and edge cases such as forbidden events API and unreadable pods.
Welcome To opensourceways CommunityHey @flysky22222 , thanks for your contribution to the community. Bot Usage ManualI'm the Bot here serving you. You can find the instructions on how to interact with me at Here . That means you can comment below every pull request or issue to trigger Bot Commands. |
CLA Signature Passflysky22222, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
Linking Issue Notice@flysky22222 , the pull request must be linked to at least one issue. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces enhanced diagnostic logging and unrecoverable container error detection when waiting for Kubernetes pods to come online, including documentation updates in the README. The feedback focuses on optimizing the polling loop in waitForPodPhases by caching the unrecoverable waiting reasons environment variable and passing the already-fetched pod object to describePodFailure and getContainerErrors to avoid redundant API calls and parsing overhead.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| export function getContainerErrors(pod: k8s.V1Pod): string[] { | ||
| const errors: string[] = [] | ||
| const unrecoverableReasons = getUnrecoverableWaitingReasons() |
There was a problem hiding this comment.
Calling getUnrecoverableWaitingReasons() inside getContainerErrors parses the environment variable ACTIONS_RUNNER_K8S_UNRECOVERABLE_WAITING_REASONS on every single polling iteration. Since environment variables do not change during the execution of waitForPodPhases, we can optimize this by parsing the reasons once before the loop and passing them as an optional parameter to getContainerErrors.
export function getContainerErrors(
pod: k8s.V1Pod,
unrecoverableReasons = getUnrecoverableWaitingReasons()
): string[] {
const errors: string[] = []| export async function describePodFailure(podName: string): Promise<string> { | ||
| let pod: k8s.V1Pod | ||
| try { | ||
| pod = await readPod(podName) | ||
| } catch (err) { | ||
| return `Could not read pod ${podName} for diagnostics: ${ | ||
| err instanceof Error ? err.message : String(err) | ||
| }` | ||
| } |
There was a problem hiding this comment.
The describePodFailure function always fetches the pod status from the Kubernetes API server using readPod(podName). However, in waitForPodPhases, we already have the fetched pod object. We can optimize this by allowing describePodFailure to accept an optional pod parameter, avoiding redundant API calls when the pod object is already available.
export async function describePodFailure(
podName: string,
providedPod?: k8s.V1Pod
): Promise<string> {
let pod = providedPod
if (!pod) {
try {
pod = await readPod(podName)
} catch (err) {
return `Could not read pod ${podName} for diagnostics: ${
err instanceof Error ? err.message : String(err)
}`
}
}| ): Promise<void> { | ||
| const backOffManager = new BackOffManager(maxTimeSeconds) | ||
| let phase: PodPhase = PodPhase.UNKNOWN | ||
| try { | ||
| while (true) { | ||
| phase = await getPodPhase(podName) | ||
| if (awaitingPhases.has(phase)) { | ||
| return | ||
| } | ||
| while (true) { | ||
| const pod = await readPod(podName) | ||
| phase = parsePodPhase(pod) | ||
| if (awaitingPhases.has(phase)) { | ||
| return | ||
| } | ||
|
|
||
| if (!backOffPhases.has(phase)) { | ||
| throw new Error( | ||
| `Pod ${podName} is unhealthy with phase status ${phase}` | ||
| ) | ||
| } | ||
| if (!backOffPhases.has(phase)) { | ||
| const details = await describePodFailure(podName) | ||
| throw new Error( | ||
| `Pod ${podName} is unhealthy with phase status ${phase}\n${details}` | ||
| ) | ||
| } | ||
|
|
||
| const containerErrors = getContainerErrors(pod) | ||
| if (containerErrors.length > 0) { | ||
| const details = await describePodFailure(podName) | ||
| throw new Error( | ||
| `Pod ${podName} has unrecoverable container errors: ${containerErrors.join( | ||
| '; ' | ||
| )}\n${details}` | ||
| ) | ||
| } | ||
|
|
||
| try { | ||
| await backOffManager.backOff() | ||
| } catch (error) { | ||
| const details = await describePodFailure(podName) | ||
| throw new Error( | ||
| `Pod ${podName} is unhealthy: timed out after ${maxTimeSeconds}s in phase ${phase}\n${details}` | ||
| ) | ||
| } | ||
| } catch (error) { | ||
| throw new Error(`Pod ${podName} is unhealthy with phase status ${phase}`) | ||
| } |
There was a problem hiding this comment.
Optimize waitForPodPhases by fetching the unrecoverable waiting reasons once before entering the polling loop, and passing the already-fetched pod object to describePodFailure and getContainerErrors to avoid redundant environment variable parsing and Kubernetes API calls.
): Promise<void> {
const backOffManager = new BackOffManager(maxTimeSeconds)
let phase: PodPhase = PodPhase.UNKNOWN
const unrecoverableReasons = getUnrecoverableWaitingReasons()
while (true) {
const pod = await readPod(podName)
phase = parsePodPhase(pod)
if (awaitingPhases.has(phase)) {
return
}
if (!backOffPhases.has(phase)) {
const details = await describePodFailure(podName, pod)
throw new Error(
`Pod ${podName} is unhealthy with phase status ${phase}\n${details}`
)
}
const containerErrors = getContainerErrors(pod, unrecoverableReasons)
if (containerErrors.length > 0) {
const details = await describePodFailure(podName, pod)
throw new Error(
`Pod ${podName} has unrecoverable container errors: ${containerErrors.join(
'; '
)}\n${details}`
)
}
try {
await backOffManager.backOff()
} catch (error) {
const details = await describePodFailure(podName, pod)
throw new Error(
`Pod ${podName} is unhealthy: timed out after ${maxTimeSeconds}s in phase ${phase}\n${details}`
)
}
}3 participants
背景
K8s 模式下，当 CI job 引用了不存在的镜像、或没有权限拉取的镜像时，Pod 会卡在 Pending。此前
waitForPodPhases只会一直 backoff 直到超时，最终只报一句笼统的 phase 错误，GitHub Actions 用户看不到真正的失败原因。改动
ImagePullBackOff、ErrImagePull、InvalidImageName、CreateContainerConfigError、CreateContainerError）。getPodPhase重构为readPod+parsePodPhase，以便复用 Pod 对象做容器检查。describePodFailure聚合 Pod phase、conditions、container statuses 和 Warning events 为诊断信息。describePodWarningEvents读取 K8s Warning 事件（可选 events 权限，缺失时优雅降级）。getUnrecoverableWaitingReasons支持通过环境变量扩展不可恢复原因列表。waitForPodPhases三条失败路径均附带完整诊断信息。测试
parsePodPhase、getContainerErrors、waitForPodPhases、getUnrecoverableWaitingReasons、describePodFailure、describePodWarningEvents及边界情况。npm run build --prefix packages/k8s通过npm run test --prefix packages/k8s中新增测试全部通过prettier@2.6.2格式检查通过关联 Issue: https://github.com/opensourceways/backlog/issues/1102