[需求] CI容器通用化k8s错误反馈机制-补充优化-的开发实现-runner-container-hooks部分#20
Conversation
…-only to 11 types - Add FailedMount to UNRECOVERABLE_WAITING_REASONS (covers waiting-state volume mount failures) - Add UNRECOVERABLE_TERMINATED_REASONS (OOMKilled, Error, FailedPostStartHookError) for Running-phase errors - Add getUnrecoverableTerminatedReasons() with env var extension support - Add getContainerTerminatedErrors() for detecting terminated containers with unrecoverable reasons - Add terminated error detection in runContainerStep catch block with describePodFailure diagnostics - Branch based on release/no_volumes (not main) per issue #1203 request UT coverage: - wait-for-pod-phases-test.ts: 10 new tests for getContainerTerminatedErrors (8) and getUnrecoverableTerminatedReasons (4) - run-container-step-terminated-test.ts: 5 new tests for runContainerStep terminated error detection (OOMKilled, Error, FailedPostStartHookError, no errors, getPodByName failure)
Welcome To opensourceways CommunityHey @flysky22222 , thanks for your contribution to the community. Bot Usage ManualI'm the Bot here serving you. You can find the instructions on how to interact with me at Here . That means you can comment below every pull request or issue to trigger Bot Commands. Contact GuideIf you have any questions, please contact the SIG: infratructure , |
CLA Signature Passflysky22222, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
|
|||||||||||||||||
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces unrecoverable container termination error detection in the Kubernetes runner step, adding support for identifying issues like OOMKilled, Error, and FailedPostStartHookError, with the ability to extend these reasons via environment variables. Feedback on these changes suggests optimizing performance by avoiding redundant Kubernetes API calls when fetching pod details, and explicitly checking that the container exit code is non-zero to prevent false positives for successful terminations.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| const pod = await getPodByName(podName) | ||
| const terminatedErrors = getContainerTerminatedErrors(pod) | ||
| if (terminatedErrors.length > 0) { | ||
| const details = await describePodFailure(podName) |
There was a problem hiding this comment.
Calling getPodByName(podName) and then describePodFailure(podName) results in two consecutive, redundant Kubernetes API calls to fetch the exact same Pod resource (since describePodFailure internally calls readPod to fetch the pod again). To optimize performance and avoid unnecessary API rate-limiting, consider refactoring describePodFailure to accept an optional pre-fetched V1Pod object, allowing you to reuse the pod instance fetched on line 123.
| ] | ||
| for (const cs of allStatuses) { | ||
| const terminated = cs.state?.terminated | ||
| if (terminated?.reason && unrecoverableReasons.has(terminated.reason)) { |
There was a problem hiding this comment.
To prevent potential false positives where a container successfully exits with code 0 but is flagged due to a whitelisted termination reason, we should explicitly check that terminated.exitCode !== 0 before reporting it as an error. This also aligns with the logic used in describePodFailure (line 939).
| if (terminated?.reason && unrecoverableReasons.has(terminated.reason)) { | |
| if (terminated?.reason && terminated.exitCode !== 0 && unrecoverableReasons.has(terminated.reason)) { |
3 participants
背景
[需求] CI容器通用化k8s错误反馈机制-补充优化（runner-container-hooks）· 开发流水线 · 开发预览阶段（代码已推 + 预览已部署 + UT 已补；门禁/对抗由 PR CI 异步跑）
注意：本 PR 基于
release/no_volumes分支（不是main），按照 @Longwt123 的要求重新创建。之前的 PR #19（基于main）已关闭。改动内容
release/no_volumes已有的 Pending 阶段错误检测（无需改动，保留）UNRECOVERABLE_WAITING_REASONS(ImagePullBackOff, ErrImagePull, InvalidImageName, CreateContainerConfigError, CreateContainerError)UNRECOVERABLE_EVENT_REASONS(FailedMount, FailedScheduling, FailedBinding)getContainerErrors(),getPodConditionErrors(),getPodEventErrors()describePodFailure(),waitForPodPhases()with full error detectionprepareJob()with improved error message formatting本轮新增：Running 阶段 terminated 错误检测 + FailedMount 补充
packages/k8s/src/k8s/index.tsFailedMounttoUNRECOVERABLE_WAITING_REASONS(per design §2.3 check-npu #1 — FailedMount can appear as container waiting reason in some k8s versions)UNRECOVERABLE_TERMINATED_REASONSSet withOOMKilled,Error,FailedPostStartHookError(per design §2.3 feat: add preferred pod affinity for app pod #3)getUnrecoverableTerminatedReasons()function with env varACTIONS_RUNNER_K8S_UNRECOVERABLE_TERMINATED_REASONSextension support (per design §2.3 feat: delete check npu script #5)getContainerTerminatedErrors()function — detects terminated containers with unrecoverable reasons, returns formatted strings with ✗ markers (per design §2.3 update runner to 2.330.0 #8)packages/k8s/src/hooks/run-container-step.tsdescribePodFailure,getContainerTerminatedErrors,getPodByNamefrom../k8sgetContainerTerminatedErrors+describePodFailurebefore rethrowing (per design §2.3 Tfh novolumes #11, adapted forrelease/no_volumes's runContainerStep flow)11 种错误 → 检测机制映射
getContainerErrorsgetContainerErrorsgetContainerErrorsgetContainerErrorsgetContainerErrors+getPodEventErrorsgetPodConditionErrors+getPodEventErrorsgetPodEventErrorsgetPodEventErrorsgetContainerTerminatedErrorsgetContainerTerminatedErrorsgetContainerTerminatedErrorsUT Coverage
packages/k8s/tests/wait-for-pod-phases-test.ts— 10 new testsgetContainerTerminatedErrors: 8 tests (no errors, no terminated state, recoverable reasons, OOMKilled, Error, FailedPostStartHookError, every reason, init containers, whitelist filtering)getUnrecoverableTerminatedReasons: 4 tests (defaults, env var extension, honors in getContainerTerminatedErrors, filters empty strings)packages/k8s/tests/run-container-step-terminated-test.ts— 5 new testsAll 15 new tests pass. All 44 existing wait-for-pod-phases tests pass. TypeScript compiles without errors.
相关 Issue
resolve https://github.com/opensourceways/backlog/issues/1203
AI 使用声明
当前 PR 是否有 AI 参与：
/ai-develop-preview