fix(k8s): FailedScheduling errors for fast-fail and Hint#26
Conversation
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
Welcome To opensourceways CommunityHey @Longwt123 , thanks for your contribution to the community. Bot Usage ManualI'm the Bot here serving you. You can find the instructions on how to interact with me at Here . That means you can comment below every pull request or issue to trigger Bot Commands. Contact GuideIf you have any questions, please contact the SIG: infratructure , |
CLA Signature PassLongwt123, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
|
|||||||||||||||||
CLA Signature PassLongwt123, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
|
/retest |
|
|||||||||||||||||
|
|||||||||||||||||
Longwt123
force-pushed
the
fix/scheduling-fast-fail
branch
from
2be9111 to
8451506
Compare
CLA Signature PassLongwt123, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
|
/retest |
|
|||||||||||||||||
|
|||||||||||||||||
Longwt123
changed the title
NotificationThis pull request source branch has changed, so removes the following label(s): gate_check_pass. |
CLA Signature PassLongwt123, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
|
|||||||||||||||||
…on readPod failure
Problem: FailedScheduling / Unschedulable fast-failed on ANY unrecognised
message, including transient resource shortages (Insufficient gpu/cpu/memory)
and unknown kube-scheduler message formats. This broke GPU workflow queuing.
Also, a transient readPod() failure crashed the waitForPodPhases loop with
no diagnostics (Risk A).
Changes:
- Replace blacklist (skip if 'Insufficient') with whitelist (fast-fail ONLY
if message matches PERMANENT_SCHEDULING_PATTERNS):
* node(s) didn't match Pod's node affinity/selector
* node(s) didn't match node affinity
* node(s) had untolerated taint
Unknown messages, absent messages, resource shortages all → queue to timeout.
- Add isPermanentSchedulingFailure() / getPermanentSchedulingPatterns()
(replaces isTransientSchedulingFailure). Extendable at runtime via
ACTIONS_RUNNER_K8S_PERMANENT_SCHEDULING_PATTERNS env var (CSV regexes).
- Apply same whitelist logic to getPodConditionErrors (Unschedulable).
- Remove dead pre-scan block in getPodEventErrors.
- Wrap readPod() in waitForPodPhases with try/catch: transient read failures
now log a warning and back off rather than crashing the loop. Timeout
produces a message that includes the read error.
- 48 tests, all passing.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… hints - Remove FailedScheduling from UNRECOVERABLE_EVENT_REASONS entirely; scheduling failures are always transient resource waits, never fast-fail - Remove PERMANENT_SCHEDULING_PATTERNS, isPermanentSchedulingFailure, getPermanentSchedulingPatterns, getPodConditionErrors (Unschedulable) - Add getWaitingReasonHint: ImagePullBackOff/ErrImagePull with registry, imagePullSecret, and network connectivity guidance; others with targeted hints - Add getTerminatedReasonHint: Error reason split by exit code (137=SIGKILL, 139=SIGSEGV, 126=no exec permission, 127=command not found, other=generic) - Add getEventReasonHint: FailedMount and FailedBinding with kubectl checklist - Update tests: 47 passing, use toContain assertions resilient to hint wording Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Longwt123
force-pushed
the
fix/scheduling-fast-fail
branch
from
9025d3e to
49028a2
Compare
NotificationThis pull request source branch has changed, so removes the following label(s): gate_check_pass. |
CLA Signature PassLongwt123, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
|
|||||||||||||||||
2 participants
描述
问题：FailedScheduling 被无差别 fast-fail，资源不足排队也立即报错。
根源：UNRECOVERABLE_EVENT_REASONS 含 FailedScheduling，但 FailedScheduling 有两种：
修改：
测试：
61 测试全过
相关 Issue
resolve https://github.com/opensourceways/backlog/issues/1203
变更类型