Skip to content

ci: add debug workflow with -finit-real=snan + fcheck=all#136

Closed
k-yoshimi wants to merge 3 commits into
developfrom
ci/debug-build-snan-trap
Closed

ci: add debug workflow with -finit-real=snan + fcheck=all#136
k-yoshimi wants to merge 3 commits into
developfrom
ci/debug-build-snan-trap

Conversation

@k-yoshimi
Copy link
Copy Markdown
Owner

@k-yoshimi k-yoshimi commented Apr 20, 2026

Summary

Diagnostic CI workflow to surface the SIGABRT / SEGV crashes that the release CI (#135) currently skips. Build each `libapi.so` with gfortran runtime traps + signaling-NaN initialization, then run the full test suite with `--forked`. Log is uploaded as an artifact.

Phase 1 of the post-#135 follow-up plan discussed with user:
diagnose → fix → land per-bug PRs until release CI can run the full test suite without `--deselect`.

Key gfortran flags

```
OFLAGS = -g -O0 -m64 -std=legacy \
-fbounds-check -fcheck=all \
-ffpe-trap=invalid,zero,overflow -fbacktrace \
-finit-real=snan # read uninit real -> FPE trap
-finit-integer=-8888 # obvious sentinel
-finit-logical=false
-finit-derived # recursively init derived-type members
```

Trigger model

  • `workflow_dispatch`: manual run from the Actions tab
  • `pull_request labeled 'debug-ci'`: apply `debug-ci` label to any PR to run a diagnostic against that PR HEAD

Why `continue-on-error: true`

The whole point is to SURFACE crashes. Forcing green status would defeat the purpose. The step uploads `pytest-debug.log` as an artifact so backtraces are reviewable from the Actions UI.

Follow-up flow after merge

  1. Trigger this workflow on develop (`workflow_dispatch`)
  2. Review the uploaded log — expect to see signaling-NaN FPEs pinpointing the first uninit read
  3. File a one-line fix PR for that specific issue (`zero-init ARRAY after ALLOCATE` or similar)
  4. Repeat until the debug workflow passes the same tests that release CI passes
  5. Remove `--deselect` exclusions from release CI

Test plan

  • CI python-tests (release) stays green (this PR only adds a new workflow file)
  • Cursor Bugbot clean
  • After merge: trigger `python-tests-debug` manually and inspect first crash backtrace

🤖 Generated with Claude Code


Note

Low Risk
Low risk: it only adds a new, explicitly triggered CI workflow and does not affect the existing release python-tests job or production code paths.

Overview
Adds a new opt-in python-tests-debug GitHub Actions workflow that can be triggered manually or by applying a debug-ci label to a PR.

The job builds the Fortran module .so libraries using strict gfortran debug/runtime-trap flags (e.g., bounds checking, -fcheck=all, -finit-*=snan, FPE traps) and runs the full pytest suite with --forked, teeing output to pytest-debug.log and uploading it as an artifact for post-crash backtrace inspection.

Reviewed by Cursor Bugbot for commit 42f0748. Bugbot is set up for automated code reviews on this repo. Configure here.

Diagnostic workflow that builds libXapi.so with gfortran runtime
traps enabled and runs the tests that currently SIGABRT/SEGV in
the release CI. The goal is to surface uninit-read / array-bounds
violations with clean backtraces so they can be fixed one at a
time.

Key flags (overriding OFLAGS so everything built with $(OFLAGS)
inherits them):
  -fbounds-check -fcheck=all
  -ffpe-trap=invalid,zero,overflow -fbacktrace
  -finit-real=snan         (read-before-write -> FPE trap)
  -finit-integer=-8888     (obvious sentinel)
  -finit-logical=false
  -finit-derived           (recursively init derived-type members)
  -O0 -g                   (no optimization, full debug symbols)

Trigger model:
- workflow_dispatch: manual run from the Actions tab
- pull_request labeled 'debug-ci': apply the label to a PR for
  a one-shot diagnostic run

The pytest step sets continue-on-error=true and uploads the
log as an artifact — the whole point is to capture crash
backtraces, not to green-check.

Phase 1 of the post-#135 followup plan discussed with user:
diagnose -> fix -> land per-bug PRs until the release CI can
run the full test suite without --deselect.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@k-yoshimi
Copy link
Copy Markdown
Owner Author

@cursor review

Comment thread .github/workflows/python-tests-debug.yml
@k-yoshimi
Copy link
Copy Markdown
Owner Author

@cursor review

Bugbot caught that `2>&1 | tee pytest-debug.log` swallows
pytest's exit code (pipeline defaults to tee's 0), making
continue-on-error dead — the step always showed green in the
UI even when every test crashed. `set -o pipefail` + explicit
`shell: bash` fixes that so Actions displays red when pytest
exited non-zero, even while the job itself continues to
subsequent steps (the `continue-on-error: true` contract).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 42f0748. Configure here.

# way a fresh run is visually distinguishable from one where
# everything crashed cleanly. `set -o pipefail` ensures the
# `| tee` does not swallow pytest's exit code.
continue-on-error: true
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

continue-on-error contradicts stated goal of red status

Medium Severity

The step-level continue-on-error: true forces the overall job status to green/success in the Actions UI, directly contradicting the comment that says "the Actions UI shows red" and the PR description stating "Forcing green status would defeat the purpose." The if: always() on the upload step already ensures the artifact is uploaded after a failure, making continue-on-error: true both redundant and counterproductive — it hides the very crashes this workflow exists to surface.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 42f0748. Configure here.

Bugbot caught that step-level `continue-on-error: true` always
produces a green step status, directly contradicting the comment
and the PR description. The artifact upload has its own
`if: always()` guard, so removing `continue-on-error` does not
lose logs on failure — it just makes the Actions UI honestly
show red when pytest exits non-zero, which is the entire point
of the workflow.

`set -o pipefail` stays (so `| tee` does not swallow pytest's
exit code before it reaches the job status).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@k-yoshimi k-yoshimi added the debug-ci Apply to trigger python-tests-debug workflow on this PR label Apr 20, 2026
k-yoshimi added a commit that referenced this pull request Apr 20, 2026
Hypothesis test for the SIGABRT class tracked in docker/ci-repro/
and in the --deselect list in the release CI: on the developer
host (gfortran 13.3.0) the tests pass; in CI (gfortran 13.2.0,
the Ubuntu 24.04 default) a subset SIGABRTs inside the Fortran
finalize path. 14.x is the closest widely-available bump and, if
the problem is a 13.2-specific codegen / runtime bug, should make
it disappear.

Pulls gfortran-14 from `ppa:ubuntu-toolchain-r/test`, points
/usr/bin/gfortran via update-alternatives so every module
Makefile picks it up without per-file patches.

If this turns CI green with the --deselect list relaxed, the
SIGABRT was a 13.2 regression. If not, the Docker repro path
(#137) + debug workflow (#136) remain the way forward.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@k-yoshimi
Copy link
Copy Markdown
Owner Author

Superseded by #140 (root-cause BPSD species_kid OOB fix located via #139's valgrind diagnostic step). The valgrind/finit-NaN/gfortran-14 instrumentation was the path that found the bug; #140 lands the actual upstream-BPSD patch + restores all deselected Layer-1 tests.

@k-yoshimi k-yoshimi closed this Apr 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

debug-ci Apply to trigger python-tests-debug workflow on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant