Skip to content

CI eqdata-fixture drift detection for trlib/eqlib/totlib equivalence tests #200

@k-yoshimi

Description

@k-yoshimi

Context

PR #195 (#192) and PR #199 introduced committed eqdata binary fixtures under:

These are exact copies of test_run/test_output/<case>/eqdata* produced by the standalone eq.x / tot/tot Fortran binaries at the time of commit. They act as the silent-SKIP safety net for Layer 1 (1e-10) equivalence tests in the three Phase L Python wrappers.

Risk

If the upstream eq.x / tot/tot binary's EQSAVE format or numerics drift (e.g. compiler upgrade, eq subroutine refactor), the committed fixtures could go stale silently: the test would PASS against the OLD baseline + OLD fixture, but the staging-path eqdata (which the test prefers when present) would differ. CI staging currently catches "fixture missing" via test -s, but not "fixture stale".

Surfaced as low-probability LOW by in-house code reviewer on PR #199; same risk applies symmetrically to all 3 libs.

Possible remediations (not yet decided)

  • A. CI checksum gate: at the end of the CI staging step, cmp (or md5) the CI-generated eqdata against the committed fixture; fail loudly on mismatch.
  • B. Periodic regen workflow: tie into PR ci+eq+tr: workflow_dispatch baseline regen + graphics-stubs (#197) #198's workflow_dispatch regen path (see .github/workflows/regen-baselines.yml) so fixtures can be refreshed on a cadence or on demand.
  • C. Acceptance: declare the eq physics layer too stable for this to matter in practice. Reconsider only when a real drift is observed.

Out of scope

Source

In-house code reviewer's LOW L2 finding on PR #199 (totlib eqdata fallback polish).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions