Skip to content

test: guard consumer-side verdict maps stay total over the enum#91

Merged
ericckzhou merged 1 commit into
devfrom
test/verdict-consumer-coverage
Jun 7, 2026
Merged

test: guard consumer-side verdict maps stay total over the enum#91
ericckzhou merged 1 commit into
devfrom
test/verdict-consumer-coverage

Conversation

@ericckzhou

Copy link
Copy Markdown
Owner

Summary:

  • Adds tests/meta/test_verdict_consumer_coverage.py, an anti-entropy guard that asserts the two consumer-side tables which turn a Verdict into a CI signal stay total over the enum.
  • render._EXIT_CODES must cover every Verdictexit_code_for indexes it directly (_EXIT_CODES[verdict]), so a verdict missing from the map is a KeyError crash the moment that verdict reaches the CLI. The contract is "one exit code your CI can gate on"; a crash is not an exit code.
  • Every verdict must map to a documented verdict-derived code (0/1/2/4); codes 3/5/6 belong to non-verdict surfaces (errors, diff) and a verdict must never resolve to one.
  • diff._QUALITY_RANK must partition the enum into ranked verdicts plus exactly the two deliberately off-ladder ones (INVALID_EVAL, INSUFFICIENT). _classify_transition reads the rank with .get(), so a new verdict missing from the map silently becomes OTHER_CHANGE and suppresses a regression signal — this forces a conscious rank-vs-off-ladder decision instead.

Validation:

  • uv run pytest tests/meta tests/unit -q → 720 passed.
  • Mutation check (in-process, no files touched): popping one verdict from each consumer map makes all three new assertions fail with actionable messages — confirms the guard bites and is not vacuously passing.
  • uv run ruff check / ruff format --check clean on the new file.

Notes:

  • Tests-only; no runtime behavior changes.
  • Closes the seam left open by the existing producer-side guard (test_resolver_branch_count.py) and the frozen-enum guard (test_verdict_models.py): the enum could grow and the resolver emit a new class while a downstream consumer map went silently out of sync, surfacing only in a user's CI.

The resolver's verdict growth is guarded (test_resolver_branch_count) and the
enum is frozen (test_verdict_models), but the consumer tables that turn a
Verdict into a CI signal had no totality guard. A new verdict class could be
wired into the enum + resolver while a downstream map silently went out of sync:

- render._EXIT_CODES is indexed directly, so a missing verdict KeyErrors at
  run time when that verdict reaches the CLI — the "one exit code your CI can
  gate on" contract becomes a crash.
- diff._QUALITY_RANK is read with .get(), so a missing verdict silently falls
  off the ladder and is reported as OTHER_CHANGE, suppressing a regression.

Add tests/meta/test_verdict_consumer_coverage.py: _EXIT_CODES must cover every
Verdict and map only to documented verdict-derived codes (0/1/2/4); _QUALITY_RANK
must partition the enum into ranked verdicts plus exactly the two deliberately
off-ladder ones (INVALID_EVAL, INSUFFICIENT). No runtime behavior changes.
@ericckzhou ericckzhou merged commit 0a47ccf into dev Jun 7, 2026
1 check passed
@ericckzhou ericckzhou deleted the test/verdict-consumer-coverage branch June 7, 2026 22:01
ericckzhou added a commit that referenced this pull request Jun 7, 2026
…num (#91) (#92)

The resolver's verdict growth is guarded (test_resolver_branch_count) and the
enum is frozen (test_verdict_models), but the consumer tables that turn a
Verdict into a CI signal had no totality guard. A new verdict class could be
wired into the enum + resolver while a downstream map silently went out of sync:

- render._EXIT_CODES is indexed directly, so a missing verdict KeyErrors at
  run time when that verdict reaches the CLI — the "one exit code your CI can
  gate on" contract becomes a crash.
- diff._QUALITY_RANK is read with .get(), so a missing verdict silently falls
  off the ladder and is reported as OTHER_CHANGE, suppressing a regression.

Add tests/meta/test_verdict_consumer_coverage.py: _EXIT_CODES must cover every
Verdict and map only to documented verdict-derived codes (0/1/2/4); _QUALITY_RANK
must partition the enum into ranked verdicts plus exactly the two deliberately
off-ladder ones (INVALID_EVAL, INSUFFICIENT). No runtime behavior changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant