docs: add spec & test shape-review checklist (presence vs shape gap)#56
Merged
Conversation
The bidirectional audit gate (scripts/check-spec-coverage.sh +
tests/meta/playwright-practices.spec.ts) enforces PRESENCE — every
requirement has a tag, every contract-bearing test has a tag pointing
at a requirement. That's load-bearing.
What the gate does NOT enforce is SHAPE — whether the test's
assertion actually fails when the contract breaks. Four distinct
shape bugs have shipped through the gate in this repo:
1. workspace-auto-join-independence — asserted "no UUID collision
across 4 independent DBs," structurally impossible to fail.
2. jwt-algorithm-confusion — toBeGreaterThanOrEqual(200), every
HTTP status is >= 200, tautology.
3. login-logout-flow — @SPEC tag pointed at a contract the
file's tests don't exercise.
4. cookie-bomb-dos — required HTTP 4xx, but the bundle's
fail-closed shape is 302-to-IDP; 6/7 entry points failed CI
while behaving correctly.
Each passed the automated gate. None would have shipped if the
author had walked a structured shape-review checklist.
Adds docs/spec-review-checklist.md:
- Part A: 5-item checklist for adding/editing a `### Requirement:`
(strong verb, observable, no implementation coupling, scenario
block, failure mode named).
- Part B: 6-item checklist for adding/editing an @spec-tagged
assertion (tag points at exercised contract, counterfactual
exists, assertion can fail, range assertions enumerate the
acceptable set, per-app generalisation handled, precondition
does not bake the contract).
- Worked examples: each of the four shape bugs walked through
the checklist as the author/reviewer would have.
- Smell tests: three quick questions that catch most cases.
Adds a pointer from CLAUDE.md (so Claude sessions consult it before
editing specs / @spec-tagged tests) and from README.md (so human
PR authors land on it).
No tooling change. Pure discipline document — the enforcement is
"reviewer walks the checklist out loud in the PR comments before
approving."
Audit + typecheck remain clean (88 covered/deferred/missing total
unchanged).
awais786
added a commit
that referenced
this pull request
Jun 3, 2026
CI surfaced one false positive after merging: CLAUDE.md line 21
contains the backtick literal `git log -- tests/`. The path-detection
heuristic was "contains a slash AND ends in .ext or /" — `tests/`
matched that even though the full literal is a shell command, not a
file path.
Tightening the heuristic to also require no whitespace inside the
backtick literal catches this class without losing real coverage:
file paths in prose never contain spaces (they're either single
tokens like `tests/auth/foo.spec.ts` or markdown links). Shell
commands, prose phrases ("`git log -- tests/`", "`cd tests/`"), and
sentence fragments all do contain spaces.
Verified against the CI-surfaced case + the two skills.md references
to docs/spec-review-checklist.md (which now exist on origin/main
after PR #56 merged but not yet on this branch's local main).
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bidirectional audit gate (scripts/check-spec-coverage.sh + tests/meta/playwright-practices.spec.ts) enforces PRESENCE — every requirement has a tag, every contract-bearing test has a tag pointing at a requirement. That's load-bearing.
What the gate does NOT enforce is SHAPE — whether the test's assertion actually fails when the contract breaks. Four distinct shape bugs have shipped through the gate in this repo:
Each passed the automated gate. None would have shipped if the author had walked a structured shape-review checklist.
Adds docs/spec-review-checklist.md:
### Requirement:(strong verb, observable, no implementation coupling, scenario block, failure mode named).Adds a pointer from CLAUDE.md (so Claude sessions consult it before editing specs / @spec-tagged tests) and from README.md (so human PR authors land on it).
No tooling change. Pure discipline document — the enforcement is "reviewer walks the checklist out loud in the PR comments before approving."
Audit + typecheck remain clean (88 covered/deferred/missing total unchanged).
Motivation / Background
This Pull Request has been created because:
Detail
This Pull Request changes:
Additional information
TIP: Provide additional information such as screenshots, benchmarks, reference to other repositories or alternative solutions
Checklist
Before submitting the PR make sure the following are checked:
[Fix #issue-number]