build: auto-generate docs/spec-coverage.md from the audit inputs#60
Merged
Conversation
SSO Spec Coverage AuditContract source:
All 88 spec requirements are accounted for. |
docs/spec-coverage.md was the highest-drift hand-maintained file in the repo's doc set. Every PR that added a test, retitled a requirement, or moved an entry in spec-coverage-deferred.md needed a matching hand-edit. The counts went stale routinely (we caught the "59 of 84" line in skills.md during PR #55; the audit table itself drifted more silently). # What Adds `scripts/gen-spec-coverage.sh` — a pure-bash generator that reads the same inputs the audit reads: - Per-module `### Requirement:` blocks from `vendor/openspec/specs/*/spec.md` and `vendor/openspec/skills/*/SKILL.md` - `// @SPEC module#slug` tags across `tests/` - Categorised deferred entries in `docs/spec-coverage-deferred.md` …and emits the full markdown table to stdout. Coverage state mirrors the audit, with one additional distinction: - ✅ — at least one @SPEC tag - 🟡 Partial — tagged AND the deferred doc carries a "Partially covered" note -⚠️ Deferred — no tag; deferred entry with rationale - ❌ Missing — neither; CI fails on this state via the existing audit script (the generator emits it for honest reporting even though the audit blocks before it can ship) # Wiring - `make spec-coverage-doc` writes the regenerated file. - `make check-spec-coverage-doc-fresh` runs the generator and `diff -u` against the committed file. Fails the PR if stale. - Added to `make pre-commit` so local runs catch drift before push. - Added a "Check spec-coverage.md is fresh" step to `.github/workflows/spec-coverage.yml`. Same gate at CI time. # Slug normalisation fix While writing the generator, found a case-mismatch between spec.md ("Layer 1 expiry SHALL ...") and the corresponding deferred entry ("... shall ..."). The audit handles this by slugifying both sides before comparison; an earlier generator draft compared raw titles and reported a false "❌ Missing" for that one entry. Fixed by slugifying titles in collect_deferred_categorised + category_for_deferred. Audit and generator now agree. # Counts (this regeneration) - 61 ✅ Covered - 2 🟡 Partial - 25⚠️ Deferred - 0 ❌ Missing = 88 total Sum (covered + partial = 63) matches the audit's "covered" count exactly. Deferred count matches. Total matches. # Out of scope (follow-ups) - The per-row nuance the old hand-curated file carried (e.g. "Outline observable; PM/Penpot/SurfSense/Twenty skip — no JS-readable session cookie") is dropped in this regeneration. That nuance belongs in the test file's head comment, which is the right place for it and where the test author already documents it. The audit table is now a clean index.
3fdf888 to
ded7893
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
docs/spec-coverage.md was the highest-drift hand-maintained file in the repo's doc set. Every PR that added a test, retitled a requirement, or moved an entry in spec-coverage-deferred.md needed a matching hand-edit. The counts went stale routinely (we caught the "59 of 84" line in skills.md during PR #55; the audit table itself drifted more silently).
What
Adds
scripts/gen-spec-coverage.sh— a pure-bash generator that reads the same inputs the audit reads:### Requirement:blocks fromvendor/openspec/specs/*/spec.mdandvendor/openspec/skills/*/SKILL.md// @spec module#slugtags acrosstests/docs/spec-coverage-deferred.md…and emits the full markdown table to stdout. Coverage state mirrors the audit, with one additional distinction:
Wiring
make spec-coverage-docwrites the regenerated file.make check-spec-coverage-doc-freshruns the generator anddiff -uagainst the committed file. Fails the PR if stale.make pre-commitso local runs catch drift before push..github/workflows/spec-coverage.yml. Same gate at CI time.Slug normalisation fix
While writing the generator, found a case-mismatch between spec.md ("Layer 1 expiry SHALL ...") and the corresponding deferred entry ("... shall ..."). The audit handles this by slugifying both sides before comparison; an earlier generator draft compared raw titles and reported a false "❌ Missing" for that one entry. Fixed by slugifying titles in
collect_deferred_categorised + category_for_deferred. Audit and generator now agree.
Counts (this regeneration)
Sum (covered + partial = 63) matches the audit's "covered" count exactly. Deferred count matches. Total matches.
Out of scope (follow-ups)
Motivation / Background
This Pull Request has been created because:
Detail
This Pull Request changes:
Additional information
TIP: Provide additional information such as screenshots, benchmarks, reference to other repositories or alternative solutions
Checklist
Before submitting the PR make sure the following are checked:
[Fix #issue-number]