Skip to content

build: auto-generate docs/spec-coverage.md from the audit inputs#60

Merged
awais786 merged 1 commit into
mainfrom
auto-gen-spec-coverage
Jun 4, 2026
Merged

build: auto-generate docs/spec-coverage.md from the audit inputs#60
awais786 merged 1 commit into
mainfrom
auto-gen-spec-coverage

Conversation

@awais786

@awais786 awais786 commented Jun 4, 2026

Copy link
Copy Markdown
Collaborator

docs/spec-coverage.md was the highest-drift hand-maintained file in the repo's doc set. Every PR that added a test, retitled a requirement, or moved an entry in spec-coverage-deferred.md needed a matching hand-edit. The counts went stale routinely (we caught the "59 of 84" line in skills.md during PR #55; the audit table itself drifted more silently).

What

Adds scripts/gen-spec-coverage.sh — a pure-bash generator that reads the same inputs the audit reads:

  • Per-module ### Requirement: blocks from vendor/openspec/specs/*/spec.md and vendor/openspec/skills/*/SKILL.md
  • // @spec module#slug tags across tests/
  • Categorised deferred entries in docs/spec-coverage-deferred.md

…and emits the full markdown table to stdout. Coverage state mirrors the audit, with one additional distinction:

  • ✅ — at least one @SPEC tag
  • 🟡 Partial — tagged AND the deferred doc carries a "Partially covered" note
  • ⚠️ Deferred — no tag; deferred entry with rationale
  • ❌ Missing — neither; CI fails on this state via the existing audit script (the generator emits it for honest reporting even though the audit blocks before it can ship)

Wiring

  • make spec-coverage-doc writes the regenerated file.
  • make check-spec-coverage-doc-fresh runs the generator and diff -u against the committed file. Fails the PR if stale.
  • Added to make pre-commit so local runs catch drift before push.
  • Added a "Check spec-coverage.md is fresh" step to .github/workflows/spec-coverage.yml. Same gate at CI time.

Slug normalisation fix

While writing the generator, found a case-mismatch between spec.md ("Layer 1 expiry SHALL ...") and the corresponding deferred entry ("... shall ..."). The audit handles this by slugifying both sides before comparison; an earlier generator draft compared raw titles and reported a false "❌ Missing" for that one entry. Fixed by slugifying titles in
collect_deferred_categorised + category_for_deferred. Audit and generator now agree.

Counts (this regeneration)

  • 61 ✅ Covered
  • 2 🟡 Partial
  • 25 ⚠️ Deferred
  • 0 ❌ Missing = 88 total

Sum (covered + partial = 63) matches the audit's "covered" count exactly. Deferred count matches. Total matches.

Out of scope (follow-ups)

  • The per-row nuance the old hand-curated file carried (e.g. "Outline observable; PM/Penpot/SurfSense/Twenty skip — no JS-readable session cookie") is dropped in this regeneration. That nuance belongs in the test file's head comment, which is the right place for it and where the test author already documents it. The audit table is now a clean index.

Motivation / Background

This Pull Request has been created because:

  • Resolves #issue-id

Detail

This Pull Request changes:

Additional information

TIP: Provide additional information such as screenshots, benchmarks, reference to other repositories or alternative solutions

Checklist

Before submitting the PR make sure the following are checked:

  • This Pull Request is related to one change. Changes that are unrelated should be opened in separate PRs
  • Commit message has a detailed description of what changed and why. If this PR fixes a related issue include it in the commit message. Ex: [Fix #issue-number]
  • Tests are added or updated if you fix a bug or add a feature
  • CHANGELOG files are updated for the behavior change or additional feature (minor bug fixes and documentation changes should not be included)
  • This PR contains API changes and API documentation is updated accordingly (for critical or behavior change, please inform related parties about them).

@github-actions

github-actions Bot commented Jun 4, 2026

Copy link
Copy Markdown

SSO Spec Coverage Audit

Contract source: vendor/openspec/specs/ (snapshot vendored in this repo)

Status Count
✅ Covered (test tag found) 65
⚠️ Deferred (in deferred doc) 23
❌ Missing (no tag, no defer) 0
Total requirements 88

All 88 spec requirements are accounted for.

docs/spec-coverage.md was the highest-drift hand-maintained file in
the repo's doc set. Every PR that added a test, retitled a
requirement, or moved an entry in spec-coverage-deferred.md needed
a matching hand-edit. The counts went stale routinely (we caught
the "59 of 84" line in skills.md during PR #55; the audit table
itself drifted more silently).

# What

Adds `scripts/gen-spec-coverage.sh` — a pure-bash generator that
reads the same inputs the audit reads:

  - Per-module `### Requirement:` blocks from
    `vendor/openspec/specs/*/spec.md` and
    `vendor/openspec/skills/*/SKILL.md`
  - `// @SPEC module#slug` tags across `tests/`
  - Categorised deferred entries in `docs/spec-coverage-deferred.md`

…and emits the full markdown table to stdout. Coverage state
mirrors the audit, with one additional distinction:

  - ✅ — at least one @SPEC tag
  - 🟡 Partial — tagged AND the deferred doc carries a
    "Partially covered" note
  - ⚠️ Deferred — no tag; deferred entry with rationale
  - ❌ Missing — neither; CI fails on this state via the existing
    audit script (the generator emits it for honest reporting
    even though the audit blocks before it can ship)

# Wiring

  - `make spec-coverage-doc` writes the regenerated file.
  - `make check-spec-coverage-doc-fresh` runs the generator and
    `diff -u` against the committed file. Fails the PR if stale.
  - Added to `make pre-commit` so local runs catch drift before
    push.
  - Added a "Check spec-coverage.md is fresh" step to
    `.github/workflows/spec-coverage.yml`. Same gate at CI time.

# Slug normalisation fix

While writing the generator, found a case-mismatch between
spec.md ("Layer 1 expiry SHALL ...") and the corresponding
deferred entry ("... shall ..."). The audit handles this by
slugifying both sides before comparison; an earlier generator
draft compared raw titles and reported a false "❌ Missing"
for that one entry. Fixed by slugifying titles in
collect_deferred_categorised + category_for_deferred. Audit and
generator now agree.

# Counts (this regeneration)

  - 61 ✅ Covered
  -  2 🟡 Partial
  - 25 ⚠️ Deferred
  -  0 ❌ Missing
  = 88 total

Sum (covered + partial = 63) matches the audit's "covered" count
exactly. Deferred count matches. Total matches.

# Out of scope (follow-ups)

  - The per-row nuance the old hand-curated file carried (e.g.
    "Outline observable; PM/Penpot/SurfSense/Twenty skip — no
    JS-readable session cookie") is dropped in this regeneration.
    That nuance belongs in the test file's head comment, which
    is the right place for it and where the test author already
    documents it. The audit table is now a clean index.
@awais786 awais786 force-pushed the auto-gen-spec-coverage branch from 3fdf888 to ded7893 Compare June 4, 2026 10:30
@awais786 awais786 merged commit cb1eddb into main Jun 4, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant