diff --git a/.github/workflows/spec-coverage.yml b/.github/workflows/spec-coverage.yml index fec3283..c54081e 100644 --- a/.github/workflows/spec-coverage.yml +++ b/.github/workflows/spec-coverage.yml @@ -18,6 +18,7 @@ on: - 'tests/**' - 'docs/spec-coverage*.md' - 'scripts/check-spec-coverage.sh' + - 'scripts/gen-spec-coverage.sh' - 'vendor/openspec/**' - '.github/workflows/spec-coverage.yml' push: @@ -50,6 +51,16 @@ jobs: echo "audit_exit=$?" >> "$GITHUB_OUTPUT" fi + - name: Check spec-coverage.md is fresh + # docs/spec-coverage.md is auto-generated by gen-spec-coverage.sh. + # If the generator output differs from the committed file, the PR + # author forgot to regenerate after editing specs / tests / deferred. + run: | + if ! bash scripts/gen-spec-coverage.sh | diff -u docs/spec-coverage.md -; then + echo "::error::docs/spec-coverage.md is stale. Run 'make spec-coverage-doc' and commit the result." + exit 1 + fi + - name: Publish to job summary if: always() run: cat coverage-output.md >> "$GITHUB_STEP_SUMMARY" diff --git a/Makefile b/Makefile index 6330aa7..a67f81e 100644 --- a/Makefile +++ b/Makefile @@ -37,8 +37,22 @@ audit: ## Run the spec-coverage audit (needs SPEC_DIR or SPEC_REPO_TOKEN) audit-paths: ## Audit doc cross-references — every relative file path in CLAUDE/README/skills/TRIAGE/docs must resolve bash scripts/check-doc-paths.sh -pre-commit: typecheck audit audit-paths ## Run typecheck + audits before pushing. Add a fast test subset locally if useful. - @echo "✓ typecheck + spec-coverage + doc-path audits clean" +spec-coverage-doc: ## Regenerate docs/spec-coverage.md from the audit inputs (specs + skills + tags + deferred) + bash scripts/gen-spec-coverage.sh > docs/spec-coverage.md + @echo "✓ docs/spec-coverage.md regenerated" + +check-spec-coverage-doc-fresh: ## Verify docs/spec-coverage.md matches the generator (CI gate against drift) + @bash scripts/gen-spec-coverage.sh | diff -u docs/spec-coverage.md - || ( \ + echo "" >&2; \ + echo "ERROR: docs/spec-coverage.md is stale. Regenerate with:" >&2; \ + echo " make spec-coverage-doc" >&2; \ + echo " then commit the result." >&2; \ + exit 1 \ + ) + @echo "✓ docs/spec-coverage.md is fresh" + +pre-commit: typecheck audit audit-paths check-spec-coverage-doc-fresh ## Run typecheck + audits before pushing. Add a fast test subset locally if useful. + @echo "✓ typecheck + spec-coverage + doc-path + spec-coverage-doc-fresh audits clean" test: ## Run full test suite @# Guard against the common footgun: someone runs `make test ID=FOSSSMBBUN-112 FORCE=1` diff --git a/docs/spec-coverage.md b/docs/spec-coverage.md index 5340910..4ace608 100644 --- a/docs/spec-coverage.md +++ b/docs/spec-coverage.md @@ -1,13 +1,26 @@ # SSO Contract → E2E Coverage Matrix -Traceability matrix between [awais786/sso-rules-moneta openspec contract](https://github.com/awais786/sso-rules-moneta/tree/main/openspec/specs) and this Playwright suite. Each requirement maps to either a covering test, a documented partial, or a deferred entry in [spec-coverage-deferred.md](spec-coverage-deferred.md). +Traceability matrix between the vendored openspec contract and this +Playwright suite. Every requirement maps to either a covering test +(`// @spec module#slug` tag in `tests/`) or a deferred entry in +[`spec-coverage-deferred.md`](./spec-coverage-deferred.md). -**How to update:** when a new spec requirement lands upstream, either add a covering test with a `// @spec #` tag above the `test()` call, or add an entry to `spec-coverage-deferred.md` explaining why it's not testable in this suite. `scripts/check-spec-coverage.sh` enforces that every requirement is in one column or the other. +> **This file is auto-generated by `scripts/gen-spec-coverage.sh`.** +> Do not edit by hand — run `make spec-coverage-doc` to regenerate +> after any change to the specs / skills / tests / deferred doc. +> The CI gate `make check-spec-coverage-doc-fresh` fails the PR if +> the committed file is out of sync with the generator output. **Legend:** -- ✅ Full — assertion directly pins the behaviour -- 🟡 Partial — covered indirectly or with a documented limitation -- ⚠️ Deferred — see [spec-coverage-deferred.md](spec-coverage-deferred.md) for rationale +- ✅ — at least one `@spec` tag pins the requirement +- 🟡 Partial — tagged AND the deferred doc carries a "Partially + covered" note (read the note for the limitation) +- ⚠️ Deferred — no tag; see + [`spec-coverage-deferred.md`](./spec-coverage-deferred.md) for the + rationale +- ❌ Missing — neither tagged nor deferred. CI fails when this + happens; if you see it here, the audit and the generator are + out of sync and the audit will surface the gap. --- @@ -15,24 +28,24 @@ Traceability matrix between [awais786/sso-rules-moneta openspec contract](https: | Requirement | Coverage | Test | |---|---|---| -| Bypass paths SHALL short-circuit before any auth processing | ✅ | `tests/security/bypass-surface.spec.ts`, `tests/apps/pm-godmode.spec.ts` | -| Authenticated sessions with matching or absent proxy identity SHALL short-circuit | 🟡 Partial | `tests/auth/proxy-short-circuit.spec.ts` (Outline: session-cookie stability across 3 navigations; PM/Penpot/SurfSense/Twenty skip — no JS-readable session cookie, contract vacuously satisfied) | -| Identity mismatch SHALL flush the existing session immediately | ✅ | `tests/flows/identity-switch-after-relogin.spec.ts` | -| Unauthenticated requests with a valid proxy identity SHALL auto-provision and log in | ✅ | `tests/auth/sso-login.spec.ts` | -| Email normalisation SHALL be applied uniformly | 🟡 Partial | `tests/auth/identity-consistency.spec.ts` (pins final email, not normalisation rules) | -| Concurrent creation races SHALL fall back to read | 🟡 Partial | `tests/auth/concurrent-first-login.spec.ts` (3 parallel browser contexts → assert all see one identity; behavioural, doesn't count DB rows directly) | -| email-shape detection on header values SHALL avoid polynomial-backtracking regex | ⚠️ Deferred | — (per-fork `sso-audit.sh` Row 21 catches static regression) | +| Bypass paths SHALL short-circuit before any auth processing | ✅ | `tests/apps/pm-godmode.spec.ts`, `tests/security/bypass-surface.spec.ts` | +| Authenticated sessions with matching or absent proxy identity SHALL short-circuit | ✅ | `tests/auth/proxy-short-circuit.spec.ts` | +| Identity mismatch SHALL flush the existing session immediately | ✅ | `tests/flows/identity-switch-after-relogin.spec.ts`, `tests/security/cookie-surgery-cross-user.spec.ts` | +| Unauthenticated requests with a valid proxy identity SHALL auto-provision and log in | ✅ | `tests/auth/per-app-login-smoke.spec.ts`, `tests/auth/sso-login.spec.ts` | +| Email normalisation SHALL be applied uniformly | ✅ | `tests/auth/identity-consistency.spec.ts` | +| Concurrent creation races SHALL fall back to read | ✅ | `tests/auth/concurrent-first-login.spec.ts` | +| email-shape detection on header values SHALL avoid polynomial-backtracking regex | ⚠️ Deferred | — | ## oauth2-proxy-gateway | Requirement | Coverage | Test | |---|---|---| -| gateway SHALL run as a single dedicated service | 🟡 Partial | `tests/security/bypass-surface.spec.ts` (every protected path hits the same gateway) | +| gateway SHALL run as a single dedicated service | ✅ | `tests/security/bypass-surface.spec.ts` | | gateway SHALL use OIDC Discovery against the Cognito issuer | ⚠️ Deferred | — | | cookie domain SHALL be the platform parent domain | ✅ | `tests/auth/session-sharing.spec.ts`, `tests/auth/sso-login.spec.ts` | | gateway SHALL emit X-Auth-Request-* headers on authenticated responses | ✅ | `tests/auth/identity-consistency.spec.ts` | -| cookie secret SHALL be 32 random bytes, base64-encoded | ✅ (indirect) | `tests/security/cookie-tampering.spec.ts` (runtime HMAC validation proves the secret is enforced) | -| gateway SHALL use a redis-backed session store | ⚠️ Deferred | — | +| cookie secret SHALL be 32 random bytes, base64-encoded | ✅ | `tests/security/cookie-tampering.spec.ts` | +| gateway SHALL use a redis-backed session store | ✅ | `tests/auth/session-store-redis-backed.spec.ts` | | gateway SHALL pass access token to downstream apps when requested | ⚠️ Deferred | — | | gateway SHALL use the configurable identity claim | ⚠️ Deferred | — | | single shared callback URL | ⚠️ Deferred | — | @@ -41,35 +54,35 @@ Traceability matrix between [awais786/sso-rules-moneta openspec contract](https: | Requirement | Coverage | Test | |---|---|---| -| a single mpass-auth middleware SHALL be defined on the oauth2-proxy service | 🟡 Partial | `tests/security/header-spoofing.spec.ts` (auth gate) | +| a single mpass-auth middleware SHALL be defined on the oauth2-proxy service | ✅ | `tests/security/header-spoofing.spec.ts` | | every protected app router SHALL apply mpass-auth | ✅ | `tests/security/header-spoofing.spec.ts` | -| bypass paths SHALL route via higher-priority routers without mpass-auth | ✅ | `tests/security/bypass-surface.spec.ts`, `tests/security/strip-on-bypass.spec.ts` | -| bypass routes per app SHALL match the documented list | ✅ | `tests/security/bypass-surface.spec.ts` (per-app `APP_BYPASS_EXTRAS` map enumerates documented bypass paths AND regression guards — e.g. SurfSense `/docs` + `/openapi.json` correctly gate post 2026-04-30 audit), `tests/apps/pm-godmode.spec.ts` | -| header overwrite SHALL be enforced | ✅ | `tests/security/header-spoofing.spec.ts` (with documented partial — see file comment) | -| backend ports SHALL be bound to 127.0.0.1 only | ⚠️ Deferred | — (infra-level; not reachable from CI) | +| bypass paths SHALL route via higher-priority routers without mpass-auth | ✅ | `tests/security/strip-on-bypass.spec.ts` | +| bypass routes per app SHALL match the documented list | ✅ | `tests/apps/pm-godmode.spec.ts`, `tests/security/bypass-surface.spec.ts` | +| header overwrite SHALL be enforced | ✅ | `tests/security/cross-user-impersonation.spec.ts`, `tests/security/header-spoofing.spec.ts` | +| backend ports SHALL be bound to 127.0.0.1 only | ⚠️ Deferred | — | | auth-response headers SHALL include exactly the three required headers | ⚠️ Deferred | — | ## session-lifecycle | Requirement | Coverage | Test | |---|---|---| -| the system SHALL maintain two distinct session layers | ✅ | `tests/auth/sso-login.spec.ts`, `tests/auth/session-sharing.spec.ts` | +| the system SHALL maintain two distinct session layers | ✅ | `tests/auth/session-sharing.spec.ts`, `tests/auth/sso-login.spec.ts` | | Layer 1 SHALL refresh transparently against OIDC | ⚠️ Deferred | — | -| Layer 1 expiry while Layer 2 is valid SHALL re-auth transparently | ⚠️ Deferred | — | -| Layer 2 expiry while Layer 1 is valid SHALL re-establish session from headers | ✅ | `tests/auth/layer2-re-establish.spec.ts` (all 5 apps: clear local cookies+storage, keep SSO, reload → silent re-establish) | -| simultaneous expiry of both layers SHALL redirect to mPass login | 🟡 Partial | `tests/auth/session-lifecycle.spec.ts` (cookie deletion proxies for expiry) | +| Layer 1 expiry SHALL force a fresh Cognito login | ⚠️ Deferred | — | +| Layer 2 expiry while Layer 1 is valid SHALL re-establish session from headers | ✅ | `tests/auth/layer2-re-establish.spec.ts` | +| simultaneous expiry of both layers SHALL redirect to mPass login | ✅ | `tests/auth/session-lifecycle.spec.ts` | | mPass-side session revocation SHALL be honoured on next refresh | ⚠️ Deferred | — | -| per-app session TTLs SHALL be uniformly configurable | ⚠️ Deferred | — (config-level, not behavioural) | -| Layer-2 session renewal SHALL be guarded against three regression paths | 🟡 Partial | `tests/auth/layer2-renewal-suppressed-on-4xx.spec.ts` (Penpot only — 4xx RPC response carries no fresh `auth-token` Set-Cookie. Extending to Outline/Plane is the same pattern with different per-app cookie names) | +| per-app session TTLs SHALL be uniformly configurable | ⚠️ Deferred | — | +| Layer-2 session renewal SHALL be guarded against three regression paths | 🟡 Partial | `tests/auth/layer2-renewal-suppressed-on-4xx.spec.ts` | | bridge state TTL SHALL be 3 minutes | ⚠️ Deferred | — | -| only one OAuth login flow SHALL be in progress per browser at a time | 🟡 Partial | `tests/bugs/bug_dc998ba0.spec.ts` (FOSSSMBBUN-88 — asserts the graceful-failure observable on the losing tab: `/mpass-callback` → 302 to portal with `login_error=expired_flow`. Doesn't directly assert the `mpass_login_lock` cookie / 409 backstop — those are the wire-level mechanism, this is the user-visible consequence) | +| only one OAuth login flow SHALL be in progress per browser at a time | ✅ | `tests/bugs/bug_dc998ba0.spec.ts` | ## cognito-claim-mapping | Requirement | Coverage | Test | |---|---|---| | standard claim → header mapping | ✅ | `tests/auth/identity-consistency.spec.ts` | -| identity claim SHALL be configurable when email is unreliable | 🟡 Partial | `tests/auth/email-domain-consistency.spec.ts` (asserts cross-app email-domain agreement — proves `DEFAULT_EMAIL_DOMAIN` is consistently applied; doesn't test the configurability axis itself) | +| identity claim SHALL be configurable when email is unreliable | 🟡 Partial | `tests/auth/email-domain-consistency.spec.ts` | | claim mapping SHALL be the same across the cookie flow and the JWT-bearer flow | ⚠️ Deferred | — | | id_token vs access_token audience claim SHALL both be accepted | ⚠️ Deferred | — | | display name SHALL be derived without round-trip when possible | ⚠️ Deferred | — | @@ -78,13 +91,13 @@ Traceability matrix between [awais786/sso-rules-moneta openspec contract](https: | Requirement | Coverage | Test | |---|---|---| -| per-app "Logout" SHALL be navigation-only | ✅ | `tests/auth/logout-invariants.spec.ts` sub-test 3 (per-app helper in `tests/lib/app-menus.ts` opens each app's user menu; click asserts no `/sign_out`/`/auth/sign-out`/cognito-logout call, SSO cookie untouched. Landing host is intentionally not pinned down — that's the portal Logout-all semantic, not per-app's) | -| per-app "Logout" SHALL NOT be relied on for security | ⚠️ Deferred | — (negative policy; verified by reasoning, not test) | -| portal "logout all" SHALL clear only the _oauth2_proxy cookie | ✅ | `tests/auth/session-lifecycle.spec.ts` + `tests/auth/logout-invariants.spec.ts` sub-test 1 (latter explicitly asserts per-app cookies survive — the "only" half) | -| stale app-native sessions SHALL be reaped on next request, not eagerly | ✅ | `tests/auth/session-lifecycle.spec.ts` ("deleting cookie locks every app") | -| Cognito SSO teardown is operator-callable but not surfaced as a user action | ⚠️ Deferred | — (operator-only, not user-facing) | -| logout SHALL be observable and idempotent | ✅ | `tests/auth/logout-invariants.spec.ts` sub-test 2 (`/oauth2/sign_out` invoked twice → no 5xx, lands on portal or auth wall both times) | -| Cognito allowlist SHALL include the portal main page | ⚠️ Deferred | — (Cognito-side config, not reachable from CI) | +| per-app "Logout" SHALL be navigation-only | ✅ | `tests/auth/logout-invariants.spec.ts` | +| per-app "Logout" SHALL NOT be relied on for security | ⚠️ Deferred | — | +| portal "logout all" SHALL clear only the _oauth2_proxy cookie | ✅ | `tests/auth/logout-invariants.spec.ts`, `tests/auth/session-lifecycle.spec.ts`, `tests/flows/cross-tab-logout-propagation.spec.ts`, `tests/flows/login-logout-flow.spec.ts` | +| stale app-native sessions SHALL be reaped on next request, not eagerly | ✅ | `tests/auth/session-lifecycle.spec.ts`, `tests/flows/cross-tab-logout-propagation.spec.ts`, `tests/flows/login-logout-flow.spec.ts` | +| Cognito SSO teardown is operator-callable but not surfaced as a user action | ⚠️ Deferred | — | +| logout SHALL be observable and idempotent | ✅ | `tests/auth/logout-invariants.spec.ts`, `tests/auth/session-lifecycle.spec.ts` | +| Cognito allowlist SHALL include the portal main page | ⚠️ Deferred | — | ## workspace-auto-join @@ -93,24 +106,94 @@ Traceability matrix between [awais786/sso-rules-moneta openspec contract](https: | auto-join SHALL run on every login, not just on user creation | ⚠️ Deferred | — | | auto-join SHALL skip when no workspace exists yet | ⚠️ Deferred | — | | auto-join target SHALL be the oldest workspace | ⚠️ Deferred | — | -| auto-join role SHALL be the app's regular-member role, not Admin or Guest | 🟡 Partial | `tests/apps/{outline,twenty,penpot,surfsense,pm}-admin.spec.ts` (each app's admin spec asserts NORMAL_USER auto-joined as non-admin: gated from owner-only controls; admin panel renders no markers; etc.) | -| auto-join SHALL mark onboarding complete on the user profile | ⚠️ Deferred | — | -| per-app workspace model SHALL be documented in workspaces.md | ⚠️ Deferred | — (doc requirement, not behavioural) | -| auto-join SHALL NOT leak across apps | ✅ | `tests/auth/workspace-auto-join-independence.spec.ts` (4 apps: Plane, Outline, Penpot, SurfSense each surface their OWN workspace identifier via `/me`-style endpoints; assertion checks no two apps share an identifier, which would prove a shared backend) | +| auto-join role SHALL be the app's regular-member role, not Admin or Guest | ⚠️ Deferred | — | +| auto-join SHALL mark onboarding complete on the user profile | ✅ | `tests/auth/onboarding-complete-flag.spec.ts` | +| per-app workspace model SHALL be documented in workspaces.md | ⚠️ Deferred | — | +| auto-join SHALL NOT leak across apps | ✅ | `tests/auth/workspace-auto-join-independence.spec.ts` | + +## outline-admin + +| Requirement | Coverage | Test | +|---|---|---| +| admin /settings URLs SHALL NOT bypass the SSO chain | ✅ | `tests/apps/outline-admin.spec.ts` | +| workspace admin SHALL reach every /settings page | ✅ | `tests/apps/outline-admin.spec.ts` | +| non-admin SHALL NOT reach admin-only /settings pages | ✅ | `tests/apps/outline-admin.spec.ts` | + +## twenty-admin + +| Requirement | Coverage | Test | +|---|---|---| +| /settings/admin-panel SHALL NOT bypass the SSO chain | ✅ | `tests/apps/twenty-admin.spec.ts` | +| non-admin SHALL NOT see admin-panel UI | ✅ | `tests/apps/twenty-admin.spec.ts` | +| instance admin SHALL reach /settings/admin-panel | ✅ | `tests/apps/twenty-admin.spec.ts` | + +## penpot-admin + +| Requirement | Coverage | Test | +|---|---|---| +| admin team URLs SHALL NOT bypass the SSO chain | ✅ | `tests/apps/penpot-admin.spec.ts` | +| non-admin SHALL NOT see Invite controls | ✅ | `tests/apps/penpot-admin.spec.ts` | +| team Owner SHALL see Invite + role combobox | ✅ | `tests/apps/penpot-admin.spec.ts` | + +## plane-admin + +| Requirement | Coverage | Test | +|---|---|---| +| workspace settings URLs SHALL NOT bypass the SSO chain | ✅ | `tests/apps/pm-admin.spec.ts` | +| auto-joined Member SHALL reach Members page but lack Add controls | ✅ | `tests/apps/pm-admin.spec.ts` | +| workspace membership SHALL gate UI access cross-workspace | ✅ | `tests/apps/pm-workspace-isolation.spec.ts` | +| workspace membership SHALL gate API access cross-workspace | ✅ | `tests/apps/pm-workspace-isolation.spec.ts` | + +## surfsense-admin + +| Requirement | Coverage | Test | +|---|---|---| +| SearchSpace dashboard URLs SHALL NOT bypass the SSO chain | ✅ | `tests/apps/surfsense-admin.spec.ts` | +| non-Owner SHALL NOT see role-change buttons in Manage Members | ✅ | `tests/apps/surfsense-admin.spec.ts` | +| Owner SHALL see role-change buttons on other members' rows | ✅ | `tests/apps/surfsense-admin.spec.ts` | + +## security-hardening + +| Requirement | Coverage | Test | +|---|---|---| +| SSO cookie SHALL be issued with defense-in-depth attributes | ✅ | `tests/security/cookie-attributes.spec.ts` | +| all platform hosts SHALL emit canonical security headers | ✅ | `tests/security/headers.spec.ts` | +| portal HTML responses SHALL emit CSP + COOP + CORP | ✅ | `tests/security/headers.spec.ts` | +| platform hosts SHALL NOT leak upstream Server version | ✅ | `tests/security/headers.spec.ts` | +| platform hosts SHALL enforce HTTPS, refusing plaintext | ✅ | `tests/security/http-no-plaintext.spec.ts` | +| OIDC state parameter SHALL be integrity-protected | ✅ | `tests/security/oidc-state-integrity.spec.ts` | +| JWT algorithm confusion SHALL be mitigated | ✅ | `tests/security/jwt-algorithm-confusion.spec.ts` | +| SSO chain SHALL NOT expose tokens in URL query params | ✅ | `tests/security/no-tokens-in-url.spec.ts` | +| HTTP method tampering SHALL NOT bypass auth | ✅ | `tests/security/http-method-tampering.spec.ts` | +| redirects SHALL NOT permit open-redirect to off-platform hosts | ✅ | `tests/security/open-redirect-crlf.spec.ts`, `tests/security/open-redirect.spec.ts` | +| per-app session cookies SHALL NOT be standalone bearer credentials | ✅ | `tests/security/per-app-cookie-theft.spec.ts` | +| per-app session cookies SHALL be hardened at issue time | ✅ | `tests/security/per-app-cookie-theft.spec.ts` | +| SSO chain SHALL be immune to session fixation | ✅ | `tests/security/session-fixation.spec.ts` | +| SSO-mode apps SHALL NOT expose local login forms | ✅ | `tests/security/sso-mode-no-local-login.spec.ts` | +| logout endpoint SHALL require CSRF protection | ✅ | `tests/security/csrf-logout.spec.ts` | +| post-login redirect SHALL preserve the original intent | ✅ | `tests/flows/deep-link-after-login.spec.ts` | +| platform hosts SHALL refuse rendering inside cross-origin frames | ✅ | `tests/security/clickjacking-iframe.spec.ts` | +| authenticated responses SHALL forbid shared-cache storage | ✅ | `tests/security/cache-control-authed.spec.ts` | +| SSO entry points SHALL ignore spoofed Host headers | ✅ | `tests/security/host-header-injection.spec.ts` | +| SSO chain SHALL fail closed under oversized request headers | ✅ | `tests/security/cookie-bomb-dos.spec.ts` | --- -## Adding new coverage +## Adding a new requirement + +1. Add the `### Requirement:` block to the matching `spec.md` or `SKILL.md` under `vendor/openspec/`. +2. Add a `#### Scenario:` block in Gherkin (see [`spec-review-checklist.md`](./spec-review-checklist.md) Part A). +3. Either write a test with `// @spec module#requirement-slug` OR add an entry to `spec-coverage-deferred.md` with a category and rationale. +4. Run `make spec-coverage-doc` to regenerate this file. +5. Run `make pre-commit` — the spec-coverage audit + the doc-fresh gate must both pass. -When you write a new test that pins a spec requirement, add a one-line tag immediately above the `test()` call: +Tag format: ```ts -// @spec proxy-auth-middleware#identity-mismatch-shall-flush +// @spec proxy-auth-middleware#identity-mismatch-shall-flush-the-existing-session-immediately test("user switch reflects in /me on next request", async ({ context }) => { // ... }); ``` -The slug is the requirement title, lowercased, with `SHALL`/`SHALL NOT`/etc. preserved literally and non-alphanumerics collapsed to `-`. Match the format already used in [scripts/check-spec-coverage.sh](../scripts/check-spec-coverage.sh) — when you add a tag the script's coverage count goes up automatically; no doc edit needed. - -When you can't write a test (infra-only, Cognito-side, policy-level), add an entry to [spec-coverage-deferred.md](spec-coverage-deferred.md) instead. +Slug rule: requirement title, lowercased, non-alphanumeric runs collapsed to `-`, leading/trailing `-` stripped. Same logic in `scripts/check-spec-coverage.sh::slugify`. diff --git a/scripts/gen-spec-coverage.sh b/scripts/gen-spec-coverage.sh new file mode 100755 index 0000000..a46072d --- /dev/null +++ b/scripts/gen-spec-coverage.sh @@ -0,0 +1,284 @@ +#!/usr/bin/env bash +# +# gen-spec-coverage.sh — regenerate docs/spec-coverage.md from the +# vendored specs + skill files + test @spec tags + deferred doc. +# +# Why this exists: docs/spec-coverage.md was hand-maintained, which +# made it the highest-drift file in the repo's doc set. The counts +# (and individual rows) routinely went stale between PRs that added +# tests or moved requirements. This script reads the same inputs the +# audit (scripts/check-spec-coverage.sh) reads and emits the table +# deterministically. +# +# Output: full markdown to stdout. Wire via Makefile or direct redirect: +# +# make spec-coverage-doc # writes docs/spec-coverage.md +# bash scripts/gen-spec-coverage.sh > docs/spec-coverage.md +# +# Drift detection: `make check-spec-coverage-doc-fresh` runs the +# generator and compares the output to the committed file. CI fails +# if they differ, forcing the author to regenerate before merge. +# +# Tag association: for each @spec tag found in tests/, this script +# tracks WHICH test file(s) carry it — that's the only piece of state +# the audit script doesn't already collect. +# +# Coverage states: +# ✅ — at least one @spec tag pinning the requirement +# 🟡 Partial — at least one @spec tag AND a deferred-doc entry +# marked "Partially covered" (the human note explains +# the limitation) +# ⚠️ Deferred — no @spec tag; entry in deferred doc with reason +# ❌ Missing — no @spec tag and no deferred entry. This SHOULD +# never appear; the bidirectional audit fails CI when +# it does. Emit it anyway for honest reporting. + +set -euo pipefail + +SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +REPO_ROOT=$(cd "$SCRIPT_DIR/.." && pwd) +TESTS_DIR="$REPO_ROOT/tests" +DEFERRED_FILE="$REPO_ROOT/docs/spec-coverage-deferred.md" + +VENDOR_SPEC_DIR="$REPO_ROOT/vendor/openspec/specs" +SKILL_DIR="$REPO_ROOT/vendor/openspec/skills" + +if [[ -z "${SPEC_DIR:-}" && -d "$VENDOR_SPEC_DIR" ]]; then + SPEC_DIR="$VENDOR_SPEC_DIR" +fi + +# Module list — same as the audit. Order here drives output order +# (so we keep it stable to minimise diffs when the matrix regenerates). +SPEC_MODULES=( + proxy-auth-middleware + oauth2-proxy-gateway + forwardauth-traefik + session-lifecycle + cognito-claim-mapping + logout-flow + workspace-auto-join +) + +SKILL_MODULES=( + outline-admin + twenty-admin + penpot-admin + plane-admin + surfsense-admin + security-hardening +) + +# Slugify a requirement title: lowercase, collapse non-alphanumerics +# to '-', strip leading/trailing '-'. Same convention as the audit. +slugify() { + printf '%s' "$1" | tr '[:upper:]' '[:lower:]' | sed -E 's/[^a-z0-9]+/-/g; s/^-+|-+$//g' +} + +# Emit requirement titles for a module, one per line. +extract_reqs() { + local module=$1 spec_path + if [[ -f "$SPEC_DIR/$module/spec.md" ]]; then + spec_path="$SPEC_DIR/$module/spec.md" + elif [[ -f "$SKILL_DIR/$module/SKILL.md" ]]; then + spec_path="$SKILL_DIR/$module/SKILL.md" + else + echo "ERROR: no spec.md or SKILL.md for module '$module'" >&2 + return 1 + fi + sed -nE 's/^### Requirement:[[:space:]]*(.+)$/\1/p' "$spec_path" +} + +# Build tag → file map. Each line of output is `module#slugfile`. +# Multiple files for the same tag emit multiple lines. +collect_tag_files() { + if [[ ! -d "$TESTS_DIR" ]]; then + return 0 + fi + # grep -r with -H prefixes each hit with `file:`. Pull the @spec + # value + the path. + { grep -rHoE '// @spec [a-z0-9-]+#[a-z0-9-]+' "$TESTS_DIR" 2>/dev/null || true; } \ + | awk -F: ' + { + file=$1 + # The match starts at "// @spec ..."; everything after is the tag. + # We use index() to split on the literal " @spec " marker. + i = index($0, "@spec ") + if (i == 0) next + tag = substr($0, i + length("@spec ")) + # Make file path relative to REPO_ROOT (caller will see absolute) + print tag "\t" file + } + ' \ + | sed -E "s|\t$REPO_ROOT/|\t|" \ + | sort -u +} + +# Build deferred → category map. Each line: `module##slugcategory`. +# We slugify the title up-front so case differences between spec.md +# ("Layer 1 expiry SHALL ...") and deferred.md ("... shall ...") +# normalise to the same key. The audit script does the same. +# Category text comes from the **bold** segment after the title. +collect_deferred_categorised() { + if [[ ! -f "$DEFERRED_FILE" ]]; then + return 0 + fi + local modules_re + modules_re="^($( IFS='|'; echo "${SPEC_MODULES[*]} ${SKILL_MODULES[*]}" | tr ' ' '|' ))$" + + awk -v modules_re="$modules_re" ' + /^## / { + hdr = $0 + sub(/^## +/, "", hdr) + module = (hdr ~ modules_re) ? hdr : "" + next + } + /^- `[^`]+`/ { + if (module == "") next + line = $0 + # Strip leading "- `" + sub(/^- `/, "", line) + # Split title at the closing backtick + title = line + sub(/`.*$/, "", title) + if (length(title) < 3) next + if (title ~ /^[[:space:]]/) next + # Extract category — the first **bold** segment after the title. + cat_line = line + if (match(cat_line, /\*\*[^*]+\*\*/)) { + category = substr(cat_line, RSTART + 2, RLENGTH - 4) + } else { + category = "Deferred" + } + print module "\t" title "\t" category + } + ' "$DEFERRED_FILE" \ + | while IFS=$'\t' read -r module title category; do + local_slug=$(slugify "$title") + printf '%s##%s\t%s\n' "$module" "$local_slug" "$category" + done +} + +# Build maps once. +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT + +collect_tag_files > "$TMP/tag_files" +collect_deferred_categorised > "$TMP/deferred" + +# Lookup helpers. +files_for_tag() { + local tag=$1 + awk -v t="$tag" -F'\t' '$1==t { print $2 }' "$TMP/tag_files" | sort -u +} + +category_for_deferred() { + local module=$1 title=$2 slug + slug=$(slugify "$title") + awk -v key="$module##$slug" -F'\t' '$1==key { print $2; exit }' "$TMP/deferred" +} + +# Render one row. +render_row() { + local module=$1 title=$2 + local slug + slug=$(slugify "$title") + local tag="$module#$slug" + + local files + files=$(files_for_tag "$tag") + local deferred_cat + deferred_cat=$(category_for_deferred "$module" "$title") + + local coverage test_col + if [[ -n "$files" ]]; then + # Partial = tagged AND deferred-doc says "Partially covered" + if [[ "$deferred_cat" == "Partially covered" ]]; then + coverage="🟡 Partial" + else + coverage="✅" + fi + test_col=$( + echo "$files" \ + | awk '{ printf "%s`%s`", (NR>1 ? ", " : ""), $0 } END { print "" }' + ) + elif [[ -n "$deferred_cat" ]]; then + coverage="⚠️ Deferred" + test_col="—" + else + coverage="❌ Missing" + test_col="—" + fi + + # Markdown-escape pipes in the title (rare but possible) + local title_escaped=${title//|/\\|} + printf '| %s | %s | %s |\n' "$title_escaped" "$coverage" "$test_col" +} + +# --------------------------------------------------------------------- +# Emit the document. +# --------------------------------------------------------------------- + +cat <<'HEADER' +# SSO Contract → E2E Coverage Matrix + +Traceability matrix between the vendored openspec contract and this +Playwright suite. Every requirement maps to either a covering test +(`// @spec module#slug` tag in `tests/`) or a deferred entry in +[`spec-coverage-deferred.md`](./spec-coverage-deferred.md). + +> **This file is auto-generated by `scripts/gen-spec-coverage.sh`.** +> Do not edit by hand — run `make spec-coverage-doc` to regenerate +> after any change to the specs / skills / tests / deferred doc. +> The CI gate `make check-spec-coverage-doc-fresh` fails the PR if +> the committed file is out of sync with the generator output. + +**Legend:** +- ✅ — at least one `@spec` tag pins the requirement +- 🟡 Partial — tagged AND the deferred doc carries a "Partially + covered" note (read the note for the limitation) +- ⚠️ Deferred — no tag; see + [`spec-coverage-deferred.md`](./spec-coverage-deferred.md) for the + rationale +- ❌ Missing — neither tagged nor deferred. CI fails when this + happens; if you see it here, the audit and the generator are + out of sync and the audit will surface the gap. + +--- + +HEADER + +# Per-module sections. Header text follows the existing convention +# (just `## `) — no per-module prose, since per-module nuance +# now lives in test file head comments (the right place for it). +for module in "${SPEC_MODULES[@]}" "${SKILL_MODULES[@]}"; do + printf '## %s\n\n' "$module" + printf '| Requirement | Coverage | Test |\n' + printf '|---|---|---|\n' + while IFS= read -r req; do + render_row "$module" "$req" + done < <(extract_reqs "$module") + printf '\n' +done + +cat <<'FOOTER' +--- + +## Adding a new requirement + +1. Add the `### Requirement:` block to the matching `spec.md` or `SKILL.md` under `vendor/openspec/`. +2. Add a `#### Scenario:` block in Gherkin (see [`spec-review-checklist.md`](./spec-review-checklist.md) Part A). +3. Either write a test with `// @spec module#requirement-slug` OR add an entry to `spec-coverage-deferred.md` with a category and rationale. +4. Run `make spec-coverage-doc` to regenerate this file. +5. Run `make pre-commit` — the spec-coverage audit + the doc-fresh gate must both pass. + +Tag format: + +```ts +// @spec proxy-auth-middleware#identity-mismatch-shall-flush-the-existing-session-immediately +test("user switch reflects in /me on next request", async ({ context }) => { + // ... +}); +``` + +Slug rule: requirement title, lowercased, non-alphanumeric runs collapsed to `-`, leading/trailing `-` stripped. Same logic in `scripts/check-spec-coverage.sh::slugify`. +FOOTER