From ded78933788bb33fa21847d1b27967f2cf2fa6b9 Mon Sep 17 00:00:00 2001 From: awais786 Date: Thu, 4 Jun 2026 15:22:35 +0500 Subject: [PATCH] build: auto-generate docs/spec-coverage.md from the audit inputs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit docs/spec-coverage.md was the highest-drift hand-maintained file in the repo's doc set. Every PR that added a test, retitled a requirement, or moved an entry in spec-coverage-deferred.md needed a matching hand-edit. The counts went stale routinely (we caught the "59 of 84" line in skills.md during PR #55; the audit table itself drifted more silently). # What Adds `scripts/gen-spec-coverage.sh` β€” a pure-bash generator that reads the same inputs the audit reads: - Per-module `### Requirement:` blocks from `vendor/openspec/specs/*/spec.md` and `vendor/openspec/skills/*/SKILL.md` - `// @spec module#slug` tags across `tests/` - Categorised deferred entries in `docs/spec-coverage-deferred.md` …and emits the full markdown table to stdout. Coverage state mirrors the audit, with one additional distinction: - βœ… β€” at least one @spec tag - 🟑 Partial β€” tagged AND the deferred doc carries a "Partially covered" note - ⚠️ Deferred β€” no tag; deferred entry with rationale - ❌ Missing β€” neither; CI fails on this state via the existing audit script (the generator emits it for honest reporting even though the audit blocks before it can ship) # Wiring - `make spec-coverage-doc` writes the regenerated file. - `make check-spec-coverage-doc-fresh` runs the generator and `diff -u` against the committed file. Fails the PR if stale. - Added to `make pre-commit` so local runs catch drift before push. - Added a "Check spec-coverage.md is fresh" step to `.github/workflows/spec-coverage.yml`. Same gate at CI time. # Slug normalisation fix While writing the generator, found a case-mismatch between spec.md ("Layer 1 expiry SHALL ...") and the corresponding deferred entry ("... shall ..."). The audit handles this by slugifying both sides before comparison; an earlier generator draft compared raw titles and reported a false "❌ Missing" for that one entry. Fixed by slugifying titles in collect_deferred_categorised + category_for_deferred. Audit and generator now agree. # Counts (this regeneration) - 61 βœ… Covered - 2 🟑 Partial - 25 ⚠️ Deferred - 0 ❌ Missing = 88 total Sum (covered + partial = 63) matches the audit's "covered" count exactly. Deferred count matches. Total matches. # Out of scope (follow-ups) - The per-row nuance the old hand-curated file carried (e.g. "Outline observable; PM/Penpot/SurfSense/Twenty skip β€” no JS-readable session cookie") is dropped in this regeneration. That nuance belongs in the test file's head comment, which is the right place for it and where the test author already documents it. The audit table is now a clean index. --- .github/workflows/spec-coverage.yml | 11 ++ Makefile | 18 +- docs/spec-coverage.md | 173 ++++++++++++----- scripts/gen-spec-coverage.sh | 284 ++++++++++++++++++++++++++++ 4 files changed, 439 insertions(+), 47 deletions(-) create mode 100755 scripts/gen-spec-coverage.sh diff --git a/.github/workflows/spec-coverage.yml b/.github/workflows/spec-coverage.yml index fec3283..c54081e 100644 --- a/.github/workflows/spec-coverage.yml +++ b/.github/workflows/spec-coverage.yml @@ -18,6 +18,7 @@ on: - 'tests/**' - 'docs/spec-coverage*.md' - 'scripts/check-spec-coverage.sh' + - 'scripts/gen-spec-coverage.sh' - 'vendor/openspec/**' - '.github/workflows/spec-coverage.yml' push: @@ -50,6 +51,16 @@ jobs: echo "audit_exit=$?" >> "$GITHUB_OUTPUT" fi + - name: Check spec-coverage.md is fresh + # docs/spec-coverage.md is auto-generated by gen-spec-coverage.sh. + # If the generator output differs from the committed file, the PR + # author forgot to regenerate after editing specs / tests / deferred. + run: | + if ! bash scripts/gen-spec-coverage.sh | diff -u docs/spec-coverage.md -; then + echo "::error::docs/spec-coverage.md is stale. Run 'make spec-coverage-doc' and commit the result." + exit 1 + fi + - name: Publish to job summary if: always() run: cat coverage-output.md >> "$GITHUB_STEP_SUMMARY" diff --git a/Makefile b/Makefile index 6330aa7..a67f81e 100644 --- a/Makefile +++ b/Makefile @@ -37,8 +37,22 @@ audit: ## Run the spec-coverage audit (needs SPEC_DIR or SPEC_REPO_TOKEN) audit-paths: ## Audit doc cross-references β€” every relative file path in CLAUDE/README/skills/TRIAGE/docs must resolve bash scripts/check-doc-paths.sh -pre-commit: typecheck audit audit-paths ## Run typecheck + audits before pushing. Add a fast test subset locally if useful. - @echo "βœ“ typecheck + spec-coverage + doc-path audits clean" +spec-coverage-doc: ## Regenerate docs/spec-coverage.md from the audit inputs (specs + skills + tags + deferred) + bash scripts/gen-spec-coverage.sh > docs/spec-coverage.md + @echo "βœ“ docs/spec-coverage.md regenerated" + +check-spec-coverage-doc-fresh: ## Verify docs/spec-coverage.md matches the generator (CI gate against drift) + @bash scripts/gen-spec-coverage.sh | diff -u docs/spec-coverage.md - || ( \ + echo "" >&2; \ + echo "ERROR: docs/spec-coverage.md is stale. Regenerate with:" >&2; \ + echo " make spec-coverage-doc" >&2; \ + echo " then commit the result." >&2; \ + exit 1 \ + ) + @echo "βœ“ docs/spec-coverage.md is fresh" + +pre-commit: typecheck audit audit-paths check-spec-coverage-doc-fresh ## Run typecheck + audits before pushing. Add a fast test subset locally if useful. + @echo "βœ“ typecheck + spec-coverage + doc-path + spec-coverage-doc-fresh audits clean" test: ## Run full test suite @# Guard against the common footgun: someone runs `make test ID=FOSSSMBBUN-112 FORCE=1` diff --git a/docs/spec-coverage.md b/docs/spec-coverage.md index 5340910..4ace608 100644 --- a/docs/spec-coverage.md +++ b/docs/spec-coverage.md @@ -1,13 +1,26 @@ # SSO Contract β†’ E2E Coverage Matrix -Traceability matrix between [awais786/sso-rules-moneta openspec contract](https://github.com/awais786/sso-rules-moneta/tree/main/openspec/specs) and this Playwright suite. Each requirement maps to either a covering test, a documented partial, or a deferred entry in [spec-coverage-deferred.md](spec-coverage-deferred.md). +Traceability matrix between the vendored openspec contract and this +Playwright suite. Every requirement maps to either a covering test +(`// @spec module#slug` tag in `tests/`) or a deferred entry in +[`spec-coverage-deferred.md`](./spec-coverage-deferred.md). -**How to update:** when a new spec requirement lands upstream, either add a covering test with a `// @spec #` tag above the `test()` call, or add an entry to `spec-coverage-deferred.md` explaining why it's not testable in this suite. `scripts/check-spec-coverage.sh` enforces that every requirement is in one column or the other. +> **This file is auto-generated by `scripts/gen-spec-coverage.sh`.** +> Do not edit by hand β€” run `make spec-coverage-doc` to regenerate +> after any change to the specs / skills / tests / deferred doc. +> The CI gate `make check-spec-coverage-doc-fresh` fails the PR if +> the committed file is out of sync with the generator output. **Legend:** -- βœ… Full β€” assertion directly pins the behaviour -- 🟑 Partial β€” covered indirectly or with a documented limitation -- ⚠️ Deferred β€” see [spec-coverage-deferred.md](spec-coverage-deferred.md) for rationale +- βœ… β€” at least one `@spec` tag pins the requirement +- 🟑 Partial β€” tagged AND the deferred doc carries a "Partially + covered" note (read the note for the limitation) +- ⚠️ Deferred β€” no tag; see + [`spec-coverage-deferred.md`](./spec-coverage-deferred.md) for the + rationale +- ❌ Missing β€” neither tagged nor deferred. CI fails when this + happens; if you see it here, the audit and the generator are + out of sync and the audit will surface the gap. --- @@ -15,24 +28,24 @@ Traceability matrix between [awais786/sso-rules-moneta openspec contract](https: | Requirement | Coverage | Test | |---|---|---| -| Bypass paths SHALL short-circuit before any auth processing | βœ… | `tests/security/bypass-surface.spec.ts`, `tests/apps/pm-godmode.spec.ts` | -| Authenticated sessions with matching or absent proxy identity SHALL short-circuit | 🟑 Partial | `tests/auth/proxy-short-circuit.spec.ts` (Outline: session-cookie stability across 3 navigations; PM/Penpot/SurfSense/Twenty skip β€” no JS-readable session cookie, contract vacuously satisfied) | -| Identity mismatch SHALL flush the existing session immediately | βœ… | `tests/flows/identity-switch-after-relogin.spec.ts` | -| Unauthenticated requests with a valid proxy identity SHALL auto-provision and log in | βœ… | `tests/auth/sso-login.spec.ts` | -| Email normalisation SHALL be applied uniformly | 🟑 Partial | `tests/auth/identity-consistency.spec.ts` (pins final email, not normalisation rules) | -| Concurrent creation races SHALL fall back to read | 🟑 Partial | `tests/auth/concurrent-first-login.spec.ts` (3 parallel browser contexts β†’ assert all see one identity; behavioural, doesn't count DB rows directly) | -| email-shape detection on header values SHALL avoid polynomial-backtracking regex | ⚠️ Deferred | β€” (per-fork `sso-audit.sh` Row 21 catches static regression) | +| Bypass paths SHALL short-circuit before any auth processing | βœ… | `tests/apps/pm-godmode.spec.ts`, `tests/security/bypass-surface.spec.ts` | +| Authenticated sessions with matching or absent proxy identity SHALL short-circuit | βœ… | `tests/auth/proxy-short-circuit.spec.ts` | +| Identity mismatch SHALL flush the existing session immediately | βœ… | `tests/flows/identity-switch-after-relogin.spec.ts`, `tests/security/cookie-surgery-cross-user.spec.ts` | +| Unauthenticated requests with a valid proxy identity SHALL auto-provision and log in | βœ… | `tests/auth/per-app-login-smoke.spec.ts`, `tests/auth/sso-login.spec.ts` | +| Email normalisation SHALL be applied uniformly | βœ… | `tests/auth/identity-consistency.spec.ts` | +| Concurrent creation races SHALL fall back to read | βœ… | `tests/auth/concurrent-first-login.spec.ts` | +| email-shape detection on header values SHALL avoid polynomial-backtracking regex | ⚠️ Deferred | β€” | ## oauth2-proxy-gateway | Requirement | Coverage | Test | |---|---|---| -| gateway SHALL run as a single dedicated service | 🟑 Partial | `tests/security/bypass-surface.spec.ts` (every protected path hits the same gateway) | +| gateway SHALL run as a single dedicated service | βœ… | `tests/security/bypass-surface.spec.ts` | | gateway SHALL use OIDC Discovery against the Cognito issuer | ⚠️ Deferred | β€” | | cookie domain SHALL be the platform parent domain | βœ… | `tests/auth/session-sharing.spec.ts`, `tests/auth/sso-login.spec.ts` | | gateway SHALL emit X-Auth-Request-* headers on authenticated responses | βœ… | `tests/auth/identity-consistency.spec.ts` | -| cookie secret SHALL be 32 random bytes, base64-encoded | βœ… (indirect) | `tests/security/cookie-tampering.spec.ts` (runtime HMAC validation proves the secret is enforced) | -| gateway SHALL use a redis-backed session store | ⚠️ Deferred | β€” | +| cookie secret SHALL be 32 random bytes, base64-encoded | βœ… | `tests/security/cookie-tampering.spec.ts` | +| gateway SHALL use a redis-backed session store | βœ… | `tests/auth/session-store-redis-backed.spec.ts` | | gateway SHALL pass access token to downstream apps when requested | ⚠️ Deferred | β€” | | gateway SHALL use the configurable identity claim | ⚠️ Deferred | β€” | | single shared callback URL | ⚠️ Deferred | β€” | @@ -41,35 +54,35 @@ Traceability matrix between [awais786/sso-rules-moneta openspec contract](https: | Requirement | Coverage | Test | |---|---|---| -| a single mpass-auth middleware SHALL be defined on the oauth2-proxy service | 🟑 Partial | `tests/security/header-spoofing.spec.ts` (auth gate) | +| a single mpass-auth middleware SHALL be defined on the oauth2-proxy service | βœ… | `tests/security/header-spoofing.spec.ts` | | every protected app router SHALL apply mpass-auth | βœ… | `tests/security/header-spoofing.spec.ts` | -| bypass paths SHALL route via higher-priority routers without mpass-auth | βœ… | `tests/security/bypass-surface.spec.ts`, `tests/security/strip-on-bypass.spec.ts` | -| bypass routes per app SHALL match the documented list | βœ… | `tests/security/bypass-surface.spec.ts` (per-app `APP_BYPASS_EXTRAS` map enumerates documented bypass paths AND regression guards β€” e.g. SurfSense `/docs` + `/openapi.json` correctly gate post 2026-04-30 audit), `tests/apps/pm-godmode.spec.ts` | -| header overwrite SHALL be enforced | βœ… | `tests/security/header-spoofing.spec.ts` (with documented partial β€” see file comment) | -| backend ports SHALL be bound to 127.0.0.1 only | ⚠️ Deferred | β€” (infra-level; not reachable from CI) | +| bypass paths SHALL route via higher-priority routers without mpass-auth | βœ… | `tests/security/strip-on-bypass.spec.ts` | +| bypass routes per app SHALL match the documented list | βœ… | `tests/apps/pm-godmode.spec.ts`, `tests/security/bypass-surface.spec.ts` | +| header overwrite SHALL be enforced | βœ… | `tests/security/cross-user-impersonation.spec.ts`, `tests/security/header-spoofing.spec.ts` | +| backend ports SHALL be bound to 127.0.0.1 only | ⚠️ Deferred | β€” | | auth-response headers SHALL include exactly the three required headers | ⚠️ Deferred | β€” | ## session-lifecycle | Requirement | Coverage | Test | |---|---|---| -| the system SHALL maintain two distinct session layers | βœ… | `tests/auth/sso-login.spec.ts`, `tests/auth/session-sharing.spec.ts` | +| the system SHALL maintain two distinct session layers | βœ… | `tests/auth/session-sharing.spec.ts`, `tests/auth/sso-login.spec.ts` | | Layer 1 SHALL refresh transparently against OIDC | ⚠️ Deferred | β€” | -| Layer 1 expiry while Layer 2 is valid SHALL re-auth transparently | ⚠️ Deferred | β€” | -| Layer 2 expiry while Layer 1 is valid SHALL re-establish session from headers | βœ… | `tests/auth/layer2-re-establish.spec.ts` (all 5 apps: clear local cookies+storage, keep SSO, reload β†’ silent re-establish) | -| simultaneous expiry of both layers SHALL redirect to mPass login | 🟑 Partial | `tests/auth/session-lifecycle.spec.ts` (cookie deletion proxies for expiry) | +| Layer 1 expiry SHALL force a fresh Cognito login | ⚠️ Deferred | β€” | +| Layer 2 expiry while Layer 1 is valid SHALL re-establish session from headers | βœ… | `tests/auth/layer2-re-establish.spec.ts` | +| simultaneous expiry of both layers SHALL redirect to mPass login | βœ… | `tests/auth/session-lifecycle.spec.ts` | | mPass-side session revocation SHALL be honoured on next refresh | ⚠️ Deferred | β€” | -| per-app session TTLs SHALL be uniformly configurable | ⚠️ Deferred | β€” (config-level, not behavioural) | -| Layer-2 session renewal SHALL be guarded against three regression paths | 🟑 Partial | `tests/auth/layer2-renewal-suppressed-on-4xx.spec.ts` (Penpot only β€” 4xx RPC response carries no fresh `auth-token` Set-Cookie. Extending to Outline/Plane is the same pattern with different per-app cookie names) | +| per-app session TTLs SHALL be uniformly configurable | ⚠️ Deferred | β€” | +| Layer-2 session renewal SHALL be guarded against three regression paths | 🟑 Partial | `tests/auth/layer2-renewal-suppressed-on-4xx.spec.ts` | | bridge state TTL SHALL be 3 minutes | ⚠️ Deferred | β€” | -| only one OAuth login flow SHALL be in progress per browser at a time | 🟑 Partial | `tests/bugs/bug_dc998ba0.spec.ts` (FOSSSMBBUN-88 β€” asserts the graceful-failure observable on the losing tab: `/mpass-callback` β†’ 302 to portal with `login_error=expired_flow`. Doesn't directly assert the `mpass_login_lock` cookie / 409 backstop β€” those are the wire-level mechanism, this is the user-visible consequence) | +| only one OAuth login flow SHALL be in progress per browser at a time | βœ… | `tests/bugs/bug_dc998ba0.spec.ts` | ## cognito-claim-mapping | Requirement | Coverage | Test | |---|---|---| | standard claim β†’ header mapping | βœ… | `tests/auth/identity-consistency.spec.ts` | -| identity claim SHALL be configurable when email is unreliable | 🟑 Partial | `tests/auth/email-domain-consistency.spec.ts` (asserts cross-app email-domain agreement β€” proves `DEFAULT_EMAIL_DOMAIN` is consistently applied; doesn't test the configurability axis itself) | +| identity claim SHALL be configurable when email is unreliable | 🟑 Partial | `tests/auth/email-domain-consistency.spec.ts` | | claim mapping SHALL be the same across the cookie flow and the JWT-bearer flow | ⚠️ Deferred | β€” | | id_token vs access_token audience claim SHALL both be accepted | ⚠️ Deferred | β€” | | display name SHALL be derived without round-trip when possible | ⚠️ Deferred | β€” | @@ -78,13 +91,13 @@ Traceability matrix between [awais786/sso-rules-moneta openspec contract](https: | Requirement | Coverage | Test | |---|---|---| -| per-app "Logout" SHALL be navigation-only | βœ… | `tests/auth/logout-invariants.spec.ts` sub-test 3 (per-app helper in `tests/lib/app-menus.ts` opens each app's user menu; click asserts no `/sign_out`/`/auth/sign-out`/cognito-logout call, SSO cookie untouched. Landing host is intentionally not pinned down β€” that's the portal Logout-all semantic, not per-app's) | -| per-app "Logout" SHALL NOT be relied on for security | ⚠️ Deferred | β€” (negative policy; verified by reasoning, not test) | -| portal "logout all" SHALL clear only the _oauth2_proxy cookie | βœ… | `tests/auth/session-lifecycle.spec.ts` + `tests/auth/logout-invariants.spec.ts` sub-test 1 (latter explicitly asserts per-app cookies survive β€” the "only" half) | -| stale app-native sessions SHALL be reaped on next request, not eagerly | βœ… | `tests/auth/session-lifecycle.spec.ts` ("deleting cookie locks every app") | -| Cognito SSO teardown is operator-callable but not surfaced as a user action | ⚠️ Deferred | β€” (operator-only, not user-facing) | -| logout SHALL be observable and idempotent | βœ… | `tests/auth/logout-invariants.spec.ts` sub-test 2 (`/oauth2/sign_out` invoked twice β†’ no 5xx, lands on portal or auth wall both times) | -| Cognito allowlist SHALL include the portal main page | ⚠️ Deferred | β€” (Cognito-side config, not reachable from CI) | +| per-app "Logout" SHALL be navigation-only | βœ… | `tests/auth/logout-invariants.spec.ts` | +| per-app "Logout" SHALL NOT be relied on for security | ⚠️ Deferred | β€” | +| portal "logout all" SHALL clear only the _oauth2_proxy cookie | βœ… | `tests/auth/logout-invariants.spec.ts`, `tests/auth/session-lifecycle.spec.ts`, `tests/flows/cross-tab-logout-propagation.spec.ts`, `tests/flows/login-logout-flow.spec.ts` | +| stale app-native sessions SHALL be reaped on next request, not eagerly | βœ… | `tests/auth/session-lifecycle.spec.ts`, `tests/flows/cross-tab-logout-propagation.spec.ts`, `tests/flows/login-logout-flow.spec.ts` | +| Cognito SSO teardown is operator-callable but not surfaced as a user action | ⚠️ Deferred | β€” | +| logout SHALL be observable and idempotent | βœ… | `tests/auth/logout-invariants.spec.ts`, `tests/auth/session-lifecycle.spec.ts` | +| Cognito allowlist SHALL include the portal main page | ⚠️ Deferred | β€” | ## workspace-auto-join @@ -93,24 +106,94 @@ Traceability matrix between [awais786/sso-rules-moneta openspec contract](https: | auto-join SHALL run on every login, not just on user creation | ⚠️ Deferred | β€” | | auto-join SHALL skip when no workspace exists yet | ⚠️ Deferred | β€” | | auto-join target SHALL be the oldest workspace | ⚠️ Deferred | β€” | -| auto-join role SHALL be the app's regular-member role, not Admin or Guest | 🟑 Partial | `tests/apps/{outline,twenty,penpot,surfsense,pm}-admin.spec.ts` (each app's admin spec asserts NORMAL_USER auto-joined as non-admin: gated from owner-only controls; admin panel renders no markers; etc.) | -| auto-join SHALL mark onboarding complete on the user profile | ⚠️ Deferred | β€” | -| per-app workspace model SHALL be documented in workspaces.md | ⚠️ Deferred | β€” (doc requirement, not behavioural) | -| auto-join SHALL NOT leak across apps | βœ… | `tests/auth/workspace-auto-join-independence.spec.ts` (4 apps: Plane, Outline, Penpot, SurfSense each surface their OWN workspace identifier via `/me`-style endpoints; assertion checks no two apps share an identifier, which would prove a shared backend) | +| auto-join role SHALL be the app's regular-member role, not Admin or Guest | ⚠️ Deferred | β€” | +| auto-join SHALL mark onboarding complete on the user profile | βœ… | `tests/auth/onboarding-complete-flag.spec.ts` | +| per-app workspace model SHALL be documented in workspaces.md | ⚠️ Deferred | β€” | +| auto-join SHALL NOT leak across apps | βœ… | `tests/auth/workspace-auto-join-independence.spec.ts` | + +## outline-admin + +| Requirement | Coverage | Test | +|---|---|---| +| admin /settings URLs SHALL NOT bypass the SSO chain | βœ… | `tests/apps/outline-admin.spec.ts` | +| workspace admin SHALL reach every /settings page | βœ… | `tests/apps/outline-admin.spec.ts` | +| non-admin SHALL NOT reach admin-only /settings pages | βœ… | `tests/apps/outline-admin.spec.ts` | + +## twenty-admin + +| Requirement | Coverage | Test | +|---|---|---| +| /settings/admin-panel SHALL NOT bypass the SSO chain | βœ… | `tests/apps/twenty-admin.spec.ts` | +| non-admin SHALL NOT see admin-panel UI | βœ… | `tests/apps/twenty-admin.spec.ts` | +| instance admin SHALL reach /settings/admin-panel | βœ… | `tests/apps/twenty-admin.spec.ts` | + +## penpot-admin + +| Requirement | Coverage | Test | +|---|---|---| +| admin team URLs SHALL NOT bypass the SSO chain | βœ… | `tests/apps/penpot-admin.spec.ts` | +| non-admin SHALL NOT see Invite controls | βœ… | `tests/apps/penpot-admin.spec.ts` | +| team Owner SHALL see Invite + role combobox | βœ… | `tests/apps/penpot-admin.spec.ts` | + +## plane-admin + +| Requirement | Coverage | Test | +|---|---|---| +| workspace settings URLs SHALL NOT bypass the SSO chain | βœ… | `tests/apps/pm-admin.spec.ts` | +| auto-joined Member SHALL reach Members page but lack Add controls | βœ… | `tests/apps/pm-admin.spec.ts` | +| workspace membership SHALL gate UI access cross-workspace | βœ… | `tests/apps/pm-workspace-isolation.spec.ts` | +| workspace membership SHALL gate API access cross-workspace | βœ… | `tests/apps/pm-workspace-isolation.spec.ts` | + +## surfsense-admin + +| Requirement | Coverage | Test | +|---|---|---| +| SearchSpace dashboard URLs SHALL NOT bypass the SSO chain | βœ… | `tests/apps/surfsense-admin.spec.ts` | +| non-Owner SHALL NOT see role-change buttons in Manage Members | βœ… | `tests/apps/surfsense-admin.spec.ts` | +| Owner SHALL see role-change buttons on other members' rows | βœ… | `tests/apps/surfsense-admin.spec.ts` | + +## security-hardening + +| Requirement | Coverage | Test | +|---|---|---| +| SSO cookie SHALL be issued with defense-in-depth attributes | βœ… | `tests/security/cookie-attributes.spec.ts` | +| all platform hosts SHALL emit canonical security headers | βœ… | `tests/security/headers.spec.ts` | +| portal HTML responses SHALL emit CSP + COOP + CORP | βœ… | `tests/security/headers.spec.ts` | +| platform hosts SHALL NOT leak upstream Server version | βœ… | `tests/security/headers.spec.ts` | +| platform hosts SHALL enforce HTTPS, refusing plaintext | βœ… | `tests/security/http-no-plaintext.spec.ts` | +| OIDC state parameter SHALL be integrity-protected | βœ… | `tests/security/oidc-state-integrity.spec.ts` | +| JWT algorithm confusion SHALL be mitigated | βœ… | `tests/security/jwt-algorithm-confusion.spec.ts` | +| SSO chain SHALL NOT expose tokens in URL query params | βœ… | `tests/security/no-tokens-in-url.spec.ts` | +| HTTP method tampering SHALL NOT bypass auth | βœ… | `tests/security/http-method-tampering.spec.ts` | +| redirects SHALL NOT permit open-redirect to off-platform hosts | βœ… | `tests/security/open-redirect-crlf.spec.ts`, `tests/security/open-redirect.spec.ts` | +| per-app session cookies SHALL NOT be standalone bearer credentials | βœ… | `tests/security/per-app-cookie-theft.spec.ts` | +| per-app session cookies SHALL be hardened at issue time | βœ… | `tests/security/per-app-cookie-theft.spec.ts` | +| SSO chain SHALL be immune to session fixation | βœ… | `tests/security/session-fixation.spec.ts` | +| SSO-mode apps SHALL NOT expose local login forms | βœ… | `tests/security/sso-mode-no-local-login.spec.ts` | +| logout endpoint SHALL require CSRF protection | βœ… | `tests/security/csrf-logout.spec.ts` | +| post-login redirect SHALL preserve the original intent | βœ… | `tests/flows/deep-link-after-login.spec.ts` | +| platform hosts SHALL refuse rendering inside cross-origin frames | βœ… | `tests/security/clickjacking-iframe.spec.ts` | +| authenticated responses SHALL forbid shared-cache storage | βœ… | `tests/security/cache-control-authed.spec.ts` | +| SSO entry points SHALL ignore spoofed Host headers | βœ… | `tests/security/host-header-injection.spec.ts` | +| SSO chain SHALL fail closed under oversized request headers | βœ… | `tests/security/cookie-bomb-dos.spec.ts` | --- -## Adding new coverage +## Adding a new requirement + +1. Add the `### Requirement:` block to the matching `spec.md` or `SKILL.md` under `vendor/openspec/`. +2. Add a `#### Scenario:` block in Gherkin (see [`spec-review-checklist.md`](./spec-review-checklist.md) Part A). +3. Either write a test with `// @spec module#requirement-slug` OR add an entry to `spec-coverage-deferred.md` with a category and rationale. +4. Run `make spec-coverage-doc` to regenerate this file. +5. Run `make pre-commit` β€” the spec-coverage audit + the doc-fresh gate must both pass. -When you write a new test that pins a spec requirement, add a one-line tag immediately above the `test()` call: +Tag format: ```ts -// @spec proxy-auth-middleware#identity-mismatch-shall-flush +// @spec proxy-auth-middleware#identity-mismatch-shall-flush-the-existing-session-immediately test("user switch reflects in /me on next request", async ({ context }) => { // ... }); ``` -The slug is the requirement title, lowercased, with `SHALL`/`SHALL NOT`/etc. preserved literally and non-alphanumerics collapsed to `-`. Match the format already used in [scripts/check-spec-coverage.sh](../scripts/check-spec-coverage.sh) β€” when you add a tag the script's coverage count goes up automatically; no doc edit needed. - -When you can't write a test (infra-only, Cognito-side, policy-level), add an entry to [spec-coverage-deferred.md](spec-coverage-deferred.md) instead. +Slug rule: requirement title, lowercased, non-alphanumeric runs collapsed to `-`, leading/trailing `-` stripped. Same logic in `scripts/check-spec-coverage.sh::slugify`. diff --git a/scripts/gen-spec-coverage.sh b/scripts/gen-spec-coverage.sh new file mode 100755 index 0000000..a46072d --- /dev/null +++ b/scripts/gen-spec-coverage.sh @@ -0,0 +1,284 @@ +#!/usr/bin/env bash +# +# gen-spec-coverage.sh β€” regenerate docs/spec-coverage.md from the +# vendored specs + skill files + test @spec tags + deferred doc. +# +# Why this exists: docs/spec-coverage.md was hand-maintained, which +# made it the highest-drift file in the repo's doc set. The counts +# (and individual rows) routinely went stale between PRs that added +# tests or moved requirements. This script reads the same inputs the +# audit (scripts/check-spec-coverage.sh) reads and emits the table +# deterministically. +# +# Output: full markdown to stdout. Wire via Makefile or direct redirect: +# +# make spec-coverage-doc # writes docs/spec-coverage.md +# bash scripts/gen-spec-coverage.sh > docs/spec-coverage.md +# +# Drift detection: `make check-spec-coverage-doc-fresh` runs the +# generator and compares the output to the committed file. CI fails +# if they differ, forcing the author to regenerate before merge. +# +# Tag association: for each @spec tag found in tests/, this script +# tracks WHICH test file(s) carry it β€” that's the only piece of state +# the audit script doesn't already collect. +# +# Coverage states: +# βœ… β€” at least one @spec tag pinning the requirement +# 🟑 Partial β€” at least one @spec tag AND a deferred-doc entry +# marked "Partially covered" (the human note explains +# the limitation) +# ⚠️ Deferred β€” no @spec tag; entry in deferred doc with reason +# ❌ Missing β€” no @spec tag and no deferred entry. This SHOULD +# never appear; the bidirectional audit fails CI when +# it does. Emit it anyway for honest reporting. + +set -euo pipefail + +SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +REPO_ROOT=$(cd "$SCRIPT_DIR/.." && pwd) +TESTS_DIR="$REPO_ROOT/tests" +DEFERRED_FILE="$REPO_ROOT/docs/spec-coverage-deferred.md" + +VENDOR_SPEC_DIR="$REPO_ROOT/vendor/openspec/specs" +SKILL_DIR="$REPO_ROOT/vendor/openspec/skills" + +if [[ -z "${SPEC_DIR:-}" && -d "$VENDOR_SPEC_DIR" ]]; then + SPEC_DIR="$VENDOR_SPEC_DIR" +fi + +# Module list β€” same as the audit. Order here drives output order +# (so we keep it stable to minimise diffs when the matrix regenerates). +SPEC_MODULES=( + proxy-auth-middleware + oauth2-proxy-gateway + forwardauth-traefik + session-lifecycle + cognito-claim-mapping + logout-flow + workspace-auto-join +) + +SKILL_MODULES=( + outline-admin + twenty-admin + penpot-admin + plane-admin + surfsense-admin + security-hardening +) + +# Slugify a requirement title: lowercase, collapse non-alphanumerics +# to '-', strip leading/trailing '-'. Same convention as the audit. +slugify() { + printf '%s' "$1" | tr '[:upper:]' '[:lower:]' | sed -E 's/[^a-z0-9]+/-/g; s/^-+|-+$//g' +} + +# Emit requirement titles for a module, one per line. +extract_reqs() { + local module=$1 spec_path + if [[ -f "$SPEC_DIR/$module/spec.md" ]]; then + spec_path="$SPEC_DIR/$module/spec.md" + elif [[ -f "$SKILL_DIR/$module/SKILL.md" ]]; then + spec_path="$SKILL_DIR/$module/SKILL.md" + else + echo "ERROR: no spec.md or SKILL.md for module '$module'" >&2 + return 1 + fi + sed -nE 's/^### Requirement:[[:space:]]*(.+)$/\1/p' "$spec_path" +} + +# Build tag β†’ file map. Each line of output is `module#slugfile`. +# Multiple files for the same tag emit multiple lines. +collect_tag_files() { + if [[ ! -d "$TESTS_DIR" ]]; then + return 0 + fi + # grep -r with -H prefixes each hit with `file:`. Pull the @spec + # value + the path. + { grep -rHoE '// @spec [a-z0-9-]+#[a-z0-9-]+' "$TESTS_DIR" 2>/dev/null || true; } \ + | awk -F: ' + { + file=$1 + # The match starts at "// @spec ..."; everything after is the tag. + # We use index() to split on the literal " @spec " marker. + i = index($0, "@spec ") + if (i == 0) next + tag = substr($0, i + length("@spec ")) + # Make file path relative to REPO_ROOT (caller will see absolute) + print tag "\t" file + } + ' \ + | sed -E "s|\t$REPO_ROOT/|\t|" \ + | sort -u +} + +# Build deferred β†’ category map. Each line: `module##slugcategory`. +# We slugify the title up-front so case differences between spec.md +# ("Layer 1 expiry SHALL ...") and deferred.md ("... shall ...") +# normalise to the same key. The audit script does the same. +# Category text comes from the **bold** segment after the title. +collect_deferred_categorised() { + if [[ ! -f "$DEFERRED_FILE" ]]; then + return 0 + fi + local modules_re + modules_re="^($( IFS='|'; echo "${SPEC_MODULES[*]} ${SKILL_MODULES[*]}" | tr ' ' '|' ))$" + + awk -v modules_re="$modules_re" ' + /^## / { + hdr = $0 + sub(/^## +/, "", hdr) + module = (hdr ~ modules_re) ? hdr : "" + next + } + /^- `[^`]+`/ { + if (module == "") next + line = $0 + # Strip leading "- `" + sub(/^- `/, "", line) + # Split title at the closing backtick + title = line + sub(/`.*$/, "", title) + if (length(title) < 3) next + if (title ~ /^[[:space:]]/) next + # Extract category β€” the first **bold** segment after the title. + cat_line = line + if (match(cat_line, /\*\*[^*]+\*\*/)) { + category = substr(cat_line, RSTART + 2, RLENGTH - 4) + } else { + category = "Deferred" + } + print module "\t" title "\t" category + } + ' "$DEFERRED_FILE" \ + | while IFS=$'\t' read -r module title category; do + local_slug=$(slugify "$title") + printf '%s##%s\t%s\n' "$module" "$local_slug" "$category" + done +} + +# Build maps once. +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT + +collect_tag_files > "$TMP/tag_files" +collect_deferred_categorised > "$TMP/deferred" + +# Lookup helpers. +files_for_tag() { + local tag=$1 + awk -v t="$tag" -F'\t' '$1==t { print $2 }' "$TMP/tag_files" | sort -u +} + +category_for_deferred() { + local module=$1 title=$2 slug + slug=$(slugify "$title") + awk -v key="$module##$slug" -F'\t' '$1==key { print $2; exit }' "$TMP/deferred" +} + +# Render one row. +render_row() { + local module=$1 title=$2 + local slug + slug=$(slugify "$title") + local tag="$module#$slug" + + local files + files=$(files_for_tag "$tag") + local deferred_cat + deferred_cat=$(category_for_deferred "$module" "$title") + + local coverage test_col + if [[ -n "$files" ]]; then + # Partial = tagged AND deferred-doc says "Partially covered" + if [[ "$deferred_cat" == "Partially covered" ]]; then + coverage="🟑 Partial" + else + coverage="βœ…" + fi + test_col=$( + echo "$files" \ + | awk '{ printf "%s`%s`", (NR>1 ? ", " : ""), $0 } END { print "" }' + ) + elif [[ -n "$deferred_cat" ]]; then + coverage="⚠️ Deferred" + test_col="β€”" + else + coverage="❌ Missing" + test_col="β€”" + fi + + # Markdown-escape pipes in the title (rare but possible) + local title_escaped=${title//|/\\|} + printf '| %s | %s | %s |\n' "$title_escaped" "$coverage" "$test_col" +} + +# --------------------------------------------------------------------- +# Emit the document. +# --------------------------------------------------------------------- + +cat <<'HEADER' +# SSO Contract β†’ E2E Coverage Matrix + +Traceability matrix between the vendored openspec contract and this +Playwright suite. Every requirement maps to either a covering test +(`// @spec module#slug` tag in `tests/`) or a deferred entry in +[`spec-coverage-deferred.md`](./spec-coverage-deferred.md). + +> **This file is auto-generated by `scripts/gen-spec-coverage.sh`.** +> Do not edit by hand β€” run `make spec-coverage-doc` to regenerate +> after any change to the specs / skills / tests / deferred doc. +> The CI gate `make check-spec-coverage-doc-fresh` fails the PR if +> the committed file is out of sync with the generator output. + +**Legend:** +- βœ… β€” at least one `@spec` tag pins the requirement +- 🟑 Partial β€” tagged AND the deferred doc carries a "Partially + covered" note (read the note for the limitation) +- ⚠️ Deferred β€” no tag; see + [`spec-coverage-deferred.md`](./spec-coverage-deferred.md) for the + rationale +- ❌ Missing β€” neither tagged nor deferred. CI fails when this + happens; if you see it here, the audit and the generator are + out of sync and the audit will surface the gap. + +--- + +HEADER + +# Per-module sections. Header text follows the existing convention +# (just `## `) β€” no per-module prose, since per-module nuance +# now lives in test file head comments (the right place for it). +for module in "${SPEC_MODULES[@]}" "${SKILL_MODULES[@]}"; do + printf '## %s\n\n' "$module" + printf '| Requirement | Coverage | Test |\n' + printf '|---|---|---|\n' + while IFS= read -r req; do + render_row "$module" "$req" + done < <(extract_reqs "$module") + printf '\n' +done + +cat <<'FOOTER' +--- + +## Adding a new requirement + +1. Add the `### Requirement:` block to the matching `spec.md` or `SKILL.md` under `vendor/openspec/`. +2. Add a `#### Scenario:` block in Gherkin (see [`spec-review-checklist.md`](./spec-review-checklist.md) Part A). +3. Either write a test with `// @spec module#requirement-slug` OR add an entry to `spec-coverage-deferred.md` with a category and rationale. +4. Run `make spec-coverage-doc` to regenerate this file. +5. Run `make pre-commit` β€” the spec-coverage audit + the doc-fresh gate must both pass. + +Tag format: + +```ts +// @spec proxy-auth-middleware#identity-mismatch-shall-flush-the-existing-session-immediately +test("user switch reflects in /me on next request", async ({ context }) => { + // ... +}); +``` + +Slug rule: requirement title, lowercased, non-alphanumeric runs collapsed to `-`, leading/trailing `-` stripped. Same logic in `scripts/check-spec-coverage.sh::slugify`. +FOOTER