From e1b70888c71f74353ffae9ce17228651abcf48c0 Mon Sep 17 00:00:00 2001 From: awais786 Date: Thu, 4 Jun 2026 01:30:40 +0500 Subject: [PATCH 1/3] =?UTF-8?q?build:=20doc-path=20audit=20=E2=80=94=20ver?= =?UTF-8?q?ify=20every=20prose=20file-ref=20points=20at=20a=20real=20file?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit # Why The bidirectional spec-coverage audit handles drift for tests ↔ specs. What it doesn't handle is drift for prose ↔ files: a file gets renamed, a doc gets moved to docs/archive/, a helper changes location — and CLAUDE.md / README.md / skills.md / TRIAGE.md / docs/*.md keep referencing the old paths long after the link is dead. PR #55 (docs-consolidation) made this concrete: archiving 3 historical files could have silently broken any cross-doc reference to them. The gate would not have caught it. # What Adds `scripts/check-doc-paths.sh` — a pure-bash audit: - Walks CLAUDE.md, README.md, skills.md, TRIAGE.md, TESTS.md, and every current docs/*.md (the archive dir + test-review snapshots are explicitly excluded — they're time-bound). - Extracts two kinds of file references: 1. Markdown-link targets: `[text](./path)` / `[text](path)` 2. Backtick literals that look like paths (contain `/` AND end in a known extension or `/`) - Resolves each candidate first relative to the doc, then relative to the repo root (the prose convention in docs/ is repo-root-relative). - Ignores: URLs, glob/brace patterns, placeholder interpolation, HTTP/API endpoint paths (`/api/users/me`, `/oauth2/sign_out`, `/god-mode/`, ...), generated artefacts (`playwright-report/`, `node_modules/`), and a small documented allowlist for cross-repo refs (foss-server-bundle/) + not-yet-built files (`.github/workflows/e2e-prod.yml` — referenced in README but the production workflow hasn't shipped). - Returns non-zero if any ref points at a missing file. Output lists each break as `:: missing → `. Wired into: - `make audit-paths` (callable on its own) - `make pre-commit` (alongside typecheck + spec-coverage audit) - `.github/workflows/doc-path-audit.yml` (runs on PRs that touch any *.md, the script, or its workflow; runs on push to main) # Fixed by this run The audit immediately surfaced 3 real broken refs in the existing docs: - CLAUDE.md:116 — `app-rules/RULES.md` → correct path is `vendor/openspec/skills/app-rules/RULES.md` - README.md:292 — `security/` → should be `tests/security/` - TRIAGE.md:41 — `security/headers.spec.ts` → should be `tests/security/headers.spec.ts` Each found and fixed in this commit. # Counts Final audit run: 270 references checked across 7 files, 0 missing. Pre-commit gate green (typecheck + spec-coverage + doc-path). # Out of scope (follow-up notes) - `.github/workflows/e2e-prod.yml` is referenced in README but not yet built. Added to IGNORED_PATTERNS with a TODO comment; remove the ignore once the production workflow ships. - `docs/test-review-*.md` and `docs/archive/*` are excluded by design (time-bound snapshots). If we want them audited too, revisit the exclusion list. --- .github/workflows/doc-path-audit.yml | 38 +++++ CLAUDE.md | 2 +- Makefile | 7 +- README.md | 2 +- TRIAGE.md | 2 +- scripts/check-doc-paths.sh | 247 +++++++++++++++++++++++++++ 6 files changed, 293 insertions(+), 5 deletions(-) create mode 100644 .github/workflows/doc-path-audit.yml create mode 100755 scripts/check-doc-paths.sh diff --git a/.github/workflows/doc-path-audit.yml b/.github/workflows/doc-path-audit.yml new file mode 100644 index 0000000..f9b5800 --- /dev/null +++ b/.github/workflows/doc-path-audit.yml @@ -0,0 +1,38 @@ +name: Doc-path audit + +# Verifies every relative file-path reference in CLAUDE.md, README.md, +# skills.md, TRIAGE.md, TESTS.md, and docs/*.md points at a file that +# actually exists in the repo. +# +# Failure mode this catches: a file gets renamed, moved, or archived, +# and a prose reference somewhere in the doc set still points at the +# old location. The bidirectional spec-coverage audit handles drift for +# tests ↔ specs; this one handles drift for prose ↔ files. +# +# Same lightweight shape as spec-coverage.yml — pure-bash script, no +# network, no node setup. + +on: + pull_request: + paths: + - '**/*.md' + - 'scripts/check-doc-paths.sh' + - '.github/workflows/doc-path-audit.yml' + push: + branches: [main] + workflow_dispatch: + +permissions: + contents: read + +jobs: + doc-path-audit: + runs-on: ubuntu-latest + permissions: + contents: read + steps: + - name: Checkout + uses: actions/checkout@v6 + + - name: Run doc-path audit + run: bash scripts/check-doc-paths.sh diff --git a/CLAUDE.md b/CLAUDE.md index 3a246ba..a53b65b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -113,7 +113,7 @@ structure and how to update it. | **IDP / mPass** | The unauthenticated landing on the IDP host is a **method picker** ("QR Code Login" / "Password Login") — NOT a direct username/password form. `cognitoLogin()` clicks "Password Login" internally before filling fields. Tests calling `cognitoLogin()` shouldn't pre-assert on `input[type="password"]` visibility before the helper runs — assert on the method picker (e.g. `getByText(/password login/i)`) or skip the pre-assertion and trust `cognitoLogin` to find the right form. | | **Multi-tab login mutex** | Only **one** OAuth login flow can be in progress per browser at a time (FOSSSMBBUN-88 / `session-lifecycle#single-flight`). The portal Login button reads a non-HttpOnly `mpass_login_lock` cookie set by `/authorize`; mpass-auth-proxy returns **409** as a server-side backstop. TTL = 600s. Tests that start a login but **don't complete it** must either complete or close the context before starting a second concurrent flow — otherwise the second `/authorize` 409s. The lock cookie is cleared on successful `/mpass-callback`, on failed `/mpass-callback` (any path), and on `/mpass/logout`. | | **Identities** | Two SSO users with an **asymmetric workspace surface**, NOT a role-split. Both auto-join the shared SMB workspace `fossarbisoft` as Member; FOSS_USER also belongs to additional workspaces/teams that NORMAL_USER doesn't. Verified via direct API probes against the live sandbox. **`FOSS_USER` ("User A", multi-workspace surface):** Plane Member of `fossarbisoft` + Admin of `aa` (`FOSS_USER_PRIVATE_WORKSPACE_SLUG`); Outline workspace admin of own team (UUID `1a2e0bad-…` per `OUTLINE_TEAM_ID`); Penpot Owner of personal "Default" team (UUID `c16a7502-…` per `PENPOT_TEAM_ID`) plus Editor on the SMB `fossarbisoft` team; SurfSense Owner of search space id 1 (`SURFSENSE_SEARCH_SPACE_ID`); Twenty `canAccessFullAdminPanel=true`. **`NORMAL_USER` ("User B", SMB-workspace-only surface):** Plane Member of `fossarbisoft` only (no `aa`); Outline workspace admin of own (different) team; Penpot Owner of own personal Default; no broader admin scope. **Use cases:** Where FOSS_USER and NORMAL_USER overlap in `fossarbisoft` they have the SAME role (both Member) — so the per-app `*-admin` tests are about WORKSPACE-LEVEL admin (FOSS_USER has it on `aa`, NORMAL_USER doesn't); about each user's own-team admin (vacuously equal); or about cross-workspace isolation (NORMAL_USER's UI/API access to `aa` must be refused). `tests/apps/pm-workspace-isolation.spec.ts` pins the isolation contract for Plane. `PLANE_ADMIN_USER`/`PLANE_ADMIN_PASS` is the local-creds god-mode admin (separate from SSO). `foss-server-bundle`'s `scripts/provision-admin/*` (PR #63) can promote a chosen email to SMB-workspace admin in each app; not currently run with FOSS_USER's email. | -| **Spec source — vendored** | The SSO openspec lives in-tree at `vendor/openspec/specs/`. The audit reads from there — no network or token at CI time. Updates land as normal PR diffs against the markdown files. `vendor/openspec/skills/` carries the per-app admin contracts (one `SKILL.md` per app) + the devstack `app-rules/RULES.md` — vendored as reference; the audit doesn't parse them yet (they don't use the `### Requirement:` format). | +| **Spec source — vendored** | The SSO openspec lives in-tree at `vendor/openspec/specs/`. The audit reads from there — no network or token at CI time. Updates land as normal PR diffs against the markdown files. `vendor/openspec/skills/` carries the per-app admin contracts (one `SKILL.md` per app) + the devstack `vendor/openspec/skills/app-rules/RULES.md` — vendored as reference; the audit doesn't parse them yet (they don't use the `### Requirement:` format). | ## What to work on (live) diff --git a/Makefile b/Makefile index 26ba6d6..6330aa7 100644 --- a/Makefile +++ b/Makefile @@ -34,8 +34,11 @@ typecheck: ## Run TypeScript type-check audit: ## Run the spec-coverage audit (needs SPEC_DIR or SPEC_REPO_TOKEN) bash scripts/check-spec-coverage.sh -pre-commit: typecheck audit ## Run typecheck + audit before pushing. Add a fast test subset locally if useful. - @echo "✓ typecheck + spec-coverage audit clean" +audit-paths: ## Audit doc cross-references — every relative file path in CLAUDE/README/skills/TRIAGE/docs must resolve + bash scripts/check-doc-paths.sh + +pre-commit: typecheck audit audit-paths ## Run typecheck + audits before pushing. Add a fast test subset locally if useful. + @echo "✓ typecheck + spec-coverage + doc-path audits clean" test: ## Run full test suite @# Guard against the common footgun: someone runs `make test ID=FOSSSMBBUN-112 FORCE=1` diff --git a/README.md b/README.md index b9eadd9..ba2a0a5 100644 --- a/README.md +++ b/README.md @@ -290,7 +290,7 @@ tests/ `FOSS_BASE_URL`. `skills.md` — local invariant contract (what every app must satisfy). [vendored openspec](./vendor/openspec/) — canonical edge-layer + -per-app rules organised by capability spec; the `security/` and +per-app rules organised by capability spec; the `tests/security/` and `identity-consistency` tests verify these on the live deployment. ## Auth architecture diff --git a/TRIAGE.md b/TRIAGE.md index 2d52e81..1954908 100644 --- a/TRIAGE.md +++ b/TRIAGE.md @@ -38,7 +38,7 @@ Don't triage the skipped tests. They're not the problem. Fix the smoke. | `per-app-login-smoke[]` returns 2xx but no usable email | App middleware authed the request but didn't synthesise the email — likely `DEFAULT_EMAIL_DOMAIN` misconfig | Check the app container's env. See `vendor/openspec/skills/app-rules/RULES.md` for the canonical pattern. | | `outline-admin.spec.ts /settings/` failed once, passed on retry | Outline's chunk-loader 429 burst flake | Known flake. Documented in `CLAUDE.md` → Deployment gotchas → Outline. Not actionable. | | `concurrent-first-login.spec.ts Outline: 3 parallel first-time visits resolve to ONE identity` | The race-condition test occasionally needs FOSS_USER to be a *fresh* user; if they're already provisioned in Outline, the test can flake | Re-run; if it stays red across re-runs, it's a real regression. | -| `security/headers.spec.ts` portal CSP / COOP / CORP / Server | Bundle-side; portal nginx config | Waiting on `foss-server-bundle#66`. Suite-side fix already in (`tests/security/headers.spec.ts` scoped to portal only). | +| `tests/security/headers.spec.ts` portal CSP / COOP / CORP / Server | Bundle-side; portal nginx config | Waiting on `foss-server-bundle#66`. Suite-side fix already in (`tests/security/headers.spec.ts` scoped to portal only). | | `tests/meta/playwright-practices.spec.ts` failed | Convention violation in a spec file | Read the failure — it names the file, line, and rule. Fix the spec, not the meta. | | `tests/meta/...` "every contract-bearing spec carries at least one @spec tag" failed | A new test file lacks an `@spec` tag and isn't in `UNTAGGED_ALLOWLIST` | Add `// @spec #` pointing at a vendored requirement, OR add a new `### Requirement:` to the matching SKILL.md / spec.md and tag against it, OR add the file to `UNTAGGED_ALLOWLIST` in the meta spec with a one-line rationale. | | `spec-coverage` check failed with `❌ Missing` rows | New requirement in `vendor/openspec/` but no test pins it | Either add a `// @spec module#slug` tag above the test that covers it, or add it to `docs/spec-coverage-deferred.md` with a category + rationale. | diff --git a/scripts/check-doc-paths.sh b/scripts/check-doc-paths.sh new file mode 100755 index 0000000..d6d528e --- /dev/null +++ b/scripts/check-doc-paths.sh @@ -0,0 +1,247 @@ +#!/usr/bin/env bash +# +# check-doc-paths.sh — verify every relative file-path reference in the +# repo's load-bearing docs points at a file that actually exists. +# +# Why this exists: CLAUDE.md, README.md, skills.md, TRIAGE.md, and the +# docs/ tree carry path references that decay silently. A spec file gets +# renamed, a doc gets moved to docs/archive/, a helper changes location +# — and the prose still says `tests/auth/foo.spec.ts` or +# `[skills.md](./skills.md)` long after the link is dead. The +# bidirectional spec-coverage audit handles drift for tests ↔ specs; +# this one handles drift for prose ↔ files. +# +# Scope: +# - Markdown links of shape `[text](./path)` or `[text](../path)` or +# `[text](path)` where the target looks like a file path (not URL). +# - Backtick-quoted refs that look like paths and have an extension +# (e.g. `tests/auth/sso-login.spec.ts`, `docs/spec-coverage.md`). +# +# Not in scope: +# - URLs (`https://...`, `http://...`, `mailto:`) +# - Wildcard / glob refs (e.g. `tests/auth/*.spec.ts`, `tests/**/*.ts`) +# - Backtick refs that don't have a slash AND extension (`Outline`, +# `auth-token`, `_oauth2_proxy`) +# - Refs whose path component is interpolated (e.g. +# `tests/apps/${app}-admin.spec.ts`) +# - Synthetic / generated paths (e.g. `playwright-report/`) +# +# Exit codes: +# 0 — every checked reference resolves +# 1 — at least one broken reference; details printed to stderr +# +# Add or remove input files via the `DOC_FILES` array below. The intent +# is to cover the docs a contributor lands on first (CLAUDE / README / +# skills / TRIAGE) and everything under docs/. To skip a specific +# false-positive add it to `IGNORED_PATTERNS`. + +set -euo pipefail + +SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +REPO_ROOT=$(cd "$SCRIPT_DIR/.." && pwd) + +# Docs to audit. Glob expansion happens inside the loop so a missing +# docs/ entry doesn't break anything. +DOC_FILES=( + "$REPO_ROOT/CLAUDE.md" + "$REPO_ROOT/README.md" + "$REPO_ROOT/skills.md" + "$REPO_ROOT/TRIAGE.md" + "$REPO_ROOT/TESTS.md" +) +# Include every CURRENT .md under docs/. The archive is excluded — +# its contents are time-bound snapshots that reference paths as they +# existed at the time of writing; auditing them would surface false +# positives whenever a file moves or is renamed. Same reason +# `test-review-*` snapshots aren't audited. +while IFS= read -r f; do + case "$f" in + */docs/archive/*) continue ;; + */docs/test-review-*) continue ;; + esac + DOC_FILES+=("$f") +done < <(find "$REPO_ROOT/docs" -name '*.md' 2>/dev/null | sort) + +# Patterns that look like file paths but aren't (won't be checked). +# Each line is an extended-regex applied to the candidate path. +IGNORED_PATTERNS=( + # External URLs + '^https?://' + '^mailto:' + # Glob patterns (something/*, **, etc.) + '\*' + # Brace expansion (e.g. `tests/apps/{outline,twenty,penpot}-admin.spec.ts`) + '\{[^}]*,' + # Placeholder interpolation + '\$\{' + '<[a-z-]+>' + # Anchors-only (e.g. "#section") + '^#' + # HTTP / API endpoint paths (server URLs, not files) — refs prose + # like `/api/users/me/`, `/oauth2/sign_out`, `/god-mode/`, + # `/auth/email`. These all start with a slash AND a known endpoint + # prefix. + '^/(api|oauth2|auth|sign_in|sign_out|signin|signup|sign-in|sign-up|users|workspaces|settings|home|search|drafts|archive|admin|admin-panel|callback|authorize|callback|god-mode|members|teams|invitations|onboarding|create-workspace|profile|dashboard|me)(/|$)' + # Generated artefacts that don't live in the repo + '^playwright-report/' + '^test-results/' + '^node_modules/' + # The .env file is user-created, not in the repo + '^\.env(\.example)?$' + # External repo paths (foss-server-bundle etc. — these are + # cross-repo doc pointers, not refs to files in this repo) + '^foss-server-bundle/' + # `data/` is a banned directory called out by CLAUDE.md + # ("Don't add data/ ..."). It's a negative-shaped reference; the + # file SHOULDN'T exist, and the audit shouldn't complain. + '^data/$' + # Example placeholders in the bug-spec format + '^tests/seed\.spec\.ts$' + # OpenAPI / generated JSON endpoints referenced in prose + '/openapi\.json$' + # Cross-repo refs into foss-server-bundle's provisioning scripts — + # prose mentions them ("foss-server-bundle's `scripts/provision-admin/plane.py`") + # without the bundle prefix, but the file lives in the other repo. + '^scripts/provision-admin/' + # `.github/workflows/e2e-prod.yml` is documented in README but not + # yet built (the production workflow is planned, not shipped). + # Remove this ignore once the file lands. + '^\.github/workflows/e2e-prod\.yml$' +) + +# Extract candidate paths from a single doc file. Emits one +# tab-separated line per candidate: +# +# \t +# +# Two patterns are matched: +# +# 1. Markdown link: `](./path)` or `](path)` — group is the part +# between `](` and `)`. +# 2. Backtick literal that looks like a path: contains `/` AND ends +# with a known extension OR is a dir-shaped ref ending in `/`. +extract_candidates() { + local file=$1 + # Pattern 1 — markdown links. + grep -nE '\]\(\.?\.?/?[^)]+\)' "$file" 2>/dev/null \ + | sed -E 's/^([0-9]+):.*\]\(([^)]+)\).*/\1\t\2/' \ + || true + + # Pattern 2 — backtick refs with slash and extension. + # The grep below pulls backtick literals that contain a slash AND + # either end in a common extension or end in a slash (dir). + awk -F: ' + { + line=$1; rest=substr($0, length(line)+2) + n = 0 + while (match(rest, /`[^`]+`/)) { + token = substr(rest, RSTART+1, RLENGTH-2) + # path-shaped: contains a slash and (ends in .ext or ends in /) + if (token ~ /\// && (token ~ /\.(md|ts|tsx|js|json|sh|yml|yaml|cfg|env|sql|py|nginx|conf|md|ipynb)$/ || token ~ /\/$/)) { + print line "\t" token + } + rest = substr(rest, RSTART + RLENGTH) + } + } + ' < <(grep -nE '`[^`]+`' "$file" 2>/dev/null) || true +} + +# Check whether a candidate (raw path string from the doc) resolves to +# an existing file/dir. Resolution is relative to the doc's directory. +# Returns 0 if exists, 1 if missing or unparseable. +# +# Emits the resolved path on success via stdout so the caller can use +# it for diagnostics; emits the cleaned raw path on missing so the +# error message is meaningful. +candidate_exists() { + local doc=$1 raw=$2 + # Strip trailing fragment (#section) and any quotes. + local clean=${raw%%#*} + clean=${clean%\"} + clean=${clean#\"} + clean=$(printf '%s' "$clean" | sed -E 's/[[:space:]]+$//') + if [[ -z "$clean" ]]; then + printf '%s' "$raw" + return 1 + fi + local doc_dir + doc_dir=$(dirname "$doc") + # Try two resolutions, in order, both via bash's path-tolerant [[ -e ]]: + # 1. doc-relative (the markdown-link convention — `./foo.md`, + # `../sibling/file.md`) + # 2. repo-root-relative (the prose convention in docs/ files — + # `tests/auth/foo.spec.ts` interpreted as `${REPO_ROOT}/tests/...`) + # Both are common in this repo; checking both eliminates false + # positives from refs in docs/ that mean repo-root paths. + if [[ "$clean" == /* ]]; then + # Absolute (rare) — accept as-is. + if [[ -e "$clean" ]]; then + printf '%s' "$clean"; return 0 + fi + else + if [[ -e "$doc_dir/$clean" ]]; then + printf '%s' "$doc_dir/$clean"; return 0 + fi + if [[ -e "$REPO_ROOT/$clean" ]]; then + printf '%s' "$REPO_ROOT/$clean"; return 0 + fi + fi + printf '%s' "$clean" + return 1 +} + +# Check whether the candidate matches any IGNORED_PATTERNS — i.e. +# should be skipped entirely. +is_ignored() { + local candidate=$1 + for pat in "${IGNORED_PATTERNS[@]}"; do + if [[ "$candidate" =~ $pat ]]; then + return 0 + fi + done + return 1 +} + +# Main loop. +total=0 +broken=0 +declare -a broken_lines=() + +for doc in "${DOC_FILES[@]}"; do + if [[ ! -f "$doc" ]]; then + continue + fi + rel_doc=${doc#"$REPO_ROOT/"} + while IFS=$'\t' read -r line_no raw_path; do + if [[ -z "$raw_path" ]]; then + continue + fi + if is_ignored "$raw_path"; then + continue + fi + total=$((total + 1)) + if candidate_exists "$doc" "$raw_path" >/dev/null; then + continue + fi + broken=$((broken + 1)) + broken_lines+=("$rel_doc:$line_no: missing → $raw_path") + done < <(extract_candidates "$doc" | sort -u) +done + +if (( broken == 0 )); then + echo "✓ doc-path audit clean — $total path references checked across ${#DOC_FILES[@]} files" + exit 0 +fi + +{ + echo "✗ doc-path audit — $broken / $total references point at missing files:" + echo "" + for line in "${broken_lines[@]}"; do + echo " $line" + done + echo "" + echo "Either restore the file, update the reference, or — if the" + echo "false positive is structural (wildcard, glob, placeholder) —" + echo "extend IGNORED_PATTERNS in scripts/check-doc-paths.sh." +} >&2 +exit 1 From 95c8d8e102b2ce6c2ba85f081ed74a0b1210d4fe Mon Sep 17 00:00:00 2001 From: awais786 Date: Thu, 4 Jun 2026 01:35:12 +0500 Subject: [PATCH 2/3] fix(check-doc-paths): skip backtick literals that contain whitespace MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI surfaced one false positive after merging: CLAUDE.md line 21 contains the backtick literal `git log -- tests/`. The path-detection heuristic was "contains a slash AND ends in .ext or /" — `tests/` matched that even though the full literal is a shell command, not a file path. Tightening the heuristic to also require no whitespace inside the backtick literal catches this class without losing real coverage: file paths in prose never contain spaces (they're either single tokens like `tests/auth/foo.spec.ts` or markdown links). Shell commands, prose phrases ("`git log -- tests/`", "`cd tests/`"), and sentence fragments all do contain spaces. Verified against the CI-surfaced case + the two skills.md references to docs/spec-review-checklist.md (which now exist on origin/main after PR #56 merged but not yet on this branch's local main). --- scripts/check-doc-paths.sh | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/scripts/check-doc-paths.sh b/scripts/check-doc-paths.sh index d6d528e..867969d 100755 --- a/scripts/check-doc-paths.sh +++ b/scripts/check-doc-paths.sh @@ -136,8 +136,10 @@ extract_candidates() { n = 0 while (match(rest, /`[^`]+`/)) { token = substr(rest, RSTART+1, RLENGTH-2) - # path-shaped: contains a slash and (ends in .ext or ends in /) - if (token ~ /\// && (token ~ /\.(md|ts|tsx|js|json|sh|yml|yaml|cfg|env|sql|py|nginx|conf|md|ipynb)$/ || token ~ /\/$/)) { + # path-shaped: contains a slash, has no whitespace (rules out + # shell commands like `git log -- tests/` and prose phrases), + # and either ends in .ext or ends in / (directory). + if (token ~ /\// && token !~ /[[:space:]]/ && (token ~ /\.(md|ts|tsx|js|json|sh|yml|yaml|cfg|env|sql|py|nginx|conf|md|ipynb)$/ || token ~ /\/$/)) { print line "\t" token } rest = substr(rest, RSTART + RLENGTH) From 5bf4d8bc444f69be10c516da40d013011d6231ed Mon Sep 17 00:00:00 2001 From: awais786 Date: Thu, 4 Jun 2026 01:38:43 +0500 Subject: [PATCH 3/3] docs: retire TESTS.md + archive 2 untracked test-review snapshots MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit TESTS.md was already flagged as "hand-maintained, may lag — git log is the truth" in CLAUDE.md after PR #55. That's a soft deprecation; this commit follows through. The catalog's content is redundant with: - @spec tags above each test - the audit table at docs/spec-coverage.md - per-file head comments Drift between TESTS.md and reality has been the highest of any doc in the suite (every test add/remove needs a hand-edit). Retiring it removes the maintenance burden without losing information. Changes: - TESTS.md → docs/archive/TESTS.md (preserved for provenance) - docs/test-review-2026-05-{22,30}.md → docs/archive/ (the 2 review snapshots from prior sessions that were sitting untracked at the repo root level of docs/; same time-bound shape as the test-review-2026-05-20 that PR #55 already archived) - CLAUDE.md "Read these in order": point at file-head comments + `grep -nA1 "// @spec" tests/` instead of TESTS.md - README.md opening highlight: point at docs/spec-coverage.md (which has the requirement-by-requirement table) instead of TESTS.md (which had the per-test catalog by another name) - scripts/check-doc-paths.sh: drop TESTS.md from DOC_FILES (no longer at the repo root) After this: - Live doc set at root: CLAUDE.md, README.md, skills.md, TRIAGE.md (4, down from 5) - Archive: 4 historical files + the new TESTS.md --- CLAUDE.md | 2 +- README.md | 5 +- TESTS.md => docs/archive/TESTS.md | 0 docs/archive/test-review-2026-05-22.md | 85 +++++++++++ docs/archive/test-review-2026-05-30.md | 200 +++++++++++++++++++++++++ scripts/check-doc-paths.sh | 1 - 6 files changed, 289 insertions(+), 4 deletions(-) rename TESTS.md => docs/archive/TESTS.md (100%) create mode 100644 docs/archive/test-review-2026-05-22.md create mode 100644 docs/archive/test-review-2026-05-30.md diff --git a/CLAUDE.md b/CLAUDE.md index a53b65b..3958762 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -18,7 +18,7 @@ enforced, …) not a single product. Apps are upstream; we don't own their UI. 2. [`docs/spec-coverage.md`](./docs/spec-coverage.md) — audit table: which test pins which requirement 3. [`docs/spec-coverage-deferred.md`](./docs/spec-coverage-deferred.md) — gap pile + categories 4. [`skills.md`](./skills.md) — findings (F1–F11), link-coverage rules, per-app admin gating narrative, adding-a-new-app guide, bug-staging workflow -5. [`TESTS.md`](./TESTS.md) — per-test catalog (hand-maintained, may lag — `git log -- tests/` is the truth) +5. To answer "what does test X do?" — read the test file's head comment + `@spec` tag, OR run `grep -nA1 "// @spec" tests/` for the full inventory. The retired per-test catalog (`TESTS.md`) was hand-maintained and went stale; it's archived at `docs/archive/TESTS.md` for provenance. When CI is red, [`TRIAGE.md`](./TRIAGE.md) is the 2-minute "what is this and what do I do" runbook — failure pattern → cause → action. diff --git a/README.md b/README.md index ba2a0a5..7f5c527 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,8 @@ # FOSS E2E — Playwright Test Suite -End-to-end tests for the FOSS platform. See **[`TESTS.md`](./TESTS.md)** for -a top-to-bottom catalog of every test in the suite. Highlights: SSO chain, +End-to-end tests for the FOSS platform. See +**[`docs/spec-coverage.md`](./docs/spec-coverage.md)** for the +requirement-by-requirement audit table. Highlights: SSO chain, multi-app session sharing, cookie expiry bounds, session lifecycle (logout / invalidation / replay / deletion), per-app link coverage, the Plane god-mode admin escape hatch, Outline's admin `/settings/*` SSO-gating diff --git a/TESTS.md b/docs/archive/TESTS.md similarity index 100% rename from TESTS.md rename to docs/archive/TESTS.md diff --git a/docs/archive/test-review-2026-05-22.md b/docs/archive/test-review-2026-05-22.md new file mode 100644 index 0000000..60f50fb --- /dev/null +++ b/docs/archive/test-review-2026-05-22.md @@ -0,0 +1,85 @@ +# Test-suite review — 2026-05-22 + +Scope: `tests/` (excluding `tests/bugs/`) — **44 files reviewed**, **306 expanded tests** (83 `test()` declarations × parameterization). + +## Bottom line + +The suite is in **good shape**. One Important finding worth fixing in a hygiene PR; no Critical findings. The 306-test count is justified — it expands from 83 distinct `test()` declarations, each parameterized for per-app + per-payload precision. Removing the multipliers would lose the "which exact attack vector / which exact app" failure-localisation signal. + +## Critical + +(none) + +## Important + +- **`tests/auth/proxy-short-circuit.spec.ts:34`** — hardcoded `_oauth2_proxy` literal + - **Why:** Convention (Mechanical check 2 from this skill): import `AUTH_COOKIE` from `constants.ts` rather than hardcoding the cookie name. The only files allowed to use the literal are `cookie-tampering` and `session-fixation` (which assert tampering signals). + - **Fix:** Replace `const SSO_COOKIE_NAMES = new Set(["_oauth2_proxy"]);` with `const SSO_COOKIE_NAMES = new Set([AUTH_COOKIE]);` and add `AUTH_COOKIE` to the existing `constants.ts` import. + +## Nit + +(none) + +## Findings discarded (false positives worth recording so future runs skip them) + +- **S2 (login choreography duplication):** 5 grep hits, all legitimate. + - `tests/security/sso-mode-no-local-login.spec.ts` looks for password inputs to assert they DON'T exist (the test's whole point). + - `tests/auth/session-lifecycle.spec.ts` checks if a password input is visible (re-auth detection signal). + - `tests/apps/pm-godmode.spec.ts` uses local creds at `/god-mode/` — explicitly NOT SSO, per CLAUDE.md. +- **S4 (sleep / `delay()` / `node:timers/promises`):** all 30+ hits are `test.setTimeout(...)` or `raw.setTimeout(...)` — Playwright's per-test timeout setter, not a sleep call. Grep pattern needs to be narrower (`page.waitForTimeout` is the actual ban, already covered by Mechanical check 3). +- **S6 (cookie surgery):** all hits are in tests where cookie state IS the system-under-test: + - `cookie-tampering.spec.ts` + `session-fixation.spec.ts` — explicitly listed as allowed in CLAUDE.md. + - `layer2-re-establish.spec.ts` — tests session re-establishment when Layer 1 cookies/storage are cleared. The cookie surgery is the setup, not a shortcut around the contract. + - `session-lifecycle.spec.ts` — tests session lifecycle by clearing/restoring `AUTH_COOKIE`. Same reasoning. +- **S3 (bare `toBe(true|false)`):** 27 files flagged by raw count, but all sampled cases use the `expect(value, "descriptive message").toBe(...)` form — the message arg is on the line above. The skill's grep matches the closing parens only and gives a noisy signal. Skip this check on future runs, or rewrite the grep to detect missing-message-arg directly (e.g. AST-based, since `expect(X).toBe(Y)` vs `expect(X, "msg").toBe(Y)` is hard to distinguish with regex). + +## On the "do we have too many tests?" question + +| Concern | Reality | +|---|---| +| Raw count | 306 tests across 44 files | +| Source count | 83 `test()` declarations | +| Expansion ratio | 3.7× average, driven by **per-app loops** (×5) and **per-payload loops** (×4 for redirects, ×5 for OIDC state values) | + +### Highest-multiplier files + +| File | Tests | Pattern | +|---|---|---| +| `apps/outline-admin.spec.ts` | 39 | 13 Outline settings paths × 3 contracts (cold-bounce / non-admin role / admin role) | +| `security/headers.spec.ts` | 27 | Targets × `HEADER_RULES` + the new HTML-hardening block | +| `security/oidc-state-integrity.spec.ts` | 25 | 5 apps × 5 invalid-state payloads | +| `security/bypass-surface.spec.ts` | 21 | Bypass paths × apps × statuses | +| `security/open-redirect.spec.ts` | 20 | 5 apps × 4 evasion payloads | +| `security/open-redirect-crlf.spec.ts` | 20 | 5 apps × 4 CRLF payloads | +| `security/http-method-tampering.spec.ts` | 20 | 5 apps × 4 methods | + +Each line in this table is "one test per attack vector × per app." Collapsing them (e.g. one test that loops internally and aggregates failures) would hide which specific app/payload combination failed — the most useful signal during a security regression. + +### Specs to leave alone (high count is justified) + +All of them. + +### Specs to consider trimming (none currently) + +- A reasonable rule-of-thumb threshold is `tests-per-file > 40` for non-security specs. The only file approaching that is `outline-admin.spec.ts` at 39, but its three contracts × 13 paths shape is intentional and reviewed (see the file's leading comment about Outline's chunk-loader 429 flake handling). + +## Summary + +| Severity | Count | +|---|---| +| Critical | 0 | +| Important | 1 | +| Nit | 0 | + +**Action items:** + +1. Open a hygiene PR for the single Important finding (`proxy-short-circuit.spec.ts:34` — use `AUTH_COOKIE` import). + +That's it. No further action needed; the suite size and shape are correct. + +## Cross-references + +- `CLAUDE.md` — suite conventions (the rulebook this review measured against) +- `skills.md` §1 / §2 — invariant-based test organisation +- `scripts/check-spec-coverage.sh` — separate audit for `@spec` coverage (last run: 27 ✅ / 25 ⚠️ / 0 ❌ / 52 total) +- `.claude/skills/test-reviewer/SKILL.md` — the skill that produced this report diff --git a/docs/archive/test-review-2026-05-30.md b/docs/archive/test-review-2026-05-30.md new file mode 100644 index 0000000..62c09ae --- /dev/null +++ b/docs/archive/test-review-2026-05-30.md @@ -0,0 +1,200 @@ +# Test-suite review — 2026-05-30 + +Scope: `tests/` (excluding `tests/bugs/`) — 54 files reviewed. +Focus: scenario correctness (does each test actually pin the right invariant?), plus convention compliance. + +All 8 mechanical convention checks pass cleanly (no hardcoded foss.arbisoft.com, no hardcoded `_oauth2_proxy` outside the two allowlisted tampering specs, no `waitForTimeout`, no POM, no `data/`, no `tests/auth/` files missing `@spec`, no `networkidle`, no `expect(await page.title())`). The remaining findings are semantic — they're about whether each test's *scenario* actually catches the failure mode the test claims to pin. + +--- + +## Critical +(Tests that don't pin what they claim, or could pass even if the contract is broken.) + +- `tests/auth/workspace-auto-join-independence.spec.ts:138-192` — Independence assertion is structurally vacuous. + - **Why (S-correctness, S5):** The test's load-bearing assertion is "no two apps report the same identifier." But the probes pull each app's own primary key from its own DB: Outline's `team.id` (UUID generated by Outline), Plane's `last_workspace_id` (UUID generated by Plane), Penpot's `~:default-team-id` (UUID generated by Penpot), and **SurfSense's user `id`** (a per-app user PK, not a workspace at all — the file-level comment even acknowledges this with "if cross-app sync somehow assigned the same user PK across apps, this would coincidentally match another app's UUID (extremely unlikely with separate per-app DBs)"). Four independent DBs generating random UUIDs will *never* collide regardless of whether the auto-join independence contract is upheld. A regression that introduced a shared workspace backend would not flip this assertion — the apps would each still return their own primary key. The test passes for the wrong reason. + - **Fix:** Either (a) reframe the test to assert a positive shape — for the same SSO user, each app's primary workspace identifier maps to a *per-app-named* workspace whose name/slug matches the bundle's expected per-app workspace (Outline's team slug = `SMB_DEFAULT_WORKSPACE_NAME`, Plane's workspace slug = `PLANE_WORKSPACE_SLUG`, etc.), or (b) downgrade the test to deferred and document that proving "no cross-app sync" from a black-box e2e probe requires shape-correlated probes (the same-UUID collision check is dominated by birthday-paradox math against 4 independent generators and is not informative). The current shape is a placeholder that will never fire. + +- `tests/security/jwt-algorithm-confusion.spec.ts:80-83, 117-120` — `expect(status).toBeGreaterThanOrEqual(200)` is a tautology. + - **Why (S-correctness):** Both tests' else-branch assertion is `expect(status, ...).toBeGreaterThanOrEqual(200)`. Every HTTP status code Playwright returns is ≥ 200, so this assertion does nothing. The intent (per the comment "Any non-2xx is fine — proves the token was rejected") is to assert `status >= 400` or `!== 200`. As written, an alg=none / alg=HS256 token that returned 200 with no email in the body would pass the if-branch's body check, but a 200 *with* the forged email is the only assertion that actually fires — meaning a regression that returned a forged identity *in a different field name* (e.g. `userEmail` instead of `email`) would slip through both branches. + - **Fix:** Change the else-branch assertion to `expect(status, ...).not.toBe(200)` (the contract is: token rejected, so non-2xx is the load-bearing signal). Keep the 200-branch body check as a defence-in-depth backstop. + +- `tests/flows/login-logout-flow.spec.ts:1-2` — `@spec` tag mismatches the file's tests. + - **Why (S-correctness):** The file is tagged `@spec logout-flow#per-app-logout-shall-be-navigation-only`, but the three tests it contains are: (1) "login once, every app loads authed", (2) "portal Logout-All kills SSO and every app bounces to IDP", (3) "/oauth2/sign_out clears SSO cookie and redirects". None of these exercise per-app navigation-only logout — that contract is pinned by `tests/auth/logout-invariants.spec.ts` sub-test 3 (which uses `openLogoutMenu[app.name]` to click each app's in-app logout control and asserts no `/sign_out` is called). The tag is misleading: a regression in per-app navigation-only logout could be reported as "covered" by this file even though the file doesn't test it. + - **Fix:** Replace the tag with the requirements the tests actually pin — `logout-flow#portal-logout-all-shall-clear-only-the-oauth2-proxy-cookie` and `logout-flow#stale-app-native-sessions-shall-be-reaped-on-next-request-not-eagerly`. Keep `tests/auth/logout-invariants.spec.ts` as the sole owner of the navigation-only tag. + +--- + +## Important +(Convention drift, scenario-shape weaknesses. Fix in a hygiene PR.) + +- `tests/security/host-header-injection.spec.ts:155-168` — Body-reflection check is too permissive. + - **Why (S-correctness — new spec):** The test rejects responses where `result.body.includes(ATTACKER_HOST)`, but only sends `Host: attacker.example`. Some servers (Traefik default) will echo the request Host in error pages even when they refuse the request — and `attacker.example` is a public TLD-shaped string that could appear by coincidence in a 502 body containing the request line. More importantly, the test only catches reflection of the *exact* attacker string. A server that builds redirect URLs from `request.Host` but rewrites the host portion to omit the public part (or substitutes its own template logic) would not reflect `attacker.example` literally but would still be exploitable. + - **Fix:** Tighten: (1) use a randomized attacker hostname per test invocation so an in-body match is provably from the inbound header, not background body content; (2) add a positive assertion that any Location header on a 3xx response starts with `https://` (or is a relative path) — that's the actually-load-bearing defense, the body echo is a secondary smoke check. + +- `tests/security/cookie-bomb-dos.spec.ts:62-118` — Doesn't exercise the worst failure mode it claims to catch. + - **Why (S-correctness — new spec):** The file's header comment names two failure modes — "5xx storm" and "header parser truncation" leading to "successful authz check runs against truncated header state, opening an authz bypass." The test asserts only the 5xx case. The truncation/authz-bypass failure mode (where the chain returns a 2xx with truncated headers, possibly the more dangerous case) is not asserted at all — the test specifically excludes 2xx from being a failure ("2xx is unexpected ... but it's not a failure of THIS contract"). + - **Fix:** Either narrow the file header comment to "we only pin the 5xx fail-closed shape; truncation-bypass is out of scope" or add a second assertion: when an oversized cookie is sent alongside a *valid* `_oauth2_proxy` cookie, the request must still be properly authenticated (i.e. the real cookie isn't silently truncated). The current asymmetry between the threat model in the header comment and the assertion is misleading. + +- `tests/security/clickjacking-iframe.spec.ts:75-133` — The two probes (`contentDocument` and `contentWindow.location.href`) measure the same-origin policy, not XFO/CSP enforcement. + - **Why (S-correctness — new spec):** The data: URL parent has an opaque origin. The cross-origin policy ALWAYS prevents `contentDocument` access and `contentWindow.location.href` access from an opaque-origin parent reading any HTTPS-origin frame, regardless of XFO or CSP frame-ancestors. So both assertions pass for **any** target — they would even pass for a target that explicitly allows framing (`X-Frame-Options: ALLOWALL`). The test as written cannot distinguish a properly-protected target from an unprotected one. + - **Fix:** To actually test XFO/CSP enforcement: (1) check whether the iframe loaded at all by reading `iframe.contentWindow?.length` or `iframe.contentDocument?.URL` (which are blocked at frame-load time when XFO triggers, vs. blocked by same-origin policy when the frame loads but is cross-origin), or (2) listen for the iframe's `load` event and combine with the `__playwright` page response listener to detect whether the response was actually rendered (a frame blocked by XFO emits no `load` for the inner doc; same-origin-blocked content still loads but is unreadable). Easier alternative: parse the response headers directly in the parent context — that's what `tests/security/headers.spec.ts` already does, and the file header itself notes "headers asserted in isolation are necessary but not sufficient." If the goal is "actual browser behaviour" the test needs a real load-event probe, not the same-origin-policy probes it uses today. + +- `tests/security/oidc-state-integrity.spec.ts:81-95` — Accepts 5xx as a valid rejection. + - **Why (S-correctness):** The test treats `(status >= 400 && status < 600)` as acceptable, with an explicit "Observed in practice: oauth2-proxy returns 500 on forged-state callbacks" note. Treating 5xx as a valid rejection is a real softening — a 5xx-on-bad-input is itself a security-relevant bug (a remote unauthenticated user can crash the callback handler at will), and accepting it means the test will go green even if oauth2-proxy starts panic-ing on every callback. A correct security defense is a clean 4xx; the current shape lets a regression that turns clean 4xx → noisy 5xx pass silently. + - **Fix:** Narrow acceptance to `(status >= 400 && status < 500)`. If oauth2-proxy returns 500 on the live deployment, file an upstream bug; the test should fail until it does, or the test should be carved into "expected 4xx" and "expected 5xx until upstream fix" so the latter has a tracking link. + +- `tests/security/cross-user-impersonation.spec.ts:106-109` — `.not.toContain(fossEmail)` after `.toBe(normalEmail)` is redundant given the preceding strict equality. + - **Why (S-correctness):** The first assertion already requires `spoofed.toLowerCase() === normalEmail`. If that passes, `spoofed.toLowerCase() !== fossEmail` is trivially true (assuming normalEmail !== fossEmail, which the test already asserts as a pre-condition). The defense-in-depth comment justifies it as "catches a backend that returns BOTH identities joined" — but if `spoofed` is a single email string from a `/me` JSON field, joining is structurally impossible. The check is decorative. + - **Fix:** Either drop the assertion, or change it to scan a different artifact (e.g. the full response body, not just the email field) so it pins something the first assertion doesn't. + +- `tests/auth/session-lifecycle.spec.ts:81-118` — Replay-after-logout test accepts visible "Sign in" button as a pass signal. + - **Why (S-correctness):** The assertion is `redirectedToAuthWall || loginVisible`. The latter is satisfied by any visible button matching `/sign\s*in/i` — but the *portal's own homepage* (which is where the user lands after sign_out completes) has a "Login" button. So a regression where the cookie is genuinely still valid but the user happens to also see a sign-in button somewhere on the landing page would pass. A more conclusive check would be to attempt a privileged-API call (e.g. Plane `/api/users/me/`) with the replayed cookies and assert it returns 401/403 — that directly tests whether the server-side session is invalidated, which is the actual contract. + - **Fix:** Add a request-level probe (cookie jar → `/api/users/me/`) to confirm the replayed cookie is server-side rejected, in addition to the URL-based check. + +- `tests/security/no-tokens-in-url.spec.ts:31-44` — Includes `code` in TOKEN_PARAM_NAMES but only checks final URL. + - **Why (S-correctness):** The comment says "We only flag it if it persists past the final settle" — but the test only ever inspects the post-`expect.poll` settled URL, never the redirect chain. So `code` will be in the list but never trigger. That's fine for `code` (it legitimately appears mid-flight) but means `code` adds no test coverage. The bigger gap: the test only inspects `settled` (a single URL). A SPA that briefly stashes a JWT into the URL, then SPA-rewrites it away before the poll fires, would not be caught — and that's the most-likely shape of a real regression. + - **Fix:** Capture every URL the page visits via `page.on('framenavigated', ...)` from before the goto, then sweep all of them at the end. Drop `code` from the list since it's intentionally not flagged. + +- `tests/auth/sso-login.spec.ts:99-122` — "Session survives reload + revisit" cookieAfter.value === cookieBefore.value can pass when oauth2-proxy refreshes the cookie unnecessarily. + - **Why (S-correctness):** The test asserts the cookie value is unchanged across reload + revisit, but oauth2-proxy's `cookie_refresh` window will rotate the value silently if the cookie age crosses the threshold during the test. With the worker fixture's storage state being injected at start-of-worker, there's no guarantee the cookie's age during these particular tests falls below the refresh window. If the cookie rotates, the test fails — not because the contract is broken, but because timing. This is a stability issue rather than a contract issue, but it's worth noting that the assertion intends "no IDP round-trip occurred" and what it actually pins is "the cookie value bytes are identical." + - **Fix:** Soften the assertion to "cookie still exists, value non-empty, no IDP bounce occurred" by listening on `page.on('request', ...)` for any IDP-host requests during the reload/revisit. The current bytes-equal check is too strict for the actual contract. + +- `tests/auth/proxy-short-circuit.spec.ts:58-63` — Self-skip on header-only apps masks per-app coverage gaps. + - **Why (S-correctness):** The test self-skips with "vacuously satisfied" when no JS-readable session cookie is observed. For Penpot, Twenty, and SurfSense this is likely — meaning the contract is only actually validated on Outline (and maybe Plane). The block-on-smoke pattern from `app-health-probes` exists, but this test skips silently with vacuous-pass framing. From CI output a green run looks like "5 apps validated" but in reality only 1-2 were. + - **Fix:** When the skip fires, emit a more pointed message ("Penpot does not expose a JS-readable session cookie; short-circuit invariant can be validated only via server-side response timing or per-request DB-write count — out of scope for this test"). Consider tracking this as a deferred entry rather than a silent skip. + +--- + +## Nit +(Stylistic / minor refinements.) + +- `tests/apps/twenty-admin.spec.ts:89` — Block (B) comment says "SSO-authed as FOSS_USER (non-admin)" but the block actually uses `NORMAL_USER` (line 136). FOSS_USER is the admin in Twenty per `constants.ts` and the file's own (C) block. Stale comment. + +- `tests/security/cookie-tampering.spec.ts:43-44` — `flipChar` mapping is too narrow — if the byte at `idx` isn't `A`/`a`/`0`, it falls back to `X`. If `original[idx]` is already `X` (a real possibility for v2-cookie base64), the tampered value equals the original and the `expect(tampered).not.toBe(original)` assertion will fail confusingly. Low probability but trivially-fixable: flip to any character that's guaranteed different from the input (e.g. `c === 'X' ? 'Y' : 'X'`). + +- `tests/security/headers.spec.ts:36` — HSTS lower bound is hardcoded to 15552000 (~180 days). The contract comment says "≥ 6 months" but 180 days < 6 months in months-of-30/31-days. Cosmetic; the value matches OWASP's "180 days" recommendation either way. + +- `tests/auth/session-sharing.spec.ts:158` — Builds a regex inline (`new RegExp(\`^https?://${targetHost.replace(/\\./g, "\\\\.")}\`)`). The shared `escapeHostForRegex(host)` helper exists exactly for this — substitute it for consistency with the rest of the suite. (Mechanical check 1 caught zero violations, but this is the same pattern at the regex-construction layer.) + +- `tests/flows/identity-switch-after-relogin.spec.ts:51-55` — `SURFACED_EMAIL_RE` uses `(\\d{10,})@…` for User identity matching. The literal `{10,}` assumes Cognito subs are always ≥10 digits; the live deployment uses `1020010000019120` (16 digits), so this is fine today, but if Cognito ever issues a UUID-shaped sub the regex misses entirely. Document the assumption inline or use a more shape-agnostic match. + +- `tests/security/oidc-state-integrity.spec.ts:38-44` — `INVALID_STATES` includes `'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'` (32 chars, all 'a') as "32 chars but trivially guessable." This will never match a real-issued state value (which is randomly generated), so it's structurally identical to the other invalid-state probes — adds no extra coverage. Cosmetic. + +--- + +## Per-test scenario notes +(For tests where the scenario is correct but could be tightened, or where the reviewer wants to flag a specific judgment call.) + +### tests/auth/ + +- `tests/auth/identity-consistency.spec.ts` — Solid. The `upstreamUser` cross-check at line 213-225 is exactly the anti-vacuous-pass armour the test needs. The Twenty-omitted-by-design caveat is well-documented. + +- `tests/auth/per-app-login-smoke.spec.ts` — Correctly scoped (one test per app, named-loud failure). Pre-condition warming via `page.goto(baseUrl)` before `probeApp` matches the contract. + +- `tests/auth/email-domain-consistency.spec.ts` — Largely a strict subset of `identity-consistency.spec.ts`. The "different failure-mode shape" justification in the file header is fair but the duplication of cookie/JSON probe helpers across both files (`cookieHeaderFor`, `postJSON`, `getJSON`, the `PROBES` table) is significant. Consider moving the shared probe definitions to `tests/lib/identity-probes.ts` (which already has its own `IDENTITY_PROBES` table — currently used only by header-spoofing.spec.ts) and have both consistency specs read from it. + +- `tests/auth/concurrent-first-login.spec.ts` — The shape-by-email-set comparison is a sound proxy for "single identity returned," but the test only catches HARD divergence (two contexts reporting different emails). The file's own header comment acknowledges this is "intentionally conservative." Document this as the test's known scope; the alternative (DB-row count) genuinely requires admin access. + +- `tests/auth/layer2-re-establish.spec.ts` — Strong shape (clear, single-invariant). The `.catch(() => {})` swallowing of mid-evaluate "Execution context was destroyed" at line 67-68 is well-commented; the reload after is the load-bearing check. + +- `tests/auth/layer2-renewal-suppressed-on-4xx.spec.ts` — Scope is honest (one app, one cookie name). The file head clearly states the extension path. Good. + +- `tests/auth/logout-invariants.spec.ts` — Three sub-tests, three invariants — clean. Sub-test 1's per-app cookie survival check (line 117-128) handles the vacuous case (no cookies at all) with `test.skip` at line 131, which is correct. Sub-test 3's forbidden-paths regex includes `/cognito.*\/logout/i` — note this matches any URL with "cognito" + "/logout" in it, including the host portion; that's fine for the live deployment (Cognito host contains "cognito") but worth noting. + +### tests/security/ + +- `tests/security/cache-control-authed.spec.ts` (new) — `evaluateCacheControl` correctly rejects `private` alone but accepts `private + no-cache` or `private + max-age=0`. The acceptance logic is sound. One edge case: a response with both `Cache-Control: public, max-age=0` and a separate `Pragma: no-cache` header would be rejected here but is also a real (if old) shape that protects shared caches. The reject is conservative but defensible. + +- `tests/security/clickjacking-iframe.spec.ts` (new) — See Important above. The probe shape doesn't catch the failure mode the file header claims. + +- `tests/security/host-header-injection.spec.ts` (new) — See Important above. Worth a second pass to tighten the body-reflection check. + +- `tests/security/cookie-bomb-dos.spec.ts` (new) — See Important above. The 5xx-only assertion is correct for the named contract but misses the truncation-bypass mode. + +- `tests/security/header-spoofing.spec.ts` — The (B) "strip middleware (authed)" block's file-head LIMITATIONS section is exemplary documentation of what the test does and doesn't catch. The "different attacker email" check in the worker pre-condition (line 107-109) is a good anti-vacuous-pass. + +- `tests/security/cookie-surgery-cross-user.spec.ts` — Strong shape: cookie surgery from a closed source context, plant into a fresh victim context, assert identity-flush fires. The 4-app coverage (Twenty omitted) matches the rest of the suite's per-app probe shape. + +- `tests/security/cookie-attributes.spec.ts` — Tight, single-invariant: every defense-in-depth attribute checked individually with a clear failure message per attribute. Good. + +- `tests/security/per-app-cookie-theft.spec.ts` — Two complementary describes (A: not standalone, B: hardening attributes). The (A) block's pre-condition warming is correct. Twenty's exclusion from (A) is properly self-skipped via `test.skip` for header-only apps. + +- `tests/security/bypass-surface.spec.ts` — Per-app `APP_BYPASS_EXTRAS` table is a great pattern. The "gated regression guard" lane for SurfSense `/docs` and `/openapi.json` is exactly the right shape for tracking paths that were intentionally moved off bypass. + +- `tests/security/strip-on-bypass.spec.ts` — File-head comment is honest about what it does and doesn't prove. The "test grows teeth if a future bypass surface starts reading identity" framing is forward-looking. + +- `tests/security/sso-mode-no-local-login.spec.ts` — The 404/5xx-fails-loud branches at lines 73-95 are correct anti-rot armour. The narrow `LOCAL_AUTH_TEXT_RE` is justified inline. + +- `tests/security/csrf-logout.spec.ts` — Single-flow, two attack shapes (img + fetch with credentials), both pinning the SameSite=Lax defense. Good. + +- `tests/security/session-fixation.spec.ts` — The pre-login HMAC-rejection probe at line 62-76 is excellent — it specifically defends against "test passes because login overwrote the planted cookie anyway" vacuous outcomes. + +- `tests/security/no-tokens-in-url.spec.ts` — See Important above. The shape is right; the coverage of the redirect chain (not just the settled URL) is the gap. + +- `tests/security/http-no-plaintext.spec.ts` — Three acceptable outcomes, well-documented. The `ENOTFOUND` re-throw catches environment drift loudly. Good. + +- `tests/security/headers.spec.ts` — HEADER_RULES is well-structured; per-rule `why` strings are the right kind of context for on-call. The portal-only CSP/COOP/CORP scope is correctly justified. + +- `tests/security/http-method-tampering.spec.ts` — Solid. The "redirect into the app vs. into auth chain" sub-check at line 62-74 catches a subtle bypass shape. + +- `tests/security/open-redirect.spec.ts` and `open-redirect-crlf.spec.ts` — Both correctly scoped; the CRLF spec exercises the second attack class that the host-validation spec doesn't. + +- `tests/security/oidc-state-integrity.spec.ts` — See Important above for the 5xx-as-rejection concern. Otherwise good shape. + +- `tests/security/cross-user-impersonation.spec.ts` — See Important above for the redundant assertion. The pre-condition email-distinct check (line 51-54) is good. + +- `tests/security/jwt-algorithm-confusion.spec.ts` — See Critical above for the tautological else-branch assertion. + +### tests/apps/ + +- `tests/apps/outline-admin.spec.ts` — Three-block structure (cold/non-admin/admin) is the right shape. The 429-retry choreography is well-documented and pacing-only. The `gotoSettingsPath` click-nav vs. direct-goto split at line 116-120 is a thoughtful adaptation to Outline's chunk-loader behavior. + +- `tests/apps/twenty-admin.spec.ts` — Three-block structure same as outline-admin. ADMIN_UI_MARKERS regex set is a fragile contract (Twenty could rename "Health Status" → "System health" and break the test), but the markers are unlikely to all change at once and the count-zero-vs-non-zero shape is sensible. + +- `tests/apps/penpot-admin.spec.ts` — Member-vs-owner gate via Invite-people locator is the right granularity given Penpot's UI shape. + +- `tests/apps/surfsense-admin.spec.ts` — The 2026-05 UI redesign comment is well-tracked. Role-change-button count assertion is the load-bearing signal. + +- `tests/apps/pm-admin.spec.ts` — Auto-join-as-Member-not-Admin shape is correct. The "Workspace not found" negative-shell check (line 145-147) is a good positive-membership proof. + +- `tests/apps/pm-workspace-isolation.spec.ts` — UI + API probes in one file; the API probe is the actually-load-bearing test (UI gating could collapse without exposing data, but API-layer membership check is the contract). Good. + +- `tests/apps/pm-godmode.spec.ts` — 5 sub-tests, each a discrete invariant. Good. Sub-test 5 (independence) is the load-bearing signal that god-mode and SSO are separate session universes. + +- `tests/apps/pm-project-create.spec.ts` — Out-of-scope for SSO contract (documented). The `pendingErrorReads` async-drain pattern at line 60-75, 199 is a good way to surface the body on failure without forcing a local repro. Teardown via `context.request.delete` at line 234-246 is correct. + +- `tests/apps/outline.spec.ts`, `pm.spec.ts`, `penpot.spec.ts`, `surfsense.spec.ts`, `twenty.spec.ts` — All shell registrations for link-coverage; covered by `UNTAGGED_ALLOWLIST` rationale. Penpot's hash-route check is the right shape for an app without anchor-nav. + +### tests/flows/ + +- `tests/flows/login-logout-flow.spec.ts` — See Critical for the `@spec` tag mismatch. Test bodies themselves are sound. + +- `tests/flows/cross-tab-logout-propagation.spec.ts` — Two-tab shared-context pattern is the right model for browser cross-tab semantics. Good. + +- `tests/flows/identity-switch-after-relogin.spec.ts` — Rich haystack-collection (storage + cookies + DOM + JWT payload decode) is the right approach for catching the reported bug across all 5 apps. Penpot skip is properly documented at the per-app level (SKIP_APPS_KNOWN_STALE). + +- `tests/flows/deep-link-after-login.spec.ts` — Per-app DEEP_PATHS table with two apps explicitly excluded for upstream product reasons (Penpot hash, SurfSense /login override). Honest. + +### tests/zap/ and tests/meta/ + +- `tests/zap/sso-chain.spec.ts` — Recording driver only; correctly excluded from default discovery and from the `@spec` tag gate. + +- `tests/meta/playwright-practices.spec.ts` — The bidirectional spec-gate (`@spec` tag check + UNTAGGED_ALLOWLIST) is exactly the right shape. The 8 rules in RULES are aligned with the conventions documented in CLAUDE.md. + +--- + +## Summary + +| Severity | Count | +|---|---| +| Critical | 3 | +| Important | 8 | +| Nit | 5 | + +**Top 3 recommended fixes (in priority order):** + +1. **Fix `tests/auth/workspace-auto-join-independence.spec.ts`** — The current "no shared identifier" assertion is structurally vacuous (4 independent UUID generators will never collide regardless of the contract). Either re-shape the test to assert per-app-named workspaces, or downgrade to deferred with a clear note that this invariant isn't testable from black-box e2e. + +2. **Fix `tests/security/jwt-algorithm-confusion.spec.ts`** — Replace the tautological `toBeGreaterThanOrEqual(200)` with a real rejection assertion (`not.toBe(200)`). The current shape lets a "forged token returns 200 with the forged email in a different field name" regression slip through both branches. + +3. **Re-tag `tests/flows/login-logout-flow.spec.ts`** — Replace `@spec logout-flow#per-app-logout-shall-be-navigation-only` with the two tags the file's tests actually pin (`portal-logout-all-shall-clear-only-the-oauth2-proxy-cookie` and `stale-app-native-sessions-shall-be-reaped-on-next-request-not-eagerly`). Keep `tests/auth/logout-invariants.spec.ts` as sole owner of the navigation-only tag so a per-app-logout regression isn't silently reported as covered by a portal-logout test. diff --git a/scripts/check-doc-paths.sh b/scripts/check-doc-paths.sh index 867969d..358e881 100755 --- a/scripts/check-doc-paths.sh +++ b/scripts/check-doc-paths.sh @@ -47,7 +47,6 @@ DOC_FILES=( "$REPO_ROOT/README.md" "$REPO_ROOT/skills.md" "$REPO_ROOT/TRIAGE.md" - "$REPO_ROOT/TESTS.md" ) # Include every CURRENT .md under docs/. The archive is excluded — # its contents are time-bound snapshots that reference paths as they