Skip to content

feat(bench): opt-in CDP tracing in bench.spec.ts (unblocks flame-graph perf diagnostics)#143

Merged
blove merged 5 commits into
mainfrom
bench-harness-cdp-tracing
May 13, 2026
Merged

feat(bench): opt-in CDP tracing in bench.spec.ts (unblocks flame-graph perf diagnostics)#143
blove merged 5 commits into
mainfrom
bench-harness-cdp-tracing

Conversation

@blove
Copy link
Copy Markdown
Contributor

@blove blove commented May 13, 2026

Summary

  • Adds an opt-in Chrome DevTools-format trace capture path to apps/bench/tests/bench.spec.ts, gated by PLAYWRIGHT_PERF_TRACE=1. Output: status/traces/<stem>.cdp.json (sibling to the Playwright .trace.zip, gitignored).
  • Closes the tooling gap surfaced by PR docs(research): pretable wrapped-text filter perf diagnostic #142's wrapped-text filter perf-diag: Playwright's default tracing.start({ screenshots, snapshots }) produces an SDK action trace, not a JS flame graph. CDP Tracing.start over disabled-by-default-devtools.timeline + v8 + disabled-by-default-v8.cpu_profiler produces DevTools-loadable JSON.
  • Best-effort additive: CDP attach failures, end timeouts, JSON write failures all log + continue. Env-unset path is byte-identical to current behavior — the existing spec test still passes unchanged.

Manual verification

  • Captured one CDP trace locally for pretable / S2 / hypothesis / filter-text. File: status/traces/chromium-pretable-default-s2-hypothesis-filter-text-2026-05-13t17-55-56-984z.cdp.json (30 KB, 145 events).
  • jq confirms well-formed shape: every event has ph, ts, cat, name. Categories present: __metadata, disabled-by-default-devtools.timeline, disabled-by-default-v8.cpu_profiler, v8 — the standard DevTools profiling set, as expected.
  • DevTools-load verification deferred to the reviewer. The captured JSON is in DevTools trace-event format ({ traceEvents: [...] }) and should load in Chrome → DevTools → Performance → "Load profile…".

Known limitation (consumer side, not the harness)

The bench app starts running its interaction script on page-load. CDP attaches after the adapter-label visibility check, so the very first moments of the interaction may run before tracing starts. For hypothesis-scale runs the captured window is sparse (~145 events). The wiring + format are correct; the next perf-diag consumer (the deferred wrapped-text filter investigation) will likely need either a wider scale or a triggered-start variant in the bench app. Documented in the spec, docs/research/repo-memory.md, and called out as out-of-scope here.

What's NOT in this PR

  • Matrix-runner integration (would multiply matrix wall-clock 5–10×).
  • Speedscope export.
  • Automated test for the CDP path (requires a browser-in-CI setup beyond what we have; manual verification is the bar).
  • Wrapped-text filter perf-fix v2 (separate follow-up; this PR unblocks the tooling).

Files touched

  • apps/bench/tests/bench.spec.ts (+62)
  • docs/research/repo-memory.md (+28 / -1)
  • docs/superpowers/specs/2026-05-13-bench-harness-cdp-tracing-design.md (new)
  • docs/superpowers/plans/2026-05-13-bench-harness-cdp-tracing.md (new)

No `packages/` changes. No public-API surface.

Gates

  • `pnpm -w typecheck` — passes
  • `pnpm -w test` — 191 tests pass
  • `pnpm -w lint` — passes
  • `pnpm format` — clean
  • Manual verification: CDP JSON well-formed (`jq` shape check + category enumeration)

🤖 Generated with Claude Code

blove and others added 5 commits May 13, 2026 10:46
Opt-in CDP-level tracing path for apps/bench/tests/bench.spec.ts.
Unblocks future perf diagnostics that need flame-graph data (the
wrapped-text filter investigation from PR #142 hit this exact wall).
No matrix-runner integration; opt-in via PLAYWRIGHT_PERF_TRACE=1 env.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Four-task plan: wire CDP-tracing branch behind PLAYWRIGHT_PERF_TRACE=1
env opt-in, manual verification, repo-memory entry, gates + PR.
Auto-mergeable.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…TRACE

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@blove blove enabled auto-merge (squash) May 13, 2026 18:14
@vercel
Copy link
Copy Markdown

vercel Bot commented May 13, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
pretable Ready Ready Preview, Comment May 13, 2026 6:14pm

@blove blove merged commit e558bc5 into main May 13, 2026
11 checks passed
@blove blove deleted the bench-harness-cdp-tracing branch May 13, 2026 18:16
@github-actions
Copy link
Copy Markdown
Contributor

Vercel preview ready

Preview: https://pretable-p57kyj3hj-cacheplane.vercel.app
Commit: 36cc0b21a328b7ac643422e0006b58785db85c1d

Updated automatically by the deploy-preview job.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant