Runtime of tests is 3x to 6x slower on Windows than it is on Linux

[cboos]

This started to become annoying... I got Claude to think about a few approaches.

[cboos]

(Claude) Write-up of the investigation and the PRs it produced.

TL;DR — three PRs, split by concern so they can land independently

• Speed up integration-test fixtures on Windows (shrink copied transcripts; skip symlink tests when unavailable) #249 — test fixtures (test-only): shrink the oversized fixture
  transcripts in the copied data; skip symlink tests where unavailable.
  test_integration_realistic.py −38%, full unit suite −21%
  (local Windows).
• Cut pytest-xdist per-worker import cost on Windows unit runs #250 — pytest-xdist per-worker import cost (test config): stop
  importing playwright/textual and the pytest-playwright plugin on
  every unit worker. Collect-only −26%; per-worker savings that
  compound under Windows' process model.
• Speed up the SQLite metadata-cache build (connection reuse + synchronous=NORMAL) #251 — SQLite cache build (production code): reuse one DB
  connection per build instead of ~189 opens; synchronous=NORMAL. Cache
  build −25%, connection opens 189 → 14.

How the time actually splits (the surprising part)

A profiling pass on the worst offenders (the test_integration_realistic.py
family — 18 of the 25 slowest tests) showed:

1. It is not the fixture copy. shutil.copytree of the ~20 MB /
   ~100-file fixture set measured ~0.15s; teardown ~0.10s. The natural
   hypothesis — that the function-scoped copytree was the cost — was
   refuted. The entire per-test cost is process_projects_hierarchy
   (render + cache build), re-run over the large transcripts every test.

2. Two distinct causes drive the Windows/Linux ratio (as opposed to
   absolute slowness):

   • pytest-xdist worker startup. Windows spawns workers and
     re-imports the whole module graph per worker; Linux forks. A single
     trivial test costs ~4.5s at -n0 vs ~18.6s at -n auto (12 workers)
     — ~14s of pure spawn + re-import. Module import alone is ~12.9s.
     Browser/TUI test modules, plus the auto-loaded pytest-playwright
     plugin, were imported on every unit worker even though unit tests
     never use them.
   • SQLite connection churn. The cache opened ~189 connections per
     single-project build (every method, reads included), each ~3.9ms —
     heaviest on Windows, where connection/file overhead is highest.

Ideas tried — what worked, what didn't

Idea	Verdict
Share the copytree across tests (function→module scope)	Dropped — copytree was never the cost (~0.15s).
Shrink the large fixture transcripts (copy-only, DAG-safe prefix truncation)	Worked → #249. Absolute win on both platforms.
synchronous=NORMAL/WAL on cache connections	Minor (~145ms) alone — kept as part of #251. The "fsync is 66–73% of the cost" reading turned out to be a measurement artifact (toggling use_cache under load); contention-immune counts corrected it.
Reuse one connection per build (CacheManager.batch())	Worked → #251. 189→14 opens, −25% build. Catch: a persistent connection holds .db-wal/.db-shm open and breaks Windows rmtree (WinError 32) — batch() closes at scope end and on exception; the default stays open-per-call.
Skip browser/TUI module import + drop pytest-playwright on unit runs	Worked → #250.
Cap xdist worker count on CI	Not committed — the spawn tax scales with workers, but the right fix is cheaper workers, not fewer. Optimal -n depends on the runner (windows-latest ~4 vCPUs vs a 12-core dev box) and needs an A/B on the real runner.

Absolute speedup vs the 3× ratio

#249 reduces rendered volume, so it speeds Linux up proportionally — it
lowers absolute Windows CI time but does not shrink the ratio. The
ratio is closed by #250 and #251, which target costs that are structurally
heavier on Windows (per-worker re-import under spawn; connection/file
overhead). #251 also helps real (non-test) cache builds on every platform.

Open item

Worker-count tuning on CI (#250) should be validated on the actual
windows-latest runner before committing a -n value — deliberately left
out of the PR.

[cboos]

The 3 PRs have been merged; let's see what the combined pay-off is in GitHub CI.

[cboos]

Local before/after benchmark on Linux

Branch fully restored: HEAD back at 985938c, all 102 staged WIP changes + 3 untracked files exactly as before (temp commit b19a1ff now unreferenced). Benchmark done.

Phase-2 AFTER (9cd5b7f = #249 + #250 + #251) — all rc=0

run	total just ci	rc
1	70s	0
2	68s	0
3	69s	0
mean	≈69.0s	all pass

Baseline → After deltas (means of 3)

phase	baseline	after	Δ	Δ%
unit (#250)	13.76s	13.34s	−0.42s	−3.1%
tui	7.43s	7.44s	+0.01s	~0%
browser	15.30s	14.70s	−0.60s	−3.9%
integration (#249)	48.84s	21.41s	−27.4s	−56.2%
benchmark	4.41s	4.17s	−0.24s	−5.4%
pytest subtotal	89.75s	61.06s	−28.7s	−32.0%
total ci	98.33s	69.0s	−29.3s	−29.8%

Findings

• The entire win is Speed up integration-test fixtures on Windows (shrink copied transcripts; skip symlink tests when unavailable) #249 (integration fixtures): −27s / −56%. It's the dominant phase on Linux and the only one that moved beyond noise. ~30s off total ci, ~all of it here.
• Cut pytest-xdist per-worker import cost on Windows unit runs #250 (xdist import-cost) shows no measurable Linux benefit — unit moved −0.4s, within run-to-run noise. This is exactly the "faster on Windows, not Linux" signal you flagged: xdist worker startup is expensive on Windows (spawn) but cheap on Linux (fork), so the import-cost cut pays off there and is ~invisible here. Worth confirming on the Windows numbers.
• Speed up the SQLite metadata-cache build (connection reuse + synchronous=NORMAL) #251 (sqlite-cache) not separately visible — no isolated cache-heavy phase; any gain is submerged in integration/unit and below Linux noise.
• Skip counts unchanged (integration 7 skipped, browser 1 skipped — identical both phases). Speed up integration-test fixtures on Windows (shrink copied transcripts; skip symlink tests when unavailable) #249's "skip symlink tests when unavailable" adds no skips on Linux → symlink tests still run here, as you predicted. So the 56% integration drop is real work getting faster on the same executed test set, not tests skipped away.
• Test count: unit 2128→2136 (+8 from the PRs); integration identical (78), so its speedup is apples-to-apples. Low variance both phases (base 95–101s, after 68–70s).

[cboos]

Local before/after benchmark on Windows

Combined effect of all 3 PRs — local Windows (12-core), 4bd633f → 9cd5b7f

Clean apples-to-apples metric: just test (unit suite) — pytest runs to completion in both states, so this is the valid cross-state comparison:

just test (unit)	run 1	run 2	run 3	min	median
Baseline 4bd633f	177.9s	173.3s	160.7s	160.7s	173.3s
Post-PR 9cd5b7f	101.5s	89.2s	92.1s	89.2s	92.1s
Δ				−44%	−47%

So the three PRs together cut the Windows unit-suite wall time ~45% — more than any single PR, because they stack (smaller render+cache volume × fewer per-worker imports × less SQLite churn).

Headline framing: post-PR just test-all (the entire suite — unit + tui + browser, exit 0) runs in 153–168s — i.e. the full suite now finishes in about the same time the baseline took to run the unit subset alone (160–178s).

The test-all / ci numbers need a caveat (as monk flagged)

• Baseline test-all/ci = 45–49s, exit 1 — these are time-to-abort, not full-suite: the recipe chain dies on the first failing step (the symlink WinError 1314 tests), so they never ran the whole suite. Not comparable to post-PR.
• Post-PR test-all = 153–168s, exit 0 (full run, symlink tests now skip).
• Post-PR ci = 162–192s, exit 1 — still red, but for an unrelated reason: the pre-existing ty diagnostics (27, red on main independently), not symlinks. Tests themselves complete.

On the Linux/Windows ratio

This is a local Windows measurement; the −45% is a real absolute win driven mostly by the two Windows-disproportionate code/config branches (per-worker import + SQLite churn) plus the test-data shrink. A matching Linux baseline would be needed to state the new ratio precisely, but the levers that closed it are exactly the Windows-specific ones.

[daaain]

Great work, I wondered why the Windows runners are so slow in CI!

[cboos]

Thanks - yes, this looks a bit better now (Windows jobs between 11m and 15m), though that can still feel a bit excessive...

We might want to continue to improve upon this in the future, from a different perspective, but for now we can close this iteration.