Skip to content

Cut pytest-xdist per-worker import cost on Windows unit runs#250

Merged
cboos merged 5 commits into
mainfrom
dev/xdist-import-cost
Jun 30, 2026
Merged

Cut pytest-xdist per-worker import cost on Windows unit runs#250
cboos merged 5 commits into
mainfrom
dev/xdist-import-cost

Conversation

@cboos

@cboos cboos commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator

Part of #248.

Problem

On Windows, pytest-xdist spawns each worker and re-imports the whole
module graph per worker (Linux forks and skips this). Measured:

  • module import alone: ~12.9s serial
  • a single trivial test: -n0 4.5s vs -n auto 18.6s → ~14s of pure
    worker-spawn + re-import (12 workers)

Two heavy dependencies were imported on every unit worker even though
unit tests never use them:

  • pytest-playwright auto-loads via its entry point (~1s/worker)
  • browser/TUI test modules import playwright/textual at module top
    level, and pytest imports every collected module before marker
    deselection (~0.8s/worker for textual)

Change

  • test/conftest.py: a pytest_ignore_collect hook that skips
    *_browser.py / test_tui*.py modules only when they would be
    deselected anyway
    , evaluated via pytest's own marker Expression
    (so -m browser, -m tui, -m "not slow", and unmarked full runs all
    still collect them). Verified no unmarked test hides in those modules,
    so no coverage is lost.
  • -p no:playwright on every unit-invoking command (justfile
    test/test-all/test-cov + the CI unit step); browser/TUI steps are
    untouched so they still load the plugin.

Results (local Windows, 12-core)

  • collect-only wall (best-of-5, -n auto): 4.2s → 3.1s (~26%)
  • selection unchanged end-to-end (unit / -m browser / -m tui counts
    identical; browser + TUI runs still execute)

Worker count (deliberately not changed)

The spawn/import tax scales with worker count, but the right fix is to
make each worker cheaper, not to cap workers, so -n auto is kept. The
optimal CI worker count depends on the runner (GitHub windows-latest
has ~4 vCPUs vs a 12-core dev box), so no CI -n value is hard-committed
here — it should be A/B'd on the real runner. Full measurements in
work/xdist-import-cost.md.

Note: the collect-skip matcher keys off the filename pattern; a one-line
comment near it documents that coupling (a browser-marked test in a
non-matching filename stays collected on unit runs — harmless, just a
missed optimization).

Summary by CodeRabbit

  • Tests
    • Reduced unit test collection overhead by skipping browser and TUI tests earlier when they aren’t selected.
    • Disabled an unused browser testing plugin during unit test runs, keeping the same test coverage and filters.
  • Chores
    • Updated test commands and CI test steps to use the faster unit-test setup consistently.
    • Added notes documenting the test runtime improvement approach.

cboos and others added 5 commits June 25, 2026 11:19
pytest imports every test module during collection and only then applies
`-m` deselection, so plain unit runs imported the heavy optional deps
(playwright via 10 *_browser.py, textual via test_tui*.py) for nothing.
On Windows each xdist worker spawns a fresh interpreter and re-imports
the world, so this tax is paid per worker, every CI run.

- conftest: pytest_ignore_collect hook skips *_browser.py / test_tui*.py
  when their marker can't be selected by the active -m expression (uses
  pytest's own Expression evaluator). Removes the per-worker textual
  import (~0.8s/worker) from unit runs; selection is unchanged.
- justfile / CI unit step: `-p no:playwright` skips the unused
  pytest-playwright plugin on unit runs (~1s/worker playwright import).
  Kept out of pyproject addopts so browser runs still load it.
- work/xdist-import-cost.md: findings, measurements, and the
  worker-count recommendation (keep -n auto; CI Windows has ~4 vCPUs —
  the gap-closer is per-worker import cost, not worker count).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jun 29, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 5ee1ce0e-88a4-4619-893e-ad43966807e3

📥 Commits

Reviewing files that changed from the base of the PR and between 4bd633f and 78b17f0.

📒 Files selected for processing (4)
  • .github/workflows/ci.yml
  • justfile
  • test/conftest.py
  • work/xdist-import-cost.md

📝 Walkthrough

Walkthrough

Two optimizations reduce pytest startup and collection overhead for unit test runs: a new pytest_ignore_collect hook in conftest.py skips browser and TUI modules when their markers are excluded by the active -m expression, and -p no:playwright is added to all unit test invocations in justfile and the CI workflow.

Changes

Pytest collection and plugin optimizations

Layer / File(s) Summary
pytest_ignore_collect hook
test/conftest.py
Adds Optional import and a new hook that infers a module's marker from its filename, evaluates the active -m expression, and returns True to skip the module when the marker cannot be selected.
-p no:playwright flag in all unit test invocations
justfile, .github/workflows/ci.yml, work/xdist-import-cost.md
Adds -p no:playwright to the test, test-all, and test-cov targets in justfile and to the CI coverage step; includes a WIP investigation note documenting the root cause and measurements.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐇 No playwright here, no browser to load,
Just markers and hooks along the fast road.
We skip what we must, collect only true,
And Windows xdist runs faster now too!
Hip hop hooray, the imports are few! 🎉

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: reducing pytest-xdist per-worker import overhead on Windows unit test runs.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch dev/xdist-import-cost

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@cboos cboos merged commit f6e19ef into main Jun 30, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant