Skip to content

fix(cpp): degrade gracefully when progress Manager cannot spawn (#339)#348

Draft
breimanntools wants to merge 1 commit into
masterfrom
fix/339-mp-manager-eof
Draft

fix(cpp): degrade gracefully when progress Manager cannot spawn (#339)#348
breimanntools wants to merge 1 commit into
masterfrom
fix/339-mp-manager-eof

Conversation

@breimanntools

Copy link
Copy Markdown
Owner

Closes #339.

Summary

CPP.run(..., n_jobs>1) crashed in non-interactive contexts (python -c, heredocs, some subprocess
shells) where multiprocessing.Manager() raised EOFError/OSError while setting up the
cross-process progress bar. The only workaround was n_jobs=1.

Details

  • _get_mp_shared() now creates the Manager best-effort: on failure it leaves all module globals
    untouched (no half-initialized state, no bumped refcount, half-started Manager shut down) and
    returns None, so _resolve_shared() degrades to the thread-safe DEFAULT_SHARED_* progress path.
  • Because the assign / pre-filter dispatch uses joblib's process backend, a new _worker_shared()
    helper hands Manager proxies to workers unchanged (byte-identical happy path) but passes None for
    the thread-safe fallbacks (a threading.Lock cannot be pickled to a spawned process), so each
    worker self-resolves its own process-local progress and the run completes.
  • Progress is cosmetic-only; output is unchanged when the Manager is available.

Ripple

  • tests/unit/cpp_tests/test_progress_backend.py — Manager failure/degradation arm, clean-globals
    invariant, resolve fallback, and an end-to-end CPP.run(n_jobs=2) under a forced Manager failure.
  • release-notes Unreleased Fixed entry.

Part of epic #336.

🤖 Generated with Claude Code

CPP.run(..., n_jobs>1) crashed in non-interactive contexts (python -c,
heredocs, some subprocess shells) where multiprocessing.Manager() raised
EOFError/OSError while setting up the cross-process progress bar. The only
workaround was n_jobs=1.

_get_mp_shared() now creates the Manager best-effort: on failure it leaves
all module globals untouched (no half-initialized state, no bumped refcount,
half-started Manager shut down) and returns None, so _resolve_shared()
degrades to the thread-safe DEFAULT_SHARED_* progress path. Because the
assign / pre-filter dispatch uses joblib's process backend, a new
_worker_shared() helper hands Manager proxies to workers unchanged (byte-
identical happy path) but passes None for the thread-safe fallbacks (a
threading.Lock cannot be pickled to a spawned process) so each worker
self-resolves its own process-local progress and the run completes. Progress
is cosmetic-only; output is unchanged when the Manager is available.

Adds tests/unit/cpp_tests/test_progress_backend.py covering the Manager
failure/degradation arm, clean-globals invariant, resolve fallback, and an
end-to-end CPP.run(n_jobs=2) under a forced Manager failure.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@codecov

codecov Bot commented Jul 4, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.66667% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.83%. Comparing base (7dcc8d8) to head (3e601f9).
⚠️ Report is 7 commits behind head on master.

Files with missing lines Patch % Lines
...ure_engineering/_backend/cpp/_filters/_progress.py 90.90% 2 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##           master     #348   +/-   ##
=======================================
  Coverage   94.83%   94.83%           
=======================================
  Files         196      196           
  Lines       18767    18787   +20     
  Branches     3175     3177    +2     
=======================================
+ Hits        17797    17817   +20     
- Misses        633      635    +2     
+ Partials      337      335    -2     
Files with missing lines Coverage Δ
...ature_engineering/_backend/cpp/_filters/_assign.py 98.85% <100.00%> (+<0.01%) ⬆️
..._engineering/_backend/cpp/_filters/_stat_filter.py 92.72% <100.00%> (+0.04%) ⬆️
...ure_engineering/_backend/cpp/_filters/_progress.py 85.50% <90.90%> (+5.11%) ⬆️
Components Coverage Δ
cpp_core 95.08% <91.66%> (+0.13%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CPP.run(n_jobs>1) crashes with multiprocessing.Manager() EOFError outside notebooks

1 participant