cache() + Await + Spawn/Gather hangs at N>~100 tasks

[proboscis]

Summary

When cache(lifecycle="persistent") wraps a @do function containing yield Await(asyncio.sleep(...)), and many instances are dispatched via Spawn+Gather, the program hangs indefinitely at around N>100 tasks.

The Await inside compute_and_cache() never returns — tasks enter compute start but never reach compute done.

Reproduction

Self-contained test added in tests/test_cache_await_spawn_hang.py:

@cache(lifecycle="persistent")
@do
def _cached_async_task(i: int) -> EffectGenerator[str]:
    yield slog(msg=f"[{i}] compute start", level="info")
    yield Await(asyncio.sleep(0.01))
    yield slog(msg=f"[{i}] compute done", level="info")
    return f"result-{i}"

@do
def _spawn_gather_n(task_factory, n: int) -> EffectGenerator[list]:
    tasks = []
    for i in range(n):
        t = yield Spawn(task_factory(i), daemon=False)
        tasks.append(t)
    return list((yield Gather(*tasks)))

# Run with sqlite_cache_handler
wrapped = WithHandler(sqlite_cache_handler(None), _spawn_gather_n(_cached_async_task, 500))
r = run(wrapped, handlers=default_handlers())

uv run pytest tests/test_cache_await_spawn_hang.py -v --timeout=60

Results

Test	N	Result
no cache + Await + Spawn	500	✅ PASS (46s)
cache + Await + Spawn	50	✅ PASS (4s)
cache + Await + Spawn	100	✅ PASS
cache + Await + Spawn	200	❌ HANG (timeout)
cache + Await + Spawn	500	❌ HANG (timeout)

Observations

• Without cache(), the same Await + Spawn+Gather pattern works at any N
• With cache(), tasks that hit cache miss enter compute_and_cache(), the inner yield Await(asyncio.sleep(0.01)) never completes
• Log shows [i] compute start but never [i] compute done
• This blocks real-world pipelines (e.g., 754 cached LLM calls via Spawn+Gather)
• The threshold is somewhere between N=100 and N=200

Impact

This blocks any pipeline that uses cache() on async operations (LLM calls, HTTP requests) with Spawn+Gather at scale. Discovered while running a news indexing pipeline with ~754 events.

[proboscis]

Update: fix does not resolve the hang on Python 3.12

Tested with latest main (dd2a05a) on proboscis-ema (Python 3.12.5):

• p_100 (all cache hits from prior run): ✅ OK in 1.5s
• p_500 (100 cache hits + 400 cache miss): ❌ Still hangs
  • 3 tasks enter compute start (cache miss)
  • yield Await(asyncio.sleep(0.01)) never returns
  • Only "Acquiring semaphore" messages after that

Note: the doeff test suite on Python 3.14t also shows coroutine 'sleep' was never awaited warning and no outer handler for effect: PyShared(Handle) error — the Rust sync_await_handler may not be processing Await effects inside cache() compute_and_cache() context.

[proboscis]

Update after uv sync --upgrade --reinstall

Partial improvement: the first batch of 3 tasks (matching concurrency=3) now completes compute start → Await(asyncio.sleep) → compute done → Cache: stored. Previously compute done was never reached.

However, the 4th task onwards never starts. After the first 3 complete and release their semaphore slots, no new tasks acquire the semaphore. The log just stops after storing the 3rd result.

[231] compute start (cache miss)
[232] compute start (cache miss)  
[233] compute start (cache miss)
[231] compute done              ← NEW: these now complete!
[232] compute done
[233] compute done
Cache: stored result ...
Cache: stored result ...
Cache: stored result ...
                                ← HANG: no [234] compute start ever appears

So the Await inside cache now works for the initial batch, but the semaphore/scheduler doesn't continue dispatching subsequent tasks after the first batch completes.

[proboscis]

Reproducer now works in doeff repo (Python 3.14)

Updated tests/test_cache_await_spawn_hang.py — uses from doeff.effects import Await and _fake_work coroutine pattern (matching existing tests).

Results with latest main (dd2a05a):

Test	Result
no_cache 100	✅ PASS
no_cache 500	✅ PASS
cached 50	✅ PASS
cached 100	✅ PASS
cached 200	✅ PASS ← improved!
cached 500	❌ HANG

The threshold moved from ~100 to somewhere between 200-500. The scheduler fix helped but didn't fully resolve it.

On proboscis-ema (Python 3.12) with the same doeff version, cached 500 also still hangs (first 3 tasks complete but no subsequent tasks are dispatched).

[proboscis]

Key finding: it's OOM, not a hang!

Running test_cached_500 with verbose output and 120s timeout revealed:

[pytest-mem-guard] RSS 8198 MiB exceeded limit 8192 MiB. Requesting immediate pytest abort.

The test isn't hanging — it's consuming 8+ GB of memory with just 500 tasks doing Await(asyncio.sleep(0.01)) inside cache().

For reference: test_no_cache_500 (same tasks without cache()) completes in 46 seconds with normal memory.

This suggests cache() + Spawn + Await causes O(N²) or worse memory growth in VM internal state (trace entries? continuation chains?).

[proboscis]

Root cause found: GetExecutionContext() is O(N²) with Spawn

Memory bisect test results (N=500 Spawn+Gather tasks):

Variant	RSS delta	Result
bare Await	+217 MB	✅ OK
GetExecutionContext	+8500 MB	❌ OOM

GetExecutionContext() at cache.py:344 returns active_chain which apparently includes all 500 spawned continuations. Each of the 500 tasks calls it → O(N²) memory.

In cache(), GetExecutionContext() is only used for call-site info in error messages (line 344-347). The fix would be to either:

1. Make it lazy (only call on error path)
2. Return a lightweight summary instead of the full chain
3. Remove it from the hot path entirely

Bisect test added in tests/test_cache_memory_bisect.py.