Next.js dev server OOM crash — JavaScript heap out of memory (~4GB) on ao start

[i-trytoohard]

Bug Report

Reporter: @yyovil
Environment: macOS, Nix-managed Node.js 22.22.1 (nodejs-slim via Nix store), branch feat/droid-agent-plugin

Summary

Running ao start causes the Next.js dev server to consume ~4GB+ of heap memory and crash with a fatal OOM error. The process aborts and the user is forced to kill all running applications to recover system memory.

Crash Details

[next] FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Pre-crash errors observed in terminal:

[direct-terminal] [SessionBroadcaster] fetchSnapshot error: This operation was aborted
[direct-terminal] [MuxServer] Mux connection closed

GC data (last cycles before crash):

Scavenge / Mark-Compact — heap usage ~4087 MB to ~4135 MB

Stack trace highlights:

• V8 engine heap allocation failures (v8::internal::Heap::AllocateRawWithLightRetrySlowPath)
• v8::internal::Runtime_AllocateInYoungGeneration
• Builtins_StringSlowFlatten, Builtins_PromiseFulfillReactionJob
• libuv event loop (uv__io_poll, uv_run)
• Final: abort ao start — process exits with code null

Full native stack trace (37 frames):

1: 0x10429c9d4 node::OOMErrorHandler
2: 0x1033d7c40 v8::internal::V8::FatalProcessOutOfMemory
3: 0x103522a30 v8::internal::Heap::AllocateRawWithLightRetrySlowPath
...
11: 0x1033599d4 v8::internal::Runtime_AllocateInYoungGeneration
...
37: 0x1874c3da4 start [/usr/lib/dyld]
[1] 10191 abort ao start

All native frames resolve to /nix/store/0hwvnfndqy5jzdy05hq3rzwvbs9yask2-nodejs-slim-22.22.1/bin/node.

Impact

• Severity: High — renders ao start unusable on the affected branch/setup
• User had to quit all running applications to recover from the memory leak
• SessionBroadcaster fetchSnapshot errors and MuxServer disconnections precede the crash, suggesting a memory leak in the SSE/WebSocket broadcasting layer

Possible Root Causes

1. SessionBroadcaster snapshot polling — repeated aborted fetch operations may be leaking memory (unaborted fetches accumulating in memory)
2. MuxServer connection handling — closed mux connections may not be properly cleaned up
3. Next.js dev server + SSE stream — the 5s-interval SSE stream may accumulate uncollected response objects
4. Node.js slim build — nodejs-slim from Nix may have different memory management characteristics

Steps to Reproduce

1. Run ao start on branch feat/droid-agent-plugin
2. Observe terminal for SessionBroadcaster/MuxServer errors
3. Monitor memory usage — expect rapid growth to ~4GB
4. Process aborts with OOM

[i-trytoohard]

Crash Screenshots

Screenshot 1 — Heap OOM fatal error

Screenshot 2 — GC trace + exit code

Screenshot 3 — Full native stack trace (37 frames)

[i-trytoohard]

Root Cause Analysis

Investigated the codebase. Found multiple memory leak vectors, with one primary culprit.

🔴 PRIMARY: SessionBroadcaster fire-and-forget fetches (packages/web/server/mux-websocket.ts)

Every WebSocket client connect triggers an immediate fetchSnapshot() (~line 105) that is never aborted on disconnect. Each fetch:

• Creates its own AbortController with a 4s timeout
• Holds the HTTP response body (parsed JSON with session patches) in memory
• Captures closure references to subscriber callbacks

The immediate fetch and the 3s polling interval can run concurrently (no shared abort signal). On browser reconnects, HMR reloads, or flaky connections, these pile up. disconnect() clears the interval but does NOT abort in-flight fetches.

This matches the pre-crash errors:

[direct-terminal] [SessionBroadcaster] fetchSnapshot error: This operation was aborted
[direct-terminal] [MuxServer] Mux connection closed

🔴 SECONDARY: MuxServer per-connection memory

Each mux WebSocket connection creates closures capturing subscriptions Map, winPipes Map, winPipeBuffers Map, plus a 15s heartbeat interval. On rapid reconnects, peak memory spikes.

🟡 CONTRIBUTING: LifecycleManager internal Maps never cleared on stop()

packages/web/src/lib/services.ts — invalidatePortfolioServicesCache() calls lifecycleManager.stop() which kills the timer but never clears internal Maps: states, activityStateCache, reactionTrackers, prEnrichmentCache, branchAdoptionReservations, prEnrichmentCache. Old instances linger.

🟡 CONTRIBUTING: processedIssues Set grows forever

packages/web/src/lib/services.ts line 176 — module-level Set<string> that entries are added to but never evicted. No cap, no TTL.

---

Recommended Fixes

1. Share an AbortController in SessionBroadcaster — create one per broadcaster instance, abort it in disconnect(), and pass its signal to all fetchSnapshot() calls (both the immediate one in subscribe() and the polling interval).
2. Add stop() methods to SessionBroadcaster/NotificationBroadcaster that abort in-flight fetches and clear intervals. Call them in createMuxWebSocket() when the WebSocketServer closes.
3. Clear internal Maps in the lifecycle manager's stop() method — this.states.clear(), this.activityStateCache.clear(), etc.
4. Cap processedIssues at a reasonable size (e.g., 1000 entries) or use a TTL-based eviction.

Updated Screenshot — System OOM

The memory pressure is so severe it triggers macOS system-level warnings:

• Terminal: 6.83 GB (the ao start Node.js process)
• Comet (browser): 6.63 GB (dashboard SSE/WebSocket client)
• VS Code: 5.02 GB
• Figma: 2.74 GB

[yyovil]

Updated triage: why this shows up on ao start

New local report: ao start restored ao-orchestrator, then opening the restored dashboard URL eventually showed the session-detail timeout UI and the Next child crashed with V8 OOM.

Analyzed against: installed AO 0.9.0-nightly-a66a087ef678313bf3a78dfe70aadee9adc25935; local checkout 50dc18ffa1fe1e7944e214a5f854fba87b1afe61 with dirty Codex-plugin changes, so I fetched but did not pull over local work. upstream/main at time of triage: 1ed0719770c07081dbcb526419baca996b67c116.

Key clarification

This is not caused by the CLI startup loop alone. ao start creates the conditions and then the first dashboard/session request triggers the expensive path:

1. ao start launches ao-web/dist-server/start-all.js.
2. start-all launches Next and direct-terminal-ws.
3. The browser connects to /projects/<project>/sessions/ao-orchestrator and /mux.
4. The dynamic /projects layout and session-detail client load ask for dashboard/session data.
5. Those request paths fan out across all restored sessions and run agent metadata enrichment.

On this machine, the restored project has 67 AO session metadata files. The Codex history corpus has 979 JSONL files under ~/.codex/sessions, totaling ~2.0 GB, with individual files up to ~92 MB.

Root cause refinement

The prior #1935 hypothesis focused on SessionBroadcaster fetch leaks. That is a real amplifier, and #1936 is still a useful mitigation, but it is not the whole root cause.

The heavier trigger is request-path session enrichment over a large Codex JSONL corpus:

• packages/web/src/lib/dashboard-page-data.ts calls sessionManager.listCached() during dynamic project layout rendering.
• packages/web/src/app/api/sessions/route.ts uses sessionManager.list() whenever fresh=true, bypassing the cache.
• packages/core/src/session-manager.ts fans out enrichment over active session records.
• packages/plugins/agent-codex/src/index.ts currently caches session-file lookup by workspacePath only. In upstream/main it still calls findCodexSessionFile(workspacePath), which recursively lists ~/.codex/sessions/**/*.jsonl and opens prefix chunks to match session_meta.cwd, even when AO metadata already has codexThreadId.

The local crash report matched this: next-server (v15.5.18) died through node::OOMErrorHandler / v8::FatalProcessOutOfMemory, with hot frames in Builtins_StringSlowFlatten and Builtins_StringIndexOf. That lines up with repeated string searching/parsing over JSONL data, plus overlapping dashboard/API/mux requests.

Why it appears specifically during ao start

ao start restores the existing session set and starts the dashboard with a cold session cache. The first dashboard open after startup then needs to build sidebar/session state from scratch. Because all 67 sessions are restored in metadata and the cache is cold, the initial request can trigger the worst-case all-session enrichment path immediately.

After the Next child OOMs, ao start remains half-alive because start-all.ts only restarts direct-terminal (restart: true) and does not restart the next child. In the observed process state, direct-terminal-ws was still listening on 14801, while port 8000 was no longer listening.

Existing related work

• fix(web): coalesce session snapshot fetches #1936 (fix(web): coalesce session snapshot fetches) should reduce duplicate /api/sessions/patches pressure, but will not by itself eliminate full Codex corpus scans from session/detail/dashboard request paths.
• fix(agent-codex): close codex JSONL streams on abort #1977 (fix(agent-codex): close codex JSONL streams on abort) is merged and included in the installed nightly used for this report; the OOM still occurred, so stream cleanup is necessary but insufficient.
• bug(web): session-detail page connection-pool starvation — RSC sidebar prefetches + sync probes + first-failure-fatal cascade produces "Failed to load session" #1855 tracks the user-visible session-detail timeout family. The current report is the same family plus the Next OOM failure mode.

Fix direction

1. Make persisted codexThreadId a hard fast path in agent-codex: resolve by filename and cache by thread:<id> before any cwd scan. Fallback cwd scans should be byte/line bounded and cancellable.
2. Stop using uncached fresh=true session list calls on session-detail/sidebar refresh paths unless there is a deliberate user action requiring a synchronous re-probe.
3. Keep fix(web): coalesce session snapshot fetches #1936 or equivalent snapshot coalescing to avoid duplicated mux-driven pressure.
4. Change start-all behavior so a Next crash is visible: either restart Next with a cap or make ao start fail instead of leaving a half-alive mux-only dashboard.

Confidence: high for the startup trigger path and OOM class; medium-high for the exact allocation hotspot because the native stack only gives V8 builtin frames, not JS source frames.

[yyovil]

Triage split: OOM is produced by several cooperating bugs

Follow-up from the 2026-05-22 local ao start / ao-orchestrator investigation. This issue remains the correct place for the Next V8 OOM on first dashboard load after ao start. The root is not one leak only; several bugs line up:

1. bug(core,web): dashboard request paths run expensive agent enrichment for terminated/runtime-lost sessions #1991 — dashboard request paths perform expensive agent/runtime enrichment over persisted session records, including runtime-lost/terminated rows. This puts observation work on Next request handlers instead of the lifecycle manager/background state store.
2. bug(agent-codex): dashboard enrichment ignores persisted codexThreadId and scans global Codex history #1990 — Codex enrichment does not use persisted codexThreadId as the hard fast path and can scan the global ~/.codex/sessions corpus. On the reporting machine that corpus was ~2.0 GB / 979 JSONL files.
3. bug(web): session-detail page connection-pool starvation — RSC sidebar prefetches + sync probes + first-failure-fatal cascade produces "Failed to load session" #1855 — session-detail page issues fresh=true/client fetches that contend with RSC/layout/session loading and can hit the 8s timeout.
4. fix(web): coalesce session snapshot fetches #1936 — uncoalesced mux snapshot fetches amplify /api/sessions/patches pressure. This is a mitigation PR, but not sufficient by itself.
5. fix: dashboard terminal endpoints die without supervision causing recurring /ao outages #417 — process supervision gap: after the Next child crashes, ao start can remain half-alive with direct-terminal-ws still listening while dashboard HTTP is dead.

The important correction from the local state: there were not 67 live tmux sessions. There were 67 persisted AO session metadata files and only one live tmux session (ao-orchestrator). The OOM occurs because the dashboard read path still treats those persisted rows as candidates for request-time enrichment.

[yyovil]

Post-#1992 update: OOM still reproduces, narrowed to request-path summary enrichment

After #1992 merged and was installed locally, ao start initially loaded the dashboard, then Next crashed again after a few minutes with V8 OOM:

[next] FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
[next] ... Builtins_StringSlowFlatten
[next] ... Builtins_StringIndexOf
[next] ... node::fs::AfterInteger
[next] exited with code null

Crash report:

~/Library/Logs/DiagnosticReports/node-2026-05-22-023651.ips

It confirms next-server (v15.5.15) died through node::OOMErrorHandler / v8::internal::V8::FatalProcessOutOfMemory.

Important correction: #1992 fixed Codex JSONL discovery by codexThreadId, but the dashboard still performs Codex getSessionInfo() summary enrichment on request paths for many terminated/runtime-missing sessions. On this machine there are 25 Codex AO sessions with codexThreadId, 24 terminated/runtime-missing, and none have persisted summaries. The exact matched JSONL files total ~39 MB. SSR/API enrichment races this work against a 3s timeout via settlesWithin, but that helper does not cancel the underlying JSONL streams, so repeated dashboard/API/mux traffic can leave old enrichment work running and accumulate heap until Next OOMs.

Detailed root-cause update is on #1991: #1991 (comment)

The #417 half-alive behavior also reproduced again: after Next exited, ao start and direct-terminal-ws remained alive, port 14801 was listening, and port 8000 was not.

[yyovil]

Post-#1994 regression evidence: session detail still times out and Next still OOMs

Reported from: local AO dogfooding after installing #1994 from durable worktree commit 63ad054566e706eed989d1d9ce0b696bda58a4b6
Date: 2026-05-22 IST
Installed AO path: /Users/tanishqpalandurkar/Projects/worktrees/agent-orchestrator-pr-1994-install/packages/ao
Current upstream/main fetched: d5249228498fc5cb8f14609fcd81baa3630e5076
HAR: ~/ao_timeout_again.har
Crash report: ~/Library/Logs/DiagnosticReports/node-2026-05-22-032350.ips

What happened

After ao stop / ao start, the dashboard loaded initially. Reloading:

/projects/agent-orchestrator_48321dec7a/sessions/ao-orchestrator

again produced:

Failed to load session
The session request is taking too long...

The ao start logs then showed repeated mux snapshot aborts followed by Next V8 OOM:

[direct-terminal] [SessionBroadcaster] fetchSnapshot error: This operation was aborted
[next] FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
[next] ... Builtins_StringSlowFlatten
[next] ... Builtins_StringIndexOf
[next] ... node::fs::AfterInteger
[next] exited with code null

After the crash, ao start and direct-terminal-ws were still alive, but port 8000 was no longer listening.

HAR evidence

Relevant entries from ~/ao_timeout_again.har:

Request	Result
GET /projects/agent-orchestrator_48321dec7a/sessions/ao-orchestrator	200, ~60.2s total, TTFB ~0.66s, receive phase ~59.6s
duplicate document request	200, ~60.2s total
GET /api/sessions/ao-orchestrator	status=0, ~8003ms, client-aborted at the page's 8s timeout
second GET /api/sessions/ao-orchestrator	status=0, ~8001ms
repeated GET /api/sessions	~2.5-3.8s entries, some client-aborted
GET ws://localhost:14801/mux	101, long-lived

The failure is still the session-detail timeout family: the JSON detail request never completes before the client aborts, while the document request streams slowly.

Why #1994 was not enough

#1994 is installed and its built dist contains the metadata-only Codex fast path. A direct core/plugin reproduction outside Next is fast and low-memory:

sessionManager.get('ao-orchestrator')  ~70ms, RSS ~76MB
sessionManager.get('ao-70')            ~57ms, RSS ~76MB
sessionManager.listCached()           ~213ms, RSS ~77MB

So the basic Codex plugin fix is present and working for direct core calls.

The remaining failing path is the web/session-detail request pattern:

• packages/web/src/app/projects/[projectId]/sessions/[id]/page.tsx still polls every 2s.
• For an orchestrator session, it calls /api/sessions?project=<projectId>&fresh=true for project zone counts.
• The non-project session page also calls /api/sessions?fresh=true for the sidebar.
• packages/web/src/app/api/sessions/route.ts maps fresh=true to sessionManager.list(...), bypassing listCached().
• Those requests overlap with /api/sessions/:id and mux /api/sessions/patches fetches.

That means #1994 removed one expensive Codex parsing case, but the page still keeps forcing uncached all-session request-path work under load. The settlesWithin(...) callers still only stop waiting; they do not cancel already-started server-side enrichment.

Current diagnosis

This is a cooperating failure between:

1. This issue's session-detail fresh=true/request contention behavior.
2. bug(core,web): dashboard request paths run expensive agent enrichment for terminated/runtime-lost sessions #1991's broader bug: request paths still own observation/enrichment work that should be background/persisted.
3. Next.js dev server OOM crash — JavaScript heap out of memory (~4GB) on ao start #1935's Next OOM surface: repeated timed-out request work can still push next-server to the ~4GB V8 heap limit.
4. fix: dashboard terminal endpoints die without supervision causing recurring /ao outages #417's supervision gap: after the Next child exits, AO stays half-alive with only direct-terminal running.

#1994 should still be merged/kept because it removes a real Codex-plugin cost center, but it is not the final fix for this issue. The next practical fix should remove fresh=true from session-detail/sidebar polling and make those paths use cached/persisted session state by default.