bug(web): session-detail page connection-pool starvation — RSC sidebar prefetches + sync probes + first-failure-fatal cascade produces "Failed to load session"

[i-trytoohard]

Bug

Loading /projects/<projectId>/sessions/<sessionId> intermittently fails with the dashboard error:

Failed to load session. The session request is taking too long, so the page stopped waiting instead of spinning forever.

The failure is not a real server error — every aborted request in the HAR shows wait=0ms, blocked=5000–8140ms and status=0, meaning the browser client-aborted them after they sat queued in the HTTP/1.1 connection pool (Chrome caps at 6 concurrent per origin). The browser never actually opened a TCP connection for them.

The same page sometimes loads fine and sometimes shows the error, on the same machine, same code, same data, because the trigger is purely a race between RSC-rendering wall-clock time and the page's 8s AbortController deadline.

Source: live debugging session with HAR captures
Reported by: @i-trytoohard
Date: 2026-05-15
Analyzed against: 7d324b53 (origin/main HEAD)
AO version: 0.9.0 (PR #1849 / fix-1848 dist swapped — bug independent of this swap)
Environment: macOS, Node 25.9.0, Chromium-based browser
Confidence: High — root cause directly observed in two HAR captures + cross-traced to source.

What two HARs revealed

SUCCESS load (eventually rendered)

72 entries total
  /api/sessions/ao-187 calls : 19  (3 client-aborted, 16 succeeded)
  /api/sessions?fresh=true   : 14  (12 client-aborted, 2 succeeded)
  total wall time           : 212.3s of request time

FAILED load (error rendered)

38 entries total
  /api/sessions/ao-187 calls : 5  (5/5 client-aborted)
  /api/sessions?fresh=true   : 9  (9/9 client-aborted)
  total wall time           : 179.4s of request time

In both loads, all status=0 aborts have blocked ≈ deadline:

• /api/sessions/ao-187: blocked=8001-8140ms → matches SESSION_FETCH_TIMEOUT_MS=8000
• /api/sessions?fresh=true: blocked=5000-6002ms → matches PROJECT_SIDEBAR_FETCH_TIMEOUT_MS=5000

So the timeouts are client-side (AbortController), not server unresponsiveness. Browsers cap HTTP/1.1 same-origin connections at 6. The 6 slots are pinned, queued requests time out, page bails.

Root cause (multi-layer)

Layer 1 — 5 RSC project prefetches pin 5/6 connection slots

The session-detail page's sidebar renders <Link prefetch> (Next.js default) for every registered project in the sidebar:

• /projects/integrator?_rsc=10mlo
• /projects/workos?_rsc=10mlo
• /projects/agent-orchestrator?_rsc=10mlo
• /projects/mercury_8c1e44e68c?_rsc=10mlo
• /projects/eng-hiring_dd1da96287?_rsc=10mlo

These fire concurrently on mount and each take 1.3–7.3 seconds to complete because the server-side rendering of each project tree calls into enrichSessionsMetadata → agent plugin's getActivityState → findClaudeProcess/findCodexProcess → ps -eo pid,tty,args (the slow path partially mitigated by #1838 but still ≥200ms each).

Layer 2 — /mux WebSocket pins 1 more slot

The mux upgrade in the same HAR (status=101, time=55-74 seconds) pins 1 connection for its entire lifetime. Combined with Layer 1, 6/6 connection slots are now occupied.

Layer 3 — API requests queue behind them and hit deadlines

/api/sessions/<id> and /api/sessions?fresh=true get queued. After 8000ms / 5000ms they hit their AbortController deadlines and abort.

Layer 4 — fresh=true cascade compounds

useSessionEvents.scheduleRefresh() (packages/web/src/hooks/useSessionEvents.ts) fires /api/sessions?fresh=true on cadence (every 30s and on membership change). The HARs show 9 of these in the failed load — none with backoff or coordination. Each one would have triggered a full re-probe of all sessions server-side.

Layer 5 — First timeout becomes terminal UI state

packages/web/src/app/sessions/[id]/page.tsx:531:

setRouteError(err instanceof Error ? err : new Error("Failed to load session"));

One AbortError → terminal error UI. No retry-with-backoff. Yet the SUCCESS HAR shows successful fetches in later entries (#41, 43, 44, 47, 54, 55, 57, 60, 61, 63, 65, 69, 71) — data would have arrived if the page weren't locked in error state.

File:line references

What	Where
Client timeout constants	packages/web/src/app/sessions/[id]/page.tsx:86-88 (SESSION_FETCH_TIMEOUT_MS=8000, PROJECT_SIDEBAR_FETCH_TIMEOUT_MS=5000, PROJECTS_FETCH_TIMEOUT_MS=5000)
First-failure-fatal	packages/web/src/app/sessions/[id]/page.tsx:531 setRouteError(...)
Mount-time session fetch	packages/web/src/app/sessions/[id]/page.tsx:505
HTTP refresh that contends with mux	packages/web/src/hooks/useSessionEvents.ts:scheduleRefresh()
Server-side enrichment path	packages/web/src/app/api/sessions/[id]/route.ts → enrichSessionsMetadata() → agent plugin probe
RSC prefetch in sidebar	packages/web/src/components/ProjectSidebar.tsx — <Link> defaults to prefetch={true}

Reproduction

1. Have several projects registered (5+ to saturate the 6-slot pool reliably).
2. Have several running sessions (ao spawn 5-10).
3. From the dashboard, click any session to navigate to /projects/<projectId>/sessions/<id>.
4. Observe (with DevTools Network panel open): 5 RSC prefetches fire, sit at varying blocked=X times; session and sidebar fetches queue behind them.
5. Every few attempts the page shows the "Failed to load session" error; refreshing might succeed depending on probe timing.

Suggested fixes (in priority order)

1. <Link prefetch={false}> on sidebar project links. Or restrict prefetch to the currently-active project. Single-line change in ProjectSidebar.tsx. Frees 4 connection slots immediately and fixes the user-visible symptom for everyone with >2 registered projects.

2. Read sessions from disk index, not from probes. Background lifecycle-manager already writes enriched state to each <id>.json. /api/sessions/<id> should just deserialize that file (~5ms) — no synchronous probe. This kills the multiplicative slowdown that makes the RSC fetches slow in the first place. Pure indexing, no caching.

3. Concurrent-fetch dedup on the client. A Map<key, Promise> for in-flight fetches in client-fetch.ts so 15 simultaneous /api/sessions/ao-187 calls become 1 HTTP request. Not caching — purely concurrent-request coalescing.

4. First failure not fatal. Require N=3 consecutive failures (or N seconds of no progress) before setRouteError(...). With this, transient queue pressure becomes a hidden retry instead of a hard error.

5. Don't HTTP-poll when mux is connected and recent. useSessionEvents's scheduleRefresh() should skip when mux.status === "connected" && Date.now() - lastSnapshotAt < STALE_THRESHOLD. Eliminates the redundant ?fresh=true storm.

(1) alone likely fixes the user-visible bug entirely. (2) is the architectural win. (3) and (4) are belt-and-suspenders.

Related

• bug(web): sidebar shows "Failed to load sessions" on session detail page #1410 — sidebar "Failed to load sessions" — same connection-pool starvation but a different UI surface; same root cause path. The fix here would resolve that issue's sidebar manifestation too.
• bug(web): dashboard renders session as "Exited" when terminatedAt is set, regardless of lifecycle.session.state #1832 — dashboard renders Exited from wrong field — adjacent: about what the page shows when load succeeds, not whether load succeeds.
• ao status is slow and reports inconsistent session count vs tmux ls #1850 — ao status is slow — CLI-side manifestation of the same slow-probe family root cause. Different surface.
• bug(agent-plugins): ps timeout conflated with "process not found" — bulk-terminates all sessions when ps exceeds 5s #1838 — ps timeout = "process gone" (fixed in fix(agent-plugins,lifecycle): distinguish indeterminate probe from "not found" + bump ps timeout (closes #1838) #1839/fix(cli): reap daemon children on stop+SIGINT, sweep orphans on start (closes #1848) #1849). Made probes slow rather than fatal; this issue is what's left exposed once bug(agent-plugins): ps timeout conflated with "process not found" — bulk-terminates all sessions when ps exceeds 5s #1838's safety net is in place.
• lifecycle: terminated sessions never revive when probes later disagree — causes dashboard state flicker (idle → killed → ready) #1454 — terminated sessions never revive. Independent but in the same neighborhood — both root-cause-class bugs we've been chasing in tandem.

Impact

• Any user with >2 registered projects intermittently can't open session-detail pages.
• Especially visible after AO has been up a while and the process table is dense, slowing every RSC server-render.
• Workarounds: hard-refresh after seeing the error. Or open the URL directly (skips some prefetch). Or ao session attach from the terminal.

[i-trytoohard]

Regression — still occurring after #1856 (HAR-traced, High confidence)

Reported by: @prateek (orchestrator dogfooding) | Date: 2026-05-16 | Analyzed against: 667d1ded (sits on top of e6ad078d = #1856) | Confidence: High — Chrome DevTools HAR captures of two failing loads, correlated to exact code lines.

#1856 is deployed and this exact failure ("Failed to load session — the session request is taking too long, so the page stopped waiting") still reproduces. #1856 fixed the RSC prefetch storm + dedupe + retry, but did not close the dominant cost path: an explicit cache-bypass that the suffering page issues against itself.

HAR evidence (two failing loads)

/tmp/mer26_load_fail.har — GET /projects/mercury_8c1e44e68c/sessions/mer-26:

Request	Count	Time	Status
RSC document …/sessions/mer-26	1	55,449 ms (server wait/TTFB 33,246 ms)	200
GET /api/sessions/mer-26	7×	~8,000 ms each	0 (aborted at 8s client timeout)
GET /api/sessions?fresh=true	~8×	~6,000 ms each	0 (aborted)
GET /api/projects	1	5,786 ms	0 (aborted)

/tmp/int3_load_fail.har — GET /projects/integrator/sessions/int-3: RSC document 74,898 ms (server wait/TTFB 43,007 ms); GET /api/sessions/int-3 8,001 ms → status 0; GET /api/sessions?fresh=true ×2 → status 0; /api/projects → status 0.

Root cause: the session-detail page bypasses the listCached() cache against itself

packages/web/src/app/sessions/[id]/page.tsx:623-624 and :685 request:

/api/sessions?project=<p>&fresh=true
/api/sessions?fresh=true

packages/web/src/app/api/sessions/route.ts:87-88:

const coreSessions = fresh
  ? await sessionManager.list(requestedProjectId)        // UNCACHED — full probe fan-out
  : await sessionManager.listCached(requestedProjectId); // the #1737 cache

fresh=true ⇒ uncached sessionManager.list() ⇒ for every session: tmux has-session + ps runtime probe + gh PR/metadata enrichment (the #1858/#1865 fan-out, plus the unbounded request-path enrichment from #1685). The session-detail page fires this on open and polls it, several concurrent in-flight. They saturate next-server's limited concurrency, so:

1. The RSC document itself queues behind the fan-out → 33–43 s server TTFB.
2. Client GET /api/sessions/<id> aborts at SESSION_FETCH_TIMEOUT_MS = 8000 (page.tsx:86), retries 7×, all abort → the "taking too long" error UI (page.tsx:158).

#1737 added listCached() precisely to absorb this storm — but fresh=true deliberately defeats the cache, on the very page that is being starved by it. That is why #1856 (prefetch dedupe) did not fix it: the prefetch was deduped, but each retained request still forces the full uncached fan-out.

Fix direction

• Immediate: stop the session-detail page from passing fresh=true for the session-list/sidebar fetch (page.tsx:623-624,685) — use the cached path; 5 s SSE cadence does not need a forced uncached re-probe every tick. (Pairs with bug(web): newly spawned sessions don't propagate to sidebar — session API hangs + listCached() is process-local (regression from #1737) #1883 cause 2: make listCached cross-process-aware so cached data is still fresh enough.)
• Structural: the uncached list() should not fan-out synchronous per-session tmux/ps/gh probes on a request path at all — bug(core): centralize tmux/ps runtime probes behind a shared observation store #1858 (shared observation store) + bug(core): replace full process-table scan fanout with a benchmarked process observation strategy #1865 (process-scan strategy) + Session detail PR card shows +0/-0 instead of real diff stats #1685 (bound request-path enrichment). The page-data path must read persisted/observed state, not recompute it (same single-writer principle as sm.list() bypasses lifecycle probe decision matrix — single isAlive() failure writes terminated to disk #1735).
• Raising SESSION_FETCH_TIMEOUT_MS is not a fix — it would only convert fast failures into 30–70 s hangs (which is what the RSC document already does).

Requesting this be reopened — the user-visible failure in the title is reproducing in production on the latest build.

Related

• bug(web): newly spawned sessions don't propagate to sidebar — session API hangs + listCached() is process-local (regression from #1737) #1883 — sibling (session API hang + listCached process-local); shares the fan-out root.
• bug(web): make project RSC prefetch route cheap and prefetch-safe #1857 — make project RSC prefetch cheap/prefetch-safe (adjacent).
• bug(core): centralize tmux/ps runtime probes behind a shared observation store #1858 / bug(core): replace full process-table scan fanout with a benchmarked process observation strategy #1865 — probe fan-out = the actual per-request cost.
• Session detail PR card shows +0/-0 instead of real diff stats #1685 — unbounded request-path PR enrichment (adds to TTFB).
• fix(core): sm.list() no longer writes terminated state to disk #1737 — added listCached() that fresh=true bypasses.