WIP Fix: recover from network drops without restarting

[teocns]

Problem

After the network drops and comes back, teamclaude keeps failing every request
until you restart it. Accounts show as throttled or errored and Claude reports
API errors, even though nothing is actually rate limited.

The cause is a poisoned connection pool. teamclaude reuses a shared pool of
connections to the API across all requests and all accounts. When the network
drops the connections die, but nothing tells teamclaude they died, and because a
request has no time limit it waits on a dead connection forever instead of
failing. A hung request never errors, so the dead connection is never dropped
from the pool, and every retry and every account keeps landing on the same poison.

sequenceDiagram
    participant C as Claude
    participant T as teamclaude
    participant P as Connection pool
    participant A as API

    Note over P,A: network drops, pooled connections die silently

    C->>T: request
    T->>P: get a connection
    P-->>T: dead connection (looks alive)
    T->>A: send request on dead connection
    Note over T,A: no response, no time limit
    A--xT: nothing ever comes back (hangs)

    C->>T: times out, retries
    T->>P: get a connection
    P-->>T: another dead connection
    Note over T,A: hangs again, same poison

    Note over C,A: only restarting teamclaude builds a fresh pool

Loading

Fix

Add time limits so a stuck request fails quickly instead of hanging. The failure
lets teamclaude drop the dead connection from the pool, the client retries, and
the next request connects fresh. No restart needed.

• API requests now fail if no response starts within 120 seconds. The limit only
  covers the start of the response, so long replies keep streaming normally. It
  measures time to the first byte, which streaming delivers within seconds, so a
  slow or overloaded upstream that still answers (including a 429) is unaffected.
• Token refresh now fails after 30 seconds instead of hanging.

Both limits are adjustable with TEAMCLAUDE_UPSTREAM_HEADERS_TIMEOUT_MS and
TEAMCLAUDE_REFRESH_TIMEOUT_MS. No new dependencies.

sequenceDiagram
    participant C as Claude
    participant T as teamclaude
    participant P as Connection pool
    participant A as API

    C->>T: request
    T->>A: send request on dead connection
    Note over T,A: no response within 120s
    T->>P: drop dead connection from pool
    T-->>C: fail fast (error)
    C->>T: retry
    T->>A: send request on a fresh connection
    A-->>T: response
    T-->>C: response

Loading

Notes

Recovery is per-socket: each fast failure evicts one dead connection, so after a
flap with several in-flight requests it can take a few failed-then-retried
requests to fully drain. It always converges without a restart.

Tests added in test/upstream-timeout.test.js cover fast-fail on a dead socket,
same-origin socket eviction and reconnect, and that a slow response body is not
cut off once headers arrive. Full suite passes (141) and lint is clean.

[teocns]

Closing this: sandbox testing falsified the diagnosis.

I reproduced the failure in Docker (silent packet blackhole via iptables between a pooled client and upstream). Findings:

• Node's fetch pool self-heals after a network drop. New requests reconnect instantly once the network returns, in the same process. No restart needed at the socket layer.
• Idle pooled sockets are closed by undici after about 4 seconds anyway, so an idle drop leaves nothing to poison.
• The real "works only after restarting teamclaude" wedge is elsewhere: a 429 burst can put every account into an in-memory rate-limit hold (up to 1 hour). One request can throttle all accounts via failover. Nothing revalidates the hold while it lasts, so teamclaude keeps refusing even when the upstream is healthy again, and a restart "fixes" it only because the hold is not persisted.

A replacement PR with the evidence and the actual fix (probe revalidation for throttled accounts) is coming shortly.