Fix: revalidate throttled accounts so a stale 429 hold cannot require a restart

[teocns]

Symptom

After an internet drop and reconnect, teamclaude sometimes keeps failing every request even though the network and the API are fine again. All accounts show throttled, Claude reports API errors, and the only thing that helps is restarting teamclaude.

This replaces #73, which blamed dead pooled connections. Sandbox testing (below) disproved that diagnosis and found the real one.

What testing showed

All of this was measured in a Docker sandbox: a mock Anthropic upstream, teamclaude in a container, and a true silent network blackhole (iptables dropping all packets, no RST or FIN, exactly what a wifi drop looks like).

Belief	Measured reality
Dead pooled connections poison the pool until restart	False. After reconnect, the next request opened a fresh connection and succeeded in 10ms, same process
Idle sockets go stale during long idle periods	False. Node closes idle pooled sockets after about 4s, so an idle drop leaves nothing to poison
Restart is needed because of sockets	False at the socket layer. The restart ritual is explained by in-memory account state, below

The actual bug

A 429 response arms an in-memory hold on the account (rateLimitedUntil, up to 1 hour from retry-after). Two properties turn that into a lockout:

1. Failover amplifies. One client request during a 429 burst gets retried across accounts, so a single request can throttle every account at once.
2. Nothing revalidates a hold. While it lasts, the account is excluded from routing and from probing. If the 429s were transient (for example the retry burst right after a reconnect), the fleet stays locked long after upstream is healthy.

The hold is not persisted. That is the whole reason restarting "fixes" it: the restart wipes the hold, nothing else.

sequenceDiagram
    participant C as Client
    participant T as teamclaude
    participant A as API

    Note over C,A: network drops, then reconnects
    C->>T: request (retry burst)
    T->>A: try account alpha
    A-->>T: 429, retry-after 3600
    T->>A: failover to account beta
    A-->>T: 429, retry-after 3600
    Note over T: both accounts held in memory for 1h
    T-->>C: 429 "all accounts exhausted"

    Note over A: API fully healthy again
    C->>T: request
    T-->>C: 429 (hold still active, nothing rechecks it)
    Note over C,T: dead until hold expires or teamclaude restarts

Loading

Reproduced end to end in the sandbox: healthy → 200, then 429 burst → both accounts held 3600s, then upstream healthy again → still refused every request, then restart → instant 200.

Fix

Reuse the existing quota-probe machinery (the same mechanism that revalidates stale soft-quota reads). After a floor (TEAMCLAUDE_THROTTLE_PROBE_FLOOR_MS, default 60s), a throttled account becomes probe-eligible when no account is available, limited to one probe per probe interval. The probe is a real client request, so it costs nothing extra:

• any non-429 response clears the hold and puts the account back in rotation
• another 429 re-arms the hold with a fresh retry-after and pushes the next probe out a full floor

A genuine long rate limit is still respected: the fleet sends at most one request per minute against it, and every re-arm is based on what upstream just said rather than a stale snapshot.

Same sandbox, with the fix:

[TeamClaude] Account "acct-alpha" rate limited for 3600s
[TeamClaude] Account "acct-beta" rate limited for 3600s
[TeamClaude] All accounts unavailable — revalidating throttled "acct-beta" with a live request
[TeamClaude] Account "acct-beta" revalidated — rate limit no longer applies, back in rotation

Recovery without restart, on the first request past the floor. Unit tests cover the floor, the probe gating, clearing on non-429, re-arming on 429, and hold expiry.