feat: add circuit breaker for upstream provider overload protection #75

kacpersaw · 2025-12-11T13:51:13Z

Summary

Implement per-provider circuit breakers that detect upstream rate limiting (429/503/529 status codes) and temporarily stop sending requests when providers are overloaded.

This completes the overload protection story by adding the aibridge-specific component that couldn't be implemented as generic HTTP middleware in coderd (since it requires understanding upstream provider responses).

Key Features

Per-provider circuit breakers - Separate circuit breakers for Anthropic and OpenAI since upstream rate limits are provider-specific
Configurable parameters - Failure threshold, time window, cooldown period, half-open max requests
Half-open state - Allows gradual recovery testing after cooldown
Prometheus metrics - State gauge, trips counter, rejects counter for monitoring
Thread-safe - Proper mutex protection for concurrent access
Disabled by default - For backward compatibility, must be explicitly enabled

Circuit Breaker States

┌─────────┐  failures >= threshold  ┌─────────┐
│ CLOSED  │ ──────────────────────► │  OPEN   │
│(normal) │                         │(reject) │
└─────────┘                         └────┬────┘
     ▲                                   │
     │                          cooldown expires
     │                                   │
     │  successes >= threshold     ┌─────▼─────┐
     └──────────────────────────── │ HALF-OPEN │
                                   │ (testing) │
     ┌─────────────────────────────┤           │
     │         any failure         └───────────┘
     │
     ▼
┌─────────┐
│  OPEN   │
│(reject) │
└─────────┘

Status Codes That Trigger Circuit Breaker

Code	Description
429	Too Many Requests
503	Service Unavailable
529	Anthropic Overloaded

Other error codes (400, 401, 500, 502, etc.) do not trigger the circuit breaker since they indicate different issues that circuit breaking wouldn't help with.

Default Configuration

Parameter	Default	Description
`Enabled`	`false`	Must be explicitly enabled
`FailureThreshold`	`5`	Failures needed to trip circuit
`Window`	`10s`	Time window for counting failures
`Cooldown`	`30s`	Time to wait before testing recovery
`HalfOpenMaxRequests`	`3`	Requests allowed in half-open state

New Prometheus Metrics

aibridge_circuit_breaker_state{provider} - Current state (0=closed, 1=open, 2=half-open)
aibridge_circuit_breaker_trips_total{provider} - Total times circuit opened
aibridge_circuit_breaker_rejects_total{provider} - Requests rejected due to open circuit

Files Changed

circuit_breaker.go - Core circuit breaker implementation
circuit_breaker_test.go - Comprehensive test suite (13 tests)
bridge.go - Integration into RequestBridge
interception.go - Apply circuit breaker to intercepted requests
metrics.go - Add Prometheus metrics

Testing

All tests pass:

go test -count=1 -short ./...
go vet ./...
go build ./...

Implement per-provider circuit breakers that detect upstream rate limiting (429/503/529 status codes) and temporarily stop sending requests when providers are overloaded. Key features: - Per-provider circuit breakers (Anthropic, OpenAI) - Configurable failure threshold, time window, and cooldown period - Half-open state allows gradual recovery testing - Prometheus metrics for monitoring (state gauge, trips counter, rejects counter) - Thread-safe implementation with proper state machine transitions - Disabled by default for backward compatibility Circuit breaker states: - Closed: normal operation, tracking failures within sliding window - Open: all requests rejected with 503, waiting for cooldown - Half-Open: limited requests allowed to test if upstream recovered Status codes that trigger circuit breaker: - 429 Too Many Requests - 503 Service Unavailable - 529 Anthropic Overloaded Relates to: coder/internal#1153

circuit_breaker.go

interception.go

circuit_breaker.go

…solation - Replace custom circuit breaker implementation with sony/gobreaker - Change from per-provider to per-endpoint circuit breakers (e.g., OpenAI chat completions failing won't block responses API) - Simplify API: CircuitBreakers manages all breakers internally - Update metrics to include endpoint label - Simplify tests to focus on key behaviors Based on PR review feedback suggesting use of established library and per-endpoint granularity for better fault isolation.

Rename fields to match gobreaker naming convention: - Window -> Interval - Cooldown -> Timeout - HalfOpenMaxRequests -> MaxRequests - FailureThreshold type int64 -> uint32

circuit_breaker.go

bridge.go

interception.go

go.mod

interception.go

metrics.go

circuit_breaker_test.go

…onfigs Address PR review feedback: 1. Middleware pattern - Circuit breaker is now HTTP middleware that wraps handlers, capturing response status codes directly instead of extracting from provider-specific error types. 2. Per-provider configs - NewCircuitBreakers takes map[string]CircuitBreakerConfig keyed by provider name. Providers not in the map have no circuit breaker. 3. Remove provider overfitting - Deleted extractStatusCodeFromError() which hardcoded AnthropicErrorResponse and OpenAIErrorResponse types. Middleware now uses statusCapturingWriter to inspect actual HTTP response codes. 4. Configurable failure detection - IsFailure func in config allows providers to define custom status codes as failures. Defaults to 429/503/529. 5. Fix gauge values - State gauge now uses 0 (closed), 0.5 (half-open), 1 (open) 6. Integration tests - Replaced unit tests with httptest-based integration tests that verify actual behavior: upstream errors trip circuit, requests get blocked, recovery after timeout, per-endpoint isolation. 7. Error message - Changed from 'upstream rate limiting' to 'circuit breaker is open'

- Add CircuitBreaker interface with Allow(), RecordSuccess(), RecordFailure() - Add NoopCircuitBreaker struct for providers without circuit breaker config - Add gobreakerCircuitBreaker wrapping sony/gobreaker implementation - CircuitBreakers.Get() returns NoopCircuitBreaker when provider not configured - Add http.Flusher support to statusCapturingWriter for SSE streaming - Add Unwrap() for ResponseWriter interface detection

bridge.go

circuit_breaker.go

- Changed CircuitBreaker interface to Execute(fn func() int) (statusCode, rejected) - Use gobreaker.Execute() to properly handle both ErrOpenState and ErrTooManyRequests - NoopCircuitBreaker.Execute simply runs the function and returns not rejected - Simplified middleware by removing separate Allow/Record pattern

…Bridge

bridge.go

circuit_breaker.go

…e gobreakerCircuitBraker along with the interface and noop struct

bridge.go

Co-authored-by: Paweł Banaszewski <pawel@coder.com>

circuit_breaker.go

circuit_breaker_test.go

circuit_breaker.go

circuit_breaker_integration_test.go

dannykopping

It's looking better now, but still a few things which need addressing please.

dannykopping · 2025-12-22T15:07:55Z

bridge.go

 	for _, provider := range providers {
+		// Create per-provider circuit breaker if configured
+		cfg := provider.CircuitBreakerConfig()
+		onChange := func(endpoint string, from, to gobreaker.State) {}


We should always log when the CB changes state.

You can move the check for cfg != nil && metrics != nil into onChange and only update metrics if needed.

dannykopping · 2025-12-22T15:10:41Z

bridge.go

-			mux.HandleFunc(path, newInterceptionProcessor(provider, recorder, mcpProxy, logger, metrics, tracer))
+			handler := newInterceptionProcessor(provider, recorder, mcpProxy, logger, metrics, tracer)
+			// Wrap with circuit breaker middleware (nil cbs passes through)
+			wrapped := CircuitBreakerMiddleware(cbs, metrics)(handler)


The PR description is now stale, btw.

dannykopping · 2025-12-22T15:13:40Z

circuit_breaker.go

+}
+
+// DefaultCircuitBreakerConfig returns sensible defaults for circuit breaker configuration.
+func DefaultCircuitBreakerConfig() CircuitBreakerConfig {


Not used, right?

dannykopping · 2025-12-22T15:17:31Z

circuit_breaker.go

+	switch statusCode {
+	case http.StatusTooManyRequests, // 429
+		http.StatusServiceUnavailable, // 503
+		529:                           // Anthropic "Overloaded"


I had a comment here before about this not being provider-specific; not sure what happened to it.
Each provider may handle rate-limits differently. That needs to go into the Provider interface.

dannykopping · 2025-12-22T15:26:35Z

config.go

 type ProviderConfig struct {
-	BaseURL, Key string
+	BaseURL, Key   string
+	CircuitBreaker *CircuitBreakerConfig


How do you imagine this will be configured?

dannykopping · 2025-12-22T15:35:39Z

circuit_breaker.go

+		}
+
+		return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			endpoint := strings.TrimPrefix(r.URL.Path, "/"+cbs.provider)


If two providers supported the same endpoint, you'd get one provider influencing another.

This was changed so ProviderCircuitBreakers struct is created per provider, separate sync.Map per provider.

dannykopping · 2025-12-22T15:40:19Z

circuit_breaker_integration_test.go

+	"go.opentelemetry.io/otel"
+)
+
+func TestCircuitBreaker_WithNewRequestBridge(t *testing.T) {


This is just testing Anthropic; we should support all providers.

dannykopping · 2025-12-22T15:41:24Z

circuit_breaker_integration_test.go

+	}
+
+	// First 2 requests hit upstream, get 429
+	for i := 0; i < 2; i++ {


Nit: reference the config (FailureThreshold) so the relationship is clear.

dannykopping · 2025-12-22T15:41:58Z

circuit_breaker_integration_test.go

+	assert.Equal(t, 1.0, state, "CircuitBreakerState should be 1 (open)")
+
+	rejects := promtest.ToFloat64(metrics.CircuitBreakerRejects.WithLabelValues(aibridge.ProviderAnthropic, "/v1/messages"))
+	assert.Equal(t, 1.0, rejects, "CircuitBreakerRejects should be 1")


This is just testing that a circuit-breaker will close, but not open again, so this isn't a comprehensive integration test.

dannykopping · 2025-12-22T15:43:44Z

circuit_breaker.go

+				if metrics != nil {
+					metrics.CircuitBreakerRejects.WithLabelValues(cbs.provider, endpoint).Inc()
+				}
+				http.Error(w, "circuit breaker is open", http.StatusServiceUnavailable)


Requests expecting JSON responses will be broken by plaintext responses.

kacpersaw marked this pull request as draft December 11, 2025 13:52

chore: apply make fmt

aad288c

kacpersaw marked this pull request as ready for review December 15, 2025 11:18

kacpersaw requested review from dannykopping and pawbana December 15, 2025 11:18

dannykopping requested changes Dec 16, 2025

View reviewed changes

circuit_breaker.go Show resolved Hide resolved

pawbana reviewed Dec 16, 2025

View reviewed changes

interception.go Outdated Show resolved Hide resolved

circuit_breaker.go Show resolved Hide resolved

kacpersaw added 3 commits December 16, 2025 10:27

refactor: align CircuitBreakerConfig fields with gobreaker.Settings

8cf2d18

Rename fields to match gobreaker naming convention: - Window -> Interval - Cooldown -> Timeout - HalfOpenMaxRequests -> MaxRequests - FailureThreshold type int64 -> uint32

refactor: remove CircuitState, use gobreaker.State directly

8e44145

kacpersaw requested review from dannykopping and pawbana December 16, 2025 11:01

dannykopping requested changes Dec 17, 2025

View reviewed changes

kacpersaw added 5 commits December 17, 2025 08:51

docs: clarify noop behavior when provider not configured

521df9b

Update go.mod

c85b836

fix: update metrics help text to reflect 0/0.5/1 gauge values

e446954

pawbana reviewed Dec 17, 2025

View reviewed changes

kacpersaw added 7 commits December 17, 2025 10:27

refactor: remove unused circuitBreakers field and getter from Request…

6a7d578

…Bridge

use per-provider maps for endpoints

b0ff0eb

make fmt

bee7a4d

use mux.Handle for cb middleware

98c7b7a

Move CircuitBreakerConfig to the Provider struct

7733266

Update tests

7c7c85b

kacpersaw requested a review from pawbana December 17, 2025 14:38

pawbana reviewed Dec 17, 2025

View reviewed changes

bridge.go Outdated Show resolved Hide resolved

bridge.go Outdated Show resolved Hide resolved

circuit_breaker.go Outdated Show resolved Hide resolved

default noop func for onChange

8943ef0

create CircuitBreakers per Provider instead of a global one and remov…

7d2dcb1

…e gobreakerCircuitBraker along with the interface and noop struct

pawbana reviewed Dec 17, 2025

View reviewed changes

bridge.go Show resolved Hide resolved

kacpersaw and others added 3 commits December 17, 2025 17:27

Update bridge.go

e3438f4

Co-authored-by: Paweł Banaszewski <pawel@coder.com>

fix format

a32f246

Apply review suggestions

e929098

pawbana reviewed Dec 18, 2025

View reviewed changes

circuit_breaker.go Outdated Show resolved Hide resolved

circuit_breaker_test.go Outdated Show resolved Hide resolved

circuit_breaker_test.go Outdated Show resolved Hide resolved

kacpersaw added 3 commits December 18, 2025 12:44

Apply review suggestions and add proper integration tests

ab08de4

Add test to check circuit breaker config

161db92

Remove test

33ea4ae

pawbana reviewed Dec 18, 2025

View reviewed changes

circuit_breaker.go Show resolved Hide resolved

Remove TestCircuitBreaker_HalfOpenAndRecovery

dbfab23

pawbana reviewed Dec 18, 2025

View reviewed changes

circuit_breaker_integration_test.go Outdated Show resolved Hide resolved

pawbana approved these changes Dec 18, 2025

View reviewed changes

kacpersaw requested a review from dannykopping December 22, 2025 15:04

dannykopping reviewed Dec 22, 2025

View reviewed changes

feat: add circuit breaker for upstream provider overload protection #75

Are you sure you want to change the base?

feat: add circuit breaker for upstream provider overload protection #75

Conversation

kacpersaw commented Dec 11, 2025

Summary

Key Features

Circuit Breaker States

Status Codes That Trigger Circuit Breaker

Default Configuration

New Prometheus Metrics

Files Changed

Testing

Related

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dannykopping left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants