Skip to content

troubleshooting.md

Latest commit

 

History

History
111 lines (79 loc) · 3.23 KB

troubleshooting.md

File metadata and controls

111 lines (79 loc) · 3.23 KB

Troubleshooting

503 Errors / "High Demand" Message

When a developer sees "We're currently experiencing high demand, which may cause temporary errors", the error can come from two sources:

1. OpenAI (upstream)

The OpenAI API itself returns a 503 or error message during high load. The proxy forwards it unchanged.

How to verify:

./scripts/check-response-origin.sh "high demand"
./scripts/check-response-origin.sh "temporary errors"

If the script finds the text in S3 logs, the origin is OpenAI. Nothing to do on the proxy side.

2. Lambda throttling

If all reserved concurrency slots (25) are in use, new requests get throttled.

Check:

aws cloudwatch get-metric-statistics --region us-east-1 \
  --namespace AWS/Lambda --metric-name Throttles \
  --dimensions Name=FunctionName,Value=codex-proxy \
  --start-time "$(date -u -v-1H '+%Y-%m-%dT%H:%M:%SZ')" \
  --end-time "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" \
  --period 300 --statistics Sum

Fix: Increase reserved concurrency in deploy.sh (change --reserved-concurrent-executions).

Token Rejected (401)

The proxy returns 401 for:

  • Missing Authorization header: Developer didn't set OPENAI_API_KEY (which should contain their proxy token)
  • Invalid or disabled token: Token not in DynamoDB or enabled = false

Check token in DynamoDB:

aws dynamodb get-item --region us-east-1 --table-name codex-proxy-tokens \
  --key '{"token": {"S": "the-token"}}'

Path Normalization

Codex CLI may send requests to /responses instead of /v1/responses. The proxy automatically normalizes paths without /v1 prefix by prepending /v1.

Windows Setup

Option A: PowerShell script (recommended)

powershell -ExecutionPolicy Bypass -File scripts\setup-windows.ps1

Option B: Manual configuration

  1. Create %USERPROFILE%\.codex\config.toml:
model_provider = "codexproxy"
forced_login_method = "api"

[model_providers.codexproxy]
name = "Codex Proxy"
base_url = "https://YOUR_FUNCTION_URL.lambda-url.REGION.on.aws/v1"
env_key = "OPENAI_API_KEY"  # Codex reads this env var; the value is the proxy token, not an OpenAI key

  1. Set user environment variables:

    • OPENAI_BASE_URL = https://YOUR_FUNCTION_URL.lambda-url.REGION.on.aws
    • OPENAI_API_KEY = your proxy token (not an OpenAI API key)

  2. Open a new terminal and run codex.

Checking Error Origin

The proxy stores the full response_body from OpenAI in S3. To determine if an error message comes from OpenAI or the proxy:

# Search recent logs for a specific error message
./scripts/check-response-origin.sh "high demand"

# Search more logs
CHECK_ORIGIN_MAX=50 ./scripts/check-response-origin.sh "temporary errors"

Proxy-generated errors have these specific messages:

  • missing or invalid Authorization header
  • invalid or disabled token
  • token lookup failed
  • upstream request failed
  • internal error

Any other error text in the response comes from OpenAI.

Validation

Run the full validation suite:

./scripts/validate.sh

This checks:

  1. Lambda exists and has recent invocations
  2. CloudWatch Logs have recent entries
  3. DynamoDB has registered tokens
  4. S3 has today's logs
  5. Secrets Manager has the OpenAI key