README.md.draft

# CodeAgentLens

CodeAgentLens is a local model-endpoint gateway and observability workbench for code agents. It gives Claude Code, Codex, and other OpenAI/Anthropic-compatible clients one local gateway while keeping traces, prompts, request artifacts, usage, endpoint decisions, and logs inspectable on the same machine.

This project is an independently maintained derivative of ccNexus.

Upstream: https://github.com/lich0821/ccNexus

## What CodeAgentLens Provides

- A loopback Gateway/API at `http://127.0.0.1:3010` for code-agent model traffic.
- A separate Debug Portal at `http://127.0.0.1:3011/debug/obs` for observability and local artifact inspection.
- OpenTelemetry traces and metrics exported to a bundled local stack.
- Local request dumps with prompt extraction, headers, request/response bodies, transformed bodies, stream events, and usage files.
- Endpoint routing, retry, credential rotation, and token usage accounting across Claude, OpenAI Chat, and OpenAI Responses style clients.
- Source-controlled local validation commands for native and Docker runtime modes.

## Architecture

```mermaid
flowchart LR
  subgraph Agents[Code agent clients]
    Claude[Claude Code\n/v1/messages]
    Codex[Codex CLI\n/v1/responses or /v1/chat/completions]
    Other[Other local clients\nOpenAI-compatible HTTP]
  end

  subgraph Gateway[CodeAgentLens Gateway :3010]
    API[Gateway HTTP API]
    Resolver[Endpoint resolver\nactive endpoint selection]
    Transformer[Protocol transformers\nClaude / OpenAI Chat / OpenAI Responses]
    Proxy[Proxy and retry engine]
    Usage[Usage extractor\ntokens and credential usage]
    Store[(SQLite config, stats, credentials)]
  end

  subgraph Obs[Local observability runtime]
    Recorder[Request recorder]
    Dump[Local dump writer\nruns/traces/requests]
    OTel[OpenTelemetry tracer and meter]
    Portal[Debug Portal :3011]
  end

  subgraph Upstream[Model endpoints]
    ProviderA[Provider A]
    ProviderB[Provider B]
    Smoke[Local synthetic smoke upstream]
  end

  Claude --> API
  Codex --> API
  Other --> API
  API --> Resolver --> Transformer --> Proxy --> Upstream
  Proxy --> Usage --> Store
  API --> Recorder
  Proxy --> Recorder
  Recorder --> Dump
  Recorder --> OTel
  Dump --> Portal
  OTel --> Portal
```

## Deployment Diagram

```mermaid
flowchart TB
  User[User browser]
  Agent[Code agent CLI]

  subgraph Host[Developer machine]
    subgraph Runtime[CodeAgentLens runtime]
      Gateway[Gateway/API\n127.0.0.1:3010]
      DebugPortal[Debug Portal\n127.0.0.1:3011/debug/obs]
      Data[(SQLite DB\nconfig, endpoints, stats)]
      Artifacts[(Observability dump root\nruns/*/traces/*/*)]
    end

    subgraph Stack[Bundled local observability stack]
      OTelCollector[OTel Collector\n4317 / 4318 / 8888]
      Jaeger[Jaeger\n16686]
      Grafana[Grafana\n13000]
      Prometheus[Prometheus\n9090]
      Tempo[Tempo\n3200]
    end
  end

  Agent -->|LLM API traffic| Gateway
  Gateway -->|traceparent + spans + metrics| OTelCollector
  Gateway -->|local artifacts| Artifacts
  Gateway --> Data
  User -->|inspect requests and prompts| DebugPortal
  DebugPortal --> Artifacts
  DebugPortal -->|wrapper links| Jaeger
  DebugPortal -->|wrapper links| Grafana
  OTelCollector --> Jaeger
  OTelCollector --> Prometheus
  OTelCollector --> Tempo
  Grafana --> Tempo
  Grafana --> Prometheus
```

Default local entry points:

| Component | URL | Purpose |
| --- | --- | --- |
| Gateway/API | `http://127.0.0.1:3010` | Local code-agent model endpoint. |
| Debug Portal | `http://127.0.0.1:3011/debug/obs` | Main UI for traces, prompts, artifacts, and tool links. |
| Jaeger | `http://127.0.0.1:16686` | Trace search and span timing. |
| Grafana | `http://127.0.0.1:13000` | Dashboards and Tempo/Prometheus queries. |
| Prometheus | `http://127.0.0.1:9090/graph` | Metrics query UI. |
| Tempo | `http://127.0.0.1:3200/status` | Trace storage status. |
| OTel Collector | `http://127.0.0.1:8888/metrics` | Collector diagnostics. |

## Observability Features

CodeAgentLens separates traffic handling from observability access. The Gateway/API listens on `3010` and agent clients only need that URL. The Debug Portal listens on `3011` and is the place to inspect what happened after a request flows through the gateway.

### Integrated Modules

| Module | What it does | Where to look |
| --- | --- | --- |
| Gateway listener | Receives Claude, OpenAI Chat, and OpenAI Responses style requests from code agents. | `http://127.0.0.1:3010/health` and gateway logs. |
| Endpoint resolver | Chooses the active enabled endpoint, supports manual switching, and protects in-flight endpoint requests during rotation. | Web UI and request logs. |
| Protocol transformers | Converts between supported client and upstream formats, including Claude, OpenAI Chat, and OpenAI Responses paths. | Request artifacts named `transform.request.*` and transformed response artifacts. |
| Proxy and retry engine | Sends upstream requests with connection pooling, retries transient failures, tracks active requests, and records endpoint failures. | Debug Portal request detail, logs, and usage artifacts. |
| Token and usage extractor | Normalizes input/output token usage from non-streaming and streaming responses. | `usage.json` inside each request artifact directory. |
| Request recorder | Creates request IDs, trace IDs, span IDs, obs references, viewer URLs, and OTel attributes. | `/debug/obs/trace/{trace_id}` and Jaeger span attributes. |
| Local dump writer | Writes manifest-bound request artifacts under `runs/{run_id}/traces/{trace_id}/{request_id}`. | `/debug/obs/request/{request_id}`. |
| Prompt extractor | Extracts system, developer, user, and other prompt roles from supported API bodies. | `/debug/obs/prompts` and `/debug/obs/request/{request_id}/prompts`. |
| Stream event capture | Records raw and transformed SSE events when stream capture is enabled. | Stream artifacts and `/debug/obs/request/{request_id}`. |
| Debug Portal | Provides the local UI for recent requests, prompt sessions, errors, trace links, raw artifacts, and integrated tools. | `http://127.0.0.1:3011/debug/obs`. |
| Jaeger integration | Shows distributed trace timing and links back to CodeAgentLens artifacts through `code-agent-lens_obs_ref`. | `/debug/obs/tool/jaeger` or native Jaeger. |
| Grafana integration | Opens local dashboards and Tempo/Prometheus views while preserving Portal navigation. | `/debug/obs/tool/grafana` or native Grafana. |
| Prometheus metrics | Exposes counters and histograms for requests, errors, retries, endpoint rotations, stream events, credential refreshes, and token totals. | Prometheus or Grafana. |
| Validation CLI | Produces evidence for stack profiles, local debug policy, synthetic traces, auth simulations, and port checks. | `go run ./cmd/code-agent-lens obs ...`. |

### Captured Artifacts

For each observable request, CodeAgentLens can write a manifest-bound directory containing:

- `prompt.index.json`: request metadata, prompt role manifest, file manifest, trace ID, span ID, request ID, and run ID.
- `ingress.request.headers.json` and `ingress.request.body.raw`: client-side request data, with secret headers redacted unless explicitly enabled for local debugging.
- `prompt.system.txt`, `prompt.developer.txt`, `prompt.user.txt`, and numbered variants: extracted prompt text by role.
- `transform.request.input.raw` and `transform.request.output.raw`: transformer input and output snapshots.
- `upstream.request.*` and `upstream.response.*`: upstream URL, headers, body, and transformed response data.
- `stream.raw.events.jsonl` and `stream.transformed.events.jsonl`: streaming event evidence when enabled.
- `usage.json`: normalized input and output token counts plus the source of the usage data.

The Debug Portal serves artifacts through manifest-bound routes, not arbitrary filesystem paths.

## Quickstart

Run static source validation:

```powershell
go run ./cmd/code-agent-lens obs validate --deployment-profile local_debug --profile deploy/observability/stack.local.yaml --evidence-dir .tmp/release-gate/observability/native
```

Start the native local debug runtime when ports `3010` and `3011` are free:

```powershell
go run ./cmd/code-agent-lens obs up --mode native --deployment-profile local_debug --profile deploy/observability/stack.local.yaml --evidence-dir .tmp/release-gate/observability/native
```

Run full-chain synthetic trace validation after the runtime is up:

```powershell
go run ./cmd/code-agent-lens obs validate --mode native --deployment-profile local_debug --profile deploy/observability/stack.local.yaml --synthetic --trace --evidence-dir .tmp/release-gate/observability/native
```

Stop the native runtime:

```powershell
go run ./cmd/code-agent-lens obs stop --mode native --profile deploy/observability/stack.local.yaml --evidence-dir .tmp/release-gate/observability/native-stop
```

Check that the official ports are free:

```powershell
go run ./cmd/code-agent-lens obs ports --expect-free 3010,3011 --evidence-dir .tmp/release-gate/observability/native-stop
```

For Docker mode, use the bundled compose profile after the official ports are available:

```powershell
docker compose -f deploy/observability/docker-compose.full.yaml up -d --build
```

## Common User Cases: Q/A

### Q: I want to know what prompt my code agent actually sent. Where do I look?

Open `http://127.0.0.1:3011/debug/obs/prompts`. Use the search, role, and run filters to find the request. Open the request's prompt details to see extracted system, developer, user, and other prompt roles.

### Q: I have a trace ID. How do I open the related request artifacts?

Open `http://127.0.0.1:3011/debug/obs/trace/{trace_id}`. The trace page lists the request IDs found under that trace and links to request details, prompt details, raw bodies, and artifact views.

### Q: I have an `obs_ref` from Jaeger. How do I jump back to CodeAgentLens?

Open `http://127.0.0.1:3011/debug/obs/ref/{trace_id}/{request_id}`. Jaeger can also use `docs/observability/jaeger-ui-config.json` to make `code-agent-lens_obs_ref` clickable.

### Q: I want to see whether the agent used Claude Messages, OpenAI Chat, or OpenAI Responses format.

Open the request detail page and inspect the ingress body plus transformer artifacts. The Gateway detects client format from paths such as `/v1/messages`, `/v1/chat/completions`, and `/v1/responses`.

### Q: I want to inspect endpoint decisions or why a request went to a provider.

Start from `/debug/obs`, open the request, then inspect upstream request artifacts and logs. Endpoint state is also represented in the Web UI and persisted in the local SQLite-backed configuration and stats store.

### Q: I want to compare raw upstream data with transformed output.

Open `/debug/obs/request/{request_id}`. Use the artifact links for ingress request body, transform input, transform output, upstream request body, upstream response body, and upstream response transformed body.

### Q: I need token usage for a single agent request.

Open `/debug/obs/request/{request_id}` and inspect `usage.json`. It records normalized input and output tokens plus whether usage came from upstream data or fallback extraction.

### Q: I need to debug a streaming response.

Open the request detail page and inspect stream artifacts. When stream capture is enabled, CodeAgentLens writes raw and transformed stream event JSONL files so you can compare upstream SSE shape with client-facing SSE shape.

### Q: I want to know why a code-agent request was slow.

Open `/debug/obs/tool/jaeger` or native Jaeger at `http://127.0.0.1:16686`. Search for the CodeAgentLens service and follow spans such as gateway request handling, ingress capture, upstream request, usage extraction, and stream processing.

### Q: I want dashboards or metrics, not raw traces.

Open `/debug/obs/tool/grafana` or native Grafana at `http://127.0.0.1:13000`. Prometheus metrics include request totals, error totals, request duration, upstream duration, retries, endpoint rotations, token counters, stream events, and credential refresh/failure counters.

### Q: I want to confirm the Portal is separate from the Gateway/API.

Gateway traffic uses `127.0.0.1:3010`. Debug Portal traffic uses `127.0.0.1:3011/debug/obs`. The gateway deliberately returns not found for `/debug/obs` so agent tokens and model-call traffic do not open the Portal listener.

### Q: I only want local debugging. Will CodeAgentLens publish my prompts to a remote service?

The Debug Portal reads local dump files from the configured dump root. Full prompt/body artifacts are local debug artifacts. OpenTelemetry exports traces and metrics to the configured local collector in the bundled local stack.

### Q: What should I do if `3010` or `3011` is already occupied?

Stop the process that owns the official port, or run a separate diagnostic environment that does not count as formal release evidence. The official local debug profile uses `3010` for the Gateway/API and `3011` for the Debug Portal.

### Q: How do I prove the observability path works end to end?

Run `obs up`, then `obs validate --synthetic --trace`. The validation sends a fixture-driven OpenAI Responses request through the local gateway, verifies the response, checks local dump artifacts, verifies prompt roles and usage, and writes evidence under the chosen evidence directory.

### Q: How do I configure Claude Code or Codex to send traffic through CodeAgentLens?

Point the agent's base URL at `http://127.0.0.1:3010` or `http://127.0.0.1:3010/v1` according to the client. Use `obs configure-agents --backup` first as a dry run to preview agent configuration without modifying user config files.

## Security Model

The default `local_debug` profile is for loopback-only development. It can use full local capture so a developer can inspect prompts and model payloads on the same machine. Non-loopback profiles must fail closed unless auth, RBAC, capture policy, and tool exposure rules pass validation.

Important boundaries:

- `local_debug` is the only profile that permits disabled Portal auth, and only for loopback bind.
- `lan_team` and `public_server` use metadata-first capture defaults and require auth/RBAC simulation to pass.
- Secret headers such as `Authorization` and `X-API-Key` are redacted unless explicit local debug capture is enabled.
- Raw artifact routes are manifest-bound and reject path traversal.
- Agent Gateway traffic and Portal traffic use separate listeners.

## Maintainer

Maintainer: milome

Security reports: GitHub private vulnerability reporting: https://github.com/milome/code-agent-lens/security/advisories/new