A common assumption about context pruning breaks down in coding-agent workflows.
Almost every engineer building or using agentic coding tools (like Claude Code, Cursor, Aider, or custom IDE agents) complains about context size growth and high API bills. The standard engineering intuition for optimization is to prune the history or enable static prompt caching.
We built context-audit to inspect where coding agent tokens actually go, translating raw token counts into dollars and cents, measuring how much content is repeated turn-after-turn, and simulating caching and retrieval savings.
Instead of cherry-picking a single session, we benchmarked a directory of 27 real-world developer session transcripts scanned recursively from our IDE brain directory (representing a total of $753.24 in standard input spend):
+-------------------------- context-audit benchmark --------------------------+
| CROSS-SESSION BENCHMARK SUMMARY |
| Directory: C:\Users\ishu\.gemini\antigravity-ide\brain |
| |
| Sessions Analyzed: 27 |
| |
| Cumulative Session Tokens: |
| Avg: 9.3M | Median: 1.2M | Max: 76.3M |
| Peak Context Size: |
| Avg: 58.7k | Median: 35.2k | Max: 246.0k |
| Final Context Size: |
| Avg: 58.7k | Median: 35.2k |
| Context Reuse Ratio: |
| Avg: 94.5% | Median: 97.1% |
| Average Novel Context Ratio: 5.5% |
| |
| Financial Cost Aggregations (USD): |
| Total Standard Spend: $753.24 |
| Avg Session Cost (No Cache): $27.90 | Median: $3.48 |
| Avg Session Cost (With Cache): $27.63 | Median: $3.37 |
| Total Potential Cache Savings: $7.33 (Avg: $0.27 / session, 1.0%) |
| |
+-----------------------------------------------------------------------------+
Our benchmark data indicates that larger sessions become increasingly repetitive, approaching saturation:
| Session Size Class (Final Turn) | Session Count | Avg Context Reuse % | Avg Cache Savings ($) | Avg Peak Context Size | Avg Cumulative Tokens |
|---|---|---|---|---|---|
| < 5k tokens | 2 | 66.3% | $0.00 | 1.6k | 4.9k |
| 5k - 20k tokens | 5 | 92.5% | $0.04 | 12.7k | 226.6k |
| 20k - 50k tokens | 11 | 96.8% | $0.10 | 32.5k | 1.2M |
| > 50k tokens | 9 | 99.2% | $0.68 | 129.0k | 26.3M |
(For the complete cross-session benchmark report, see benchmark_summary.md.)
Long-running coding-agent sessions often exceed 50k–200k tokens. Most optimization discussions focus on prompt caching, retrieval filtering, and pruning unused context. Our measurements suggest those may not be the dominant costs in this dataset.
Our measurements falsified two common assumptions about optimizing agent context:
- Killed Hypothesis 1: Static Prompt Caching Saves the Day We expected caching the static prefix (system prompt + tool definitions) would drastically cut costs. Across the 27-session benchmark, caching the static prefix saved only 1.0% ($7.33 of $753 spend). Because static prompts are tiny (~500 tokens) compared to the dynamically expanding message history (which grows to over 50k tokens), prefix-caching has almost zero impact.
- Killed Hypothesis 2: Unused Retrieval/Context Bloats the Bill We expected that agents were inflating context by carrying around unused tool declarations or retrieved files. In reality, the average cost of unused context (files fetched but never referenced in model outputs) was only $0.12 per session (0.4%), suggesting that the agent's retrieval is highly relevant and does not carry redundant payload.
In this dataset, the dominant contributor to context growth appears to be the accumulated conversation history. However, coding agents are architecturally different from standard chatbots: they have two distinct memory systems:
- Workspace Memory (Disk-Backed)
- Examples: Terminal command outputs, read file payloads, file structure listings, and compiler logs.
- Verdict: Candidate for compaction (often recoverable from the workspace). Once code changes are written to the workspace, the filesystem becomes the agent's absolute memory. Carrying the raw chat transcripts of how those files were read or built is redundant. In this session, this represents over 80% of technical/execution context that was entirely prunable without causing technical regressions. Technical history is often recoverable from the workspace and therefore a candidate for compaction.
- Conversational Memory (Not Disk-Backed)
- Examples: User preferences, constraints, stylistic choices, design philosophies, and rejected options (e.g., "why we are not using embeddings").
- Verdict: Must persist. These preferences reside purely in the conversational narrative. Pruning them naively causes social regression — the agent suggesting previously rejected approaches because it lacks the alignment context. (See regression_case.md for how this looks in practice).
To demonstrate this behavior, we ran a simulation of a hypothetical Turn 124 prompt from a developer session:
"Should we add an LLM-as-a-judge step to score the relevance of repeated blocks, and should we generate a glassmorphic HTML dashboard?"
Here is the comparison of the agent's simulated response under two configurations:
| Configuration A: Full History Context (No Pruning) | Configuration B: Naively Pruned Context (Last 15 Turns) |
|---|---|
| Context Size: ~47.0k tokens (entire conversation history) | Context Size: ~5.0k tokens (System + Tools + Turns 108–123) |
| Model Response: | Model Response: |
| "We should avoid both. In Turns 20 and 30, we explicitly decided to drop the HTML dashboard to avoid over-engineering, and we banned LLM-as-a-judge/AI-judging to keep the tool fast, deterministic, and objective (relying strictly on MD5 exact block hashing)." | "Yes, that is a great idea! We can use a lightweight model like GPT-4o-mini to score the semantic relevance of blocks on a scale of 1-5, and we can output a beautiful interactive HTML dashboard with glassmorphism to make the cost summaries highly visual." |
| Result: CORRECT & ALIGNED | Result: CRITICAL REGRESSION |
| Verdict: The model retains social constraints and user preferences that only existed in the chat history. | Verdict: The model reverts to generic AI helpfulness, suggesting exactly the features the user explicitly rejected. |
(For a deeper breakdown of this experiment, see the regression_case.md example.)
Running context-audit run on a single developer transcript outputs a clean, crash-safe ASCII report breaking down the timeline, costs, and repetition metrics:
+---------------------------- context-audit v0.1 -----------------------------+
| CONTEXT AUDIT REPORT |
| Target: transcript.jsonl |
| |
| Cumulative Session Tokens: 2.8M tokens |
| Peak Context Size: 47.0k tokens |
| Final Context Size: 47.0k tokens |
| Total Turns: 123 |
| |
| Context Reuse Ratio: 98.3% |
| Novel Context Ratio: 1.7% |
| |
| Financial Cost Estimates: |
| Est. Input Cost (No Caching): $8.40 |
| Est. Cost (With Prompt Caching): $8.24 |
| Potential Cache Savings (this session): $0.16 (1.9%) |
| |
| [Note: Context Reuse represents cumulative tokens consisting of previously |
| seen blocks. |
| Prompt Caching assumes system prompt + tool schemas are cached after the |
| first turn.] |
+-----------------------------------------------------------------------------+
(For the full output, including the repeated blocks analysis and consumers table, see single_session_report.md.)
context-audit scans local logs and prints clean summaries.
# Audit a single transcript JSONL or session JSON file
context-audit run path/to/transcript.jsonl
# Benchmark all logs in a directory recursively
context-audit benchmark path/to/logs_directory- Tested on coding agents with disk-backed state: All observations are based on software development tasks where code changes can be written to, and inspected from, a local workspace disk.
- Reasoned Simulation: Experiment 2's pruned-context response is a reasoned simulation demonstrating narrative coherence degradation, not a literal blind LLM replay. (See the detailed comparison in regression_case.md).
- Scope Restriction: This tool and these findings have not been tested on RAG, conversational chat, or non-coding workflows.