Claude PR reviewer reloads the full coding-standards ruleset every run instead of caching it

[kacper-mikolajczak]

Problem

The Claude PR reviewer dispatches a code-inline-reviewer subagent whose definition tells it to Glob and Read every coding-standards rule file (~25K tokens across 28 files) on every run. Those reads land in the conversation (message) layer, beneath a system prompt that embeds per-checkout sections (cwd, git state, OS).

Prompt-cache hits require a byte-identical prefix, and the message layer sits below that ever-changing prefix, so the ruleset is never a stable cacheable prefix. As a result it is re-paid at the cache-write rate (1.25x) on every PR run rather than read from cache (0.1x), and ~29 extra tool round-trips are spent each run just loading the rules.

Solution

Move the ruleset into the reviewer's system prompt as a deterministic, byte-stable prefix:

• A small deterministic script concatenates the reviewer instructions and every rule file into one system-prompt file (stable order, fixed separators - so identical bytes across runs, which is what prompt-cache prefix hits require), supplied to the reviewer via --append-system-prompt-file.
• Add --exclude-dynamic-system-prompt-sections so the per-checkout system-prompt sections (cwd, git, OS) stop invalidating the prefix.
• Run the reviewer as a single agent so the rules sit in its own system prompt.

Net effect: the rules load once per run instead of via ~29 tool calls, and the now byte-stable prefix becomes eligible for cross-run cache reads (0.1x) instead of writes (1.25x) when runs fall within the cache TTL. Reviewer behavior is unchanged - the same rules are enforced.

Implemented in #93661.

[kacper-mikolajczak]

Claude PR Reviewer — Prompt-Cache Prefix

Status: validated on a test fork; change open as draft PR #93661.

1. Problem

The Claude reviewer loads the full coding-standards ruleset (28 files, ~26K tokens) into context on every PR review. The rules are pulled in at runtime by the code-inline-reviewer sub-agent and land in the conversation's message layer, below a per-checkout system prompt — so they are not a cacheable prefix and are reprocessed on every run.

2. Idea

Put the ruleset in a byte-stable, cacheable prompt prefix so prompt caching serves it at the read rate (0.1×) instead of reprocessing — when the prefix is byte-identical and a prior run is within the cache TTL.

3. Solution — two changes, both required

#	Change	Why it's needed
1	Concatenate instructions + rules into one file, inject via --append-system-prompt-file	Moves rules into the system layer (cacheable position), not the volatile message layer
2	--exclude-dynamic-system-prompt-sections	Strips per-checkout cwd/git/OS that otherwise change every run and bust the prefix

flowchart TB
  subgraph OLD["OLD — current reviewer"]
    direction TB
    O1["tools"]:::stable --> O2["system: preset<br/>+ dynamic sections cwd / git / OS<br/>(change every run)"]:::volatile
    O2 --> O3["messages:<br/>Glob + Read 28 rule files (~29 tool calls)<br/>then PR diff"]:::volatile
  end
  subgraph NEWREV["NEW — cache-prefix reviewer"]
    direction TB
    N1["tools"]:::stable --> N2["system: preset (dynamic EXCLUDED)<br/>+ 28 rule files inlined<br/>(byte-stable → CACHED)"]:::cached
    N2 --> N3["messages:<br/>PR diff only"]:::volatile
  end
  classDef cached fill:#d4f8d4,stroke:#2e7d32,color:#1b5e20;
  classDef volatile fill:#ffe0e0,stroke:#c62828,color:#b71c1c;
  classDef stable fill:#e8eaf6,stroke:#3949ab,color:#1a237e;

Loading

4. Implementation — draft PR #93661

File	Change
.github/scripts/pack-reviewer-prompt.sh	new — deterministic concat of CI instructions + all rule files
.github/reviewer/ci-instructions.md	new — CI reviewer instructions (rules come from the prompt)
.github/workflows/claude-review.yml	pack step + the two flags; single agent (no Task sub-agent)
.claude/agents/code-inline-reviewer.md	unchanged — interactive/local agent still works as before

3 files, +97 / −2. Nothing generated is committed. Needs bundled CLI ≥ 2.1.98 — App is on claude-code-action@v1.0.149 (✓). 1h cache TTL recommended.

5. Cache behavior

State	When	Prefix cost
Cold	first run / > TTL gap	written at 1.25×
Warm	prior review within TTL	read at 0.1×

TTL 5 min default, 1 h opt-in (ENABLE_PROMPT_CACHING_1H=1) — required since a run takes ~5-6 min. Cache key = prefix bytes + model + org, independent of the PR diff.

6. Test framework

Aspect	Choice
Fixture	PR with trivial diff → low output variance, cost reflects the prefix cache
Conditions	{OLD, NEW} × {cold, warm}
n	NEW-warm 6, OLD-warm 5, cold 1+ each
Measurement	total_cost_usd + direct cache_read/cache_creation tokens (show_full_output)
Controls	model claude-opus-4-8, action v1.0.149, ubuntu-latest, 1h TTL, single session; cold/warm verified per run from tokens

7. Results (code-step cost, USD)

Condition	n	Median	Range	cache_creation	cache_read
NEW — warm	6	$0.19	0.174–0.214	~0 (0.6–4.7K)	~262–266K
NEW — cold	1+	$0.81	0.81–0.97	~64–68K	—
OLD — warm	5	$0.28	0.280–0.290	~10.8K	~73K
OLD — cold	1+	$0.53	0.49–0.71	~30K	partial

Comparison	NEW	OLD	Δ
Warm (steady state)	$0.19	$0.28	NEW −33%
Cold	$0.81	$0.53	NEW +53%
Rule-loading tool round-trips/run	0	~29	NEW avoids
5 seeded violations detected	5/5	5/5	parity

8. Economics

OLD(h)=0.28h+0.53(1−h) · NEW(h)=0.19h+0.81(1−h)

xychart-beta
    title "Cost per review vs warm-hit-rate (break-even at h ~ 0.75)"
    x-axis "warm-hit-rate h" [0, 0.25, 0.5, 0.75, 1.0]
    y-axis "USD per review" 0 --> 0.9
    line "OLD" [0.53, 0.468, 0.405, 0.343, 0.28]
    line "NEW" [0.81, 0.655, 0.50, 0.345, 0.19]

Loading

h	OLD	NEW	cheaper
0.00	$0.53	$0.81	OLD
0.50	$0.41	$0.50	OLD
0.75	$0.34	$0.35	~tie
0.90	$0.31	$0.25	NEW
1.00	$0.28	$0.19	NEW

Break-even ≈ 75% warm.

Updated SUMMARY.md. The two changed sections now read:

9. Verdict for App repo

Adopt — with the 1h cache TTL enabled. The reviewer's measured run cadence puts it deep in the warm regime, well past the break-even.

Run history (claude-review.yml, 28,489 runs over 268 days, ~106/day):

Metric	Value
Median gap between runs	4.3 min
p90 / p95 gap	31 / 53 min
Warm-hit-rate h @ 1h TTL	95.8% (only 4.2% cold starts)
Warm-hit-rate h @ 5-min TTL	53.6%

At h ≈ 0.96: OLD ≈ $0.29/review, NEW ≈ $0.22/review → ~25% saving (~$0.07/run; ~$8/day ≈ ~$2.8K/yr on the code reviewer alone), plus lower latency (skips ~29 rule-loading tool round-trips/run).

Monitor via the action's execution_file output.

[kacper-mikolajczak]

Hi @Julesssss @rlinoz! 👋 One of the possible enhancements to the reviewer setup - this time coming with possible non-functional improvement on time and savings. Let me know if you have any thoughts 🙏

[Julesssss]

Oh I'd like to look at this more closely, very interesting to see how we could make our prompting more efficient. To be honest the savings aren't that exciting here though, and it comes with some overhead of needing to maintain and be detect regressions to the caching. The idea is interesting though.

[kacper-mikolajczak]

Thanks. I agree this is more of an end-level optimisation that comes with a cost.

However, it could be beneficial to monitor this approach as it will become more and more valid as the rule-set grows - both from duration (currently rules are loaded explicitly by prompting, not programmatically) and savings perspective.

Prompting efficiency is topic that I'd definitely want to look into more deeply in spare time. Let's get ourselves posted if we come up with anything interesting. :)

[kacper-mikolajczak]

With that said, here's a possible alternative - a lightweight monitoring proposal we could use to gauge over time how much caching actually helps vs adds cost, and catch it if the savings quietly erode (a rules edit, action/CLI bump, model change etc.).

1. In-run emit - the reviewer prints one counts-only line per run (CACHE_USAGE creation=… read=… cost=…). No message content; safe in public logs.
2. Scheduled collector - scans recent reviewer runs' logs for that line and tracks the warm-hit / cost trend over time, flagging if it drifts cold. Logs only, no artifacts.

Draft PR: callstack-internal#57

[Julesssss]

@kacper-mikolajczak to be honest I think we can just get you to re-investigate later and focus on device-agent. If you are interested in prompting efficiency we could instead look into Melvin costs. We have a cost analysis script/summary here that you may have some thoughts on?

[kacper-mikolajczak]

@Julesssss Makes total sense - parking this for now and focusing on agent-device first 👍

[Bakomebandias]

Picking this up! 👋

Fix plan: Move the coding-standards ruleset from the message layer into the system prompt as a static, byte-stable prefix so Anthropic's prompt cache can hit on every PR run.

This will eliminate the ~29 extra tool round-trips and reduce cost from 1.25x (cache-write) to 0.1x (cache-read) per run.

Will submit PR shortly. @Bakomebandias

[melvin-bot]

📣 @Bakomebandias! 📣
Hey, it seems we don’t have your contributor details yet! You'll only have to do this once, and this is how we'll hire you on Upwork.
Please follow these steps:

1. Make sure you've read and understood the contributing guidelines.
2. Get the email address used to login to your Expensify account. If you don't already have an Expensify account, create one here. If you have multiple accounts (e.g. one for testing), please use your main account email.
3. Get the link to your Upwork profile. It's necessary because we only pay via Upwork. You can access it by logging in, and then clicking on your name. It'll look like this. If you don't already have an account, sign up for one here.
4. Copy the format below and paste it in a comment on this issue. Replace the placeholder text with your actual details.
   
   Format:

Contributor details
Your Expensify account email: <REPLACE EMAIL HERE>
Upwork Profile Link: <REPLACE LINK HERE>

[Bakomebandias]

Withdrawing my attempt — I see PR #93661 already addresses this. Good work @kacper-mikolajczak! 🙏