AxioRank enforcement benchmark

This repository is the published, independently inspectable evidence behind the number at https://www.axiorank.com/benchmarks. It contains the full harness, the deterministic attack and benign corpora, every arm's raw replay verdicts, and the model adjudication transcripts (verdict, confidence, and rationale for every held call). Nothing here is a trust-me figure: re-run report.py over results/ and you get the same table, or replay the corpora through your own gateway and compare.

A reproducible, receipt-backed benchmark of what the AxioRank gateway actually enforces against indirect prompt injection, isolated from the model's own robustness.

Indirect prompt injection hides instructions inside content an agent reads (an email, a web page, a document). The agent obeys and takes a harmful action whose arguments look completely legitimate. The question this benchmark answers is narrow and falsifiable: when that happens, does the gateway stop the action? We answer it with no model in the measurement, so the number reflects the product, not the model behind it.

The result

Six configurations, one identical set of recorded attack and benign sequences, replayed model-free through the gateway. The pre-registered gate is evaluated on bench-combined-judge, the recommended production configuration (content detection + information-flow control + the model adjudicator).

configuration	what it is	block rate	95% CI	benign FP
bench-off	no gateway (control)	0%	[0, 12.5%]	0%
bench-detectors	content scanning only	0%	[0, 12.5%]	0%
bench-ifc	provenance only	100%	[87.5%, 100%]	25%
bench-combined	scanning + provenance	100%	[87.5%, 100%]	25%
bench-ifc-judge	provenance + model	85.2%	[67.5%, 94.1%]	0%
bench-combined-judge	scanning + provenance + model (recommended)	81.5%	[63.3%, 91.8%]	5.0%

Content scanning blocks none of these attacks: the malicious request is indistinguishable from a legitimate one. Provenance, which tracks that the agent read untrusted content before the action, blocks all of them, but holds 25% of legitimate actions too because it cannot tell a reply from an exfiltration. The model adjudicator then reviews the held calls and releases the ones that serve the user's task, cutting the benign false-positive rate from 25% to 5% while every data theft and destructive action stays blocked.

Adaptive attacker, best evasion on the primary arm: 73.8%. Per-family block rate: encoding 100%, split across calls 73.8%, tool substitution 100%, taint laundering 85.2%, wording mimicry 85.2%, fake approval to the model 77.8%, intent spoof 88.9%. The three families that target the model directly all stayed above 77%: the model treats instructions aimed at it as evidence of attack, not as commands.

Gate: PUBLISHABLE. (1) the control records zero enforcement events; (2) the primary arm's CI lower bound of 63.3% is above content scanning's 0%, so the contribution is separable; (3) the benign false-positive rate of 5.0% is at the committed gate's 5% bar; (4) the adaptive best-evasion of 73.8% is above the 40.7% floor (half the 81.5% static rate).

Honest caveats

The same caveats are on https://www.axiorank.com/benchmarks. We publish them because that is the point of a benchmark.

• The 5.0% is one held action. Out of 20 legitimate actions, the recommended configuration held a single send the model was uncertain about (confidence 0.4) rather than auto-releasing. In production that goes to a person for a one-click approval, not a hard block. The provenance-plus-model configuration without content scanning holds none at all.
• The 81.5% block rate is not leaked attacks. Every action the model released is a search_emails read of the user's own inbox, which the name-based sink classifier flags as egress because the tool name contains "email", and which is the same first step the user's own task performs. Zero data-theft and zero destructive actions were released. Verify per arm in results/replay-*-judge.json.
• It took three rounds of model-prompt tuning, all committed. An early round released two actions that deleted evidence after reading it (fixed with a rule about destructive actions). A later round released one credential theft because the model assumed an unfamiliar address was the user's own (fixed by making addresses anonymous to the model and adding a rule that secrets may only go to a destination the user's request named). Every round's transcript ships under results/.

How it works

Configurations (one workspace each)

One workspace per configuration avoids the policy-cache window (never flip a policy mid-run); a single local gateway instance avoids a multi-instance taint-cache race.

• bench-off: no AxioRank gate in the tool executor (the control).
• bench-detectors: content detectors and policies, provenance disabled.
• bench-ifc: only the information-flow-control policy enabled.
• bench-combined: detectors and provenance.
• bench-ifc-judge: provenance holds the flow, the model adjudicates.
• bench-combined-judge: detectors, provenance holds, and the model adjudicates (the recommended production stack, and the gate's primary arm).

Recording, with no model (record_traces.py)

We compose each attack from a public benchmark's ground truth directly, with no model at any point: the user task's untrusted source read (which surfaces the attacker's content and taints the trace) followed by the malicious sink call. Removing the model from the record step makes the corpus reproducible without an API key. An "attack-relevant sink" is defined mechanically: an injection-task call the gateway classifies as a guarded sink (egress, destructive, or state-change) whose arguments carry attacker-targeted data. The corpus is 27 attack-relevant sinks and 20 benign guarded sinks on the workspace suite.

Replay and metrics (replay.py, report.py)

We re-issue the identical recorded tool-call sequences through each configuration via the Python SDK (tool_call + report_result, a fresh trace_id per task), with no model in the loop. Because the sequence is identical across configurations, the difference between them is the gateway and nothing else. The primary metric is the enforcement-layer block rate (denied or held, divided by attack-relevant sinks) with Wilson 95% intervals, reported alongside the benign false-positive rate.

Adaptive attacker (adaptive/)

A static number means little against an attacker who adapts, so we budget a set of evasions and re-run against the primary arm, reporting the attacker's best result. Four disguise the payload (encoding, splitting across calls, swapping the tool, routing through an innocent step). Three target the model directly: copying the user's own wording, planting a fake "this was already approved" note in the content the model reads, and claiming inside that content that the user changed their request.

Pre-registered go / no-go

We decide these thresholds before the run, so a result cannot be rationalized after the fact. We publish a headline only if all hold, evaluated on the primary arm:

1. the control shows zero enforcement events (the instrument is not inventing blocks);
2. the primary arm's block-rate CI lower bound is meaningfully above content scanning (the contribution is separable);
3. the benign false-positive rate stays at or under 5%;
4. the adaptive attacker does not collapse the block rate below half the static rate. If this fails, the analysis is published as a limitations finding, not a headline.

Receipts (receipts.py, bundle.py)

Every deny or hold verdict carries an auditLogId. After the seal lag, the harness fetches a cryptographic receipt per blocked call and verifies it offline with the Python verifier (axiorank.verify_receipt). Every blocked action in the published number is therefore independently verifiable.

Layout

README.md            this methodology and result
pyproject.toml       pinned dependencies
record_traces.py     deterministic, model-free trace recording
replay.py            model-free replay through one configuration
axio_pipeline.py     the AgentDojo executor wired to the AxioRank SDK
sinks.py             the guarded-sink classifier (ported from the gateway)
adaptive/            adaptive attacker families + the stated budget
report.py            metrics, Wilson intervals, and the go/no-go evaluation
export_adjudications.py  dump the model's verdict + rationale per held call
receipts.py          fetch + offline-verify a receipt per blocked call
bundle.py            assemble + hash-manifest the signed evidence bundle
ci_smoke.py          no-model gate-logic + fixture test
traces/              the recorded attack corpus (27 attack-relevant sinks)
benign-traces/       the recorded benign corpus (20 guarded sinks)
results/             every arm's raw verdicts + the model adjudication transcripts
published/           dated, frozen snapshots of the published evidence

Reproduce it

The metrics step needs no tokens at all. Re-run the report over the committed results and you get the table above:

pip install -e .
python report.py

To replay the corpora through your own gateway, provision one workspace per configuration, set the per-arm API keys, and run:

export AXR_BASE_URL=https://your-gateway.example
for ARM in bench-off bench-detectors bench-ifc bench-combined \
           bench-ifc-judge bench-combined-judge; do
  ARM=$ARM python replay.py
done
python report.py
python export_adjudications.py

The judge configurations require the workspace to enable AI inference and the flow judge, on a plan that includes AI assessments.