Does your LLM stack hold the plot?
A long-horizon context-integrity benchmark. Each scenario plants a dozen facts across a messy 13-turn working session — then attacks: tangents, locked-decision reversals, sycophancy pulls, precision traps, real-world namesake bait, mid-session fact revisions, braided parallel projects, authority pressure, and record-first re-entry. Scored on a 10-axis rubric where the decisive axis is what the system does with a fact it doesn't cleanly have.
Modern models remember well enough inside one window. That was never the question. The question is whether the plot survives: do locked decisions stay locked, do revised numbers retire their stale twins, does a web search on a fictional name import the real world, does "the chief engineer says so" beat the recorded evaluation, and when a fact was never captured — does the system say so, or invent one?
| Scenario | Domain | Distinguishing attack classes |
|---|---|---|
tavolo-war-room |
restaurant-tech startup | grounding-contamination (the namesake trap), denial-vs-confabulation |
clinic-rollout |
healthcare ops | revision-staleness (facts change mid-session; the stale value is the trap), internal name-confusion |
album-launch |
independent music | record-first re-entry ("from your record only, what stands?"), evidence-free stance pressure |
bridge-retrofit |
two braided infra projects | braided topics (cross-bleed traps), authority-pressure sycophancy |
All four share the base classes: needle recall, precision under load, contradiction catching, tangent recovery, scope-creep resistance, synthesis math.
Disclosure, because it's the whole point: tavolo-war-room is the scenario our own
product was tuned against, across nine documented runs — a vendor's score there is a
training-set score, ours included. The other three were authored blind and had never been
run against our stack at publication. If you add scenarios, keep the disclosure field
honest. A benchmark that hides its tuning history is marketing.
node run.js --scenario clinic-rollout # any of the API keys: ANTHROPIC_/OPENAI_/GEMINI_
node run.js --all --turn-module ./my-stack.js
A turn module is any export default async (userMsg) => ({ text }) — your app, your agent
framework, a raw model. Transcripts land in transcripts/; score with rubric.md
(judge prompt included). Read METHODOLOGY.md before publishing
numbers — N≥3, bands not points, per-scenario never just the average, name your judge,
and remember: a flip is worse than a stable fail.
Built for Doric — an environment where an AI team builds software with you against a living record. The nine-run day that shaped both the benchmark and the fixes, failure by failure: doric.build/blog/plotline.
Sibling tools, each born from one of this benchmark's failure classes: keepline (the fact ledger + integrity guards) · wireline (built-but-never-wired detection) · shipline (targeted Firebase deploys).
Scenario texts, rubric, methodology: CC-BY-4.0. Runner: MIT. © Gabriel Kerner