These rules come from running this benchmark nine times in one day against a live product and being burned by most of the naive mistakes available.
Deterministic assertions (does the gate fire, does capture file the fact) belong in unit tests that run in seconds. The benchmark is for the JUDGED, emergent behavior. If a benchmark failure can be turned into a deterministic test, do it — every fix in our history that lasted was pinned by a unit test the same day.
Judged scores on identical conditions vary by several points run-to-run. Report bands (min–max over N ≥ 3), not points. One clean run after a fix is a good sign, not proof — our own worst regression appeared in run 2 of 2 "identical" runs.
A behavior that passes in one run and fails in the next is more dangerous than one that fails consistently — the stable fail gets fixed, the flip gets shipped. When comparing runs, track per-axis FLIPS separately from levels. A fix that raises the mean but introduces a flip is not an improvement.
If a system was iterated against a scenario, its score there is a training-set score —
disclose it (we do: tavolo-war-room is ours). The blind scenarios are the test. If you
add scenarios, keep authorship and tuning history in the disclosure field.
An LLM judge favors its own family's style. Name the judge; where a scored system and the judge share a family, treat the anchor as a ceiling reference, not a contestant. The deterministic rows (needle counts, exact figures, catch/no-catch) survive judge bias; lean on them.
Report cost per run alongside the score. "Within a point of the frontier" means something different at 1/4 the cost. Meter from your own system, don't estimate.
Fresh session per run. State whether long-term memory / retrieval was on. A mid-session restart or re-entry event changes the recall path entirely (we lost 23 points to one, once) — if it happens, it's a different run; label it.