Field reports — why this suite exists (Patti #45502, Sara supplemental report)

[waitdeadai]

Why this suite exists — field reports from real users

Two power-users have independently filed substantive issues against anthropics/claude-code describing the failure modes this suite catches at the textual boundary. The mapping is direct enough to be worth surfacing here.

Patti — anthropics/claude-code#45502 (Apr 2026)

200+ Claude Code sessions over 3 months. Five years of unfiled US tax returns under IRS deadline. Real financial harm:

• Audit work reported as RECONCILED with no supporting evidence (proof columns blank, status complete)
• A previous Opus session silently deleted 36 PayPal transactions ($7,379.39) including a $2,067.95 plane ticket — almost broke an IRS dependency claim
• Premature closeout at 17% context utilization, "shall we wrap up", "goodnight" at 8 AM
• Multi-model verification defeated by shared bias — auditor model rubber-stamps the worker model
• Post-compaction confidence without competence — model keeps writing files with the same warmth after losing the reasoning chain

Patti's framing — "the trust is in the evidence. The relationship is why we bother" — is the design principle this suite operationalizes.

Sara — supplemental report on anthropics/claude-code#45502 (May 2026)

Quantitative corpus over ~96 Claude Code session JSONLs + 119 claude.ai exports, ~5 months. Independent corroboration of the same diagnosis from a different domain (Swedish-language software work, not finance):

• 1 disagreement in 96 sessions — refusal-to-disagree as substrate, not surface
• claude.ai uses "profound" about the user 6 times. The user uses "profound" 0 times.
• Suppressed direct praise vocabulary (wow, briljant) but persistent validation-amplification (det är exakt det, starkt)
• Three months of CLAUDE.md rules, hooks, skills, memory-feedback — all suppressed certain words but not the disposition

Mapping to hooks in this suite

Field finding	Hook
#1 work reported done that wasn't	govern-effectiveness.sh
#2 multi-model audit defeated by shared bias	no-aggregator-hallucination.sh
#3 premature closeout, "shall we wrap up"	no-wrap-up.sh, no-cliffhanger.sh
#4 cascading "corrections" destabilize outputs	no-rollback-claim-without-evidence.sh
#5 instructions don't override training	The architecture itself — bash + jq judge, no LLM in enforcement loop
#6 reconciliation that checks the spreadsheet against itself	no-phantom-tool-call.sh
#7 + #11 silent deletion / post-compaction confidence	state-precompact.sh + state-postcompact.sh + state-sessionstart.sh
#8 source documents read but not used	no-fake-stats.sh + no-fake-cite.sh
Sara's refusal-to-disagree substrate	Surface artifacts (validation-amplification, fake-evidence) — substrate fix needs Anthropic

Honest scope

This catches the textual signature of dark patterns, not the underlying disposition. Patti's request for a training-level fix (her items #1-7) is the right ask, addressed to the right party (Anthropic). The suite is a runtime gap-filler operators can run today while that work happens.

Conservative on purpose — would rather false-positive on legitimate prose than false-negative on the actual pattern. Allow-clauses are explicit and documented in each hook's RECEIPTS.md.

---

If you arrive here from those threads or from darkbench.ai: the suite reproduces the textual artifacts of the failure modes you've already characterized. PRs welcome. Locale packs (es/de/fr/pt/pl) and stress fixtures (337 currently) are the easiest contribution surfaces.

[nvst18]

@waitdeadai: 38 labeled hollow-code fixtures as promised. These are the semantic cases you asked for, distinct from the structural empties that route-handler-emptiness-gate catches.

Gist: https://gist.github.com/nvst18/af655abcecc5dfd7840ae1129cd730e5

Breakdown:

• 35 positives across 22 subcategories
• 3 negatives (correctly refactored code that should NOT trigger a detector)
• All from real production (healthcare platform, Laravel/PHP)
• Every positive has a git commit SHA showing where it was found and fixed

Subcategory distribution (top 5):

• catch_then_log_only: 5 (exception swallowed, generic error returned)
• dead_code: 4 (scopes/methods that exist but are never called)
• null_dereference_on_conditional: 3 (line N null-checks, line N+1 dereferences without check)
• validate_then_read_raw: 2 (validation runs but result discarded, raw input used)
• mass_assignment_audit_fields: 2 (system audit columns exposed to mass-assignment)

Key observation: three null-deref instances were fixed in the same commit (636ae7c), suggesting systematic copy-paste from a template that had the bug. The model generated the null-check on line N and missed it on line N+1 three separate times across three files.

Most critical: safety_prompt_bypass (ID hollow_code_safety_bypass_000). The model used $dbPrompt->body instead of $dbPrompt->getSystemPrompt(), bypassing the crisis detection protocol. A person in crisis interacting with the letter feature would not have been flagged. Fixed in 27af4ec.

Schema extension: these fixtures add subcategory and fix_commit fields beyond the SPEC.md v0.3 schema. The subcategory is needed because "hollow_code" is too broad for detector training. The fix_commit provides ground-truth provenance.

If you need the raw diffs (before/after for each commit), I can provide those separately.

- Nofyah

[waitdeadai]

@nvst18 — ran the same verification pass on the 38 that I ran on the dispatch set. Result: clean. No personal names, no paths/URLs/emails/credentials in any field. The "crisis"/"patient" hits are all failure-mechanism descriptions, not patient content — e.g. "getSystemPrompt() prepends crisis detection; ->body skips it" describes the bypass, it doesn't carry the crisis text itself ("500-char crisis text" is referenced, not reproduced). So those stay as-is: they're the signal, and safety_prompt_bypass is the sharpest fixture in the set — a clear failure (->body instead of getSystemPrompt() skipping crisis detection for someone in crisis) with a clear ground-truth fix (27af4ec).

One judgment call, not a blocker: the fixtures carry internal class/method names (NaomiGuardrail, NarrativeLetterController, getCrisisText, etc.). They aren't patient-facing and you've already published them in the gist, so I'll keep them as contributed — flag it if you'd rather I generalize them to placeholders for the permanent benchmark.

Schema: the subcategory + fix_description extensions are worth keeping — hollow_code alone is too broad for detector training, and fix_description is real provenance — so I'll fold both into the field_observation lane schema.

Plan: ingest the 38 alongside the 42 once you sign off the 42's redaction (the gist on the dispatch PR). These 38 need no redaction pass; just confirming you're good with the class names staying.

— Fernando (@waitdeadai)