spec: ReasoningAssay + AssayStandard — epistemic verdicts as projections#198
Merged
Conversation
…ections Claim trustworthiness was being reduced to a flat ok/sad/bad scalar, but confidence, authenticity, and verification-vs-generation are orthogonal axes. ReasoningAssay stores five axes (method, binding, verifier, agreement, authority) and makes ok/sad/bad a render-time projection that re-derives when a verifier's reliability improves. AssayStandard records that reliability (confusion matrix, sample size, calibrated flag) as versioned canon, so a verdict must point at measured — not assumed — verifier reliability to project above 'sad'. validate-reasoning-examples enforces schema conformance AND projection soundness (recomputes assay() from the axes, fails on drift), and brings the previously-unvalidated reasoning family under make validate.
mdheller
added a commit
that referenced
this pull request
Jul 5, 2026
Add optional ReasoningReceipt.assay — the reserved receipt landing spot for the epistemic Assay (PR #198 follow-on). Run-level, render-time summary that references its run's ReasoningAssay records by URN (assayRefs, so verdicts stay independently re-projectable) and surfaces overallState, weakest-link binding, verifier calibrationRef, per-state counts, and projectedAt. A cache, never authoritative over the assays. EventEnvelope needs no structural change: each assay's authority axis already mirrors EventEnvelope.actor/integrity by reference. Backward-compatible (optional field). validate-reasoning-examples green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds two canonical schemas to the reasoning-run contract family that replace flat, stored confidence scalars on claims with a typed, re-projectable verdict.
ReasoningAssay— a verdict on a claim over five orthogonal axes:method(computed/retrieved/generated),binding(inline/post-hoc),verifier(points at a measured standard),agreement(decorrelation-weighted),authority(actor/channel authenticity). Theok/sad/badvalue (projectedState) is a render-time projection of the axes, cached and re-derivable — not the source of truth.AssayStandard— a verifier's measured, versioned reliability (confusion matrix, sample size, optional metrics + inter-rater κ,calibratedflag). Every assay'sverifier.calibrationRefmust point at one; a verdict without it cannot project abovesad. Answers "who verifies the verifier" mechanically.Why
ok/sad/badis right for operational health (mutually exclusive outcomes of one operation). It's the wrong shape for a claim, where confidence, authenticity, and verification-vs-generation are independent facts. Collapsing them into one pixel also makes history un-re-judgeable: when a verifier improves, past verdicts are stuck. Storing the axes and projecting at render time means a verdict re-projects against a newerAssayStandardwithout mutating its record — the same discipline as keeping every benchmark arm, not just the current winner. Full rationale indocs/adr/ADR-epistemic-assay-verdict-v0-1.md.Projection is executable and CI-enforced
tools/validate_reasoning_examples.pydoes two things: schema conformance for every example, and projection soundness — it recomputesassay()from each example's stored axes and fails the build if the recordedprojectedStatedoesn't follow. Verified locally that flipping an axis makes the gate bite. The newvalidate-reasoning-examplestarget also retroactively brings the existing (previously unvalidated) reasoning family undermake validate.Contents
schemas/ReasoningAssay.json,schemas/AssayStandard.jsonexamples/—assay_standard.json+ three-state proof set:reasoning_assay.json(ok),reasoning_assay.unassayed.json(sad),reasoning_assay.refuted.json(bad)tools/validate_reasoning_examples.py+ Makefile wiringmake validate→OK: validate(green).Review notes
assay()projection here is v0.1 and intentionally conservative (badis driven by authority-broken). Follow-on: populate realAssayStandardrecords from verifier calibration runs so production verdicts reference measured reliability.