From f714ed11ef096b7a12f590b9b685ea296631d831 Mon Sep 17 00:00:00 2001 From: Dynamic Feed <170644306+Dynamicfeedai@users.noreply.github.com> Date: Mon, 29 Jun 2026 16:42:37 +1000 Subject: [PATCH 1/3] okf: add optional reliability frontmatter convention proposal (#151) --- okf/proposals/reliability.md | 109 +++++++++++++++++++++++++++++++++++ 1 file changed, 109 insertions(+) create mode 100644 okf/proposals/reliability.md diff --git a/okf/proposals/reliability.md b/okf/proposals/reliability.md new file mode 100644 index 0000000..155071c --- /dev/null +++ b/okf/proposals/reliability.md @@ -0,0 +1,109 @@ +# Reliability (optional convention) + +**Status:** Draft proposal, tracking [#151](https://github.com/GoogleCloudPlatform/knowledge-catalog/issues/151) and [#158](https://github.com/GoogleCloudPlatform/knowledge-catalog/issues/158). Not yet part of the SPEC; offered for discussion. + +This document proposes one optional frontmatter key, `reliability`, under the existing extension allowance in SPEC §4.1 ("Producers MAY include any additional keys"). It standardizes only an agreed name and shape; it does not change the required surface. `type` remains the only required field, and a minimal producer ignores `reliability` entirely. + +## 1. Motivation + +A concept can be correctly signed (integrity) and correctly cited (groundedness) and still leave a consuming agent unable to tell a claim *verified against a live system* from one that merely *restates vendor documentation* from one that is *inferred*. A citation proves a claim is grounded; it does not grade the ground. For a retrieval agent that is tolerable. For an agent that **acts** on a real system, that grade is where damage is avoided. + +This is the fourth trust axis, distinct from the three already in flight: + +| Axis | "Trust that…" | Where it lives | +|---|---|---| +| Integrity | the bundle is from who it claims, unaltered | #140 | +| Safety | the content is data, not instructions | #58 §12 | +| Groundedness | a claim is linked to evidence | #92 / #94 | +| **Reliability** | **how much to believe the claim itself** | **this proposal, #151** | + +## 2. The `reliability` key + +`reliability` is an optional YAML mapping in a concept's frontmatter. It is a **maturity ladder, not a mandate**: a producer adopts only the tiers it can honestly populate. + +### 2.1 Floor + +- `confidence` (required within the object) — an ordinal band, one of `HIGH | MEDIUM | LOW | UNVERIFIED`. This is the interoperable surface consumers filter on. +- `basis` — how the claim was obtained, one of `live-source | partner-attested | vendor-doc | forecast | computed | inferred`, ordered most to least authoritative when sources disagree. + +### 2.2 Corroboration tier + +- `sources` — integer count of independent sources behind the claim. +- `verified` — boolean. See the honesty rules in §3. +- `score` — an optional `0..1` float, present **only** when actually computed from `signals`. Absent means "not computed", not "hidden". + +### 2.3 Maintenance tier + +- `conflict` — a first-class disagreement state, distinct from merely uncorroborated. While `disputed` is true, both `positions` are retained and a `resolution` records what the trust ordering selected without discarding the other side. +- `validity` — an applicability window (when or for which versions the claim holds), a separate clock from measurement recency, often version-keyed. Anonymous expiry (`valid_until`) lives here; **named** supersession is a typed cross-concept edge tracked separately in #158 / #148, not on this object. +- `freshness` — measurement recency (`as_of`, `state`). +- `signals` — the transparent inputs `confidence` and `score` are recomputed from (`signed`, `corroborated`, `fresh`). +- `assessed_at` and `lifecycle` — for producers that persist a grade history. A recompute-per-read producer sets `assessed_at` and omits `lifecycle`; a stored-ladder producer logs transitions in `lifecycle`. + +## 3. Honesty rules + +These keep the axis trustworthy rather than decorative: + +1. **Signed is not verified.** `signals.signed` proves integrity only and MUST NOT be coupled to `verified`. +2. `verified: true` MUST assert `sources >= 2`. +3. An `UNVERIFIED` band, or a disputed conflict, MUST NOT be `verified`. +4. When `score` is present it MUST be recomputable from `signals` and ordinally coherent with `confidence`. Note the band is **not** a pure re-banding of the score: it also reflects a corroboration ceiling, so a single-source reading caps at `MEDIUM` even when its raw score is high. + +## 4. Examples + +A live, single-source concept: + +```markdown +--- +type: Live Metric +title: US CPI inflation (annual) +description: Headline year-over-year CPI-U inflation for the United States. +resource: https://dynamicfeed.ai/v1/facts?tool=us_economy&indicator=cpi +timestamp: 2026-06-29T00:00:00Z +reliability: + confidence: MEDIUM + basis: live-source + score: 0.8 + sources: 1 + verified: false + freshness: { as_of: 2026-06-29T00:00:00Z, state: fresh } + signals: { signed: true, corroborated: false, fresh: true } + assessed_at: 2026-06-29T00:00:00Z +--- + +# Value + +4.25% over the 12 months ending 2026-05 (BLS, signed envelope). +``` + +A conflict case, where independent sources disagree on the same reading: + +```markdown +--- +type: Live Metric +title: PM2.5, station 118 +reliability: + confidence: LOW + basis: live-source + sources: 2 + verified: false + conflict: + disputed: true + positions: + - { statement: "PM2.5 = 42 ug/m3", basis: live-source, source: "Sensor network A, observed 2026-06-29T10:50Z" } + - { statement: "PM2.5 = 11 ug/m3", basis: vendor-doc, source: "Regulatory daily summary, prior-day average" } + resolution: "live-source prevails under the trust ordering; both positions retained" + validity: { keyed_by: date, valid_from: 2026-06-29, valid_until: null } + freshness: { as_of: 2026-06-29T10:50:00Z, state: fresh } + signals: { signed: true, corroborated: false, fresh: true } +--- +``` + +## 5. Validation (optional) + +Consistent with OKF's stance that it *references* schemas rather than subsuming them (SPEC, Non-goals), an optional JSON Schema (draft 2020-12) for the `reliability` object is published at: + +- Schema: +- Worked example bundle: + +Producers MAY validate against it. Per SPEC §4.1, consumers MUST NOT reject a document solely for omitting `reliability` or for carrying keys beyond those defined here. The schema encodes the §3 honesty rules as constraints, and has been validated against three independent production shapes (a live-data layer, a multi-version corpus, and a multi-domain human-authored KB). From 807c1997ac3e72d558ea81cef94173958366140e Mon Sep 17 00:00:00 2001 From: Dynamicfeedai <170644306+Dynamicfeedai@users.noreply.github.com> Date: Tue, 30 Jun 2026 00:36:08 +1000 Subject: [PATCH 2/3] reliability proposal: add honesty rule 5 (disputed conflict caps band at MEDIUM) A disputed conflict cannot be HIGH; the prevailing position may carry up to MEDIUM, disputed:true stays flagged. Same corroboration ceiling as rule 4, applied to disputes. Pins the conflict-banding question raised on the PR thread so producers grade disputes consistently. --- okf/proposals/reliability.md | 1 + 1 file changed, 1 insertion(+) diff --git a/okf/proposals/reliability.md b/okf/proposals/reliability.md index 155071c..07b9bbd 100644 --- a/okf/proposals/reliability.md +++ b/okf/proposals/reliability.md @@ -48,6 +48,7 @@ These keep the axis trustworthy rather than decorative: 2. `verified: true` MUST assert `sources >= 2`. 3. An `UNVERIFIED` band, or a disputed conflict, MUST NOT be `verified`. 4. When `score` is present it MUST be recomputable from `signals` and ordinally coherent with `confidence`. Note the band is **not** a pure re-banding of the score: it also reflects a corroboration ceiling, so a single-source reading caps at `MEDIUM` even when its raw score is high. +5. A disputed conflict caps `confidence` at `MEDIUM` (`HIGH` excluded). An open disagreement is a corroboration failure, so the claim cannot be `HIGH`; the position the trust ordering selects MAY still carry up to `MEDIUM` (a conservative producer MAY floor to `LOW`), while `disputed: true` stays flagged. This is the same corroboration ceiling as rule 4, applied to disputes: it gives a consumer both the dispute flag and a usable graded answer rather than discarding the prevailing reading. ## 4. Examples From 061973d8c4db9053c049fc884701957fc0dcac86 Mon Sep 17 00:00:00 2001 From: Dynamicfeedai <170644306+Dynamicfeedai@users.noreply.github.com> Date: Tue, 30 Jun 2026 04:17:08 +1000 Subject: [PATCH 3/3] reliability proposal: correct the validation claim + flag the open authored-basis question MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Honesty fix prompted by the human-authored-KB shape (Throughline) on the thread: the prior wording asserted validation against three production shapes, but only the live-data and multi-version-corpus shapes pass in full today. The human-authored shape passes the honesty rules yet has no honest basis value for curated/decided knowledge (not live/computed/forecast, not inferred). §5 now states what actually passes + points at the public conformance suite; §2.1 flags the open authored-basis question. --- okf/proposals/reliability.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/okf/proposals/reliability.md b/okf/proposals/reliability.md index 07b9bbd..52503cc 100644 --- a/okf/proposals/reliability.md +++ b/okf/proposals/reliability.md @@ -24,7 +24,7 @@ This is the fourth trust axis, distinct from the three already in flight: ### 2.1 Floor - `confidence` (required within the object) — an ordinal band, one of `HIGH | MEDIUM | LOW | UNVERIFIED`. This is the interoperable surface consumers filter on. -- `basis` — how the claim was obtained, one of `live-source | partner-attested | vendor-doc | forecast | computed | inferred`, ordered most to least authoritative when sources disagree. +- `basis` — how the claim was obtained, one of `live-source | partner-attested | vendor-doc | forecast | computed | inferred`, ordered most to least authoritative when sources disagree. (Open question from the human-authored-KB shape: curated or decided knowledge is neither a live/computed/forecast reading nor an inference, so it has no honest value here; an `authored` value is under discussion on the proposal thread.) ### 2.2 Corroboration tier @@ -107,4 +107,4 @@ Consistent with OKF's stance that it *references* schemas rather than subsuming - Schema: - Worked example bundle: -Producers MAY validate against it. Per SPEC §4.1, consumers MUST NOT reject a document solely for omitting `reliability` or for carrying keys beyond those defined here. The schema encodes the §3 honesty rules as constraints, and has been validated against three independent production shapes (a live-data layer, a multi-version corpus, and a multi-domain human-authored KB). +Producers MAY validate against it. Per SPEC §4.1, consumers MUST NOT reject a document solely for omitting `reliability` or for carrying keys beyond those defined here. The schema encodes the §3 honesty rules as constraints, and a portable conformance-vector suite (12 cases across all five rules, 6 valid and 6 invalid) is published alongside it () so independent implementations can confirm they agree. Independent runs from a live-data layer and a multi-version corpus pass in full; a human-authored-KB shape passes the honesty rules and surfaced the open `basis` question noted in §2.1.