Proposal: Knowledge Maturity Model — combining confidence, evidence, lifecycle, and verification gates

[Sliky1]

Proposal: Knowledge Maturity Model — combining confidence, evidence, lifecycle, and verification gates

This proposal builds on several active discussions:

• Proposal: Typed relationships between concepts #148 (Typed relationships) — defines rel, but doesn't address how an edge transitions from draft to published to deprecated.
• Proposal: graded confidence — an epistemic-reliability axis, distinct from integrity (#140) and citation (#92/#94) #151 (Graded confidence) — a static epistemic axis, but confidence alone doesn't capture lifecycle.
• Provenance / verification: an additive layer for OKF bundles #140 (Provenance / verification) — addresses bundle-level integrity via SHA-256. Edge-level is not covered.
• Budget-aware assembly for OKF: optional edge priority levels #139 (Budget-aware assembly) — overlaps with why lifecycle matters for token-constrained agents.

Each covers one dimension. What's missing is a unified model that connects them: a claim's reliability depends on where it sits in a maturity lifecycle, not just a static score.

---

The gap

An OKF bundle today can express See [customers](/tables/customers.md). #148 adds rel:

links:
  - target: /tables/customers.md
    rel: depends_on

But an agent still can't ask:

• Is this depends_on verified against the live system, or inferred from documentation?
• How many independent sources support it?
• Should I trust this claim enough to act on it?

We need a maturity model, not just metadata fields.

---

Proposal: Knowledge Maturity Model

                    EVIDENCE (citations + provenance)
                         │
                   CONFIDENCE (epistemic score)
                         │
                    ┌────┴────┐
   proposed ────────► active ──► expired
        │
        ▼
   low_quality / noise

Axis 1: Lifecycle (proposed → active → expired)

Status	Meaning	Example
proposed	Extracted, not yet verified	LLM-extracted candidate for review
active	Passed verification gate	Verified claim in production
expired	Superseded or invalidated	Outdated, kept for audit trail

The lifecycle is not monotonic. active can roll back to proposed if regression tests fail. proposed is periodically GC'd if it never gains enough evidence.

Axis 2: Confidence (epistemic reliability)

Floating-point [0.0, 1.0] — finer than categorical, needed by downstream agents for ranking:

Range	Meaning
≥ 0.95	keyword_list match / manually confirmed
≥ 0.75	LLM extraction with multi-document consensus
≥ 0.50	Single-document LLM extraction
< 0.50	regex pattern match (high noise, treat as suggestion)

Axis 3: Evidence (per-edge provenance)

Not just IMPORTED FROM X, but which document, which sentence, which quote:

evidence:
  - document: /tech/some-doc.md
    quote: "system X uses technology Y for data processing"
    confidence: 0.95
  - document: /business/client-interview.md
    quote: "confirmed dependency in architecture review"
    confidence: 0.85

---

Production reference implementation

Running in production on an enterprise knowledge base:

Metric	Count
Knowledge concepts (OKF concepts)	320
Vector chunks (768d)	1,354
Knowledge graph nodes	~207
Knowledge graph edges	330+ (proposed + active + expired)
Edge predicates	8 types (uses, adopts, regulates, competes, implements, depends_on, part_of, validates)
Pipeline stages	5 (extract → denoise → backfill → regress → promote)
Regression gate	Node recall ≥ 0.95, Edge recall ≥ 0.75

Core schema (SQLite)

CREATE TABLE knowledge_nodes (
    id TEXT PRIMARY KEY,
    name TEXT NOT NULL,
    node_type TEXT NOT NULL,    -- Client | Technology | Concept | Project
    description TEXT,
    properties TEXT,
    created_at TEXT,
    updated_at TEXT
);

CREATE TABLE knowledge_edges (
    id TEXT PRIMARY KEY,
    source_id TEXT NOT NULL REFERENCES knowledge_nodes(id),
    target_id TEXT NOT NULL REFERENCES knowledge_nodes(id),
    predicate TEXT NOT NULL,
    confidence REAL DEFAULT 0.0,
    edge_status TEXT DEFAULT 'proposed',  -- proposed | active | expired
    extraction_source TEXT,
    description TEXT,
    evidence_summary TEXT,
    properties TEXT,
    promotion_reason TEXT,
    created_at TEXT,
    updated_at TEXT,
    expired_at TEXT,
    valid_to TEXT,
    CHECK(edge_status IN ('proposed', 'active', 'expired'))
);

CREATE TABLE edge_evidence_links (
    id TEXT PRIMARY KEY,
    target_id TEXT NOT NULL REFERENCES knowledge_edges(id),
    document_id TEXT,
    quote TEXT,
    confidence REAL DEFAULT 0.0,
    created_at TEXT
);

Pipeline flow

watch_brain (file watcher)
    │
    ▼
Stage 1 — extract:  LLM extracts entities + relationships → stored as proposed
    │
    ▼
Stage 2 — denoise:  Rust/Python noise filter → batch-expire low-quality edges
    │                  (weak subjects, noise patterns, cross-domain isolation)
    ▼
Stage 3 — backfill:  Re-link evidence quotes to proposed edges
    │
    ▼
Stage 4 — regress:   Gold Set regression gate
    │                  node_recall ≥ 0.95 AND edge_recall ≥ 0.75?
    │                  NO  → block pipeline, flag for human review
    │                  YES → proceed
    ▼
Stage 5 — promote:   proposed → active + write wiki Markdown
    │                  Only edges with confidence ≥ 0.75, importance ≥ 4,
    │                  evidence_count ≥ 2
    ▼
Agent uses active edges; proposed edges are periodically garbage-collected

Proposed frontmatter extension

---
type: Relationship
title: "System X uses Technology Y"
predicate: uses
confidence: 0.92
lifecycle: active                             # proposed | active | expired
extraction_source: keyword_list               # how it was produced
evidence:
  - document: "/tech/some-doc.md"
    quote: "system X uses technology Y"
    confidence: 0.95
verified_by: gold_set_regression              # gate type
verified_at: "2026-06-29T10:00:00Z"
---

---

Backward compatibility

• Bundles without lifecycle, predicate, or evidence remain valid OKF.
• Consumers that don't understand these fields ignore them.
• confidence and lifecycle are semantically separable: a proposed edge can have high confidence, and an active edge can have low confidence (stale verified claim).

---

Questions for maintainers

1. Would OKF v0.2 / v1.0 consider adding an optional lifecycle field to concept frontmatter, analogous to edge_status in graph stores?
2. Should evidence be a spec-level concept (like # Citations), or remain a producer extension?
3. Is there interest in a reference implementation of the pipeline as a companion tool to the Enrichment Agent?

okf-knowledge-maturity-model-issue.md

[inkxel]

Hi @Sliky1 — this is great, and it overlaps/validates the #151/#159 reliability work that's converging. Your edge-lifecycle angle, which lines up with the supersedes/contradicts work in #148, feels genuinely additive.

[smgam29]

Lifecycle + evidence + confidence as a unified model is the right framing; treating them as three separate axes is exactly the trap I'd been falling into.

One important thing on the confidence axis: as written, the bands conflate two things:

• How the claim was produced (keyword vs LLM vs regex)
• How reliable the underlying claim is (who said it, when, does it agree with what's already in the graph).

Those correlate but aren't the same. A regex-extracted claim from a vendor's own KB can be more trustworthy than an LLM-extracted claim from an anonymous forum post.

Picking up Q3: I've been developing a producer-side framework that decomposes confidence into six weighted factors:

• Authority
• Specificity
• Recency
• Alignment
• Completeness
• Verifiability

NOTE: Recency on its own is not a good measure - it must be used with other factors, and in some knowledge spheres empirical validation is more important than recency.

Recency includes an age penalty relative to existing edges, which connects directly to your lifecycle transitions; when does active decay back to proposed? Alignment maps to the regression gate concept: does the new claim reinforce, complement, or contradict what's already there?

It's a producer methodology rather than a spec change, so wouldn't touch v0.2 frontmatter. The output is just the float that ends up in confidence:. Happy to share the full thing if it'd be useful as a companion reference implementation alongside the pipeline you describe.

Also: Interested to understand how you landed on the gold-set regression thresholds (0.95 node recall, 0.75 edge recall). In my experience those numbers are heavily corpus-dependent, and I'd be curious what shifted them for yours.

[inkxel]

@smgam29 the produced-vs-reliable split you're drawing is the same seam #159 ended up cutting — worth linking the two so the confidence axis here doesn't drift from the reliability convention converging there.

In #159 "how it was produced" became basis (an ordered enum — live-source, authored, vendor-doc, inferred…) and "how reliable" became the honesty rules plus a corroboration ceiling layered on top. Your regex-from-a-vendor-KB beating LLM-from-a-forum-post is exactly the case that forced them apart: high-authority source, low-automation extraction, and one band can't carry both.

Your six factors read to me like a recipe for the score float in that object — signals → score, recomputable, with the ordinal band as the view on top. #159 fixed the shape (score + band + basis + the rules) without prescribing how a producer computes the float, so a weighted-factor framework like yours could land as a reference producer for it rather than a separate axis. If you post the full thing I'd like to see how Authority and Alignment line up against the basis ordering and the conflict/disputed handling — Alignment sounds close to what got pinned there.

(I can't speak to the 0.95 / 0.75 thresholds — those are @Sliky1's gold set, not mine.)

[Sliky1]

@smgam29 On the 0.95 / 0.75 thresholds:
You’re right that they’re heavily corpus-dependent — and that’s exactly why I’d frame the gate mechanism (Gold Set + regression thresholds) as the thing to standardize rather than any specific numbers.

In my case, these came from running multiple rounds of pipeline iteration against a small knowledge base with ~390 nodes and ~369 edges as the Gold Set. Tuned empirically, not derived from theory.

0.95 for nodes is intentionally strict — nodes are foundational, and a missed node cascades into missed edges and broken paths. Rule-based matching catches most nodes reliably, with LLM extraction filling in the remainder. At 0.95, regressions in node coverage are consistently flagged; this has caught real regression bugs with very few false blocks.

0.75 for edges is more relaxed by design. Edge recall depends on both endpoints being identified correctly first, so some loss is cascaded from node misses rather than caused by extraction quality. I started at 0.90, but the pipeline was blocked too often by cascaded node misses rather than actual edge-extraction regressions. Lowering to 0.75 cut false blocks substantially while still catching real regressions reliably.

If a different approach works better for your corpus, happy to try it together — the shape isn’t settled yet.

“On the six-factor framework — +1 to what @inkxel said. Would love to see the full thing.”