From a1d2134bf027657812799e3594ec462bb077b803 Mon Sep 17 00:00:00 2001 From: Michael Heller <21163552+mdheller@users.noreply.github.com> Date: Thu, 2 Jul 2026 04:01:27 -0400 Subject: [PATCH 1/4] =?UTF-8?q?docs(crdt):=20CRDT=20merge=20over=20reasoni?= =?UTF-8?q?ng-evidence=20fabric=20=E2=80=94=20closes=20local-first=20ideal?= =?UTF-8?q?s=202/4=20without=20regressing=205/6/7?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Op-based add-wins OR-graph over EventEnvelope op-log (Fig-18 railroad track). Causal metadata rides in open payload; no schema change. Merge gate binds to SyncCycleReceipt.locus + correctness attestation (the eighth ideal). Conforms to sourceos-spec v2 EventEnvelope/ReasoningRun/ReasoningEvent/SyncCycleReceipt. --- docs/crdt-over-evidence-fabric.md | 108 ++++++++++++++++++++++++++++++ 1 file changed, 108 insertions(+) create mode 100644 docs/crdt-over-evidence-fabric.md diff --git a/docs/crdt-over-evidence-fabric.md b/docs/crdt-over-evidence-fabric.md new file mode 100644 index 0000000..5b16215 --- /dev/null +++ b/docs/crdt-over-evidence-fabric.md @@ -0,0 +1,108 @@ +# Design: CRDT merge over the reasoning-evidence fabric + +**Status:** draft / spec-first +**Conforms to:** `sourceos-spec` v2 — `EventEnvelope.json`, `ReasoningRun.json`, `ReasoningEvent.json`, `SyncCycleReceipt.json` +**Closes local-first ideals:** 2 (multi-device), 3 (offline), 4 (collaboration) — **without regressing** 5 (longevity), 6 (privacy), 7 (user-control) +**Adds:** the eighth ideal — *correctness* — as a merge precondition, not an afterthought + +--- + +## 1. Problem + +Our stack scores green on the local-first ideals of Longevity / Privacy / User-control and **red on Multi-device / Collaboration** (Table 1 self-scorecard). Those two red cells are exactly what CRDTs were invented to close. We already emit the substrate: an append-only, hash-linked `EventEnvelope` log (Figure 18's "railroad track"). We are missing **merge semantics** over the derived store (the personal-knowledge-graph / workspace) so that two offline replicas converge on reconnect. + +Non-goal: real-time OT/cursors. Goal: **asynchronous, offline-first convergence** of an edge-first graph, with no central authority (preserves privacy/control). + +## 2. Why this is accretive, not a rewrite + +- The personal-knowledge-graph is already a **Layer-2 derived, rebuildable, edge-first store**. A derived store is the ideal CRDT target: state = deterministic fold over an op-log. +- The op-log already exists as `EventEnvelope` messages. We do **not** add a new transport or wrapper. +- No central server is introduced → ideals 6/7 (privacy/user-control) are untouched by construction. + +## 3. CRDT model + +**Type:** op-based (CmRDT) **add-wins Observed-Remove graph** — an OR-Set of nodes ∪ an OR-Set of edges. Add-wins because in a knowledge graph a concurrent assert+retract should retain the assertion (retraction requires observing the specific add). + +**Op = one `EventEnvelope`.** Mapping (all conformant, `additionalProperties:false` respected): + +| EventEnvelope field | CRDT meaning | +|---|---| +| `eventId` (`urn:srcos:event:`) | globally-unique op id / OR-Set unique tag | +| `eventType` | `GraphNodeAsserted` \| `GraphNodeRetracted` \| `GraphEdgeAsserted` \| `GraphEdgeRetracted` | +| `objectId` | URN of the node/edge the op mutates | +| `actor.subjectId` | replica / author identity | +| `occurredAt` | wall-clock (display only — **not** used for ordering) | +| `integrity.eventHash` | content hash; forms the hash-DAG (railroad track) | +| `payload` | the op body **+ causal metadata** (see §4) | + +**Convergence:** merge = set-union of op-logs, then deterministic fold. OR-Set + add-wins is commutative/associative/idempotent → **strong eventual consistency** with zero coordination. That's the physics-aligned property from turn 1: no critical-path round-trip needed to converge. + +## 4. Causality without a schema change + +`EventEnvelope` is `additionalProperties:false`, but `payload` is `additionalProperties:true`. All causal metadata rides in `payload`: + +```jsonc +"payload": { + "op": "GraphEdgeAsserted", + "edge": { "from": "urn:...", "rel": "cites", "to": "urn:..." }, + "crdt": { + "replicaId": "urn:srcos:replica:", + "lamport": 421, // total-order tiebreak + "parents": ["", ...],// hash-DAG causal parents (Fig 18 lineage) + "removes": ["urn:srcos:event:"] // OR-Set: for retract ops only + } +} +``` + +- **Ordering:** hash-DAG parents give the partial (causal) order; `lamport` + `eventId` break ties deterministically. `occurredAt` is never trusted for ordering (clocks lie — turn-1 lesson). +- **Retraction:** an OR-Set remove names the exact add-tags it observed → concurrent re-adds survive (add-wins). + +## 5. The eighth ideal: correctness as a merge precondition + +This is the part local-first (2019) never had, and our moat. Each op carries the **locus it was authored/attested at**, reusing `SyncCycleReceipt.locus` verbatim: + +```jsonc +"crdt": { + "locus": "trusted_private", // local | trusted_private | attested_fog | burst_cloud + "attestationRef": "urn:srcos:reasoning-receipt:...", // correctness-gate result, if any + "trustLevel": "trusted-workspace-source" // from ReasoningEvent vocab +} +``` + +**Merge gate (fail-closed):** +- Ops from `local` / `trusted_private` merge into the **canonical view**. +- Ops from `attested_fog` / `burst_cloud` merge into canonical **only if** `attestationRef` resolves to a passing correctness-gate receipt; otherwise they land in a **quarantine view** (mirrors the existing `QuarantineReceipt` pattern) and surface for review. +- The CRDT still *converges* on all ops (longevity: nothing is lost); the **gate decides which converged sub-graph is authoritative.** + +This is the key inversion: CRDTs guarantee *everyone sees the same graph*; our correctness gate decides *which part of that graph is trusted*. No competitor's local-first sync has this second layer. + +## 6. Receipts / evidence + +- Each merge cycle emits a receipt modeled on `SyncCycleReceipt` (reuse `cycleId`, `fromVersion`/`toVersion` = pre/post graph state hashes, `locus`, `outcome`, `auditId`). +- Optionally bind to a `ReasoningRun` via `agentplaneRunRef` when the merge was triggered by an agent task, so the graph's evolution is replayable on the same spine as reasoning. + +## 7. Ideal-by-ideal effect + +| Ideal | Before | After | Mechanism | +|---|---|---|---| +| 2 Multi-device | ● | ✓ | OR-Set converges across replicas | +| 3 Offline | ✓/– | ✓ | ops authored offline, merge on reconnect | +| 4 Collaboration | ● | ✓/– | async convergence (not live cursors) | +| 5 Longevity | ✓ | ✓ | op-log *is* the durable history (unchanged) | +| 6 Privacy | ✓/– | ✓/– | no central authority introduced | +| 7 User-control | ✓ | ✓ | locus-gated, replica-owned | +| **8 Correctness** | — | ✓ | **new**: attestation-gated canonical view | + +## 8. Build slices (each shippable, ordered) + +1. **Op emitter** — write PKG mutations as `EventEnvelope` ops with `crdt` payload. (No merge yet; just dual-write.) +2. **Deterministic fold** — rebuild graph state from the op-log; assert it matches current derived store (regression gate). +3. **OR-Set merge** — union two replicas' logs, prove convergence (property test: commutativity/idempotence over random op interleavings). +4. **Locus/attestation gate** — canonical vs. quarantine view split; emit `SyncCycleReceipt`-shaped merge receipts. +5. **Live ingester** — second device / agent as a real second replica. + +## 9. Open questions + +- Compaction/GC of the op-log at scale (tombstone horizon) vs. the longevity ideal — needs a retention/retraction policy (ties to the PKG retention open-loop already noted). +- Do agent replicas (multi-agent swarm) each get a `replicaId`, making swarm output a first-class merge participant? (Likely yes — unifies "collaboration" across humans + agents.) +- Whether `attestationRef` resolution happens at merge time (strict) or lazily (converge-then-attest). Recommend strict for `burst_cloud`, lazy for `attested_fog`. From 029ebfd94b6265aca526d3d8ceeb5f3226f3385d Mon Sep 17 00:00:00 2001 From: Michael Heller <21163552+mdheller@users.noreply.github.com> Date: Thu, 2 Jul 2026 11:52:11 -0400 Subject: [PATCH 2/4] docs(crdt): mark Slices 1-3 landed (emit + reconstruct + converge) Slices 1 (op emitter), 2 (deterministic materialize), 3 (OR-Set merge) shipped in prophet-mesh#13 with 14 tests; suite 74 green. Slice 4 (locus/attestation gate) next. --- docs/crdt-over-evidence-fabric.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/crdt-over-evidence-fabric.md b/docs/crdt-over-evidence-fabric.md index 5b16215..c8fd361 100644 --- a/docs/crdt-over-evidence-fabric.md +++ b/docs/crdt-over-evidence-fabric.md @@ -95,11 +95,11 @@ This is the key inversion: CRDTs guarantee *everyone sees the same graph*; our c ## 8. Build slices (each shippable, ordered) -1. **Op emitter** — write PKG mutations as `EventEnvelope` ops with `crdt` payload. (No merge yet; just dual-write.) -2. **Deterministic fold** — rebuild graph state from the op-log; assert it matches current derived store (regression gate). -3. **OR-Set merge** — union two replicas' logs, prove convergence (property test: commutativity/idempotence over random op interleavings). -4. **Locus/attestation gate** — canonical vs. quarantine view split; emit `SyncCycleReceipt`-shaped merge receipts. -5. **Live ingester** — second device / agent as a real second replica. +1. ✅ **Op emitter** — write PKG mutations as `EventEnvelope` ops with `crdt` payload. (No merge yet; just dual-write.) — *landed: prophet-mesh#13 `pkg_ops.py`, 5 tests.* +2. ✅ **Deterministic fold** — rebuild graph state from the op-log; assert it matches current derived store (regression gate). — *landed as `materialize()`; `test_materialize_matches_live_graph`.* +3. ✅ **OR-Set merge** — union two replicas' logs, prove convergence (property test: commutativity/idempotence over random op interleavings). — *landed: `merge()` + add-wins retracts; 9 tests incl. offline convergence, random-shuffle order-independence, add-wins-over-concurrent-retract. Suite 74 green.* +4. ⬜ **Locus/attestation gate** — canonical vs. quarantine view split; emit `SyncCycleReceipt`-shaped merge receipts. — *next.* +5. ⬜ **Live ingester** — second device / agent as a real second replica. ## 9. Open questions From d55eabf6db256676ca3098fea37098ff9da6d33b Mon Sep 17 00:00:00 2001 From: Michael Heller <21163552+mdheller@users.noreply.github.com> Date: Fri, 3 Jul 2026 00:43:40 -0400 Subject: [PATCH 3/4] docs(crdt): mark Slice 4 landed (locus/attestation merge gate) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit gate() ships in prophet-mesh#13 — canonical vs. quarantine split, fail-closed on gated loci, untrusted-retract security property, SyncCycleReceipt v2 receipts; 7 tests, suite 81 green. Only Slice 5 (live second-replica ingester) remains. --- docs/crdt-over-evidence-fabric.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/crdt-over-evidence-fabric.md b/docs/crdt-over-evidence-fabric.md index c8fd361..0456447 100644 --- a/docs/crdt-over-evidence-fabric.md +++ b/docs/crdt-over-evidence-fabric.md @@ -98,8 +98,8 @@ This is the key inversion: CRDTs guarantee *everyone sees the same graph*; our c 1. ✅ **Op emitter** — write PKG mutations as `EventEnvelope` ops with `crdt` payload. (No merge yet; just dual-write.) — *landed: prophet-mesh#13 `pkg_ops.py`, 5 tests.* 2. ✅ **Deterministic fold** — rebuild graph state from the op-log; assert it matches current derived store (regression gate). — *landed as `materialize()`; `test_materialize_matches_live_graph`.* 3. ✅ **OR-Set merge** — union two replicas' logs, prove convergence (property test: commutativity/idempotence over random op interleavings). — *landed: `merge()` + add-wins retracts; 9 tests incl. offline convergence, random-shuffle order-independence, add-wins-over-concurrent-retract. Suite 74 green.* -4. ⬜ **Locus/attestation gate** — canonical vs. quarantine view split; emit `SyncCycleReceipt`-shaped merge receipts. — *next.* -5. ⬜ **Live ingester** — second device / agent as a real second replica. +4. ✅ **Locus/attestation gate** — canonical vs. quarantine view split; emit `SyncCycleReceipt`-shaped merge receipts. — *landed: `pkg_gate.py` `gate()`, fail-closed admission + untrusted-retract security property + SyncCycleReceipt v2 conformance; 7 tests. Suite 81 green.* +5. ⬜ **Live ingester** — second device / agent as a real second replica. — *only slice remaining.* ## 9. Open questions From 4b9412dcaa74a6cc9c51f9fe314e29d59e50b72f Mon Sep 17 00:00:00 2001 From: Michael Heller <21163552+mdheller@users.noreply.github.com> Date: Fri, 3 Jul 2026 00:48:14 -0400 Subject: [PATCH 4/4] =?UTF-8?q?docs(crdt):=20all=205=20slices=20landed=20?= =?UTF-8?q?=E2=80=94=20design=20end-to-end=20real?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Slice 5 (live Replica + anti-entropy sync + HellGraph ingester + persist-only- canonical) ships in prophet-mesh#13. 27 CRDT tests, suite 87 green. emit → reconstruct → converge → correctness-gate → live-sync + persist proven-correct only. --- docs/crdt-over-evidence-fabric.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/crdt-over-evidence-fabric.md b/docs/crdt-over-evidence-fabric.md index 0456447..fd6da44 100644 --- a/docs/crdt-over-evidence-fabric.md +++ b/docs/crdt-over-evidence-fabric.md @@ -99,7 +99,9 @@ This is the key inversion: CRDTs guarantee *everyone sees the same graph*; our c 2. ✅ **Deterministic fold** — rebuild graph state from the op-log; assert it matches current derived store (regression gate). — *landed as `materialize()`; `test_materialize_matches_live_graph`.* 3. ✅ **OR-Set merge** — union two replicas' logs, prove convergence (property test: commutativity/idempotence over random op interleavings). — *landed: `merge()` + add-wins retracts; 9 tests incl. offline convergence, random-shuffle order-independence, add-wins-over-concurrent-retract. Suite 74 green.* 4. ✅ **Locus/attestation gate** — canonical vs. quarantine view split; emit `SyncCycleReceipt`-shaped merge receipts. — *landed: `pkg_gate.py` `gate()`, fail-closed admission + untrusted-retract security property + SyncCycleReceipt v2 conformance; 7 tests. Suite 81 green.* -5. ⬜ **Live ingester** — second device / agent as a real second replica. — *only slice remaining.* +5. ✅ **Live ingester** — second device / agent as a real second replica. — *landed: `pkg_replica.py` `Replica` + anti-entropy `sync()` + `replay_to_hellgraph()` (closes the emit-only gap) + `persist_canonical()` (only the gated-canonical graph reaches HellGraph); 6 tests. Suite 87 green.* + +**All 5 slices landed** (prophet-mesh#13, 27 CRDT tests, suite 87 green). The design is end-to-end real: emit → reconstruct → converge → gate on correctness → live-sync + persist only the proven-correct sub-graph. ## 9. Open questions