chore: forward-merge dev into main (stable mirror sync)

README.md

@@ -175,6 +175,17 @@ since 0.3.0.
 → See [`docs/benchmarks.md`](docs/benchmarks.md) for reproducible
 evidence (synthetic benchmark suite runnable in ~5 minutes).
 
+### Tamper-Evident Audit Trail
+
+Opt-in hash-chain **journal** that records every thought/edge mutation as a
+SHA-256-linked, before/after entry — off by default, one config flag to enable.
+Query history with `store.journal.get_entries(...)` and validate the chain with
+`store.journal.verify_integrity()`.
+
+→ See [`docs/audit-trail.md`](docs/audit-trail.md) for enabling, querying,
+verification, and the security model (what "tamper-evident" does and does not
+guarantee).
+
 ### Multi-Service Isolation
 
 Run multiple independent databases under one `EngravaManager`:
@@ -203,6 +214,8 @@ engrava --db mydata.db export -o portable.json
 `engrava info` now renders the same metrics snapshot contract exposed by
 `await store.metrics()`.
 
+See the [CLI reference](docs/cli.md) for every command and option.
+
 ## Architecture
 
 - **SQLite** with WAL mode for concurrent reads
@@ -214,13 +227,30 @@ engrava --db mydata.db export -o portable.json
 
 ## Documentation
 
-- [Upgrade Guide](docs/upgrade.md) — compatibility matrix, backups, and troubleshooting
+- [Core Concepts](docs/concepts.md) — the mental model (thought, edge, reflection, cycle, …) — start here
+- [Positioning](docs/positioning.md) — when Engrava is (and isn't) the right tool, and how it compares
 - [Quick Start](docs/quickstart.md) — 5-minute setup guide
+- [Tutorial](docs/tutorial.md) — build a small notes memory end to end
+- [Recipes](docs/recipes/index.md) — copy-paste snippets for common tasks (store a turn, retrieve context, TTL, dedup, …)
+- [Building a memory-backed agent](docs/guides/agent-memory.md) — the end-to-end agent turn loop (ingest → retrieve → generate → consolidate)
+- [Migrating from another memory system](docs/guides/migrating-from-other-memory.md) — concept mapping, porting calls, bulk import, and scoping/multi-tenancy
+- [Embeddings](docs/guides/embeddings.md) — wiring a real embedding provider (local / OpenAI / Ollama / HuggingFace / custom)
 - [Configuration](docs/configuration.md) — YAML config format and options
+- [Upgrade Guide](docs/upgrade.md) — compatibility matrix, backups, and troubleshooting
 - [Extensions](docs/extensions.md) — Writing custom extensions and hooks
 - [Observability](docs/observability.md) — Metrics snapshot API
+- [Audit Trail](docs/audit-trail.md) — Tamper-evident hash-chain journal (enabling, querying, verifying, security model)
 - [API Reference](docs/api-reference.md) — Full protocol and class reference
+- [CLI Reference](docs/cli.md) — every `engrava` command and option
+- [Glossary](docs/glossary.md) — quick definitions of every Engrava term
 - [MindQL](docs/mindql.md) — Query language syntax and examples
+- [Troubleshooting](docs/troubleshooting.md) — symptom → cause → fix for common errors
+- [FAQ](docs/faq.md) — quick answers (LLM/keys, embeddings-optional, scale, concurrency, backups, …)
+- [Performance & Scaling](docs/performance.md) — the vector-backend switch, bulk-ingest, and dreaming cost at scale
+- [Data Lifecycle & Retention](docs/data-lifecycle.md) — lifecycle states, TTL, archive-vs-delete, GDPR erasure, disk reclamation
+- [Deployment](docs/deployment.md) — process model, database files on disk, containers, graceful shutdown
+- [Concurrency](docs/concurrency.md) — the WAL single-writer model, busy timeout, and per-service isolation
+- [Backup & Recovery](docs/backup-and-recovery.md) — WAL-safe backups, snapshot vs file copy, restore verification
 - [Known Limitations](docs/known-limitations.md) — Platform notes and constraints
 
 ## Development

docs/api-reference.md

@@ -65,6 +65,7 @@ keyword arguments and does **not** return a UUID string.
 | `await list_thoughts(...)` | `list[ThoughtRecord]` | List with filters (keyword-only) |
 | `await count_thoughts(...)` | `int` | Count with filters (keyword-only) |
 | `await delete_thought(thought_id)` | `bool` | Hard delete; `True` if a row was removed |
+| `await record_access(thought_id)` | `None` | Mark a thought as accessed — bumps `access_count` and sets `last_accessed_at`; raises `ThoughtNotFoundError` if missing. Drives the access-frequency dreaming signal. |
 
 ```python
 import uuid
@@ -135,6 +136,40 @@ await store.create_edge(
 )
 ```
 
+#### REFLECTION lineage
+
+Helpers for navigating the `CONSOLIDATED_FROM` graph that dreaming builds
+between a REFLECTION and the source thoughts it summarises.
+
+| Method | Returns | Description |
+|--------|---------|-------------|
+| `await consolidated_member_ids(reflection_id)` | `list[str]` | The thought IDs a REFLECTION was consolidated from |
+| `await consolidated_source_statuses(reflection_id)` | `list[str]` | The lifecycle statuses of those source thoughts (e.g. to detect a fully-archived, orphaned cluster) |
+| `await reflections_consolidated_from(source_id)` | `list[str]` | The REFLECTION IDs that consolidated a given source thought (the reverse direction) |
+| `await thought_exists_by_source(*, source, thought_type_value)` | `bool` | Whether any thought exists with the given `source` and type — keyword-only |
+
+```python
+# Walk a REFLECTION down to its sources, and back from a source to its REFLECTIONs.
+member_ids = await store.consolidated_member_ids(reflection_id)
+for thought_id in member_ids:
+    source = await store.get_thought(thought_id)
+    if source is not None:
+        print(source.essence)
+
+# Detect an orphaned cluster — every source archived/gone:
+statuses = await store.consolidated_source_statuses(reflection_id)
+is_orphaned = bool(statuses) and all(s != "ACTIVE" for s in statuses)
+
+# Reverse direction: which REFLECTIONs summarise this source?
+parents = await store.reflections_consolidated_from(member_ids[0])
+
+# Exact-source existence check (e.g. dreaming's idempotency guard — a REFLECTION's
+# source is "dreaming:<cluster_hash>", so match the full value, not a prefix):
+exists = await store.thought_exists_by_source(
+    source="dreaming:abc123def4567890", thought_type_value="REFLECTION"
+)
+```
+
 #### Embedding Operations
 
 | Method | Returns | Description |
@@ -167,11 +202,23 @@ returns a single `HybridSearchResult` container.
 | `await metrics()` | `EngravaMetrics` | Snapshot of thought/edge counts, storage, and search-latency percentiles (see [Observability](observability.md)) |
 | `await cleanup_expired(now=None, *, exclude_id=None)` | `CleanupResult` | Archive or delete thoughts past their `expires_at` |
 | `await verify_embedding_model()` | `None` | Raise `EmbeddingModelMismatchError` if the stored model lock disagrees with the configured provider |
+| `async with store.suspend_auto_commit():` | context manager | Defer per-call commits so a block of writes commits once (rolls back on error) — use for bulk ingest |
 | `await close()` | `None` | Close the owned connection (only when the store opened it via `from_config`) |
 
+```python
+# Bulk ingest: one transaction instead of one commit per write.
+async with store.suspend_auto_commit():
+    for record in many_records:
+        await store.create_thought(record)
+# commit happens once on clean exit; any exception rolls the whole block back
+```
+
 ### `ReadOnlyEngrava`
 
-Wrapper that raises `ReadOnlyViolationError` on any write operation.
+A composition wrapper that delegates reads to the wrapped store and raises
+`ReadOnlyViolationError` on any write. Use it to hand a retrieval-only view of
+shared memory to a component that should never mutate it — e.g. a sub-agent or
+worker whose job is only to look things up.
 
 ```python
 from engrava import ReadOnlyEngrava
@@ -299,14 +346,51 @@ extension is recommended for filtering queries (`json_extract(metadata_json, '$.
 
 ### `ActionRecord`
 
+Records an action the agent took (a tool call, a message, …), linked to the
+thought that prompted it, with execution and verification state.
+
 | Field | Type | Description |
 |-------|------|-------------|
 | `action_id` | `str` | UUID primary key |
-| `source_thought_id` | `str` | Linked thought |
+| `source_thought_id` | `str` | The thought this action originated from |
 | `action_type` | `ActionType` | Action classification |
-| `intent` | `str` | Description of intent |
-| `status` | `ActionStatus` | Current status |
+| `intent` | `str` | Description of intent (min length 1) |
+| `status` | `ActionStatus` | Current execution status |
 | `verification_status` | `VerificationStatus` | Verification state |
+| `raw_metrics_json` | `str \| None` | Optional ground-truth facts for verification |
+
+**Store methods** (on `SqliteEngravaCore`):
+
+| Method | Returns | Description |
+|--------|---------|-------------|
+| `await create_action(action)` | `ActionRecord` | Persist an `ActionRecord` |
+| `await get_actions(thought_id)` | `list[ActionRecord]` | Actions linked to a thought |
+
+`ActionStatus` is a state machine: `PLANNED → EXECUTING → CONFIRMED` / `FAILED`,
+and `PLANNED → BLOCKED → PLANNED`. `can_transition_to(...)` / `evolve(...)`
+enforce valid transitions (an illegal change raises `InvalidTransitionError`).
+
+```python
+import uuid
+from engrava import ActionRecord, ActionType, ActionStatus, VerificationStatus
+
+action = ActionRecord(
+    action_id=str(uuid.uuid4()),
+    source_thought_id=prompting_thought_id,
+    action_type=ActionType.TOOL_CALL,
+    intent="search the web for flight prices",
+    status=ActionStatus.PLANNED,
+    verification_status=VerificationStatus.PENDING,
+)
+await store.create_action(action)
+
+# advance through the lifecycle (frozen model → evolve returns a new instance):
+done = action.evolve(status=ActionStatus.EXECUTING).evolve(
+    status=ActionStatus.CONFIRMED
+)
+
+actions = await store.get_actions(prompting_thought_id)
+```
 
 ### `HybridSearchResult`