docs: expand and correct the engrava documentation set#17
Merged
Conversation
Document the opt-in journal: enabling it (off by default), which mutations are recorded automatically (thought + edge CRUD, plus TTL expiry as UPDATE_THOUGHT/DELETE_THOUGHT), the JournalEntry schema, querying via get_entries(), and verifying the chain via verify_integrity(). Includes a runnable worked example and a security-model section stating plainly what the keyless in-file chain does and does not guarantee (detects accidental corruption + naive edits; not forgery-proof against an actor with file write access) plus hardening guidance. Link it from the README feature list and docs index.
Introduce the domain model as a coherent mental model before the how-to docs: thought (essence vs content, the type taxonomy with when-to-use and which are system-made, priority, lifecycle), edge, embedding, and reflection. Define the "cycle" logical clock explicitly — it is consumer-owned, Engrava never advances or persists it, and leaving it at 0 silently disables recency and stalls dreaming. Cover provenance (KnowledgeSource; source vs source_type), visibility as the inner/outer-speech boundary, and the confidence-vs-confirmation_count distinction. Includes a small thought-graph diagram and a worked construction example. Link it first in the README docs index, before Quick Start.
…lout Fix an inaccuracy in the Core Concepts cycle section: passing current_cycle=0 does NOT skip recency. Distinguish the two real failure modes — current_cycle omitted (None) drops the recency signal entirely, whereas a constant value keeps recency active but useless (every thought's age is current_cycle minus updated_cycle, so a frozen clock makes everything look equally fresh); the frozen clock is also why dreaming's age gate never opens. Add a callout at the first use of created_cycle in the quickstart explaining the cycle is consumer-owned (0 is fine for the quickstart, increment per turn in a real agent), linking to the Core Concepts cycle section.
Document the canonical agent integration: a per-turn loop that stores the user message as a percept, retrieves relevant memory via search_hybrid (passing the cycle), builds a prompt from retrieved essences, calls the LLM, stores the reply as an utterance, records the action, advances the consumer-owned cycle, and consolidates every N turns. Covers swapping in a real LLM/embedder, scheduling consolidation, and persistence across restarts (embeddings persist; recover the cycle from the latest updated_cycle). Ship examples/agent_loop.py as a complete, dependency-free runnable version (deterministic CallbackProvider + mock LLM) and cover it with a subprocess test so it can't silently break. Link the guide from the README docs index.
- examples/agent_loop.py: replace the incorrect struct.unpack("64B"*4 ...)
pseudo-embedding (a malformed format that only worked by accident) with a
clean byte-based deterministic vector of length EMBED_DIM; drop the now-unused
struct import.
- Thread session_id and turn_index explicitly through both the guide and the
example (metadata = {**percept(...)/utterance(...), session_id, turn_index})
so memories are anchored to their conversation and turn.
- Remove the links to the not-yet-existing docs/guides/embeddings.md; point at
the existing configuration guide instead.
- Correct the search_hybrid description: query_vector is optional — when an
embedding provider is configured the store embeds the query text for you;
passing the vector is just an override.
- Make the cycle-recovery text and code agree (both use updated_cycle, matching
list_thoughts ordering).
Formatting only — collapse two function signatures that fit on one line. No behaviour change.
…embedder Add docs/guides/embeddings.md covering all five providers (SentenceTransformer / OpenAICompatible / Ollama / HuggingFace / Callback): the install extra, constructor signature and defaults, env-var handling for keys (OPENAI_API_KEY, HF_TOKEN), and the YAML provider name resolved by resolve_embedding_provider. Explain the query side — search embeds the query for you when a provider is configured, or you pass query_vector — and that the vector signal is skipped (lexical-only) when no provider is set. Replace the misleading lambda text: [0.1] * 384 placeholder in the quickstart with a real SentenceTransformerProvider example and a note about auto_embed. Link the guide from the README docs index and the quickstart.
The embeddings guide treated both search methods as auto-embedding the query, which is wrong. Distinguish them: search_hybrid(query_text, query_vector=None) embeds the query text for you when a provider is configured, and only it has the no-provider-and-no-vector lexical-only fallback; search_similar(query_vector) takes a ready vector as a required first argument and never auto-embeds, so the query must be embedded first (query_vec = await provider.embed(...)). Fix both the "query-time embedding" summary and the "query side" section to match.
Add docs/recipes.md — goal-titled copy-paste snippets for the common agent-memory tasks: store a conversation turn, retrieve context for a prompt, filter retrieval by session (over-fetch + post-filter, since ranked search has no metadata filter), set a TTL, deduplicate repeated facts, run consolidation on a schedule, and inspect the audit trail. Add docs/tutorial.md — a guided, typed-from-scratch build of a small notes memory (ingest, embed, link, search) whose final script runs end to end. Re-point the quickstart Next Steps to lead with the tutorial and recipes before the references, and link both from the README docs index. All snippets verified against the running package.
…example Address the cookbook spec: relocate the cookbook to docs/recipes/index.md (the recipes/ deliverable) and fix all relative links; repoint README, quickstart, and tutorial to it. Expand from 7 to 9 goal-titled recipes — add "Record a tool result / action" (ActionRecord + get_actions) and "Restore the cycle counter after a restart" (seed from list_thoughts' updated_cycle ordering). Ship examples/notes_memory.py as the runnable companion to the tutorial (deterministic CallbackProvider, no external services) and cover it with a subprocess test so it can't silently break.
…lk, read-only) Fill the remaining shipped-but-undocumented surface in api-reference.md: - ActionRecord: full field table (incl. raw_metrics_json and the intent min-length), the create_action / get_actions store methods, the ActionStatus lifecycle (PLANNED -> EXECUTING -> CONFIRMED/FAILED, PLANNED <-> BLOCKED) with evolve()/InvalidTransitionError, and a worked plan->execute->confirm example. - REFLECTION lineage: consolidated_member_ids / consolidated_source_statuses / reflections_consolidated_from / thought_exists_by_source for walking the CONSOLIDATED_FROM graph. - record_access (bumps access_count + last_accessed_at; drives the frequency dreaming signal). - suspend_auto_commit as the supported bulk-write batching context manager, with an example. - ReadOnlyEngrava: a use-case note (hand a retrieval-only view to a sub-agent). All signatures and behaviours verified against the running package.
Add a worked traversal snippet under the REFLECTION-lineage helper table in api-reference.md: walk a REFLECTION to its members and back, detect an orphaned cluster from consolidated_source_statuses, and use thought_exists_by_source as an exact-source check (noting a REFLECTION source is "dreaming:<cluster_hash>", so it matches the full value, not a prefix — verified). Add a callout in dreaming.md next to CONSOLIDATED_FROM pointing at these helpers and linking the API reference. Docs only.
configuration.md covered database/search/embeddings/dreaming/services but omitted five sections that config.py actually parses. Add a key/type/default reference table (and YAML example) for each: journal (enabled), ttl (strategy archive|delete, check_every_n_operations, default_ttl_seconds), ingest (deduplication_enabled), hooks (class dotted path), and manifests (list form or discover/paths mapping). Cross-link audit-trail, recipes, and extensions; note that metrics is documented in observability. Every documented key verified by loading a YAML that exercises all five sections through load_config.
hooks.class resolves via rsplit(".", 1) (module.path + ClassName), so it takes a
plain dotted path like "my_package.hooks.MyHooks" — not the module.path:ATTRIBUTE
colon form, which is specific to manifests.paths. The previous example used the
colon form, which would fail to import. Add a note contrasting the two formats.
ENGRAVA_DB / ENGRAVA_CONFIG are CLI-only env vars (cli/config.py): ENGRAVA_DB is
the fallback for --db (--db > ENGRAVA_DB > ./engrava.db), not an override of an
internal db_path attribute. Restate both in terms of the CLI flag they back.
Two new pages close the "is this the right tool / how do I move to it" gap: - docs/positioning.md: good-fit vs not-a-fit, an orientation comparison against hosted memory services, framework memory, and standalone vector DBs, and an explicit non-goals section (no LLM-side extraction; retrieval unscoped by default; not distributed; not a framework). - docs/guides/migrating-from-other-memory.md: a concept-mapping table, before/ after porting snippets, a runnable bulk-import recipe (suspend_auto_commit for one-transaction commit/rollback + deduplicate=True to collapse repeats), and a filtering/scoping/multi-tenancy section covering the three patterns for the unscoped search_* methods: over-fetch+post-filter, EngravaManager-per-tenant, and raw json_extract on metadata_json, with tradeoffs. Every API claim was verified against the running package: search_* take no scope filter; suspend_auto_commit commits on success and rolls back on error; create_thought(deduplicate=True) collapses identical content; metadata persists to the metadata_json column queryable via json_extract. The bulk-import snippet is registered as an executable doc-test, so its dedupe-collapse behaviour is asserted on every run. Linked both pages from the README documentation index.
The concept-mapping row for embeddings implied a configured provider was enough to embed on write. create_thought only embeds when embedding_provider is set AND auto_embed=True (default False); otherwise the caller stores the vector via store_embedding(thought_id, vector). State both conditions and the manual path. Add `assert total == 3` to the runnable bulk-import snippet so the executable doc-test fails if deduplicate=True ever stops collapsing identical content, instead of merely printing a wrong count.
Two new pages close the support gap for common errors and "is this supposed to work this way" questions. docs/troubleshooting.md — symptom -> cause -> fix for the six failure modes that actually trip people up, each verified against the running package: - AttributeError 'tuple' object has no attribute 'keys' -> set conn.row_factory (fails on read, not on connect/write) - ValueError '...' is not a valid ThoughtType -> use a real enum member - search returns nothing -> the signal-skipped checklist (no provider skips vector; current_cycle None or recency_weight 0 skips recency) + backends_used - dreaming promotes nothing -> the two independent bars: the age gate (current_cycle - created_cycle >= min_age_cycles) and promote_threshold - EmbeddingModelMismatchError -> model name/dimension recorded at first embed - ReferentialIntegrityError -> edge endpoint missing, and it is not re-exported from the top-level package (import from engrava.domain.exceptions) docs/faq.md — concise answers on no-LLM/no-key, no network/service, embeddings optional, corpus scale and the sqlite-vec switch, single-writer concurrency, scoping, when to enable dreaming, the cycle counter, WAL-safe backup, tamper-evident (not tamper-proof) audit, and production-readiness. Linked both from the README documentation index and the quickstart Next Steps.
Two corrections so the FAQ matches docs/concepts.md and the shipped behaviour: - current_cycle: the previous wording said leaving it at 0 "disables recency". Per the recency_active = current_cycle is not None and recency_weight > 0 contract, only current_cycle=None makes recency inactive. A constant (e.g. 0) keeps recency active but useless — every age collapses to the same value and the dreaming age gate never opens. State both cases distinctly. - dreaming: drop the "from v0.4.0 create REFLECTION summaries" version claim, which concepts.md does not make (it describes REFLECTION present-tense) and which the code supports today. Describe clustering into REFLECTION summaries present-tense and link to Core Concepts.
New docs/performance.md explains what drives cost and the two main levers, with mechanics verified against the running package rather than invented numbers: - per-signal cost table (FTS sub-linear; numpy vector linear; sqlite-vec sub-linear; recency/priority negligible; graph opt-in) - the ~100k brute-force ceiling and when to move past it - a turnkey "switch to sqlite-vec" walkthrough including migrating an existing database: from_config creates the vec0 index and back-fills existing embeddings automatically (no re-embed), new writes stay in sync; with the important caveats — only from_config wires the backend, missing/unloadable extension falls back to numpy with a warning (not a crash), the macOS system-SQLite extension block, and the dimension-match requirement - write throughput and a bulk-ingest recipe via suspend_auto_commit (one commit, rollback on error) with the deduplicate=True interaction and the embedding-cost caveat - dreaming cost at scale (off the hot path; candidates_limit; clustering_backend) Restored the migration guide's performance cross-link now that the page exists, and linked the new page from the README documentation index. Only qualitative cost guidance plus the documented ~100k figure — no fabricated latency numbers.
In the pinned sqlite-vec 0.1.x line, a vec0 query is an exhaustive k-nearest-neighbour scan over a compact, chunked columnar store — faster and more memory-efficient than the Python brute-force path, but not an approximate or sub-linear index. Calling it "ANN" / "O(log n)" / "sub-linear" overstated the guarantee. Correct the claim everywhere it appears, in the public docstrings that ship with the package as well as the guide: - performance.md: vector cost is linear for both backends (sqlite-vec just has a much smaller constant factor over the vec0 table); rewrite the ceiling and migration sections to describe a compact vec0 vector table, not an ANN index. - vector_sqlite_vec.py, engrava_core.py, config.py, extensions/__init__.py: replace "ANN" / "O(log n) approximate nearest-neighbor" wording with neutral "vec0 vector table" / "k-nearest-neighbour" language and an explicit no-ANN-guarantee note. Behaviour is unchanged (docstrings/prose only): sqlite-vec unit tests and the doc-example suite stay green. The accurate claims are kept intact — from_config wiring, automatic back-fill via sync_embeddings, graceful numpy fallback, and no re-embedding.
New docs/data-lifecycle.md, with every persistence/erasure claim verified against the running package so the privacy guidance is accurate: - the four LifecycleStatus states (CREATED/ACTIVE/DONE/ARCHIVED) - TTL: per-thought expires_at, expires_after_seconds at create time, and the ttl.default_ttl_seconds store default; expiry is not timer-driven; expired rows are excluded from count_thoughts/list_thoughts unless include_expired=True - archive-vs-delete: the default "archive" strategy marks ARCHIVED and keeps the content; "delete" removes the row. Verified: an archived thought retains its content; a deleted thought is gone from the thought table - running cleanup: cleanup_expired() -> CleanupResult(expired_count, strategy_applied, timestamp); ttl.check_every_n_operations; engrava gc --expired / --dry-run - GDPR and hard deletion: archive does not erase; with journaling on, a delete leaves the content in the journal delta (verified: both the INSERT_THOUGHT and DELETE_THOUGHT entries carry it), and backups retain it — so a correct erasure must gc/delete the row, purge the journal entries, and roll through backups - reclaiming disk space: deleting rows does not shrink the file (verified: gc deletes rows and the source never runs VACUUM); freed pages go to SQLite's free-list. Reclaiming size needs VACUUM, with its exclusive-lock / ~2x-temp / off-peak caveats Also corrects the upgrade guide's "compact" wording (gc removes rows but does not shrink the file), restores the migration guide's data-lifecycle cross-link now that the page exists, and links the page from the README documentation index.
Two accuracy fixes in data-lifecycle.md, both verified against the running CLI and core: - engrava gc --expired is strategy-dependent. With ttl.strategy: delete it deletes expired rows and then collects pre-existing ARCHIVED rows in the same pass. With the default archive strategy it archives the expired rows and STOPS — it does not also collect archived rows that run (gc returns early when the expiry pass archived rows), so removing archived rows needs a separate engrava gc or the delete strategy. The previous text wrongly said the --expired pass "does both" under archive; the GDPR section repeated that error. - ARCHIVED is not a global results filter. An archived regular thought still appears in search_hybrid/search_fts and is still counted by count_thoughts/list_thoughts (verified). Only expired thoughts (via the TTL expiry checks) and retired REFLECTIONs (a REFLECTION whose lifecycle_status is no longer ACTIVE is excluded from search by a freshness floor, type-specific to REFLECTIONs) are auto-excluded. To hide archived regular thoughts, filter on lifecycle_status or remove them with gc. Clarify the lifecycle table and add a note spelling this out.
Normalize trailing-comment whitespace inside python code blocks on the pages added in this docs expansion so they match the rest of the docs corpus (which uses a single space before #) and satisfy ruff format --preview. Whitespace in comments only — no prose, code, or behaviour change; the executable doc-test blocks are unaffected and the suite stays green. Pre-existing pages are left untouched.
Three new pages cover running Engrava in production, with every operational claim verified against the running package: - deployment.md: open one store per process at startup via from_config (which owns the connection); never share a store across event loops; the three WAL files on disk (.db / -wal / -shm) and their permission/volume implications; container and multi-worker guidance; graceful close(). - concurrency.md: the WAL many-readers/one-writer model; a single store safely serves many async tasks in one loop (aiosqlite serialises on its worker thread plus the store's internal locks); busy_timeout is the inherited Python sqlite3 default of 5000 ms (engrava does not override it) and how to raise it; multiple-process writing is unsupported — the audit journal's lock is an in-process asyncio.Lock keyed on the connection, so cross-process writers race the journal sequence_number and, after 5 retries, raise "Failed to append journal entry after 5 retries due to sequence contention"; EngravaManager gives per-service file isolation. - backup-and-recovery.md: logical snapshot vs physical file backup. Verified that engrava snapshot exports thought/edge/embedding/action records but NOT the journal_entry table, so a restored database starts with an empty audit journal — documented as an explicit warning. WAL-safe physical backup options (wal_checkpoint(TRUNCATE) then copy / copy all three files / VACUUM INTO / Online Backup API), restore verification via engrava info, and multi-service backups. Also fixes the upgrade guide's WAL-unsafe `cp` snippet (checkpoint first, or copy the -wal/-shm siblings) and adds a "snapshot excludes the audit journal" note to its downgrade flow. Linked all three pages from the README documentation index.
Three accuracy fixes verified against the running package: - deployment.md close semantics. store.close() only closes a connection the store OWNS. Verified: from_config sets _owns_connection=True so close() (or leaving the context manager) closes it; the manual SqliteEngravaCore(conn) constructor leaves _owns_connection=False, so store.close() is a no-op and the caller must close the connection it created. Split the shutdown guidance into the owned and caller-managed cases. - deployment.md ENGRAVA_DB. It is a CLI-only fallback for the engrava --db flag, not a setting read by from_config. Application code should set database.path in engrava.yaml; clarify that ENGRAVA_DB does not configure from_config. - backup-and-recovery.md WAL-safe backup. A file copy of a live database is not reliable, and copying the .db/-wal/-shm trio non-atomically while writes continue can still be inconsistent. Restructure by live-vs-stopped: for a live database recommend the SQLite Online Backup API or VACUUM INTO; only do a file copy (checkpoint-then-copy the single file, or an atomic multi-file filesystem snapshot) when writers are stopped or quiesced. Align the restore section and the deployment files-on-disk note with this.
store.close() is just await self._db.close() on the owned connection — it issues no WAL checkpoint (verified: no checkpoint call in close() or anywhere in the store class). Describe it as closing/releasing the owned connection cleanly, and point to Backup & Recovery for an explicit PRAGMA wal_checkpoint(TRUNCATE), which belongs to the backup/maintenance flow rather than connection teardown.
New docs/cli.md documents every engrava command and option, captured verbatim
from --help and verified against the running CLI:
- global options: --db (with ENGRAVA_DB fallback), --config (ENGRAVA_CONFIG),
--format {table,json,csv} (default table), --verbose
- info, query (MindQL positional arg), snapshot (-o/--output, --service),
restore (-i/--input required, --clear, --skip-embeddings, --re-embed,
--service), gc (--dry-run, --expired), migrate, export (-o/--output, --status)
- the restore --skip-embeddings / --re-embed mutual-exclusion rule, with the
exact error message it prints
- gc --expired behaviour cross-linked to the data-lifecycle strategy semantics
- snapshot/restore audit-journal exclusion note
- documents that there is no `engrava verify` command; journal verification is
the Python store.journal.verify_integrity() call
Linked the page from the README CLI section and the documentation index, and
restored the cli.md cross-links in backup-and-recovery.md and data-lifecycle.md
that were softened while the page did not yet exist.
The snapshot reference conflated the single-database and multi-service cases. Verified against the command: - default output path is mode-dependent: single database -> <db-stem>.snapshot.jsonl via Path.with_suffix (engrava.db -> engrava.snapshot.jsonl, the .db is replaced not appended); multi-service -> <data_dir>/<service>.snapshot.jsonl. - --service only applies when a services config is loaded; in that case omitting it falls back to services.default_service. With no services config, --service has no effect and the command snapshots the single --db database.
Both commands resolve --service identically (verified in the source). The prior text wrongly said --service has no effect without a services config. In fact: - explicit --service NAME targets that service even with no services config — the service database resolves in the services data_dir if a config is loaded, else in the parent directory of --db (cfg.db_path.parent); - omitted with a services config falls back to services.default_service; - omitted with no services config operates on the single --db database. Add a shared "Service resolution" section stating the three cases once, and link both snapshot and restore to it instead of repeating (and mis-stating) the rule.
New docs/glossary.md gives a one-paragraph definition of every Engrava term — thought, essence, content, edge, embedding, reflection, dreaming, consolidation, promotion, cycle, signal, gate, priority, lifecycle, provenance, confirmation, visibility, hybrid search, graph signal, percept, utterance — each linking to the page that explains it in depth. Definitions are taken from Core Concepts and the search/dreaming docs and checked against the code (enum members verified), so the glossary stays consistent with the source of truth. The lifecycle entry uses the accurate ARCHIVED behaviour (a retained, still-searchable retention state, not an automatic results filter). Linked the page from the README documentation index and added a glossary pointer at the top of Core Concepts (first use). Also normalized the pre-existing "Putting it together" example in concepts.md to single-space comments and one-per-line imports so the page is clean under ruff format --preview.
The Core Concepts lifecycle section still claimed "ARCHIVED thoughts are excluded from normal results", which contradicts the glossary and the verified behavior: an archived regular thought is not auto-hidden from search_hybrid / list_thoughts / count_thoughts and stays searchable until garbage-collected. Restate ARCHIVED as a soft-retired retention state and GC marker, note that the only auto-excluded rows are expired thoughts and retired REFLECTIONs, and link to data-lifecycle.md for the full retention/GC detail. No other semantics changed.
Expands observability.md with how to monitor a deployment, with every claim verified against the running package: - exporting the metrics() snapshot to Prometheus/OTel/StatsD (Prometheus example), listing the exact EngravaMetrics fields available (thoughts, edges, storage with db/wal/vec_index/total bytes, search_latency percentiles) - scrape cadence guidance (30-60s; the latency window already smooths spikes) - a "what to alert on" table: storage and WAL growth, search p95/p99 past the brute-force vector ceiling, expired backlog (computed from include_expired), and journal verify_integrity() failures - a health-check pattern built on metrics() - logging: the engrava.* logger namespace and the levels actually used (WARNING/INFO/DEBUG; no ERROR/CRITICAL — failures are raised as typed exceptions) - an explicit out-of-scope statement: no write/error counters, no dreaming metrics (use ConsolidationResult), no journal-size/per-event audit metrics (the journal is queried and verified directly). Verified that the snapshot dataclass contains no such fields. Also adds the PEP-8 blank line the pre-existing Quick Example block was missing so the page is clean under ruff format --preview.
All verified against the running package: - field list: reworded from "the fields available are exactly those" to "the main metric groups are", and noted the top-level schema_version and snapshot_timestamp fields the exact-claim had omitted. - audit-integrity alert: store.journal is None unless journaling is enabled, so an unconditional verify_integrity() call fails with AttributeError. Mark the check as journaling-only and add a guarded snippet that returns early when store.journal is None. - health check: metrics() returns a zero-filled snapshot without issuing SQL when metrics.enabled is false, so it cannot confirm DB/schema readability in that case (verified: metrics() on a schema-less db with metrics disabled returns zeros, no error). Replace the probe with count_thoughts(), which always queries the database, and document the metrics-disabled caveat.
JournalIntegrityResult exposes `valid` (bool), not `is_valid`. Fix the journal verification snippets in observability.md and cli.md to read result.valid. A full docs sweep confirms these were the only two occurrences.
Opens dreaming.md with "How memory consolidation works (the dreaming loop)" — a single memory's journey (ingest -> confirm -> promote -> link -> reflect -> improved retrieval) with a small loop diagram, before the existing knob-by-knob reference. The phase order matches run_consolidation (promote, then edges, then reflections) and the steps are honest about the mechanics: nothing happens until you call run_consolidation(); promotion needs both the gates and the threshold; confirmation_count (not confidence) is the recurrence signal; edges are idempotent; a REFLECTION whose source cluster leaves the active set is retired. Each step links down to its detailed section and to Core Concepts / Search / Troubleshooting. Also normalizes two pre-existing code blocks (a missing PEP-8 blank line and aligned comments) so the page is clean under ruff format --preview.
Tighten the "How memory consolidation works" wording to match DreamingExtension: - opening: ingest and confirmation happen on the normal write path; only the consolidation part is manual. One run_consolidation() call runs promote -> edge creation -> reflection clustering/creation -> orphan sweep in order (added the sweep as a visible phase in the diagram). - edge creation (step 4 + diagram): a promoted thought *may* gain ASSOCIATED edges — only when edge creation is enabled, the thought has a stored embedding, and qualifying neighbours above min_similarity are found. Kept the idempotence note. - reflections (step 5 + diagram): related thoughts *may* be clustered into REFLECTIONs — only when reflections are enabled and eligible clusters pass the clustering/quality gates. Kept the stale-REFLECTION retirement note. Conceptual flow and links unchanged; no mechanics rewrite.
Adds a "Rolling upgrades (multiple workers)" section and brings the matrix up to the shipped version, both grounded in the migration code: - migrations are versioned by PRAGMA user_version and run forward-only inside a transaction. Most steps are additive (new columns/tables/indexes), but several rebuild a table in place (create-new, copy, drop-old, rename) — so a schema-changing release is not guaranteed backward-readable. Document that a patch upgrade with an unchanged user_version is safe to roll across workers, while a minor (schema-changing) upgrade needs a writer quiesce: back up, stop old workers, run the migration once, then start the new workers. - compatibility matrix: add the 0.3.0 -> 0.3.1 row (verified no schema change — schema_core.sql is untouched between the v0.3.0 and v0.3.1 tags, user_version stays 12), plus a patch-vs-minor rule of thumb and a 0.3.0 -> 0.3.1 version note.
examples/config.yaml described the sqlite-vec backend as "ANN" — inconsistent with the rest of the docs, which (correctly) describe vec0 in the pinned 0.1.x line as a faster brute-force KNN scan, not an approximate/sub-linear index. Reword the inline comment to "faster KNN, not ANN".
engrava-release Bot
pushed a commit
that referenced
this pull request
Jun 18, 2026
## 0.4.0 (2026-06-18) * Merge bi-temporal documentation and 0.3 to 0.4 upgrade notes into v0.4.0 ([a04b75c](a04b75c)) * Merge bi-temporal valid-time storage into v0.4.0 ([534cb5c](534cb5c)) * Merge chore/release-trigger-to-dev: switch release-trigger to dev + align docs + CI on dev ([3ce07b2](3ce07b2)) * Merge convenience API (remember/recall) and cycle defaults into v0.4.0 ([bd6402f](bd6402f)) * Merge doc-block execution (tutorial end-to-end + search round-trip) into v0.4.0 ([c582bf0](c582bf0)) * Merge docs accuracy fix + documentation-example tests ([f4e1e28](f4e1e28)) * Merge documented-defaults verification and metadata-helper docs into v0.4.0 ([8ff4145](8ff4145)) * Merge embedding-input repair into v0.4.0 ([825a9fc](825a9fc)) * Merge embedding-provider transient-error retry into v0.4.0 ([f959844](f959844)) * Merge full-text query repair into v0.4.0 ([ba1b288](ba1b288)) * Merge MCP delete tools (delete_thought, delete_edge) into v0.4.0 ([76598bb](76598bb)) * Merge MCP guided memory prompts into v0.4.0 ([2afef80](2afef80)) * Merge MCP memory filters and pagination into v0.4.0 ([62b2b60](62b2b60)) * Merge MCP resources (thought, stats, recent) into v0.4.0 ([62c7639](62c7639)) * Merge MCP server (read tools) + MindQL execution entry point into v0.4.0 ([82c65f0](82c65f0)) * Merge MCP server guide and client-config examples into v0.4.0 ([7ccdbd8](7ccdbd8)) * Merge MCP structured tool errors into v0.4.0 ([8c6811b](8c6811b)) * Merge MCP write tools, read-only mode and safety annotations into v0.4.0 ([3723d10](3723d10)) * Merge MindQL correctness fixes into v0.4.0 ([3f3fbeb](3f3fbeb)) * Merge pull request #13 from sovantica/dev ([f2d4f8b](f2d4f8b)), closes [#13](#13) * Merge pull request #14 from sovantica/chore/commitlint-ignore-merge-commits ([fbfe701](fbfe701)), closes [#14](#14) * Merge pull request #15 from sovantica/dev ([f93d026](f93d026)), closes [#15](#15) * Merge pull request #18 from sovantica/dev ([3db5530](3db5530)), closes [#18](#18) * Merge pull request #21 from sovantica/dev ([f07fef4](f07fef4)), closes [#21](#21) * Merge pull request #23 from sovantica/dev ([b2f1691](b2f1691)), closes [#23](#23) * Merge pull request #27 from sovantica/release/v0.4.0 ([fcc41c1](fcc41c1)), closes [#27](#27) * Merge pull request #28 from sovantica/fix/branch-guard-semver ([0598155](0598155)), closes [#28](#28) * Merge pull request #29 from sovantica/ci/release-app-token ([5432de6](5432de6)), closes [#29](#29) * Merge pull request from dev: docs accuracy fix + documentation-example tests (WS docs) ([7d1118d](7d1118d)) * Merge quickstart remember()/recall() short path into v0.4.0 ([1a89476](1a89476)) * Merge README extras list + drop no-op dreaming extra into v0.4.0 ([13c2448](13c2448)) * Merge README quickstart remember()/recall() into v0.4.0 ([22bf027](22bf027)) * Merge reflection temporal-extent inheritance into v0.4.0 ([e220f0a](e220f0a)) * Merge search functional contract suite into v0.4.0 ([30027f4](30027f4)) * Merge session-wide test thread-pinning into v0.4.0 ([114e85a](114e85a)) * Merge sqlite pragma tuning and hot-path indexes into v0.4.0 ([66afaf0](66afaf0)) * Merge temporal query predicates and invalidate primitive into v0.4.0 ([c160736](c160736)) * Merge temporal-query performance gate into v0.4.0 ([4779c9c](4779c9c)) * ci: add secret scan + dependency audit; verify wheel data on publish (#22) ([7cfaec3](7cfaec3)), closes [#22](#22) * ci: align branch-name guard allowed types with BRANCHING.md ([a7f3e98](a7f3e98)) * ci: allow semantic-version release and hotfix branch names ([26261b4](26261b4)) * ci: cache HuggingFace models and pip to stop HF 429 (#20) ([e43f3cc](e43f3cc)), closes [#20](#20) * ci: use GitHub App token for semantic-release push to protected dev ([1a282bd](1a282bd)) * ci(ci): ignore merge commits in commitlint via JS config ([3d54f61](3d54f61)) * ci(ci): run CI on dev branch as well as main ([d28ace2](d28ace2)) * docs: add bi-temporal model guide and 0.3 to 0.4 upgrade notes ([036e225](036e225)) * docs: bring architecture + CLI docs up to 0.4.0 (bi-temporal + MCP) ([cd5df19](cd5df19)) * docs: capitalize the Engrava brand name in README prose ([1254830](1254830)) * docs: correct mindql, extensions, extension-hooks, and configuration examples ([54149ef](54149ef)) * docs: correct README, quickstart, and api-reference examples to match shipped API ([36cf61d](36cf61d)) * docs: correct reflection_boost default to 1.0; add test verifying documented config defaults ([d42a852](d42a852)) * docs: document percept/utterance/thought metadata helpers in api-reference ([2e80d04](2e80d04)) * docs: drop reference to non-existent purity-check script in CONTRIBUTING ([59dcf5d](59dcf5d)) * docs: expand and correct the engrava documentation set (#17) ([2c3fa82](2c3fa82)), closes [#17](#17) * docs: fix 0.4.0 drift found in the full doc audit ([bb5dff1](bb5dff1)) * docs: fix quickstart cycle note — remember() takes no created_cycle; use ThoughtRecord for write-sid ([8c00368](8c00368)) * docs: lead quickstart with remember()/recall() short path ([edfe620](edfe620)) * docs: lead README Basic Usage with remember()/recall() ([3f415e2](3f415e2)) * docs: list all installable extras in README (mcp + ollama/hf embeddings); note dreaming needs no ext ([c55f0ad](c55f0ad)) * docs: trim README Basic Usage to the core create-and-read example ([b2bf7bf](b2bf7bf)) * docs(mcp): add MCP server guide and client-config examples ([b2bc18b](b2bc18b)) * docs(release): align branching guide with dev release-trigger model ([7619127](7619127)) * docs(tests): de-reference internal principle name in doc-test rationale ([ed8b184](ed8b184)) * build: drop no-op 'dreaming' extra (empty deps; dreaming is in the base install) ([c6fa946](c6fa946)) * build(deps): raise pydantic floor to >=2.11 ([d7eaff0](d7eaff0)) * feat: add bi-temporal valid-time to thoughts and edges ([456bcb6](456bcb6)) * feat: add temporal query predicates and invalidate primitive ([86de77f](86de77f)) * feat: reflections inherit temporal extent from members ([8fba769](8fba769)) * feat(api): add remember() and recall() convenience methods on the store ([be9b110](be9b110)) * feat(mcp): add delete_thought and delete_edge tools ([84b87f6](84b87f6)) * feat(mcp): add guided memory prompts ([1f6fa36](1f6fa36)) * feat(mcp): add MCP server with read tools (engrava[mcp] extra) ([68a8085](68a8085)) * feat(mcp): add memory filters and pagination ([57357b7](57357b7)) * feat(mcp): add write tools, opt-in read-only mode, and per-tool safety annotations ([79d7604](79d7604)) * feat(mcp): expose memory as resources (thought, stats, recent) ([c54dcf7](c54dcf7)) * feat(mcp): map known failures to typed, actionable tool errors ([8b615cc](8b615cc)) * feat(mindql): add store-level execute_mindql entry point ([1c8ffb4](1c8ffb4)) * fix: assert plan-shape invariant for temporal queries, not scan-vs-index ([0e4e176](0e4e176)) * fix: embed full thought content without duplication or silent truncation ([36e08e7](36e08e7)) * fix: keep quoted MindQL values as strings and reject malformed conditions ([d88043d](d88043d)) * fix: let natural-language queries reach the full-text index ([bb6b729](bb6b729)) * fix: match exact table token in query-plan helpers ([cd4ecc2](cd4ecc2)) * fix(embeddings): retry transient errors with bounded backoff ([897c46f](897c46f)) * fix(mcp): keep query_memory parse errors FIND-only ([5f4ea20](5f4ea20)) * fix(mcp): map write-tool errors and complete the 0.4.0 documentation ([6794436](6794436)) * test: add documentation-example test suite ([75b977e](75b977e)) * test: add functional contract suite for search behavior ([862e7bc](862e7bc)) * test: bound temporal query overhead and confirm index use ([035e860](035e860)) * test: isolate subprocess examples (offline, single-thread, no stdin) ([c3cf339](c3cf339)) * test: pin native thread pools session-wide to stop full-suite hang ([3a0eaa3](3a0eaa3)) * test(docs): execute the tutorial end-to-end and a search round-trip ([730c2bb](730c2bb)) * perf: tune sqlite pragmas and add hot-path indexes ([1256303](1256303)) * chore: back-merge main into dev after v0.3.0 release ([4769d22](4769d22)) * chore: back-merge main into dev after v0.3.1 release ([40e45b6](40e45b6)) * chore(ci): switch release-trigger branch from main to dev ([cd2bb24](cd2bb24)) * chore(deps): bump actions/cache from 4 to 5 (#25) ([c84fb8f](c84fb8f)), closes [#25](#25) * chore(deps): bump actions/setup-python from 5 to 6 (#26) ([0a69457](0a69457)), closes [#26](#26) [skip ci]
|
🎉 This PR is included in version 0.4.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A documentation expansion bringing
docs/up to the shipped behaviour of engrava0.3.x. Adds the missing pages developers were reverse-engineering, and corrects
drift in existing pages. Docs-only (plus the doc-example test suite and a few
public docstrings) — no library behaviour changes, no version bump.
New pages
cycle, provenance, visibility) — links first from the docs index.
its non-goals, concept mapping, a runnable bulk-import recipe, and a
filtering/scoping/multi-tenancy section.
examples/agent_loop.py),Embeddings guide, Recipes + Tutorial (+
examples/notes_memory.py).retention / erasure, Deployment, Concurrency, Backup & recovery,
CLI reference, Glossary.
rolling-upgrade guidance + a refreshed compatibility matrix.
Accuracy fixes
Numerous corrections verified against the running package — e.g. the audit-trail
threat model and journal-delta residue, the cycle/recency contract,
search_fts/
search_similarreturn shapes and the query-vector vs auto-embed distinction,the ARCHIVED lifecycle (retained + searchable, not auto-hidden), the sqlite-vec
backend described as faster brute-force KNN (not ANN/sub-linear),
gc --expiredstrategy behaviour, and CLI option tables.
Tests
tests/docs/compiles and phantom-scans every documentation code block andexecutes the self-contained ones against the installed package, so the examples
are verified to run and cannot silently drift.
Release impact
None — all commits are
docs:/style:(nofeat/fix/perf, no breakingchanges), so semantic-release will not cut a version.