diff --git a/.agents/docs/adr/README.md b/.agents/docs/adr/README.md index c31220b..242a22b 100644 --- a/.agents/docs/adr/README.md +++ b/.agents/docs/adr/README.md @@ -1,12 +1,22 @@ # Architecture Decision Records -This directory records architecture decisions for `MCPContentSearch`. +This directory records historical architecture decisions for +`MCPContentSearch`. -Harness planning reads this index first, then opens only accepted ADRs directly related to the requested change. Review gates treat accepted ADRs as contracts. If a diff conflicts with an accepted ADR, either change the diff or add a new ADR that supersedes the old decision. +For current work, the maintained source of truth is +`.agents/docs/architecture.md`. Harness planning and review do not require +reading ADRs. Keep this directory as optional historical context for older plan +docs, superseded decisions, or cases where a reader explicitly wants the +original decision record. + +If an ADR and `.agents/docs/architecture.md` disagree, follow the architecture +doc and update the historical ADR wording only when that extra archival cleanup +is worth doing. ## ADR Format -Use `template.md` for new records. +Use `template.md` only when you explicitly want to preserve a standalone +decision record in this archive. Required fields: @@ -35,7 +45,10 @@ File names should be numbered and descriptive: ## When to Add or Update ADRs -Add or update an ADR for: +Add or update an ADR only when preserving the separate historical decision +record is itself useful. + +Typical cases: - New cross-module architecture patterns. - MCP tool contract strategy changes. @@ -43,6 +56,8 @@ Add or update an ADR for: - Chroma data migration or reindexing policy. - External integration strategy changes. - Configuration/secrets policy changes. -- Intentional departures from `.agents/docs/architecture.md`. +- Architecture changes worth preserving historically after updating + `.agents/docs/architecture.md`. -Do not add ADRs for ordinary local refactors, one-off bug fixes, or implementation details that do not constrain future work. +Do not add ADRs for ordinary local refactors, one-off bug fixes, or +implementation details that do not constrain future work. diff --git a/.agents/docs/architecture.md b/.agents/docs/architecture.md index 4b8d95d..49b23ea 100644 --- a/.agents/docs/architecture.md +++ b/.agents/docs/architecture.md @@ -4,11 +4,8 @@ This document maps the current slim `MCPContentSearch` architecture. Harness planning and review use it to keep changes inside the focused MCP retrieval -scope and to catch contract or data-safety regressions. - -Decision history is indexed in `.agents/docs/adr/README.md`. ADR 0006 is the -current scope decision for the slim MCP core and supersedes the website/docs -portion of ADR 0004 plus ADR 0005's Auto Wiki decision for current work. +scope and to catch contract or data-safety regressions. It is the single +maintained design reference beyond the README. ## Runtime Structure @@ -40,6 +37,35 @@ The current architecture does not include a production Web Console, Auto Wiki generation, generic website/docs crawling, dynamic web fallback, or legacy live-search/indexing MCP tools. +## Core Mental Model + +The shortest accurate description of the current system is: + +```text +configured source sync +-> normalized document identity and content hashes +-> deterministic chunking +-> Chroma semantic retrieval candidates +-> SQLite active-document validation +-> chunk evidence, grouped document browsing, or citation-gated answer helpers +``` + +Keep these design assumptions aligned with implementation: + +- Source sync is the only supported ingestion entrypoint for retained sources. +- SQLite is the lifecycle source of truth for source status, sync jobs, + document/chunk activity, and citation-safe evidence gating. +- Chroma is the retrieval accelerator, not the final authority on whether a hit + is still active or citeable. +- `search_context` is the primary chunk-level evidence surface. +- `search_documents` is the grouped browsing surface built from the same + validated retrieval path. +- `answer_with_citations` is a helper answer surface built on top of validated + evidence, not a separate retrieval stack. +- Search query rewrite is optional, disabled by default, and any egress it + performs is limited to query rewriting rather than source fetching or data + mutation. + ## Data Flow ```text @@ -56,6 +82,33 @@ list_sources / get_sync_status -> MetadataStore SQLite source/job metadata ``` +Reviewer-facing source/status fields should remain understandable from the +maintained docs. Current status payloads are expected to center on fields such +as: + +```text +latest_success_at +latest_failure_at +document_count +chunk_count +latest_failure_reason +stale_cleanup_disabled_reason +``` + +Those fields explain recent source health, retained indexed volume, and whether +cleanup is intentionally disabled for safety. + +Running-job ownership is part of that status story. A source reports as +effectively blocked when SQLite still sees an active sync owner/heartbeat for +that source, which prevents overlapping syncs from starting. + +That blocked state is intentionally recoverable rather than permanent. Recovery +distinguishes stale jobs, unowned-job grace, dead owners, and the container +PID-reuse case where an old and new container can both appear as PID `1`. +During that same-PID edge case, reclaim falls back to the running job's own +heartbeat staleness window instead of reclaiming immediately from a transient +owner mismatch. + Source sync flow: ```text @@ -69,6 +122,25 @@ sync_source -> MetadataStore SQLite source/job/document/chunk/tombstone metadata ``` +Retained sync safety rule: + +- Tombstoning stale documents is allowed only for cleanup-capable sources after + a complete successful snapshot. Failed or incomplete syncs must not tombstone + documents simply because they were absent from a partial fetch. + +Bulk source sync flow: + +```text +sync_all + -> enumerate retained configured sources + -> start one sync_source task per source + -> preserve per-source running-job guards in SQLite + -> aggregate per-source results as succeeded, failed, blocked, or skipped + -> return completed only when results are succeeded/skipped only + -> return partial for mixed success/skip/failure/block combinations + -> return failed when every source failed or was blocked by failure conditions +``` + Retrieval and answer flow: ```text @@ -126,6 +198,101 @@ New behavior should start in the module that owns the relevant responsibility. Avoid adding cross-module shortcuts in `api/tools.py` when a service boundary is more appropriate. +## Incremental Indexing and Tombstone Safety + +The retained ingestion model depends on stable document identity plus cautious +cleanup: + +- Source connectors should preserve stable document identity fields such as + source-specific external ids, canonical URLs, and version or freshness + markers when available. +- Indexing compares content hashes and chunk ids so unchanged documents can skip + unnecessary vector rewrites. +- Reappeared or reactivated documents should return to the active set through a + normal successful sync rather than through ad hoc metadata repair. +- Tombstone and stale-cleanup behavior is safety-gated. Missing documents may be + marked stale only after a cleanup-capable source completes a full successful + snapshot. Failed, partial, or byte/file-limit-truncated snapshots must not + tombstone documents simply because they were absent from that incomplete run. + +## Source Identity and Chunking Model + +The retained indexing model distinguishes document management from chunk +retrieval: + +- `DocumentModel` is the sync and lifecycle unit. +- Chunks are the search and citation unit. +- Chunk metadata carries the reviewer-visible citation context used by + `search_context`, `search_documents`, and `answer_with_citations`. + +Stable identity and version expectations stay source-aware: + +- Notion: page id drives stable identity. +- Tistory: `blog_name:post_id` drives stable identity. +- GitHub: repository path drives stable identity, while blob SHA is revision + metadata. +- Obsidian: relative note path drives stable identity, while the + `obsidian://open` URL stays the citation-friendly canonical URL. + +The lifecycle fields that matter for reviewer understanding are: + +```text +external_id +document_id +canonical_url +version_id +last_seen_at +last_seen_sync_id +deleted_at +``` + +Current chunking remains deterministic and source-aware: + +- heading-based markdown chunking when structure exists +- deterministic plain-text fallback windows when structure does not +- line-range-preserving code chunking for citeable code evidence + +Representative citation metadata per chunk should remain understandable from the +maintained docs: + +```text +chunk_id +document_id +source_id +title +url +path +chunk_index +line_start +line_end +content_hash +version_id +updated_at +``` + +That deterministic chunking plus stable identity is what makes unchanged-doc +skip behavior, reappeared-document recovery, and citation stability predictable +across syncs. + +## Four-Layer View + +ContextWiki is easiest to reason about as four layers: + +```text +MCP client +-> FastMCP tool surface +-> ingestion/search services +-> SQLite metadata plus Chroma retrieval storage +``` + +That division is intentional: + +- MCP clients interact with tools, not storage internals. +- Service boundaries own ingestion, retrieval, ranking, and answer assembly. +- Chroma finds semantically relevant candidates. +- SQLite decides whether those candidates are still active, valid, and safe to + return as evidence. + ## MCP Tool Contract Current tools: @@ -142,6 +309,13 @@ Current tools: Contract intent: - `search_context` remains the chunk-level evidence and citation surface. +- `sync_all` is an aggregate orchestration helper, not a separate ingestion + stack. It fans out retained-source `sync_source` runs concurrently, preserves + each source's existing running-job guard, and reports mixed source outcomes + truthfully instead of pretending the whole batch succeeded when one source was + blocked or failed. Disabled or unconfigured sources may surface as `skipped`, + and the top-level batch status remains `completed` only when the aggregate + outcomes are limited to `succeeded` and `skipped`. - `search_documents` is additive and document-oriented: it uses the same retained-source retrieval path but returns one representative chunk-backed row per document for browsing. @@ -156,6 +330,46 @@ Contract intent: `include_debug=False`. - `answer_with_citations` keeps `include_debug` as a true opt-in debug surface and does not mirror the `no_matching_sources` exception path. +- Retrieval policy keeps vector retrieval, metadata fallback, and rerank/debug + reporting as distinct concerns. Query rewrite, fallback candidate addition, + and final SQLite validation should stay inspectable without blurring them into + one opaque score. +- When query rewrite debug is included, the caller-visible explanation should + keep the current public fields aligned with behavior: `rewrite_enabled`, + `rewrite_attempted`, `rewrite_applied`, and `rewrite_skipped_reason` explain + whether rewrite was disabled, skipped, attempted but unused, or actually + applied before retrieval. +- The top-level `rewrite_skipped_reason` field should stay coarse and + reviewer-readable. Current values explain state such as `disabled`, + `not_needed`, `rewrite_failed`, `no_matching_sources`, and `no_term_groups`. +- The nested `debug.query_rewrite.reason` field is the retrieval-pipeline + explanation vocabulary. Current stable values include + `no_initial_candidates`, `missing_textual_match`, + `insufficient_candidate_count`, and `low_initial_score`. +- A single strong exact-match candidate can also suppress rewrite even when the + caller asked for a larger `top_k`; that guardrail keeps clearly correct + direct hits from being rewritten unnecessarily. + +Retained debug-oriented answer inspection surfaces should stay documented and +stable enough for local evaluation and reviewer use: + +- `search_context` debug explains retrieval/rewrite decisions. +- Current reviewer-facing search debug commonly includes retrieval query and + result-selection surfaces such as `retrieval_queries`, + `rewritten_queries`, and `selected_results[]`. +- Deterministic intent policy should remain readable in debug output when + present. The current retained intent vocabulary includes `strict_lookup`, + `broad_topic`, `list`, and `comparison`, and that intent is reused by + ranking and grounded answer rendering. +- `answer_with_citations` exposes helper-answer inspection surfaces such as + `citations`, `used_chunks`, `debug`, and `debug_markdown` when the current + implementation returns them. +- Public debug payloads may also surface deterministic intent and retrieval + inspection sections such as `intent.*`, `retrieval_queries`, + `rewritten_queries`, and `selected_results[]` so reviewers can explain why a + grounded result set was chosen. +- Eval and reviewer workflows should be able to explain why a retrieval or + answer path was chosen without reading raw vector-store internals. When changing a tool: @@ -184,6 +398,14 @@ ContextWiki source/job/document/chunk lifecycle and citation metadata. SQLite is the authoritative active-document gate. Stale Chroma hits must be filtered through SQLite metadata before being returned as evidence. +GitHub stale cleanup remains repository-prefix scoped under the shared +`source_github` source id. A sync for one configured repository must not +tombstone documents that belong to another repository prefix. + +Soft-delete provenance matters here: SQLite tombstone metadata must remain able +to suppress stale managed vector hits even when best-effort vector cleanup +cannot remove every old Chroma candidate immediately. + ## External Services and Local Sources Current integrations and local configured sources: @@ -222,8 +444,8 @@ print credentials or local path details. - `environments/token.py`, `.env`, shell environment variables, and API keys are sensitive. - Do not add secret values to docs, tests, logs, screenshots, or examples. -- If a configuration default changes long-term behavior, update architecture - docs or ADRs in the same work item. +- If a configuration default changes long-term behavior, update this document in + the same work item. ## Error Handling @@ -239,6 +461,24 @@ Domain exceptions live in `core/exceptions.py`. Use the smallest useful check first. +The maintained verification model is layered: + +- docs-only verification for README, harness docs, plans, and other markdown + changes +- focused syntax, import, or targeted pytest checks for the directly changed + modules +- retained functional E2E coverage for MCP-visible sync/search/fetch/answer + workflows +- full-wrapper verification through `./scripts/verify_all.sh` when the work item + needs the repo's broader default gate instead of only a narrow focused check +- optional manual live smoke through `scripts/live_query_smoke.py` only when the + user explicitly approves real configured-source validation +- retained local eval surfaces such as `python scripts/run_contextwiki_eval.py` + when the change affects a quality-sensitive retrieval or answer surface that + already has modeled eval coverage +- retained eval or higher-level verification only when the change touches an + already-modeled quality-sensitive surface such as retrieval or answer quality + - Docs-only changes: path listing, `git status --short --branch`, `git diff --check`, then stage relevant docs-only files and run `git diff --cached --check` so new docs are covered. @@ -247,7 +487,9 @@ Use the smallest useful check first. - Unit/integration tests: `uv run --locked pytest -m "not live"` when the uv workspace is healthy. - MCP contract: focused tests around `register_tools` and retained tool - functions. + functions. The strongest public contract layer uses real + `FastMCP.call_tool(...)` payload checks rather than only internal helper + assertions. - Search/indexing/storage: temp Chroma path, temp SQLite path, or mock collection; avoid user data. - Fetching: mocked Notion/Tistory/GitHub responses and temporary Obsidian vaults; @@ -255,9 +497,19 @@ Use the smallest useful check first. - Functional E2E: `./scripts/verify_functional_e2e.sh`, which must cover retained MCP sync/search/fetch/answer paths, including grouped document browsing, without browser, wiki, live API, or LLM dependencies. +- Full wrapper: `./scripts/verify_all.sh`, which includes compile, lint, type, + non-live pytest, and the functional E2E gate when that broader default repo + verification is required. +- Manual live smoke: `python scripts/live_query_smoke.py`, only with explicit + approval because it can touch real configured sources or local user data. +- Retained eval runner: `PYTHONPATH=. python scripts/run_contextwiki_eval.py` + or the repo wrapper that invokes it, used when a work item changes retrieval + or answer quality on a modeled local eval surface. +- Deterministic reviewer-visible eval artifacts should stay separate from + optional runtime or latency metrics such as `runtime_metrics.json` so repeated + runs remain comparable. ## Harness Usage `harness-plan` must read this document before choosing implementation -boundaries. Review gates must check changed files against this architecture and -directly relevant accepted ADRs. +boundaries. Review gates must check changed files against this architecture. diff --git a/.agents/docs/github-workflow.md b/.agents/docs/github-workflow.md index d9a3cbd..3aa38ca 100644 --- a/.agents/docs/github-workflow.md +++ b/.agents/docs/github-workflow.md @@ -135,7 +135,9 @@ Stop monitoring when the PR is merged/closed, a blocker needs user judgment, or ## Release Policy -This repository currently uses `main` as the default branch. If a later workflow introduces `develop` or release branches, update this document and relevant ADRs before applying that workflow. +This repository currently uses `main` as the default branch. If a later +workflow introduces `develop` or release branches, update this document and the +maintained architecture doc before applying that workflow. ## Failure Handling diff --git a/.agents/docs/harness-engineering.md b/.agents/docs/harness-engineering.md index d02c289..6745a63 100644 --- a/.agents/docs/harness-engineering.md +++ b/.agents/docs/harness-engineering.md @@ -81,19 +81,19 @@ File-changing work must create or update a plan document after branch preflight - File name: `YYYY-MM-DD-short-task-name.md` - Required sections are listed in `docs/plan/README.md`. -- Include branch preflight result, scope/non-goals, acceptance criteria, expected files, verification plan, architecture/ADR constraints, risk/rollback notes, and progress log. +- Include branch preflight result, scope/non-goals, acceptance criteria, expected files, verification plan, architecture constraints, risk/rollback notes, and progress log. - If verification or review changes the plan, update the same plan document before continuing. - Final reports should include the plan document path. -## Architecture and ADR +## Architecture Planning must read: - `.agents/docs/architecture.md` -- `.agents/docs/adr/README.md` -- Directly relevant accepted ADRs only -Review gates must check that the diff does not violate architecture docs or accepted ADRs. If a change intentionally changes long-term architecture, add or update an ADR in the same work item. +Review gates must check that the diff does not violate the maintained +architecture doc. If a change intentionally changes long-term architecture, +update `.agents/docs/architecture.md` in the same work item. ## Multi-task Orchestration @@ -130,7 +130,7 @@ explicit. After branch preflight and plan creation, the main agent should: - Ask the user only for unsafe ambiguity, credentials, destructive operations, local data mutation, unavailable delegation/review tools, or external approval. Do not ask for routine implementation choices that can be decided - from repo docs, architecture, ADRs, and the plan. + from repo docs, architecture, and the plan. Implementation/execution worker subagents and `$subagent-review-loop` reviewer subagents are separate roles. Workers may edit within their assigned boundary @@ -143,7 +143,7 @@ Use this control loop: ```text read repository instructions read harness and GitHub workflow -read architecture and relevant ADRs +read architecture run branch preflight with GitHub workflow dirty/clean worktree safeguards write or update docs/plan plan run planning phase @@ -180,7 +180,7 @@ until complete or blocked ## Review Gates -Review gates use `$subagent-review-loop` and code-review stance. Each review pass must use exactly five newly spawned reviewer subagents, and the loop continues until all five reviewers in the newest pass report no actionable findings. Findings are prioritized by correctness, regressions, missing tests, data loss, security, MCP contract mismatch, async/concurrency issues, architecture/ADR violations, and change size. +Review gates use `$subagent-review-loop` and code-review stance. Each review pass must use exactly five newly spawned reviewer subagents, and the loop continues until all five reviewers in the newest pass report no actionable findings. Findings are prioritized by correctness, regressions, missing tests, data loss, security, MCP contract mismatch, async/concurrency issues, architecture violations, and change size. Use review lenses that fit the change: diff --git a/.agents/skills/harness-engineering/SKILL.md b/.agents/skills/harness-engineering/SKILL.md index 9241b9d..c2d7927 100644 --- a/.agents/skills/harness-engineering/SKILL.md +++ b/.agents/skills/harness-engineering/SKILL.md @@ -9,7 +9,7 @@ description: Orchestrates MCPContentSearch harness work for implementation, fixe Read `.agents/docs/harness-engineering.md` and `.agents/docs/github-workflow.md` first. For file-changing work, do not edit target files until branch preflight is complete and a `docs/plan/` plan exists. -During planning, read `.agents/docs/architecture.md` and `.agents/docs/adr/README.md`. Open only accepted ADRs directly related to the change. +During planning, read `.agents/docs/architecture.md`. ## Phases diff --git a/.agents/skills/harness-functional-smoke/SKILL.md b/.agents/skills/harness-functional-smoke/SKILL.md index cc6b212..5ca19dd 100644 --- a/.agents/skills/harness-functional-smoke/SKILL.md +++ b/.agents/skills/harness-functional-smoke/SKILL.md @@ -14,8 +14,7 @@ caller surfaces, not only through unit tests or helper functions. ## Inputs Read the current plan, local diff, `.agents/docs/harness-engineering.md`, -`.agents/docs/architecture.md`, `.agents/docs/functional-smoke-matrix.md`, -and directly relevant accepted ADRs. +`.agents/docs/architecture.md`, and `.agents/docs/functional-smoke-matrix.md`. ## Build The Matrix diff --git a/.agents/skills/harness-implement/SKILL.md b/.agents/skills/harness-implement/SKILL.md index 45eb87f..dd142ae 100644 --- a/.agents/skills/harness-implement/SKILL.md +++ b/.agents/skills/harness-implement/SKILL.md @@ -7,7 +7,7 @@ description: Implementation lane for scoped MCPContentSearch changes after plann ## Input -Read the current plan, `.agents/docs/harness-engineering.md`, `.agents/docs/github-workflow.md`, `.agents/docs/architecture.md`, relevant ADRs, and the production files in scope. +Read the current plan, `.agents/docs/harness-engineering.md`, `.agents/docs/github-workflow.md`, `.agents/docs/architecture.md`, and the production files in scope. ## Work diff --git a/.agents/skills/harness-plan/SKILL.md b/.agents/skills/harness-plan/SKILL.md index c1e6462..fb3258f 100644 --- a/.agents/skills/harness-plan/SKILL.md +++ b/.agents/skills/harness-plan/SKILL.md @@ -14,8 +14,6 @@ Read: - `.agents/docs/harness-engineering.md` - Current `docs/plan/...` plan document - `.agents/docs/architecture.md` -- `.agents/docs/adr/README.md` -- Directly relevant accepted ADRs - Minimal code or docs context needed for the work Read `.agents/docs/github-workflow.md` when branch, commit, push, PR, or release work is involved. @@ -42,7 +40,7 @@ The plan must include: Notion, Tistory, GitHub, optional search LLM query rewrite, or embedding providers, if any. - Risks, open questions, environment requirements, and rollback point. -- Architecture/ADR constraints. +- Architecture constraints. - PR split or stacked PR plan if PRs are requested. - Progress table with `Phase`, `Status`, `Summary`, and `Evidence`. diff --git a/.agents/skills/harness-review/SKILL.md b/.agents/skills/harness-review/SKILL.md index cd2b9c9..52cdc04 100644 --- a/.agents/skills/harness-review/SKILL.md +++ b/.agents/skills/harness-review/SKILL.md @@ -1,6 +1,6 @@ --- name: harness-review -description: Middle and final $subagent-review-loop gate for MCPContentSearch changes, focused on bugs, regressions, tests, contracts, data safety, secrets, and architecture/ADR compliance. +description: Middle and final $subagent-review-loop gate for MCPContentSearch changes, focused on bugs, regressions, tests, contracts, data safety, secrets, and architecture compliance. --- # Harness Review @@ -22,7 +22,7 @@ only when all five reviewers in the newest pass report no actionable findings. ## Input -Read the plan, local diff, `.agents/docs/harness-engineering.md`, `.agents/docs/architecture.md`, `.agents/docs/functional-smoke-matrix.md`, `.agents/docs/adr/README.md`, relevant accepted ADRs, verification history, functional smoke matrix/results, and changed files. +Read the plan, local diff, `.agents/docs/harness-engineering.md`, `.agents/docs/architecture.md`, `.agents/docs/functional-smoke-matrix.md`, verification history, functional smoke matrix/results, and changed files. ## Subagent Review Loop @@ -65,7 +65,7 @@ Produce a checklist: | Item | Result | Notes | | --- | --- | --- | -| Architecture/ADR compliance | pass/fail/n/a | Relevant violation or n/a reason | +| Architecture compliance | pass/fail/n/a | Relevant violation or n/a reason | | Acceptance criteria | pass/fail/n/a | Missing behavior | | Tests/verification | pass/fail/n/a | Commands run or gaps | | Functional smoke matrix | pass/fail/n/a | Rows covered, blocked/gated checks, substitutes, and evidence | diff --git a/AGENTS.md b/AGENTS.md index 21d885d..4cb518b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -3,7 +3,7 @@ ## Project Harness - When the user asks to implement, add, fix, refactor, or test behavior, first read `.agents/docs/harness-engineering.md`, then follow `.agents/skills/harness-engineering/SKILL.md`. -- Harness planning and review must read `.agents/docs/architecture.md` and `.agents/docs/adr/README.md`. Read only accepted ADRs that directly affect the requested change. +- Harness planning and review must read `.agents/docs/architecture.md`. - Harness phase skills live under `.agents/skills/`: `harness-plan`, `harness-multitask`, `harness-implement`, `harness-test`, `harness-functional-smoke`, `harness-review`, `harness-refactor`, and `harness-integrate`. - Branch, commit, push, PR, and PR-watch policy is defined in `.agents/docs/github-workflow.md`. - File-changing work starts with branch preflight from the latest `main`: if the worktree is clean, switch to `main`, fast-forward it from `origin/main` when network is available, delete only safe local non-`main` work branches using `.agents/docs/github-workflow.md` safeguards, then create a fresh `feature/...` branch before target edits. @@ -14,7 +14,7 @@ - The main agent may implement directly only when the change is truly atomic. Record the reason in the plan progress log before editing target files. Shared-file overlap is not a reason to bypass workers: for non-atomic work, use a single-owner worker or sequential worker handoff instead of parallel edits. If subagent tools are unavailable for non-atomic work, or no safe worker boundary can be created, stop before target edits and ask the user for explicit approval before bypassing worker orchestration. Do not silently collapse worker orchestration into self-implementation. - Worker subagents and reviewer subagents are different roles. Workers may edit only inside their assigned boundary and must not commit, push, open PRs, inspect secrets, print secret values, inspect local Chroma/SQLite data, or mutate user data. Local Chroma/SQLite inspection or user-data mutation requires explicit user approval plus plan rationale, bounded instructions, and rollback/safety notes; secret values are not delegable. `$subagent-review-loop` reviewers are read-only and run only after verification and functional smoke. - The main agent owns integration: collect worker outputs, inspect diffs, resolve conflicts, update the plan, run verification, route actionable findings back to the responsible worker persona or a fresh replacement with the same ownership boundary, and minimize human intervention. Ask the user only when safety, credentials, destructive actions, unavailable delegation/review tools, or genuinely unclear requirements require human judgment. -- Keep `docs/contextwiki-core-understanding.md` updated when changes affect ContextWiki source connectors, source sync, document identity, chunking, tombstones, retrieval, citation metadata, or answer behavior. This note is the maintained human explanation layer and should not drift behind README, architecture docs, ADRs, or implementation. +- Keep `.agents/docs/architecture.md` updated when changes affect ContextWiki source connectors, source sync, document identity, chunking, tombstones, retrieval, citation metadata, answer behavior, or other maintained design assumptions. This document is the maintained human explanation layer and should not drift behind README or implementation. - After code-changing work, always run the relevant test or verification command before review. Use the repo-local commands in this file and `.agents/docs/harness-engineering.md`, not plugin-default paths such as `docs/superpowers/...`. - After implementation and focused tests, run the functional smoke gate before `$subagent-review-loop`: use `.agents/skills/harness-functional-smoke/SKILL.md` to exercise the task-relevant MCP/source-sync/user-visible feature inventory once through the safest real caller surfaces, not only unit-test the changed files. Record explicit safety blockers, approval needed, and nearest fake/temp substitutes in the plan. - After verification and before PR delivery for any code, configuration, documentation, or skill change, run `$subagent-review-loop`: spawn exactly five fresh reviewer subagents per pass and repeat until all five reviewers in the newest pass report no actionable findings. If subagent review is unavailable, stop and report the blocker instead of silently replacing it with self-review. @@ -32,13 +32,12 @@ LlamaIndex, ChromaDB, and SQLite metadata storage. - `api/`: MCP tool registration and tool handlers. - `core/`: shared models, exceptions, and utility code. - `environments/`: runtime configuration and secret/environment loading. -- `fetching/`: Notion, Tistory, and GitHub source connectors. +- `fetching/`: Notion, Tistory, GitHub, and Obsidian source connectors. - `indexing/`: document conversion, chunking, dedup/update detection, and vector indexing. - `search/`: ContextWiki retrieval, ranking, SQLite-backed active gates, query rewrite, and citation answer scaffolding. - `storage/`: SQLite source/job/document/chunk lifecycle metadata and active retrieval checks. -- `docs/contextwiki-core-understanding.md`: maintained learning note for the current ContextWiki data flow, source connector behavior, lifecycle metadata, retrieval gate, and limitations. - `docs/plan/`: plan documents written before file-changing harness work. -- `.agents/`: local harness docs, phase skills, and ADRs. +- `.agents/`: local harness docs and phase skills. ## Development Commands diff --git a/README.md b/README.md index 558b242..c32e4f3 100644 --- a/README.md +++ b/README.md @@ -2,25 +2,13 @@ [![CI](https://github.com/eunhwa99/MCPContentSearch/actions/workflows/ci.yml/badge.svg)](https://github.com/eunhwa99/MCPContentSearch/actions/workflows/ci.yml) -ContextWiki is a focused MCP retrieval server for private or project-specific -knowledge. It syncs a small set of configured sources into local Chroma vectors, -stores lifecycle and citation metadata in SQLite, and exposes MCP tools that let -an LLM client retrieve grounded context without reading local databases -directly. - -## πŸš€ Key Features - -- πŸ”Œ **Multi-source connectors**: Notion, Tistory, GitHub, and local Obsidian -- βš–οΈ **Hybrid retrieval architecture**: Chroma for semantic search, SQLite for - lifecycle and citation validation -- πŸ› οΈ **Practical MCP tools**: source listing, sync, status, search, fetch, and - a helper citation-backed answer preview surface -- πŸ›‘οΈ **Citation safety**: only SQLite-validated chunks are returned as evidence -- πŸ”’ **Local-first by default**: optional query rewrite stays off unless you - enable it explicitly, but fully non-egress operation still depends on your - embedding provider choice - -## πŸ—οΈ Architecture Overview +A **private knowledge retrieval MCP server** for LLM clients. It syncs +configured sources from Notion, Tistory, GitHub, and Obsidian into local +vector and metadata stores, then returns verified, citation-backed context. + +--- + +## Architecture ```text [ Sources ] @@ -31,404 +19,427 @@ directly. / \ v v [ Chroma ] [ SQLite ] - semantic hits metadata gate + semantic search metadata gate \ / - \ / - v v + v v [ Verified Context ] ``` -## πŸ› οΈ MCP Tools - -Tool handlers live in `api/tools.py`. Business logic stays in `fetching/`, -`indexing/`, `search/`, and `storage/`. - -| Tool | Purpose | -| --- | --- | -| `list_sources()` | List configured Notion, Tistory, GitHub, and Obsidian sources. | -| `sync_source(source_id)` | Sync one configured source into SQLite metadata and Chroma vectors. | -| `sync_all()` | Sync all retained sources concurrently and return aggregate results. | -| `get_sync_status(source_id="")` | Read latest source and sync-job state. | -| `search_context(query, filters=None, top_k=10, include_debug=False)` | Return structured evidence chunks after Chroma retrieval, metadata fallback when needed, and SQLite validation. | -| `search_documents(query, filters=None, top_k=10)` | Return one representative, retrieval-ranked row per matching document. | -| `fetch_context(document_id="", chunk_id="")` | Fetch a document or chunk directly from SQLite metadata. | -| `answer_with_citations(question, filters=None, top_k=5, include_debug=False)` | Build a helper evidence-gated answer preview with citations and used chunks by reusing the `search_context` retrieval path. | - -At a glance: - -- `sync_all()` syncs all retained sources in one pass. -- `sync_all()` reports aggregate `status` truthfully: `completed` for - succeed/skip-only runs, `partial` when success/skip is mixed with blocked or - failed sources, and `failed` when nothing completed successfully. -- `sync_all()` summary counts include `succeeded`, `failed`, `blocked`, and - `skipped`. Disabled sources are counted as `skipped` during bulk sync. -- `search_context(...)` always returns a `debug` key. On the default normal - path, `include_debug=False` returns `debug={}`, while `include_debug=True` - returns populated structured debug details. -- `answer_with_citations(..., include_debug=True)` exposes helper-surface debug - details through the same retrieval path. -- Today `search_context` also returns a small populated `debug` object when - `include_debug=False` if the public source filter leaves no matching sources, - including `debug.rewrite_skipped_reason=no_matching_sources`. In other words, - the normal default-path `debug` payload is empty, and only that fast path - returns populated debug data without `include_debug=True`. - `answer_with_citations` does not have that exception and exposes debug only - when `include_debug=True`. - -Production MCP clients usually use `search_context` or `search_documents` to -collect grounded evidence, then let a downstream LLM generate the final -natural-language answer. In this repo, `answer_with_citations` is intentionally -positioned as an optional helper surface for preview, debug, evaluation, and -developer inspection rather than the main production answer API. - -When validating `answer_with_citations`, inspect: - -- `citations` to confirm every visible claim points to expected evidence -- `used_chunks` to confirm the helper output stayed grounded in retrieved chunks -- `debug` to inspect retrieval/grounding state when `include_debug=True` -- `debug_markdown` to review the human-readable retrieval-to-answer trace - -## βš™οΈ Configuration - -### 1. Source connectors - -| Source | Source id | Configuration | Notes | -| --- | --- | --- | --- | -| Notion | `source_notion` | `NOTION_API_KEY` | Syncs pages/documents through the Notion fetcher. | -| Tistory | `source_tistory` | `TISTORY_BLOG_NAME` | Syncs blog posts through the Tistory fetcher. | -| GitHub | `source_github` | `CONTEXTWIKI_GITHUB_REPOSITORIES`, `CONTEXTWIKI_GITHUB_DEFAULT_REF`, `CONTEXTWIKI_GITHUB_MAX_FILES`, `CONTEXTWIKI_GITHUB_MAX_FILE_BYTES`, optional `GITHUB_TOKEN`, optional `CONTEXTWIKI_GITHUB_USER_AGENT` | Syncs bounded text/code/Markdown files from configured repositories. | -| Obsidian | `source_obsidian` | `CONTEXTWIKI_OBSIDIAN_VAULT_PATH`, `CONTEXTWIKI_OBSIDIAN_MAX_FILES`, `CONTEXTWIKI_OBSIDIAN_MAX_FILE_BYTES` | Syncs bounded Markdown notes from a configured local vault. | - -GitHub repositories are configured as comma-separated `owner/repo` entries with -an optional `@ref`. If `@ref` is omitted, ContextWiki uses -`CONTEXTWIKI_GITHUB_DEFAULT_REF` and defaults that env var to `main`. +- **Chroma** handles semantic retrieval. +- **SQLite** tracks document lifecycle and citation validity. +- Only chunks still active in SQLite are returned as evidence. -```bash -CONTEXTWIKI_GITHUB_REPOSITORIES="eunhwa99/MCPContentSearch@main" -CONTEXTWIKI_GITHUB_DEFAULT_REF=main -CONTEXTWIKI_GITHUB_MAX_FILES=200 -CONTEXTWIKI_GITHUB_MAX_FILE_BYTES=512000 -CONTEXTWIKI_GITHUB_USER_AGENT="ContextWikiBot/0.1 (+https://github.com/eunhwa99/MCPContentSearch)" -GITHUB_TOKEN=... -``` +--- -Notes: +## MCP Tools -- `GITHUB_TOKEN` is optional. Unauthenticated GitHub API access depends on the - target repository being visible without auth and is subject to lower rate - limits. -- `CONTEXTWIKI_GITHUB_MAX_FILES` and `CONTEXTWIKI_GITHUB_MAX_FILE_BYTES` - control fetch completeness. Exceeding those bounds means the connector does - not claim a complete repository snapshot for stale cleanup. -- `CONTEXTWIKI_GITHUB_USER_AGENT` is the HTTP `User-Agent` header knob used by - the GitHub fetcher. +| Tool | Description | +|------|-------------| +| `list_sources()` | List configured sources. | +| `sync_source(source_id)` | Sync a specific source. | +| `sync_all()` | Sync all configured sources in one pass. | +| `get_sync_status(source_id="")` | Get source and sync job status. | +| `search_context(query, ...)` | Semantic search with SQLite-validated chunks. | +| `search_documents(query, ...)` | Search results grouped by document. | +| `fetch_context(document_id, chunk_id)` | Fetch a specific document or chunk directly. | +| `answer_with_citations(question, ...)` | Citation-backed answer preview for preview/debug/eval use. | -```bash -CONTEXTWIKI_OBSIDIAN_VAULT_PATH="/path/to/temp-or-real-vault" -CONTEXTWIKI_OBSIDIAN_MAX_FILES=2000 -CONTEXTWIKI_OBSIDIAN_MAX_FILE_BYTES=512000 -``` +> In production, use `search_context` or `search_documents` to gather grounded +> evidence, then let a downstream LLM generate the final answer. -Raw secrets are read at runtime from environment variables. They are not stored -in SQLite, committed to docs/tests, or printed by verification commands. +When validating `answer_with_citations`, inspect `citations`, `used_chunks`, +and `debug` or `debug_markdown` when enabled. -### 2. Optional search query rewrite +--- -`search_context` can optionally ask an external LLM for short query rewrites -when initial local retrieval looks weak. `answer_with_citations` inherits the -same rewrite egress because its answer flow calls the `search_context` -retrieval path. This is disabled by default. +## Quick Start -```bash -CONTEXTWIKI_SEARCH_LLM_ENABLED=true -CONTEXTWIKI_SEARCH_LLM_PROVIDER=openai -CONTEXTWIKI_SEARCH_LLM_MODEL=gpt-4.1-mini -OPENAI_API_KEY=... -``` - -The default is `off`. When enabled, the search query and normalized query term -groups may be sent to an external LLM. This setting does not fetch source -content or mutate SQLite/Chroma. - -## ⚑ Reproducible Launch Paths - -### 1. Fresh-machine local launch +**Prerequisites:** Python `3.13`, [`uv`](https://docs.astral.sh/uv/) -Prerequisites: - -- Python `3.13` -- [`uv`](https://docs.astral.sh/uv/) - -Install dependencies, create a local env file, and start the slim MCP core: +Use this path if you want to run ContextWiki directly with local Python and +`uv`. ```bash -uv sync --locked --python 3.13 --dev +uv sync --locked cp .env.example .env uv run --locked python main.py ``` -Notes: - -- `environments/token.py` loads `.env`, so `cp .env.example .env` is the - intended local setup path. -- Leaving optional source env vars blank is valid; those sources stay disabled - for future syncs until configured. -- A disabled source does not automatically hide already indexed content from - retrieval. Existing SQLite-active documents remain retrievable until a later - cleanup or metadata change removes them. -- The packaged runtime is not keyless today. For non-demo sync/search runs, the - default embedding path typically requires `OPENAI_API_KEY` even if query - rewrite stays disabled. -- If you want a public GitHub example after first launch, set - `CONTEXTWIKI_GITHUB_REPOSITORIES=eunhwa99/MCPContentSearch@main` manually in - `.env`. -- Default local SQLite/Chroma state is created under - `~/.mcp_content_search/`. - -For a plain syntax check without contacting external services: +**Syntax check only (no external services):** ```bash python -m compileall api core environments fetching indexing search storage main.py ``` -### 2. Container launch +### Docker -Build the image: +For Docker-only usage, you need Docker Desktop or Docker Engine instead of the +local Python + `uv` setup above. -```bash -docker build -t contextwiki . -``` +The command below is the minimum Docker run example. It is enough if you are +not using the Obsidian source. -Prepare the runtime env file before starting the container: +If your `.env` already contains a host path such as +`CONTEXTWIKI_OBSIDIAN_VAULT_PATH=/absolute/path/to/your/vault`, remove it or +override it before using this minimum Docker path unless you are also mounting +that vault into the container. ```bash +docker build -t contextwiki . cp .env.example .env -``` - -For a real sync/search run, edit `.env` and set an embedding-provider key first. -With the current default runtime, that usually means setting `OPENAI_API_KEY`. - -Start the MCP server in a container: - -```bash -docker run --rm -it \ +docker run --rm -i \ --env-file .env \ -v contextwiki_data:/home/appuser/.mcp_content_search \ contextwiki ``` -This named volume avoids first-run host-permission issues for reviewers. +Because ContextWiki is a stdio MCP server rather than a long-running HTTP API, +`docker run -d ...` is not the normal integration path. -If you want to expose a real Obsidian vault to the container, mount it -read-only and point `CONTEXTWIKI_OBSIDIAN_VAULT_PATH` at the container path: +If you also want `source_obsidian` inside Docker, add a vault mount and point +`CONTEXTWIKI_OBSIDIAN_VAULT_PATH` at the container path: ```bash -docker run --rm -it \ +docker run --rm -i \ --env-file .env \ -v contextwiki_data:/home/appuser/.mcp_content_search \ - -v "/absolute/path/to/vault:/vault:ro" \ - -e CONTEXTWIKI_OBSIDIAN_VAULT_PATH=/vault \ + -v "/absolute/path/to/your/vault:/vault:ro" \ contextwiki ``` -Supported limitations: +Then set this in `.env` for the Docker run: -- This image runs the retained slim MCP core only. -- It does not add deployment automation, multi-service orchestration, or secret - management beyond `--env-file`. -- Persisted runtime data lives inside `/home/appuser/.mcp_content_search` unless you - mount that path. +```bash +CONTEXTWIKI_OBSIDIAN_VAULT_PATH=/vault +``` -## 🚦 Verification +--- -Verification is split into four explicit layers so local runs and CI describe -the same trust boundary: +## Configuration -- `Public MCP contract layer`: real `FastMCP.call_tool(...)` coverage for each retained public tool. -- `Deterministic functional E2E layer`: retained sync/search/fetch/answer flows over temporary local state. -- `Deterministic quality eval layer`: fixture-based retrieval and answer scoring plus reviewer-visible artifacts. -- `Manual live smoke layer`: local configured-runtime checks for human debugging only, not automated portfolio assurance. +### Source Connectors -No retained automated pytest currently uses the `live` marker. +| Source | Source ID | Required Env Var | +|--------|-----------|------------------| +| Notion | `source_notion` | `NOTION_API_KEY` | +| Tistory | `source_tistory` | `TISTORY_BLOG_NAME` | +| GitHub | `source_github` | `CONTEXTWIKI_GITHUB_REPOSITORIES` | +| Obsidian | `source_obsidian` | `CONTEXTWIKI_OBSIDIAN_VAULT_PATH` | -### 1. Full gate +**GitHub example** ```bash -./scripts/verify_all.sh +CONTEXTWIKI_GITHUB_REPOSITORIES=eunhwa99/MCPContentSearch@main +CONTEXTWIKI_GITHUB_DEFAULT_REF=main +CONTEXTWIKI_GITHUB_MAX_FILES=200 +CONTEXTWIKI_GITHUB_MAX_FILE_BYTES=512000 +CONTEXTWIKI_GITHUB_USER_AGENT=ContextWikiBot/0.1 (+https://github.com/eunhwa99/MCPContentSearch) +GITHUB_TOKEN=... ``` -Includes syntax checks, Ruff, mypy, Bandit, the public contract layer, the -broader non-live pytest regression run, deterministic eval artifacts, and the -functional E2E gate. The full gate defaults `IS_TESTING=1` to mirror the -current CI env shape for non-live verification commands, even though the -current retained suite does not branch on that variable. +Important GitHub notes: + +- Use `owner/repo` or `owner/repo@ref`. +- Do not wrap `CONTEXTWIKI_GITHUB_REPOSITORIES` in quotes in `.env`. +- `GITHUB_TOKEN` is optional, but unauthenticated access is more rate-limited. -### 2. Public MCP contract layer +**Tistory example** ```bash -uv run --locked pytest -q tests/contracts/test_public_mcp_contracts.py tests/test_app_composition.py +TISTORY_BLOG_NAME=devlog ``` -This verifies that each retained public MCP tool has at least one real -`FastMCP.call_tool(...)` contract path with JSON payload assertions. +Use the blog subdomain only, not the full URL. For example, use `devlog`, not +`https://devlog.tistory.com`. -### 3. Deterministic functional E2E layer +**Obsidian example** ```bash -./scripts/verify_functional_e2e.sh +CONTEXTWIKI_OBSIDIAN_VAULT_PATH=/absolute/path/to/your/vault +CONTEXTWIKI_OBSIDIAN_MAX_FILES=2000 +CONTEXTWIKI_OBSIDIAN_MAX_FILE_BYTES=512000 ``` -This covers retained MCP flows with non-live tests and temporary storage. The -script now stays on the retained end-to-end flow modules instead of also -re-running broader contract/service suites. It also defaults `IS_TESTING=1` to -mirror the current CI env shape for non-live verification commands. +Use the vault root path, not an individual `.md` file path. + +### Search Query Rewrite (Optional) -### 4. Deterministic quality eval layer +Rewrites weak queries via an external LLM. Disabled by default. ```bash -uv run --locked pytest -q tests/evals -uv run --locked python scripts/run_contextwiki_eval.py --output-dir artifacts/contextwiki-evals -uv run --locked python scripts/run_contextwiki_eval.py --output-dir artifacts/contextwiki-evals --include-latency +CONTEXTWIKI_SEARCH_LLM_ENABLED=true +CONTEXTWIKI_SEARCH_LLM_PROVIDER=openai +CONTEXTWIKI_SEARCH_LLM_MODEL=gpt-4.1-mini +OPENAI_API_KEY=... ``` -`scripts/run_contextwiki_eval.py` is the deterministic reviewer-evidence runner. -It seeds temporary SQLite fixture data, swaps in fixture retrieval behavior -instead of the normal live indexing/vector setup, exercises retained retrieval -and grounded-answer scoring paths, and writes deterministic JSON artifacts -with: +> The default embedding path also usually requires `OPENAI_API_KEY` for real +> sync and search runs. + +--- + +## Usage + +ContextWiki runs as a local stdio MCP server. Your MCP client spawns the +server process and communicates with it over stdin/stdout. + +### Recommended: Claude Desktop via local `uv` + +This is the easiest setup path on macOS because: + +- Claude Desktop can spawn the server directly. +- ContextWiki can load `.env` at startup when launched from the repo root or + via `uv --directory ...`. +- Obsidian can use your real host vault path directly. +- You do not need Docker mounts for the vault. + +Do this in order: -- group-level mixed-query metrics -- suite pass/fail summaries +1. Create `.env` in your repo root. +2. Put your real values there. +3. Add the MCP entry below to Claude Desktop. +4. Fully restart Claude Desktop. +5. In a fresh Claude Desktop chat, ask it to call `list_sources()`. -When `--include-latency` is supplied, it also writes an informational -`runtime_metrics.json` file with retrieval/answer latency summaries. CI uploads -the deterministic JSON files as the `contextwiki-evals` artifact, while the -timing file is produced only by opt-in local latency runs. +Example `.env`: -### 5. Focused checks +```bash +OPENAI_API_KEY=... +NOTION_API_KEY=... +TISTORY_BLOG_NAME=devlog +CONTEXTWIKI_GITHUB_REPOSITORIES=eunhwa99/MCPContentSearch +CONTEXTWIKI_GITHUB_DEFAULT_REF=main +CONTEXTWIKI_OBSIDIAN_VAULT_PATH=/absolute/path/to/your/vault +``` + +On macOS, add this to +`~/Library/Application Support/Claude/claude_desktop_config.json`: + +```json +{ + "mcpServers": { + "content-search-server": { + "command": "/absolute/path/to/uv", + "args": [ + "--directory", + "/absolute/path/to/MCPContentSearch", + "run", + "--python", + "3.13", + "python", + "main.py" + ] + } + } +} +``` + +On macOS, find your `uv` path with: ```bash -uv run --locked pytest -q tests/fetching/test_connectors.py -uv run --locked pytest -q tests/api/test_tools_contract.py -uv run --locked pytest -q tests/contracts/test_public_mcp_contracts.py tests/test_app_composition.py -uv run --locked pytest -q tests/e2e/test_contextwiki_flow.py -uv run --locked pytest -q tests/search/test_context_service.py tests/search/test_answer_service.py -uv run --locked pytest -q tests/storage/test_metadata_store.py tests/indexing/test_ingestion_service.py -uv run --locked pytest -q tests/scripts/test_demo_public_flow.py -uv run --locked pytest -q tests/scripts/test_live_query_smoke.py -uv run --locked pytest -q tests/evals -uv run --locked python scripts/run_contextwiki_eval.py --output-dir artifacts/contextwiki-evals -uv run --locked python scripts/run_contextwiki_eval.py --output-dir artifacts/contextwiki-evals --include-latency +which uv ``` -`tests/scripts/test_live_query_smoke.py` only verifies the CLI contract, -redaction rules, and summary wording for the manual smoke script. It does not -turn the manual live smoke layer into automated live runtime verification. +You should not need to start `uv run python main.py` manually first. Claude +Desktop should launch the configured command automatically when it needs the +MCP server. -## 🎬 One-command Demo +### Claude Desktop via Docker -Run the retained slim tool-handler and service flow against the bundled public -sample vault: +Use this only if you specifically want the server to run inside Docker. +Build the image first so the `contextwiki:latest` tag exists locally: ```bash -./scripts/demo.sh +docker build -t contextwiki . +``` + +If you are not using Obsidian, you can omit the vault mount below. +If you are using Obsidian in Docker, keep both the mount and the `/vault` +environment setting shown after the JSON example. +If your `.env` still contains a host-only +`CONTEXTWIKI_OBSIDIAN_VAULT_PATH=/absolute/path/to/your/vault` from the local +`uv` setup, remove it or override it before using the no-Obsidian Docker +client path below. +On macOS, prefer the absolute Docker binary path from `which docker` if Claude +Desktop cannot find the `docker` command from its GUI environment. + +```json +{ + "mcpServers": { + "content-search-server": { + "command": "docker", + "args": [ + "run", + "--rm", + "-i", + "--env-file", + "/absolute/path/to/MCPContentSearch/.env", + "-v", + "contextwiki_data:/home/appuser/.mcp_content_search", + "contextwiki:latest" + ] + } + } +} +``` + +If you want Obsidian in the Docker-spawned client path, add this mount to the +JSON `args` list before `contextwiki:latest`: + +```json +"-v", +"/absolute/path/to/your/vault:/vault:ro" ``` -If you have not installed dependencies yet: +Then set this in `.env`: ```bash -uv sync --locked --python 3.13 --dev -./scripts/demo.sh +CONTEXTWIKI_OBSIDIAN_VAULT_PATH=/vault +``` + +Why both the mount and the env var matter: + +- `-v /absolute/path/to/your/vault:/vault:ro` exposes your host vault inside + the container. +- `CONTEXTWIKI_OBSIDIAN_VAULT_PATH=/vault` tells the app where that vault + lives inside the container. + +If you leave `CONTEXTWIKI_OBSIDIAN_VAULT_PATH` set to a host path like +`/Users/...` while using Docker, the container will not be able to read it. + +### Cursor + +Add to `.cursor/mcp.json` in your project root: + +```json +{ + "mcpServers": { + "content-search-server": { + "command": "/absolute/path/to/uv", + "args": [ + "--directory", + "/absolute/path/to/MCPContentSearch", + "run", + "--python", + "3.13", + "python", + "main.py" + ] + } + } +} ``` -What it does: +### After connecting + +1. Ask the AI to call `sync_all()` or `sync_source("source_notion")`. +2. Once synced, ask the AI to use `search_context()` or `search_documents()` + against your indexed sources. + +### Claude Desktop Client Workflow Example + +Example prompt: + +```text +find my projects about DynamoDB and organize it with STAR method. Answer in English +``` + +This screenshot shows a Claude Desktop client workflow that uses ContextWiki as +the retrieval backend. The final STAR-form prose is Claude output built on top +of retrieved notes, not a direct server-side answer format. + +![Claude Desktop using ContextWiki MCP as a retrieval backend before Claude composes the final STAR-style response](docs/images/claude-desktop-dynamodb-star-example.png) + +### Troubleshooting + +**If Claude cannot discover the MCP server** -1. Uses `sample_vault/` as a bounded public Obsidian source. -2. Syncs it through the retained `source_obsidian` connector. -3. By default, runs `search_context` with the canonical question shown in the transcript. -4. By default, runs `answer_with_citations` with that same canonical question. -5. Prints sync, search, and helper answer preview payloads. +- Recheck your Claude Desktop MCP config file. On macOS, the default path is + `~/Library/Application Support/Claude/claude_desktop_config.json`. +- Fully restart Claude Desktop after any config change. +- If needed, run the configured `command` and `args` manually outside Claude to + verify they start successfully. -This demo is non-live and: +**If Claude only works after you start the server manually** -- it uses temporary SQLite and Chroma storage -- it uses `MockEmbedding` instead of a live embedding provider -- it does not require Notion, Tistory, GitHub, or Obsidian credentials -- it forces `CONTEXTWIKI_SEARCH_LLM_ENABLED=false` even if your shell sets it -- it normalizes generated ids and timestamps so the transcript stays stable +- Claude Desktop is probably failing to launch the configured command. +- Test the exact `command` and `args` outside Claude first. -Use the answer step here as a grounded helper preview. In production MCP usage, -downstream LLMs usually turn the retrieved evidence into the final answer. -The default demo transcript is the canonical portfolio path: the same question -is used for retrieval and helper answer preview. -If you pass only `--query`, the demo reuses that same text for the answer step. -If you override both `--query` and `--question` with different values, read the -output as separate probes rather than one validated end-to-end chain. +**If GitHub source startup fails with `Invalid GitHub repository spec`** -Optional flags: +- Remove quotes from `CONTEXTWIKI_GITHUB_REPOSITORIES` in `.env`. +- Use `owner/repo` or `owner/repo@ref`, not a full GitHub URL. + +**If Obsidian works locally but not in Docker** + +- Make sure you both mounted the host vault and set + `CONTEXTWIKI_OBSIDIAN_VAULT_PATH=/vault`. +- The Docker mount path and the app env path are intentionally different. + +**If a source still appears disabled after config changes** + +- Fully restart the MCP client after updating config. +- In Claude Desktop, a chat refresh alone is often not enough. + +**If `sync_all()` takes too long** + +- Sync sources individually with `sync_source("source_notion")`, + `sync_source("source_github")`, and so on. +- GitHub sync can be slower for larger repositories. + +--- + +## Demo + +Run the full retained flow against the bundled sample vault: + +```bash +./scripts/demo.sh +``` + +- Uses temporary SQLite and Chroma state +- Uses `MockEmbedding` +- Requires no Notion, Tistory, GitHub, or Obsidian credentials ```bash -./scripts/demo.sh --json ./scripts/demo.sh --query "Why does ContextWiki validate citations through SQLite?" -./scripts/demo.sh --query "sqlite active evidence gate" --question "Why does ContextWiki validate citations through SQLite?" +./scripts/demo.sh --json ``` -## πŸ”Ž Manual Live Smoke Layer +--- -Run a real local retrieval check against your configured ContextWiki runtime: +## Verification + +If you only want a quick product-flow check, run: ```bash -uv run --locked python scripts/live_query_smoke.py --query "How does github sync work?" +./scripts/demo.sh ``` -This is the aligned same-input smoke path: when `--question` is omitted, the -script reuses the same text for retrieval and helper answer preview. If you -supply a different `--question`, the transcript explicitly labels the output as -separate probes so reviewers do not mistake it for one validated chain. - -Useful options: +If you changed code and want the main local verification gate, run: ```bash -uv run --locked python scripts/live_query_smoke.py --query "aws startup" -uv run --locked python scripts/live_query_smoke.py --query "aws startup" --question "How do I start EC2?" -uv run --locked python scripts/live_query_smoke.py --query "github sync" --source-id source_github --top-k 3 -uv run --locked python scripts/live_query_smoke.py --query "obsidian citation" --rewrite off -uv run --locked python scripts/live_query_smoke.py --query "obsidian citation" --json +./scripts/verify_all.sh ``` -Use this only after you have configured real sources in `.env`. The public demo -above is the safer first-run path for reviewers, and it is the only documented -path here that is intentionally keyless. - -All live-smoke output should be treated as local diagnostic data rather than -public sample content. The default text summary can still reveal local -source-derived titles, citation titles, and helper-answer text. `--json` removes -raw chunk text, previews, and direct `path`/`url` fields, but titles, -identifiers, and synthesized helper answer text may still reflect local source -content. - -This smoke is for retrieval and helper-surface validation. It is manual live -diagnostic evidence, not part of the automated portfolio gate. Check -`citations`, `used_chunks`, and, when using direct MCP calls with -`include_debug=True`, `debug` plus `debug_markdown`. - -## πŸ—ΊοΈ Project Map - -- `main.py`: FastMCP server composition -- `api/`: MCP-facing tool contracts and response formatting -- `core/`: shared models, exceptions, and utilities -- `environments/`: runtime configuration and secret/environment access -- `fetching/`: Notion, Tistory, GitHub, and Obsidian connectors/fetchers -- `indexing/`: chunking, deduplication, lifecycle coordination, and Chroma writes -- `search/`: ContextWiki retrieval, ranking, active metadata gates, and answers -- `storage/`: SQLite source/job/document/chunk lifecycle metadata -- `tests/`, `scripts/`: non-live verification harnesses and reviewer utilities - -## πŸ“– Additional Docs - -- [ContextWiki Core Understanding](docs/contextwiki-core-understanding.md) +--- + +## Project Structure + +```text +main.py FastMCP server entry point +api/ MCP tool handlers +core/ Shared models, exceptions, utilities +environments/ Env var and secret loading +fetching/ Source connectors (Notion, Tistory, GitHub, Obsidian) +indexing/ Chunking, deduplication, Chroma indexing +search/ Search, ranking, metadata gate, citation answer support +storage/ SQLite lifecycle management +tests/, scripts/ Verification harnesses and utilities +``` + +--- + +## Additional Docs + - [Architecture](.agents/docs/architecture.md) -- [ADRs](.agents/docs/adr/) -- [Harness engineering](.agents/docs/harness-engineering.md) -- [GitHub workflow policy](.agents/docs/github-workflow.md) -- [Plan log](docs/plan/) diff --git a/docs/contextwiki-core-understanding.md b/docs/contextwiki-core-understanding.md deleted file mode 100644 index 565d173..0000000 --- a/docs/contextwiki-core-understanding.md +++ /dev/null @@ -1,710 +0,0 @@ -# ContextWiki Core Understanding Note - -Baseline: - -- Original baseline: `eunhwa99/MCPContentSearch` PR #2 -- Current update: slim MCP core refactor with PR #25 Obsidian restoration, - retaining GitHub/Notion/Tistory/Obsidian source sync, SQLite lifecycle - metadata, Chroma retrieval, and citation-backed answers. - -Goal: - -This note is the maintained mental model for explaining ContextWiki's current -design intent, data flow, and limitations. When source connectors, ingestion, -lifecycle metadata, retrieval, citation, or answer behavior changes, update this -note together with README, architecture docs, ADRs, or plan docs as appropriate. - ---- - -## 0. One-line Summary - -ContextWiki is a focused MCP retrieval server. - -The core flow is: - -```text -source registration --> source sync --> document fetch from Notion, Tistory, GitHub, or Obsidian --> external_id / canonical_url / version_id / last_seen metadata normalize --> content_hash computation --> deterministic source-aware chunking for chunk-id comparison --> unchanged documents skip vector reindexing when hash and chunk ids match --> new, changed, reappeared, or rechunked documents are stored in Chroma --> source / job / document / chunk / tombstone metadata is stored in SQLite --> cleanup-capable sources tombstone stale documents after complete snapshots --> search_context may optionally rewrite weak queries through a configured LLM - when CONTEXTWIKI_SEARCH_LLM_ENABLED=true --> answer_with_citations inherits that same query-rewrite egress because it - calls search_context through search_context_for_answer / search_context --> search debug output reports whether rewrite was disabled, skipped, attempted, - or applied --> retrieval starts from Chroma candidates when available, then may add - metadata-fallback candidates before SQLite validation --> retrieval policy keeps vector score and rerank score separate in debug, uses - named rerank/rewrite thresholds in code, and no longer treats a bare - lowercase long token as an implicit GitHub repo probe without explicit - repository-style evidence such as GitHub/repository wording, a strong - repo-shaped anchor, or a curated compound alias --> retrieval candidates are hydrated through SQLite before citation use --> search_context asks Chroma for candidates, may add metadata fallback, and - validates managed hits through SQLite --> search_documents groups those validated candidates by document and keeps one - representative chunk per document for browsing --> answer_with_citations returns a helper evidence-gated answer preview -``` - -Interview or README version: - -```text -ContextWiki exposes MCP tools for source sync, incremental indexing, -citation-ready chunk search, grouped document browsing, context fetch, and a -helper evidence-gated answer preview surface. - -SQLite is the source of truth for lifecycle and citation metadata, while -ChromaDB is the semantic retrieval index. - -In normal production MCP usage, downstream LLMs usually turn retrieved evidence -into the final natural-language answer. `answer_with_citations` is useful here -as a preview/debug/eval helper, not as the repository's core answer contract. -``` - ---- - -## 1. Current Source Coverage - -ContextWiki currently has source connectors for: - -| Source | Source id | How it is configured | Notes | -| --- | --- | --- | --- | -| Notion | `source_notion` | `NOTION_API_KEY` | page/document source | -| Tistory | `source_tistory` | `TISTORY_BLOG_NAME` | blog post source | -| GitHub | `source_github` | `CONTEXTWIKI_GITHUB_REPOSITORIES`, optional `@ref`, `CONTEXTWIKI_GITHUB_DEFAULT_REF`, file limits, optional `GITHUB_TOKEN` | repository file source | -| Obsidian | `source_obsidian` | `CONTEXTWIKI_OBSIDIAN_VAULT_PATH`, `CONTEXTWIKI_OBSIDIAN_MAX_FILES`, `CONTEXTWIKI_OBSIDIAN_MAX_FILE_BYTES` | local Markdown vault source | - -Example: - -```bash -CONTEXTWIKI_GITHUB_REPOSITORIES="eunhwa99/MCPContentSearch@main" -``` - -Then: - -```text -sync_source("source_github") -``` - -fetches supported text/code/Markdown files from configured repositories, -converts each file into a `DocumentModel`, chunks it with line-range citation -metadata, indexes the chunks, and stores lifecycle metadata in SQLite. - -If `@ref` is omitted from a repository spec, the GitHub connector uses -`CONTEXTWIKI_GITHUB_DEFAULT_REF`, which defaults to `main`. Fetch completeness -and stale-cleanup eligibility also depend on `CONTEXTWIKI_GITHUB_MAX_FILES` and -`CONTEXTWIKI_GITHUB_MAX_FILE_BYTES`. `GITHUB_TOKEN` is optional, and -`CONTEXTWIKI_GITHUB_USER_AGENT` controls the request header used by the fetcher. - -Bulk retained-source sync is also available: - -```text -sync_all() -``` - -This fans out one concurrent `sync_source()` run per retained source. Each -source still keeps its own SQLite running-job guard, so a source that is -already syncing is reported as blocked in the aggregate result instead of -starting a second overlapping fetch. The running-job guard also refreshes a -sync-owner heartbeat beside the per-job heartbeat during active running-job -metadata updates. When a previous owner still looks alive only because both old -and new containers report the app as PID `1`, startup recovery and -`begin_sync_job()` specifically fall back to the running job's own staleness -window before reclaiming the source, instead of treating the short unowned-job -grace as proof that the previous sync is dead. - -Obsidian example: - -```bash -CONTEXTWIKI_OBSIDIAN_VAULT_PATH="/path/to/temp-or-real-vault" -CONTEXTWIKI_OBSIDIAN_MAX_FILES=2000 -CONTEXTWIKI_OBSIDIAN_MAX_FILE_BYTES=512000 -``` - -Then: - -```text -sync_source("source_obsidian") -``` - -reads bounded `.md` notes from the configured local vault, skips hidden and -Obsidian metadata directories, preserves frontmatter-derived titles when -available, uses `obsidian://open` canonical URLs, and stores lifecycle metadata -in SQLite. If the configured vault exceeds the max file count or per-file byte -bound, sync fails as an incomplete snapshot before stale cleanup. It does not -require a live Obsidian app. - -If a retained connector is disabled for future syncs, already indexed active -documents are not automatically hidden from retrieval. They remain retrievable -until later cleanup or metadata changes mark them inactive. - ---- - -## 2. Overall Mental Model - -ContextWiki is easiest to understand as four layers. - -```mermaid -flowchart TD - Client["AI Client"] - MCP["FastMCP Server"] - Tools["MCP Tools
api/tools.py"] - Ingestion["IngestionService"] - Search["ContextSearchService"] - Answer["CitationAnswerService"] - Store["MetadataStore
SQLite"] - Vector["Vector Index
ChromaDB / LlamaIndex"] - Sources["Source Connectors
Notion / Tistory / GitHub / Obsidian"] - - Client --> MCP - MCP --> Tools - Tools --> Ingestion - Tools --> Search - Tools --> Answer - Ingestion --> Sources - Ingestion --> Store - Ingestion --> Vector - Search --> Vector - Search --> Store - Answer --> Search -``` - -Important design intent: - -```text -AI clients do not read the database directly. -AI clients call MCP tools. - -ChromaDB finds semantically relevant candidate chunks. -SQLite decides whether those chunks are currently active, citeable evidence. -``` - ---- - -## 3. MCP Tool Surface - -Current tools: - -| Tool | Use | -| --- | --- | -| `list_sources()` | see configured sources | -| `sync_source(source_id)` | refresh one source | -| `sync_all()` | refresh all retained sources concurrently and return aggregate results | -| `get_sync_status(source_id?)` | inspect source/job state | -| `search_context(query, filters, top_k, include_debug)` | find SQLite-validated evidence | -| `search_documents(query, filters, top_k)` | browse unique matching documents through one representative chunk per document | -| `fetch_context(document_id, chunk_id)` | inspect one document or chunk | -| `answer_with_citations(question, filters, top_k, include_debug)` | helper answer preview from validated evidence | - -The retained `search_context` response always includes a public `debug` key. -On the normal default path `api/tools.py` returns `debug={}` unless -`include_debug=True`, and when caller filters leave no matching public sources -it still returns a small populated `debug` object even if `include_debug=False`. - -The current debug payload includes rewrite decision fields: - -```text -rewrite_enabled -rewrite_attempted -rewrite_applied -rewrite_skipped_reason -``` - -These fields explain whether query rewrite was disabled, not needed, or tried -without producing an applied rewrite. - -Current rewrite policy keeps one more guardrail: a single high-confidence, -textually matching result may suppress rewrite even when `top_k` asks for more -rows, so `insufficient_candidate_count` alone does not force rewrite churn. - -Observed `rewrite_skipped_reason` values include: - -```text -disabled -not_needed -empty_result -rewrite_failed -no_matching_sources -no_term_groups -not_supported -``` - -Tool handlers call service boundaries and return JSON-safe values through -Pydantic `model_dump(mode="json")` where needed. - -`include_debug=True` is the retained explainability switch for retrieval and -the helper answer preview surface. It is additive and opt-in for populated retrieval reasoning -on the normal success path: default `search_context` payloads still carry -`debug={}`, while debug payloads expose structured retrieval reasoning without -dumping raw local DB contents. The current exception is the public -`no_matching_sources` fast path in `search_context`, which still emits a small -populated `debug` object even when `include_debug=False`. -`answer_with_citations` does not mirror that exception path. - -Verification layer split: - -- Public MCP contract layer: - real `FastMCP.call_tool(...)` payload checks for each retained public tool -- Deterministic functional E2E layer: - retained sync/search/fetch/answer flow modules over temp/local state via - `./scripts/verify_functional_e2e.sh` -- Deterministic quality eval layer: - fixture retrieval and answer scoring via `tests/evals` and - `scripts/run_contextwiki_eval.py` -- Manual live smoke layer: - `scripts/live_query_smoke.py` against a user's configured local runtime; - this is manual diagnostic evidence, not automated gate coverage -Retrieval split: - -```text -search_context -= chunk-level retrieval for evidence, grounding, and citations - -search_documents -= grouped document-browsing retrieval that reuses the same retained-source - candidates but collapses repeated chunks into one representative row per - document -``` - -Helper answer validation focuses on: - -```text -citations -used_chunks -debug -debug_markdown -``` - -Those fields help developers compare the response draft against the retrieved -evidence and decide whether a downstream LLM would have enough grounded context -to answer well. -`list_sources()` and `get_sync_status()` expose additive reviewer-readable -operational fields per source: - -```text -latest_success_at -latest_failure_at -document_count -chunk_count -latest_failure_reason -stale_cleanup_disabled_reason -``` - -Those fields come from SQLite lifecycle metadata plus runtime connector state. -Public error text still passes through the same redaction path used by other -sync/job payloads. - -For reviewer-driven live local verification, the repo also includes: - -```text -scripts/live_query_smoke.py -``` - -This script runs `search_context` and `answer_with_citations` against the local -configured environment through the retained tool handlers and prints a compact -safe summary of rewrite state, hits, helper answer preview status, and -citations. It is a manual live check surface, not a replacement for -deterministic non-live tests. - ---- - -## 4. Core Model Relationships - -Relevant files: - -```text -core/models.py -storage/metadata_store.py -``` - -The most important models are: - -| Model | Meaning | Main use | -| --- | --- | --- | -| `SourceModel` | Notion, Tistory, GitHub, or Obsidian source | source configuration and sync state | -| `SyncJobModel` | one source sync execution | success/failure and processing counts | -| `DocumentModel` | one original document | identity, content hash, lifecycle, source metadata | -| `ChunkModel` | searchable/citeable document segment | vector search and citations | -| `ContextSearchResult` | search response DTO | chunk + score + preview + citation metadata | -| `DocumentSearchResult` | grouped document search response DTO | one representative chunk-backed row per document | - -Key distinction: - -```text -DocumentModel = management and sync unit -Examples: one Notion page, one Tistory post, one GitHub file, one Obsidian note. - -ChunkModel = search and citation unit -Examples: a markdown section, a code line range, a plain-text window. -``` - ---- - -## 5. SQLite vs ChromaDB - -SQLite and Chroma both store chunk-related information, but they have different -jobs. - -```text -SQLite MetadataStore -= source/job/document/chunk lifecycle source of truth - -ChromaDB -= semantic retrieval candidate index -``` - -```mermaid -flowchart TD - A["Fetch from Notion / Tistory / GitHub / Obsidian"] - B["DocumentModel"] - C["DocumentChunker -> ChunkModel[]"] - D["SQLite documents
identity/content/hash/lifecycle"] - E["SQLite chunks
citation metadata and text"] - T["SQLite chunk_tombstones
historical chunk id provenance"] - F["ChromaDB
semantic vector retrieval"] - G["search_context(query)"] - H["Chroma candidate chunks"] - I["Ranking decides whether
metadata fallback is needed"] - J["Combined candidates
Chroma plus metadata fallback"] - N["SQLite active chunk validation"] - O["ContextSearchResult[]"] - K["search_documents(query)"] - L["group by document_id
pick representative chunk"] - M["DocumentSearchResult[]"] - - A --> B --> C - B --> D - C --> E - C --> F - E --> T - G --> H --> I --> J --> N --> O - K --> H - H --> I - I --> J - N --> L --> M -``` - -Search results are not trusted directly from Chroma. Retrieval starts with -vector candidates when available, may add metadata-fallback candidates when the -ranker decides lexical or source-aware recovery is needed, and then hydrates -and validates managed hits through SQLite before they become evidence, -citations, or grouped document rows. - -When `search_context(..., include_debug=True)` is used, the response now makes -that decision path visible with reviewer-readable fields such as: - -```text -intent.name / raw_name / confidence / reasons -query_rewrite.attempted / applied / reason -retrieval_queries -rewritten_queries -filters.source_ids -selected_results[] -``` - -`query_rewrite.reason` is intentionally coarse and stable. Current values are: - -```text -no_initial_candidates -insufficient_candidate_count -missing_textual_match -low_initial_score -``` - -The retrieval path now also makes a coarse intent-policy decision before final -reranking. The current retained intent classes are: - -```text -strict_lookup -broad_topic -list -comparison -``` - -This intent is deterministic and local-first. It is derived from normalized -query terms and anchors, then reused by ranking and grounded answer rendering. -It is not a separate MCP tool and does not require a live LLM call. - ---- - -## 6. Sync and Incremental Indexing - -Relevant files: - -```text -indexing/ingestion_service.py -indexing/chunker.py -indexing/indexer.py -storage/metadata_store.py -``` - -`IngestionService.sync_source()` is the core per-source business flow, and -`IngestionService.sync_all()` is the retained-source aggregate fan-out entrypoint. - -```text -sync_source(source_id) --> SourceRegistry connector lookup --> MetadataStore register_source + begin_sync_job guard --> connector.fetch_documents() --> normalize document source/id/url/version/last_seen --> compute content_hash --> deterministic source-aware chunking --> skip vector reindexing when active content_hash and chunk ids are unchanged --> index new/changed/reappeared/rechunked chunks in Chroma --> commit document + chunk metadata --> finalize successful sync --> tombstone stale documents when cleanup is safe --> best-effort vector cleanup -``` - -```text -sync_all() --> SourceRegistry retained source enumeration --> concurrent per-source sync_source() fan-out --> one aggregate summary with succeeded / failed / blocked counts --> per-source result payloads that preserve the latest job outcome -``` - -Reindexing still happens when: - -```text -- a tombstoned document reappears -- content changes -- generated chunk ids change -- document identity changes -``` - -Failed or partial syncs must not tombstone missing documents. - ---- - -## 7. Source-aware Chunking - -Relevant file: - -```text -indexing/chunker.py -``` - -Current chunking strategy: - -```text -Markdown with headings --> heading / section based chunks - -Markdown without headings --> deterministic plain-text fallback windows - -Code --> deterministic line-range chunks --> blank lines preserved --> long lines split by max_chars - -Plain text --> deterministic character windows -``` - -Each chunk carries citation metadata: - -```text -chunk_id -document_id -source_id -title -url -path -chunk_index -line_start -line_end -content_hash -version_id -updated_at -``` - ---- - -## 8. Document Identity and Versioning - -Relevant fields: - -| Field | Meaning | -| --- | --- | -| `source_id` | which ContextWiki source owns the document | -| `external_id` | stable id from the original system | -| `document_id` | internal canonical document id, usually `external_id` | -| `canonical_url` | primary URL used for citations and legacy matching | -| `version_id` | source version metadata, separate from stable identity | -| `last_seen_at` | last sync time that observed the document | -| `last_seen_sync_id` | job marker that observed the document | -| `deleted_at` | tombstone timestamp when missing from a cleanup-capable successful sync | - -Current mapping: - -```text -Notion --> external_id = page_id --> document_id = page_id - -Tistory --> external_id = blog_name:post_id --> document_id = blog_name:post_id - -GitHub --> external_id/document_id = github:owner/repo:path --> canonical_url = GitHub blob URL at the resolved commit --> version_id = blob SHA - -Obsidian --> external_id/document_id = relative/note/path.md --> canonical_url = obsidian://open URL for the configured vault note --> title = frontmatter title when present, otherwise note stem -``` - -Stable identity should not change just because content changes. `version_id` -records source revision metadata; `content_hash` records actual indexed content. - ---- - -## 9. Tombstones and Stale Vector Safety - -Tombstone means soft delete, not hard delete. - -```text -documents.deleted_at records when a document disappears from a cleanup-capable -source after a complete successful sync. -``` - -For GitHub, cleanup is limited to repository document-id prefixes fetched by the -connector, such as `github:eunhwa99/mcpcontentsearch:`. This keeps one -configured repository sync from tombstoning documents that belong to another -repository scope under the same `source_github` source id. - -For Obsidian, cleanup is allowed only after a complete local-vault snapshot. -Unreadable notes, traversal errors, or exceeded file count/byte bounds should -fail the sync before stale cleanup can tombstone missing active documents. - -Why not hard delete immediately? - -```text -Vector cleanup is best-effort. -If Chroma cleanup fails, SQLite still needs historical provenance to suppress -stale managed vector hits. -``` - -SQLite is the last defense against stale vector results. - ---- - -## 10. Answer Behavior - -`answer_with_citations` delegates retrieval to `ContextSearchService` and only -uses validated chunks as evidence. The answer response should make evidence -status explicit: - -```text -grounded / insufficient / error -``` - -The service must not invent citations from unmanaged or tombstoned chunks. If -retrieval cannot find enough active evidence, the response should say that the -available evidence is insufficient instead of fabricating an answer. - -When evidence is sufficient, the final rendered answer can now vary by request -shape while staying citation-grounded: - -```text -summary-style answer for ordinary grounded QA -Grounded List for collection requests -Grounded Comparison for comparison requests -``` - -This is still bounded synthesis, not open-ended chat. The answer service only -repackages retrieved evidence chunks and keeps the same insufficient-evidence -escape hatch when retrieval is weak. - -`search_documents` does not replace this evidence path. It is a browsing surface -for "which documents matched?" and keeps the highest-ranked chunk as the -representative row so callers can pivot into `fetch_context` or run a later -chunk-level citation workflow through `search_context` or -`answer_with_citations`. - ---- - -## 11. Current Limits - -Current intentional limits: - -- No generic website/docs crawler in production scope. -- No browser Web Console or local HTTP reviewer UI in production scope. -- No Auto Wiki generation or LLM wiki synthesis in production scope. -- No dynamic web fallback or legacy live search/index MCP tools in production - scope. -- No live Obsidian app, plugin, or API server requirement; Obsidian is a - configured local-vault source. -- Optional `CONTEXTWIKI_SEARCH_LLM_ENABLED=true` query rewrite is disabled by - default. If enabled, it may send the user query and normalized terms to the - configured provider, but it must not send source evidence, fetch external - source content, or mutate SQLite/Chroma. -- No deletion, reset, migration, or inspection of local user Chroma/SQLite data - without explicit approval. -- Live Notion, Tistory, GitHub, or embedding-provider validation is opt-in and - approval-gated. Real Obsidian vault validation is also approval-gated; - routine verification uses temporary vaults. - -Historical note: - -- ADR 0004's GitHub connector decision remains current. -- ADR 0004 now also documents the retained Obsidian local-vault connector. -- ADR 0004's website/docs connector portion is superseded for the current scope - by ADR 0006. -- ADR 0005's Auto Wiki decision is superseded for the current scope by ADR - 0006. - ---- - -## 12. Verification Model - -Retained verification layers: - -- Public MCP contract layer: - - `uv run --locked pytest -q tests/contracts/test_public_mcp_contracts.py tests/test_app_composition.py` -- Deterministic functional E2E layer: - - `./scripts/verify_functional_e2e.sh` -- Deterministic quality eval layer: - - `uv run --locked pytest -q tests/evals` - - `uv run --locked python scripts/run_contextwiki_eval.py --output-dir artifacts/contextwiki-evals` - - optional latency artifacts: - `uv run --locked python scripts/run_contextwiki_eval.py --output-dir artifacts/contextwiki-evals --include-latency` -- Manual live smoke layer: - - `uv run --locked python scripts/live_query_smoke.py --query "your query here"` -- Full local gate wrapper: - - `./scripts/verify_all.sh` - - wraps the automated layers above but does not replace the manual live smoke layer - -Functional verification should use fake or temporary persistence and must not -mutate local user Chroma/SQLite data. Live external checks require explicit -approval and should report the source used, safety plan, and whether any local -state was touched. -Obsidian verification should use temporary vault directories unless the user -explicitly approves a real vault path. - -The deterministic eval runner is the retained reviewer-evidence path for -retrieval quality changes. It uses temp SQLite fixture data, never touches user -Chroma/SQLite, and now emits deterministic JSON artifacts with: - -- mixed-query group breakdowns -- retrieval and answer suite pass/fail summaries - -When latency visibility is needed, the runner can also emit a separate -non-deterministic `runtime_metrics.json` file with wall-clock retrieval and -answer timings. That runtime file is an opt-in local artifact from the -latency-enabled command above; the current CI `contextwiki-evals` artifact -uploads only the deterministic retrieval-quality JSON files. diff --git a/docs/images/claude-desktop-dynamodb-star-example.png b/docs/images/claude-desktop-dynamodb-star-example.png new file mode 100644 index 0000000..8e24f37 Binary files /dev/null and b/docs/images/claude-desktop-dynamodb-star-example.png differ diff --git a/docs/plan/2026-06-15-readme-mcp-docker-obsidian.md b/docs/plan/2026-06-15-readme-mcp-docker-obsidian.md new file mode 100644 index 0000000..dcb4326 --- /dev/null +++ b/docs/plan/2026-06-15-readme-mcp-docker-obsidian.md @@ -0,0 +1,133 @@ +# README and Docs Simplification + +## User request + +Update the reader-facing docs so setup mistakes encountered during Docker, +Claude Desktop MCP registration, GitHub repository config, and Obsidian vault +wiring are less likely to happen again. Consolidate the maintained +architecture/explanation path onto `README.md` plus `.agents/docs/architecture.md` +so readers do not need a separate core-understanding note or ADR trail for +current setup. + +## Branch preflight result + +| Phase | Status | Summary | Evidence | +| --- | --- | --- | --- | +| Branch preflight | completed | Started from a clean detached worktree, fetched `origin/main`, and created `feature/readme-mcp-docker-obsidian` from `origin/main` because local `main` is checked out in another linked worktree. | `git status --short`; `git branch --show-current`; `git branch -vv`; `git worktree list`; `git fetch origin main`; `git switch -c feature/readme-mcp-docker-obsidian origin/main` | + +## Scope and non-goals + +- Clarify README guidance for: + - quoted `.env` pitfalls in `CONTEXTWIKI_GITHUB_REPOSITORIES` + - Tistory blog-name format expectations + - Docker Obsidian vault mount requirements + - stdio MCP behavior in Docker and Claude Desktop + - Claude Desktop launching the server automatically +- Simplify the active docs structure so `Architecture` is the single maintained + design/reference document beyond the README. + +Non-goals: + +- No runtime behavior changes +- No Claude Desktop config file edits as part of this repo change +- No connector or Dockerfile code changes +- No historical plan-doc cleanup + +## Acceptance criteria + +- README no longer suggests `.env` examples that are known to break the current + parser for GitHub repository specs. +- README explicitly explains that Dockerized Obsidian sync needs both a mount + and a container-visible vault path. +- README explains that this MCP server is stdio-based, so detached Docker runs + are not the normal MCP client integration path. +- README includes a Claude Desktop example that can use Docker as the spawned + MCP command. +- README no longer points readers to `docs/contextwiki-core-understanding.md` + or ADRs as required current docs. +- Harness docs point to `.agents/docs/architecture.md` as the active design + source of truth. +- The maintained design explanation now lives only in + `.agents/docs/architecture.md`, with the previously duplicated core + understanding content absorbed there. + +## Step breakdown + +1. Review the current README and harness/doc structure against the observed + setup failures and current duplicate-doc paths. +2. Update README examples and troubleshooting so the documented path matches the + current parser and MCP runtime model. +3. Remove active references that force readers through the extra + core-understanding and ADR path, and keep architecture as the maintained + design note. +4. Run docs-only verification checks. + +## Files likely to change + +- `README.md` +- `AGENTS.md` +- `.agents/docs/architecture.md` +- `.agents/docs/harness-engineering.md` +- `.agents/docs/adr/README.md` +- `.agents/docs/github-workflow.md` +- `.agents/skills/harness-engineering/SKILL.md` +- `.agents/skills/harness-plan/SKILL.md` +- `.agents/skills/harness-review/SKILL.md` +- `.agents/skills/harness-functional-smoke/SKILL.md` +- `.agents/skills/harness-implement/SKILL.md` +- `docs/images/claude-desktop-dynamodb-star-example.png` +- `docs/plan/README.md` +- `docs/plan/2026-06-15-readme-mcp-docker-obsidian.md` + +## Test and verification plan + +- `rg --files AGENTS.md README.md docs .agents/docs .agents/skills` +- `git status --short --branch` +- `git diff --check` +- Stage docs-only files +- `git diff --cached --check` + +## Functional smoke matrix or planned matrix rows before review + +| Surface | Planned check | Why | +| --- | --- | --- | +| README setup guidance | manual diff inspection | Docs-only request; no runtime behavior change | +| Docs structure redirects | manual diff inspection | Confirm architecture-only active-doc path, ADR archive wording, and the intentional deletion of the duplicated core-understanding file stay reader-clear | +| Harness and instruction contracts | manual diff inspection | Confirm AGENTS, harness docs, phase skills, and plan template all reflect the same architecture-first active-doc policy | + +## Architecture constraints + +- Follow `.agents/docs/architecture.md` slim MCP scope and keep README aligned + with retained MCP tools and local source connectors. +- Preserve the current documented local-vault Obsidian behavior and + client-attached stdio MCP framing. + +## Risks and rollback notes + +- Risk: over-documenting one client path while obscuring the local `uv` path. + Mitigation: keep both local and Docker/client-spawn paths explicit. +- Rollback: revert the staged docs bundle for this work item if the new + architecture/README simplification wording proves inaccurate. + +## Progress log + +| Phase | Status | Summary | Evidence | +| --- | --- | --- | --- | +| Branch preflight | completed | Created `feature/readme-mcp-docker-obsidian` from `origin/main` in this clean worktree. | Git evidence above | +| Planning | completed | Expanded the earlier README-only scope into a docs-structure simplification. Direct implementation remains appropriate because this is still an atomic docs-only change owned by one author across tightly related files. | This plan | +| README update | completed | Rebased the README structure around the simplified reader flow, then folded in the reproduced GitHub quoting, Tistory format, Docker stdio, Claude Desktop auto-launch, and Dockerized Obsidian mount pitfalls so the documented path is safer to follow verbatim. | `README.md` | +| README Docker clarification follow-up | completed | Added an explicit split between the minimum Docker run example and the Obsidian-enabled Docker run example so readers do not assume the vault mount is optional when `source_obsidian` is in use. | `README.md` | +| README Claude Desktop example | completed | Added a real Claude Desktop screenshot plus a short example prompt so readers can see a Claude client workflow using ContextWiki after setup. | `README.md`; `docs/images/claude-desktop-dynamodb-star-example.png` | +| README wording cleanup follow-up | completed | Removed the extra explanatory sentence under the Claude Desktop example so the README stays focused on the prompt plus screenshot evidence only. | `README.md` | +| Functional smoke | completed | Completed the docs-only smoke rows by manually inspecting the README setup flow, Docker/Obsidian notes, Claude Desktop example placement, architecture-only active-doc wording, ADR archive wording, and harness/instruction-contract consistency against the final staged wording. | `README.md`; `AGENTS.md`; `.agents/docs/harness-engineering.md`; `.agents/docs/github-workflow.md`; `.agents/skills/harness-engineering/SKILL.md`; `.agents/skills/harness-functional-smoke/SKILL.md`; `.agents/skills/harness-review/SKILL.md`; `.agents/docs/adr/README.md`; `.agents/docs/architecture.md`; `docs/plan/README.md`; matrix rows above | +| Review-fix pass 1 | completed | Resolved reviewer findings by converting `.agents/docs/adr/README.md` into a historical archive note instead of an active harness contract. | `.agents/docs/adr/README.md` | +| Review-fix pass 2 | completed | Clarified that the documented Claude Desktop config path is the macOS path, added the missing Docker image build prerequisite, fixed the stale AGENTS connector inventory to include Obsidian, and synced the plan file list with the actual staged scope. | `README.md`; `AGENTS.md`; `docs/plan/2026-06-15-readme-mcp-docker-obsidian.md` | +| Review-fix pass 3 | completed | Reframed the screenshot section as a Claude Desktop client workflow example rather than a direct server-answer capability, and updated the plan template wording from `Architecture/ADR constraints` to `Architecture constraints` so active plan guidance matches the simplified harness contract. | `README.md`; `docs/plan/README.md`; `docs/plan/2026-06-15-readme-mcp-docker-obsidian.md` | +| Docs structure simplification | completed | Removed the active `Core Understanding` reader path from README and harness flows, dropped current-reader ADR dependencies from active harness docs, and positioned `Architecture` as the single maintained design reference beyond the README. | `README.md`; `AGENTS.md`; `.agents/docs/architecture.md`; `.agents/docs/adr/README.md`; harness docs/skills | +| Review-fix pass 4 | completed | Separated local `uv` prerequisites from Docker prerequisites and clarified that the Docker Claude Desktop config is minimal by default and needs an extra mount only for Obsidian. | `README.md`; `docs/plan/2026-06-15-readme-mcp-docker-obsidian.md` | +| Review-fix pass 5 | completed | Added an explicit warning about reusing a host-path Obsidian env var with the minimum Docker example. | `README.md`; `docs/plan/2026-06-15-readme-mcp-docker-obsidian.md` | +| Follow-up docs merge | completed | Merged the remaining setup and runtime mental model into `.agents/docs/architecture.md`, then deleted `docs/contextwiki-core-understanding.md` per the latest request so current docs have a single maintained design document. Historical plan-doc mentions remain archival only. | `.agents/docs/architecture.md`; `docs/plan/2026-06-15-readme-mcp-docker-obsidian.md`; delete `docs/contextwiki-core-understanding.md` | +| Docs verification | completed | Reran docs-only file listing, branch status, whitespace checks, staged diff checks, and cached diff checks after the follow-up architecture merge and file deletion. | `rg --files AGENTS.md README.md docs .agents/docs .agents/skills`; `git status --short --branch`; `git diff --check`; `git add ...`; `git diff --cached --check` | +| Review-fix pass 6 | completed | Cleared the stale post-merge plan state by marking docs verification complete and keeping the verification evidence path on the real `SKILL.md` files rather than a nonexistent markdown pattern. | `docs/plan/2026-06-15-readme-mcp-docker-obsidian.md` | +| Review-fix pass 7 | completed | Expanded `.agents/docs/architecture.md` so the single maintained design doc now covers retained `sync_all` aggregate semantics, retrieval/debug policy boundaries, and the layered verification model that previously lived in the duplicated note. | `.agents/docs/architecture.md`; `docs/plan/2026-06-15-readme-mcp-docker-obsidian.md` | +| Final review gate | completed | Ran a fresh five-reviewer `$subagent-review-loop` pass after the final architecture backfill fixes. The newest pass reported no actionable findings, so the architecture-only maintained-doc transition and file deletion are review-clean. | reviewers `Ramanujan`, `Poincare`, `Locke`, `Carson`, and `Hubble` in this thread | diff --git a/docs/plan/README.md b/docs/plan/README.md index 20ff057..986b1d3 100644 --- a/docs/plan/README.md +++ b/docs/plan/README.md @@ -32,7 +32,7 @@ Each plan document should include: - Files likely to change - Test and verification plan - Functional smoke matrix or planned matrix rows before review -- Architecture/ADR constraints +- Architecture constraints - Risks and rollback notes - Progress log