This repository was archived by the owner on May 18, 2026. It is now read-only.
feat: add single-shot tool retrieval layer#1
Merged
Conversation
Embeds MCP tool descriptions at index time and uses cosine similarity
to expose only the top-K relevant tools to OpenCode per session.
New commands:
- opencode-workspace index [--force] build/update the tool corpus
- opencode-workspace stats [--last N] summarise retrieval sessions
- opencode-workspace "<prompt>" one-shot with tool filtering
Key components:
- src/index/ MCP client (stdio+remote), HuggingFace ONNX embedder,
SQLite corpus with sqlite-vec ANN + brute-force fallback
- src/retrieval/ cosine search, server-level deny-rule composer,
non-destructive temp config overlay via OPENCODE_CONFIG
- src/telemetry/ atomic JSONL sessions log, stats aggregation
- src/config.js deep-merged config with defaults (local all-MiniLM-L6-v2,
k=10, topk strategy); OpenAI/Voyage/Cohere provider stubs
- bin/smoke.js make smoke: index + assert GitHub tool is top-1 for a
known GitHub query
Kill switch: OPENCODE_WORKSPACE_RETRIEVAL=off passes through unchanged.
TUI mode (no prompt arg) is unaffected.
docs/: Gherkin feature files for all new behaviour (37 scenarios).
37 scenarios / 158 steps covering all 5 feature files:
indexing, retrieval, permissions, telemetry, configuration
Infrastructure:
- unit-tests/support/ world, hooks (isolated HOME per scenario,
process.exit stub, console capture), fixtures
(fake 384-dim vector space, canned tool lists)
- unit-tests/step-definitions/ common + one file per feature
- cucumber.js runner config (docs/*.feature, 30s timeout)
- proxyquire used for cmd modules that destructure deps at require time
.github/workflows/test.yml:
- ubuntu-latest / Node 20 LTS
- npm ci --ignore-scripts (skip postinstall tool downloads)
- npm rebuild better-sqlite3 (restore native binary)
- npx cucumber-js
make test -> npx cucumber-js (was: help + kill-switch check)
Three new retrieval modes complement the existing one-shot flow:
1. Standalone retrieve command
- src/cmd/retrieve.js: cmdRetrieve(query, { json, k }) embeds a query
and prints top-K corpus matches to stdout (text or --json).
- Dispatched as: opencode-workspace retrieve [--json] [--k N] <query>
2. TUI first-message hook (OpenCode plugin)
- lib/tool-retrieval.plugin.js: ESM plugin installed to
~/.config/opencode/plugins/ow-tool-retrieval.js by 'install'.
- Hooks message.updated; fires once per session on the first user
message; calls 'opencode-workspace retrieve --json' as a subprocess;
injects results via client.session.prompt({ noReply: true }) so the
agent has tool recommendations in context before it replies.
- Soft-fails silently on any error.
3. On-demand search_tools MCP tool
- src/mcp/tool-retrieval-server.js: MCP stdio server launched as
'opencode-workspace mcp-serve'.
- Exposes search_tools({ query, k? }) — the agent calls this
proactively whenever it needs to discover relevant MCP capabilities.
- Added to lib/opencode.json.template as the 'tool-retrieval' server.
- Listed in ALWAYS_ALLOWED (src/retrieval/permissions.js) so it is
never denied by the one-shot permission generator.
Other changes:
- bin/cli.js: retrieve and mcp-serve subcommands; plugin copy step in
cmdInstall(); updated help text.
- AGENTS.md: documents all three retrieval modes, new runtime paths,
ALWAYS_ALLOWED gotcha, plugin ESM/global-scope notes.
- docs/tui-retrieval.feature, docs/tool-retrieval-mcp.feature: BDD specs.
- unit-tests/step-definitions/indexing.steps.js: add tool-retrieval to
the all-servers-fail stub so the exit-code-1 scenario stays correct.
…eval-mcp features Refactored plugin and MCP server to expose testable CommonJS helpers, then wired up full step definitions for both new feature files. All 50 scenarios and 223 steps pass. Refactors (to enable unit testing without a live OpenCode or MCP process): - src/cmd/tui-hook.js (new): extracts first-message hook logic from the ESM plugin into a testable CommonJS module. Exports createFirstMessageHandler() (stateful, closes over seenSessions Set) and handleFirstMessage() (core injection logic with optional _searchFn override for tests). - src/mcp/search-tools-handler.js (new): extracts search_tools handler from the MCP server. Exports handleSearchTools() and formatResults(). Accepts optional _searchFn for test injection. - lib/tool-retrieval.plugin.js: rewritten to import createFirstMessageHandler via createRequire(import.meta.url) instead of spawning a subprocess. Plugin body is now a 3-line thin adapter. - src/mcp/tool-retrieval-server.js: delegates tools/call to handleSearchTools. Test infrastructure: - unit-tests/support/fixtures.js: add FETCH_TOOLS (fetch server, dim 4) and add 'fetch' to ALL_FIXTURES for 'browse/fetch URL' scenarios. - unit-tests/support/world.js: add _capturedInjections, _hookHandler, _hookLastQuery, _searchToolsResult to constructor; add runHook() and runSearchTools() helpers. New step definitions (all steps now defined — 0 undefined): - unit-tests/step-definitions/tui-retrieval.steps.js: 26 steps covering first-message injection, session dedup, role filtering, silent-fail on error, and context format validation. - unit-tests/step-definitions/tool-retrieval-mcp.steps.js: 18 steps covering ranked results, k limit, empty corpus message, missing query error, ALWAYS_ALLOWED permission behaviour, and server-specific relevance. Feature file updates: - docs/tui-retrieval.feature: renamed 'Plugin is silent when opencode-workspace is not in PATH' → 'Plugin is silent when search throws an error' to reflect the in-process (non-subprocess) architecture after the refactor.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a tool-RAG layer that embeds MCP tool descriptions at index time and uses cosine similarity to expose only the top-K relevant tools to OpenCode per session, cutting context overhead from loading every server's schema upfront.
New commands
Existing TUI mode (`opencode-workspace` with no args) is unchanged.
Kill switch: `OPENCODE_WORKSPACE_RETRIEVAL=off` passes through to opencode with no filtering.
Architecture
```
src/
index/ MCP stdio+remote client, HuggingFace ONNX embedder (all-MiniLM-L6-v2),
SQLite corpus with sqlite-vec ANN + brute-force cosine fallback
retrieval/ cosine search, server-level deny-rule composer,
non-destructive temp config overlay (OPENCODE_CONFIG, never mutates user config)
telemetry/ atomic JSONL session log, stats aggregation
config.js deep-merged config, defaults: local embedder, k=10, topk strategy
OpenAI wired; Voyage/Cohere stubbed
bin/smoke.js make smoke: index then assert GitHub tool is top-1 for a GitHub query
docs/ Gherkin feature files documenting all new behaviour (37 scenarios, 5 files)
```
Non-destructive by design
Verified
```
$ make smoke
=== Step 1: index MCP tool corpus ===
✓ notion +22
✓ gitlab +179
✓ playwright +23
✓ fetch +1
✓ semgrep +7
✓ aws-knowledge +6
✓ sequential-thinking +1
✓ github +26
⚠ brave-search failed: --brave-api-key is required (expected, no key configured)
Done. corpus: 265 tools
=== Step 2: retrieval assertion ===
PASS Corpus contains 265 tools
PASS Query returned 5 result(s)
PASS Top result: github/get_pull_request score=0.370
```