Skip to content
This repository was archived by the owner on May 18, 2026. It is now read-only.

refactor: migrate to Bun monorepo built on forked OpenCode source #5

Merged
GusCayresMindsight merged 4 commits into from
May 15, 2026
Merged

refactor: migrate to Bun monorepo built on forked OpenCode source #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
86bec58
refactor: migrate to Bun monorepo built on forked OpenCode source
GusCayresMindsight May 15, 2026
88792a7
ci: update workflow for Bun monorepo and add permissions block
GusCayresMindsight May 15, 2026
8c7ca96
test: add 70 BDD-driven unit tests; update docs and AGENTS.md
GusCayresMindsight May 15, 2026
727efc9
fix: bare 'ow' outside tmux launches workspace manager instead of TUI
GusCayresMindsight May 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
Empty file added .github/TEAM_MEMBERS
View file
Open in desktop
Empty file.
20 changes: 11 additions & 9 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,22 +5,24 @@ on:
branches: [master]
pull_request:

permissions:
contents: read

jobs:
unit-tests:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- uses: actions/setup-node@v4
- uses: oven-sh/setup-bun@v2
with:
node-version: '20'
cache: 'npm'
bun-version: "1.3.13"

- name: Install dependencies
run: npm ci --ignore-scripts

- name: Rebuild native add-on
run: npm rebuild better-sqlite3
run: bun install

- name: Run unit tests
run: npx cucumber-js
run: bun test --cwd packages/workspace

- name: Build ow binary
run: make build
24 changes: 14 additions & 10 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,19 +1,23 @@
# Dependencies
node_modules/
.opencode/node_modules/

# Build outputs
packages/opencode/dist/
packages/opencode/.artifacts/

# Local OpenCode config/state
.opencode/

# Sub-projects managed separately
# opencode:allow
workspaces/
# Local OpenCode config and state
.opencode/

# Playwright
.playwright-mcp/

# Secrets
# Secrets / env
.env

# Node dependencies
.opencode/node_modules/

# OS noise
.DS_Store
Thumbs.db

# Playwright
.playwright-mcp/
209 changes: 152 additions & 57 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,97 +1,192 @@
# OpenCode Workspace — AGENTS.md
# ow — AGENTS.md

## What this repo is
## BDD-first workflow

**Read this before touching any code.**

`docs/*.feature` files are the source of truth for this project's behavior.
Every non-`@wip` scenario MUST have a passing `bun test`.

### The rule

Before making any code change:

1. **Read** the relevant `docs/*.feature` file(s)
2. **Update** the feature file first — if the behavior you are adding or
changing does not have a scenario, write one before writing any code
3. **Write** the failing test in `packages/workspace/src/*.test.ts`
4. **Make it pass** by editing `packages/workspace/src/`
5. If the CLI surface changes, update `packages/opencode/src/cli/cmd/corpus.ts`
or `ws.ts`
6. Run `make test` — all 70+ tests must stay green

Scenarios tagged `@wip` are aspirational or require live infrastructure
(real MCP servers, tmux). Do not write unit tests for them.

A tmux workspace manager + **MCP tool-retrieval layer** for OpenCode. Tool retrieval operates in three modes:
---

## What this repo is

1. **One-shot** — before each `opencode run` session, the prompt is embedded, the corpus is searched, and deny-rules are injected into a temp config.
2. **TUI first-message hook** — an OpenCode plugin (`lib/tool-retrieval.plugin.js`, installed to `~/.config/opencode/plugins/ow-tool-retrieval.js`) fires on the user's first TUI message, runs retrieval, and injects the results as system context via `client.session.prompt({ noReply: true })`.
3. **On-demand MCP tool** — the `tool-retrieval` MCP server (launched via `opencode-workspace mcp-serve`) exposes a `search_tools(query, k?)` tool. The agent calls this proactively whenever it believes it needs additional or different MCP capabilities.
A Bun monorepo that builds the **`ow` binary** — a fork of
[anomalyco/opencode](https://github.com/anomalyco/opencode) extended with:

Plain Node.js (CommonJS, no TypeScript, no build step). Requires Node ≥ 18.
- **Semantic MCP tool retrieval** — indexes all configured MCP server tools
into a local SQLite corpus, embeds them with a local ONNX model, and
surfaces the most relevant ones at the start of every session
- **Built-in tool-retrieval plugin** — fires on the first user message,
embeds it, and injects the top-K tools as system context before the LLM
responds (no subprocess, no separate install)
- **`ow corpus` commands** — `index`, `retrieve`, `stats`, `mcp-serve`
- **`ow ws` command** — tmux workspace management (two-pane layout)

---

## Developer commands

```bash
make install # npm install -g .
make test # opencode-workspace --help (exit-code only; very shallow)
make smoke # node bin/cli.js index && node bin/smoke.js (real validation)
make update # bumps package.json "opencode.version" from GitHub API — does NOT run npm install
make install # bun install (resolves all workspace dependencies)
make build # builds ./packages/opencode/dist/ow-linux-x64/bin/ow
make dev # interpreted mode — no compile, fast iteration
make test # bun test --cwd packages/workspace (70 unit tests, ~80 ms)
make smoke # make build + ow corpus index + retrieval assertion
```

**One-shot usage:**
Bun 1.3.13 is pinned in `package.json` `packageManager`. Install it:

```bash
opencode-workspace index # incremental; builds corpus before first one-shot
opencode-workspace index --force # re-embeds every tool regardless of cache
opencode-workspace "find open PRs" # retrieval → temp config → opencode run
OPENCODE_WORKSPACE_RETRIEVAL=off opencode-workspace "any prompt" # bypass retrieval
opencode-workspace stats --last 10
opencode-workspace mcp env GITHUB_TOKEN # store MCP credential
curl -fsSL https://bun.sh/install | bash -s "bun-v1.3.13"
```

**Standalone retrieval (new):**
Symlink the built binary onto PATH after `make build`:

```bash
opencode-workspace retrieve "list GitHub pull requests" # human-readable top-K
opencode-workspace retrieve --json "run browser tests" # JSON array output
opencode-workspace retrieve --k 5 "query a database" # override top-K count
ln -sf $(pwd)/packages/opencode/dist/ow-linux-x64/bin/ow ~/.local/bin/ow
```

**Fresh-install order (matters):**
1. `npm install -g .`
2. `opencode-workspace install` — installs uv, glab, opencode 1.15.0, semgrep
3. `opencode-workspace mcp env NOTION_TOKEN` / `GITHUB_TOKEN` (if needed)
4. `opencode-workspace index` — corpus must exist before any one-shot
5. `make smoke` — asserts GitHub PR query returns a GitHub tool as top-1

---

## Architecture — what to know before editing
## Repo structure

**Indexing** (`src/cmd/index.js`): reads `lib/opencode.json.template`, spawns each MCP server (max 4 parallel, 15 s timeout), calls `listTools()`, hashes `description+inputSchema` to skip unchanged tools, embeds `"<server> / <tool_name>: <description>"`, stores in SQLite.
```
packages/
opencode/ # Fork of anomalyco/opencode — the ow binary
src/
index.ts # CLI entry: scriptName("ow"), all commands
cli/cmd/corpus.ts # ow corpus index|retrieve|stats|mcp-serve
cli/cmd/ws.ts # ow ws [term]
plugin/index.ts # INTERNAL_PLUGINS — ToolRetrievalPlugin added here
workspace/ # Tool-retrieval logic (our code)
src/
config.ts # loadConfig() / loadConfigFromFile() ← docs/configuration.feature
db.ts # openDb() / createTestDb()
hash.ts # hashTool()
corpus.ts # upsertTool / getToolHash / packF32… ← docs/indexing.feature
embedder.ts # createEmbedder() — local ONNX or OpenAI
search.ts # search() / cosineSim / bruteForceSearch ← docs/retrieval.feature
telemetry.ts # appendSession / readSessions / computeStats ← docs/telemetry.feature
mcp-client.ts # listToolsForServer / loadMcpEnvFromFile ← docs/mcp-env.feature
cmd/
index.ts # cmdIndex() — reads opencode.json, spawns MCP servers
retrieve.ts # cmdRetrieve() — runs search(), prints results
stats.ts # cmdStats() — formats telemetry
mcp-serve.ts # startMcpServer() / handleSearchTools() ← docs/tool-retrieval-mcp.feature
plugin/
tool-retrieval.ts # ToolRetrievalPlugin / handleFirstMessage() ← docs/tui-retrieval.feature
core/ # From upstream — shared utilities (@opencode-ai/core)
sdk/js/ # From upstream — HTTP client (@opencode-ai/sdk)
plugin/ # From upstream — plugin type definitions (@opencode-ai/plugin)
ui/ # From upstream — TUI component library (@opencode-ai/ui)
script/ # From upstream — build scripts (@opencode-ai/script)
docs/ # BDD feature files — the source of truth
configuration.feature
indexing.feature
retrieval.feature
telemetry.feature
mcp-env.feature
tool-retrieval-mcp.feature
tui-retrieval.feature
tui-commands.feature # all @wip (tmux)
smoke-test.feature # all @wip (integration)
prerequisites.feature
```

---

## Adding new behavior (step by step)

**One-shot** (`src/cmd/oneshot.js`): embeds prompt → cosine-searches corpus → collects unique server names from top-K → reads `~/.config/opencode/opencode.json` for existing user permissions → writes merged temp config to `/tmp/ow-<uuid>.json` with deny-rules for every server NOT in top-K → `OPENCODE_CONFIG=/tmp/ow-<uuid>.json opencode run "..."` → deletes temp file.
1. **Feature file first** — open the relevant `docs/*.feature` and add or
update a scenario. If none of the existing files fit, create a new one.

**TUI first-message hook** (`lib/tool-retrieval.plugin.js`): an OpenCode plugin installed to `~/.config/opencode/plugins/ow-tool-retrieval.js` by `opencode-workspace install`. Subscribes to the `message.updated` event. On the first user message per session, it calls `opencode-workspace retrieve --json "<text>"` as a subprocess, then calls `client.session.prompt({ noReply: true, … })` to inject the retrieved tool list as system context before the LLM responds. Soft-fails silently on any error so normal operation is never interrupted.
2. **Write the failing test** in `packages/workspace/src/*.test.ts` using
`bun:test` (`describe / test / expect`). Run `make test` and confirm it
fails for the right reason.

**On-demand retrieval tool** (`src/mcp/tool-retrieval-server.js`): a MCP stdio server launched as `opencode-workspace mcp-serve`. Always present in the template config (never denied by permission rules via `ALWAYS_ALLOWED` in `src/retrieval/permissions.js`). Exposes `search_tools(query, k?)` — the agent calls this proactively when it suspects it needs a tool it does not currently know about.
3. **Implement** in `packages/workspace/src/`. Keep functions small and
injectable (accept `_searchFn`, `_corpusSizeFn`, explicit file paths)
so they stay testable without filesystem side-effects.

**Standalone retrieval** (`src/cmd/retrieve.js`): `opencode-workspace retrieve [--json] [--k N] "<query>"`. Used by the plugin subprocess and directly by users or scripts.
4. **Wire CLI** — if the feature needs a new or changed CLI command, edit
`packages/opencode/src/cli/cmd/corpus.ts` (for corpus commands) or
`ws.ts` (for workspace commands), then re-register it in
`packages/opencode/src/index.ts` if it is a new top-level command.

**`lib/opencode.json.template`** is the single source of truth for which MCP servers exist. Editing it affects both indexing and retrieval.
5. **Run `make test`** — all tests must pass. Then `make build` to verify
the binary compiles cleanly.

---

## Runtime file locations

| Path | Purpose |
|---|---|
| `~/.config/opencode-workspace/config.json` | User config; auto-created with defaults if absent |
| `~/.config/opencode-workspace/tools.db` | SQLite corpus (265 tools when fully indexed) |
| `~/.config/opencode-workspace/sessions.jsonl` | Per-session telemetry; may not exist until first one-shot |
| `~/.config/ow/config.json` | User config — auto-defaults if absent |
| `~/.config/ow/tools.db` | SQLite tool corpus (`ow corpus index` writes here) |
| `~/.config/ow/sessions.jsonl` | Per-retrieval telemetry records |
| `~/.config/opencode/opencode.json` | MCP server list — read by `ow corpus index` |
| `~/.local/share/opencode/mcp.env` | MCP secrets (`KEY=value`, one per line) |
| `~/.config/opencode/opencode.json` | Global OpenCode config — read by this tool for permission merging |
| `~/.config/opencode/plugins/ow-tool-retrieval.js` | TUI first-message hook plugin; installed by `opencode-workspace install` |
| `/tmp/ow-<uuid>.json` | Temp per-session config; deleted after opencode exits |
| `~/.local/share/opencode/opencode.db` | OpenCode session/message database |
| `~/.cache/huggingface/` | ONNX model cache (~23 MB, auto-downloaded on first use) |
| `packages/opencode/dist/ow-linux-x64/bin/ow` | The compiled binary |

---

## Gotchas

- **No test runner**: `make test` checks help output only. `make smoke` is the real validation; requires a live indexed corpus.
- **`sqlite-vec` is optional**: absent → transparent fallback to brute-force in-process cosine search. Performance difference only.
- **`bun:sqlite` first, then `better-sqlite3`**: `db.js` tries `bun:sqlite`; the throw is caught. Do not remove the fallback.
- **Embedding text format must stay consistent**: `"<server> / <tool_name>: <description>"` — index and search must use the same string and same model. Mixing models silently produces wrong results.
- **Permissions are deny-only, server-level**: if any tool from a server is in top-K, all tools on that server stay accessible. User rules from `~/.config/opencode/opencode.json` are never overridden.
- **Permission key format**: `mcp_<server_name>_*` with underscores — server `brave-search-mcp-server` → `mcp_brave-search-mcp-server_*`.
- **All retrieval messages go to `stderr`**; opencode stdout is untouched.
- **`postinstall` runs `cmdInstall`**: `npm install` triggers dependency installation; each step fails with a warning rather than aborting.
- **`workspaces/`** at repo root is `.gitignored` — treat it as external; it is not part of this package.
- **PATH**: `cli.js` prepends `~/.local/bin` and `~/.opencode/bin` on every run; tools installed there are always found.
- **`make update`** only edits `package.json`; does not reinstall. Run `npm install -g .` manually after if you want the new binary version.
- **`docs/*.feature`** are documentation only — no step implementations exist.
- **`ALWAYS_ALLOWED` in `src/retrieval/permissions.js`**: servers listed here are never denied by the one-shot permission generator. Currently contains `tool-retrieval` so the on-demand search_tools MCP tool is always callable.
- **Plugin is global**: `ow-tool-retrieval.js` is installed into `~/.config/opencode/plugins/` (the OpenCode global plugin directory), not `~/.config/opencode-workspace/`. It fires for all opencode sessions, but soft-fails if the corpus is absent.
- **Plugin uses ES module syntax** (`export const`): OpenCode plugins are loaded by Bun (which supports ESM). The rest of this codebase uses CommonJS — do not mix them in the same file.
- **Feature files first** — if you skip step 1 above and go straight to
code, you will break the BDD contract and create tests that do not map
to any documented scenario.

- **`bun:sqlite` only** — we dropped `better-sqlite3` and `sqlite-vec`.
SQLite is always native Bun. All brute-force cosine search; no vector
extension. Fast enough for corpora ≤ ~5 000 tools.

- **BLOB type from `bun:sqlite` is `Uint8Array`** — not a Node.js
`Buffer`. `corpus.ts` uses `DataView` + `ArrayBuffer` for float32
packing; do not use `Buffer.readFloatLE`.

- **Embedding text format is a contract** — `"<server> / <tool>: <desc>"`
is used at index time AND at query time. Changing the format invalidates
the corpus and requires `ow corpus index --force`.

- **`ToolRetrievalPlugin` is in `INTERNAL_PLUGINS`** in
`packages/opencode/src/plugin/index.ts`. It runs in every ow session
automatically. To disable it for a specific session use `ow --pure`.

- **MCP config is user-owned** — `ow corpus index` reads MCP servers from
the user's `~/.config/opencode/opencode.json` (or `OPENCODE_CONFIG` env
or `.opencode/opencode.json`). There is no bundled template anymore.

- **`packages/opencode` has its own `AGENTS.md`** inherited from upstream.
Do not overwrite it. It contains Effect.ts and database conventions that
apply to the OpenCode internals.

- **Binary name in build output** — the build script outputs to
`dist/ow-linux-x64/bin/ow`. The smoke test in `script/build.ts` also
uses `ow --version`. If you change the name, update both.

- **All retrieval messages go to `stderr`** — `stdout` belongs to
structured output (`--json` mode) and is read by scripts.

- **Testability pattern** — injectable dependencies use underscore-prefix
options (`_searchFn`, `_corpusSizeFn`, explicit file paths). These are
test-only overrides; production code uses the real implementations.
52 changes: 21 additions & 31 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,42 +1,32 @@
# Usage:
# make install — install the package globally from this local repo
# make test — quick CLI sanity checks
# make smoke — end-to-end: index all MCP servers, assert top retrieval result
# make update — update pinned dependency versions to their latest releases
# make install — bun install (resolve all workspace dependencies)
# make build — build the ow binary for the current platform
# make dev — run ow in dev/interpreted mode (no compilation)
# make test — run unit tests
# make smoke — build binary + run corpus retrieval smoke test
# make update — bump @ow/workspace and ow versions

.PHONY: install test smoke update
.PHONY: install build dev test smoke update

BUN := $(HOME)/.bun/bin/bun
BINARY := packages/opencode/dist/ow-linux-x64/bin/ow

install:
npm install -g .
$(BUN) install

build:
$(BUN) run --cwd packages/opencode build -- --single --skip-embed-web-ui

dev:
$(BUN) run --cwd packages/opencode --conditions=browser src/index.ts

test:
npx cucumber-js
$(BUN) test

smoke:
smoke: build
@echo "=== Step 1: index MCP tool corpus ==="
node bin/cli.js index
$(BINARY) corpus index
@echo ""
@echo "=== Step 2: retrieval assertion ==="
node bin/smoke.js

update:
@node -e " \
const https = require('https'); \
const fs = require('fs'); \
function fetchLatest(repo, cb) { \
https.get( \
'https://api.github.com/repos/' + repo + '/releases/latest', \
{ headers: { 'User-Agent': 'opencode-workspace' } }, \
(res) => { let raw = ''; res.on('data', c => raw += c); res.on('end', () => cb(JSON.parse(raw).tag_name.replace(/^v/, ''))); } \
); \
} \
fetchLatest('anomalyco/opencode', (v) => { \
const pkg = JSON.parse(fs.readFileSync('package.json', 'utf8')); \
const prev = pkg.opencode ? pkg.opencode.version : 'none'; \
if (prev === v) { console.log('opencode already up to date: ' + v); return; } \
pkg.opencode = { version: v }; \
fs.writeFileSync('package.json', JSON.stringify(pkg, null, 2) + '\n'); \
console.log('opencode: ' + prev + ' → ' + v); \
}); \
"
$(BINARY) corpus retrieve "list GitHub pull requests" | grep -i github
@echo "Smoke test passed."
Loading
Loading