TTL caches, session pruning, REPL split + LM Studio provider abstraction by michaeldtimpe · Pull Request #2 · michaeldtimpe/luxe

michaeldtimpe · 2026-04-27T19:16:30Z

Summary

26 commits across multiple sessions. The largest themes:

Provider abstraction (today): new luxe_cli/providers/ package with a BackendProvider protocol and concrete Ollama / LMStudio / OMLX / generic OpenAI-compat impls. agents.yaml gains a top-level providers: map and per-agent provider: field; dispatch routes through cfg.resolve_endpoint(). Migration to LM Studio is now a one-line YAML diff per agent — playbook in luxe/luxe_cli/README.md.
Telemetry honesty (today): make_backend derives kind from the resolved URL instead of hardcoding "mlx". Session JSONL events get tagged with provider + base_url via Session.bind_backend(...). Multi-provider health check on REPL start (warns per-down provider instead of a single Ollama-only ping).
Repo hygiene (today): cli → luxe_cli package rename (47+ import sites) to avoid the generic-name shadowing risk. New .github/workflows/luxe-tests.yml runs pytest on push/PR. Pinned [tool.pytest] testpaths = ["tests"] so collection no longer trips on vendored repos under personal_eval/cache/. Tightened luxe/results/ gitignore against future noise.
oMLX migration + evaluation (earlier): Qwen agents moved to oMLX with composite-verdict ADOPT decision; cc-canary metrics; browser tool; secrets persistence via ~/.luxe/secrets.env.
Autonomous test infra (earlier): day-long supervised-runner pipeline with backend recovery, progress tail, and per-phase aggregation.

Test plan

uv run pytest passes (142 tests, including 4 new live LM Studio smoke tests)
CI green on f18e586
luxe --help works after the package rename
Live: LMStudioProvider().list_models() against running LM Studio returns 9 loaded models with correct context lengths
Manual: open REPL, confirm multi-provider health-check banner pings oMLX + LM Studio + Ollama as configured
Manual: flip one agent to provider: lmstudio per the playbook, confirm dispatch routes correctly + session JSONL carries "provider": "lmstudio"

🤖 Generated with Claude Code

… tool Multi-epic build-out from a 2026-04-24 session. E1 — cc-canary-style behavioral metrics - reads_per_edit + tool_loop_ratio per-subtask + totals - composite_health z-scored over trailing 10-row window - INFLECTION flag in `bench show` when health diverges by >1.5σ - backwards-compat for old history rows (renders missing fields as —) E2 — browser tool (CDP-bridged, allowlist-gated) - browse_navigate + browse_read in cli/tools/browser.py - pychrome optional dep, lazy headless Chrome via brew google-chrome - DEFAULT_BROWSER_ALLOWLIST + LUXE_BROWSER_ALLOWLIST env override - wired into research + lookup agents (with system-prompt guidance) - ToolName Literal extended in cli/registry.py E3 — oMLX integration test suite - BackendKind extended; harness/server.py _yield_omlx + _resolve_omlx_model - scripts/omlx_healthcheck.py (Phase 0 install + auth + model resolution) - benchmarks/prefix_cache_decay.py (3 prefix sizes × 10 shared-prefix queries) - scripts/omlx_verdict.py — P3 redefined to absolute TTFT after measurement showed cold/warm cache_benefit_ratio is undefined under SSD-paged caching - LUXE_BACKEND_OVERRIDE env var in cli/backend.py - OMLX_API_KEY auto-loaded by make_backend; sent harmlessly to all backends - AB harness: --config-suffix, --phase, --omlx-url, per-bench try/except E4 — speculative decoding - scripts/spec_decoding_verdict.py with synthetic perfect/terrible self-tests - scripts/llamacpp_spec_test.py (baseline vs spec-only on llama-server) - scripts/omlx_configure_dflash.py (cookie-auth admin client + draft pull) - finding: spec decoding is conditional, not a default win — DFlash and llama-server-spec optimize for opposite output-length regimes E4 Phase B migration (data-driven, replaces original "→ llama-server +spec" plan) - code → Qwen2.5-Coder-14B-Instruct-MLX-4bit on oMLX (+56% decode, +6.7pp pass rate) - review → Qwen2.5-32B-Instruct-4bit on oMLX (+54% decode, parity 93.3%) - refactor → same as review - general/lookup/image/router stay on Ollama (low-leverage workloads) - writing stays on llama-server (Gemma 3 native tool-call format) Docs - README Status: backend split, browser tool, bench-history metrics - cli/README: oMLX install + OMLX_API_KEY + per-agent backend table - LESSONS.md: 5 new sections (engine win > SSD cache, cold/warm metric collapse, conditional spec decoding, bearer-vs-cookie auth split, ToolName Literal gotcha, sticky benchmark slots) - .gitignore: .claude/ + luxe/scripts/results/ stray dir 113/113 tests passing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The 2026-04-24 oMLX migration broke `luxe` REPL invocations that didn't have OMLX_API_KEY in the shell env — calls to /v1/chat/ completions returned 401 silently mid-task. Add a small dotenv-style loader so the key persists across shells without requiring users to edit .zshrc. - cli/secrets.py: load_secrets() reads ~/.luxe/secrets.env (KEY=VALUE per line, # comments, optional quotes) into os.environ. Existing values always win — `OMLX_API_KEY=foo luxe` still works as a one-shot override. - cli/main.py: load_secrets() called before `from cli import repl` so the env is populated before any backend module captures it. warn_missing_omlx_key(cfg) prints a yellow [!] banner when an agent's endpoint references oMLX (port 8000) but no key is set. - daily_driver/secrets.env.example: template covering OMLX_API_KEY, LUXE_BACKEND_OVERRIDE_URL, LUXE_BROWSER_ALLOWLIST, LUXE_CACHE_TTL_S. - daily_driver/install_luxe.sh: stages the template at ~/.luxe/secrets.env (chmod 600) when neither file exists, and warns at install end if OMLX_API_KEY is still placeholder/missing. - cli/README: documents the precedence (file < shell export < per-invocation prefix) + install steps. 113/113 tests still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Real /review on the oMLX-served 32B-Instruct regressed catastrophically on the multi-turn workflow, even though the morning's HumanEval+ benchmark showed +54% decode tok/s. Same task, same target repo, same agent definition: subtask Ollama wall oMLX wall notes ------- ----------- --------- ----- sub 01 1m 11s 1m 07s parity (~7k prompt) sub 02 2m 04s 2m 53s +40% (~10-13k prompt) sub 03 3m 49s total >8m to first tool call catastrophic Root cause: /review's sub 03 receives the concatenated outputs of sub 01 + 02 via _augment_with_prior, so its prompt is ~13k tokens at start. The HumanEval+ benchmark prompts are ~1k. oMLX 32B's TTFT at 1k was 1.6× slower than Ollama; at 13k it was 6× slower. Decode-rate wins can't recover the prefill cost on tool-heavy multi-turn loops. Diagnosis was visible only in the per-tool-call event log (~/.luxe/tasks/<id>/log.jsonl). The bench harness's metrics didn't capture this regime — single-turn benchmarks miss the prompt-growth shape that orchestrator workflows produce. `code` (14B, smaller prompt distribution, single-turn-ish workload) stays on oMLX where the HumanEval+ pattern matches reality. LESSONS.md gains a "Single-turn benchmarks miss the multi-turn growth regime" section; README + cli/README updated to reflect the partial rollback. 113/113 tests still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…ackend Same-day reversal: the morning's rollback to Ollama (commit c6211a6) was based on a single oMLX run that looked catastrophically slow at sub 03 (~13k prompt → 8m to first tool call). Re-running on the rolled-back Ollama setup showed the same multi-turn slowness — the 8-minute gap is inherent to a 32B serving a ~13k prompt with multi- paragraph output, not an oMLX-specific TTFT regression. The morning's faster Ollama timing was likely benefiting from in-process prefix- cache state that didn't survive between sessions. Restore the migration: - review → Qwen2.5-32B-Instruct-4bit on oMLX - refactor → Qwen2.5-32B-Instruct-4bit on oMLX This recovers the +54% decode tok/s the AB sweep measured (12.30 vs 7.99 on HumanEval+, parity pass rate). The wall-time tax at ~13k prompts is paid by both backends; oMLX wins by +54% on the decode portion regardless. LESSONS.md updated to reflect the corrected attribution. The methodological lesson stands — single-turn benchmarks don't predict 13k-prompt behavior — but the directional conclusion was wrong, and the LESSON now records the premature-rollback episode + the rule that fixed it: re-run the same-session A/B before reversing a migration the benchmark earned. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…view sweep) Builds the framework for a hands-off overnight backend evaluation across ollama × oMLX × LM Studio × llama-server, capturing the multi-turn /review regime that single-turn benchmarks missed. Designed to run 6-12 hours unattended with per-phase try/except + signal-based timeouts + persistent state.json so partial runs are still useful. Backend wiring: - harness/backends.py: BackendKind extended to include "lmstudio"; Backend.__post_init__ auto-loads LMSTUDIO_API_KEY for kind="lmstudio" (same auth pattern as oMLX). - harness/server.py: _yield_lmstudio + _resolve_lmstudio_model. Mirrors _yield_omlx — externally-managed daemon, fuzzy model resolution from /v1/models, optional Bearer auth. - scripts/run_ab_benchmark.py: lmstudio added to _CONFIG_IDS; --lmstudio-url CLI flag (default http://127.0.0.1:1234). New scripts: - scripts/lmstudio_healthcheck.py: Phase-0 probe — endpoint reachability, required models present, chat completion round-trip, best-effort spec- decoding API probe (LM Studio's draft-model support is undocumented). - scripts/run_overnight.py: top-level orchestrator. Six phases: preflight → synthetic baseline → spec decoding → 3-repo /review sweep → DFlash for writing/calc → verdicts. Per-phase signal timeout, no interactive prompts, --dry-run + --skip-phase + --resume flags. - scripts/composite_verdict.py: aggregates omlx_verdict + spec_decoding_verdict + Phase 3 multi-turn data into one per-agent recommendation table. Self-test covers 4 review-decision cases + 3 code-decision cases. Test corpus: - elara, never-say-yes, neon-rain cloned to luxe/. Diverse (Python / Python+Docker / JavaScript-heavy). All gitignored. .gitignore: .claude/, scripts/results/ stray dir, never-say-yes, neon-rain, results/overnight_*/. 113/113 tests pass. Dry-run produces clean preflight.json with all backends + repos accounted for. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…nv var scripts/overnight_tail.py: live-tail an in-progress overnight run. Watches three sources concurrently: - results/overnight_<ts>/state.json — phase transitions - results/overnight_<ts>/<phase>.log — active phase stdout - ~/.luxe/tasks/<id>/log.jsonl — per-tool-call events for any running /review (Phase 3 spawns these as background processes) ANSI color (auto-disabled when stdout isn't a TTY), heartbeat after N seconds of quiet, attaches with a small rewind so the user sees recent context on first connect (not just net-new appends). Env var rename: LMSTUDIO_API_KEY → LM_API_TOKEN. Per LM Studio's official auth docs, LM_API_TOKEN is their env-var name. Backward compat: LMSTUDIO_API_KEY still works as a fallback. Auth is OFF by default in LM Studio's local server, so neither is required for the common case. Files: harness/backends.py + harness/server.py + scripts/lmstudio_ healthcheck.py + daily_driver/secrets.env.example all updated to prefer LM_API_TOKEN, fall through to LMSTUDIO_API_KEY. 113/113 tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Three related fixes for cross-backend orchestration: - _BACKEND_OVERRIDE_URLS now includes lmstudio (port 1234). Without this, LUXE_BACKEND_OVERRIDE=lmstudio silently fell through to the agent's default endpoint (oMLX :8000), which 404'd on the LM Studio model tag. Discovered when every lmstudio /review sub-chunk in the 2026-04-26 overnight produced "404 Not Found" at :8000. - New LUXE_MODEL_OVERRIDE env var redirects the model name alongside the URL. The agents.yaml model name (e.g. Qwen2.5-32B-Instruct-4bit for review/refactor) is the oMLX-internal tag; Ollama and LM Studio serve the same weights under different names. Without this, a URL redirect alone hits the wrong tag on the target backend. - New `ignore_override=True` on make_backend opts out of both URL and model overrides. Planner uses it: the router agent's tiny model (qwen2.5:7b-instruct) is only reliably tagged on Ollama, so when an overnight run redirects the WORKLOAD agent (review/refactor) to oMLX/LM Studio, the planner must keep its own routing intact. Pre- fix symptom: oMLX runs produced 1-subtask degenerate plans because the planner's qwen2.5:7b-instruct call to oMLX returned text the JSON extractor couldn't parse, falling back to the single-task path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Add a per-agent_loop signature counter: when the same (tool name, canonical args) signature appears LOOP_REPEAT_LIMIT (=3) times, the 4th and later identical calls are refused with an explicit ERROR result instructing the model to stop and pick a different action. Motivated by LM Studio's Qwen 32B looping on identical tool calls 20 times in a row in long-context multi-turn /review subtasks (2026-04-26 data: every (repo × lmstudio) sub-chunk burned step budget on identical-tool loops). The error returns in-band as a role=tool message — much stronger signal than a side-channel system message, which an earlier weaker version proved the model ignores. Ollama and oMLX never trip this guard in normal use, so it's a no-op for the working backends. Smoke-test with the previous weaker (nudge-only) version on neon-rain × lmstudio produced 3/7 subtasks done (vs 2/7 baseline) — the refuse version is in but unverified end-to-end. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Three changes that ship together because the data flow connects them. run_overnight.py - Per-phase backend liveness probes (_probe_backend, _wait_for_backend) before synthetic_baseline / spec_decoding / multi_turn_reviews / dflash_long_output. preflight may have run hours ago; oMLX could have crashed since (see project_omlx_metal_crashes memory). The probe waits up to 5 min for launchd KeepAlive to recover the backend before giving up on a slot. - Sanity guard on multi_turn_reviews: when a /review task completes with wall_s<60 and subtasks_done==0, re-probe and record status="backend_died_mid_run" instead of letting it look like a clean done. Catches the 09:15:29 incident where every (repo, backend) launched at the same instant against a backend that had crashed minutes earlier. - New CLI flags --only PHASE, --repo NAME, --backend NAME so a single (repo, backend) chunk can be initiated and supervised on its own. --only requires --resume except for preflight. - Pass repo URL (not local path) to start_review_task; the path-as- URL bug was making resolve_repo bail with "origin does not match" for every multi_turn run. - LUXE_MODEL_OVERRIDE wired per backend (qwen2.5:32b-instruct for ollama, qwen2.5-32b-instruct for lmstudio, none for omlx). - Synthetic_baseline budget bumped 90 -> 120 min (2026-04-26 hit the 90-min limit at 95% data captured). - Verdicts phase now invokes aggregate_multi_turn.py first. aggregate_multi_turn.py (new) - Walks ~/.luxe/tasks/T-…/state.json, joins to multi_turn_reviews.log by chronological proximity to extract (repo, backend) labels for each task. Writes multi_turn_runs.jsonl. Necessary because each --only multi_turn_reviews invocation overwrites state.json's result.runs with just that sub-chunk's record — the jsonl is the durable cross-(repo, backend) view. composite_verdict.py - _load_multi_turn now prefers multi_turn_runs.jsonl, falls back to state.json. When multiple runs exist for the same (repo, backend) — pre-fix degenerate plus post-fix real — keep the one with most subtasks_done, ties broken by latest started_at. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Operational scaffolding from the 2026-04-26 overnight investigation. run_overnight_supervised.sh - Walks the 6 overnight phases interactively; multi_turn_reviews fans out into 9 sub-chunks (3 repos × 3 backends) so each is individually supervisable. Honors $TS env var to resume into an existing results dir without re-running preflight. run_overnight_catchup.sh - Unattended re-runner for the slots that produced no usable data the first time: synthetic_baseline patch, 3 omlx + 3 lmstudio multi_turn sub-chunks, then verdicts. run_lmstudio_recheck.sh - Targeted re-runner for just the LM Studio sub-chunks after the loop guard fix. Less needed now that the smoke test showed only partial recovery — keeping for future probe rounds. tail_progress.sh - Live overview of the catchup run. Refreshes every 5s with the last few catchup.log lines and per-subtask status of the most recently created /review task. Alias-bypasses ls (user has eza). probe_lmstudio_{tools,review,stream}.py - Three diagnostic probes used to confirm LM Studio's tool-call protocol IS correctly handled at the harness layer — basic 2-turn, full 17-tool review system prompt, and raw SSE stream dump. All succeed. The loop bug is reachable only inside the agent loop's long-context multi-subtask conversation. See project_lmstudio_loop memory for follow-up direction. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Aggregator dedupe used to keep the LATEST run when (repo, backend) had multiple equally-complete entries. That biased toward warm-cache runs against backends that always cold-start (oMLX under launchd KeepAlive). 2026-04-26 elara × ollama had a 56-min cold first run plus a 31-min warm retry; keeping the warm one made the median unfair to oMLX. New rule: tie-break on EARLIEST started_at. Cold-cache wins. Headline impact on 2026-04-26 review/refactor verdict: before: oMLX 36.4m vs Ollama 31m → +15.6%, medium confidence after: oMLX 36.4m vs Ollama 46m → -21.7%, high confidence oMLX is no longer just "within tolerance" — it's measurably faster on real multi-turn /review wall, matching the synthetic-baseline decode lead (1.43×). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

End-to-end summary of the oMLX vs Ollama vs LM Studio investigation: the bugs uncovered, the verdict numbers (oMLX -21.7% on multi-turn wall, high confidence), the harness improvements that landed, and the open items for follow-up. Self-contained — written so a fresh session can pick this up cold. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Backed by 2026-04-26 overnight verdict (oMLX 21.7% faster than Ollama on multi-turn /review wall, 1.43× decode). Bundles together: - agents.yaml: router/general/web_search/research/writing/image/calc all point at oMLX (Qwen2.5-{7B,32B}-Instruct-4bit, gemma-3-27b-it-4bit). Cross-backend comparison harness env vars (LUXE_BACKEND_OVERRIDE / LUXE_MODEL_OVERRIDE) kept; ignore_override is now unused in practice but left in cli/backend.make_backend for future meta-orchestration. - cli/tasks/plan_cache.py: per-(repo, mode) JSON cache with 24h TTL, wired into planner via cache_key kwarg. Addresses non-deterministic cross-backend decompositions (SESSION_REPORT open item #3). main.py exposes --no-plan-cache; review.start_review_task takes use_plan_cache. - main.py / router.py: honor agent.endpoint over cfg.ollama_base_url so the oMLX migration takes effect without env-var override. - LM Studio dropped: probe scripts moved to scripts/archive/, removed from run_overnight.py / run_ab_benchmark.py / supervised + catchup runners (Qwen 32B tool-loop bug is downstream-fixable only). - scripts/launchd/com.luxe.omlx-restart.plist: daily 4am brew restart to dodge the latent MLX/Metal gpu::check_error crash. Plus an in-run proactive_restart guard in run_overnight.py that recycles oMLX before any multi-turn slot if uptime ≥4h. - scripts/probe_omlx_swap.py / probe_omlx_gemma.py / verify_jsonl_schema.py: one-shot diagnostics (model-swap latency for single-server vs two-server topology; gemma sanity; jsonl schema verifier for open item #4). - gitignore: nohup.out / overnight.log / catchup.log. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Each agent now carries one fixed num_ctx value chosen against the RoPE/PE scan and the workload. With one model loaded at a time on oMLX and known hardware budget, tier-driven ctx sizing was solving a problem we don't have. Wall-time tiering stays — that one still depends on input size. Values: - router/image: 8k (tiny prompts) - general/lookup: 16k (chat + snippets, 7B) - code/calc/research/review/refactor: 32k (Qwen2.5 32k native; tool-calling correctness degrades at extended ctx — LM Studio Qwen-32B loop bug as precedent) - writing: 131k (gemma-3-27b full native; long-form drafts) Plumbing removed: - _resize_for_cwd in cli/agents/code.py (dead with fixed ctx). - num_ctx field from BudgetDecision and _TIER_TABLE; size_budgets now returns (tier, task_max_wall_s, rationale) only. - Task.num_ctx_override and Subtask.num_ctx_override (planner never emitted the latter). load() pops the legacy key so older state.json files still rehydrate. - num_ctx_override CSV column in scripts/summarize_runs.py. - The num_ctx branch of _cfg_with_task_overrides; max_tokens_per_turn + analyzer_languages branches stay. Note: extra_body.options.num_ctx in cli/agents/base.py is preserved — Ollama-effective only; oMLX/llama-server honor their server-side --max-kv-size instead. Set --max-kv-size 131072 on the brew-managed mlx_lm.server so writing can use its full window. Docs (AGENTS.md, ARCHITECTURE.md, LESSONS.md, README.md, cli/README.md) updated; TurboQuant link cited in LESSONS.md as KV-quant background. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…ignore Bundles four review-driven cleanups: - Rename `luxe/cli/` → `luxe/luxe_cli/` to avoid the generic-name collision risk with any dep that ships a `cli` module. Updates 47+ imports, the typer entry point, the hatch packages list, and doc path references. - Add `.github/workflows/luxe-tests.yml` running `uv run pytest -x` on Python 3.11 / ubuntu-latest for any push or PR touching `luxe/**`. Status badge added to the root README. - Pin `[tool.pytest.ini_options] testpaths = ["tests"]` so pytest no longer trips on test files inside vendored repos under `personal_eval/cache/`. The five "pre-existing compression-repo failures" called out in prior commit messages were collection errors, not real failures — the suite was always green when scoped. - Tighten `.gitignore` under `luxe/results/`: catch *.log, *.tmp, *.partial, and replay_inputs/ artifacts that scripts may drop outside the directories already ignored. Tracked outputs (REPORT/VERDICT files, history.jsonl, eval markdown) stay tracked. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

actions/checkout@v4 (Node 20) and astral-sh/setup-uv@v3 (also Node 20) trip the Sept 2026 Node.js 20 deprecation annotation. Pin to checkout@v5 and setup-uv@v8.1.0 (latest tagged release; astral-sh publishes only fully-qualified version tags, no floating major). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

make_backend hardcoded kind="mlx" regardless of where it was actually pointing, so telemetry/A-B comparisons across providers couldn't tell Ollama runs from oMLX runs from llama-server runs. The Backend dataclass already supports the right BackendKind literal — the factory just wasn't using it. Adds _kind_for_url(url) → BackendKind that matches the resolved base_url against _BACKEND_OVERRIDE_URLS (now including lmstudio at :1234 in prep for the upcoming migration). Falls back to "ollama" for unknown URLs to preserve the previous default's intent. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Defines the introspection surface that varies per provider — ping, list_models, context_length, parameter_size, prewarm, pull_stream. Backend (chat-completion transport) is already provider-agnostic because every supported provider speaks /v1/chat/completions; this protocol covers the parts that don't. Methods that don't apply to a given provider (e.g. pull_stream on LM Studio, where downloads are GUI-driven) raise NotImplementedError. Concrete Ollama and LM Studio implementations land in #7. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Concrete provider classes behind the BackendProvider protocol added in c02339a. OllamaProvider is a thin wrapper around the existing luxe_cli.backend module functions (no logic duplication; consumer migration to the new instance API happens in #9). LMStudioProvider is new — uses /v1/models for listing, /v1/models/{id} for metadata, and shares the TTL caches in luxe_cli.backend. estimate_kv_ram_gb returns None on LM Studio: /v1/models/{id} doesn't expose head_count / block_count the way Ollama's /api/show does, so we'd be guessing. pull_stream raises NotImplementedError because LM Studio model downloads are GUI-driven. get_provider(kind, base_url=None) is the single construction point. Defaults base_url to the canonical local port from _BACKEND_OVERRIDE_URLS so callers can `get_provider("lmstudio")` and get the right port without repeating themselves. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

LuxeConfig.providers is a dict[str, ProviderConfig] of named backend endpoints (base_url + kind). AgentConfig.provider names one of those keys. Migrating the writing-agent's separate-endpoint hack into the same shape — dispatch (#9) will read resolve_endpoint() instead of agent.endpoint directly. resolve_endpoint precedence: 1. agent.endpoint (legacy direct URL — explicit wins for migration) 2. providers[agent.provider] 3. providers[default_provider] 4. ollama_base_url with /v1 stripped (legacy fallback) agents.yaml gains a providers map declaring ollama, omlx, lmstudio, llamacpp at their canonical local ports + default_provider: omlx. Existing per-agent endpoint: keys keep working unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Replace every `agent.endpoint or cfg.ollama_base_url` with `cfg.resolve_endpoint(agent)` so dispatch goes through the new provider-aware resolution: explicit endpoint > providers[provider] > providers[default_provider] > legacy ollama_base_url. Behavior unchanged on the current config (every agent still resolves to oMLX :8000 via its existing endpoint: field). The migration to provider: keys is now incremental — flip one agent at a time without touching code. Side fix: tasks/clarify.py was building the router model against cfg.ollama_base_url (:11434, Ollama port) while the router itself runs on oMLX (:8000), so it would have hit the wrong endpoint. Now correctly resolved through the router agent's config. Touched: main.py, router.py, runner.py, tasks/{clarify,planner}.py, repl/{core,status}.py. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The /models, /variants, and /pull commands hardcoded cfg.ollama_base_url (legacy field, port 11434) even though every agent runs on oMLX :8000. Switch to a _default_provider(cfg) lookup that returns the right provider instance for cfg.default_provider, falling back to Ollama on the legacy URL when default_provider is unset. Behavior changes: - /models: lists from the active provider (oMLX/LM Studio/Ollama). - /variants on a non-Ollama provider: shows just the live list of loaded models. The MODEL_VARIANTS catalog is Ollama-tag-specific and would be misleading. - /variants on Ollama: unchanged. Column headers retitled "installed (live)" / "available (catalog)" so the source is explicit. - /pull on a non-Ollama provider: prints a friendly message instead of trying (LM Studio downloads are GUI-only; oMLX has no pull). Refactor: extracted OpenAICompatProvider as the base for any /v1/models-style provider. LMStudioProvider, OMLXProvider, and the generic llamacpp/mlx fallbacks all use it. Each subclass picks its auth env vars (LM_API_TOKEN vs OMLX_API_KEY). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Old behavior: hardcoded ping("http://127.0.0.1:11434") (Ollama port) on REPL start; failed-fast or proceeded based only on Ollama health. Wrong now that agents target oMLX :8000 and an LM Studio agent could target :1234. New behavior: _check_provider_health(cfg) iterates the unique endpoints across enabled agents (deduped — same provider only pings once even if 9 agents share it), and returns reachable + unreachable buckets. Bail only when nothing is reachable; otherwise warn per-down provider and proceed so reachable agents still work. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Adds Session.bind_backend(kind, base_url) context manager. Every session.append() inside the block carries provider + base_url fields so cross-backend A/B analysis can filter by which provider served each turn — required when LM Studio joins Ollama and oMLX as concurrent providers (see #13). Wiring: - runner.dispatch wraps the specialist call with bind_backend(...) so the agent loop's many session.append callsites pick up tagging without per-callsite plumbing. - router.route mutates _backend_kind/_backend_url before the routing turn so router decisions get tagged too. Outer dispatch's bind still restores its own values via the context manager. Backwards-compatible: untagged events (no active bind) get no extra fields, so old session JSONLs and any non-router/non-dispatch appenders still parse identically. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

LM Studio's standard /v1/models returns minimal data ({id, object, owned_by}). The rich metadata (max_context_length, arch, quantization, capabilities) lives at /api/v0/models/{id}. Override LMStudioProvider._model_info to hit v0; parameter_size now also parses "27b" / "32b" out of the model id when /api/v0 doesn't expose the count directly. tests/test_lmstudio_smoke.py: - 2 static tests (always run): provider:lmstudio agent dispatches with kind="lmstudio" + correct base_url, and session events get tagged correctly. - 2 live tests (skipped if :1234 is down): real list_models() and context_length lookups against the running LM Studio server. Verified live against 9 loaded models — context lengths and parameter sizes resolve cleanly. luxe_cli/README.md gains a "Provider migration" section showing the exact one-line YAML diff to flip an agent + the validation recipe. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

ARCHITECTURE.md: - Tree diagram lists the new providers/ package + its 4 modules. - "Backend abstraction" section split into transport (Backend) and introspection (BackendProvider) layers; documents the URL→kind derivation (was hardcoded "mlx"). - "Configuration" section names providers map, default_provider, and the per-agent provider field; documents resolve_endpoint() precedence and Session.bind_backend tagging. - Fix typo: luxe_luxe_cli → luxe_cli. AGENTS.md "adding a new agent" step 4 mentions provider: as the preferred per-agent override (endpoint: kept as legacy). LESSONS.md oMLX migration section points at the new provider: syntax and references the LM Studio playbook. Root README.md mentions the full provider matrix (Ollama + oMLX + LM Studio + llama.cpp) instead of just Ollama + llama.cpp. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

/review and /refactor previously cloned into Path.cwd() — running luxe from the project root scattered repo clones (elara, neon-rain, zoleb, never-say-yes) at /luxe/<name>/. They were gitignored individually but littered the project tree. LuxeConfig.local_cache_dir (default "local-cache") plus LuxeConfig.cache_dir() helper expand the path (absolute, ~/, or relative to cwd) and ensure the directory exists. Both resolve_repo callsites in luxe_cli/repl/review.py and luxe_cli/review.py now pass cfg.cache_dir() instead of Path.cwd(). .gitignore: replace the per-repo entries (/ecliptic/, /luxe/zoleb/, /luxe/elara/, etc.) with a single `local-cache/` rule. Sibling projects /ecliptic/ and /warrens/ that aren't luxe-managed get their own short ignore block. Cleaned up the existing local tree: moved luxe/{elara,neon-rain, never-say-yes,zoleb} → luxe/local-cache/. Nothing was tracked in those dirs, so the move is a pure filesystem reorg. agents.yaml documents the new field with a one-line comment + points users at ~/.luxe/cache as an alternative. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

When users activate a project venv ($VIRTUAL_ENV) different from luxe's install, they often can't tell whether their venv has every dep luxe needs. luxe_cli/venv_check.py probes the active interpreter once at startup with a -c that imports each runtime dep, and prints one actionable line if any are missing — including the exact `uv pip install -e .` command to install luxe into their venv. Silent when: - No $VIRTUAL_ENV is set. - Active venv is the same one luxe runs from. - Active venv has every dep importable. - LUXE_NO_VENV_CHECK=1. Deliberately does NOT re-exec into the user's venv. If their venv has an older luxe_cli installed, that would silently take over — PR #2 thread covers the trade-off. The warning surfaces the choice without making it for them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…ning Updates RESUME.md and lessons.md to reflect the 2026-05-06 session work: the v1 paired-mechanism n=75 succeeded on every floor except new_file_in_diff = 0 (8 escape paths under write_pressure actuation), and the Forbids tightening (commit e062bab) is the v1.5.0 ship gate. RESUME.md now points at the v2 rerun as the remaining work: - Steps 1-3 (variance probe / Step 2 wiring / first n=75) marked done - New Step 1 launches v2 with --no-write-pressure ablation hatch + the paired-mechanism env wiring already baked in (no shell munging) - Step 3 ship-floor check is a HARD blocker; stop conditions explicit (no third round of broad-glob whack-a-mole; escalate to creation-only forbids if v2 still leaks) - v1.6 backlog re-ranked: early-bail intervention #1 (addresses 10 of 14 v1 empty_patch), creation-only forbids #2, retrieval #3 lessons.md adds a full v1 paired-mechanism postmortem entry covering: - The 3 ship-relevant commits (paired-mechanism env, gpg-sign override, Forbids tightening) - v1 n=75 numbers vs predictions vs baseline - Diagnosis of the 8 escape patterns into 3 clusters - Acknowledged broad-glob risk + path forward via creation-only forbids - Four durable rules of thumb (paired-mechanism category, bench-side commit-signing immunity, whack-a-mole stops at iteration 2, failure-mode analysis-driven priorities) Plan + failure analysis: ~/.claude/plans/humble-prancing-patterson.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Two anchors landed back-to-back on 2026-05-10/11: - Raw mode (regression check, ~6h): 948/1240 = 76.45%, +0.16pp vs pre-SpecDD 76.29%. No infra drift across the v1.4.1 → v1.6 ship. - Agent mode (one-shot v1.6 datapoint, 8.47h): 1038/1240 = 83.71%, +7.26pp vs raw v1.6. Parallel cliff +17pp (parallel) and +16.5pp (parallel_multiple) is the dominant lift; irrelevance regresses −6.25pp from loop framing priming tool-eagerness. BFCL agent adapter does NOT wire `.sdd` injection or the Lever 1 spec validator yet — the +7.26pp is loop-vs-single-shot, not SpecDD-driven. That wiring is now v1.7 priority #2 (slotted after early-bail intervention), with an explicit baseline to beat: agent 83.71% total, parallel_multiple 64.5%, irrelevance 85.83%. Signal that Lever 1 is doing real work in BFCL: parallel_multiple ↑ AND irrelevance recovers toward 92%. Side lesson captured: BFCL probes on the prefix of a non-randomized subset aren't representative. parallel_multiple n=50 probe showed 86%; full n=200 was 64.5% — a 21.5pp methodological gap. Future probes are either random-sampled or framed strictly as infrastructure validation. Files: README.md (BFCL section gets the three-anchor table + agent-mode caveat), RESUME.md (v1.7 priorities reordered; items 1 + 2 from "v1.6 loose ends" marked DONE), lessons.md (2026-05-11 entry: parallel-cliff finding, irrelevance regression, probe-prefix lesson). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Tag v1.6.1 at 0a964bf (local only, not pushed). 652 tests passing. Patch on v1.6.0 capturing substrate hardening from m5max_moe bake-off, SpecDD Lever 2 extended into maintain_suite, and BFCL v3 anchors (raw 76.45%, agent 83.71%). v1.7 priorities #1 (early-bail) and #2 (BFCL Lever 1 wiring + irrelevance abstain) are next. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Two new RequirementKinds in src/luxe/spec.py: - expects_zero_calls: agent must emit zero tool calls (irrelevance) - min_tool_calls: agent must emit >= min_matches calls (parallel*) src/luxe/spec_validator.py: - validate() gains a tool_calls kwarg; diff parsing is conditional on the spec actually containing a diff-predicate requirement. - _eval_expects_zero_calls + _eval_min_tool_calls evaluators. - _compiled_pattern lru_cache so regex predicates re-use compiled patterns across calls. Latency contract: <1ms p95 in agent-loop hot path (verified by test_validator_latency_under_1ms_p95). src/luxe/agents/loop.py: - run_agent() gains a `spec: Spec | None = None` parameter. - Mid-loop reprompt at the same checkpoint as write_pressure / early_bail: fires expects_zero_calls reprompts after the first tool call (immediate violation). - Loop-break reprompt for min_tool_calls: when the model emits a text-only response with fewer than min_matches calls, inject a reprompt and resume the loop (each requirement fires at most once). - Suppression hook: when the spec contains any expects_zero_calls requirement, LUXE_EARLY_BAIL and LUXE_WRITE_PRESSURE are forced off. Both are tool-eagerness amplifiers that would corrupt the abstain signal. benchmarks/bfcl/adapter.py: - _spec_from_problem(problem, category, ground_truth): derives a Lever 1 Spec from BFCL GT structure. Irrelevance -> expects_zero_calls; parallel/parallel_multiple with GT length >= 2 -> min_tool_calls(n). Single-call categories return None (no Lever 1 needed). - _system_prompt_for(category): per-category prompt override. Irrelevance gets an abstain-tolerant prompt ("decline and explain"); everything else falls back to the default tool-eagerness phrasing. - run_problem_agent() now accepts `category` + `ground_truth` and threads spec + system_prompt to run_agent(). benchmarks/bfcl/run.py: thread category and ground_truth to run_problem_agent. Fairness — Lever 1 leaks the *count* of expected calls from BFCL GT structure (not the values). Documented inline in adapter._spec_from_problem + RESUME.md raw-vs-agent caveat + project_external_benchmark_program.md. Post-v1.7 raw-vs-agent deltas measure [loop + Lever 1] vs [no loop], not loop alone. 29 new tests: 4 spec round-trip, 7 validator (incl. <1ms p95 + lru_cache hit-count + diff-skip), 9 BFCL adapter (system prompts + _spec_from_problem + end-to-end), 9 loop spec gate (reprompt fires, fire-once, suppression of early_bail/write_pressure). 687 passing. Plan: ~/.claude/plans/bubbly-plotting-gosling.md Phases C.1-C.6. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

michaeldtimpe and others added 28 commits April 24, 2026 21:01

michaeldtimpe merged commit ff6eab7 into main Apr 27, 2026
2 checks passed

michaeldtimpe deleted the luxe/ttl-caches-session-pruning-repl-split branch April 27, 2026 19:35

michaeldtimpe mentioned this pull request Jun 1, 2026

fix(ablation): harness import path, cell fault-isolation, harness stats #3

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TTL caches, session pruning, REPL split + LM Studio provider abstraction#2

TTL caches, session pruning, REPL split + LM Studio provider abstraction#2
michaeldtimpe merged 28 commits into
mainfrom
luxe/ttl-caches-session-pruning-repl-split

michaeldtimpe commented Apr 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

michaeldtimpe commented Apr 27, 2026

Summary

Test plan

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant