experimental: Add external host driver runtime

[steve-calvert-glean]

Summary

This PR adds an experimental external_host eval mode that runs the existing dataset, expectation, iteration, and reporter flow through real MCP host applications instead of only the SDK-based mcp_host simulation path.

The implementation is capability-composed rather than host-runner-specific:

• Adds structured external host driver identity, for example anthropic.claude.cowork.desktop-app.macos.
• Adds a runtime that loads host configs into composed control, input, completion, trace, and normalize capabilities.
• Adds built-in Claude Desktop drivers for separate Chat and Cowork surfaces.
• Adds reusable macOS desktop app driving via AppleScript as builtin:desktop.macos.accessibilitySubmit, plus deterministic Cowork surface activation as builtin:anthropic.claude.activateCoworkSurface.
• Adds high-confidence Claude Cowork trace collection from local-agent metadata, audit.jsonl, and embedded .claude/projects/**/*.jsonl transcripts.
• Adds evidence gating so tool-call assertions require structured trace evidence rather than DOM/screenshot/accessibility text.
• Adds external-host schema/reference helpers for docs, autocomplete, and agent-generated configs.
• Updates reporter data and UI to show host identity, trace source/confidence, evidence source, artifacts, session metadata, usage, cost, and host/automation failure kinds.

Why

The current eval path only approximates an MCP host. For real confidence, we need to run eval cases through actual hosts such as Claude Cowork, Claude Desktop Chat, CLI tools, browser hosts, and future desktop apps, while preserving the same dataset and expectation model.

This PR establishes the foundation for that by making the external-host runner declarative and capability-driven. New hosts should be addable by composing or registering capabilities instead of adding one-off imperative runners.

How It Works

An eval case can now specify:

{
  "mode": "external_host",
  "scenario": "Please reply with exactly: external host integration acknowledged.",
  "externalHost": {
    "driver": "anthropic.claude.cowork.desktop-app.macos",
    "timeoutMs": 60000
  },
  "expect": {
    "containsText": "external host integration acknowledged"
  }
}

The runtime generates a unique run marker, submits the marked scenario to the configured host, waits for host evidence, normalizes the response into the existing expectation-compatible result shape, and attaches external-host metadata for the reporter.

Claude Cowork is configured as a composition of capabilities:

capabilities: {
  control: [
    { uses: 'builtin:platform.macos' },
    {
      uses: 'builtin:anthropic.claude.activateCoworkSurface',
      with: { appName: 'Claude' },
    },
  ],
  input: {
    uses: 'builtin:desktop.macos.accessibilitySubmit',
    with: { appName: 'Claude', createNewConversation: true },
  },
  completion: {
    uses: 'builtin:anthropic.claude.localAgentTrace',
    provides: ['trace'],
  },
  normalize: {
    uses: 'builtin:anthropic.claude.localAgentNormalize',
  },
}

Custom/project-local host capabilities can also be loaded dynamically:

{
  "capabilities": {
    "control": {
      "uses": "module:file:///path/to/fake-external-host.mjs#fakeExternalHost",
      "provides": ["input", "completion", "trace", "normalize"]
    }
  }
}

Cowork driver design

The Cowork driver is intentionally keyboard-only — no coordinate-based clicks, no recursive accessibility tree walks. The submit sequence is:

1. Activate Claude.app and verify it actually came to the foreground. tell-to-activate is unreliable on multi-monitor / multi-Space setups when another app holds focus precedence; the driver retries set frontmost to true for up to 2 seconds and errors fast with a clear message if the OS refuses, rather than letting downstream keystrokes route to the wrong app and surfacing as a 90-second eval timeout.
2. Send Cmd+2 to switch to the Cowork surface (idempotent — no-op if already on Cowork).
3. Send Cmd+N to open a new conversation. Cowork's React app autofocuses the composer when a new conversation view renders, even though Chromium's macOS accessibility bridge does not expose AXFocusedUIElement for that focus.
4. Send Cmd+V to paste the marked scenario from the clipboard. Keystrokes route to whatever has DOM focus inside the active window.
5. Send Return to submit.
6. Watch ~/Library/Application Support/Claude/local-agent-mode-sessions/ for a fresh session containing the run marker, then parse its audit.jsonl and embedded .claude/projects/**/*.jsonl transcript for trace evidence.

This avoids two known fragility classes: coordinate-based composer targeting (which breaks across window position, monitor placement, and layout drift) and recursive AX walks against Electron's Chromium accessibility bridge (which on a fully-loaded Cowork window can take 30+ seconds per query).

Validation

Validated locally with:

• npm run build
• npm run typecheck
• npm run format:check
• npm run lint
• npm test
• Focused external-host unit tests
• Deterministic package-boundary e2e demo: npm test -- tests/mcp-tests.spec.ts

Manual end-to-end run against real Claude Cowork on macOS: a two-case dataset (echo + Glean MCP search tool call) passes end-to-end with traceConfidence: high and traceSource: host-local-transcript on both cases, regardless of which monitor Claude is on once it can be activated.

{
  "pass": true,
  "driverSlug": "anthropic.claude.cowork.desktop-app.macos",
  "traceSource": "host-local-transcript",
  "traceConfidence": "high",
  "artifacts": 3
}

Notes

GUI-based host evals depend on macOS automation:

• Cowork must be running and signed in.
• The terminal running the eval needs Accessibility/Automation permission for Claude.app.
• The driver verifies foreground activation up to 2 seconds; if focus-prevention blocks the activation it errors fast rather than silently timing out.
• external_host runs are inherently more dependent on the host process than direct or mcp_host runs. CI use should run on an isolated desktop session or VM where window state is predictable.

[aditya-scio]

LGTM

[aditya-scio]

LGTM

[aditya-scio]

This is a meaningful step up in eval realism