Wire OpenShell sandbox lifecycle into the orchestrator

[saichandrapandraju]

What this adds

Fills in the lifecycle stubs introduced in #67 (setup, exec_agent, snapshot, teardown) so --protocol openshell runs end-to-end against real NVIDIA OpenShell MicroVM sandboxes. The orchestrator drives the full evaluation cycle — create sandbox, seed workspace, run agent, collect workspace diff and OCSF kernel events, grade — with the backend owning the sandbox lifecycle.

Architecture

orchestrator.run_task()
  ├── backend.provision(injections)       # pure: render workspace files
  ├── backend.setup(pre_env)             # create MicroVM, seed /sandbox/workspace, inject providers
  │     └── SandboxClient.from_active_cluster()  # picks up mTLS from ~/.config/openshell/
  ├── agent_client.send_task(prompt)     # exec agent_command inside sandbox, return stdout
  ├── backend.snapshot()                 # workspace diff + OCSF events → OpenShellEnvironment
  ├── PUT /evaluations/{id}/environment  # post-env to control plane
  └── backend.teardown()                 # delete sandbox (always runs, even on error)

OpenShellAgentClient is a 5-line wrapper that calls backend.exec_agent() and returns stdout. The sandbox lifecycle lives entirely in the backend.

New files

File	Purpose
src/midojo/openshell_logs.py	OCSF shorthand log parser (NET/HTTP/PROC/FINDING events)
src/midojo/verifiers/openshell.py	Workspace diff and OCSF predicates (moved out of builtin)
suites/document_assistant/	Demo suite: Q4 financial report with curl-exfiltration and staging-cache injection tasks
tests/test_shell_predicates.py	Tests for the workspace/OCSF predicates

Suite YAML

environment:
  backend:
    type: openshell
    image: pi                             # community sandbox with pi pre-installed
    # providers: [openai]                 # uncomment and set your registered provider(s)
    agent_command: ["pi", "-p", "--no-session"]

• image is a community sandbox name — resolved to ghcr.io/nvidia/openshell-community/sandboxes/pi:latest automatically.
• policy is omitted so the image's own /etc/openshell/policy.yaml applies (comprehensive allowlist for all major providers).
• providers injects registered OpenShell provider credentials as env vars into the sandbox (e.g. OPENAI_API_KEY).
• agent_command is configurable per-suite; model selection is left to the user's pi config or provider defaults.

CLI changes

• --agent-url renamed to --agent-uri (more accurate: for pi it's a directory path, not a URL).
• --openshell-endpoint removed — --agent-uri serves as the gateway endpoint for --protocol openshell. Omit it to use the CLI-registered active gateway (SandboxClient.from_active_cluster()).
• backend parameter typed as EnvironmentBackend | None in run_task() and run_benchmark().
• PUT /environment now calls raise_for_status() — silent 4xx failures no longer cause a successful attack to grade as "agent secure".

Running it

Prerequisites (one-time)

Install OpenShell (macOS — Homebrew, auto-starts the gateway):

curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh
openshell status   # → Status: Connected

Install the Python SDK into the midojo venv:

uv pip install openshell

Register your inference provider:

OPENAI_API_KEY=sk-... openshell provider create --name openai --type openai --from-existing
# or: ANTHROPIC_API_KEY=sk-ant-... openshell provider create --name anthropic --type anthropic --from-existing

Run the demo suite

Uncomment and set providers in suites/document_assistant/suite.yaml:

providers: [openai]   # or [anthropic], matching your registered provider name

Then:

# Terminal 1 — control plane
uv run midojo-serve --suite document_assistant --port 8090

# Terminal 2 — benchmark (no --agent-uri needed; active gateway is picked up automatically)
uv run midojo-run \
  --protocol openshell \
  --suite document_assistant \
  --control-url http://localhost:8090

Each evaluation creates a fresh MicroVM sandbox from the pi image, seeds /sandbox/workspace/ with the injected financial report, runs the pi agent against it, collects workspace diff and OCSF kernel events, and deletes the sandbox. Each evaluation takes ~30–60 s on first run (image pull) and ~15–30 s subsequently.

Known limitation

OCSF unavailability grades as False, not N/A. When GetSandboxLogs fails (gateway timeout, OCSF stream unavailable), OCSF predicates (process_ran, network_call_to, etc.) silently return False — "attack failed" — rather than N/A. A silent OCSF failure can make a successful attack appear to have been resisted. Fixing this properly requires predicates to return bool | None, which is a verifier-protocol change deferred to a follow-up.

agent_command is effectively required. The suite YAML field agent_command is technically optional (defaults to None), but the fallback when unset runs the prompt text as a shell command — not the image's entrypoint. The pi image's ENTRYPOINT is /bin/bash, so without agent_command nothing useful runs. This means midojo currently owns a piece of agent configuration that arguably belongs to the image. The right long-term fix is for community sandbox images to adopt a standard headless entrypoint that accepts a prompt argument, making agent_command unnecessary. Until then it is load-bearing.

[saichandrapandraju]

Hi @dmaniloff , would like to get early feedback on this, thanks!

[dmaniloff]

there's already a pi image and a pi policy for pi.dev agents.

it's the one that gets used when you say openshell sandbox create --from pi

let's use that instead.

and let's also add an agent_command that defaults to the image's entrypoint eg

agent_command: ["pi", "-p", "--no-session"] # optional, defaults to image's entrypoint

[dmaniloff]

though i may have a different opinion about this now ... let's chat when you get a moment.

[dmaniloff]

the any_of/all_of combinators call parse_predicate recursively, which doesn't dispatch to named verifiers

nice call out @saichandrapandraju -- this is a missing piece in Predicate.evaluate. i am adding that in #71

let's please rebase this on top of that

[dmaniloff]

once rebased, the question is whether we want to keep the observations field or just do what you are doing which is to gather observations from the orchestrator and populate environment fields like files etc.

maybe that is cleaner sine the fields are self-explanatory? the other benefit of that is that we can remove the POST /observations endpoint in the control api.

thoughts?

[saichandrapandraju]

I think -

VerificationContext.observations can stay - it's the right extension point for future backends that surface event streams that don't fit neatly into env fields.

Everything OpenShell-specific around it can be removed:

• backend.observations() — its only job was formatting the OCSF dict to push to the control plane, which we don't need since env fields cover all current predicates
• The orchestrator step calling backend.observations() and POST /observations
• The POST /evaluations/{id}/observations control API endpoint
• evaluation.observations in the state model

At least for Openshell, the env fields on OpenShellEnvironment are sufficient. If a future backend needs richer grading data that doesn't map cleanly to env fields, then ctx.observations and the endpoint comes back.

Happy to make the changes if this sounds right to you.

[dmaniloff]

agree w/ you @saichandrapandraju ! let's make those changes.

[dmaniloff]

lets leave this in as required and make it the openshell endpoint when using --protocol openshell

[dmaniloff]

lets use the agent-url for this. maybe we can generalize the name a bit. eg., in the case of pi the url is actually the path to the pi TS extension. so we can maybe rename agent-url to agent-uri ?

[dmaniloff]

this should be of type EnvironmentBackend right?

[dmaniloff]

same here, EnvironmentBackend ?

[dmaniloff]

These two pushes are what grading reads from — if either fails (e.g. a 422 from env-model version skew, or a 404 when the server was started with a different suite), httpx won't raise on the 4xx, the grade endpoint falls back to the pristine provisioned environment, and every security predicate evaluates False over empty fields. A successful attack then grades as "agent secure" with no error anywhere. Better to fail the eval loudly:

Suggested change

	await client.put(
	f"{control_url}/runs/{run_id}/evaluations/{eval_id}/environment",
	json=post_env.model_dump(), # type: ignore[union-attr]
	)
	await client.post(
	f"{control_url}/runs/{run_id}/evaluations/{eval_id}/observations",
	json={"source": "openshell", "data": obs_data},
	)
	env_resp = await client.put(
	f"{control_url}/runs/{run_id}/evaluations/{eval_id}/environment",
	json=post_env.model_dump(), # type: ignore[union-attr]
	)
	env_resp.raise_for_status()
	obs_resp = await client.post(
	f"{control_url}/runs/{run_id}/evaluations/{eval_id}/observations",
	json={"source": "openshell", "data": obs_data},
	)
	obs_resp.raise_for_status()

[dmaniloff]

is there a way we can avoid the inline import?

[dmaniloff]

we should think about a way to surface this so that during grading we can say N/A rather than "attack failed"

[dmaniloff]

thinking back to my suggestion re: agent_command ... i think we should probably try to avoid all this.

the concern of midojo stops at the agent boundary. in other words, agent config (like its command and providers etc) should not be what we deal with. this should all exist somewhere else already.

in the case of an openshell env it seems that there's some coupling between the agent's config & the env so maybe the way around it is via an image or another openshell native mechanism. let's chat about this.