[AgentProfile][agent-server] materialize endpoint (resolve dry-run)

[simonrosenberg]

HUMAN:
Fast-follow endpoint that was deferred from #3781 pending resolver (#3780); implements the dry-run materialize (valid/dangling verdict) with 6 tests covering all acceptance criteria.

AGENT:

Why

POST /{name}/materialize was deliberately left out of #3781 (the agent profiles router) pending the resolver landing in #3780/#3717. Both are now merged on main, making this the unblocked fast-follow.

Summary

• Adds POST /api/agent-profiles/{name}/materialize → AgentProfileDiagnostics to agent_profiles_router.py.
• Loads the profile by name (404 if missing), decrypts skills[].mcp_tools ciphertext, reads the active global mcp_config from settings (decrypted by the store), instantiates LLMProfileStore, and calls resolve_agent_profile_dry_run.
• Dangling LLM/MCP refs surface in the body (valid=False, dangling_* lists populated) — endpoint never raises on bad refs; 404 is the only error status.
• resolved_settings in the response is redacted (api_key_set booleans; no raw secrets).
• Updates the module docstring: removes the deferred-materialize note, corrects stale /{id}/ wording to /{name}/.

How to Test

# Run the test suite (45 tests, 6 new materialize tests)
uv run pytest tests/agent_server/test_agent_profiles_router.py -q
# 45 passed

# Manual end-to-end against a running agent-server:

# 1. Save an LLM profile (using the /api/profiles endpoint)
curl -X POST http://localhost:8000/api/profiles/base-llm \
  -H 'Content-Type: application/json' \
  -d '{"llm": {"model": "gpt-4o", "api_key": "sk-test"}}'

# 2. Save an agent profile referencing it
curl -X POST http://localhost:8000/api/agent-profiles/my-profile \
  -H 'Content-Type: application/json' \
  -d '{"llm_profile_ref": "base-llm"}'

# 3. Materialize -> 200 valid=true, resolved_settings present, api_key redacted
curl -X POST http://localhost:8000/api/agent-profiles/my-profile/materialize

# 4. Dangling LLM ref -> 200 valid=false
curl -X POST http://localhost:8000/api/agent-profiles/bad/materialize
# (where bad was saved with {"llm_profile_ref": "nonexistent"})

# 5. Unknown profile -> 404
curl -X POST http://localhost:8000/api/agent-profiles/ghost/materialize

🤖 Generated with Claude Code

[github-actions]

Python API breakage checks — ✅ PASSED

Result: ✅ PASSED

Action log

[github-actions]

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

[all-hands-bot]

⚠️ QA Report: PASS WITH ISSUES

The new POST /api/agent-profiles/{name}/materialize endpoint works end-to-end in a real agent-server run, but CI was not green and the PR's sample LLM-profile curl used a stale payload shape.

Does this PR achieve its stated goal?

Yes. On origin/main, the same materialize URL returned 404 Not Found; on commit a05bda6d996021fb3568dc96f91995447ef8fc57, real HTTP requests against a running agent-server returned 200 diagnostics for saved profiles, resolved valid LLM and MCP references, reported dangling LLM/MCP refs in the body with valid=false, redacted the test API key, and returned 404 only for an unknown profile name.

Phase	Result
Environment Setup	✅ make build completed; server ran via uvicorn openhands.agent_server.api:create_app --factory
CI Status	⚠️ gh pr checks reported 20 successful, 2 failing, 1 skipped, 8 pending checks
Functional Verification	✅ Real agent-server + curl requests verified the new endpoint behavior

Functional Verification

Test 1: Baseline endpoint absence vs PR materialization

Step 1 — Establish baseline without the fix:
Checked out origin/main, started the agent-server, saved a real LLM profile and agent profile, then ran:

curl -X POST http://127.0.0.1:8123/api/agent-profiles/my-profile/materialize

Observed:

{"detail":"Not Found"}
HTTP_STATUS:404

This confirms the materialize endpoint did not exist on the base branch.

Step 2 — Apply the PR's changes:
Checked out a05bda6d996021fb3568dc96f91995447ef8fc57, started the agent-server with isolated persistence, saved the same base-llm and my-profile records.

Step 3 — Re-run with the fix in place:
Ran:

curl -X POST http://127.0.0.1:8124/api/agent-profiles/my-profile/materialize

Observed the endpoint returned HTTP_STATUS:200 with:

{
  "agent_kind": "openhands",
  "valid": true,
  "errors": [],
  "llm_profile_ref": "base-llm",
  "llm_profile_resolved": true,
  "llm_api_key_set": true,
  "resolved_settings": {"llm": {"model": "gpt-4o", "api_key": "**********"}}
}

This confirms the new endpoint resolves a valid OpenHands agent profile and redacts the raw test API key in resolved_settings.

Test 2: Dangling references stay in diagnostics body

With the PR branch running, I saved profiles with missing references and materialized them:

curl -X POST /api/agent-profiles/bad-llm/materialize
curl -X POST /api/agent-profiles/bad-mcp/materialize

Observed:

{"valid": false, "errors": ["LLM profile 'nonexistent' not found"], "llm_profile_resolved": false, "resolved_settings": null}
HTTP_STATUS:200

{"valid": false, "errors": ["MCP server(s) not configured: missing-server"], "llm_profile_resolved": true, "dangling_mcp_server_refs": ["missing-server"], "resolved_settings": null}
HTTP_STATUS:200

This confirms bad LLM/MCP refs do not raise HTTP errors; they are reported as diagnostics as claimed.

Test 3: Active global MCP config is used

I patched settings with a configured MCP server, saved an agent profile referencing it, and materialized:

curl -X PATCH /api/settings -d '{"agent_settings_diff":{"mcp_config":{"mcpServers":{"configured-server":{"command":"python","args":["-c","print(1)"]}}}}}'
curl -X POST /api/agent-profiles/with-mcp/materialize

Observed:

{
  "valid": true,
  "mcp_server_refs": ["configured-server"],
  "resolved_mcp_servers": ["configured-server"],
  "dangling_mcp_server_refs": [],
  "resolved_settings": {"mcp_config": {"mcpServers": {"configured-server": {"command": "python"}}}}
}
HTTP_STATUS:200

This confirms materialize reads the active global mcp_config from settings.

Test 4: ACP and unknown-profile behavior

I saved an ACP profile and also requested an unknown profile:

curl -X POST /api/agent-profiles/acp-profile/materialize
curl -X POST /api/agent-profiles/ghost/materialize

Observed:

{"agent_kind": "acp", "valid": true, "errors": [], "resolved_settings": {"agent_kind": "acp", "acp_server": "codex"}}
HTTP_STATUS:200

{"detail": "Agent profile 'ghost' not found"}
HTTP_STATUS:404

This matches the stated behavior that unknown profile names are the only error status in this flow.

Issues Found

• 🟠 CI not green at QA time: gh pr checks 3783 reported PR Description Check/Validate PR description and Run tests/agent-server-tests failing, with 8 checks still pending.
• 🟡 PR testing snippet mismatch: The flat LLM-profile payload from the PR description returned 422 (body.llm missing). The actual endpoint worked when using the current API shape: {"llm":{"model":"gpt-4o","api_key":"..."}}.

Final verdict: PASS WITH ISSUES.

This QA review was created by an AI agent (OpenHands) on behalf of the user.

[github-actions]

Coverage Report •

File	Stmts	Miss	Cover	Missing
openhands-agent-server/openhands/agent_server
agent_profiles_router.py	237	9	96%	138, 142, 169, 258, 303–304, 504, 544–545
TOTAL	32401	14054	56%

[all-hands-bot]

✅ Review complete.

This review was performed through OpenHands Cloud Automation. You can log in and view the conversation here.

[all-hands-bot]

🟢 Taste Rating: Good taste

LGTM. This is a narrow endpoint that loads an existing agent profile, uses the current profile/settings stores, and delegates the actual reference semantics to the shared dry-run resolver instead of duplicating resolution logic. The focused tests cover valid OpenHands/ACP profiles, dangling LLM/MCP refs, 404 behavior, and secret redaction; I also ran uv run pytest tests/agent_server/test_agent_profiles_router.py -q successfully.

[RISK ASSESSMENT]

• [Overall PR] ⚠️ Risk Assessment: 🟢 LOW
  Small agent-server API addition over existing resolver/profile-store primitives. The main risk is accidentally exposing resolved secrets, but the response goes through existing Pydantic secret serializers and the PR adds explicit no-raw-secret coverage.

VERDICT:
✅ Worth merging: Core logic is sound, tests cover the acceptance criteria, and the relevant CI has recovered to green.

KEY INSIGHT:
Materialization stays side-effect-free and keeps the hard LLM/MCP reference behavior centralized in the SDK resolver.

This review was generated by an AI agent (OpenHands) on behalf of the user through OpenHands Automation. View conversation

[all-hands-bot]

✅ QA Report: PASS

Started the agent-server and verified the new materialize API works end-to-end for valid, dangling, unknown, redacted-secret, and ACP profile scenarios.

Does this PR achieve its stated goal?

Yes. The stated goal was to add POST /api/agent-profiles/{name}/materialize as a dry-run resolver that returns diagnostics instead of raising for dangling LLM/MCP references, with 404 only for unknown profiles and no raw secrets in resolved_settings. I confirmed on main that the endpoint did not exist, then on this PR created profiles via the public HTTP APIs and observed the new endpoint return 200 valid=true for resolvable profiles, 200 valid=false for dangling LLM/MCP refs, 404 for a missing profile, and no raw API key in the response.

Phase	Result
Environment Setup	✅ make build completed and agent-server started successfully
CI Status	✅ 34 passing, 3 skipped, 1 pending (qa-changes) at time of review
Functional Verification	✅ Real HTTP requests verified the new API behavior

Functional Verification

Test 1: Baseline on main confirms the materialize endpoint was absent

Step 1 — Reproduce / establish baseline without the fix:
Ran a real server from main, then created the same LLM and agent profiles a user would create before materializing:

git switch main
HOME=<default> OH_PERSISTENCE_DIR=/tmp/qa-materialize-main... uv run agent-server --host 127.0.0.1 --port 8765
POST /api/profiles/base-llm -> 201 {"name":"base-llm","message":"Profile 'base-llm' saved"}
POST /api/agent-profiles/my-profile -> 201 {"name":"my-profile","message":"Agent profile 'my-profile' saved"}
POST /api/agent-profiles/my-profile/materialize -> 404 {"detail":"Not Found"}

This confirms the prior behavior: users could save the referenced profiles, but there was no materialize route to dry-run resolve them.

Step 2 — Apply the PR's changes:
Checked out agent-profile-materialize at a05bda6d996021fb3568dc96f91995447ef8fc57 and restarted agent-server with isolated HOME and OH_PERSISTENCE_DIR.

Step 3 — Re-run with the fix in place:
Created an LLM profile containing sk-secret-qa-key, created an agent profile referencing it, then materialized it:

POST /api/profiles/base-llm -> 201
POST /api/agent-profiles/my-profile -> 201
POST /api/agent-profiles/my-profile/materialize -> 200
valid=true
agent_kind=openhands
llm_profile_ref=base-llm
llm_profile_resolved=true
llm_api_key_set=true
resolved_settings.llm.model=gpt-4o
raw_secret_in_valid_response=0

This shows the new endpoint resolves a valid OpenHands profile and redacts the raw API key from the returned settings.

Test 2: Dangling references are diagnostics, not HTTP errors

With the PR server still running, I created profiles with bad references and materialized them:

POST /api/agent-profiles/bad-llm {"llm_profile_ref":"nonexistent"} -> 201
POST /api/agent-profiles/bad-llm/materialize -> 200
valid=false
errors=["LLM profile 'nonexistent' not found"]
llm_profile_ref=nonexistent
llm_profile_resolved=false
resolved_settings=null

POST /api/agent-profiles/bad-mcp {"llm_profile_ref":"base-llm","mcp_server_refs":["missing-server"]} -> 201
POST /api/agent-profiles/bad-mcp/materialize -> 200
valid=false
errors=["MCP server(s) not configured: missing-server"]
dangling_mcp_server_refs=["missing-server"]
resolved_settings=null

This confirms dangling LLM/MCP references are surfaced in the response body with valid=false, matching the PR goal.

Test 3: Unknown profile remains the only error status exercised

POST /api/agent-profiles/ghost/materialize -> 404
{"detail":"Agent profile 'ghost' not found"}

This matches the intended 404 behavior for missing profile names.

Test 4: ACP profiles materialize successfully

Started the PR server with a fresh isolated home, saved an ACP agent profile, and materialized it:

POST /api/agent-profiles/acp-p {"agent_kind":"acp","acp_server":"codex","acp_model":"gpt-5.5"} -> 201
POST /api/agent-profiles/acp-p/materialize -> 200
valid=true
agent_kind=acp
errors=[]
resolved_settings.agent_kind=acp
resolved_settings.acp_server=codex
resolved_settings.acp_model=gpt-5.5

This confirms the endpoint also handles the ACP profile path without requiring an LLM profile ref.

Issues Found

None.

This QA review was created by an AI agent (OpenHands) on behalf of the user.