feat: OpenAI-compatible /v1/chat/completions gateway for the agent-server

[smolpaws]

🐾 This is smolpaws, @enyst's agent. Posting this proposal on his behalf after a design discussion we had today.

---

The idea

Add an OpenAI-compatible /v1/chat/completions endpoint to the agent-server. This would let any client that speaks the OpenAI protocol talk to an OpenHands agent — not just an LLM, but a full agent with tools, terminal, file editing, browser, and memory.

Why

The OpenAI chat completions protocol is the de facto standard for AI interactions. Everything speaks it: chat UIs, IDE extensions, voice platforms, evaluation harnesses, other agents. If the agent-server exposes this interface, the boundary between "asking a model" and "asking an agent" dissolves.

The caller sends a chat completions request. Behind that endpoint is the full OpenHands agent runtime. The agent can investigate, run code, read files, search — and then answer. From the caller's perspective, it looks like a very capable "model."

What this unlocks

• Voice agents — platforms like ElevenLabs Conversational AI route voice calls to a chat completions endpoint. Connect one to an OpenHands agent and your coding agent picks up the phone. Hermes Agent already does this via their gateway.
• Any chat UI — Open WebUI, LibreChat, Chatbot UI, or any frontend that supports OpenAI-compatible backends could use an OpenHands agent as its "model."
• IDE integrations — tools like Continue, Cursor, or VS Code Copilot Chat that speak the OpenAI protocol could talk to an agent instead of a raw LLM.
• Agent-to-agent — one agent could call another as if it were an LLM. Delegation via protocol, not framework coupling.
• Eval frameworks — tools that benchmark LLMs could benchmark agents on the same harness.

Prior art

Hermes Agent implements exactly this with their hermes gateway command. It wraps the Hermes agent runtime as an OpenAI-compatible server. ElevenLabs published a guide for connecting the two. The architecture is clean: voice/chat frontend handles UX, the agent runtime handles tools and reasoning, and they communicate via POST /v1/chat/completions.

Implementation shape

This would likely be a new endpoint in the agent-server:

• POST /v1/chat/completions — accepts OpenAI-format messages, runs them through the agent, returns the response in OpenAI format
• Support streaming (stream: true) for incremental responses
• Map between OpenAI message format and the agent's internal event model

Design questions worth discussing

1. Streaming granularity — agents work over many internal turns. Do we stream intermediate tool calls or just the final text response?
   • enyst: I don’t think so, but maybe worth later; for voice, low first-token latency matters.
2. Statefulness — the OpenAI protocol is stateless (full message history in each request). Agents are stateful (conversation persists). The gateway could either manage state internally (mapping conversation threads) or operate statelessly per request.
   • enyst: I assume stateless 🤔
3. Tool visibility — should the caller see tool calls in the response (as OpenAI function calls) or get a clean text response?
   • enyst: no need, I don’t think
4. Model name — the endpoint needs a model name. Could be configurable (e.g., openhands-agent) or derived from the underlying LLM.
   • enyst: openhands_{profilename} maybe

Scope

This is a proposal — looking for feedback on whether this direction makes sense and what the right scope for a first implementation would be.

[smolpaws]

🐾 Scoping notes after studying the agent-server codebase and the Hermes gateway implementation (~4,200 lines for their full API server adapter).

Implementation scope — first PR

One new router file, non-streaming only. No SSE in v1.

New file

openhands-agent-server/openhands/agent_server/openai_router.py (~150 lines)

Two endpoints:

• POST /v1/chat/completions — the main endpoint (non-streaming)
• GET /v1/models — lists available profiles as models (openhands_{profile_name})

How /v1/chat/completions works

1. Parse OpenAI request body (messages, model)
2. Map model name (e.g. openhands_opus-46) → LLM profile → Agent + LLM config
3. Separate system messages from user/assistant messages
4. Create an ephemeral conversation via conversation_service.start_conversation() with the last user message as initial_message
5. Poll event_service.get_state() until execution_status leaves RUNNING
6. Call event_service.get_agent_final_response() for the text
7. Return OpenAI-formatted response:

{
  "id": "chatcmpl-xxx",
  "object": "chat.completion",
  "created": 1234567890,
  "model": "openhands_opus-46",
  "choices": [{
    "index": 0,
    "message": {"role": "assistant", "content": "..."},
    "finish_reason": "stop"
  }]
}

8. Tear down the ephemeral conversation

What it reuses

All the heavy lifting is already in the agent-server:

Concern	Existing code
Conversation lifecycle	conversation_service.py
Agent execution + waiting	event_service.py (.run(), .get_agent_final_response())
Profile resolution	profiles_router.py / settings service
Auth	Same session_api_key — maps naturally to OpenAI Authorization: Bearer

Route mounting: ~5 lines in api.py to add app.include_router(openai_router).

What is NOT in v1

• No streaming / SSE (follow-up PR)
• No tool call forwarding to caller
• No session continuity between requests (fully stateless)
• No /v1/responses or /v1/runs

Open questions

1. Ephemeral workspace — what workspace for per-request conversations? Shared temp dir, or the server's default workspace?
2. Profile resolution — exact mechanics of mapping openhands_{name} to a profile. The profiles system exists; need to check the lookup path.
3. Request timeout — agents can run longer than typical LLM calls. Need a reasonable default (60-120s?) with the option to configure.

Prior art

Hermes does this in gateway/platforms/api_server.py as a platform adapter. Their _handle_chat_completions does the same parse → run agent → format response flow. Their _run_agent creates an AIAgent and calls agent.run_conversation() in a thread executor — analogous to our event_service.run(). Their streaming path uses an asyncio queue with delta callbacks; we can study that for the SSE follow-up.

[smolpaws]

🐾 Follow-up from design discussion — the history problem and why Responses API matters.

The conversation history problem

For Chat Completions, the client sends the full message history with every request. But the agent-server's send_message() only accepts role="user" messages — there's no existing API to pre-seed a conversation with prior assistant turns. This means we can't faithfully replay an OpenAI-style multi-turn history into the agent's event stream.

For Chat Completions v1, this is fine: each request creates an ephemeral conversation with just the last user message. Stateless, no history needed. Good for one-shot questions where the agent investigates via tools ("did last night's tests pass?").

Responses API is the right fit for agents

The OpenAI Responses API is natively stateful — the server owns the conversation state, and the client chains requests via previous_response_id:

POST /v1/responses  {input: "what's 2+2?"}
→ {id: "resp_abc", output: [...]}

POST /v1/responses  {input: "now multiply by 10", previous_response_id: "resp_abc"}
→ same conversation, full context

This maps 1:1 to how the agent-server already works:

• response_id maps to conversation_id
• Each request is a send_message() on an existing conversation
• No history injection needed — the conversation accumulates events naturally
• The agent has full context from all prior turns

Ingress ID mapping

Different frontends can map their native thread/session IDs to conversation IDs:

• Slack thread_ts → deterministic conversation_id
• WhatsApp chat → conversation_id
• ElevenLabs voice call → conversation_id
• Direct API → use previous_response_id

Proposed roadmap

1. Chat Completions (first PR) — stateless, last-message-only, ephemeral conversations. Wide client compatibility. Good for one-shot agent queries.
2. Responses API (follow-up) — stateful, maps to real conversations. The right protocol for multi-turn voice calls, chat threads, and anything that needs the agent to remember context across turns.

[smolpaws]

🐾

PR 2 update: statefulness via header, not Responses API.

For multi-turn conversations over Chat Completions, the client can pass X-OpenHands-ServerConversation-ID (already used in the cloud backend for MCP routing).

• No header → ephemeral conversation (PR 1 behavior)
• With header → reuse existing conversation, server has full event history from prior turns

This sidesteps the history-seeding problem entirely — no need to replay messages or reconstruct tool calls. The server already has the real events.

Responses API remains a future option but isn't needed for this.

[enyst]

@OpenHands wanna try your hand at this? you can clone yourself hermes but remember it’s more complex than we need here; KISS! open PR(s) ; you can use openai for short live test, probably gpt-5-nano still works; you can use the eval litellm proxy with haiku (find its base url in the codebase somewhere) to see how it works with base url too.

make a .pr/ directory with live test artefacts and a .md file documenting those tests so I can see it

[openhands-ai]

I'm on it! enyst can track my progress at all-hands.dev

[openhands-ai]

OpenHands encountered an error: **Failed to send message to agent server: HTTP 503 error: no available server
**

See the conversation for more information.

[enyst]

Opened draft PR #3545: #3545

Implemented a KISS non-streaming OpenAI-compatible gateway with POST /v1/chat/completions, GET /v1/models, profile-backed openhands_{profile} model IDs, Bearer auth, and optional X-OpenHands-ServerConversation-ID reuse. Included .pr/ live-test artifacts for OpenAI gpt-5-nano and the eval LiteLLM proxy Haiku smoke test.

This comment was created by an AI agent (OpenHands) on behalf of the user.

[enyst]

@OpenHands Review this issue, all comments and the linked PR. That PR is first PR.

Draft two things:

• add a quick page under Arch on enyst.github.io with the end goal design here
• draft an implementation plan for the rest of PRs and post it as a comment here. Focus on usability, that is, the most commonly needed first.

Clone yourself hermes if you wanna gene-transfuse it.

Note: tool calls streaming is extremely low priority, I want to say end of the list if there at all.

[openhands-ai]

Uh oh! There was an unexpected error starting the job :(

[enyst]

I reviewed the issue thread, the follow-up comments, and PR #3545. I also drafted the requested Arch page as a companion PR here:

• Add OpenHands OpenAI gateway architecture note enyst/enyst.github.io#2

Read of PR #3545

PR #3545 is the right first cut: a thin OpenAI-compatible ingress over the existing agent-server rather than a second runtime. It already covers the highest-value compatibility path:

• GET /v1/models exposes saved profiles as openhands_{profile}.
• POST /v1/chat/completions handles non-streaming OpenAI-shaped calls.
• OpenAI-style Authorization: Bearer maps to the existing session key auth.
• X-OpenHands-ServerConversation-ID gives stateful reuse without reconstructing OpenAI message history.
• Tool activity stays internal; the caller gets the final assistant text.
• Usage is wired from conversation metrics when available.
• There is an example and live/QA evidence for real OpenAI-compatible clients.

That is enough to prove the architecture and should stay small.

Implementation plan for the remaining PRs

PR 2 — Usability docs and client recipes

Most users will ask: “what base URL, API key, and model name do I put into my client?” Make that path obvious first.

Deliverables:

• Agent-server docs page for the OpenAI-compatible gateway.
• Copy/paste examples for:
  • curl
  • OpenAI Python SDK
  • OpenAI JS SDK
  • Open WebUI / LibreChat-style custom OpenAI backend setup
  • a simple voice/webhook shape using the conversation id header
• Explain the two modes clearly:
  • no X-OpenHands-ServerConversation-ID → one-shot agent run
  • with X-OpenHands-ServerConversation-ID → continue that OpenHands conversation
• Document model naming: openhands_{profile_name}.
• Document current limitations: non-streaming only, final text only, no OpenAI tool-call surface.

Why first: this is the highest usability unlock and does not require more server semantics.

PR 3 — Compatibility hardening for common clients

Make the gateway boring for real clients before adding new protocol surfaces.

Deliverables:

• Verify and fix behavior against a short matrix:
  • OpenAI Python SDK
  • OpenAI JS SDK
  • Open WebUI or LibreChat
  • one voice-style caller
• Add any missing CORS/preflight behavior needed by browser-based OpenAI-compatible clients.
• Tighten OpenAI-shaped error responses where common clients expect them.
• Ensure unsupported request fields are ignored or rejected intentionally, not accidentally.
• Add regression tests for the compatibility cases found above.

PR 4 — Conversation reuse ergonomics

The raw conversation UUID header works, but integrations often start from a platform thread id or call id. Make stateful usage easier without replaying OpenAI history.

Deliverables:

• Keep X-OpenHands-ServerConversation-ID as the canonical low-level mechanism.
• Consider a small, explicit mapping layer for external thread ids, for example X-OpenHands-External-Thread-ID, if the native conversation service can support it cleanly.
• Document recommended mappings:
  • Slack thread timestamp → conversation id
  • ElevenLabs call/session id → conversation id
  • IDE workspace session id → conversation id
• Define cleanup/lifetime expectations for reused conversations.

This improves multi-turn usability for voice, chat, and IDE integrations.

PR 5 — Operational polish

Make the endpoint safe to operate in production-like environments.

Deliverables:

• Clear timeout and cancellation behavior for long agent runs.
• Better user-facing errors for paused, confirmation-needed, stuck, and failed runs.
• Request/run correlation ids in logs and responses where useful.
• Metrics/tracing around gateway calls: model/profile, duration, terminal status, token usage when available.
• Tag conversations with e.g. openai-completions

PR 6 — Final-answer streaming

Add stream: true, focused on what ordinary OpenAI clients can consume.

Deliverables:

• SSE chat.completion.chunk responses for final assistant text.
• Possibly coarse progress/status chunks only if they are compatible with common clients.
• Preserve the non-streaming implementation as the simpler path.
• Do not stream internal tool calls in this PR.

Streaming is common sense, expected feature; it helps voice/chat perceived latency.

PR 7 — Responses API

Responses API maps naturally to server-owned state, but Chat Completions plus the conversation id header already covers the immediate use case.

Deliverables:

• POST /v1/responses with response_id / previous_response_id mapped to OpenHands conversations.
• Final text output in Responses shape.
• Optional streaming only if PR 6 gives us a clean reusable streaming layer.

Last / maybe never — tool-call streaming

Tool-call streaming should stay at the end of the list unless concrete demand needs it.

Reasons:

• Most OpenAI-compatible clients want assistant text, not OpenHands internal event semantics.
• Mapping arbitrary OpenHands tool events to OpenAI tool calls can misrepresent what happened.
• Native OpenHands clients already have the richer event stream.

If we ever do it, it should likely be an opt-in debug/specialist mode rather than the default /v1/chat/completions behavior.

This comment was created by an AI agent (OpenHands) on behalf of the user.

[enyst]

@OpenHands focus on Point 2 here: #3540 (comment)

FInd on docs repo (clone it, too, from the same org) a PR that does add some docs for this new endpoint. I think you will find it's a very limited PR to document the examples/ file. Make a new PR stacked on it, where you add in agent-server section (sdk tab) OpenAI-compatible Endpoint and document it as noted here. Also, add a note about "access the OpenHands agent via an OpenAI-compatible endpoint" in the SDK homepage/overview, to be visible - it's a really different thing than other endpoints!