[Bug] Cannot use an OpenAI-compatible gateway (LiteLLM / OpenRouter / vLLM) out of the box - several friction points

[k-nacion]

HealthLog version

v1.26.0

Deployment type

Self-hosted (Docker)

What happened?

Cannot use an OpenAI-compatible gateway (LiteLLM / OpenRouter / vLLM) out of the box - several friction points

Summary

HealthLog cannot cleanly use a generic OpenAI-compatible LLM gateway (LiteLLM proxy, OpenRouter, vLLM in OpenAI mode, LM Studio, or any cloud gateway that speaks POST /v1/chat/completions with Bearer auth and response_format). I got it working eventually, but only through the admin AI settings path and after several blockers, one of which needed a patch to the compiled build.

Most of these are provable just by reading the source (file/line refs below) - you do not need my specific gateway to confirm them. Where a live endpoint helps, any OpenAI-compatible gateway reproduces the behavior; the quickest is a local LiteLLM proxy (pip install 'litellm[proxy]' && litellm --model gpt-4o, which serves http://localhost:4000/v1), or OpenRouter (https://openrouter.ai/api/v1).

I'm reporting these for the maintainers to prioritize and fix as they see fit.

Environment

• HealthLog version: v1.26.0 (build SHA 3bceb61aed99)
• Image: ghcr.io/mbombeck/healthlog:latest (digest sha256:250f76c749d862648e4c45e67361c9e1f38deec612fa1d0efd5323a69020c34a, created 2026-07-01)
• Runtime: Node 22.23.1
• Gateway under test: an OpenAI-compatible LiteLLM proxy in front of hosted models. Reproducible with any OpenAI-compatible gateway.

General reproduction setup

1. Stand up any OpenAI-compatible gateway (local LiteLLM proxy is easiest; OpenRouter works too).
2. In HealthLog, configure it as described per blocker below.
3. Trigger insight generation (POST /api/insights/generate, e.g. the "Regenerate insights" / "Generate briefing" buttons) and watch the app logs.

---

Blocker 1 - The user-level "Local (OpenAI-compatible)" provider sends Ollama-style format: "json"

LocalOpenAICompatibleClient (src/lib/ai/local-client.ts) adds a top-level format: "json" field when responseFormat === "json":

...(params.responseFormat === "json" ? { format: "json" } : {}),

That is the Ollama JSON flag, not OpenAI's. A strict OpenAI-compatible gateway does not recognize a top-level format field and rejects the whole request with a 400.

Reproduce (no HealthLog needed) - show the exact body shape HealthLog sends is rejected:

# Against any OpenAI-compatible gateway (LiteLLM/OpenRouter/vLLM/...):
curl -sS $BASE_URL/chat/completions \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"model":"'"$MODEL"'","messages":[{"role":"user","content":"hi (json)"}],"format":"json"}'
# -> 400, complaining about an unknown/unexpected "format" parameter
# The same request with "response_format":{"type":"json_object"} instead succeeds.

(Representative errors, varying by backend: Unknown parameter: 'format', or format: Extra inputs are not permitted.)

Reproduce (in HealthLog): configure a user-level "Local" provider pointing at any OpenAI-compatible gateway, then generate insights -> every call 400s because insight generation always requests JSON.

Impact: the "Local" provider - the only user-level provider type that accepts a custom base URL - cannot talk to a real OpenAI-compatible gateway at all.

Suggested fix: send OpenAI's response_format: { type: "json_object" } instead of Ollama's format: "json", or make the JSON-mode dialect selectable per provider (Ollama vs OpenAI-compatible).

Blocker 2 - The user-level "OpenAI" provider ignores the custom base URL

buildUserProvider() / resolveProviderForType() (src/lib/ai/provider.ts) hardcode the OpenAI base URL and deliberately drop any stored aiBaseUrl:

case "OPENAI": {
  if (!row.aiOpenaiKeyEncrypted) return null;
  return new OpenAIClient({
    apiKey: decrypt(row.aiOpenaiKeyEncrypted),
    model: row.aiModel ?? "gpt-4o",
    baseUrl: "https://api.openai.com/v1",   // <-- custom URL ignored
  });
}

So the combination a gateway actually needs - OpenAI wire format (response_format) + a custom base URL - is not reachable from the per-user settings at all. Blocker 1 (Local) has the custom URL but the wrong wire format; Blocker 2 (OpenAI) has the right wire format but no custom URL. This one is confirmable by reading the code; no gateway required.

Workaround that does work: the admin AI settings path (resolveAdminProvider()) uses OpenAIClient with settings.adminAiBaseUrl, i.e. the correct combination. Wiring the gateway through admin settings + enabling the admin-openai chain entry is the only way I found to make it work.

Suggested fix: add a first-class "OpenAI-compatible" user provider type (custom base URL + response_format), or allow the OpenAI type to honor a custom base URL. A host allowlist (same idea as ADMIN_AI_BASE_URL_ALLOWLIST) would cover the "don't forward the key to a stray URL" concern.

Blocker 3 - Daily Briefing hardcodes maxTokens: 1500 -> truncated JSON -> 422

The briefing generation uses a fixed token cap (src/app/api/insights/generate/route.ts, and the same value in src/lib/insights/comprehensive-generate.ts):

temperature: 0.3, maxTokens: 1500, timeoutMs: ..., responseFormat: "json"

When the model's full briefing JSON exceeds 1500 completion tokens, the reply is cut off mid-string and JSON.parse fails, surfacing as 422 "AI response was not valid JSON". Short metric cards fit and succeed, so it looks intermittent (only the full briefing fails). This is model-agnostic - it reproduces on real OpenAI too, not just gateways.

Reproduce (no HealthLog needed): ask any model for a large JSON object with max_tokens: 1500 and response_format: json_object:

curl -sS $BASE_URL/chat/completions \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"model":"'"$MODEL"'","max_tokens":1500,"response_format":{"type":"json_object"},
       "messages":[{"role":"user","content":"Return a large JSON object: 10 recommendations, each with a 3-sentence rationale. Verbose. (json)"}]}'
# -> "finish_reason":"length", completion_tokens 1500, body cut off mid-string
# -> JSON.parse on that content fails: "Unterminated string in JSON"

Suggested fix: make the cap configurable (e.g. INSIGHTS_MAX_TOKENS, default higher), and/or on finish_reason === "length" return a specific "response truncated - raise token limit" message instead of the generic invalid-JSON 422.

Blocker 4 - Anthropic models via LiteLLM return JSON in a tool call, so admin path + Claude yields empty content

When the admin path (OpenAIClient + response_format: json_object) is pointed at an Anthropic/Claude model proxied through LiteLLM, the reply comes back as:

finish_reason: "tool_calls", message.content: "{}"

LiteLLM implements Anthropic JSON mode via a synthesized tool call, so the structured output lands in tool_calls, not message.content. HealthLog parses message.content, so Claude models produce empty insights on this path (silently). An OpenAI-family model returns usable message.content.

Reproduce: run a local LiteLLM proxy with an Anthropic/Claude model, point HealthLog's admin provider at it, generate insights -> empty output. Same request against an OpenAI-family model works.

Suggested fix: when message.content is empty and tool_calls is present, parse the tool-call arguments as the JSON payload (the common OpenAI-compat shim behavior), or document that Anthropic-via-LiteLLM isn't supported.

Blocker 5 (UX) - Grounding gate silently discards the briefing; button appears to "do nothing"

The daily-briefing number-grounding gate (src/lib/ai/briefing-grounding.ts) strips the briefing (dailyBriefing = null) when a restated number can't be matched to features.signalsOfDay, after one corrective retry. The route then returns 200 with an empty briefing, and the UI shows the generic "No briefing yet" empty state.

For an account with sparse data (small signalsOfDay set) and a chatty model, this fires on most runs - so clicking Generate briefing looks like it does nothing, even though a briefing was generated and then discarded. It's non-deterministic (temperature 0.3), so it occasionally succeeds, which is confusing.

Reproduce: on a sparse account, click "Generate briefing" repeatedly - most clicks return 200 with an empty card (server logs show ungroundedCount > 0 and the briefing stripped), an occasional click renders a briefing.

Suggested fix: surface the outcome to the user - e.g. a toast/annotation like "Briefing omitted: it referenced a figure we couldn't verify against your data." - rather than a silent empty state. (The gate itself is reasonable; the missing feedback is the problem.)

Minor - Misleading error categorization on the Test button

classifyTestFailure() (src/app/api/ai/test/route.ts) maps any non-401/403/429/5xx to unreachable -> "Could not reach the AI provider." A 400 (e.g. the format/param rejections in Blocker 1) therefore displays as a network/unreachable error even though the host was reached and answered. This sent me down a connectivity rabbit hole for what was a request-shape problem. Consider a distinct bad_request category for 400/422.

---

Configuration that was genuinely required (not bugs, but worth documenting)

For anyone wiring an OpenAI-compatible gateway via admin AI settings:

1. Set ADMIN_AI_BASE_URL_ALLOWLIST=<gateway-host> (custom hosts are otherwise rejected on save).
2. Pick an OpenAI-family model (response_format returned in message.content); Anthropic-via-LiteLLM hits Blocker 4.
3. INSIGHTS_RATE_LIMIT_PER_HOUR default of 10 is easy to exhaust while iterating; the 429 surfaces to users as a generic AI failure.

Net

The gateway side is a standard OpenAI Chat Completions endpoint - a plain curl with response_format: json_object returns valid JSON in ~1s, and everything reproduces against any OpenAI-compatible gateway (LiteLLM/OpenRouter/vLLM). The blockers are all on the HealthLog side (wire-format assumptions, hardcoded token cap, silent gate), and Blockers 1-3 are confirmable straight from the source. Blockers 1-3 actually prevent a working setup; 4-6 are quality-of-life. Leaving the fixes to the maintainers to handle.

Expected behavior

All the findings are above

Steps to reproduce

All the findings are above

Relevant logs / screenshots

Browser / OS (if applicable)

No response

Pre-flight checks

• I confirmed this is not a duplicate of an existing issue.
• I redacted any secrets, tokens, or personal data from logs.

[MBombeck]

All of this shipped in v1.28.28 (on :latest now) — thank you for a report that was essentially a fix list with receipts: every blocker had a file reference and a curl repro, which made this a same-day turnaround.

What changed, mapped to your blockers:

1. The Local provider now sends the standard response_format: {"type":"json_object"} — and if an endpoint rejects it, it falls back once and remembers that endpoint's dialect. LiteLLM, OpenRouter, vLLM, LM Studio and plain Ollama all work from the same form now.
2. Deliberately unchanged: the OpenAI provider keeps its pinned base URL (your key should never be forwardable to a custom host). With 1 fixed, the Local provider is the gateway path — its form and the provider docs now say so explicitly (Bearer key optional).
3. INSIGHTS_MAX_TOKENS is configurable (default raised to 2500, clamped 500–8000), and a reply cut off mid-JSON now returns a distinct "response truncated — raise the token limit" error instead of the generic invalid-JSON 422.
4. Empty message.content with a synthesized tool call (the LiteLLM/Anthropic shim) is parsed from the tool-call arguments now.
5. A briefing withheld by the number-grounding check says so on the card, with a retry, instead of a silent empty state.
6. The test button answers 4xx with "the provider rejected the request (HTTP n)" instead of claiming the host was unreachable.

If you get a chance to re-test against your gateway, I'd love a short confirm — especially 1 and 3 on your LiteLLM setup. Thanks again; this one made the platform better for every gateway user at once.