Grammar enforcement not applied when thinking is enabled (response_format + enable_thinking)

[shanevcantwell]

Summary

When response_format (JSON schema) is used with enable_thinking: true, grammar enforcement is completely inactive. The model generates unconstrained output. With enable_thinking: false, grammar works correctly.

This was previously raised in #12276 (closed as stale, with a request to reopen). Filing fresh with a clean reproduction on a current build that includes the autoparser (#18675).

Evidence: grammar not enforced for ANY model with thinking ON

Tested two models with the same json_schema response_format requesting fields plan_summary, steps, acceptance_criteria:

Model	Thinking ON + response_format	Grammar enforced?
Qwen3.5-35B-A3B	500 "Failed to parse input" (fenced JSON in error body)	No — grammar didn't prevent fences, PEG parser rejects
Qwen3-VL-8B	200 but wrong schema (task_id, subtasks instead of requested fields)	No — grammar didn't enforce schema, but bare JSON passes PEG
Qwen3.5-35B-A3B (thinking OFF)	200, correct schema	Yes

The 500 with Qwen3.5 is just the loud symptom — that model wraps JSON in markdown fences, which the PEG parser rejects. The quieter failure (Qwen3-VL) shows the real bug: grammar is not applied at all when thinking is enabled. Models that naturally produce bare JSON appear to succeed but silently ignore the schema.

Reproduction

Build: includes #18675 (autoparser), commit d088d5b

# Start llama-server with thinking ON
llama-server --chat-template-kwargs '{"enable_thinking": true}' \
  --model Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf \
  --ctx-size 32768 --cont-batching --n-gpu-layers 999 --port 8081

curl -s http://localhost:8081/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
  "model": "Qwen3.5-35B-A3B-UD-Q4_K_XL",
  "messages": [
    {"role": "system", "content": "You are a systems architect. Produce a task plan as JSON matching the schema provided."},
    {"role": "user", "content": "Design a caching layer for a multi-tenant SaaS application with Redis and Memcached backends."}
  ],
  "temperature": 0,
  "max_tokens": 8192,
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "task_plan",
      "strict": true,
      "schema": {
        "type": "object",
        "properties": {
          "plan_summary": {"type": "string"},
          "steps": {
            "type": "array",
            "items": {
              "type": "object",
              "properties": {
                "step_number": {"type": "integer"},
                "description": {"type": "string"},
                "specialist": {"type": "string"}
              },
              "required": ["step_number", "description", "specialist"]
            }
          },
          "acceptance_criteria": {
            "type": "array",
            "items": {"type": "string"}
          }
        },
        "required": ["plan_summary", "steps", "acceptance_criteria"]
      }
    }
  }
}'

Result (thinking ON, Qwen3.5-35B-A3B) — 500:

{"error":{"code":500,"message":"Failed to parse input at pos 778: ```json\n{\"project\":{\"name\":\"Multi-Tenant SaaS Caching Layer\"...","type":"server_error"}}

Result (thinking OFF, same request) — correct schema:

{"finish_reason":"stop","content":"{\"plan_summary\":\"Develop a basic user authentication system.\",\"steps\":[{\"step_number\":1,...}],\"acceptance_criteria\":[...]}","usage":{"completion_tokens":69}}

Deterministic at temperature=0. Reproduced multiple times.

Environment

• llama-server built from d088d5b (includes autoparser Autoparser - complete refactoring of parser architecture #18675)
• Models tested: Qwen3.5-35B-A3B-UD-Q4_K_XL, Qwen3-VL-8B-Instruct-Q8_0
• --reasoning-format deepseek
• Linux, RTX 8000

Related

• Feature Request: grammar / json schema with reasoning format. Allow model free to think but strict to answer. #12276 — Original feature request for grammar + reasoning (closed as stale, reopen requested)
• Misc. bug: llama-server fails open when JSON schema grammar parsing fails #19051 — Grammar fails silently when GBNF parsing errors occur (closed as stale)
• Autoparser - complete refactoring of parser architecture #18675 — Autoparser (merged, present in this build, doesn't resolve this)
• vLLM and SGLang both support grammar enforcement with reasoning models

[pwilkin]

You literally retested one commit before the patch to this exact problem :)

62b8143

[shanevcantwell]

Thanks! I've been battling the same kind of problem in LM Studio for months. The reddit auto template announcement had me rushing to finally try out llama-server. Rebuilding now...

[shanevcantwell]

That cleared it up - thanks again for putting it together!

[danielaustralia1]

Testing update — Apr 30, 2026
Tested grammar passthrough on Lemonade 10.2.0 (RPM, Fedora, Strix Halo / gfx1151) using llama.cpp b8668 (bundled), b8940, and b8981 (via distrobox).
Grammar works on non-thinking models — confirmed on Llama-3.2-3B-Instruct-GGUF via Lemonade port 13305. Clean structured output, finish_reason: stop, correct token count.
Grammar is broken on Qwen3.x thinking models — this is a confirmed upstream llama.cpp bug: #20345. Grammar constraints leak into reasoning_content, causing the model to loop generating garbage while content stays empty. Workarounds attempted (--reasoning-budget 0, --chat-template-file with hardcoded enable_thinking=False) all failed — llama.cpp overrides at architecture level (thinking = 1 in startup logs regardless).
Conclusion: The Lemonade passthrough request is still valid and grammar works correctly on non-thinking models. The Qwen3 interaction is a separate upstream issue.

[danielaustralia1]

Additional reproduction — Apr 30, 2026 (b8981, Vulkan, gfx1151 Strix Halo)
Tested all available workarounds for grammar + thinking interaction on Qwen3.6-35B-A3B-Q8_0.gguf:

1. Raw grammar (no lazy) — grammar leaks into reasoning_content, model loops generating garbage (e.g. "nessness..."), content stays empty, hits token limit.
2. grammar_lazy + grammar_triggers: — the grammar itself blocks from being generated (the > char isn't in the allowed set), so the model can never exit the thinking block. Infinite loop in reasoning_content.
3. --reasoning-budget 0 — server crash: Unexpected empty grammar stack after accepting piece: / (14). Aborted with core dump.
4. --chat-template-file with hardcoded enable_thinking=False — extracted GBNF template from GGUF, modified to default thinking off, passed via --chat-template-file. Server loads the file but reports thinking = 1 at startup — overridden at architecture level regardless of template.
   Root cause: The grammar engine has no awareness of the thinking/non-thinking token boundary. It needs to apply only to content tokens, not reasoning_content tokens. None of the current workarounds bridge this gap cleanly