Skip to content

Harden AI plan parsing against truncated LLM JSON responses#21

Merged
billrichards merged 4 commits into
mainfrom
copilot/improve-llm-response-parsing
Jun 2, 2026
Merged

Harden AI plan parsing against truncated LLM JSON responses#21
billrichards merged 4 commits into
mainfrom
copilot/improve-llm-response-parsing

Conversation

Copilot AI commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

AI plan generation could fail when the LLM returned near-valid JSON with a truncated string/container (e.g., Unterminated string). This change makes parsing tolerant of common truncation patterns so valid plans can still be recovered instead of aborting planning.

  • Parsing resilience

    • Added a targeted JSON repair path in AIPlannerClient._parse_json:
      • parse normally first (json.loads)
      • on JSONDecodeError, attempt truncation repair
      • accept repaired payload only if it parses to a top-level object
    • Kept existing strict failure behavior for non-recoverable or structurally invalid payloads.
  • Truncation repair helper

    • Added _repair_truncated_json(text) to close:
      • unterminated string literals
      • unmatched { / [ containers
    • Rejects malformed mismatched closers instead of over-correcting.
  • Regression coverage

    • Added focused test for a truncated warning-string response to ensure parser recovery returns a valid TestPlan with preserved warning content.
try:
    data = json.loads(stripped)
except json.JSONDecodeError:
    repaired = _repair_truncated_json(stripped)
    if repaired is not None:
        data = json.loads(repaired)  # accepted only when dict

Copilot AI changed the title [WIP] Fix parsing of LLM responses to handle invalid JSON Harden AI plan parsing against truncated LLM JSON responses Jun 2, 2026
Copilot AI requested a review from billrichards June 2, 2026 23:26
…nsive tests

- Fix escape sequence bug by returning None for mid-escape truncation
- Remove dead code assignment in JSON parsing error handler
- Add logging to track successful truncated JSON repairs
- Add comprehensive test coverage for edge cases:
  * Mid-escape truncation (verifies it fails safely)
  * Nested structure truncation
  * Valid JSON (verifies repair returns None)
  * Single unclosed string repair
  * Multiple unclosed containers
  * Mismatched brackets rejection
  * Non-object JSON rejection

Co-authored-by: openhands <openhands@all-hands.dev>
The test was using double-backslash which created a literal backslash
character instead of a newline escape sequence. Changed to use actual
newline character so json.dumps creates the \n escape sequence that can
be truncated for proper testing.

Co-authored-by: openhands <openhands@all-hands.dev>
@billrichards billrichards marked this pull request as ready for review June 2, 2026 23:54
@billrichards billrichards merged commit 1d01ea9 into main Jun 2, 2026
15 checks passed
@billrichards billrichards deleted the copilot/improve-llm-response-parsing branch June 5, 2026 00:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LLM returned invalid JSON: Unterminated string

3 participants