fix(gemini): base64-encode thoughtSignature bypass token to fix Vertex AI empty-response loop

[edelauna]

Related GitHub Issue

Closes: #536

Description

Gemini 3.x (including gemini-3.1-pro-preview and gemini-3.5-flash) validates Part.thoughtSignature strictly on Vertex AI. The field is documented by the Google GenAI SDK as "Encoded as base64 string". The fallback bypass token in gemini-format.ts was being sent as a plain string ("skip_thought_signature_validator"), while the same token in lite-llm.ts was already correctly base64-encoded.

Vertex AI enforces base64 validation more strictly than the direct Gemini API. When the plain string was sent on turn 2+ (after a tool call), Vertex returned an empty response, triggering the infinite retry loop reported in #536.

How:

• Added GEMINI_THOUGHT_SIGNATURE_BYPASS as a named export in src/api/transform/gemini-format.ts — the single source of truth for the base64-encoded bypass token.
• Updated gemini-format.ts to use the constant instead of the inline plain string.
• Updated lite-llm.ts to import and use the same constant, eliminating the duplicate Buffer.from(...) computation.

Reviewers should note:

• The direct Gemini API accepts both plain and base64 forms (confirmed via live API test), but Vertex AI requires base64. The fix is safe for both paths.
• thoughtSignature round-trip from real API responses is unaffected — only the fallback/bypass path changed.

Test Procedure

TDD approach: failing tests written first, then fix applied.

• src/api/transform/__tests__/gemini-format.spec.ts — updated existing tool-use test to assert base64-encoded bypass token; added assertion documents the exact Vertex AI failure mode.
• src/api/providers/__tests__/gemini.spec.ts — added thoughtSignature round-trip (issue #536) suite covering: capture from stream, round-trip through history on turn 2, base64 fallback for cross-model history, disabled-reasoning edge case, and no-tools boundary.
• src/api/providers/__tests__/lite-llm.spec.ts — existing tests continue to pass unchanged (they already asserted the base64 value).

pnpm exec vitest run api/transform/__tests__/gemini-format.spec.ts api/providers/__tests__/gemini.spec.ts api/providers/__tests__/lite-llm.spec.ts
# 3 files, 74 tests, all pass

Pre-Submission Checklist

• Issue Linked: This PR is linked to an approved GitHub Issue (see "Related GitHub Issue" above).
• Scope: My changes are focused on the linked issue (one major feature/fix per PR).
• Self-Review: I have performed a thorough self-review of my code.
• Testing: New and/or updated tests have been added to cover my changes (if applicable).
• Documentation Impact: I have considered if my changes require documentation updates (see "Documentation Updates" section below).
• Contribution Guidelines: I have read and agree to the Contributor Guidelines.

Documentation Updates

• No documentation updates are required.

Get in Touch

edelauna@gmail.com

Summary by CodeRabbit

• Bug Fixes

  • Improved compatibility with Gemini tool calls by preserving and reusing required reasoning metadata across streamed turns.
  • Added a safe fallback for cases where no prior signature is available, helping avoid validation issues in Gemini responses.
  • Ensured tool-related behavior remains consistent even when reasoning settings are disabled.

• Tests

  • Expanded coverage for Gemini streaming and tool-call scenarios, including edge cases around missing signatures.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds a shared GEMINI_THOUGHT_SIGNATURE_BYPASS base64-encoded constant in gemini-format.ts, used as fallback for Gemini functionCall.thoughtSignature when no prior signature exists. The lite-llm.ts provider now imports and reuses this constant instead of computing it inline. Tests updated/added accordingly.

Changes

thoughtSignature bypass token fix

Layer / File(s)	Summary
Bypass constant and core conversion logic src/api/transform/gemini-format.ts, src/api/transform/__tests__/gemini-format.spec.ts	Exports GEMINI_THOUGHT_SIGNATURE_BYPASS (base64-encoded "skip_thought_signature_validator") and uses it as the fallback functionCall.thoughtSignature in convertAnthropicContentToGemini; tests assert the base64-encoded value is produced.
LiteLLM provider bypass usage src/api/providers/lite-llm.ts	Imports and reuses the shared bypass constant for dummySignature in injectThoughtSignatureForGemini, replacing inline base64 computation, with updated comments.
GeminiHandler thoughtSignature round-trip tests src/api/providers/__tests__/gemini.spec.ts	Adds tests for capturing, persisting, and re-sending thoughtSignature across turns, fallback bypass token behavior, reasoningEffort-disabled behavior, and a no-tools boundary case.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant GeminiHandler
  participant GeminiAPI

  Client->>GeminiHandler: send turn 1 with tools
  GeminiHandler->>GeminiAPI: generateContentStream
  GeminiAPI-->>GeminiHandler: stream chunk with thoughtSignature + functionCall
  GeminiHandler->>GeminiHandler: store thoughtSignature via getThoughtSignature()
  Client->>GeminiHandler: send turn 2 (tool result)
  GeminiHandler->>GeminiAPI: generateContentStream with stored thoughtSignature on functionCall
  Note over GeminiHandler,GeminiAPI: falls back to GEMINI_THOUGHT_SIGNATURE_BYPASS if no stored signature

Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~15 minutes

Possibly related issues

• [BUG] Vertex AI Gemini 3.1/3.5: empty response loop after first tool-calling turn #536: Directly addresses the Vertex AI Gemini 3.1/3.5 empty-response loop caused by stale/invalid thoughtSignature, fixing the bypass token format to be properly base64-encoded.
• [BUG] Gemini Thought Summaries Degrade into Raw Reasoning Over the Zoo LiteLLM Provider #736: Both address Gemini thoughtSignature handling in the LiteLLM/Gemini pipeline, including the bypass token used on tool calls.

Suggested reviewers

• taltas
• navedmerchant
• hannesrudolph

A rabbit hops through streams of thought,
Signatures lost, then bravely caught.
Base64-wrapped, the bypass true,
No more loops to stumble through.
🐇✨ Hop on, Gemini, turn anew!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly states the core fix: encoding the Gemini thoughtSignature bypass token to resolve the Vertex AI empty-response loop.
Description check	✅ Passed	The description covers the issue link, implementation details, testing, checklist items, and reviewer notes, with only noncritical template sections omitted.
Linked Issues check	✅ Passed	The changes address #536 by base64-encoding the bypass token, preserving thoughtSignature round-tripping, and adding tests for the affected flows.
Out of Scope Changes check	✅ Passed	The diffs stay focused on the Gemini thoughtSignature fix and its tests, with no unrelated feature work or broad refactors.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch issue/536

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[coderabbitai]

🧹 Nitpick comments (3)

src/api/providers/__tests__/gemini.spec.ts (1)

176-176: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Duplicate base64 computation instead of importing the shared constant.

Same issue as in gemini-format.spec.ts: Buffer.from("skip_thought_signature_validator").toString("base64") reimplements GEMINI_THOUGHT_SIGNATURE_BYPASS instead of importing it from gemini-format.ts.

♻️ Proposed fix

+import { GEMINI_THOUGHT_SIGNATURE_BYPASS } from "../../transform/gemini-format"
...
-			const expectedBypass = Buffer.from("skip_thought_signature_validator").toString("base64")
+			const expectedBypass = GEMINI_THOUGHT_SIGNATURE_BYPASS

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/api/providers/__tests__/gemini.spec.ts` at line 176, The gemini test is
re-creating the bypass token instead of using the shared source of truth; update
the `gemini.spec.ts` test to import `GEMINI_THOUGHT_SIGNATURE_BYPASS` from
`gemini-format.ts` and use that constant in place of the inline
`Buffer.from(...).toString("base64")` computation. This keeps the expectation
aligned with the production value and avoids duplicating the base64 logic in the
test.

src/api/transform/__tests__/gemini-format.spec.ts (1)

126-130: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Import the shared constant instead of recomputing it.

expectedBypassToken reimplements the same Buffer.from(...).toString("base64") formula used in gemini-format.ts rather than importing GEMINI_THOUGHT_SIGNATURE_BYPASS. This makes the assertion tautological against the constant's own definition — it won't catch drift if the constant's value changes.

♻️ Proposed fix

-import { convertAnthropicMessageToGemini } from "../gemini-format"
+import { convertAnthropicMessageToGemini, GEMINI_THOUGHT_SIGNATURE_BYPASS } from "../gemini-format"
...
-		const expectedBypassToken = Buffer.from("skip_thought_signature_validator").toString("base64")
+		const expectedBypassToken = GEMINI_THOUGHT_SIGNATURE_BYPASS

Also applies to: 141-141

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/api/transform/__tests__/gemini-format.spec.ts` around lines 126 - 130,
The test in gemini-format.spec.ts is recomputing the bypass token instead of
verifying the shared value, so update the assertion to import and use
GEMINI_THOUGHT_SIGNATURE_BYPASS from gemini-format.ts (or the module that
defines it) rather than calling Buffer.from(...).toString("base64"). Keep the
expectation tied to the shared constant in the relevant test cases, including
the repeated assertion around the later line referenced in the review, so the
test will fail if the constant drifts.

src/api/providers/lite-llm.ts (1)

193-198: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Stale comment doesn't mention base64 encoding.

This comment (unchanged by this PR) still describes "skip_thought_signature_validator" as the bypass value without noting it must be base64-encoded — the exact distinction this PR fixes elsewhere. Worth aligning for consistency, though low priority since it's not on the changed lines.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/api/providers/lite-llm.ts` around lines 193 - 198, Update the stale
Gemini comment in lite-llm.ts so it matches the new behavior: the note around
isGeminiModel and the thought-signature bypass should explicitly mention that
"skip_thought_signature_validator" must be base64-encoded. Keep the existing
explanation about Gemini 3 tool-call validation, but align the wording with the
fix applied elsewhere so the comment is accurate and consistent.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/api/providers/__tests__/gemini.spec.ts`:
- Line 176: The gemini test is re-creating the bypass token instead of using the
shared source of truth; update the `gemini.spec.ts` test to import
`GEMINI_THOUGHT_SIGNATURE_BYPASS` from `gemini-format.ts` and use that constant
in place of the inline `Buffer.from(...).toString("base64")` computation. This
keeps the expectation aligned with the production value and avoids duplicating
the base64 logic in the test.

In `@src/api/providers/lite-llm.ts`:
- Around line 193-198: Update the stale Gemini comment in lite-llm.ts so it
matches the new behavior: the note around isGeminiModel and the
thought-signature bypass should explicitly mention that
"skip_thought_signature_validator" must be base64-encoded. Keep the existing
explanation about Gemini 3 tool-call validation, but align the wording with the
fix applied elsewhere so the comment is accurate and consistent.

In `@src/api/transform/__tests__/gemini-format.spec.ts`:
- Around line 126-130: The test in gemini-format.spec.ts is recomputing the
bypass token instead of verifying the shared value, so update the assertion to
import and use GEMINI_THOUGHT_SIGNATURE_BYPASS from gemini-format.ts (or the
module that defines it) rather than calling Buffer.from(...).toString("base64").
Keep the expectation tied to the shared constant in the relevant test cases,
including the repeated assertion around the later line referenced in the review,
so the test will fail if the constant drifts.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 2230a2c8-0516-4e7e-956e-98212f135049

📥 Commits

Reviewing files that changed from the base of the PR and between 67df9f9 and 6864ee1.

📒 Files selected for processing (4)

• src/api/providers/__tests__/gemini.spec.ts
• src/api/providers/lite-llm.ts
• src/api/transform/__tests__/gemini-format.spec.ts
• src/api/transform/gemini-format.ts

[taltas]

Clean PR, nice