Skip to content

fix: parse Gemma 4 <thought> reasoning tags alongside <think> #324

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
edelauna merged 7 commits into from
Jun 26, 2026
Merged

fix: parse Gemma 4 <thought> reasoning tags alongside <think> #324

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
999ce64
fix: parse Gemma 4 <thought> reasoning tags alongside <think>
May 25, 2026
ae1a395
fix: support nested reasoning tags in TagMatcher and add comprehensiv…
May 26, 2026
7d66f89
test(tag-matcher): add tests for unmatched closing tags
May 26, 2026
f24b964
Apply suggestion from @edelauna
sagidM Jun 25, 2026
63c1066
refactor(providers): updating tag matcher
edelauna Jun 26, 2026
c7e3a5a
test(TagMatcher): adding tests for new logic
edelauna Jun 26, 2026
3711c0d
test(tag-matcher): consolidate reasoning tag tests into tag-matcher.s…
edelauna Jun 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 71 additions & 7 deletions src/api/providers/__tests__/base-openai-compatible-provider.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,75 @@ describe("BaseOpenAiCompatibleProvider", () => {
])
})

it("should handle reasoning tags (<thought>) from stream", async () => {
Comment thread
sagidM marked this conversation as resolved.
mockCreate.mockImplementationOnce(() => {
return {
[Symbol.asyncIterator]: () => ({
next: vi
.fn()
.mockResolvedValueOnce({
done: false,
value: { choices: [{ delta: { content: "<thought>Deep thought" } }] },
})
.mockResolvedValueOnce({
done: false,
value: { choices: [{ delta: { content: " here</thought>" } }] },
})
.mockResolvedValueOnce({
done: false,
value: { choices: [{ delta: { content: "Result: 42" } }] },
})
.mockResolvedValueOnce({ done: true }),
}),
}
})
const stream = handler.createMessage("system prompt", [])
const chunks = []
for await (const chunk of stream) {
chunks.push(chunk)
}
expect(chunks).toEqual([
{ type: "reasoning", text: "Deep thought" },
{ type: "reasoning", text: " here" },
{ type: "text", text: "Result: 42" },
])
})

it("should not close <think> tag with </thought> tag", async () => {
mockCreate.mockImplementationOnce(() => {
return {
[Symbol.asyncIterator]: () => ({
next: vi
.fn()
.mockResolvedValueOnce({
done: false,
value: { choices: [{ delta: { content: "<think>Thinking" } }] },
})
.mockResolvedValueOnce({
done: false,
value: { choices: [{ delta: { content: " but closing with wrong tag</thought>" } }] },
})
.mockResolvedValueOnce({
done: false,
value: { choices: [{ delta: { content: " still thinking" } }] },
})
.mockResolvedValueOnce({ done: true }),
}),
}
})
const stream = handler.createMessage("system prompt", [])
const chunks = []
for await (const chunk of stream) {
chunks.push(chunk)
}
// The </thought> tag should be treated as text since it doesn't match the active <think> tag
expect(chunks).toEqual([
{ type: "reasoning", text: "Thinking" },
{ type: "reasoning", text: " but closing with wrong tag</thought>" },
{ type: "reasoning", text: " still thinking" },
])
})

it("should handle complete <think> tag in a single chunk", async () => {
mockCreate.mockImplementationOnce(() => {
return {
Expand Down Expand Up @@ -153,13 +222,8 @@ describe("BaseOpenAiCompatibleProvider", () => {
chunks.push(chunk)
}

// TagMatcher should handle incomplete tags and flush remaining content
expect(chunks.length).toBeGreaterThan(0)
expect(
chunks.some(
(c) => (c.type === "text" || c.type === "reasoning") && c.text.includes("Incomplete thought"),
),
).toBe(true)
// TagMatcher should flush incomplete reasoning content on stream end
expect(chunks).toContainEqual({ type: "reasoning", text: "Incomplete thought" })
})

it("should handle text without any <think> tags", async () => {
Expand Down
169 changes: 169 additions & 0 deletions src/api/providers/__tests__/openai.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -636,6 +636,175 @@ describe("OpenAiHandler", () => {
const callArgs = mockCreate.mock.calls[0][0]
expect(callArgs.max_completion_tokens).toBe(4096)
})

describe("TagMatcher reasoning tags", () => {
it("should treat stray closing tag as plain text when no tag is open", async () => {
mockCreate.mockImplementationOnce(() => ({
[Symbol.asyncIterator]: () => ({
next: vi
.fn()
.mockResolvedValueOnce({
done: false,
value: { choices: [{ delta: { content: "final</think>text" } }] },
})
.mockResolvedValueOnce({ done: true }),
}),
}))

const stream = handler.createMessage(systemPrompt, messages)
const chunks: any[] = []
for await (const chunk of stream) {
chunks.push(chunk)
}

expect(chunks).toEqual([{ type: "text", text: "final</think>text" }])
})

it("should treat extra closing tag after a closed block as plain text", async () => {
mockCreate.mockImplementationOnce(() => ({
[Symbol.asyncIterator]: () => ({
next: vi
.fn()
.mockResolvedValueOnce({
done: false,
value: {
choices: [{ delta: { content: "<think>thinking</think>final</think>text" } }],
},
})
.mockResolvedValueOnce({ done: true }),
}),
}))

const stream = handler.createMessage(systemPrompt, messages)
const chunks: any[] = []
for await (const chunk of stream) {
chunks.push(chunk)
}

expect(chunks).toEqual([
{ type: "reasoning", text: "thinking" },
{ type: "text", text: "final</think>text" },
])
})

it("should handle nested mixed tags with correct closure matching", async () => {
mockCreate.mockImplementationOnce(() => ({
[Symbol.asyncIterator]: () => ({
next: vi
.fn()
.mockResolvedValueOnce({
done: false,
value: { choices: [{ delta: { content: "<think>outer" } }] },
})
.mockResolvedValueOnce({
done: false,
value: { choices: [{ delta: { content: "<thought>inner</thought>" } }] },
})
.mockResolvedValueOnce({
done: false,
value: { choices: [{ delta: { content: " middle</think>" } }] },
})
.mockResolvedValueOnce({
done: false,
value: { choices: [{ delta: { content: "final text" } }] },
})
.mockResolvedValueOnce({ done: true }),
}),
}))

const stream = handler.createMessage(systemPrompt, messages)
const chunks: any[] = []
for await (const chunk of stream) {
chunks.push(chunk)
}

// With the tag stack fix, </thought> closes <thought> inner tag,
// and </think> correctly closes the outer <think> tag.
// inner content inside <thought> is reasoning, middle is still reasoning under <think>
expect(chunks).toEqual([
{ type: "reasoning", text: "outer" },
{ type: "reasoning", text: "<thought>inner</thought>" },
{ type: "reasoning", text: " middle" },
{ type: "text", text: "final text" },
])
})

it("should handle nested <think> tags with correct stack unwinding", async () => {
mockCreate.mockImplementationOnce(() => ({
[Symbol.asyncIterator]: () => ({
next: vi
.fn()
.mockResolvedValueOnce({
done: false,
value: { choices: [{ delta: { content: "<think>outer" } }] },
})
.mockResolvedValueOnce({
done: false,
value: { choices: [{ delta: { content: "<think>inner</think>" } }] },
})
.mockResolvedValueOnce({
done: false,
value: { choices: [{ delta: { content: " middle</think>" } }] },
})
.mockResolvedValueOnce({
done: false,
value: { choices: [{ delta: { content: "final text" } }] },
})
.mockResolvedValueOnce({ done: true }),
}),
}))

const stream = handler.createMessage(systemPrompt, messages)
const chunks: any[] = []
for await (const chunk of stream) {
chunks.push(chunk)
}

// With the tag stack fix, </thought> closes <thought> inner tag,
// and </think> correctly closes the outer <think> tag.
// inner content inside <thought> is reasoning, middle is still reasoning under <think>
expect(chunks).toEqual([
{ type: "reasoning", text: "outer" },
{ type: "reasoning", text: "<think>inner</think>" },
{ type: "reasoning", text: " middle" },
{ type: "text", text: "final text" },
])
})

it("should handle reasoning_content alongside tag matching", async () => {
mockCreate.mockImplementationOnce(() => ({
[Symbol.asyncIterator]: () => ({
next: vi
.fn()
.mockResolvedValueOnce({
done: false,
value: { choices: [{ delta: { reasoning_content: "native reasoning" } }] },
})
.mockResolvedValueOnce({
done: false,
value: { choices: [{ delta: { content: "<think>tag based</think>" } }] },
})
.mockResolvedValueOnce({
done: false,
value: { choices: [{ delta: { content: " final output" } }] },
})
.mockResolvedValueOnce({ done: true }),
}),
}))

const stream = handler.createMessage(systemPrompt, messages)
const chunks: any[] = []
for await (const chunk of stream) {
chunks.push(chunk)
}

expect(chunks).toEqual([
{ type: "reasoning", text: "native reasoning" },
{ type: "reasoning", text: "tag based" },
{ type: "text", text: " final output" },
])
})
})
})

describe("error handling", () => {
Expand Down
2 changes: 1 addition & 1 deletion src/api/providers/base-openai-compatible-provider.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ export abstract class BaseOpenAiCompatibleProvider<ModelName extends string>
const stream = await this.createStream(systemPrompt, messages, metadata)

const matcher = new TagMatcher(
"think",
["think", "thought"],

@edelauna edelauna Jun 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LmStudioHandler (lm-studio.ts:107) and NativeOllamaHandler (native-ollama.ts:217) both extend BaseProvider directly with their own createMessage() and TagMatcher instances — they still pass only "think". Should those be updated to ["think", "thought"] too?

@sagidM sagidM Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

// class TagMatcher
constructor(
	tagName: string | string[],
	readonly transform?: (chunks: TagMatcherResult) => Result,
	readonly position = 0,
) {
	this.tagNames = Array.isArray(tagName) ? tagName : [tagName]
}

should not matter I guess

(chunk) =>
({
type: chunk.matched ? "reasoning" : "text",
Expand Down
2 changes: 1 addition & 1 deletion src/api/providers/lm-studio.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ export class LmStudioHandler extends BaseProvider implements SingleCompletionHan
}

const matcher = new TagMatcher(
"think",
["think", "thought"],
(chunk) =>
({
type: chunk.matched ? "reasoning" : "text",
Expand Down
2 changes: 1 addition & 1 deletion src/api/providers/native-ollama.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -215,7 +215,7 @@ export class NativeOllamaHandler extends BaseProvider implements SingleCompletio
]

const matcher = new TagMatcher(
"think",
["think", "thought"],
(chunk) =>
({
type: chunk.matched ? "reasoning" : "text",
Expand Down
2 changes: 1 addition & 1 deletion src/api/providers/openai.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ export class OpenAiHandler extends BaseProvider implements SingleCompletionHandl
}

const matcher = new TagMatcher(
"think",
["think", "thought"],
(chunk) =>
({
type: chunk.matched ? "reasoning" : "text",
Expand Down
Loading
Loading