fix(examples-chat): handler hooks on_llm_new_token (not on_chat_model_stream)#264
Merged
Merged
Conversation
…at_model_stream) PR #262 originally hooked into on_chat_model_stream, but that method does not exist on AsyncCallbackHandler — the canonical streaming-token callback (fired by ChatOpenAI when streaming=True) is on_llm_new_token, which delivers each token plus an optional ChatGenerationChunk whose message field carries the AIMessageChunk with tool_call_chunks. Result of the bug: the handler was wired but never invoked at runtime, so adispatch_custom_event never fired and the frontend bridge stayed dormant. Live smoke at /embed confirmed: zero a2ui-partial events on the wire across a 622KB / 412-chunk stream. This hotfix: - Renames the override to on_llm_new_token with the canonical signature (token: str, *, chunk, run_id, parent_run_id, tags, **kwargs). - Reads tool_call_chunks from chunk.message (ChatGenerationChunk wraps the AIMessageChunk in its message field). - Gracefully handles chunk=None (legacy LLM path with no chunk object). - Updates tests to wrap AIMessageChunk in ChatGenerationChunk and call the new method name. Adds a 6th test asserting chunk=None is a no-op.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR #262 wired the streaming callback handler to a method that doesn't exist.
AsyncCallbackHandlerhas noon_chat_model_stream— the canonical streaming-token callback (fired byChatOpenAI(streaming=True)) ison_llm_new_token. The handler was attached but never invoked, soadispatch_custom_eventnever fired.Smoking gun from live verification: 622 KB of stream traffic / 412 SSE chunks during a representative GenUI turn, zero
a2ui-partialevents on the wire. Frontend bridge sat dormant; the surface still rendered correctly via PR #255's wrapped-AIMessage classifier fallback, but the per-component fallback transition (the whole point of this work) was invisible.Fix
on_llm_new_tokenwith the canonical signature(token: str, *, chunk, run_id, parent_run_id, tags, **kwargs).tool_call_chunksfromchunk.message(the AIMessageChunk wrapped insideChatGenerationChunk).chunk=None(legacy LLM path).Tests
chunk=Noneis a no-op (defends against legacy LLM emitters).Test plan
pytest tests/test_a2ui_partial_handler.py tests/test_streaming_smoke.py— 7/7 greena2ui-partialevents appear on the SSE stream and the frontend bridge populatesliveSurfaceStoremid-stream