Skip to content

Preemptive responses #57#98

Open
tanwir1703 wants to merge 5 commits into
think41:mainfrom
tanwir1703:main
Open

Preemptive responses #57#98
tanwir1703 wants to merge 5 commits into
think41:mainfrom
tanwir1703:main

Conversation

@tanwir1703

Copy link
Copy Markdown

Please find the updates below on the preemptive response issue number #57 against the requirements.
Requirements:

  1. Allow configuration of one or more preemptive phrases (global + intent-level) - Done ☑️
  2. Trigger a preemptive response based on a configurable latency threshold (e.g., 300ms) - Done ☑️
  3. Support fallback behavior (e.g., skip preemptive response if actual response is ready quickly) - Almost Done ☑️ (Please note that while the fallback behaviour is working fine for large thresholds like 6+s, when I am giving a threshold of 3-4 seconds, the preemptive response is sometimes triggered after the LLM response)
  4. Support TTS integration for seamless voice playback - Done ☑️
  5. Optional: Use different fillers depending on the user intent (context-aware preemptives) - Done ☑️ (Last user text is being fetched using TranscriptionUpdateFrame inside the preemptive block, and intent-driven preemptive responses are provided to the user. However, the intent derivation logic is currently elementary and can be easily configured in the future.)
  6. Should be integrated within the voice orchestration layer (likely between STT and LLM) - Done ☑️
  7. Ensure preemptive responses don’t conflict or overlap with final responses - Done ☑️
  8. Possible need for an interrupt or cancel mechanism if preemptive and actual responses collide - Done ☑️
  9. Should work well across both streaming and non-streaming response types - Partially Done ☑️ (I couldn't test it for streaming responses, but it works well with non-streaming responses).

Steps to test it out:

  1. basic_agent.json consists of necessary preemptive phrases and threshold values.
  2. Run main.py(to access /ws endpoint) inside the examples folder, following the steps in readme.md
  3. Navigate to foundation-voice/examples/websocket endpoint, run the client and server to test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant