Skip to content

Lo/failsafe for utterance end #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Moodbot11 wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Lo/failsafe for utterance end #1

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0e6e568
feat: create failsafe for incomplete speech final
lukeocodes Mar 6, 2024
7f2202e
chore: add logging to failsafe
lukeocodes Mar 6, 2024
e22fba2
chore: reduce failsafe timeout
lukeocodes Mar 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 54 additions & 4 deletions app/components/Conversation.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,9 @@ export default function Conversation(): JSX.Element {
useEffect(() => {
const onTranscript = (data: LiveTranscriptionEvent) => {
let content = utteranceText(data);
if (content || data.speech_final) {

// i only want an empty transcript part if it is speech_final
if (content !== "" || data.speech_final) {
/**
* use an outbound message queue to build up the unsent utterance
*/
Expand Down Expand Up @@ -240,14 +242,16 @@ export default function Conversation(): JSX.Element {
};
}, [addTranscriptPart, connection]);

const [currentUtterance, setCurrentUtterance] = useState("");
const [currentUtterance, setCurrentUtterance] = useState<string>();

const getCurrentUtterance = useCallback(() => {
return transcriptParts.filter(({ is_final, speech_final }, i, arr) => {
return is_final || speech_final || (!is_final && i === arr.length - 1);
});
}, [transcriptParts]);

const [lastUtterance, setLastUtterance] = useState<number>();
Copy link
Copy Markdown

@sourcery-ai sourcery-ai Bot Aug 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (complexity): Consider simplifying the incomplete speech handling logic to reduce complexity.

The current implementation adds unnecessary complexity to handle incomplete speech scenarios. Instead of introducing new state variables and effects, consider simplifying the approach:

  1. Remove the lastUtterance state and the separate effect for the incomplete speech failsafe.
  2. Modify the existing useEffect that handles transcripts to include a simple timeout check:
useEffect(() => {
  const parts = getCurrentUtterance();
  const content = parts
    .map(({ text }) => text)
    .join(" ")
    .trim();

  if (!content) return;

  setCurrentUtterance(content);

  let timeoutId;

  if (last && last.speech_final) {
    sendUtterance(content);
  } else {
    // Set a timeout for incomplete speech
    timeoutId = setTimeout(() => {
      sendUtterance(content);
    }, 1500);
  }

  return () => {
    if (timeoutId) clearTimeout(timeoutId);
  };
}, [getCurrentUtterance, last]);

const sendUtterance = (content) => {
  append({
    role: "user",
    content,
  });
  clearTranscriptParts();
  setCurrentUtterance(undefined);
};

This approach simplifies the logic by:

  • Eliminating the need for a separate state variable and effect
  • Using a single timeout that's cleared if speech_final is received
  • Consolidating the utterance sending logic into a single function

This maintains the desired functionality of handling incomplete speech while reducing overall complexity and improving readability.


useEffect(() => {
const parts = getCurrentUtterance();
const last = parts[parts.length - 1];
Expand All @@ -256,20 +260,66 @@ export default function Conversation(): JSX.Element {
.join(" ")
.trim();

if (content === "") return;
/**
* if the entire utterance is empty, don't go any further
* for example, many many many empty transcription responses
*/
if (!content) return;
Copy link
Copy Markdown

@sourcery-ai sourcery-ai Bot Aug 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (code-quality): Use block braces for ifs, whiles, etc. (use-braces)

Suggested change
if (!content) return;
if (!content) {


ExplanationIt is recommended to always use braces and create explicit statement blocks.

Using the allowed syntax to just write a single statement can lead to very confusing
situations, especially where subsequently a developer might add another statement
while forgetting to add the braces (meaning that this wouldn't be included in the condition).


/**
* display the concatenated utterances
*/
setCurrentUtterance(content);

/**
* record the last time we recieved a word
*/
if (last.text !== "") {
setLastUtterance(Date.now());
}

/**
* if the last part of the utterance, empty or not, is speech_final, send to the LLM.
*/
if (last && last.speech_final) {
append({
role: "user",
content,
});
clearTranscriptParts();
setCurrentUtterance("");
setCurrentUtterance(undefined);
}
}, [getCurrentUtterance, clearTranscriptParts, append]);

/**
* incomplete speech final failsafe
*/
useEffect(() => {
if (!lastUtterance || !currentUtterance) return;
Copy link
Copy Markdown

@sourcery-ai sourcery-ai Bot Aug 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (code-quality): Use block braces for ifs, whiles, etc. (use-braces)

Suggested change
if (!lastUtterance || !currentUtterance) return;
if (!lastUtterance || !currentUtterance) {


ExplanationIt is recommended to always use braces and create explicit statement blocks.

Using the allowed syntax to just write a single statement can lead to very confusing
situations, especially where subsequently a developer might add another statement
while forgetting to add the braces (meaning that this wouldn't be included in the condition).


const interval = setInterval(() => {
const timeLived = Date.now() - lastUtterance;

console.log(timeLived, timeLived > 1500, currentUtterance);

if (currentUtterance !== "" && timeLived > 1500) {
console.log("failsafe fires! pew pew!!");

append({
role: "user",
content: currentUtterance,
});
clearTranscriptParts();
setCurrentUtterance(undefined);
}
}, 100);

return () => {
clearInterval(interval);
};
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [lastUtterance, currentUtterance]);

/**
* barge-in
*/
Expand Down