Transcription confidence indicators

[madebysan]

Summary

Show a visual cue when Whisper is less confident about a transcription.

Details

• WhisperKit returns confidence/probability scores per segment
• Show an overall confidence indicator after transcription (e.g., green/yellow/red dot in history)
• Optionally highlight low-confidence words in the history view

Why

Helps users know when to double-check a transcription, especially for technical or unusual content.

[m13v]

We show confidence indicators in our transcription UI and here is what we found useful.

WhisperKit returns avgLogprob per segment in the TranscriptionResult. The mapping to a user-visible indicator: anything above -0.3 is high confidence (green), -0.3 to -0.7 is medium (yellow), below -0.7 is low (red). These thresholds are based on empirical testing - they correlate well with actual transcription accuracy in practice.

For per-word highlighting, WhisperKit also gives you token-level probabilities via tokenLogProbs in the result. You can map each token back to its word and highlight the low-confidence ones. The tricky part is that WhisperKit tokens do not always align cleanly with word boundaries (subword tokenization), so you need to merge adjacent low-confidence tokens into word groups.

One UX thing we learned: showing per-word confidence highlights by default is too noisy. It makes every transcript look unreliable because Whisper is always slightly unsure about some words. We show the overall segment confidence (the colored dot you described) by default, and only show per-word highlights when the user clicks on a segment to inspect it.

Also worth considering: confidence drops significantly at the edges of audio chunks. The first and last 0.5 seconds of each 30-second Whisper window tend to have lower confidence even when the transcription is correct. If you are chunking long audio, do not count edge tokens toward the segment confidence score.

[m13v]

Our transcription service that processes WhisperKit confidence scores and handles chunk edge effects: https://github.com/m13v/fazm/blob/main/Desktop/Sources/TranscriptionService.swift