GuidesUnderstanding AI

AI voice assistant latency: where the wait comes from, and how to cut it

By the Banzai Labs teamUpdated on 2 min read

SpeaksIn sequenceListeningThinkingVoiceStreamingListeningThinkingVoiceanswer soonerEach step starts before the previous one has finished
With streaming, listening, thinking and voice overlap and the answer arrives sooner.

The steps

  1. 1End of speechKnowing the person has finished.
  2. 2TranscriptionVoice becomes text.
  3. 3ThinkingThe model prepares the answer.
  4. 4SynthesisText becomes a voice.

In sequence or streaming

Two ways to chain the steps
In sequence
  • Each step waits for the previous one
  • Delays add up
  • Simpler to build
Streaming
  • The voice starts from the first sentence
  • Steps overlap
  • The answer starts much sooner

The levers

Lever Effect
Streamed model response Synthesis starts on the first sentence
“Short sentences” instruction The first chunk arrives sooner
Tuned end-of-speech detection Neither cutting in nor needless waiting
Faster model for conversation The largest kept for hard cases
Servers close to users Less network time

The technique: streaming AI responses. Monitoring: observing an AI application.

Frequently asked questions

What delay feels natural?
In conversation, people usually reply in under a second. Beyond that, the other person feels misunderstood. Measure the time between the end of the sentence and the first sound of the answer.
Do speech-to-speech models solve the problem?
They remove steps and cut the delay. In exchange, you have less fine control over each step (transcribed text, voice choice, logging).
What about during a long tool call?
Say a short holding phrase (“Let me look up your order”) while the tool works, rather than staying silent.

Read next

All guides