Blog · · 6 min read

Barge-In and Turn-Taking in Voice AI

A phone AI sounds natural when it stops as soon as the caller speaks and replies only once they've finished. How barge-in and turn detection work, and why they misfire.

Key points

  • A voice agent sounds natural when it does two things well, falling silent as soon as the caller starts talking (barge-in) and speaking only once the caller has actually finished (turn detection).
  • Barge-in has to stop speech synthesis and also flush audio already queued on the telephony side, for example with the clear message in Twilio Media Streams.
  • False interruptions and talking over the caller are not only an audio problem; short replies, one question at a time and a single read-back at the end reduce them too.

Most of what makes a phone bot feel robotic comes down to turn-taking. If the agent stops talking the instant the caller cuts in (barge-in), and only starts talking once the caller is really done (turn detection), the conversation feels close to talking with a person. This post explains how both mechanisms work, the usual ways they fail, and what you can fix in conversation design rather than in audio code.

Two decisions behind every turn

Throughout a call, the agent is constantly deciding whose turn it is. That breaks down into two separate decisions.

Decision Question it answers What failure sounds like
Barge-in Has the caller started speaking while the agent is talking? The agent talks over the caller, or goes quiet because of a cough
Turn detection (endpointing) Has the caller finished, or are they just pausing? The agent jumps in mid-sentence, or replies sluggishly

Both start from voice activity detection (VAD), which classifies short frames of audio as speech or not speech. VAD alone can't tell a thoughtful pause from the end of a sentence, so modern voice stacks combine it with a model that looks at the words and the way they were said to judge whether the turn is complete.

What barge-in has to stop

When the caller starts speaking over the agent, three things need to stop, not one:

  1. Speech synthesis. Don't generate the rest of the sentence.
  2. The LLM response. Drop the unspoken remainder and wait for what the caller is saying now.
  3. Audio already sent to the phone. Flush anything queued for playback.

The third one is easy to miss. Voice agents normally stream synthesized audio ahead of playback, so even after synthesis stops, buffered audio keeps playing and the caller hears an agent that "won't shut up." With Twilio Media Streams you send a clear message over the WebSocket to interrupt audio you've already sent (Twilio docs).

It also matters what you record in the conversation history. If the agent was cut off halfway through a sentence, the caller only heard the first half. If you store the full sentence, the LLM will assume it already delivered the information and won't repeat it. Twilio's mark messages, which tell you when a chunk of audio has finished playing, let you record only what actually reached the caller.

Why barge-in misfires

Barge-in can be too sensitive or not sensitive enough. These are the patterns that show up most often.

Symptom Usual cause Direction of the fix
A cough or background noise silences the agent Noise classified as speech Require speech to last a minimum duration before treating it as an interruption
"Uh-huh" or "yeah" stops the agent Backchannels treated as interruptions Keep agent turns short, especially right after confirmation questions
Caller talks, agent keeps going Detection too slow, or queued audio not flushed Stop synthesis and flush the telephony buffer together
Agent interrupts itself Speaker output leaking into the mic (mostly web and mobile) Turn on echo cancellation; suggest headphones

On phone calls the handset usually removes most of the echo. In browsers you need echoCancellation enabled on getUserMedia, otherwise the agent's own voice comes back through the mic and triggers barge-in (MDN).

Endpointing is a trade-off

Turn detection trades speed against accuracy. Wait longer after the caller goes quiet and you cut people off less often, but every reply feels slower. Wait less and replies feel snappy, but the agent starts talking while the caller is still reading out a phone number.

Mid-turn pauses cluster in predictable places:

  1. Numbers. People read phone numbers, dates and order IDs in chunks, with gaps between them.
  2. Recall. "Um, I think it was next... Thursday?"
  3. Spelling. "That's Smith, S-M-I-T-H."

Judging completeness from the words, not just from silence, cuts down on interruptions in exactly these moments. For where the rest of the response time goes, see Voice AI latency.

Fixes that live in conversation design

You can prevent a lot of turn-taking problems in the prompt, without touching audio code.

  1. Keep replies to one or two sentences. If the caller interrupts a long explanation, you lose less, and there are fewer moments where a backchannel lands mid-speech.
  2. Ask one question at a time. "Can I get your name and phone number?" makes callers pause after the name, which invites the agent to jump in.
  3. Tell callers they can take their time. Adding "no rush" before asking for a number reduces rushed, choppy delivery.
  4. After an interruption, answer the caller; don't resume. Responding to what they just said sounds natural. Restarting the cut-off sentence doesn't.
  5. Read back once, at the end. Confirming each field separately creates many spots for "yes" to collide with the agent's next sentence. A single read-back at the end is calmer.

Speaking rate matters too. Too fast and callers can't find a gap to cut in; too slow and they start talking before the agent is done. See text-to-speech for phone calls for picking a voice and pace.

Doing this with voicast

voicast stops talking and listens as soon as the caller starts speaking. Endpointing, stopping synthesis and flushing audio on the telephony side are handled by the voice server, so there's nothing to build on your side. In the agent config, you shape turn length and question order in persona, and choose a pace with speaking_rate.

If you use a text-only session (output: 'text', for pairing voicast with an avatar or another speaker), the data channel delivers { type: 'voicast.interrupted' } when the caller starts talking. Tell your speaker to stop the reply in progress when you receive it; otherwise whatever is producing the voice keeps going.

FAQ

Can I turn barge-in off?

For phone agents, we recommend leaving it on. If the agent keeps talking while the caller is speaking, the caller's only options are to repeat themselves or hang up. False interruptions are usually tuned by requiring a minimum speech duration and by keeping agent replies short.

Backchannels like "uh-huh" keep stopping the agent. What helps?

Backchannels come when the agent talks for a long time and right after confirmation questions. Shorter replies and a single read-back at the end remove most of the moments where it happens.

In my web app, the agent interrupts itself. Why?

The microphone is almost certainly picking up the speaker. Pass echoCancellation: true to getUserMedia and suggest headphones where you can. This rarely happens on phone calls.

Related posts