Turn-taking, endpointing, and barge-in: how voice agents know when to speak

A voice agent knows when to speak by combining audio-level speech detection, streaming transcript evidence, timing rules, and a conversation state machine. Endpointing decides that the caller's current turn is complete; the response pipeline starts work and plays audio; and barge-in logic keeps monitoring the caller so it can stop that playback when the caller begins a new turn. No single silence threshold solves the problem: reliable turn-taking comes from coordinating several imperfect signals while accounting for network and processing delay.

The call is a pair of continuous media streams

Once a phone or browser call is connected, audio usually arrives as a sequence of encoded media packets. The receiving side decodes those packets into short frames and feeds them to several consumers at once: speech detection, transcription, recording, and sometimes echo control or audio-quality monitoring. The outbound side generates speech, encodes it, and schedules packets back onto the call.

That matters because a conversation is not naturally divided into request-and-response messages. The media stream does not contain a marker that says, "the caller's sentence ended here." A caller may pause to remember an account number, breathe between clauses, or wait for a noisy truck to pass. The system has to infer turn boundaries from the stream.

The first layer is usually voice activity detection, or VAD. A VAD classifies short spans of audio as speech or non-speech using signals such as energy, spectral shape, and learned speech patterns. It can provide events like speech_started and speech_stopped, but those labels are only acoustic observations. They do not establish that a thought is complete.

Endpointing turns acoustic evidence into a decision

Endpointing is the policy that converts incoming evidence into "the caller has finished this turn." A basic endpoint waits for speech followed by a configured duration of non-speech. That works for clean, command-like input, but it creates two opposite errors:

• A short timeout cuts off deliberate speakers or treats an internal pause as the end of a turn.
• A long timeout avoids premature replies but makes every exchange feel sluggish.

Production endpointing therefore tends to use more than one clock. A start-of-speech timer handles a caller who says nothing. A trailing-silence timer starts after speech. A maximum-utterance timer prevents one turn from remaining open forever. Some systems also use an absolute response deadline so a stalled downstream component cannot leave the caller in silence.

Streaming transcription adds linguistic evidence. Partial transcripts arrive while the caller is speaking and may be revised as more audio supplies context. An endpointer can ask whether the latest words form a syntactically or semantically complete request. "Can you book me for" followed by a pause is probably incomplete; "Can you book me for Tuesday morning?" is much stronger evidence of completion. Acoustic and linguistic evidence can be combined into a score, or applied as rules that shorten or extend the silence window.

The transcript is not ground truth. Background speech can produce words, and the final transcript can differ from the last partial. A robust implementation associates every partial and final result with an utterance or turn identifier. Late results from an old turn must not overwrite the transcript for a newer one.

A conversation controller owns the floor

The cleanest mental model is a state machine. Exact names vary, but a useful minimal set is:

1. Listening: inbound audio is monitored and transcribed. No answer audio is playing.
2. Committing: the endpoint policy has closed the caller's turn. The controller freezes the input used for the response.
3. Responding: response text or another action is being produced.
4. Speaking: outbound audio is being generated, buffered, and played.
5. Interrupted: new caller speech has invalidated some or all of the current response.

The controller, rather than any individual speech component, should decide who owns the conversational floor. It knows whether detected speech is likely to be the caller, leaked playback, a brief acknowledgement, or audio that arrived too late to belong to the current state. It also knows whether a tool action can be canceled safely.

Generation and playback are distinct. An answer may be produced faster than it can be spoken, so the application can have text waiting, synthesized audio waiting, audio in a network buffer, and audio already heard by the caller. Canceling only the text generator leaves queued speech playing. Clearing only the local audio queue may still leave packets buffered elsewhere. Interruption handling needs explicit cancellation and flush operations at each stage, plus turn IDs so stale callbacks are ignored.

Barge-in is controlled cancellation, not just speech detection

Barge-in lets a caller interrupt while the agent is talking. During outbound playback, the input path remains open. When the system concludes that the caller has started a real utterance, it lowers or stops playback, cancels work that is no longer useful, and returns the controller to listening.

The hard part is deciding what counts as a real interruption. If every VAD-positive frame stops the response, a cough, keyboard tap, line noise, or a small amount of outbound echo will constantly cut it off. If the detector waits too long for confirmation, the agent talks over the first words of the interruption.

Implementations commonly combine several safeguards:

• Require speech-like audio to persist across more than one frame.
• Use echo cancellation or compare inbound audio with the known outbound signal.
• Treat very short sounds differently from an utterance that yields stable transcript tokens.
• Adjust the decision when the caller says a clear stop phrase or begins a substantive request.
• Use a brief playback-suppression period around transitions, while preserving enough input to avoid clipping the caller.

After confirmed barge-in, the transcript must reflect what the caller actually heard. Suppose the planned sentence was "Your appointment is Tuesday at three, and I have sent a confirmation," but playback stopped after "Tuesday." Saving the full planned sentence as spoken history gives the next response false context. Better systems track playback progress and commit only the words or semantic content likely delivered before the stop.

Not every sound should seize the floor. A caller may say "okay," "right," or "mm-hm" to show attention without asking the speaker to stop. This is backchanneling: speech that supports the current turn rather than taking it. Distinguishing a backchannel from an interruption depends on duration, wording, timing, and context. A conservative system might continue after a brief acknowledgement but stop for a longer utterance. The trade-off is unavoidable because human speakers make the same inference from incomplete evidence.

Latency changes the apparent turn boundary

Turn-taking errors are often blamed on endpoint settings when the underlying problem is accumulated delay. Inbound audio crosses a network, may wait in a jitter buffer, is decoded and classified, and then reaches transcription. The response travels through another series of processing and buffering stages. By the time the caller hears the first response audio, they may have started speaking again because the line appeared idle.

This creates an important distinction between decision latency and experience latency. The endpoint may decide promptly relative to the audio it received, yet the whole system can still reply late. Conversely, an aggressive endpoint can make the system look fast in a lab while repeatedly cutting off real callers. A complete latency investigation follows the timestamps through the entire signal path, as described in the guide to latency in AI phone calls.

Network delay also complicates interruption. Caller speech detected now may have been spoken before the latest outbound words were heard. Systems should timestamp media in a consistent clock domain where possible and measure when audio entered, left, and was acknowledged by each stage. Without that timeline, it is difficult to tell whether a bad exchange came from endpointing, playback buffering, or the network.

Choosing policies for the conversation

There is no universally correct endpoint duration. The policy should follow the task and expected speaking style.

Short, constrained prompts such as "Say yes or no" can use tighter endpointing. Requests involving addresses, medication names, serial numbers, or open-ended explanations need more tolerance for internal pauses. A prompt that asks three questions at once encourages long, hesitant answers and makes endpointing harder; asking one clear question reduces ambiguity at the source.

Barge-in can also be selective. Legal notices, safety instructions, or a short confirmation may need different interruption rules from a long explanation. The application should define whether an interruption cancels the whole response, pauses it for possible resumption, or preserves a pending action while replacing only the spoken wording. An irreversible operation should not be canceled merely because its narration was interrupted.

The response itself affects the control loop. Long answers increase the chance of interruption. Dense wording gives the caller fewer natural openings. Short clauses, explicit questions, and occasional pauses make floor changes easier to interpret. Turn-taking is therefore partly a dialogue-design problem, not only a speech-engineering problem.

How to test turn-taking under realistic conditions

Single, clean microphone tests miss the failure modes that matter. A useful test set varies speaker pace, pause length, accent, line noise, network delay, and the point at which an interruption begins. Include callers who trail off, self-correct, speak over the first response word, give backchannels, and remain silent.

For each run, record an event timeline: inbound speech start and stop, partial and final transcript times, endpoint decision, response start, first outbound audio, interruption decision, cancellation, and final playback stop. Also preserve audio so a reviewer can compare machine events with what each side heard. Reusable scenarios and pass criteria are covered in how to test a voice agent.

Evaluate errors by type instead of averaging them into one latency number. Premature endpoints, delayed endpoints, false barge-ins, missed barge-ins, leaked post-cancel audio, and context mismatches have different causes and fixes. A policy that improves one may worsen another.

FAQ

Is endpointing the same as voice activity detection?

No. Voice activity detection estimates whether audio contains speech. Endpointing uses that estimate, plus timing and sometimes transcript meaning, to decide that a conversational turn is complete.

Why does a voice agent sometimes interrupt a caller after a pause?

Its endpoint policy likely interpreted the pause as trailing silence after a complete utterance. The fix may be a longer or context-sensitive endpoint, a clearer prompt, or removal of latency elsewhere in the path.

Why does the agent keep talking after I interrupt?

The system may have detected the interruption late, mistaken it for noise, or stopped generation without clearing buffered audio. Correct barge-in requires coordinated cancellation across generation, synthesis, playback queues, and the media connection.

Should every caller sound stop playback?

No. Immediate stopping is responsive but vulnerable to echo, coughs, noise, and backchannels. A practical policy confirms that the sound is likely to be a new caller turn, with thresholds chosen for the call's purpose.