Where the milliseconds go: latency in AI phone calls

Latency in an AI phone call is not one model's runtime; it is the accumulated delay from audio capture, packetization, network transport, jitter buffering, speech recognition, turn detection, response generation, speech synthesis, and return playback. The delay a caller feels also depends on what is being measured: mouth-to-ear latency describes audio transport, while response latency runs from the caller finishing a turn to hearing the first audio of the reply. Finding the real bottleneck requires timestamps at every boundary, not one end-to-end stopwatch.

Two timelines share the same call

A live conversation contains at least two latency paths. The media path continuously moves sound from one participant to the other. Its useful measure is mouth-to-ear delay: how long a sound at the sender takes to emerge at the receiver. The response path begins when one participant has said enough for the system to act and ends when the reply becomes audible. It includes media transport in both directions plus recognition and decision-making.

These paths overlap. A streaming recognizer can process the beginning of an utterance while later words are still arriving. A response generator can produce an opening clause before it has completed the whole answer. A speech synthesizer can emit audio for that clause while more text is being produced. This concurrency is why summing the full runtime of every component overstates the caller's wait. The critical path is the chain of work that must finish before the first playable reply frame exists.

The inverse is also true: a fast component does not guarantee a fast conversation. Waiting to decide that the caller has stopped can dominate the gap even when recognition and synthesis are streaming. Latency in voice AI is therefore a property of the complete turn loop.

Inbound audio: capture, frames, and the network

The first delay appears before any intelligence runs. A device samples microphone audio and usually collects a small frame before encoding it. A codec configured for 20-millisecond frames cannot send the complete frame until those 20 milliseconds of sound exist. Some codecs also need look-ahead or multiple frames to encode efficiently. Smaller frames reduce this wait but increase packet rate and relative header overhead.

The encoded frame is placed in RTP or another real-time transport. It then crosses access networks, routers, firewalls, media relays, and possibly a telephone gateway. Physical distance creates an irreducible propagation floor; routing and serialization add more. Queueing creates the variable part. A packet that waits behind bulk traffic can arrive substantially later than its neighbors even when the route itself has not changed.

The receiver cannot play packets at their irregular arrival times. It uses a jitter buffer to reorder packets and schedule a steady output. The buffer deliberately adds delay so that modestly late packets can still be used. If it waits too little, packets become late loss even though they eventually arrive. If it waits too long, audio is smooth but conversation feels sluggish. Adaptive buffers estimate recent jitter and move the playback point, but abrupt route changes can still force a choice between a glitch and more delay.

Decoding, decrypting, resampling, noise processing, and echo cancellation also consume time. Each may be small in isolation, yet hidden buffers between them can be larger than the computation. Audio frameworks often read and write fixed-size blocks; a slow consumer can allow a queue to grow by several blocks without reporting an error.

Recognition is streaming, but commitment takes time

Streaming speech recognition does not wait for a complete recording. It converts incoming frames into acoustic features, updates token hypotheses, and emits partial text. Early words can be available while the caller is still speaking. This lets a dialogue system begin lightweight work such as entity extraction or routing before the turn ends. The full mechanism is covered in how speech recognition works on calls.

Partial text is provisional. A recognizer may revise recent words when later sounds provide more context. Acting on an unstable hypothesis can call the wrong tool or answer a sentence the caller has not finished. Waiting for a final transcript is safer but moves all downstream work later. Systems usually combine stability signals, intent-specific rules, and cancellation so safe preparation can happen early without committing irreversible actions.

Recognition runtime is only part of this stage. Audio has to arrive, pass through its buffers, and be scheduled for processing. Under load, batching may improve total throughput while making an individual stream wait. Averages can hide that queueing: one overloaded worker can create a long tail even if most turns are quick.

Endpointing is a decision under uncertainty

The system does not observe a clean “turn ended” event in the waveform. It infers one. A voice activity detector estimates whether each audio region contains speech; endpointing decides when accumulated evidence is strong enough to close the turn. They are related, but silence is not always an endpoint.

Callers pause between clauses, search for a word, spell identifiers, or wait while reading. Background noise can look speech-like, while quiet consonants can look like silence. An endpoint policy that closes quickly improves apparent speed but cuts off hesitant speakers. A conservative policy preserves the utterance but creates a dead-air gap after ordinary turns.

This is one of the most consequential latency trade-offs because it is intentional waiting. Better policies use more than a fixed silence timer: recognition stability, syntax, prosody, whether the utterance appears complete, and conversation state can all inform the decision. A question expecting a one-word confirmation can close differently from an address-collection step.

Response generation: first useful clause beats full completion

Once the turn is actionable, the dialogue layer assembles context, chooses what to do, and produces a response. Pure conversational turns may begin generating text immediately. Tool-backed turns must sometimes wait for a database, calendar, or business API. Network handshakes, retries, connection-pool waits, and slow upstream queries then become part of the caller's perceived latency.

The first generated token is not automatically ready for speech. A synthesizer needs enough text to normalize ambiguous forms and choose a stable pronunciation and rhythm. Sending one token at a time can cause unnatural breaks or force the system to revise text whose audio has already played. Waiting for the entire answer makes phrasing easier but adds avoidable delay. Most real-time designs use clause-aware chunking: accumulate a speakable unit, dispatch it, and continue producing later units in parallel.

An acknowledgement can mask a slow tool call, but it should carry conversational value rather than exist solely as filler. “I'll check that date” establishes what is happening; a generic delay phrase repeated every turn makes the system feel slower, not faster. The real fix is still to reduce tool and queue latency or redesign the interaction so independent work overlaps.

Synthesis and outbound playback

Text-to-speech has its own time to first audio: the interval from submitting speakable text until the first usable audio samples are returned. Text normalization, pronunciation, acoustic generation, waveform generation, and streaming serialization all contribute. How text-to-speech works explains those stages in detail.

The first returned bytes may not yet be audible. The application may buffer enough audio to survive small production stalls, resample it to the call's format, encode it, and packetize it. The far end then applies its own network transport and jitter buffer. Measuring only synthesis service time excludes this tail.

Buffering protects continuity. Without it, a brief synthesis stall causes underrun: playback runs out of samples and inserts silence or a discontinuity. With too much prebuffer, every response starts late and cancellation wastes more generated audio. The useful operating point depends on how consistently audio is produced and how variable the outbound network is.

Interruptions create a second latency budget

Natural phone conversations are not alternating recordings. A caller may say “wait” while the response is playing. Supporting barge-in requires the system to detect new near-end speech despite its own outbound audio, stop playback, cancel work that no longer matters, and reopen recognition context.

Several delays become visible here:

• Echo cancellation must keep synthesized audio from being mistaken for the caller.
• Speech detection must distinguish a real interruption from a cough, line noise, or short acknowledgement.
• Queued audio must be discarded at every layer, not just at the synthesizer.
• Already-sent packets may remain in a gateway or endpoint buffer and play after cancellation.

A system can generate responses quickly yet feel unresponsive if it takes too long to stop speaking. Interruption latency should be measured independently from response-start latency. The design tension is similar to endpointing: trigger too eagerly and normal noises cancel speech; wait too long and the caller has to talk over the system.

How to instrument the critical path

End-to-end recordings reveal symptoms, but component timestamps reveal causes. For each turn, capture a monotonic timestamp for at least these events:

1. First and last inbound speech frames received.
2. First partial transcript and final or committed transcript.
3. Endpoint decision.
4. Response generation requested and first speakable text available.
5. Tool request and response, when applicable.
6. Synthesis requested and first audio frame received.
7. First outbound audio frame queued and sent.
8. Playback cancellation requested and completed for interruptions.

Use one clock where possible. Wall clocks on separate machines can be skewed or corrected mid-call; monotonic clocks are safer for durations. When distributed tracing crosses hosts, synchronize clocks and record enough causal identifiers to reconstruct the sequence. RTP timestamps describe media time, not absolute wall time, so they must be mapped carefully.

Report distributions rather than one average. Median shows the ordinary turn; upper percentiles expose cold connections, slow tools, overloaded workers, and network route changes. Split results by call route, codec, region, turn type, and whether a tool ran. A single combined number can make an infrastructure problem look like a recognition problem.

Also track queue depth and buffer occupancy. If computation takes little time but start timestamps trail request timestamps, the delay is scheduling or admission control. If the outbound queue remains full after cancellation, the issue is buffering. If endpointing is late only on noisy calls, the policy or audio conditioning is the likely boundary.

Reducing latency without breaking conversation quality

The largest safe improvement is usually eliminating unnecessary serial work. Stream audio and hypotheses; establish reusable connections; start reversible preparation from stable partials; run independent lookups concurrently; send complete clauses to synthesis as they become available; and avoid decode, resample, or relay hops that add no function.

Then tune intentional buffers with failure data. Shrinking every buffer to zero produces gaps, false endpoints, and packet loss. A responsive system needs enough margin for variance, plus backpressure so queues cannot grow without bound. Set explicit deadlines for external tools and define a useful fallback rather than leaving the caller in silence.

Finally, optimize the whole loop. Saving time in generation does little if endpointing waits longer, and reducing first-audio time does little if a gateway prebuffers the result. The caller experiences the critical path and the worst interruptions, not the speed of the component with the best dashboard.

FAQ

Is network latency the main cause of slow AI calls?

Not necessarily. Transport is only one segment. Endpoint waiting, tool calls, processing queues, synthesis startup, and media buffers can each dominate a turn. Instrument the boundaries before choosing what to optimize.

Why not use the shortest possible silence threshold?

Because pauses occur inside valid turns. A very short threshold answers quickly but can split a sentence or capture an incomplete address. Endpointing must trade response speed against interruption risk.

Does streaming remove response latency?

Streaming overlaps stages; it does not make them free. The system still needs enough stable input to act and enough stable output to speak, and its first audio still crosses buffers and a network.

What latency number should a team monitor?

Monitor several: mouth-to-ear delay, end-of-speech to first audible response, endpoint decision time, synthesis first-audio time, tool time, and interruption-stop time. Their distributions explain more than one blended average.