How text-to-speech works in real-time calls

Text-to-speech in a real-time call turns response text into playable audio by normalizing written forms, resolving pronunciations, assigning rhythm and intonation, generating a speech waveform, and streaming that waveform through the call's codec and media transport. The system must begin before the full response is known, yet avoid speaking text that may change; that makes clause boundaries, pronunciation context, buffering, and cancellation as important as the voice generator itself.

The input is text, but not yet speech

Written language leaves many decisions unstated. Dr. can mean “doctor” or “drive.” 1/4 can be a date, a fraction, or part of an identifier. A written currency amount needs words, ordering, and a currency reading. A URL, acronym, phone number, and street address each require different grouping. Humans infer the intended spoken form from context; a synthesis pipeline must make those decisions explicitly.

The first stage is text normalization. It classifies spans such as numbers, dates, measurements, abbreviations, symbols, and markup, then expands them into a speakable representation. Normalization is language- and locale-dependent. 05/06 does not identify the same month and day under every convention, and decimal separators vary. Business applications should pass structured values with known types where possible rather than hope a synthesizer infers intent from an ambiguous display string.

Normalization also defines what should not be spoken. Formatting characters, Markdown, hidden control text, and raw tool output must be removed or converted deliberately. A dialogue system that sends an unfiltered JSON object to synthesis has failed before acoustic generation begins.

Pronunciation maps symbols to sound units

After normalization, the system determines how the expanded text should be pronounced. A grapheme-to-phoneme process maps letters and context to a sequence of phonemes, the contrastive sound units of a language. The mapping is not one-to-one. English letter sequences change sound across words, and heteronyms such as “lead” depend on syntax and meaning.

A lexicon supplies known pronunciations; predictive rules or learned sequence models handle unseen words. Language identification may select a different phoneme inventory and pronunciation model. Names, product terms, and abbreviations remain hard because conventional spelling often provides too little evidence. A pronunciation dictionary can map a known term to phonemes or a respelling, but the entry must match the active language and phonetic alphabet expected by the engine.

Pronunciation is more than individual words. Connected speech changes sounds at boundaries, reduces unstressed syllables, and chooses where to place emphasis. A system that pronounces each token independently may be intelligible but sound disjointed. The linguistic front end therefore represents phrase structure and context along with the phoneme sequence.

Prosody makes the sequence a sentence

Prosody covers timing, stress, rhythm, phrasing, and pitch movement. It distinguishes a statement from a question, separates list items, highlights contrast, and signals whether a thought is complete. The words “you booked Tuesday” can sound like confirmation, surprise, or a question depending on their contour.

Punctuation provides useful but incomplete cues. A comma may indicate a short phrase break; a period often indicates a stronger one. Generated conversational text may contain punctuation chosen for reading rather than speaking. Long parenthetical clauses, bullet syntax, or repeated dashes can produce unnatural pauses. A real-time dialogue layer should produce speech-ready sentences, not merely pass visual prose downstream.

Modern neural text-to-speech systems learn prosodic patterns from speech data and conditioning signals. Internally, an acoustic model converts linguistic units into a representation of how the utterance should sound over time. That representation may be a mel spectrogram or another learned sequence. Duration prediction decides how long sounds last; pitch and energy prediction shape emphasis and intonation. Some architectures generate acoustic frames sequentially, while others predict many positions in parallel.

The model can infer ordinary phrasing, but it does not know the application's intention unless the text or controls express it. If a confirmation code must be read digit by digit, that is a content requirement, not a voice-style preference. The application should encode it deliberately.

From acoustic representation to waveform

An acoustic representation is not yet audio. A vocoder converts it into waveform samples whose pressure variations can drive a speaker. Earlier vocoders used explicit source-filter and signal-processing assumptions. Neural vocoders learn the mapping from acoustic features to waveforms, often producing audio in blocks that can be streamed as soon as enough conditioning context exists.

Some synthesis architectures generate waveform-like units directly rather than exposing a conventional spectrogram between stages. The boundary may differ, but the computational problem remains: produce a temporally precise sample sequence consistent with linguistic content and voice characteristics.

Waveform generation has a latency-throughput trade-off. Larger batches and more context can improve computational efficiency or continuity, but the first block arrives later. Very small chunks begin quickly but increase scheduling overhead and can reveal seams if state is not carried across them. Real-time systems preserve model state, overlap adjacent blocks, or crossfade carefully so chunk boundaries do not click.

The generated waveform usually has to be resampled and encoded for the call. A synthesis engine may produce wideband audio while a telephone leg accepts only G.711 narrowband. Low-pass filtering and downsampling then remove frequencies the route cannot carry. Sending pristine full-band synthesis into a narrowband gateway does not make the far end wideband; it makes the conversion boundary responsible for preserving intelligibility without aliasing.

Streaming starts before the whole response exists

In an offline workflow, the system can synthesize a complete paragraph with full right-hand context. A live call cannot always wait. The response generator emits text incrementally, and the synthesis controller must decide when a prefix is stable and speakable.

Sending every token immediately is unsafe. The sequence Your appointment is at 1 is ambiguous until more context reveals 1:30, 15 Main Street, or an item number. The generator may also revise formatting or complete a unit after several tokens. Once audio has played, it cannot be edited in the caller's past.

Waiting for an entire multi-sentence answer is also costly. A common design buffers until a strong clause or sentence boundary, normalizes that unit, and sends it to synthesis while later text is still being generated. This creates a pipeline:

1. The dialogue layer produces a stable, speech-ready clause.
2. Normalization and pronunciation resolve the clause.
3. Synthesis begins generating waveform blocks.
4. The first blocks are converted to the call format and queued.
5. Later clauses follow while the first is playing.

The interval from submission to the first usable samples is time to first audio. Caller-perceived response time also includes how long the controller waited to form the clause, output prebuffering, packet transport, and the receiver's jitter buffer. Call latency must be measured across that complete path.

Chunking needs linguistic awareness. Splitting after “Dr.” because it contains a period can create a false sentence boundary. Splitting a number from its unit can yield the wrong expansion. A controller should recognize abbreviations, balanced punctuation, numeric spans, and incomplete tool results before declaring text speakable.

Media transport must remain continuous

Once audio begins, the call expects a steady stream. RTP timestamps should advance according to the media clock even if synthesis produced the samples in irregular bursts. The sender packetizes frames at the negotiated cadence and maintains sequence continuity. If generation falls behind playback, the queue underruns and the caller hears silence or a discontinuity.

Prebuffering several audio blocks absorbs small production stalls, but it adds start delay. Deep queues also make interruption sluggish because already-buffered speech must be discarded. The controller should expose queue depth, synthesis production rate, and underrun events. Without those signals, a pause caused by media starvation may be misdiagnosed as slow response generation.

Bridged calls can add another format boundary. The synthesis side may send one codec to a media server, while the phone leg uses another. Mixing, recording, and recognition may require PCM internally. Each conversion should have an explicit sample rate, channel count, and timing model; silent assumptions cause doubled speed, halved speed, drift, or periodic gaps.

Barge-in is a cancellation problem

When the caller interrupts, stopping the text generator is not sufficient. Text may be waiting in the clause buffer, synthesis requests may be running, decoded PCM may be queued, encoded packets may be in a media server, and packets already sent may be waiting in the far endpoint's jitter buffer. Responsive barge-in requires a cancellation signal that crosses every stage.

The system should associate generated text and audio with a turn identifier. When that turn is canceled, downstream components reject late results instead of appending them to the next response. Audio queues are flushed, RTP timing is resumed cleanly for future speech, and conversational state records what the caller actually heard rather than the complete text that was intended.

That last distinction affects the next turn. If the caller interrupted after hearing “Your appointment is Tuesday at,” the dialogue state should not assume the time was delivered. Tracking playback progress by character or word is approximate because phoneme durations do not align uniformly with text. Time-aligned synthesis metadata, when available, gives a better estimate, but applications still need conservative logic for consequential information.

Controlling pronunciation and delivery

SSML can express pauses, emphasis, say-as behavior, pronunciation, and other synthesis instructions. Support varies by implementation, so an application should use a tested subset and escape user-provided text. Treating arbitrary content as markup can cause parse failures or let input change how unrelated text is spoken.

Controls should solve repeatable communication requirements. Useful examples include reading a phone number digit by digit, inserting a short pause before a confirmation code, or disambiguating a brand name through a dictionary. Excessive manual prosody markup becomes brittle when the sentence changes and can create inconsistent output across languages.

Pronunciation caches can reduce repeat work for stable phrases, but cache keys must include every setting that changes audio: voice, language, speaking style, sample rate, and text normalization version. Caching dynamic statements is risky when they contain personal or time-sensitive data, and any retained audio should follow the same access and retention discipline as other call data.

Failure modes and how to diagnose them

TTS defects are easier to isolate when the text, normalized form, acoustic output, and phone-leg recording can be inspected separately:

• Wrong words spoken: inspect text generation and normalization before the acoustic model.
• Name or acronym mispronounced: add or correct language-specific dictionary entries and negative tests.
• Odd pauses or emphasis: inspect punctuation, chunk boundaries, and prosody controls.
• Muffled but otherwise correct audio: confirm the negotiated phone codec and every resample step.
• Clicks between chunks: verify waveform continuity, model state, overlap, and packet timestamps.
• Slow start: separate clause-buffer wait, synthesis first-audio time, conversion, prebuffering, and network delay.
• Speech continues after interruption: trace cancellation through text, synthesis, PCM, packet, and gateway queues.

Do not evaluate only a clean downloaded waveform. The caller hears the result after encoding, network transport, jitter buffering, and handset playback. Test the completed call path, including the routes and devices that matter to the application.

Evaluating real-time speech output

Naturalness is only one dimension. A call-oriented evaluation should cover intelligibility over the actual phone route, correctness of names and structured values, appropriate phrasing, consistency across chunks, start latency, underrun frequency, and stop latency during interruptions. Task-critical prompts deserve exact expected-output tests even when open-ended replies are judged more broadly.

Use a fixed sentence set with dates, currencies, abbreviations, alphanumeric identifiers, addresses, questions, lists, and domain terms. Add generated conversations to test context-dependent chunking. Listen to both the source waveform and a captured round trip. A regression that is invisible in full-band audio may become obvious after narrowband conversion.

The best real-time synthesis pipeline is not simply the one with the most expressive isolated voice. It is the one that says the intended content correctly, begins at the right conversational moment, stays continuous over the media path, and stops when the caller takes the floor.

FAQ

Why can the same text be pronounced differently?

Language, surrounding words, normalization rules, dictionary entries, and prosody settings all affect the spoken result. Voice generation may also be nondeterministic unless the system fixes its sampling configuration.

Can text-to-speech stream one word at a time?

It can, but isolated words lack right-hand context and create unstable normalization, pronunciation, and rhythm. Clause-sized chunks usually provide a better balance between fast start and coherent speech.

Does higher-fidelity source audio guarantee better phone audio?

No. The narrowest codec and device on the route limit what reaches the caller. Correct filtering and conversion are still important, but removed frequency content cannot survive a narrowband leg.

What should happen when a caller interrupts?

The system should stop playback promptly, cancel or ignore outstanding synthesis, clear queued audio, preserve clean media timing, and update conversation state based on what was actually heard.