One line of text beat upgrading our AI phone agent from Scribe v2 to v3

ThunderPhone's AI phone agents answer and make business calls using a mix of transcription models tuned for each agent, including ElevenLabs' Scribe v2 (the old model).

ElevenLabs released Scribe v3 (the new model) yesterday, and we tested swapping it in right away.

ElevenLabs reports 27% fewer word errors on tests of people reading aloud. We wanted to know whether that held up on real business calls.

The test

We used 451 caller turns from real customer calls. A person listened to each turn and wrote down what was said. We focused on difficult moments: spelled-out names, addresses, emails, phone numbers, and noisy or quiet audio, leaving out calls that might contain health information.

We called a transcript "right for the call" if our agent would understand the caller the same way as from the person's version. We tried audio alone, a list of key words in keyterms (accepted by both APIs), and sending the agent's last line as context.

ElevenLabs' during-the-call transcription API accepts previous_text with the first piece of audio. We sent: A phone call between an AI agent and a caller. The agent just said: "...", replacing the dots with the last thing the voice agent said.

During calls: the agent's last line beat upgrading

During calls, the models stayed within about 2 percentage points of each other in every setup. Neither had a clear edge.

Sending the agent's last line improved the old model's accuracy by 7.1 percentage points during calls. It helped the new model too, by less.

Recordings: better accuracy, more guesses

On recordings transcribed after the call, the new model reached 86.5% accuracy with key words.

The old model reached 82.7% in the same setup. Unlike during calls, the new model had a clear edge here.

A phone agent must transcribe while the caller waits, whereas full recordings are transcribed after the call finishes. The versions used during calls and on recordings are separate models, so a win on recordings doesn't mean a win during calls.

Price wasn't a factor: both versions cost $0.22 per hour during calls.

For recordings after the call, both cost $0.20 per hour.

On recordings with key words, the new model got 36 turns right that the old model missed.

But it also missed 19 turns the old model got right. Its gains mostly came from names, yes-or-no answers, everyday conversation, phone numbers, and emails. A common fix was a short answer that the old model turned into the wrong word or an invented phrase.

The catch showed up in garbled audio. We checked 10 clips where a person couldn't make out any words. During calls, neither model had a clear edge.

On recordings, though, the new model supplied words for 9 of those clips.

The old model did so for 5. A blank transcript lets the agent ask the caller to repeat instead of responding to words the model guessed.

Here's one of those clips, where the old model left the transcript blank and the new model wrote "Roll in 800.":

Where both models struggled

We grouped the 451 turns by what callers said; a turn could fit more than one group. Both models did best on yes-or-no answers, ordinary conversation, addresses, and general numbers and dates.

Emails, company names, and personal names were hardest for both models. During calls, the old model was slightly ahead on names, spelled-out letters, and phone numbers, though the gaps were too small to call.

Phone numbers and emails each had only 12 turns, so don't read much into those results.

The useful change was giving the model the last thing the voice agent said. We already had that text.

How we tested

• Sample: 451 caller turns from 302 real calls across 16 businesses (about 18% chosen at random, the rest leaning toward hard moments); calls that might contain health information were excluded.
• Setup: One run per setup in November 2026, during calls and after the call, testing keyterms and previous_text.
• Grading: A person listened to every turn and wrote down what was said; a transcript counts as right if the agent would understand the caller the same way.
• Garbled audio: 10 turns a person confirmed nobody could make out.