How to compare voice AI performance — and what ThunderPhone publishes

If you are choosing a phone-agent platform, "which one performs better?" is a reasonable question with an inconvenient answer: public, apples-to-apples performance benchmarks for voice AI platforms barely exist. Most numbers you will encounter are self-reported by vendors, measured under undisclosed conditions, and unverifiable from the outside. This page explains why the comparison is hard, states the one performance number ThunderPhone publishes and how to audit it, notes what competitors publish, and describes a testing process a buyer can actually run.

Why performance comparison is hard

"Performance" for a phone agent is not one number. At minimum it spans three different axes:

• Latency — how quickly the agent starts speaking after the caller stops. This shapes whether a call feels like a conversation or an IVR.
• Comprehension — whether the agent correctly understood what the caller said, across accents, background noise, phone-line audio quality, and languages.
• Instruction-following — whether the agent does what its prompt and tools tell it to do: asks the required questions, calls the right tool, escalates when it should, and stays within policy.

A platform can be fast and inaccurate, or accurate and slow, so a single headline figure cannot rank platforms on all three. Worse, the figures that do get published lack shared methodology. When a vendor reports a latency number, the public page typically does not state what was measured (first token of audio, or the full first sentence?), from where (server-side, or as heard on a real phone leg?), under what load, with which configuration, or at which percentile. Two vendors' numbers measured differently are not comparable, and neither is reproducible by a buyer reading the page. Accuracy claims add one more problem: they depend heavily on the test scenarios chosen, which vendors rarely publish.

The honest baseline for this page: the research behind it found no neutral, shared benchmark that ranks phone-agent platforms end to end, and any comparison built from mismatched self-reported figures would be misleading.

What ThunderPhone publishes

ThunderPhone publishes one performance number: its Storm engine scores 99.4% on Big Bench Audio, an open audio-reasoning benchmark. Big Bench Audio poses reasoning problems as spoken audio, so a system is graded on whether it understood the spoken input and produced the correct answer — a comprehension-and-reasoning measure, not a latency measure.

What distinguishes this figure is that the evaluation dataset is public, on Hugging Face at kolchinski/bba-storm-eval. A buyer — or a competitor — can inspect the benchmark items, responses, and grading rather than taking the percentage on faith. Publishing the eval set does not make the number universal (it is one benchmark, on one axis, for one engine), but it makes the claim auditable — a property none of the competitor figures reviewed for this page carries.

ThunderPhone does not publish a platform latency figure, and this page will not invent one. Latency depends on engine, voice, language, and telephony path, and the right way to assess it is to measure it on your own calls, as described below.

What competitors publish

From the competitor pages reviewed for this site's research file:

• Retell AI states a latency figure of roughly ~600ms on its homepage ([retellai.com](https://www.retellai. This is Retell's own self-reported claim; the public page does not include a methodology that would let an outside party reproduce the measurement — which is typical for the category, not a criticism specific to Retell.
• Vapi and Bland AI: no comparable published performance figure — no benchmark score and no latency measurement — was found on the pages reviewed (vapi.ai and bland.ai, accessed 2026-08-18). That is a statement about what those pages show, not a claim about how those platforms perform.

None of the reviewed vendors published a result on a shared, open benchmark with an inspectable evaluation set, so no cross-platform score comparison can honestly be drawn from public pages alone.

How a buyer should actually evaluate performance

Since public numbers cannot decide this for you, the workable approach is to run the same test on every platform you are considering:

1. Script identical scenarios. Write down your five to ten most important call flows — caller goals, objections, interruptions, edge cases — and run the same scripts against each platform's trial agent. ThunderPhone is built for exactly this workflow: AI-caller simulations, bot-to-bot and SIP-loopback test calls, reusable graded scenarios, and regression suites with minimum pass-rate gates let a scripted scenario be replayed and graded rather than judged by ear. Simulations are billable real calls; the interface shows the charge before a run.
2. Measure what matters on your calls. Time first-response latency yourself on real phone legs — a stopwatch on ten test calls per platform tells you more than any homepage figure — and record task completion: did the agent finish the job the call was for?
3. Test your languages and audio conditions. Demo calls are usually clean-audio English. If your callers speak other languages, call from noisy environments, or arrive through specific carriers or SIP trunks, test those conditions specifically; platform differences can be largest exactly there.
4. Re-test on every change. Model updates, prompt edits, and voice changes all move performance. Regression suites with pass-rate gates catch a degradation before your callers do, which matters more over the life of a deployment than the launch-day measurement.

The result of this process is a performance comparison that is actually valid for your use case — something no vendor page, this one included, can hand you.

FAQ

Is there a standard benchmark for comparing voice AI phone platforms?

None was found in the research behind this page. Open benchmarks such as Big Bench Audio measure audio comprehension and reasoning for underlying engines, but no widely adopted neutral benchmark ranking full phone-agent platforms on latency, comprehension, and instruction-following together was identified.

What does ThunderPhone's 99.4% on Big Bench Audio mean?

It means the Storm engine answered 99.4% of Big Bench Audio's spoken reasoning problems correctly, and the run is public. The evaluation set is published at kolchinski/bba-storm-eval on Hugging Face so the claim can be audited. It measures comprehension and reasoning, not latency or your specific call flows.

How fast is ThunderPhone compared to Retell, Vapi, or Bland?

No honest cross-platform answer is available from public sources. Retell's homepage states a ~600ms latency figure ([retellai.com](https://www.retellai. The reliable comparison is to time first-response latency yourself on test calls to each platform.

Why don't vendors publish comparable performance numbers?

Because there is no shared methodology and configurations vary enormously. Latency and accuracy depend on model, voice, language, telephony path, and load, so a single honest platform-wide number is hard to produce — and a self-reported one measured under favorable conditions is easy to produce. Treat any figure without a published, reproducible method as marketing input, not evidence.

How can I test performance before committing to a platform?

Run identical scripted scenarios on each candidate and grade the results. On ThunderPhone, use AI-caller simulations, bot-to-bot or SIP-loopback test calls, and reusable graded scenarios to replay the same flows repeatedly; simulations are billable real calls, with the charge shown before a run. Measure first-response latency and task completion on your own flows, languages, and audio conditions.

Does a high benchmark score guarantee good performance on my calls?

No. A benchmark measures comprehension and reasoning under the benchmark's conditions; your calls add your vocabulary, your callers' accents and audio quality, your tools, and your latency constraints. A strong benchmark score is evidence about the engine — an auditable one more so — but the deciding test is a graded run of your own scenarios.