BenPicks
Search BenPicks

Tutorial · Evidence checked 2026-08-29

How to Benchmark a Text-to-Speech API Without Inventing a Winner

A practical, evidence-led decision guide. Product capabilities and limits are separated from anything that would require hands-on testing.

Most text-to-speech benchmarks fail before the first request: the script is too easy, the models are not recorded, latency is measured from different points and listeners grade “naturalness” without agreeing what a defect sounds like. A useful benchmark is smaller and stricter. It tells you whether one named voice/model can deliver one real workload, how repairs behave and what the accepted audio costs.

Reproducible evaluation workflow with inputs, checks and decision record

Build a script that can expose failure

Use 1,200–2,000 characters drawn from the work you will actually publish. A useful fixture includes:

Add the expected reading next to every risky token before synthesis. If multilingual output matters, use native-language reviewers and separate fixtures; translated text and language quality should not be collapsed into one score.

Keep the script within the strictest candidate request limit. Deepgram Aura documents a 2,000-character request ceiling, while other services have different limits. If production text is longer, create a second chunking test rather than quietly giving one API an invalid request.

Record the identity of every run

For each output, retain provider, endpoint, model, voice, region, date, audio format, sample rate, speed, SSML/controls and request ID where exposed. “Google TTS” or “Cartesia” is not enough: model generations and supported controls can differ. Save the raw response before editing, normalising or mastering it.

Use the same input and equivalent output settings. When equivalence is impossible, document the difference. A benchmark remains useful with different native formats; it becomes misleading only when the difference is hidden.

Measure latency from the user's point of view

Capture at least four timestamps:

  1. request sent;
  2. first playable audio received;
  3. final audio byte received;
  4. playback or file ready in the application.

For conversational products, time to first playable audio and p95 matter more than the average completion time. Run a warm-up separately, then at least 30 measured turns at single concurrency and at expected peak concurrency. Log errors, retries and audio received after cancellation. For batch narration, total completion time and throughput may matter more.

Listen for named defects, not “AI-ness”

Blind the files where practical. Ask two listeners to mark observable events: wrong phoneme in a name, number read incorrectly, missing word, clipped ending, pause outside an allowed range, unstable loudness or an emotion/control instruction not followed. Keep preference—“I like voice B”—in a separate column.

A critical defect should not disappear inside an average. If commercial audio must say a medicine, legal term or customer name correctly, one unrepaired failure can reject the configuration even when every other score is high.

Make the API repair its own mistake

The correction round is where products separate. Fix the proper name using the service's supported route: lexicon, SSML, IPA control, prompt/instruction or text rewrite. Record human actions, regenerated characters, elapsed time and whether the fix changes adjacent speech.

Then change a number after approval. Measure whether the workflow can regenerate a segment or must recreate the whole asset. Run the corrected request ten times to detect whether the repair is stable. The accepted-output cost includes every failed and corrected request, not only the final file.

Calculate cost per accepted minute

Keep provider-native units first: characters, tokens, credits or included plan allowance. Then use the observed fixture to calculate:

`(synthesis charges + retries + correction charges + reviewer labour + integration overhead) ÷ accepted finished minutes`

For example, if a 100,000-character batch needs 20,000 characters of regeneration, price 120,000 submitted characters. Do not describe that as “20% more expensive” unless subscription, allowance, model rate and overage have also been normalised. It is first a 20% consumption increase.

At the published Polly rates, the same one-million-character workload can be $4, $16, $30 or $100 depending on engine. That spread is why every result must carry its model or engine. For Cartesia, translate the selected plan's credits and model consumption into the actual accepted minutes. For Google Cloud and Deepgram, attach the current selected-model rate rather than a generic brand price.

Use a result sheet that can survive review

MeasureCandidate ACandidate B
Exact model / voice / region
First-audio p50 / p95
Completion p50 / p95
Errors and retries
Critical pronunciation defects
Stable repairs / attempts
Submitted units, including rework
Accepted finished minutes
Cost per accepted minute
Rights/data/security gatepass / fail / openpass / fail / open

The decision should name a configuration, not merely a vendor. A service can pass for short agent turns and fail for long narration, or pass in one language and remain unknown in another.

Stop the benchmark when the evidence is broken

Call the run inconclusive if the model or voice is missing, fixtures changed, audio was manually repaired before preservation, listeners heard different files, latency clocks started at different points or billed units cannot be reconciled. Repeat after fixing the protocol. A clean rerun is cheaper than a procurement decision built on incomparable outputs.

What a defensible recommendation sounds like

Avoid “Voice A sounds the most human.” Prefer: “Model A passed every required name and number, produced first audio at p95 within our threshold, and needed 8% regeneration; Model B missed the proper-name gate in three of ten repaired runs.” That statement is narrower, but a buyer can act on it and reproduce it.

Choose the first two configurations to test

Start with the architecture, not the brand list. For batch narration inside AWS, use the Amazon Polly buying analysis to select the exact engine and keep Google Cloud Text-to-Speech as the cloud-native contrast. For low-latency agents, compare the documented request and control boundaries in Deepgram Aura and Cartesia Sonic before you send the first fixture.

Do not add a third candidate because its demo sounds impressive. Add one only when it represents a missing deployment, language or control model. Then run the same retained script, clocks and correction round. The next best step is to benchmark two named configurations this week and take the one that passes every critical gate into a production-shaped pilot.

Official sources used for the example boundaries