Word error rate is not enough: scoring task success on voice

Word error rate is the most common metric in voice AI evaluation, and it is also one of the most misapplied. WER measures the difference between the reference transcript and the hypothesis transcript produced by your speech-to-text engine. A lower WER means the transcription was more accurate. That is a useful thing to know. It is not the same as knowing whether the call worked.

A perfect WER of zero means the speech-to-text engine produced an exact transcript of everything said on the call. The agent could still have failed to complete the booking, failed to understand the caller's intent, delivered the wrong information, or frustrated the caller into hanging up. The transcript is right. The call is wrong. WER gives you no signal on any of those outcomes.

What WER actually measures

WER is a transcription accuracy metric. You compute it by comparing a reference transcript (the ground truth, usually human-annotated) to the hypothesis transcript (the output of your ASR engine). The error rate is the number of insertions, deletions, and substitutions divided by the number of words in the reference.

This metric was developed to evaluate speech-to-text systems, not to evaluate the behavior of a conversational agent. When you use it to evaluate a voice agent, you are applying a transcription metric to a problem that is fundamentally about intent, completion, and experience. It answers the wrong question cleanly.

WER tells you whether the microphone and the model heard correctly. It does not tell you whether the call went correctly. Those are different measurements of different things.Nora Halvorsen

The gap between transcription and task success

Consider a caller who wants to reschedule an appointment. The ASR engine produces a perfect transcript of the conversation. Now ask: did the caller's appointment get rescheduled? That question requires evaluating whether the agent understood the intent, whether it asked for the right information, whether it confirmed the new time correctly, and whether the caller expressed satisfaction or frustration at the end. None of those dimensions are in the transcript accuracy score.

Task success evaluation asks: given the scenario defined for this call (reschedule an appointment), did the agent complete the required steps? It operates on the semantic meaning of the conversation, not on the phonetic accuracy of the transcription. A high task-success score means the call worked for the caller. A high WER score means the microphone and the model heard correctly.

Where they diverge most sharply

• Accent and dialect variation: WER often increases for non-standard accents. But a caller with a strong regional accent whose intent was clearly expressed may have a lower task success than the WER implies, because the agent failed to understand the intent, not because the ASR failed to transcribe it.
• High-WER but successful calls: a noisy line with multiple corrections and repetitions may have a high error rate but still complete the task. The transcript is messy. The caller got what they called for.
• Low-WER but failed calls: a clean transcript on a call where the agent gathered all the wrong information, confirmed a booking for the wrong date, or provided incorrect information. Zero errors in transcription. Complete failure in the task.
• Caller repair and repetition: when a caller repeats information, the transcript contains redundant words. This can inflate WER if the reference does not include the repetitions, while masking the fact that the repetition itself is a signal the agent mishandled the first exchange.

Building a task-success score

Task-success scoring starts with a definition of what success looks like for each scenario. A booking call succeeds when the caller's appointment is confirmed at the right time with the right details. A support call succeeds when the caller's issue is resolved or correctly escalated. A qualification call succeeds when the required information is gathered and the call ends with the appropriate next step.

From that definition, you build a rubric with scored dimensions. The dimensions typically include: intent recognition (did the agent understand what the caller wanted), information gathering (did the agent collect the required fields), task completion (was the goal achieved), caller experience (tone, latency, barge-in events, caller sentiment at close), and error recovery (how the agent handled misunderstandings, corrections, and unexpected inputs).

Using language models to score at scale

Human review produces the highest-quality task-success scores, but it does not scale. Scoring thousands of simulated calls requires automation. Language models are well suited to this because they can read a transcript, apply a rubric, and produce a structured score with an explanation. The key is to give the model the rubric explicitly, provide the expected outcome for the scenario, and ask for a score on each dimension separately rather than a single pass-fail judgment.

This approach has limits. Language models can miss failures that require audio-level signals: a correct transcript produced during a call where the agent's response latency exceeded the caller's patience threshold, for example. The best scoring pipelines combine LM-based rubric scoring on the transcript with signal-based scoring on the audio and timing data, so that the final score reflects both what was said and how the call actually unfolded.

A practical path forward

Stop treating WER as the primary signal for voice agent quality. Keep it as one signal for the ASR component in your stack, where it belongs. Add task-success scoring as the primary measure of whether your agent is doing its job.

Start with your two or three most important scenarios, write a rubric for each, and run scored evaluations on a batch of simulated calls. Compare the results to your WER numbers. You will almost certainly find calls where the signals disagree, and those disagreements are where the real quality work lives. A high-WER call that succeeded anyway tells you your agent is robust. A low-WER call that failed tells you the failure is in the logic, not the transcription, and that WER alone would never have found it.