Testing methodology
Every call, measured before it ships
Vexa runs your voice agent through a library of 182 scenario types across 47 voices and accents, scores every turn in real time, detects regressions across releases, and gates your CI pipeline. Here is exactly how.
182 ways to break a voice agent before a caller does
The library covers five categories. Together they approximate the distribution of conditions a real production agent will encounter. You can run all of them or select the subset that matches your use case.
47
Voices and accents
Calibrated voice-accent pairs drawn from real call-center populations, sampled to match the distribution of your user base.
18
Background noise
Real noise profiles at controlled SNR levels, from 5 dB (nearly unintelligible) to 25 dB (light background). Injected into the audio path before the agent's STT.
31
Behavioral patterns
Caller behaviors that expose agent fragility: fast speech, long pauses, barge-in sequences, trailing off, overlapping speech and topic pivots.
64
Call scenarios
End-to-end call scripts spanning the most common voice-agent use cases. Each script includes a goal, caller constraints, and a success criterion.
22
Edge cases
Unusual but real conditions that regularly trip production agents: code-switching, silence, IVR handoff, low-bandwidth audio and voicemail detection.
You can also define custom scenarios in YAML and contribute them to your workspace library. See the scenario authoring guide in the docs.
Inside a simulated call
Every call in a Vexa suite follows the same five-step sequence. The whole thing is deterministic, reproducible from the scenario seed, and runs in real time over a live audio path.
Load the scenario
Vexa reads the scenario definition: caller persona, accent region, speech rate, emotional state, call goal, behavioral flags and edge case injections. Each scenario is a deterministic seed, so reruns are reproducible.
Dial the agent
Vexa connects to your SIP endpoint or WebSocket URI as if it were a real caller. The audio path is identical to what your real callers use: the same codec, the same bitrate, the same regional latency profile.
Drive the conversation
The caller model generates utterances turn by turn, following the persona and pursuing the goal. It does not try to help. It asks questions, changes its mind, barge-ins at scripted points and occasionally says the wrong thing.
Score every turn
As each agent response arrives, a separate judge model scores task success and tone. Latency is measured from audio timestamps, not from model output. WER is estimated by comparing your STT transcript to what the caller said.
Aggregate and compare
Turn scores roll up to a call score, call scores to a suite result. The result is compared to your baseline, any regressions are flagged, and the failing turns are grouped by scenario and voice.
Quiero reservar una mesa para dos, el sábado a las siete.
es-US-bilingual · speech rate 184 WPM
Con mucho gusto. ¿Me puede confirmar la fecha exacta, este sábado el 26?
latency 418 ms · tone 91
Sí, el 26... ¿hay, espera, ¿hay mesas junto a la ventana?
barge-in injected at 1.2 s
Disculpe, no le escuché bien. ¿Puede repetir, por favor?
latency 730 ms · task success 44% · recovery failed
What the judge looks at on each turn
Every turn is scored by a language model judge that receives the full call context. The rubric is the same across all scenarios, voices and accents.
The judge evaluates whether the agent made meaningful progress toward the caller's stated goal on this turn. A turn where the agent confirms a booking detail scores high. A turn where the agent asks the caller to repeat scores low.
Suite-level threshold
The suite passes if the mean task success across all turns and all calls in the suite meets or exceeds the configured threshold.
Default threshold: Configurable minimum, default 85
The judge scores professionalism (no dead air fillers, no clipped responses), warmth (natural phrasing, appropriate empathy) and adherence to persona guidelines if a persona YAML was provided for the agent.
Persona override
If you supply a persona YAML for your agent (e.g. friendly concierge), the judge uses it to calibrate the tone rubric rather than the generic baseline.
Default threshold: Informational, not threshold-gated by default
Measured from the last audio byte of the caller's utterance to the first audio byte of the agent's response. Reported as p50 and p95 across the suite. Vexa's own overhead is documented per region and subtracted from reported values.
Calibration data
Vexa provides per-region calibration files that quantify the SIP/WebRTC overhead so your reported latency reflects the agent, not Vexa's infrastructure.
Default threshold: p95 default maximum 900 ms
Estimated by comparing what the simulated caller said (the exact words from the caller model) against what your speech-to-text layer transcribed. Vexa does not have access to your STT internals; it receives only the transcript your system produces.
Estimation limit
WER estimation is most reliable when caller speech is clean. In high-noise scenarios, a WER spike may reflect the noise injection rather than a real STT regression.
Default threshold: Maximum 8.0% by default
Vexa injects an interruption mid-utterance at a scripted point. The agent must stop mid-response, avoid repeating itself, and recover the task thread within two turns. The barge-in score is the recovery rate across all injected interruptions in the suite.
Injection timing
The injection point is randomized within a ±0.8 s window around the scripted position to approximate real caller behavior rather than a deterministic trigger.
Default threshold: Minimum recovery rate 70% by default
All thresholds are configurable per suite. You can also create custom rubric extensions for domain-specific criteria (e.g. medical disclaimer adherence or PCI compliance language) via the Scoring API.
What changed between v1.4.2 and v1.4.3
Every Vexa suite run is stored against the git ref that triggered it. Regressions are computed as the delta between the current run and the baseline you have pinned.
| Metric | v1.4.2 | v1.4.3 | Delta | Status |
|---|---|---|---|---|
| Task success | 91.4% | 73.6% | −17.8 pp | Regression |
| WER | 5.8% | 5.9% | +0.1 pp | Stable |
| p95 Latency | 614 ms | 618 ms | +4 ms | Stable |
| Barge-in recovery | 74.0% | 56.2% | −17.8 pp | Regression |
| Tone | 88.2 | 87.9 | −0.3 | Stable |
The baseline can be pinned to any prior run by run ID, or to the last passing run automatically. Regressions are computed per metric independently: a drop in task success blocks the deploy even if WER and latency are stable.
Gate your deploy on a voice-test pass
Drop a vexa.yml into your repo root. Vexa runs as a CI step, posts results to your PR and blocks the merge if any threshold is breached.
How the gate works
When a PR is opened, Vexa runs the suites defined in your vexa.yml against the agent endpoint you specify. If every threshold passes, the Vexa check turns green and the merge is unblocked. If any metric breaches its threshold, Vexa posts a failure summary to the PR comment thread and blocks the merge.
The failure summary links directly to the failing turns in the Vexa console, so the engineer who opened the PR can see exactly which scenario, which voice and which turn caused the block, without switching tools.
What you can threshold-gate
- Task success minimum (suite mean, per scenario, or both)
- WER maximum (per suite, or per voice class)
- Latency p95 maximum in milliseconds
- Barge-in recovery rate minimum across injected interruptions
- Tone minimum, when a persona YAML is configured
Each threshold can be global or scenario-specific. A barge-in suite can have a tighter latency threshold than a background-noise suite, because your callers expect faster recovery from interruptions.
See the CI integration guide for the GitHub Actions and GitLab CI step definitions.
# vexa.yml - add to your repo root
suite: booking-flow
agent: wss://voice.myapp.com/ws
thresholds:
task_success_min: 85 # block the deploy below this %
wer_max: 8.0 # block the deploy above this %
latency_p95_ms: 900 # block above this value in ms
barge_in_recovery: 70 # block below this %
voices:
- en-US-standard
- en-IN-wavenet-F
- en-GB-polyglot
- es-US-bilingual
- zh-Hans-CN-accent
scenarios:
- booking-intent
- refund-dispute
- warm-transfer
- background-noise-coffeeshop
- barge-in-mid-utterance
- fast-talker-high-wpm
on_regression:
block: true
notify:
slack: "#voice-alerts"
pagerduty_key: $PD_ROUTING_KEYMore detail on how Vexa works
Every voice, tested.
Connect one agent and run your first suite of simulated calls in a minute, free.