How the AI works
An honest account of what Vexa does
Language models are the engine of Vexa, not a feature bolted onto a test runner. Without modern AI, this product does not exist. Here is an accurate, technical account of how the caller model, the judge model and the routing layer work, and what they get wrong.
Simulate the caller
A language model role-plays a realistic caller for each turn of the conversation. It does not try to be helpful. It tries to accomplish a goal, the way a real caller would.
A persona, not a prompt
The caller model receives a persona definition: accent region, speech rate class, emotional baseline, call goal and behavioral flags. It does not receive instructions to be helpful. It behaves as a real caller would: with a goal, constraints, occasional impatience and a tendency to change its mind mid-sentence.
Realistic, not perfect
Real callers trail off, barge in, pivot topics and give incomplete information. The caller model reproduces these behaviors from a behavioral flag library rather than generating them randomly. This keeps the distribution of behaviors consistent across reruns while still being realistic.
Turn-by-turn, not scripted
The caller model generates each utterance after receiving the agent's last response as a text transcript. The conversation is genuinely dynamic: the caller model responds to what the agent actually said, not to a scripted agent turn. This catches agent failures that scripted tests would miss.
Accent and speech rate
Accent and speech rate are applied at the synthesis layer, not modeled in the text. The caller model generates the words; a speech synthesis model renders them at the specified speech rate and in the specified accent variant. Vexa currently supports 12 accent classes across 47 voice configurations.
What the caller model receives (illustrative)
persona:
name: Rosa
accent: es-US-bilingual
speech_rate: fast # > 220 WPM
emotional_baseline: mild-impatience
call:
goal: "Book a table for two on Saturday evening around 7 PM"
constraints:
- "Would prefer a window seat if available"
- "Has a nut allergy, will mention if food comes up"
behavioral_flags:
- barge-in-at-turn-3 # interrupt the agent mid-response on turn 3
- topic-pivot-allowed # may ask about parking before completing booking
agent_endpoint: wss://voice.myapp.com/wsThe caller model uses this definition to generate utterances but is not given the agent's response text verbatim. It receives a transcript produced by the agent's own STT layer.
Score the conversation
A separate judge model scores every agent turn as it arrives. The judge is kept deliberately isolated from the caller model to prevent it from being lenient on turns the caller found acceptable.
Task success
The judge receives the caller persona, the caller's goal, the transcript of the current turn and the turn history. It answers: did the agent make meaningful progress toward satisfying the caller's goal on this turn? The answer is scored 0 to 100 and reasons are attached.
Tone
The judge receives the agent's utterance and, if present, a persona guideline YAML for the agent. It evaluates professionalism, warmth and adherence to persona. Tone scoring does not require audio; it operates on the STT transcript of the agent's response.
What the judge cannot score
Latency is not scored by the judge. It is measured directly from audio timestamps: the gap between the caller's last audio byte and the agent's first audio byte. WER is estimated by string comparison between the caller model's utterance text and the STT transcript, not by a model.
Why two separate models
Using the same model to play the caller and judge the agent creates a conflict: the caller model that accepted the agent's response is unlikely to also mark it as failing. Vexa runs the caller model and the judge model as separate inference calls with separate system prompts and separate context windows. The judge never sees what the caller model generated internally.
The judge model also receives the original persona definition and the call goal, so it can evaluate task success against what the caller was actually trying to accomplish rather than against a generic completion criterion.
Judge prompt structure (illustrative)
You are evaluating a voice agent's response.
Caller goal: {caller_goal}
Caller persona: {persona_summary}
Turn history: {prior_turns}
Agent's response this turn: {agent_utterance}
Score task_success 0–100:
- 90–100: goal clearly advanced
- 70–89: partial progress, no regression
- 40–69: no progress or confusion introduced
- 0–39: goal regressed or caller likely to abandon
Score tone 0–100 against persona guidelines.
Attach a one-sentence reason for each score.
Do not be lenient because the response sounds polite.Detect regressions across releases
Scores from the current suite run are compared against a baseline pinned to a prior git ref. A regression is flagged when any metric drops by more than the configured delta, not just when it falls below an absolute threshold.
Compare runs, not states
Regressions are detected by comparing two runs of the same suite against the same scenarios, not by comparing an absolute score to a threshold. A task success drop from 91% to 74% is a regression even if 74% is above the configured minimum, because the relative drop signals something changed.
Grouped by root cause
Vexa groups failing turns by scenario type, voice class and behavioral flag to surface the root cause rather than the symptom. A barge-in handling bug will cluster under every barge-in scenario, making it immediately distinguishable from a latency regression that affects only certain voices.
Pinned to a git ref
Every suite run is stored against the git commit that triggered it. You can pin your baseline to a specific run ID, the last passing run, or a named release tag. Regressions are always computed against the baseline, not against an absolute threshold.
Example regression report (CLI output, illustrative)
vexa run --suite booking-intent --baseline v1.4.2 --compare v1.4.3
✓ WER 5.8% → 5.9% +0.1 pp within threshold
✓ Latency p95 614ms → 618ms +4ms within threshold
✓ Tone 88.2 → 87.9 −0.3 within threshold
✗ Task success 91.4% → 73.6% −17.8 pp REGRESSION (threshold: max −5 pp)
✗ Barge-in recovery 74.0% → 56.2% −17.8 pp REGRESSION (threshold: max −5 pp)
2 regressions detected.
Failing scenarios: barge-in-mid-utterance (14 turns), fast-talker-high-wpm (6 turns)
Failing voices: en-IN-wavenet-F, es-US-bilingual
Deploy blocked. Review failing turns at https://vexavoice.tech/console/run/4218The right model for each task
Vexa is not tied to one model or one vendor. A small routing layer maps each task type to the model best suited to it, with a fallback for availability. You get the judgment, not a single provider's ceiling.
Why model-agnostic routing matters
Simulating a caller turn is a different task from evaluating whether the agent's response succeeded. Generating a fast, persona-consistent utterance requires speed and instruction-following. Judging a response requires reasoning, nuance and a high tolerance for ambiguity. Using the same model for both tasks wastes money on the first and compromises quality on the second.
Vexa maintains a route registry that maps each task type to the model best suited to it, with a declared fallback for availability. The current default uses gpt-4.1-mini for caller simulation and wer estimation, gpt-4.1 for turn scoring and regression summaries, and claude-sonnet-4 for failure narratives where plain language explanation is the primary output.
Adding a provider is a config change
If you want to route caller simulation through a self-hosted model, you add it to the route registry in your workspace settings and point it at your endpoint. Vexa will use it for all future runs in that workspace. You do not need to contact us to add a provider.
The routing layer also means Vexa improves without changes on your side. When a new model performs better at turn scoring, we update the registry and your suites benefit automatically on the next run.
No model routing information is exposed in the public API or in the Vexa console by default. You see the scores, not the vendor that produced them.
# Vexa model route registry (illustrative)
tasks:
caller_simulation:
model: gpt-4.1-mini # fast; strong persona adherence; low cost at 4k calls/run
fallback: claude-haiku-4
turn_scoring:
model: gpt-4.1 # stronger judgment for nuanced task-success evaluation
fallback: gpt-4.1-mini
wer_estimation:
model: gpt-4.1-mini # straightforward comparison task; cheap enough per turn
fallback: claude-haiku-4
regression_summary:
model: gpt-4.1 # interprets score deltas across a full suite run
fallback: gpt-4.1-mini
failure_narrative:
model: claude-sonnet-4 # best at explaining multi-turn failure chains plainly
fallback: gpt-4.1What Vexa gets wrong, and how you catch it
AI judges can be wrong. Simulated callers are approximations. Scores are directional, not literal. This page exists because you should know that before you gate a production deploy on a Vexa suite.
Simulated callers approximate real ones
The Vexa caller model reproduces a defined distribution of behaviors. Real caller populations are larger and more varied. Edge cases you have not defined will not appear in your suite. Running your suite regularly and adding new scenarios from real call transcripts is the right way to close this gap over time.
Task success scores are directional
A task success score of 91% does not mean 91% of your real callers will complete their goal. It is the judge model's assessment of task progress per turn, averaged across a defined scenario set. Use it as a relative signal: if it drops, something regressed. Do not treat it as a literal prediction of real-world completion rates.
WER estimation is approximate
Vexa estimates WER by comparing what the caller model said to what your STT layer transcribed. It does not have access to your STT internals or a ground-truth audio transcript. In high-noise scenarios, a WER spike may reflect the noise injection rather than a genuine STT regression. Check the noise-free equivalent before concluding an STT bug.
Latency measurement includes network overhead
The latency Vexa reports includes the SIP/WebRTC round-trip overhead between Vexa's infrastructure and your agent endpoint. Per-region calibration files are published in the docs so you can subtract Vexa's overhead from the reported value to isolate your agent's true end-to-end latency.
The judge can be wrong
Language model judges are not perfect. A judge can mark a technically correct agent turn as failing because the phrasing was ambiguous, or mark a failing turn as passing because the agent used confident language. Review failing turns in the console before making release decisions. The judge is a first-pass filter, not a final verdict.
Always keep a human in the loop for release decisions
A Vexa suite can tell you whether scores dropped and which turns failed. It cannot tell you whether your real customers will notice or care. Review the failing turns in the console, listen to the call replays, and bring your own judgment before making a release decision. Vexa is a first-pass filter, not a final verdict.
How the design handles uncertainty
Every failing turn in the Vexa console shows the caller utterance that triggered the failure, the agent response that was scored, and the judge's reasoning: a one-sentence explanation of why the score was assigned. You can read the judge's reasoning, check whether you agree, and mark the turn as a false positive if you disagree. False positive flags feed into how the judge is calibrated for your workspace over time.
Latency and WER do not go through the judge. They are computed directly from audio timestamps and string comparisons. If you see a WER spike, you can inspect the caller's text and your STT's transcript side by side in the console.
Vexa also publishes the per-region latency overhead calibration data so you can subtract it yourself if you prefer not to rely on our subtraction. The calibration method is documented in the latency methodology guide.
Data handling
Vexa does not record your agent's audio or transcripts beyond the duration of a suite run. Call-level transcripts and scores are stored in your workspace and are yours. We do not use your suite results to train models, and we do not share workspace data across accounts. The full detail is on the Security page.
Every voice, tested.
Connect one agent and run your first suite of simulated calls in a minute, free.