How to Test AI Voice Agents Before Production

The agent introduced itself, disclosed that it was an AI and asked permission to continue. It failed the compliance check anyway.

The failure was not in the voice agent. It was in the test.

The speech-to-text provider had split one sentence at an utterance boundary. The evaluator read the fragment “Hey. My name is Max.” as the complete opening and marked the required disclosure missing. The caller had heard the full sentence. The testing system had not.

This is the central problem in AI voice agent testing: a test can produce a clean verdict while measuring the harness, transcription or timing model instead of the agent. A polished demo proves even less. “It worked when I tried it” is one call against one cooperative person, and a generative voice agent is not a deterministic script.

AdaptiveX built the Agent Proving Ground to make difficult callers repeatable. It puts a production-configured voice agent on the phone with synthetic callers hundreds of times before it calls a real person. Every simulated call is recorded, transcribed, timed and scored against a scorecard the account curates. Failed checks are then read against the agent's own instructions and turned into specific prompt edits with quoted evidence.

The shortest explanation is this: it is a flight simulator for voice agents, and it files the snag report after the flight.

What is AI voice agent testing?

AI voice agent testing is the repeatable evaluation of a complete calling system across synthetic callers, scenarios, audio behaviour and measurable checks. A useful test asks more than whether the agent reached an outcome. It examines what the agent said, when it said it, how the caller responded, which rules held and whether the result can be reviewed.

That requires the whole voice path. A deployed agent depends on its prompt, flow, language model, speech recognition, text-to-speech, turn-taking, tools and post-call records. Testing only a transcript or a direct model response cannot establish how the phone call behaved.

NIST's AI Risk Management Framework says AI systems should be tested before deployment and regularly while in operation. It calls for objective, repeatable or scalable test, evaluation, verification and validation processes with documented test sets, metrics and methods. The Proving Ground applies that principle to the details that exist only in a live conversation: silence, endpointing, interruption, mishearing, latency and the caller's decision to continue or hang up.

Why manual voice agent testing does not scale

Before the Proving Ground, three problems kept recurring:

• A prompt change could reach a live customer after one save, with no repeatable test of what else had changed.
• A friendly internal caller rarely produced the failures that matter: interruption, misunderstanding, resistance, over-agreement or an unexpected route through the call.
• A person could make ten test calls, but not the same ninety calls next week, as the same difficult customer, under the same scorecard.

The answer is not simply “use another chatbot as the caller”. Two models exchanging text do not reproduce a phone call. The answer is a test library that separates who the caller is, what happens on the call and how the agent is judged.

The three objects in a voice agent test library

Persona: who the caller is

A persona defines the synthetic caller's manner and life: how quickly they follow, how much they volunteer, whether they interrupt, how well they hear, which languages they speak and whether they code-switch.

A persona carries no goal. “Hard of hearing on mobile calls” is a persona trait because it would remain true in any conversation. “Keeps pushing for a lower price” is a scenario because it describes what the caller wants this time.

A caller is who, never what they want.

Each persona can use its own speech-to-text, language model and text-to-speech configuration. The system warns when a synthetic caller falls back to the agent's providers because shared models and voices can create shared blind spots and unreadable recordings.

Scenario: what the caller wants

A scenario defines the goal, refusal or pressure applied during one call. It also includes a plausible contact record so the agent's prompt receives the same merge fields it expects in production. Without it, an unresolved field such as a name or company can be read aloud and create a failure caused by the simulator itself.

Examples include refusing an identity check, asking to skip a required step, pushing for an unauthorised concession or raising an objection named in the agent's own instructions.

Scorecard: how the call is judged

A scorecard is the curated set of checks applied to calls in a run. The Proving Ground supports three kinds:

The design rule is simple: the cheapest check that can settle the question wins. A required phrase should not be handed to a model that can score the same transcript differently on different days.

Separating the objects makes the library scale. Four scenarios across three personas create twelve simulated calls from seven reusable objects. Add a persona and every scenario can be exercised by it. Add a scenario and every caller type can face it.

How a Proving Ground run works

A run is a grid: scenarios on one axis, personas on the other. Every intersection is one simulated call, or cell.

The complete grid is created as pending rows before calls begin. The interface can therefore show the full run immediately and fill each cell as results arrive, rather than appearing to hang while rows are created.

For each cell:

1. Two audio endpoints are joined mouth-to-ear on a shared clock.
2. The agent starts a real session with its actual flow, prompt, model and configured providers.
3. The synthetic caller answers using its persona, scenario, model, ears and voice.
4. They speak in real time, with endpointing, interruption and barge-in active.
5. The caller can end the call with a hang_up tool, or the agent or wall-clock limit can end it.
6. The call is recorded as stereo audio through the production recording path.
7. Every applicable scorecard check is evaluated.
8. The grid refreshes its totals and pass rate.
9. After the run, failed checks and transcripts are passed to the prompt advisor.

The Proving Ground has two modes. Text mode is for iterating on a prompt. Voice mode is for signing one off.

Text mode uses the real prompt, model and flow but removes the audio edges, making a large run much faster and cheaper. Voice mode runs speech in both directions at wall-clock speed and can expose interruption, endpointing, mispronunciation, mishearing, latency and talk-ratio problems that do not exist in text.

A phone line with no phone network

The most important engineering detail is the loopback line.

The agent's audio goes to another AI instead of a carrier, but it is still carrier-shaped audio: 8 kHz μ-law released one 20-millisecond frame at a time at wall-clock speed. The line also produces the continuous silence a real phone line carries because speech-to-text endpointing depends on receiving it.

If audio is drained as quickly as text-to-speech can generate it, the transcript can look fine while the conversation never happened. Both speakers finish instantly, so interruption and barge-in are impossible. Turn-taking is a timing behaviour; without a clock, it has not been tested.

The line also limits catch-up after a host pause. Releasing a second of delayed audio in one burst would look like a telephony glitch to the transcription provider, not normal speech.

Making a synthetic caller behave like a person

A model given a character description tends to act like written dialogue: it announces the character immediately and then becomes generic. Real phone behaviour is the opposite. A person answers with very little and reveals their traits over several minutes.

AdaptiveX wraps each persona in a caller harness that requires it to:

• speak only the words said aloud, without narration or stage directions;
• remain in character through repetition, misunderstanding and interruption;
• start with an ordinary greeting rather than announcing its mood or backstory;
• demonstrate a trait instead of describing it;
• preserve the trait through the tenth turn, not only the first;
• use a tool call to hang up, with the reason retained on the call.

This matters because a caller who says “I am impatient” gives the agent a problem no real caller presents. A caller who is impatient for one turn and cooperative thereafter lets a weak agent pass.

The opening line receives its own validation. It must be a short, real greeting in the caller's language. That prevents caricatured openings from loading the entire persona into the first sentence before the agent has spoken.

Turning an agent prompt into a test specification

The most relevant test cases are already implicit in the agent's instructions. The Proving Ground generator reads those instructions as evidence, not orders, and looks for branches to test:

• every call stage that can be refused, skipped or misunderstood;
• every objection the prompt says the agent can handle;
• every instruction containing “always” or “never”;
• every required disclosure or template field;
• every gap where the agent has no stated path;
• and the straight, cooperative call that confirms the agent can achieve its purpose.

An agent is at its worst off its own map, and a gap in a prompt is invisible until a caller walks into one.

Generated personas, scenarios and checks return as drafts for human review. They are deduplicated against the existing library, and personas are linted for leaked goals. This avoids a scenario grid in which the caller's hidden agenda contradicts the scenario it has been paired with.

Scoring that a human can challenge

When a model judge is necessary, it receives the full call and one closed question. It must return a verdict, a one-sentence reason and the agent's own words as quoted evidence for a failure.

The judge follows three constraints:

1. Judge only the question asked.
2. Judge only the agent, not the synthetic caller.
3. If a failure cannot be supported by quoted evidence, pass the check.

Unexpected or errored evaluations remain visibly unscored. They are never guessed from loose prose and never converted into a failure that blames the agent for the evaluator's fault.

The transcript used for judgement also matters. The evaluator reads the agent's own conversation record for what it said, combined with the caller's turns as the agent received them. That avoids the split-sentence failure from the opening of this article, where a lossy merged transcript made a correct disclosure appear incomplete.

A human can listen to the recording, review the transcript and overrule any verdict. The human decision is stored beside the machine decision, not over it. The run's pass rate updates, and the disagreement remains available for measuring the judge.

A judge that nobody can contradict is a judge nobody can measure.

Measuring latency the way a caller feels it

Time to first model token and time to first synthesis byte are component metrics. Neither is the pause the caller hears.

Caller-perceived latency starts when the caller stops speaking and ends when agent audio begins. Much of that interval can be spent in endpointing before the model is called. The Proving Ground measures the gap directly at the loopback, using audio-frame indices as the clock.

It retains every gap and reports p50, p95 and maximum latency with the sample count. It also identifies overlap, caller interruption, unanswered turns and host stalls. Keeping the samples means a new threshold can be applied to a completed call later without rerunning it.

Multilingual calls need multilingual failure handling

Personas carry languages explicitly, and code-switching is instructed rather than assumed. Without that constraint, a synthetic caller may use the target language briefly and drift back to English, producing a “multilingual” test that proves very little.

Some scripts create another problem. Thai, Lao, Khmer, Burmese and CJK languages do not use spaces between words in the same way English does, while transcription providers can emit fragmented or syllable-spaced text. Feeding that fragmentation back into the caller model can create a degradation loop in which each turn becomes less intelligible.

The Proving Ground repairs spacing only between characters of an unspaced script while preserving legitimate spaces around Latin names and product terms. It also tells the caller to write normally rather than copy transcription fragmentation.

Any completed call can be translated into English for review, but the translation is stored beside the original. It never replaces the record on which the verdict was reached.

From a failed check to a prompt edit

A scorecard shows what failed. The prompt advisor tries to locate why.

It receives the agent's instructions, failed checks and relevant transcripts. Each proposed change includes severity, the problem, quoted evidence, the checks it explains and a literal before-and-after edit.

Suggestions must fix the cause rather than the symptom, match the surrounding prompt and address one problem at a time. The advisor does not propose prompt changes for latency, voice or transcription failures because those problems sit outside the prompt.

The advisor proposes; it never applies. An operator can edit the wording and approve the final change. The application is refused if the source text has changed, appears more than once or the same suggestion was already applied. Changes are written to the current prompt, while the copy attached to the completed run remains unchanged as the record of what was tested.

That makes the improvement loop traceable:

Change the prompt → run the same grid → inspect the evidence → approve a specific edit → run it again.

What the Proving Ground can and cannot claim

The Proving Ground reports the pass rate across a documented number of simulated calls. It records the checks, calls and evidence behind the result. It does not certify an agent, guarantee safety or claim statistical significance where none was calculated.

Publishing a completed run makes its scorecard and supporting calls visible to the team. It does not automatically block a deployment. Humans remain responsible for deciding whether the evidence is sufficient for the intended use.

That is a narrower claim than “this agent is proven safe”, and a far more useful one:

This version faced these reproducible callers, in these scenarios, over this voice path, against this scorecard—and these are the calls and verdicts behind its pass rate.

The AdaptiveX Agent Proving Ground turns voice agent testing from a friendly demo into an operating discipline. To see how it connects to live decision rules, explore the Behavioural Engine, or bring us a difficult caller to add to a real test grid.