Most contact centres now run at least one AI voice agent, and almost none of them score it. The scorecard they already own was written for humans — it asks whether the agent showed empathy, used the customer's name, avoided jargon. Point that at a voice bot and you get numbers that look fine while the bot talks over callers and invents refund policies.
Voice agents fail in their own particular ways. They keep talking when someone interrupts. They pause just long enough that the caller says "hello?". They answer confidently with something that was never in the knowledge base. None of that appears on a human QA form.
This is the framework we use in Xperia to score AI voice agents: 18 signals in four groups. It is deliberately separate from human scoring, because mixing the two produces averages that describe nobody.
Why human scorecards mislead you
A human QA form measures judgement, warmth and language. A voice agent has none of those and does not need them. What it needs is to take turns properly, respond fast enough to feel alive, understand what was asked, and never assert something it cannot support.
Score a bot on a human form and two things go wrong. Signals that do not apply get marked as passes, inflating the score. And the failures that actually lose customers — a two-second dead air before every reply, a confidently wrong answer — have no field to be recorded in.
| Question | Right for a human | Right for a voice bot |
|---|---|---|
| Did the agent show empathy? | Yes | No — measure tone matching instead |
| Was grammar correct? | Yes | No — measure phonetic accuracy |
| Did the agent stop when interrupted? | Rarely asked | Critical — barge-in |
| How long before the reply began? | Rarely asked | Critical — latency |
| Did the agent state anything unsupported? | Rarely asked | Critical — hallucination |
The four categories
Eighteen individual signals is too many to manage as a flat list, so they roll up into four categories. Each answers a different question about the agent.
1. Conversational fluidity — does it take turns like a person?
This is the category callers notice first and the one most bots fail. Six signals:
- Barge-in handling — does the agent stop speaking when the caller interrupts, or talk over them to the end of its sentence?
- Turn-taking — are the two parties alternating cleanly, or overlapping and colliding?
- Filler sensitivity — does an "um" or a pause get misread as the caller finishing their turn?
- Dynamic pacing — does delivery speed adapt, or is every sentence the same metronome?
- Latency — how long between the caller stopping and the agent starting? Past roughly a second, callers start repeating themselves.
- Transitions — does it move between topics and hand off cleanly, or restart context each time?
2. Emotional design — does it read the room?
- Tone matching — a frustrated caller and a cheerful caller should not receive identical delivery.
- Conciseness — does it answer the question, or deliver a paragraph where a sentence would do?
3. Acoustic quality — can the caller actually hear it?
Easy to overlook because it sounds fine in testing on good headphones, then falls apart on a mobile connection in a moving car.
- Volume normalisation — consistent level, no jumps between phrases.
- Speech-to-silence ratio — how much of the call is actually speech versus dead air.
- Voice clarity — intelligibility over a degraded line.
- Phonetic accuracy — names, addresses and reference numbers pronounced correctly. This one drives repeat calls.
- Prosody and intonation — does it sound like speech, or like a list being read out?
4. NLU intelligence — does it understand, and does it make things up?
- Intent accuracy — did it correctly identify what the caller wanted?
- Entity extraction — did it capture the order number, the date, the amount?
- Fallback handling — when it did not understand, did it recover gracefully or loop?
- Context retention — does it still know, in turn eight, what was established in turn two?
- Hallucination — did it assert anything it had no grounding for? For most teams this single signal justifies the whole exercise.
Hallucination is the one to start with
If you measure only one thing, measure this. A slow bot is annoying. A bot that invents a returns window, quotes a price that does not exist, or confirms a booking it never made creates a commitment your business then has to honour or deny. Both outcomes cost more than the call did.
A bot that is wrong 2% of the time is not 98% useful. It is a policy you have not written, applied at random.
Score bots and humans separately, report them together
Bot calls and human calls need different criteria but the same dashboard. Keeping them in separate systems hides the thing you most need to see: what happens at the handover. A bot that scores well by deflecting everything to a human is not performing well, and you can only see that when both sides sit in one view.
In Xperia, AI-handled calls are segmented from human-handled calls across analytics, call history and configuration — each scored against its own criteria, both visible in one account.
How to start this week
- Pull 50 bot-handled calls at random. Not the ones flagged as problems — a random sample, or you will only confirm what you already suspect.
- Score them on latency, barge-in and hallucination only. Three signals, not eighteen. You are establishing a baseline, not building a programme.
- Count how many contained an unsupported claim. This number is usually the surprise.
- Compare against the same period's human calls for resolution rate. This tells you whether the bot is deflecting or resolving.
- Then widen to the full eighteen once you know where the damage is.
AI voice agents are being deployed far faster than they are being measured. The teams that get ahead are not the ones with the most sophisticated bot — they are the ones who noticed their bot was interrupting callers and inventing policies, because they were looking.
You do not need eighteen signals on day one. Start with latency, barge-in and hallucination on fifty random calls. That will tell you more than any vendor demo.