Calibration is the practice of checking that different reviewers score the same call the same way. Skip it and you have a number that means something different depending on who produced it — which makes comparing agents across teams meaningless.
It takes ninety minutes and it is the highest-return activity in a QA programme.
How to run one
- Pick three calls. Not the worst ones — one straightforward, one messy, one where you expect disagreement.
- Everyone scores independently. No discussion, no shared screen. Three to six reviewers is ideal.
- Collect the scores before discussing anything. The spread is the data; discussion contaminates it.
- Work through parameter by parameter, not total by total. Totals hide which parameter caused the divergence.
- For each disagreement, ask what the parameter's wording permitted. Usually both readings were defensible.
- Rewrite the wording. This is the output — not agreement on these three calls, but a scorecard that cannot be read two ways.
Reading the spread
| Spread on total score | What it means | Action |
|---|---|---|
| Under 5 points | Well calibrated | Spot-check quarterly |
| 5–10 points | Normal for a new scorecard | Fix the two or three worst parameters |
| 10–20 points | Scorecard is ambiguous | Rewrite before trusting any comparison |
| Over 20 points | Reviewers are applying different standards | Stop reporting agent comparisons until resolved |
The parameters that always cause it
The same categories fail calibration everywhere:
- Anything with an adverb — "responded promptly", "listened actively". Promptly by whose stopwatch?
- Composite parameters — "greeted the customer and identified themselves" is two checks. Reviewers split on partial completion.
- Outcome-based parameters — "resolved the issue" depends on information the recording may not contain.
- Anything requiring the reviewer to infer intent — "showed empathy" is a judgement about an internal state.
The fix is always the same: replace the judgement with an observable. Not "was the agent patient" but "did the agent interrupt the customer".
What calibration does not fix
It aligns reviewers on interpretation. It does nothing about coverage — six perfectly calibrated reviewers scoring 2% of calls are still describing 2% of your operation. And it does not tell you whether the scorecard measures the right things, only that it is measured consistently.
Calibrating against automated scoring
If you run automated scoring, calibration gains a second use: comparing human scores against the system's on the same calls. Persistent divergence on a parameter usually means its definition is ambiguous — which is exactly what calibration is for. Xperia keeps the original AI score after any supervisor override, so those disagreements stay visible rather than being overwritten.
Uncalibrated QA scores are not comparable, and any ranking built on them is partly measuring who did the reviewing.
Run one session, expect a wide spread, and treat every disagreement as a defect in the scorecard rather than in the reviewer. The wording is almost always the problem.