Home
/
Learn
/
Voice agent testing and evaluation
/
QA Scorecard Template for Voice Agent Call Quality
Template
8
 MIN READ

QA Scorecard Template for Voice Agent Call Quality

QA scorecard template for AI voice agents: eight weighted criteria, a scoring formula, auto-fail rules, and how to automate call quality scoring at scale.

ON THIS PAGE

A voice agent QA scorecard is a fixed set of criteria, weights, and pass rules that every reviewed call is graded against, so call quality is measured the same way no matter who, or what, does the reviewing. This QA scorecard template is built for AI voice agents. It keeps the parts of a human-agent scorecard that still apply and adds the checks AI agents need, such as tool use, factual accuracy, and response time.

What goes on a voice agent QA scorecard?

Use the eight criteria below as your starting template. Weights add up to 100, and you should adjust them to match what a failed call costs your business.

CriterionPass looks likeCheck it inWeight
Task resolvedOutcome confirmed in your systemSystem of record: the booking, lead or ticket25
AccuracyNo invented or wrong factsTranscript against instructions and knowledge base20
Tool and data useRight tool, right inputsTool call log, not what the agent said about it15
Required stepsVerification and disclosures doneTranscript or recording. A fail usually fails the call15
EscalationRight transfer, with contextTransfer record and transcript10
ResponsivenessMedian delay under your limitLatency in call data5
Turn-takingYields when the caller interruptsOverlapping speech in call data5
Tone and clarityClear, concise, on-brandRecording5
Total100

Define each criterion in terms your reviewers can check from the recording, transcript, and system logs:

  • Task resolved means the outcome exists in your system of record. For a lead qualification agent, that is a qualified lead routed to the right team with the required fields filled, not a transcript where the agent said "I've passed that along."
  • Accuracy means every fact the agent stated is supported by its instructions, knowledge base, or the caller. Invented case numbers, wrong hours, and contradictions all fail.
  • Tool and data use means the agent called the right function with the right inputs. Check the tool call itself, not just what the agent said about it.
  • Required steps covers anything mandatory: identity verification, a recording notice, a disclosure. This row is pass or fail, and a fail usually fails the whole call.
  • Escalation means the agent transferred when it should have, reached the right destination, and passed context so the caller did not repeat themselves.
  • Responsiveness, turn-taking, and tone cover how the call felt: response delay, talking over the caller, and clear, concise wording.

How do you score a call with the template?

Score each criterion pass or fail, add up the weights of the criteria that passed, then apply your critical-error rule.

Weighted QA score: QA score = (Sum of weights for passed criteria ÷ Sum of all weights) × 100

A call passes everything except tool use (15) and turn-taking (5): (100 − 20) ÷ 100 × 100 = 80. The same call skips a required disclosure, a critical error, so it scores 0 and fails.

Apply critical errors after the math, because an auto-fail sets the score to zero.

Critical errors override the weighted score. Decide your list before you score the first call, and treat any of these as an automatic fail:

  • Skipped identity verification or a required disclosure.
  • Stated a price, policy, or eligibility the agent was not given.
  • Took a real action (booking, payment, update) with the wrong details.
  • Collected sensitive data the workflow does not need.
  • Refused or ignored a clear request for a person.
  • Ended the call before the caller's request was handled.

How is scoring an AI agent different from scoring a human agent?

Human QA exists to coach individuals. AI QA exists to find patterns and fix them once. That changes how you use the scorecard in four ways.

  • Failures repeat. An AI agent that mishandles a request on one call will usually mishandle it on similar calls. Score enough calls to see the pattern, then fix the cause.
  • The fix is a change, not a conversation. A failed criterion leads to a prompt, flow, or knowledge-base edit, followed by a regression test, rather than a coaching session.
  • The transcript is not enough. You also need tool calls and flow steps. In one run of Cekura's public voice agent benchmark, the transcript showed the caller's phone number correctly, but the agent sent a different number to the tool. A transcript-only review would have passed that call.
  • Timing is measurable. Response delay and overlapping speech can be read from call data instead of a reviewer's impression. Our guide to voice AI latency benchmarks explains which percentile to set limits on.

How many calls should you review?

There is no industry-standard sample size for AI agents, so set it by risk. A practical pattern is to review every call that fails an automated check, plus a random sample of calls that passed, so you catch errors the automation missed. Review more heavily in the first weeks after launch and after every significant change.

Keep a person in the loop for calibration. Each week, have a reviewer grade a small set of calls by hand and compare the results with the automated scores. Where they disagree, either the criterion wording is ambiguous or the threshold is wrong. Fix the definition, not just the score. The same scorecard also works on human-handled calls, which is the basis of most call quality monitoring programs, so you can compare AI and human calls on equal terms.

How do you automate this scorecard in Athena?

Athena's AI QA maps closely to this template. You create a cohort of calls (filtered by agent, date range, call length, and other fields), set a sampling percentage and a weekly maximum, and then define resolution criteria in two parts:

  • AI-evaluated conditions for the judgment rows: task resolved, required steps completed, escalation handled correctly. Each is a named check with a prompt that describes success.
  • Performance metrics with thresholds for the measurable rows: latency, agent hallucination, tool call inaccuracy, node transition inaccuracy, word error rate, overlapping speech, agent naturalness, and sentiment.

Turn on weighted scoring to assign the weights from the template and set the score a call must reach to pass. Reviewers can calibrate an individual call to success or failure, which changes that call's score without changing the criteria. The AI QA metrics reference defines how each metric is calculated. As one reference point, Athena's QA troubleshooting guide treats a median end-to-end latency above 2.5 seconds as high.

For outcome fields you need on every call, Post Call Extraction writes structured results such as a call summary, a success flag, and custom fields onto each call, and those fields can filter a QA cohort. Alert rules can notify you by email or webhook when the number of calls failing QA rises.

What should you do when a call fails the scorecard?

Treat a failure as the start of a fix, not the end of a review.

  1. Review several failed calls on the same criterion to confirm the pattern before changing anything.
  2. Make the change in a draft version of the agent.
  3. Turn the failed call into a test case and rerun your regression batch.
  4. Publish the version, then watch the criterion's pass rate on new calls.

Track the scorecard's pass rate per criterion over time on a dashboard. Our voice agent KPI dashboard template shows how to lay it out. If you are still choosing a platform, use this scorecard as the grading sheet for your pilot, as described in our guide to voice agent evaluation before go-live.

What does a weekly QA review routine look like?

A weekly QA review routine is one fixed cycle, run by one owner, that turns scorecard results into fixes: sample the week's calls, read them against the scorecard, flag patterns, change the agent, and check the next week whether the change held. The scorecard tells you what failed. The routine makes sure someone acts on it.

Diagram of a five-stage weekly QA review routine for an AI voice agent. 1 Sample: every call that failed a check or hit a critical error, a random set of passes, and calls on any version published this week. 2 Read: transcript, tool calls and the outcome in your system of record, one caller intent at a time. 3 Flag: every critical error, a criterion failing on several calls with the same intent, and reviewer versus score disagreements. 4 Change: one fix per pattern in a draft version, with the failed call turned into a test case. 5 Verify: rerun the test batch, publish, then track the criterion's pass rate. In Athena the stages map to an AI QA cohort, the call QA sheet and Post Call Extraction fields, calibration, simulation testing, and versions and tags. One in-house owner, such as a CX automation engineer, runs the cycle.
A weekly QA review routine: five stages, one owner, and where each stage happens in Athena. Diagram by Athena, October 2026.
  • What to sample. Start with the sample described above: every call that failed an automated check or hit a critical error, plus a random set of calls that passed. Add calls on any agent version published that week, and calls that ended in a transfer or an early hang-up. In Athena, a QA cohort with a start date and no end date keeps adding new calls as they come in, and the weekly max caps how many are scored.
  • What to read in transcripts. Begin with the caller questions that resolve least, since AI QA groups similar questions and shows a resolution rate for each. Then read failed calls one intent at a time. Compare what the agent said with the tool call it made and the outcome in your system of record. Check names, numbers, and dates against the recording, since one misheard digit can send the rest of the call the wrong way. For each hallucination, note whether the agent invented a fact, contradicted its instructions, or misunderstood the request, because each needs a different fix.
  • What to read in post-call analysis. Check the call summary and success flag against the transcript. If calls are marked successful but no outcome exists in your system, fix the success definition before you touch the agent. Group failures by your own extraction fields, such as issue type, to see which call reasons fail most.
  • What to flag. Every critical error, even a single one. Any criterion that fails on several calls with the same intent. Any call where the reviewer and the automated score disagree. Mark failures the agent could not have prevented, such as a caller hanging up early or a transfer the caller asked for, and keep them off the fix list.
  • What to change. One change per flagged pattern, run through the fix steps above: a draft version, a test case built from the failed call, a regression batch, then publish. If reviewers keep calibrating the same kind of call the same way, change the criterion wording or threshold instead of overriding call by call. Leave the weights alone unless the cost of a failure has changed.

Give the routine one named owner. In a growing number of teams that owner is a CX automation engineer, the in-house person who runs and improves the agents directly instead of waiting on a vendor to make each fix. A second reviewer grades a few calls by hand each week to keep the automated scores honest, and the owner closes the week with a short note: pass rate per criterion, what changed, and what to watch next week.

To pull outcome fields from each call into the scorecard, use post-call analysis.

A scorecard only improves quality if your team can act on what it finds. Athena lets your own team run QA, testing, and versioning on its agents, and you can prove it on a pilot with real calls before any contract. Talk to the Athena team about a pilot.

Frequently asked questions

What is a QA scorecard?

A QA scorecard is a fixed set of criteria, weights, and pass rules that every reviewed call is graded against, so quality is measured the same way whoever, or whatever, does the review. A scorecard for AI voice agents adds checks a human-agent scorecard lacks, such as tool use, factual accuracy, and response time.

How do you calculate a QA score?

Score each criterion pass or fail, add the weights of the criteria that passed, divide by the total of all weights, and multiply by 100. Then apply your critical errors: any automatic fail sets the score to 0, whatever the weighted total was.

How often should you review QA scorecard results?

Weekly is a practical default for an AI voice agent: sample and read calls, flag patterns, and ship fixes once a week. Review more heavily in the first weeks after launch and after any significant change, and use alerts to catch sudden drops between reviews.

Who should own QA for an AI voice agent?

One named in-house owner, often a CX automation engineer, who runs the weekly review, makes fixes in a draft version, and tests them before publishing. A second reviewer grades a few calls by hand each week so the automated scores stay calibrated.

Related articles