TL;DR: A QA rubric defines how to score one behavior across performance levels. Start with observable descriptors: each level names what the agent said or did, and each line item carries a weight mapped to a business outcome. The hardest part is writing those descriptors, because vague terms like "professional" cannot be scored consistently by humans or AI. That observable-descriptor standard is what makes a rubric survive calibration and agent disputes. Rewrite every vague term as an observable behavior before the rubric goes into calibration or automated QA.
When two evaluators score the same call differently and neither can point to the transcript moment that justified the number, the QA process breaks down. That is what a QA rubric exists to prevent.
A QA rubric is a scoring guide that defines what good looks like at each performance level for a single behavior, such as empathy or compliance. It differs from a scorecard, which combines multiple scored line items. A checklist records whether something happened.
Building a reliable QA rubric
A reliable rubric is one that two evaluators can apply to the same conversation and reach the same score.
The four components of a rubric
The behavior is what you are scoring. The performance levels set the scale, typically three to five levels. The observable descriptors define what the agent said or did at each level, not what the evaluator felt. The weight or critical-fail flag determines how much the behavior matters to the overall score.
Writing observable descriptors takes more effort up front than using vague terms like "professional," but it is what makes scores defensible. A rubric that says "agent showed empathy" gives an LLM or another evaluator nothing to work with. By contrast, a rubric that says "agent acknowledged customer frustration with a specific phrase within two turns of the customer expressing the complaint" can be evaluated consistently by humans and AI.
Five steps to build a functional rubric
A functional rubric lets two evaluators score the same conversation the same way. Build one in five steps:
- Name the behavior. Choose one specific behavior to score, such as empathy or compliance adherence.
- Set performance levels. Use three to five levels, from "did not do it" to "exceeded the standard."
- Write observable descriptors. Describe what the agent said or did at each level, not what the evaluator felt.
- Assign a weight or critical-fail flag. Set how much the behavior matters to the overall score.
- Test against real conversations. Have two evaluators score the same conversation independently and compare results.
The Line Item Builder in evaluagent drafts automated line items from a plain-language description or an existing scorecard, so teams do not need to write prompts.
Alt text: Line Item Builder assistant offering to generate, upload or create line items for financial services QA scorecards
How observable descriptors make scores defensible
Agents dispute scores they see as subjective, and calibration between evaluators is hard to keep consistent. Observable descriptors give every evaluator the same reference point: what the agent actually said.
Every AutoQA score in evaluagent shows the conversation passage behind it, the rule it mapped to, and how it rolled up, so teams can defend each score to agents, executives, compliance, and clients. With evaluagent, Capital on Tap increased interaction coverage to 95%.
Rubric vs scorecard vs checklist: key differences
A rubric scores one behavior, a scorecard combines many scored line items, and a checklist records whether something happened.
| Tool | Purpose | Question it answers | Output | Example |
|---|---|---|---|---|
| Rubric | Define how to score one behavior | How well did the agent do this? | Performance level per line item | Empathy rubric with four levels |
| Scorecard | Combine multiple scored line items | What is the overall quality score? | Composite weighted score | 87/100 across all line items |
| Checklist | Record whether something happened | Did it happen? Yes/No | Binary pass/fail status | Compliance disclosures completed |
Build compliance checklists first, then add rubrics for soft-skill scoring to complete the scorecard.
Key metrics for your QA scorecard
Scorecard metrics work best when they map to business outcomes rather than agent behaviors alone. Metrics such as first contact resolution (FCR), compliance adherence, empathy, accuracy, and resolution are common starting points. Weighting resolution and compliance more heavily reflects their direct link to customer outcomes and regulatory risk.
evaluagent xMetrics, part of Conversation Intelligence, give predicted scores including xNPS, xCSAT, xCES, xRepeats, xResolution, and xVulnerability, alongside agent and customer sentiment, for each agent's conversations.
Key focus areas for checklists
Checklists focus on binary, verifiable items. Common examples include disclosures, data protection, and required language such as vulnerable-customer identification, FCA Consumer Duty disclosures, PCI redaction, and GDPR consent language.
evaluagent compliance checking verifies required disclosures and compliance language on every conversation, with an exportable audit trail.
Choosing the right QA tool
Choose a tool that supports your rubric logic, not one that forces you to rebuild it. Criteria include explainable scores, calibration support, a dispute route, multi-channel coverage, and independence from the CCaaS vendor.
Bundled CCaaS QA covers one vendor's channels, and that vendor is grading its own homework. evaluagent is independent of the call recording, CCaaS, and ticketing platforms it evaluates, so it gives one view of quality across voice, chat, email, messaging, and bots, whatever your tech stack. With evaluagent, 1st Central cut QA planning time by 95%.
Essential elements for building custom scorecards
A custom scorecard needs weighted line items, scoring criteria for each line item, a pass rate, and critical fails.
How to structure your scoring tiers
Use three to five performance levels, each defined by an observable behavior. An example structure: 0 = did not do it, 1 = attempted but incomplete, 2 = met the standard, 3 = exceeded the standard.
Mapping rubric scores to outcomes
Map each rubric score to a business outcome, such as compliance risk, customer effort, or resolution. This mapping helps QA leads show executives which behaviors drive those outcomes, making scores easier to defend. Conversation Intelligence in evaluagent, available alongside AutoQA, surfaces Reason for Contact to show which demand drivers are avoidable.
How to weight your QA line items
Weight line items by their impact on customer outcomes and compliance risk, not by how easy they are to score. One illustrative split weights compliance and accuracy highest, with empathy and resolution lower. The right split depends on your strategy and regulatory environment. evaluagent scorecards support configurable weightings and critical fails.
Practical rubric templates for call centers
The templates below are starting points. Adapt the descriptors to your own procedures and definition of good.
How to measure agent empathy
One common approach is to measure empathy through observable behaviors such as acknowledgment, validation, and action.
The example below is an illustrative starting point rather than a published standard.
| Level | Score | Observable descriptor | Example from transcript |
|---|---|---|---|
| Did not do it | 0 | No acknowledgment of customer emotion | Agent jumps straight to troubleshooting |
| Attempted but incomplete | 1 | Generic acknowledgment without specifics | "I understand this is frustrating" |
| Met the standard | 2 | Acknowledged customer frustration with a specific phrase | "I can hear how frustrating it is to wait three days for a refund" |
| Exceeded the standard | 3 | Acknowledged frustration and validated the customer's concern | "You're right to expect faster service, and here's what I'll do to fix it" |
Automated line items reference evaluagent's Context Engine to check empathy against your own framework.
Mapping regulations to QA scorecards
Map each regulatory requirement to a specific, observable line item. The FCA enforced the Consumer Duty in July 2023, requiring firms to act to deliver good outcomes for retail customers, including those in vulnerable circumstances.
Map these requirements to observable scorecard criteria. Score whether the agent identified vulnerability indicators, adapted communication by simplifying language or adjusting pace, demonstrated understanding of customer needs before offering solutions, and documented the outcome achieved.
Defining success in chat rubrics
Chat rubrics typically need to account for written tone, response time, and resolution in a text-only channel.
The example below is an illustrative starting point rather than a published standard. Set thresholds and descriptors that match your own operation.
| Criterion | Performance Level 1 | Performance Level 2 | Performance Level 3 |
|---|---|---|---|
| Response Time | >60 seconds | 30-60 seconds | <30 seconds |
| Resolution | Unresolved or escalated | Partial resolution | Full resolution, no follow-up needed |
| Tone | Robotic, dismissive | Professional, neutral | Warm, personalized |
| Accuracy | Incorrect information provided | Minor errors corrected | All information accurate |
evaluagent evaluates calls, chats, emails, messaging, and bot conversations with the same automated quality framework.
Alt text: evaluagent chat evaluation flagging an incorrect cancellation fee quote, with SmartScore analysis and improvement actions
Email rubrics for better coaching
Email rubrics typically focus on clarity, accuracy, and complete resolution, because the customer cannot ask a follow-up in real time. Score whether the agent explained what happens next and by when, whether any commitment made was specific enough to be verified later, and whether the case notes would let the next person continue without additional information.
Drafting clear criteria for agent evaluations
Clear criteria are observable, specific, and tied to a single behavior.
Identify key quality assurance goals
Start with the business outcome, then work backward to the behavior.
- Name the outcome. For example, first contact resolution.
- Identify the behaviors that produce it. For example, accurate diagnosis, complete answer, clear next step.
- Write line items for each behavior. For example, "agent provided a complete answer to the customer's question."
Create measurable performance scales
A measurable scale has observable anchors at each level. One illustrative scale for accuracy runs: 0 = gave incorrect information, 1 = gave partial information, 2 = gave correct information, 3 = gave correct information and anticipated a follow-up question.
Define observable agent behaviors
Observable behaviors let you point to specific moments in a transcript: "used the customer's name," "acknowledged the delay," "offered a specific next step," "confirmed the customer understood." Contrast these with non-observable terms like "professional," "courteous," and "helpful."
Draft objective criteria for fair grading
Objective criteria remove the evaluator's opinion from the score. Test a criterion by asking: can two evaluators point to the same transcript passage and agree on the score?
Validate rubric criteria with real calls
Test draft criteria against real calls before rolling them out.
- Select a sample of calls. Choose a representative sample of conversations.
- Have two evaluators score independently. Each evaluator scores the same conversation without seeing the other's scores.
- Compare scores. Identify where the evaluators disagree.
- Revise descriptors where scores diverge. Rewrite the descriptors that caused disagreement.
The testing console in evaluagent lets you try scoring changes against real conversations before they go live, showing AI-versus-human agreement on the scored conversations.
How to validate and adjust your scoring criteria
Validation requires ongoing calibration, inter-rater reliability checks, and descriptor revision. Calibration sessions can reveal where evaluators disagree, and those disagreements often point to descriptors that need rewriting.
Align evaluators through calibration
Calibration sessions align evaluators by having them score the same conversation and discuss differences. Select two or three conversations, have each reviewer score them independently before the meeting, open with the criteria where scores disagreed most, and end by rewriting whatever rubric wording caused the disagreement.
evaluagent supports four calibration types, including Check the Checker. evaluagent also includes a formal dispute route. Check the Checker enables evaluators to score blind, then compare against the calibrated score agreed in the session. Learn how to set up calibration on evaluated contacts or facilitate a calibration session.
Calculating inter-rater reliability scores
Inter-rater reliability measures how often two evaluators assign the same score to the same conversation. The simple version is percentage agreement, but that does not account for agreement occurring by chance. Cohen's κ is designed to measure the degree of agreement between raters beyond what is expected by chance, according to research on Cohen's kappa.
One widely cited set of kappa benchmarks (Landis and Koch) treats values above about 0.80 as almost perfect, 0.61 to 0.80 as substantial, 0.41 to 0.60 as moderate, and below 0.40 as fair to poor, according to Cogn-IQ's inter-rater reliability guide.
Measure agreement with two numbers: the percentage of criteria where reviewers scored identically, and the average score spread on each criterion.
Detecting alignment issues in rubrics
Alignment issues show up as consistent score divergence on specific line items. If evaluators agree on compliance but disagree on empathy, the empathy descriptors need rewriting.
SmartScore performance reports in evaluagent let you track AI-versus-human agreement over time, so you can spot when scoring consistency drifts.
Remove bias from scorecard terms
Bias enters through vague terms that let evaluators apply their own standards. As examples, vague terms can be replaced with observable behaviors: "professional" might become "used the customer's name and confirmed understanding," "helpful" might become "offered a specific next step," and "courteous" might become "acknowledged the customer's concern before offering a solution."
If the descriptors are clear and disagreement persists, review whether evaluators have received consistent training on the rubric.
Standardizing how evaluators score conversations
Standardization comes from the same rubric, the same calibration process, and the same evidence for every score.
evaluagent AI scores show the evidence and rationale behind them and can run hidden while the team calibrates. Learn how to track and prepare for calibration or use the performance report.
"EvaluAgent is very user-friendly and easy to navigate. It helps us track performance efficiently and provides clear feedback on quality scores. The evaluation criteria are well-structured, and it makes monitoring calls simple and transparent." - Mahesh B. on G2
Building defensible QA scores with observable rubrics
Observable descriptors and calibration together produce scores that survive agent disputes, compliance audits, and the move to automated QA. Descriptors give every evaluator the same reference point, and calibration finds the wording that still causes disagreement. Start with observable behaviors, validate through calibration, and choose a platform that preserves your rubric logic when you scale to 100% coverage.
Book a personalized demo and see AutoQA working on your own conversations.
FAQs
How is a rubric different from a scorecard?
A scorecard is built from line items, and each line item is scored with a rubric. The comparison table above shows how rubrics, scorecards, and checklists differ.
How many performance levels should a rubric have?
Many rubrics use three to five levels. More levels can give finer discrimination, but they typically make calibration harder.
Can rubrics be used for automated QA?
Yes. Rubrics with observable descriptors translate cleanly into automated QA. evaluagent scores against the same observable criteria your human evaluators would use, so the AI applies your rubric logic to every conversation.
How often should rubrics be updated?
A common practice is to review rubrics when procedures, regulations, or customer expectations change. Calibration sessions often reveal descriptors that need revision.
What makes a good rubric descriptor?
A good descriptor names the exact behavior the agent showed, such as "acknowledged the customer's frustration directly," not a vague term like "professional."
Key terms glossary
QA rubric: The scoring guide for a single behavior, defined in the introduction.
Automated QA (AutoQA): Quality assurance scoring performed by AI against the same observable criteria a human evaluator would use.
Scorecard: The evaluation that combines weighted line items into one score. See the comparison table above.
Checklist: The yes/no counterpart to a rubric. See the comparison table above.
Line item: A single scored criterion within a scorecard.
Observable descriptor: A description of what the agent said or did at a specific performance level, rather than what the evaluator felt.
Calibration: A process where evaluators score the same conversation and align on the correct score.
Inter-rater reliability: A measure of how often two evaluators assign the same score to the same conversation.