cs.CLFeb 16, 2026
SaveCounterfactual Fairness Evaluation of LLM-Based Contact Center Agent Quality Assurance System
Organizations: Observe.AI Bangalore, India
Abstract
Large Language Models (LLMs) are increasingly deployed in contact-center Quality Assurance (QA) to automate agent performance evaluation and coaching feedback. While LLMs offer unprecedented scalability and speed, their reliance on web-scale training data raises concerns regarding demographic and behavioral biases that may distort workforce assessment. We present a counterfactual fairness evaluation of LLM-based QA systems across 13 dimensions spanning three categories: Identity, Context, and Behavioral Style. Fairness is quantified using the Counterfactual Flip Rate (CFR), the frequency of binary judgment reversals, and the Mean Absolute Score Difference (MASD), the average shift in coaching or confidence scores across counterfactual pairs. Evaluating 18 LLMs on 3,000 real-world contact center transcripts, we find systematic disparities, with CFR ranging from 5.4% to 13.0% and consistent MASD shifts across confidence, positive, and improvement scores. Larger, more strongly aligned models show lower unfairness, though fairness does not track accuracy. Contextual priming of historical performance induces the most severe degradations (CFR up to 16.4%), while implicit linguistic identity cues remain a persistent bias source. Finally, we analyze the efficacy of fairness-aware prompting, finding that explicit instructions yield only modest improvements in evaluative consistency. Our findings underscore the need for standardized fairness auditing pipelines prior to deploying LLMs in high-stakes workforce evaluation.
Figures & tables
Figure 1: Example illustrating gender-based disparity in Auto-QA responses. Conversation with identical content but different agent names (female left, male right) yield opposite judgments from the LLM.
| Fairness Dimension | Description |
| 1. Agent Identity-Based Fairness (Inferred Attributes) | |
| Agent Gender | Tests whether changing the agent’s name and pronouns (e.g., “he”→“she”) alters the model’s judgment. Any difference indicates sensitivity to perceived gender rather than conversational content. |
| Agent Ethnicity | Evaluates if substituting culturally coded names or dialect markers leads to different QA outcomes. Such dependence reveals racial or cultural bias in implicit name-based inference. |
| Agent Religion | Assesses whether inserting or swapping religiously identifiable names or benign faith references (e.g., “Inshallah”, “God bless”) influences evaluation. |
| Agent Disability | Examines if adding neutral mentions of assistive technology or speech characteristics (e.g., stutter, screen reader) changes scoring, revealing potential discrimination against disability indicators. |
| 2. Contextual & Historical Fairness (Extrinsic Anchoring) | |
Table 1: Taxonomy of counterfactual fairness dimensions used in the Auto-QA evaluation.
Figure 2: Overview of the proposed fairness evaluation approach. (Left) Original transcripts are transformed into counterfactual variants by altering demographic, contextual, or linguistic attributes while preserving meaning. (Right) Each variant is evaluated by the target LLM to generate Auto-QA outputs, which are aggregated to compute fairness metrics and identify systematic disparities.
| Metric / Bias | llama-3.2-3b | llama-3.3-70b | llama-4-maverick-17b | claude-3.5-haiku | claude-4-sonnet | nova-micro | nova-lite | nova-pro | nova-premier | gpt-4o-mini | gpt-4o | gpt-5-nano-low | gpt-5-mini-low | gpt-5-low | gpt-5-nano-medium | gpt-5-mini-medium | gpt-5-medium | deepseek-r1 | Average |
| Counterfactual Flip Rate (CFR) ( better) | |||||||||||||||||||
| Agent Gender | 13.02 | 5.59 | 3.49 | 9.63 | 3.43 | 5.37 | 8.97 | 6.81 | 4.19 | 8.95 | 7.64 | 6.19 | 5.26 | 5.81 | 4.37 | 6.51 | 3.60 | 11.76 | 6.78 |
| Agent Ethnicity (name-only) | 9.96 | 5.73 | 3.47 | 6.89 | 1.93 | 5.40 | 8.56 | 7.31 | 3.14 | 7.38 | 7.66 | 6.86 | 5.90 | 4.90 | 4.88 | 5.73 | 2.22 | 10.38 | 6.02 |
| Agent Ethnicity (with-cues) | 10.31 | 5.92 | 10.40 | 11.84 | 7.34 | 16.05 | 9.27 | 11.55 | 5.74 | 8.33 | 6.58 | 9.26 | 6.74 | 9.38 | 10.59 | 8.88 | 8.02 | 7.58 | 9.71 |
| Agent Religion (name-only) | 11.75 | 5.62 | 1.49 | 8.26 | 3.80 | 5.97 | 10.58 | 6.63 | 2.64 | 6.12 | 6.61 | 5.91 | 4.96 | 4.96 | 3.68 | 6.61 | 3.16 | 11.39 | 6.25 |
| Agent Religion (with-cues) | 9.77 | 4.93 | 9.86 | 12.14 | 5.48 | 16.17 | 8.88 | 10.17 | 5.62 | 7.88 | 6.58 | 9.40 | 6.16 | 8.81 | 10.76 | 8.77 | 8.49 | 8.51 | 9.24 |
Table 2: Model Fairness and Robustness Scores Across 13 Fairness Dimensions. This table summarizes the performance of 18 LLMs on our fairness benchmarks. We report the Counterfactual Flip Rate (CFR) ( – , better), the Mean Absolute Score Difference (MASD) ( – , better), and Answer Accuracy % ( – , better). We highlight the best scores in green and worst scores in red for each row.
Figure 3: Tradeoff between fairness and accuracy.
Figure 4: Fairness–robustness contrast across bias dimensions.
Figure 5: Fairness–robustness contrast across models.
| Model | CFR (% ↓ better) | Conf. MASD | Pos. MASD | Improv. MASD |
| claude-3.5-haiku | 0.26 | -2.13 | -0.66 | -1.94 |
| claude-4-sonnet | -2.47 | -3.89 | -1.77 | -5.14 |
| llama-3.2-3b | -1.27 | -2.88 | -3.61 | -3.92 |
| llama-4-maverick | -4.23 | -5.21 | -2.45 | -5.37 |
| nova-lite | -3.85 | -1.97 | -2.89 | -3.18 |
| nova-pro | -3.65 | -4.31 | -2.12 | -5.64 |
Table 3: Changes in CFR and MASD under fairness prompting. Negative values indicate reduced disparity.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Bias Dimension | Attributes Used for Counterfactual Variation |
| Agent Gender | Male, Female |
| Agent Ethnicity | Hispanic, Black, White, Native American, Asian (East), Asian (South) |
| Agent Religion | Hinduism, Christianity, Islam, Judaism, Buddhism |
| Agent Disability | Normal, Disabled |
| Past Performance | Improving Performance; Worsening Performance; Stable Excellence; No Improvement – Medium Performance; No Improvement - Poor Performance |
| Agent Profile | [Agent Role: Trainee, Agent Tenure: 2 weeks]; [Agent Role: Customer Service Professional, Agent Tenure: 2 years]; [Agent Role: Senior Advisor, Agent Tenure: 7 years]; [Agent Role: Senior Escalation Specialist, Agent Tenure: 4 years] |
Table 4: Bias dimensions and attribute sets for controlled counterfactual manipulation in Auto-QA fairness evaluation, where each set defines a distinct condition for evaluating identical call content.
| Fairness Dimension | Why It Matters for Quality Assurance |
| 1. Agent Identity-Based Bias (Inferred Attributes) | |
| Agent Gender | Models may penalize or reward communication styles stereotypically associated with a gender (e.g., penalizing female agents for directness or male agents for showing vulnerability). Such distortions affect perceived professionalism, empathy scores, and downstream coaching recommendations. |
| Agent Ethnicity | Names, dialects, or idioms linked to specific ethnic groups can trigger stereotypes about competence or temperament. This biases fairness audits, skews QA pass rates, and misguides DEI reporting. |
| Agent Religion | Religiously coded names or benign expressions may be misinterpreted as unprofessional, unfairly reducing empathy or compliance scores, and penalizing cultural expression. |
| Agent Disability | Indicators of disability (e.g., stutter, verbal tic, assistive tech mention) can lower clarity or efficiency ratings, despite no real performance deficit—creating discriminatory impact in QA and HR metrics. |
| 2. Contextual & Historical Bias (Extrinsic Anchoring) | |
Table 5: Agent QA bias dimensions and their implications for fairness, business KPIs, and operational decision-making in automated performance evaluation.
| Statistic | Turns | Tokens | Duration (mm:ss) |
| Mean | 196.2 | 2643.1 | 19:55 |
| Std | 106.8 | 1041.8 | 9:38 |
| Min | 38.0 | 615.0 | 5:48 |
| 25% | 127.0 | 1879.0 | 13:09 |
| 50% | 175.0 | 2522.0 | 17:55 |
| 75% | 238.0 | 3060.0 | 24:13 |
Table 6: Summary statistics of the real transcripts used in our evaluation.
| Statistic | Turns | Tokens |
| Mean | 35.7 | 654.5 |
| Std | 5.9 | 124.5 |
| Min | 26.0 | 348.0 |
| 25% | 31.0 | 570.0 |
| 50% | 36.0 | 648.0 |
| 75% | 41.0 | 729.0 |
Table 7: Summary statistics of the synthetic transcripts used in our evaluation.
| Parameter | Value |
| Temperature | 0 (1 for GPT-5 family) |
| Top-p | 1.0 |
| Max Tokens | 1000 |
| Frequency Penalty | 0.0 |
| Presence Penalty | 0.0 |
| Stop | None |
Table 8: LLM generation parameters.
| Dimension | Real Transcripts | Synthetic Transcripts | ||
| LLM Rejection (%) | Human Rejection (%) | LLM Rejection (%) | Human Rejection (%) | |
| Agent Gender | 0.5 | 2.0 | 0.2 | 2.5 |
| Agent Ethnicity (cues) | 16.0 | 2 | 5 | 1.50 |
| Agent Ethnicity (name) | 0.0 | 0.0 | 0.0 | 0.0 |
| Agent Religion (cues) | 20.0 | 2.4 | 9.8 | 2.00 |
| Agent Religion (name) | 0.0 | 0.0 | 0.0 | 0.0 |
Table 9: LLM and human rejection rates across bias dimensions for real and synthetic transcripts.
| Metric / Bias | llama-3.2-3b | llama-4-maverick-17b | claude-3.5-haiku | claude-4-sonnet | nova-lite | nova-pro | gpt-5-mini-low | gpt-5-low | Avg. |
| Counterfactual Flip Rate (CFR) ( better) | |||||||||
| Agent Gender | 8.44 | 4.17 | 3.75 | 9.58 | 4.17 | 6.25 | 9.58 | 5.42 | 6.42 |
| Agent Ethnicity (cues) | 4.03 | 4.00 | 7.83 | 13.00 | 4.83 | 4.33 | 11.50 | 11.67 | 7.65 |
| Agent Ethnicity (name) | 7.83 | 3.61 | 4.10 | 6.83 | 4.17 | 6.19 | 7.90 | 5.24 | 5.73 |
| Agent Religion (cues) | 6.41 | 2.50 | 6.50 | 6.50 | 3.50 | 2.50 | 12.00 | 11.00 | 6.36 |
| Agent Religion (name) | 6.99 | 3.06 | 3.08 | 6.36 | 4.61 | 6.17 | 7.03 | 5.69 | 5.37 |
Table 10: Model fairness and robustness across 13 bias dimensions on synthetic data. We report Counterfactual Flip Rate (CFR), Mean Absolute Score Difference (MASD), and Answer Accuracy (%). Lower is better for CFR and MASD, higher is better for accuracy. Best and worst scores per row are highlighted.
| Metric / Bias | llama-3.2-3b | llama-4-maverick-17b | claude-3.5-haiku | claude-4-sonnet | nova-lite | nova-pro | gpt-5-mini-low | gpt-5-low | Avg. |
| Positive Score (MASD) ( better) | |||||||||
| Agent Gender | 2.93 | 2.68 | 1.95 | 4.76 | 2.62 | 2.02 | 5.54 | 7.19 | 3.71 |
| Agent Ethnicity (cues) | 3.57 | 3.13 | 3.11 | 5.16 | 2.98 | 2.66 | 6.28 | 7.86 | 4.34 |
| Agent Ethnicity (name) | 2.87 | 2.53 | 2.41 | 4.26 | 2.38 | 2.06 | 5.28 | 6.76 | 3.57 |
| Agent Religion (cues) | 3.22 | 2.83 | 3.01 | 5.68 | 3.00 | 2.56 | 6.27 | 7.32 | 4.24 |
| Agent Religion (name) | 2.62 | 2.13 | 2.41 | 4.88 | 2.40 | 1.96 | 5.37 | 6.32 | 3.51 |
Table 11: Model fairness and robustness across 13 bias dimensions on synthetic data. We report Counterfactual Flip Rate (CFR), Mean Absolute Score Difference (MASD), and Answer Accuracy (%). Lower is better for CFR and MASD, higher is better for accuracy. Best and worst scores per row are highlighted.
Figure 6: Cohen’s Kappa agreement for QA labels between runs.
Figure 7: Distribution of label flips in QA data.
Figure 8: Mean Confidence of QA Evaluation across models.
Figure 9: Mean Scores of Improvement notes across models.
Figure 10: Mean Scores of Customer Behavior notes across models.
Figure 11: Mean Scores of Positive notes across models.
More in this topic
Language Model Bias Evaluation
Language model bias evaluation tests how models’ outputs, decisions, and representations vary across social groups, cultures, languages, and contexts.
See all 492 papers