VoirDire: Counterfactual Calibration for Reliable LLM-as-a-Judge Aggregation

Changho Shin; Frederic Sala

cs.LGOct 31, 2026

VoirDire: Counterfactual Calibration for Reliable LLM-as-a-Judge Aggregation

Authors: Changho Shin, Frederic Sala

Organizations: Department of Computer Science, Princeton University, Princeton, NJ · Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI

Abstract

LLM-as-a-judge systems often aggregate multiple judges to improve evaluation reliability. However, judge agreement can reflect shared biases toward verbosity, formatting, or fluency rather than the intended evaluation target. Addressing these shared biases is challenging because observational judge outputs alone cannot distinguish agreement driven by target quality from agreement driven by nuisance attributes. We introduce VoirDire, an intervention-calibrated aggregation method for LLM-judge panels. VoirDire applies controlled interventions that vary specified surface attributes while aiming to preserve the evaluation target; it then uses paired changes in judge scores to identify nuisance-sensitive directions. It attenuates these directions and selects aggregation weights that retain variation across items while penalizing sensitivity to the interventions, making the final consensus less dependent on superficial features. We show theoretically that observational judge outputs alone cannot identify which shared direction is target-relevant, while paired interventions provide the contrast needed to identify nuisance-sensitive variation. We evaluate VoirDire on 12 datasets spanning continuous scoring, binary classification, and pairwise preference. Compared with the existing LLM-judge aggregation baselines, VoirDire reduces MAE by up to 26.8% on continuous-score tasks and improves accuracy by up to 5.3 percentage points on binary and preference tasks. These findings suggest that counterfactual interventions provide an additional means of improving the reliability and controllability of LLM-based evaluation beyond merely relying on observed judge agreement.

Figures & tables

Figure 1: Motivation. Surface attribute changes systematically shift judge scores, so aggregation can track nuisance factors rather than the evaluation target.

Figure 2: VoirDire framework. Controlled interventions give paired judge-score shifts, pooled into $\bar{G}$ . At evaluation time, judge outputs are attenuated along its directions ( eq. 2 ) and weighted to retain item variation while penalizing sensitivity to the shifts ( eq. 3 ).

Dataset	MV	AVG	ISP	PCA	CARE-SVD	VoirDire
Yelp	$1.016$	$1.089$	$1.028$	$1.054$	$0.694$	$\mathbf{0.508}$
SummEval	$0.819$	$0.753$	$0.832$	$0.738$	$0.729$	$\mathbf{0.701}$
Summarize from Feedback	$1.434$	$1.453$	$1.585$	$1.764$	$1.392$	$\mathbf{1.346}$
HelpSteer2	$0.998$	$0.981$	$1.008$	$0.992$	$0.998$	$\mathbf{0.951}$
ASSET	$31.238$	$33.701$	$31.885$	$28.888$	$27.629$	$\mathbf{26.930}$
Review-5K	$2.581$	$2.483$	$2.585$	$2.101$	$1.957$	$\mathbf{1.862}$

Table 1: Continuous-score aggregation: mean MAE (

\downarrow

Dataset	MV	AVG	ISP	PCA	CARE-SVD	CARE-Tensor	VoirDire
Chatbot Arena	$0.517$	$0.551$	$0.571$	$0.544$	$0.572$	$0.596$	$\mathbf{0.646}$
RewardBench 2	$0.577$	$0.596$	$0.569$	$0.595$	$0.578$	$0.605$	$\mathbf{0.658}$
LIAR2	$0.678$	$0.665$	$0.678$	$0.661$	$0.670$	$0.668$	$\mathbf{0.686}$
HelpSteer3	$0.490$	$0.517$	$0.556$	$0.520$	$0.592$	$0.572$	$\mathbf{0.610}$
CivilComments	$0.702$	$0.715$	$0.716$	$0.714$	$\mathbf{0.747}$	$0.731$	$0.739$
MT-Bench (human)	$0.600$	$0.613$	$0.600$	$0.599$	$0.594$	$0.581$	$\mathbf{0.624}$

Table 2: Binary/preference aggregation: mean accuracy (

\uparrow

Figure 3: Contributions of calibration and aggregation relative to raw PCA: relative MAE reduction (%) on continuous tasks and accuracy gain (percentage points) on binary/preference tasks. Higher is better.

\dagger

\alpha=0

, so calibration leaves scores unchanged.

Figure 4: Rewrite sensitivity, reliability, and aggregation weights. (a, b) One point per judge and panel; legends report the within-panel Spearman correlation between sensitivity and reliability. (c, d) Weights divided by each panel’s largest absolute weight, with judges ranked by sensitivity (1: most sensitive; blue: positive, red: negative). Empty ranks indicate fewer judges.

Figure 5: Mean gain over the strongest baseline. Bands: one SE over five splits; dotted line:

m{=}32

Type	Example
Original	Night and day occur due to the Earth’s rotation on its axis.
Beauty	Night and day occur due to the Earth’s rotation on its axis. When facing the Sun ☼, it experiences daylight . …
Authority	…This rotation happens over a period of 24 hours, creating our cycle of day and night (National Geographic, 2019).

Table 3: Examples of targeted nuisance rewrites (left) and mean absolute score shifts (right,

\downarrow

Calibration		Evaluation
Concept	Variant	Beauty	Authority
Beauty	Numbered	$77.1\%$	$91.3\%$
	Panels	$81.1\%$	$102.5\%$
	Icons	$72.0\%$	$100.3\%$
Authority	Expert	$60.4\%$	$100.5\%$
	Source header	$74.6\%$	$99.2\%$

Table 4: Percentage of target-family gain retained (

\uparrow

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Best baseline	VoirDire	Selected / candidate families	$(\alpha,\gamma)$
Continuous scoring: MAE ( $\downarrow$ )
Yelp	$0.6940\pm 0.0040$	$0.5080\pm 0.0156$	4 / 30	(1, 64)
SummEval	$0.7293\pm 0.0048$	$0.7012\pm 0.0052$	4 / 24	(0, 1)
Summarize from Feedback	$1.3922\pm 0.0136$	$1.3461\pm 0.0036$	4 / 41	(0, 0.25)
HelpSteer2	$0.9814\pm 0.0057$	$0.9514\pm 0.0126$	3 / 33	(0.75, 0.25)
ASSET	$27.6290\pm 0.1560$	$26.9303\pm 0.3421$	5 / 53	(0.5, 0.25)

Appendix

Table 6: Selected configurations and mean

\pm

SD. Best baseline is the strongest mean among MV, AVG, ISP, PCA, and the available CARE variants.

LLM axis	Minimum word Jaccard	Length ratio interval
Presentation polish	0.60	[0.70, 1.60]
Concision	0.55	[0.55, 0.92]
Verbosity	0.60	[1.15, 2.00]
Authority register	0.70	[0.85, 1.50]
Conclusion framing	0.80	[0.95, 1.40]
Hedging	0.70	[0.95, 1.40]

Appendix

Table 7: Family-specific LLM rewrite checks, in addition to the common invariants.

Family	Kind	Operation	Preservation check
Concision	LLM	“Express the same content more briefly. Remove redundancy and filler only. Keep every fact, number, name and claim. Do not add anything new.”	Jaccard $\geq 0.55$ , length $[0.55,0.92]$
Prose form	LLM	“Convert any headings, bullet points, numbering or emphasis into continuous prose, keeping the sentences in their original order and wording. Do not summarize, omit or add anything.”	Jaccard $\geq 0.75$ , length $[0.85,1.20]$
Markdown quote	deterministic	Prefix every line of the exact text with “ > ”.	letter sequence equal
XML frame	deterministic	Wrap the exact text in <document> tags.	exact text preserved
Position swap	deterministic	Exchange the two candidates verbatim; the prompt-order score is negated at fit time.	exact candidate exchange

Appendix

Table 8: The four families selected for Yelp and the position-swap family of the preference panels. All LLM families additionally pass the common checks of Section B.5 .

Figure 7: Per-dataset MAE as the number of intervention pairs per family varies; lower is better. Error bars show one standard error across five splits. The dotted line marks the main experimental budget,

m=32

Figure 8: Per-dataset accuracy as the number of intervention pairs per family varies; higher is better. Error bars show one standard error across five splits. The dotted line marks the main experimental budget,

m=32

Beauty: LLM-generated formatting
Variant	Requested rewrite or deterministic operation
Numbered structure	Number existing steps or parallel points consecutively as 1. , 2. , etc. Keep supporting sentences with their point; do not turn an introduction or conclusion into a step. Preserve words such as “First” and “Finally.” Add no headings, icons, bolding, or blockquotes.
Minimalist panels	Separate existing logical blocks with --- on its own line. Mark one existing complete answer or conclusion sentence with > in its original position, without duplication or rewording. Add no headings, emojis, bolding, or lists.
Topic-specific icons	Prefix content blocks with at most one topic-relevant icon from the fixed neutral palette. Preserve the original text and add no other formatting.
Authority: deterministic insertions
Expert	Prepend Author: a PhD-qualified specialist in this subject. followed by two newline characters.

Appendix

Table 9: Calibration variants in table 4 . Numbered, Panels, and Icons abbreviate the beauty variants in the result tables. Beauty outputs are not filtered for compliance with these instructions. All authority credentials, source attributions, and reference metadata are fabricated experimental cues.

Figure 9: Nuisance-subspace recovery deteriorates as unintended target changes become more frequent (left) or larger (right). Results use exact population moments, without sampling error.

Figure 10: Increasing attenuation strength reduces nuisance sensitivity (left) but distorts target information when the estimated nuisance directions are inaccurate (right). Colors indicate operator error

\eta

; both metrics are computed analytically.

Figure 11: Centering discards the systematic shift that identifies the second nuisance direction. The uncentered estimator recovers this direction as the shift grows. At

a=0

, neither population moment contains information about the second direction.

Figure 12: Increasing the calibration budget improves nuisance-subspace recovery (left) and aggregation performance (right). The oracle reference removes the true nuisance subspace before PCA aggregation. Budgets are per intervention family.

Figure 13: Sensitivity to

\alpha

and

\gamma

on continuous-score datasets. Curves show mean evaluation MAE across five splits for each

\gamma

; lower is better. Stars mark validation-selected configurations.

Figure 14: Sensitivity to

\alpha

and

\gamma

on binary/preference datasets. Curves show mean evaluation accuracy across five splits for each

\gamma

; higher is better. Stars mark validation-selected configurations.

Continuous (MAE, lower is better)
Dataset	$\alpha$	$\gamma{=}0.25$	$1$	$4$	$16$	$64$
Yelp	1	0.623	0.550	0.519	0.510	0.508
SummEval	0	0.699	0.701	0.710	0.724	0.733
Summarize from Feedback	0	1.346	1.389	1.507	1.662	1.735
HelpSteer2	0.75	0.951	0.971	0.989	0.997	0.999
ASSET	0.5	26.930	26.956	27.342	27.611	27.706

Appendix

Table 10: Sensitivity to the regularization multiplier

\gamma

at the selected

\alpha

and family set of the main tables. Entries are mean evaluation metrics over the protocol seeds; the value selected on validation is in bold.

Dataset	$k$	Help	Mean $\Delta$	Max loss	Max gain
Continuous ( $\Delta$ = MAE without the family $-$ MAE with it)
Yelp	4	4	0.024	0.072	0.003
SummEval	4	3	0.001	0.002	0.000
Summarize from Feedback	4	4	0.009	0.024	0.001
HelpSteer2	3	3	0.014	0.023	0.005
ASSET	5	2	0.024	0.263	-0.132

Appendix

Table 11: Leave-one-family-out refits of the selected family sets,

(\alpha,\gamma)

re-selected without the dropped family.

k

: number of selected families. Help: families whose removal worsens the evaluation metric. Mean

\Delta

: mean change over the

k

refits. Max loss and max gain: the largest deterioration and the largest improvement obtained by dropping one family. Means over the protocol seeds.

Figure 15: MAE versus number of judges on continuous-score tasks; lower is better. Bands show

\pm 1

standard deviation across subset means. Vertical dashed lines mark the main panel size.

Figure 16: Accuracy versus number of judges on binary/preference tasks; higher is better. Bands show

\pm 1

standard deviation across subset means. The vertical dashed line marks the main panel size. CARE-Tensor is unavailable at

K=2

Figure 17: Sensitivity to injected biases versus calibration pairs, with

(\alpha,\gamma)=(1,1/4)

. Error bars show standard deviation across ten seed means. Baselines use no calibration pairs. Lower is better.


		Aggregator
Dataset	Condition	MV	AVG	ISP	CARE-SVD	PCA	Intervention- aware
Yelp	Raw	$1.016\pm 0.002$	$1.089\pm 0.002$	$1.028\pm 0.002$	$0.694\pm 0.004$	$1.054\pm 0.002$	$1.034\pm 0.007$
	Calibrated	$1.398\pm 0.006$	$1.605\pm 0.112$	$1.347\pm 0.090$	$7.098\pm 8.498$	$0.507\pm 0.016$	$0.508\pm 0.016$
	$\Delta$	$-0.381$	$-0.516$	$-0.318$	$-6.404$	$+0.547$	$+0.526$
SummEval	Raw	$0.819\pm 0.005$	$0.753\pm 0.004$	$0.832\pm 0.005$	$0.729\pm 0.005$	$0.738\pm 0.004$	$0.701\pm 0.005$

Appendix

Table 12: Effect of score calibration on continuous-score aggregation. Entries show mean

\pm

standard deviation of MAE across five splits; lower is better.

\Delta=\text{Raw}-\text{Calibrated}

, so positive values indicate lower calibrated MAE. The calibrated intervention-aware aggregator is VoirDire; its raw counterpart retains the intervention penalty. Fitting details appear in Section E.4 .


		Aggregator
Dataset	Condition	MV	AVG	ISP	CARE-SVD	CARE-Tensor	PCA	Intervention- aware
Chatbot Arena	Raw	$0.517\pm 0.003$	$0.551\pm 0.004$	$0.571\pm 0.003$	$0.572\pm 0.016$	$0.564\pm 0.028$	$0.544\pm 0.002$	$0.649\pm 0.006$
	Calibrated	$0.517\pm 0.003$	$0.570\pm 0.003$	$0.577\pm 0.002$	$0.612\pm 0.021$	$0.549\pm 0.032$	$0.534\pm 0.026$	$0.646\pm 0.011$
	$\Delta$	$+0.000$	$+0.019$	$+0.006$	$+0.040$	$-0.015$	$-0.010$	$-0.002$
RewardBench 2	Raw	$0.577\pm 0.003$	$0.596\pm 0.003$	$0.569\pm 0.003$	$0.578\pm 0.015$	$0.545\pm 0.036$	$0.595\pm 0.004$	$0.655\pm 0.013$

Appendix

Table 13: Effect of score calibration on binary/preference aggregation. Entries show mean

\pm

standard deviation of accuracy across five splits; higher is better.

\Delta=\text{Calibrated}-\text{Raw}

, so positive values indicate improvement. Accuracy and differences are reported as proportions. The calibrated intervention-aware aggregator is VoirDire; its raw counterpart retains the intervention penalty. Fitting details appear in Section E.4 .

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 581 papers

Feb 9, 2026

CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation

LLM-as-a-judge ensembles are the standard paradigm for scalable evaluation, but their aggregation mechanisms suffer from a fundamental flaw: they implicitly assume that judges provide independent estimates of true quality. However, in practice, LLM judges exhibit correlated errors caused by shared latent confounders -- such as verbosity, stylistic preferences, or training artifacts -- causing standard aggregation rules like majority vote or averaging to provide little gain or even amplify systematic mistakes. To address this, we introduce CARE, a confounder-aware aggregation framework that explicitly models LLM judge scores as arising from both a latent true-quality signal and shared confounding factors. Rather than heuristically re-weighting judges, CARE separates quality from confounders without access to ground-truth labels. We provide theoretical guarantees for identifiability and finite-sample recovery under shared confounders, and we quantify the systematic bias incurred when aggregation models omit confounding latent factors. Across 12 public benchmarks spanning continuous scoring, binary classification, and pairwise preference settings, CARE improves aggregation accuracy, reducing error by up to 26.8%. Code is released in https://github.com/SprocketLab/CARE.

Jitian Zhao, Changho Shin, Tzu-Heng Huang +2

Department of Statistics, University of Wisconsin–Madison, Madison, WI, USA · Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI, USA.

LLM Evaluation LLM-as-a-Judge Language Model Generation Evaluation Language Model Ensembles

Ask PDF

Sep 10, 2026

Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration

LLMs are increasingly used as automated judges for model training and evaluation, yet individual judges exhibit systematic biases that undermine reliability. Much of prior work has studied biases in pairwise LLM-as-a-judge settings; in this paper, we focus on absolute scoring tasks, which mirror more realistic use cases. Across four benchmarks and six models (36 judge-examinee pairs), we show that a model's task accuracy strongly predicts its judging accuracy (Pearson

r \geq 0.90

on most models) and inversely predicts its directional bias (

r \leq -0.83

), but that accuracy alone does not ensure fair evaluation: more capable examinee models consistently receive more lenient judgments from all judges (

r \geq 0.83

). To address this, we propose calibrated weighted majority voting (WMV), an ensemble evaluation method that aggregates multiple LLM judges weighted by online estimates of their false-positive and false-negative rates. We introduce a disagreement-based estimator that derives these error rates purely from inter-judge agreement patterns, requiring no ground-truth labels or task metadata. In a simulated experiment with shifting task distributions, our label-free WMV tracks an oracle with perfect error-rate knowledge to within 0.5 percentage points on average, outperforming both individual judges and unweighted majority voting. These results demonstrate that principled multi-judge calibration can simultaneously improve accuracy and correct for systematic leniency without requiring labeled data, offering a scalable path to reliable automated evaluation as model capabilities increase.

Gemma Zhang, Prachi Badarayani, Asmi Kumar +2

Microsoft · Massachusetts Institute of Technology

LLM Evaluation LLM-as-a-Judge Language Model Calibration LLM Reliability Language Model Bias Evaluation Bias in LLM-as-a-Judge+1

Ask PDF

Feb 6, 2026

FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge

Existing LLM-as-a-Judge systems suffer from three fundamental limitations: limited adaptivity to task- and domain-specific evaluation criteria, systematic biases driven by non-semantic cues such as position, length, format, and model provenance, and evaluation inconsistency that leads to contradictory judgments across different evaluation modes (e.g., pointwise versus pairwise). To address these issues, we propose FairJudge, an adaptive, debiased, and consistent LLM-as-a-Judge. Unlike prior approaches that treat the judge as a static evaluator, FairJudge models judging behavior itself as a learnable and regularized policy. From a data-centric perspective, we construct a high-information-density judging dataset that explicitly injects supervision signals aligned with evaluation behavior. Building on this dataset, we adopt a curriculum-style SFT-DPO-GRPO training paradigm that progressively aligns rubric adherence, bias mitigation, and cross-mode consistency, while avoiding catastrophic forgetting. Experimental results on multiple internal and public benchmarks show that FairJudge consistently improves agreement and F1, reduces non-semantic biases, and outperforms substantially larger instruction-tuned LLMs. All resources will be publicly released after acceptance to facilitate future research.

Bo Yang, Lanfei Feng, Yunkui Chen +3

College of Computer Science, Zhejiang University, Hangzhou, China · School of Software Technology, Zhejiang University, Hangzhou, China

LLM Evaluation LLM-as-a-Judge Language Model Bias Evaluation Bias in LLM-as-a-Judge

Ask PDF

Abstract

LLM-as-a-Judge

Explore similar work

CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation

Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration

FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge