A Statistical Framework for Auditing Behavioral Dependence and Induced Bias in LLM Judges

Chenchen Kuai; Jiwan Jiang; Zihao Zhu; Hao Wang; Keshu Wu; Zihao Li; Yunlong Zhang; Chenxi Liu

cs.AIApr 8, 2026

A Statistical Framework for Auditing Behavioral Dependence and Induced Bias in LLM Judges

Authors: Chenchen Kuai, Jiwan Jiang, Zihao Zhu, Hao Wang, Keshu Wu, Zihao Li, Yunlong Zhang, Chenxi Liu,

Organizations: Texas A&M University · Marquette University · University of Utah

Abstract

The rapid growth of the large language model (LLM) ecosystem raises a critical question: are seemingly diverse models truly independent? Shared pretraining data, distillation, and alignment pipelines can induce hidden behavioral dependencies, or latent entanglement, that undermine multi-model systems such as LLM-as-a-judge pipelines and ensemble verification, which implicitly assume independent signals. In practice, this manifests as correlated reasoning patterns and synchronized failures, where apparent agreement reflects shared error modes rather than independent validation. To address this, we develop a statistical framework for auditing behavioral entanglement among black-box LLMs. Our approach introduces a multi-resolution hierarchy that characterizes the joint failure manifold through two information-theoretic metrics: (i) a Difficulty-Weighted Behavioral Entanglement Index (BEI), which amplifies synchronized failures on easy tasks, and (ii) a Cumulative Information Gain (CIG) metric, which captures directional alignment in erroneous responses. Through experiments on 18 LLMs from six model families, we identify statistically significant behavioral entanglement. Such behavioral dependence is associated with judge over-endorsement bias on a disjoint MMLU-Pro evaluation set (rho = 0.508 for BEI and rho = 0.520 for CIG; p < 0.01). The association further transfers to the MATH-500 benchmark (rho = 0.441 for BEI and rho = 0.457 for CIG; p < 0.05), providing cross-benchmark evidence that the identified dependency structure generalizes beyond the data and response format used for its estimation. Finally, we demonstrate a practical use case of entanglement through de-entangled verifier ensemble reweighting, achieving 3.5 and 2.6 percentage-point gains in accuracy and precision, respectively, over majority voting.

Figures & tables

Figure 1: A Multi-Resolution Entanglement Hierarchy for Behavioral Dependence

Figure 2: Behavioral entanglement graph of LLMs on MMLU-Pro: (a) co-failure entanglement (BEI); and (b) directional entanglement (CIG). Reported are only statistically significant BEI and CIG values and FDR corrected relationships (q-value $<0.05$ ).

Co-failure audit
Metric	MMLU-pro		MATH-500
Metric	$\rho$	$p$ -value	$\rho$	$p$ -value
Corr.	0.063	0.724	0.187	0.360
MI	0.113	0.525	0.208	0.309
Jaccard	0.428	0.012 ∗	0.275	0.175
BEI	0.508	0.002 ∗∗	0.441	0.024 ∗

Table 1: Metrics and their Spearman correlations with

\Delta{\mathrm{Prec}}

on MMLU-pro and MATH-500.

$w$	MMLU-pro $\rho$	MATH-500 $\rho$
0	0.289	0.032
1	0.508 ∗∗	0.441 ∗
2	0.490 ∗∗	0.275
3	0.504 ∗∗	0.231
4	0.465 ∗∗	0.236
5	0.363 ∗	0.180

Table 2: Sensitivity of BEI correlations to the difficulty-weighting exponent

w

Aggregation	Acc	F1	Precision	$\Delta$ Acc
Majority Vote	0.847	0.901	0.870	–
Accuracy-based Reweight	0.881	0.919	0.891	+0.034
Entangle-based Reweight	0.882	0.922	0.896	+0.035

Table 3: Verifier aggregation performance on MMLU-Pro.

Figure 3: Calibrated verifier ensemble weights.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Mean	Std.	Range
AUC	0.865	0.050	[0.752, 0.934]
ECE	0.041	0.012	[0.022, 0.077]
Brier	0.130	0.039	[0.065, 0.192]

Appendix

Table C.1: Calibration diagnostics for the difficulty-response models used in BEI.

Figure C.1: Reliability curve of the difficulty-response models used for BEI calibration.

Model 1	Model 2	BEI	$p$ -value	$q$ -value
Llama-3-70B	Llama-3_1-70B	0.0525	5.00E-05	9.50E-04
Llama-2-70b-hf	Qwen1.5-110B	0.0398	5.00E-05	9.50E-04
Llama-3-70B	Qwen1.5-110B	0.0352	5.00E-05	9.50E-04
Qwen1.5-14B-Chat	Qwen1.5-72B-Chat	0.0333	5.00E-05	9.50E-04
Llama-2-70b-hf	Llama-3-70B	0.0326	5.00E-05	9.50E-04
Llama-3_1-70B	Qwen1.5-110B	0.0325	5.00E-05	9.50E-04

Appendix

Table C.2: Top 10 pairwise BEI values with raw

p

-values and FDR-adjusted

q

-values.

Model 1	Model 2	CIG	$p$ -value	$q$ -value
Llama-3-70B	Llama-3_1-70B	0.1332	5.00E-05	1.09E-03
Claude-3-5-sonnet-20241022	GPT-4o-2024-08-06	0.0525	5.00E-05	1.09E-03
GPT-4o-2024-08-06	GPT-4o-mini	0.0523	5.00E-05	1.09E-03
DeepSeek-chat-v2_5	Gemini-1.5-flash-002	0.0502	5.00E-05	1.09E-03
Claude-4-6-Sonnet	Gpt-5	0.0471	5.00E-05	1.09E-03
Gemini-1.5-flash-002	Gemini-1.5-pro-002	0.0454	5.00E-05	1.09E-03

Appendix

Table C.3: Top pairwise CIG values with raw

p

-values and FDR-adjusted

q

-values.

Target	Test	Null hypothesis	$p$ -value	Interpretation
$\Delta\mathrm{Prec}$	Paired $t$ -test	$\mathbb{E}[p_{t}(V_{2})-p_{t}(V_{1})]=0$	0.421	No mean shift
Ranking	Equivalence test	$\rho\leq 1-\epsilon,\ \epsilon=0.05$	$7.6\times 10^{-5}$	Ranking consistent

Appendix

Table C.4: Joint-prompt versus independent-prompt evaluation ablation.

Correlated Errors in Language Models

See all 8 papers

Sep 10, 2026

Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration

LLMs are increasingly used as automated judges for model training and evaluation, yet individual judges exhibit systematic biases that undermine reliability. Much of prior work has studied biases in pairwise LLM-as-a-judge settings; in this paper, we focus on absolute scoring tasks, which mirror more realistic use cases. Across four benchmarks and six models (36 judge-examinee pairs), we show that a model's task accuracy strongly predicts its judging accuracy (Pearson

r \geq 0.90

on most models) and inversely predicts its directional bias (

r \leq -0.83

), but that accuracy alone does not ensure fair evaluation: more capable examinee models consistently receive more lenient judgments from all judges (

r \geq 0.83

). To address this, we propose calibrated weighted majority voting (WMV), an ensemble evaluation method that aggregates multiple LLM judges weighted by online estimates of their false-positive and false-negative rates. We introduce a disagreement-based estimator that derives these error rates purely from inter-judge agreement patterns, requiring no ground-truth labels or task metadata. In a simulated experiment with shifting task distributions, our label-free WMV tracks an oracle with perfect error-rate knowledge to within 0.5 percentage points on average, outperforming both individual judges and unweighted majority voting. These results demonstrate that principled multi-judge calibration can simultaneously improve accuracy and correct for systematic leniency without requiring labeled data, offering a scalable path to reliable automated evaluation as model capabilities increase.

Gemma Zhang, Prachi Badarayani, Asmi Kumar +2

Microsoft · Massachusetts Institute of Technology

LLM Evaluation LLM-as-a-Judge Language Model Calibration LLM Reliability Language Model Bias Evaluation Bias in LLM-as-a-Judge+1

Ask PDF

Oct 13, 2026

A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge

Large language models (LLMs) are increasingly used as judges to predict which of two candidate answers a human evaluator would prefer, providing a scalable alternative when human evaluation is too costly. Unfortunately, LLM judgments can exhibit systematic biases. Prior work has documented position, verbosity or model family of the candidate answers influencing the judge decision rather than their quality, while additional biases may remain undocumented.

\textit{Multicalibration}

provides a natural framework to address this problem: each bias corresponds to miscalibration on an identifiable group,

\textit{e.g.}

, family bias manifests as overconfidence in answers generated by models belonging to the same family of the judge. The challenge is to fix miscalibration simultaneously across many, potentially unknown groups. To this end, we introduce

\mathrm{CaLLM}

, a post hoc framework that multicalibrates an LLM judge, thereby bounding all such biases simultaneously. We prove that

\mathrm{CaLLM}

bounds the bias of every group whose error structure is captured by an available representation, and that augmenting it with explicit bias groups further reduces the residual bias. We then derive theoretical conditions under which

\mathrm{CaLLM}

improves the judge's accuracy, rather than trading it for calibration. Finally, we evaluate

\mathrm{CaLLM}

across pairwise preference benchmarks, showing that it reduces the multicalibration error on both specified and unspecified bias groups, while improving predictive performance of the judge.

Cesare Barbera, Luca Stradiotti, Lorenzo Cascioli +6

University of Pisa · University of Trento · KU Leuven +1

Bias in LLM-as-a-Judge LLM-as-a-Judge Language Model Calibration Multicalibration

Ask PDF

Sep 18, 2026

Agreement Overstates Evidence: Error Dependence in LLM Judge Consensus

Consensus among LLM judges is often taken as strong evidence that a decision is correct. This assumes that judges make their errors independently. In practice, LLM judges are often trained and evaluated in similar ways, so they can make the same mistakes. We study how this dependency affects the reliability of consensus. We find substantial error correlation across both open-weight and frontier LLM judges. In our main bank of ten judges, the average pairwise correlation between judge errors is 0.21. As a result, the ten judges only provide roughly as much statistical information as 3.5 independent judges. The dependency is even stronger among the high-accuracy frontier judges we evaluate, including judges from different providers. In up to 28% of our comparisons, ignoring shared errors leads to the conclusion that one system is significantly better, while accounting for them does not. We also find that the pattern of errors matters. Errors shared by most judges and errors concentrated among a smaller group affect consensus differently and favor different voting methods. Measuring the overall amount of correlation alone is therefore insufficient. Our results suggest a simple approach: use a small set of trusted examples to estimate judge accuracy and identify shared mistakes. These shared errors should then be considered when analyzing the results, and the voting method should be chosen using trusted examples before it is applied to new data.

Elias Hossain, Niloofar Yousefi, Ser-Nam Lim

University of Central Florida

LLM-as-a-Judge LLM Reliability Language Model Ensembles

Ask PDF

Abstract

Correlated Errors in Language Models

Explore similar work

Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration

A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge

Agreement Overstates Evidence: Error Dependence in LLM Judge Consensus