When Accuracy Gaps Fail to Certify: Auditing Cross-Domain Recalibration of LLM Judges

Fariya Afrin; Ibne Farabi Shihab

cs.LGAug 23, 2026

When Accuracy Gaps Fail to Certify: Auditing Cross-Domain Recalibration of LLM Judges

Authors: Fariya Afrin, Ibne Farabi Shihab

Organizations: Department of Computer Science, Kalinga Institute of Industrial Technology · Department of Computer Science, Iowa State University

Abstract

A scalar recalibration map fitted for an LLM judge on one task can fail when the task distribution changes, but the source-target accuracy gap is often treated as a proxy for that failure. We test what this gap can predict and what it can certify across thirteen judges, two generators, eight domains, and 1,176 predeclared transfers. After accounting for mean score shift, the gap yields a population lower bound on target calibration error, yet identical gaps can induce opposite transfer outcomes. Exact importance weighting recovers target proper loss under covariate shift, so failure of an estimated weighting pipeline does not by itself establish conditional shift. A finite-sample simultaneous lower certificate converts the population bound into a one-sided rejection rule using audit labels disjoint from evaluation outcomes. The leak-free gap correlation is 0.25 (95% CI [-0.09, 0.55]), falls to 0.09 on the second generator, and does not support a generator-invariant association. The certificate retains nominal coverage but has power 0.13 even at m=1024, whereas target-domain temperature scaling with 16 labels reaches harm rate 0.09, compared with 0.34 for source-fitted Platt scaling. Accuracy gaps are therefore weak warning signals for scalar probability transfer, not deployment certificates.

Figures & tables

Domain	Task type	Correctness rule (fixed a priori)	$n$
GSM8K	Mathematical reasoning	last-number match as floats (tolerance $10^{-6}$ )	40
MMLU	Multiple-choice knowledge	option-letter exact match	40
SQuAD	Extractive QA	span containment or token-F1 $\geq 0.60$	40
BoolQ	Binary QA	normalized yes/no match	40
SAMSum †	Summarization	ROUGE-L $F\geq 0.30$ ; margin $[0.15,0.30)$ referred to a small verifier	40
HumanEval	Code generation	sandboxed execution of official unit tests	40

Table 1: Domains and correctness labels as implemented. HumanEval is evaluated by unit-test execution, not by an LLM. † Proxy-label domains: overlap thresholds are deterministic but imperfect proxies for correctness, with a single small LLM verifier (Qwen2.5-1.5B) deciding only borderline-overlap items. No human annotation is used. These two domains therefore support robustness analyses rather than deterministic-contract calibration claims; threshold sensitivity is reported in Section 6.5 .

Method	$\Delta$ Brier $\downarrow$ [95% CI]	$\Delta$ log loss $\downarrow$ [95% CI]	ECE $\downarrow$	harm rate $\downarrow$ [95% CI]	AUROC
Raw	—	—	0.393	—	0.611
Temperature	$-0.147$ [ $-0.231,-0.081$ ]	$-3.87$ [ $-5.30,-2.61$ ]	0.281	0.13 [0.02, 0.26]	0.611
Platt, $a\geq 0$	$-0.052$ [ $-0.149,+0.005$ ]	$-2.56$ [ $-3.75,-1.51$ ]	0.371	0.34 [0.18, 0.48]	0.588
Beta calibration	$-0.046$ [ $-0.143,+0.011$ ]	$-2.18$ [ $-3.42,-1.21$ ]	0.375	0.34 [0.19, 0.49]	0.602
Isotonic	$-0.078$ [ $-0.198,-0.009$ ]	$-3.11$ [ $-4.68,-1.74$ ]	0.349	0.33 [0.18, 0.46]	0.589
Multi-domain temp.	$-0.163$ [ $-0.245,-0.090$ ]	$-3.92$ [ $-5.35,-2.63$ ]	0.271	0.17 [0.03, 0.37]	0.611

Table 2: Cross-domain calibration transfer on untouched target-test records: 448 primary transfer units, paired means with two-way judge/domain cluster-bootstrap 95% intervals (

B{=}2000

) for both primary proper losses and harm rate. Negative

\Delta

improves on raw. Harm rate is the fraction of transfer units with worse target Brier than raw. Equal-label target baselines use exactly the certificate’s

m{=}16

target-probe budget. AUROC is inherited from raw wherever the fitted map is strictly increasing (Proposition 1 holds exactly; float-level sigmoid saturation would otherwise create spurious ties) and computed directly for isotonic and slope-collapsed fits, which are the only genuine departures.

Predictor	Coefficient (95% CI)	$p_{\mathrm{boot}}$
Platt $\times$ probe gap	$+0.244$ ( $-0.062,+0.618$ )	$0.10$
Temperature $\times$ probe gap	$+0.056$ ( $-0.031,+0.244$ )	$0.17$
Score Wasserstein distance	$-0.099$ ( $-0.344,+0.280$ )	$0.51$
Probe-split raw Brier	$+0.270$ ( $+0.078,+0.429$ )	$0.007$
Log effective sample size	$+0.224$ ( $-4.78,+9.55$ )	$0.69$

Table 3: Adjusted analysis of calibrated target-test Brier (ANCOVA; outcome shares no labels with any predictor), stacking the Platt and temperature contrasts over the 448 primary transfers with method fixed effects. Inference: coefficients refit in each of

B{=}2000

two-way judge/domain cluster-bootstrap replicates; percentile CIs and bootstrap

p

-values. Effective dependence: 8 judges, 8 domains, 64 judge–target cells. A mixed-effects check with judge, source, target, judge

\times

target, and transfer variance components gives the same point estimates with narrower (likelihood-based) intervals; we report the more conservative cluster-bootstrap intervals as primary.

Shift control	Plain Platt	Estimated IW	Oracle IW
Covariate, misspecified (pos.)	0.324	0.256	0.257
Covariate, mild	0.232	0.231	0.234
Label only	0.233	0.214	0.173
Conditional only	0.314	0.313	0.314
Natural NLP transfers	0.362	0.362	n/a

Table 4: Mean target Brier under identification controls (30 replications

\times

12 local judges; truth known by construction). Row 1 is the designed positive control: weighting must and does help. “Estimated IW” in the label-only row is the EM prior correction. Natural row: means over the 448 primary transfers; no oracle weights exist there.

Target labels	Simult. coverage	Harm detected	False accept
$m=32$	1.000	0.000	1.000
$m=128$	1.000	0.000	1.000
$m=512$	1.000	0.023	0.977
$m=1024$	1.000	0.128	0.872

Table 5: Certificate at

1-\delta=0.95

\gamma=0.10

K{=}6

, on synthetic populations with known truth (448 transfers, 60 audit repetitions, equal audit sizes

m

). “Harm detected” is power over cells with true target CE

>\gamma

(nearly all cells). “False accept” is descriptive: Theorem 7 certifies only rejection.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Primary judges (1–7B)
Short name	Repository ID	Params.
Phi-3-mini	microsoft/Phi-3-mini-4k-instruct	3.8B
TinyLlama	TinyLlama/TinyLlama-1.1B-Chat-v1.0	1.1B
Zephyr	HuggingFaceH4/zephyr-7b-beta	7B
MiniCPM	openbmb/MiniCPM-2B-dpo-bf16	2.4B
StableLM	stabilityai/stablelm-zephyr-3b	3B

Appendix

Table 6: Reproducibility manifest. All judges are loaded in 4-bit NF4 double quantization (bitsandbytes) with bf16 compute, eager attention, greedy decoding, at most 8 new tokens, and 512-token input truncation. Candidate generators: Qwen/Qwen2.5-1.5B-Instruct (primary) and meta-llama/Llama-3.2-1B-Instruct (revision 92131767 , replication), both bf16. Revision hashes were not captured for the primary judges in the original Qwen-generator scoring (disclosed gap); the second-generator rescore recorded revisions and per-judge parse-failure counts for all eight (phi3 44, gemma2 46, minicpm 4, stablelm 2, others 0, of 320; failures score the predeclared

0.5

), and the four replication judges have recorded revisions with zero failures.

Method	$\Delta$ Brier $\downarrow$ [95% CI]	$\Delta$ log loss $\downarrow$ [95% CI]	ECE $\downarrow$	harm rate $\downarrow$ [95% CI]	AUROC
Raw	—	—	0.369	—	0.610
Temperature	$-0.143$ [ $-0.214,-0.087$ ]	$-3.76$ [ $-5.05,-2.61$ ]	0.203	0.08 [0.01, 0.16]	0.610
Platt, $a\geq 0$	$-0.058$ [ $-0.120,-0.015$ ]	$-1.94$ [ $-2.89,-1.12$ ]	0.305	0.24 [0.10, 0.40]	0.590
Beta calibration	$-0.051$ [ $-0.112,-0.007$ ]	$-1.65$ [ $-2.63,-0.80$ ]	0.320	0.25 [0.10, 0.41]	0.605
Isotonic	$-0.113$ [ $-0.203,-0.046$ ]	$-3.25$ [ $-4.67,-1.95$ ]	0.240	0.22 [0.08, 0.36]	0.592
Multi-domain temp.	$-0.152$ [ $-0.222,-0.092$ ]	$-3.78$ [ $-5.07,-2.64$ ]	0.192	0.09 [0.00, 0.22]	0.610

Appendix

Table 7: Second-generator (Llama-3.2-1B) calibration transfer on untouched target-test records: 448 transfer units, all eight primary judges, paired means with two-way judge/domain cluster-bootstrap 95% intervals (

B{=}2000

), same protocol and columns as Table 2 . Source-fitted transfer is milder here than on the primary generator (lower harm rates), the EM label-shift correction is again the only harmful method, and equal-budget target temperature remains the best row.

Gen.	Treatment	$r_{\mathrm{gap}}$	$\Delta$ B temp	$\Delta$ B tgt-t	OLS raw
Pri.	Impute $0.5$	0.253	$-0.147$	$-0.171$	$+0.270$
Pri.	Exclude flagged	0.271	$-0.152$	$-0.177$	$+0.283$
G2	Impute $0.5$	0.085	$-0.143$	$-0.155$	$+0.242$
G2	Exclude flagged	0.093	$-0.149$	$-0.163$	$+0.238$
G2	Constrained dec.	0.067	$-0.136$	$-0.141$	$+0.094$

Appendix

Table 8: Parse-failure sensitivity: headline quantities under the committed

0.5

imputation, exclusion of flagged item–judge pairs, and (second generator) digit-constrained decoding. Point estimates; the seed-0 cluster intervals in the main text apply to the committed rows.

Variant	$r_{\mathrm{gap}}$	$\Delta$ B temp	$\Delta$ B tgt-t	harm tgt-t	fail
A (committed)	0.253	$-0.147$	$-0.171$	0.09	0.044
B (paraphrase)	0.317	$-0.128$	$-0.141$	0.11	0.107
C (paraphrase)	0.287	$-0.134$	$-0.152$	0.13	0.018
Aswap (order)	0.246	$-0.165$	$-0.191$	0.09	0.041

Appendix

Table 9: Prompt/order robustness on the primary generator (448 transfers per variant; point estimates). “fail” is the overall parse-failure rate, which is itself prompt-sensitive (prompt B drives Gemma-2 to

59\%

Language Model Calibration

Language model calibration studies whether models’ confidence estimates match their actual accuracy across tasks, prompts, and deployment settings.

See all 386 papers

Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration
Sep 10, 2026Gemma Zhang, Prachi Badarayani, Asmi Kumar +2LLM Evaluation LLM-as-a-Judge
Microsoft · Massachusetts Institute of Technology
Bias and Uncertainty in LLM-as-a-Judge Estimation
May 7, 2026James FiedlerLLM Evaluation LLM-as-a-Judge
Indeed Inc.
A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge
Oct 13, 2026Cesare Barbera, Luca Stradiotti, Lorenzo Cascioli +6Bias in LLM-as-a-Judge LLM-as-a-Judge
University of Pisa · University of Trento · KU Leuven +1

Sep 10, 2026J/K move · Enter open · S save

Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration

Gemma Zhang, Prachi Badarayani, Asmi Kumar, Sadid Hasan+1

Microsoft · Massachusetts Institute of Technology

LLM Evaluation LLM-as-a-Judge Language Model Calibration LLM Reliability Language Model Bias Evaluation Bias in LLM-as-a-Judge+1

Open paper AskPDF arXiv

LLMs are increasingly used as automated judges for model training and evaluation, yet individual judges exhibit systematic biases that undermine reliability. Much of prior work has studied biases in pairwise LLM-as-a-judge settings; in this paper, we focus on absolute scoring tasks, which mirror more realistic use cases. Across four benchmarks and six models (36 judge-examinee pairs), we show that a model's task accuracy strongly predicts its judging accuracy (Pearson

r \geq 0.90

on most models) and inversely predicts its directional bias (

r \leq -0.83

), but that accuracy alone does not ensure fair evaluation: more capable examinee models consistently receive more lenient judgments from all judges (

r \geq 0.83

). To address this, we propose calibrated weighted majority voting (WMV), an ensemble evaluation method that aggregates multiple LLM judges weighted by online estimates of their false-positive and false-negative rates. We introduce a disagreement-based estimator that derives these error rates purely from inter-judge agreement patterns, requiring no ground-truth labels or task metadata. In a simulated experiment with shifting task distributions, our label-free WMV tracks an oracle with perfect error-rate knowledge to within 0.5 percentage points on average, outperforming both individual judges and unweighted majority voting. These results demonstrate that principled multi-judge calibration can simultaneously improve accuracy and correct for systematic leniency without requiring labeled data, offering a scalable path to reliable automated evaluation as model capabilities increase.

Abstract

Language Model Calibration

Explore similar work

Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration

Bias and Uncertainty in LLM-as-a-Judge Estimation

A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge