Noise-Response Calibration: A Causal Intervention Protocol for LLM-Judges

Maxim Khomiakov; Jes Frellsen

cs.LGMar 17, 2026

Noise-Response Calibration: A Causal Intervention Protocol for LLM-Judges

Authors: Maxim Khomiakov, Jes Frellsen

Organizations: Technical University of Denmark, Kgs. Lyngby, Denmark · Normal Computing Corporation, New York, NY, USA · Pioneer Centre for Artificial Intelligence, Copenhagen, Denmark

Abstract

Large language models (LLMs) are increasingly used as automated judges and synthetic labelers, especially in low-label settings. Yet these systems are stochastic and often overconfident, which makes deployment decisions difficult when external ground truth is limited. We propose a practical calibration protocol based on controlled input interventions: if noise severity increases, task performance should exhibit a statistically significant deterioration trend. We operationalize this with a slope-based hypothesis test over repeated trials, using signal-to-noise-ratio (SNR) perturbations for tabular data and lexical perturbations for text data. Across UCI tabular benchmarks and four text classification datasets, we find clear modality-dependent behavior. Our results reveal a modality gap: while text-based judges degrade predictably, the majority of tabular datasets show a lack of statistically significant performance deterioration even under significant signal-to-noise reduction. Interestingly we find that model performance is lower on datasets that are insensitive to noise interventions. We present a reproducible methodology and reporting protocol for robust LLM-judge calibration under distribution shift.

Figures & tables

Step / criterion	Classification	Regression
Benchmark inventory declared in this paper	222	143
After $>50\%$ numeric feature filter	$190$ ( $-32$ )	$127$ ( $-16$ )
After missing-value drop-off	$185$ ( $-5$ )	$125$ ( $-2$ )
Complete datasets included in this study	138	21

Table 1: Stepwise UCI exclusion flow. First filter keeps datasets with

>50\%

numerical covariates; second filter applies missing-value incomputability rules; the final row reports the subset used for per-dataset slope analyses in Appendix Table 5 .

Figure 1: UCI classification accuracy trajectories grouped by noise sensitivity (Sensitive vs Insensitive). Curves are group means across datasets with 95% CI shading, shown separately for uncorrelated and correlated noise.

Figure 4: Clean test-set baseline comparison for classification datasets (sensitive vs insensitive): ECDFs for median accuracy, trial-to-trial standard deviation, and trial range.

Metric	Sensitive	Insensitive	Ratio	$\Delta$ Median	95% Bootstrap CI
Accuracy	0.691	0.618	1.1 $\times$	$+0.073$	[-0.036, +0.186]
Std Dev	0.0421	0.0178	2.4 $\times$	$+0.0243$	[-0.0013, +0.0394]
IQR	0.0233	0.0116	2.0 $\times$	$+0.0117$	[-0.0021, +0.0490]
Range	0.1121	0.0487	2.3 $\times$	$+0.0634$	[-0.0067, +0.0956]

Table 2: Test-set baseline comparison (classification). Sensitivity labels derived from training-set perturbation. Test-set uses clean data only (5 trials/dataset). All values are medians; CIs from 10 000 bootstrap resamples.

Model	0-shot	5-shot	10-shot	20-shot
gpt-4o	0.483 $\pm$ 0.235	0.509 $\pm$ 0.241	0.514 $\pm$ 0.240	0.523 $\pm$ 0.245
gpt-4o-mini	0.412 $\pm$ 0.210	0.435 $\pm$ 0.209	0.440 $\pm$ 0.212	0.443 $\pm$ 0.213
gpt-5-nano	0.486 $\pm$ 0.257	0.503 $\pm$ 0.258	0.509 $\pm$ 0.263	0.515 $\pm$ 0.266
gpt-5-mini baseline	–	–	–	0.561 $\pm$ 0.253

Table 3: Ablation summary (mean primary metric

\pm

std across datasets).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Lexical $\beta_{1}$	One-sided $p$	Decision
Financial PhraseBank	-0.3687	$<10^{-10}$	Reject $H_{0}$
IMDB	-0.2100	$<10^{-10}$	Reject $H_{0}$
SST-2	-0.4199	$<10^{-10}$	Reject $H_{0}$
Yelp Review Full	-0.4056	$<10^{-10}$	Reject $H_{0}$

Appendix

Table 4: Lexical slope estimates (

\beta_{1}

), one-sided

p

-values, and

H_{0}

decisions.

Task	Dataset	$\beta_{1}$ (Uncorr.)	$p$ (Uncorr.)	$\beta_{1}$ (Corr.)	$p$ (Corr.)	Bucket
Classification	abalone	-0.0002	0.1896	-0.0007	0.0119	Sensitive (corr-only)
Classification	acute_inflammations	-0.0002	0.4039	-0.0000	0.4726	Insensitive
Classification	adult	+0.0008	0.8151	-0.0012	0.1322	Insensitive
Classification	android_permissions	+0.0000	0.7318	-0.0001	0.2699	Insensitive
Classification	annealing	+0.0009	0.6829	+0.0023	0.8607	Insensitive
Classification	auction_verification	-0.0000	0.1738	-0.0000	0.1047	Insensitive

Appendix

Table 5: Per-dataset UCI slope estimates (

\beta_{1}

) from linear regression of primary metric on noise severity (

40-\mathrm{SNR}_{\mathrm{dB}}

), with one-sided

p

-values for

H_{1}:\beta_{1}<0

. Buckets are derived using the sensitivity/insensitivity deterioration protocol defined in Algorithm 1 . Classification (n=138): Insensitive=89, Sensitive (both)=32, Sensitive (corr-only)=11, Sensitive (uncorr-only)=6. Regression (n=21): Insensitive=16, Sensitive (both)=3, Sensitive (corr-only)=0, Sensitive (uncorr-only)=2.

Figure 5: 2D datapoint and density-shift illustration under noise perturbations Left: clean baseline. Middle: uncorrelated Gaussian noise. Right: correlated Gaussian noise.

					Breakdown (Count)
Factor / Level	Task	N	Sens.	Sensitive %	Both	Corr.	Unc.
Area of Application
Biology	C	14	8	57.1	3	3	2
	R	1	0	0.0	0	0	0
Business	C	13	3	23.1	2	1	0
	R	2	1	50.0	0	0	1

Appendix

Table 6: Sensitivity analysis: Relationship between dataset characteristics and performance deterioration. Bars represent relative sensitivity (%). (C: Classification, R: Regression)

Panel of Figure 6 — Figure 6: Lexical perturbation progression for a random IMDB sample ( test_022423 , label: positive), generated with the same corruption pipeline used in the experiments. As severity increases, token-level corruption and masking rapidly erase lexical signal.

Language Model Calibration

Language model calibration studies whether models’ confidence estimates match their actual accuracy across tasks, prompts, and deployment settings.

See all 381 papers

A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge
Oct 13, 2026Cesare Barbera, Luca Stradiotti, Lorenzo Cascioli +6Bias in LLM-as-a-Judge LLM-as-a-Judge
University of Pisa · University of Trento · KU Leuven +1
When Noise Fabricates Bias: The Fragility of LLM-as-a-Judge Bias Measurement under Noisy Text
Sep 10, 2026DongHyun Ryu, Jaehyeok Lee, YeongJun Hwang +1LLM-as-a-Judge Social Bias in Language Models
Sungkyunkwan University, Suwon, South Korea
Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration
Sep 10, 2026Gemma Zhang, Prachi Badarayani, Asmi Kumar +2LLM Evaluation LLM-as-a-Judge
Microsoft · Massachusetts Institute of Technology

Oct 13, 2026J/K move · Enter open · S save

A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge

Cesare Barbera, Luca Stradiotti, Lorenzo Cascioli, Lorenzo Perini+5

University of Pisa · University of Trento · KU Leuven+1

Bias in LLM-as-a-Judge LLM-as-a-Judge Language Model Calibration Multicalibration

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used as judges to predict which of two candidate answers a human evaluator would prefer, providing a scalable alternative when human evaluation is too costly. Unfortunately, LLM judgments can exhibit systematic biases. Prior work has documented position, verbosity or model family of the candidate answers influencing the judge decision rather than their quality, while additional biases may remain undocumented.

\textit{Multicalibration}

provides a natural framework to address this problem: each bias corresponds to miscalibration on an identifiable group,

\textit{e.g.}

, family bias manifests as overconfidence in answers generated by models belonging to the same family of the judge. The challenge is to fix miscalibration simultaneously across many, potentially unknown groups. To this end, we introduce

\mathrm{CaLLM}

, a post hoc framework that multicalibrates an LLM judge, thereby bounding all such biases simultaneously. We prove that

\mathrm{CaLLM}

bounds the bias of every group whose error structure is captured by an available representation, and that augmenting it with explicit bias groups further reduces the residual bias. We then derive theoretical conditions under which

\mathrm{CaLLM}

improves the judge's accuracy, rather than trading it for calibration. Finally, we evaluate

\mathrm{CaLLM}

across pairwise preference benchmarks, showing that it reduces the multicalibration error on both specified and unspecified bias groups, while improving predictive performance of the judge.

Abstract

Language Model Calibration

Explore similar work

A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge

When Noise Fabricates Bias: The Fragility of LLM-as-a-Judge Bias Measurement under Noisy Text

Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration