Cheap, Consistent, Calibrated, but Biased? Typed Decision Models as Judges

Fabian Lukassen; Christoph Weisser; Alexander Silbersdorff

cs.CLOct 13, 2026

Cheap, Consistent, Calibrated, but Biased? Typed Decision Models as Judges

Authors: Fabian Lukassen, Christoph Weisser, Alexander Silbersdorff

Organizations: University of Göttingen · Bielefeld University of Applied Sciences and Arts (HSBI)

Abstract

Typed decision models (TDMs) return a probability for each allowed answer instead of generating text, which makes them candidates for cheaply replacing LLM judges that compare two answers. Whether they share the biases of LLM judges is untested. We run the CALM benchmark's bias tests on eight TDMs and seven LLM judges of 2024 and 2026. Every TDM is biased, in three to eight of twelve tests. Emotional tone affects them most, and rewriting the better answer in such a tone lowers the accuracy of a typical TDM from 99% to 42%. TDMs are more biased than 2026 LLM judges. All eight change verdicts when the answers swap places, which the 2026 LLM judges hardly do, and several side with an answer that most people allegedly prefer or that a famous model allegedly wrote. Yet both kinds of judge are tripped up by the same prompts. The TDMs' probabilities show that most tests mainly flip verdicts the TDM was unsure of, and that good calibration does not protect against bias. As TDMs are more consistent and 37 times cheaper than 2026 LLM judges, our results suggest they can be attractive judges if their biases are tested for the task.

Figures & tables

Bias Type	Description	Example
Position	LLM judges exhibit a propensity to favor one answer at certain position over others.	Turn 1: $R_{1}$ : 3.8 $>$ 3.11 $R_{2}$ : 3.11 $>$ 3.8 Turn 2: $R_{2}$ : 3.11 $>$ 3.8 $R_{1}$ : 3.8 $>$ 3.11
Verbosity	LLM judges favor longer responses, even if they are not as clear, high-quality, or accurate as shorter alternatives.	$R_{1}$ : 3.8 $>$ 3.11 (Shorter) $R_{2}$ : As we all know, in mathematics, 3.11 is greater than 3.8 (Longer)
Compassion fade	The tendency to observe different behaviors when given well-known model’s name as opposed to anonymized aliases.	Llama-7B ( $R_{1}$ ): 3.8 $>$ 3.11 GPT-4 ( $R_{2}$ ): 3.11 $>$ 3.8
Bandwagon	The tendency to give stronger preference to the majority’s beliefs regardless of whether they are correct or not.	I : 90% believe that $R_{2}$ is better. $R_{1}$ : 3.8 $>$ 3.11 $R_{2}$ : 3.11 $>$ 3.8
Distraction	The inclination to give more attention to irrelevant or unimportant details.	I : $R_{2}$ loves eating pasta, especially with homemade tomato sauce. $R_{1}$ : 3.8 $>$ 3.11 $R_{2}$ : 3.11 $>$ 3.8
Fallacy oversight	LLM judges may ignore logical errors in reasoning steps and only focus on the correctness of final results.	$R_{1}$ : 3.8 has fewer digits, so 3.8 $>$ 3.11. (Reasoning broken) $R_{2}$ : 0.8 is greater than 0.11, so 3.8 $>$ 3.11.

Table 1: The nine biases we test, adapted from Ye et al. (2025, Table 1) .

R_{1}

= better (correct),

R_{2}

= worse.

Figure 1: Left: Excess

E\times 100

per judge and test. Bias:

\mathrm{CI}_{95}(E)>\delta

. Unclear:

\delta\in\mathrm{CI}_{95}(E)

. No bias:

\mathrm{CI}_{95}(E)<\delta

(

\delta=0.05

). † Our additions. ∗

E=\mathrm{RR}+\mathrm{CR}-1

, the overlooked flaws. Right: Tests with a bias per judge.

Figure 2: (a) Mean shift

\Delta p

of the targeted response. (b) Prompts on which the median TDM’s verdict flips, or stays while its probability moves by more than 0.10. (c) Flipped verdicts by the confidence on the original prompt. † Our additions. Fallacy oversight is left out (median

\Delta p=-0.83

	Mean excess ( $E\times 100$ )			Alignment set					Fact-related set					Layout change (%)		Latency	Cost per 1,000
Judge	CALM	native	dec.	Acc	$\widehat{\mathrm{Acc}}$	$\mathrm{CR}$	ECE	Brier	Acc	$\widehat{\mathrm{Acc}}$	$\mathrm{CR}$	ECE	Brier	native	dec.	(s)	judgments (US$)
Jev 1.13	10.9	11.0	11.5	78	71	98.2	0.08	0.17	100	84	100.0	0.03	0.01	0.8	1.3	0.25	0.04
Kev 4B	13.4	14.1	15.6	65	58	100.0	0.10	0.24	99	84	100.0	0.20	0.06	6.9	7.4	1.90	0.03
Tev1 4B	11.7	13.5	15.1	66	62	100.0	0.09	0.21	99	90	100.0	0.10	0.02	4.7	5.0	0.56	0.04
Solar Decide	7.8	6.3	7.8	67	63	90.5	0.25	0.28	96	87	98.3	0.03	0.03	10.4	9.3	2.87	0.06
D1	11.7	11.3	11.9	75	72	99.8	0.07	0.18	99	83	99.8	0.08	0.02	4.0	3.5	0.38	0.03

Table 3: Acc,

\widehat{\mathrm{Acc}}

\mathrm{CR}

, ECE and Brier score as defined in Section 3.3 , the first three in percent.

\widehat{\mathrm{Acc}}

is averaged over the tests, without fallacy oversight. Layout change: verdicts on the original prompts that change against the CALM format. Latency: median.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

	CALM	This paper
Judge and request
Judges	six LLMs at temperature 0.7	eight TDMs, and seven LLM judges on the same prompts at temperature 0 where the provider allows it (Section 3.2 )
Judge output	a short explanation, then [[A]] or [[B]]	TDMs: a probability for A and for B, no text. LLM judges: as in CALM
Form of the request	one text prompt	TDMs: three typed formats (Section 3.2 ). LLM judges: CALM’s text prompt
Order of the two responses	better response first in the released data, shuffling not reported	shuffled per item, better response first in half of the items
Ties	a tie option exists but is off by default, so the judge writes [[A]] or [[B]]	no tie option. A 50/50 answer of a TDM and an LLM reply without a verdict give no verdict (Appendix D.2 )

Appendix

Table 4: How our setup compares with CALM’s, at every point where CALM leaves a choice open or where our setup differs from it. Section 3 and Appendices B and C give the reasons. CALM’s examples are Figures 12 and 13 of Ye et al. (2025) .

Judge	Model as returned	Route	Temperature	Reasoning	Verdicts (%)
GPT-3.5 Turbo	openai/gpt-3.5-turbo	OpenRouter	0	off	99.4
GPT-4o	openai/gpt-4o-2024-05-13	OpenRouter	0	off	99.9
GPT-6.1 Sol	openai/gpt-6.1-sol	OpenRouter	default	low, required	100.0
Claude Sonnet 5.5	claude-sonnet-5-5	Anthropic API	default	off	100.0
Gemini 3.8 Flash	google/gemini-3.8-flash	OpenRouter	0	low, required	99.6
Qwen3.8-27B	qwen/qwen3.8-27b	OpenRouter	0	off	99.9

Appendix

Table 5: Settings of the LLM judges. Every judge received all 15,195 requests of the design. Verdicts: share of requests whose reply names a verdict, [[A]] or [[B]] . The rest are refusals, empty replies, replies without a verdict and failed requests (Appendix D.1 ).

Panel of Figure 3 — Figure 3: The exact requests for one bandwagon item in the three formats, and the answers of Jev 1.13. The perturbation is the sentence “90% of people believe that Assistant B is better in this question.” Panel (c) shows the first of six questions. Long lines are wrapped for display.

			Verdicts (%)
Model	Version as returned	Built on	CALM	native	decomposed
Jev 1.13	typesafe/jev-1.13-20260917	not disclosed	99.6	99.5	99.9
Kev 4B	jaredpalmer/kev-4b-20260924	Qwen	100.0	100.0	100.0
Tev1 4B	togethercomputer/tev1-4b-experimental-20260923	Qwen	97.3	98.0	99.8
Solar Decide	upstage/solar-decide-20260928	Solar Mini 4	99.2	99.2	100.0
D1	liquid/d1-20260930	not disclosed	98.3	98.3	99.9

Appendix

Table 6: Details of the TDMs. Every model received all 15,195 requests of the design in each of the three formats through OpenRouter’s Decisions API. Verdicts: share of requests with a verdict, that is, without a 50/50 answer or a failed request (Appendix D.2 ).

	50/50 answers
Model	CALM	native	option returned
Tev1	414	307	always A
D1	251	263	always A
Solar	117	117	always A
Perplexity	0	0
Jev	65	70	A or B

Appendix

Table 7: Answers with

|p_{A}-0.5|<10^{-5}

out of 15,195 per model and format, and the option the API returns for them.

Cue	Chance	Probability	Verdict
no	no	$(1-q)(1-r)$	same
yes	no	$q\,(1-r)$	changed
no	yes	$(1-q)\,r$	changed
yes	yes	$q\,r$	same (changed twice)

Appendix

Table 8: The four cases for a perturbed prompt if the cue and chance act independently: does each of them change the verdict?

	Fact-related set				Alignment set							Our additions
Judge	Ver.	Fal.	Sen.	$\mathrm{CR}$	Pos.	Com.	Ban.	Aut.	Dst.	Div.	$\mathrm{CR}$	Aut. (b.)	Ver. (b.)	Neutral
TDMs, CALM format
Jev 1.13	1.00	0.98	0.63	1.00	0.90	0.96	0.78	0.87	0.93	0.96	0.98	0.70	0.91	0.95
Kev 4B	1.00	0.99	0.61	1.00	0.81	0.85	0.81	0.85	0.94	0.91	1.00	0.78	0.91	0.95
Tev1 4B	0.99	0.93	0.75	1.00	0.76	0.79	0.96	0.85	0.95	0.94	1.00	0.77	0.97	0.95
Solar Decide	0.93	0.92	0.75	0.98	0.76	0.83	0.87	0.81	0.88	0.89	0.91	0.78	0.95	0.88

Appendix

Table 9: Robustness rate

\mathrm{RR}

of each judge per test: the TDMs in the CALM format and the LLM judges on the same prompts (Section 3.2 ). Higher is better. Bold : a bias, meaning that the whole 95% interval of the excess

\mathrm{CR}-\mathrm{RR}

lies above

\delta=0.05

. For fallacy oversight (Fal.) we give the share of correct reactions

1-\mathrm{RR}

, and bold marks a judge whose share of overlooked flaws beyond chance,

\mathrm{RR}+\mathrm{CR}-1

, lies above

\delta

\mathrm{CR}

is the consistency rate on the original prompts of each dataset. Ver. verbosity, Sen. sentiment, Pos. position, Com. compassion fade, Ban. bandwagon, Aut. authority, Dst. distraction, Div. diversity, (b.) on better, Neutral neutral note.

Test	Jev	Kev	Tev1	Solar	D1	Perplexity	Clef	Clef Flash
Sentiment	37 [34, 39]	39 [37, 41]	25 [22, 28]	24 [21, 27]	35 [32, 37]	29 [26, 32]	29 [26, 32]	27 [25, 29]
Authority (on better) †	29 [22, 36]	22 [17, 28]	23 [17, 30]	12 [6, 18]	30 [23, 37]	38 [31, 46]	34 [27, 41]	8 [5, 11]
Bandwagon	20 [16, 24]	19 [16, 22]	4 [3, 6]	3 [0, 6]	7 [5, 9]	26 [22, 30]	27 [23, 31]	12 [9, 15]
Authority	12 [8, 17]	15 [11, 20]	15 [10, 20]	8 [3, 14]	18 [12, 24]	16 [11, 22]	17 [11, 23]	12 [7, 16]
Position	8 [6, 11]	19 [15, 23]	24 [20, 29]	15 [11, 20]	12 [9, 16]	8 [5, 10]	14 [11, 17]	19 [16, 23]
Compassion fade	2 [0, 4] ∘	15 [12, 18]	21 [17, 25]	8 [4, 12]	8 [5, 11]	4 [3, 6]	5 [3, 8]	7 [5, 10]

Appendix

Table 10: Excess in changed verdicts per 100 beyond chance, with its 95% bootstrap interval over items, TDMs in the CALM format. Bold : a bias at

\delta=0.05

. ∘ : not significant in the permutation test after the Benjamini–Hochberg correction (fallacy oversight has no permutation test). † Our additions. ∗ For fallacy oversight,

\mathrm{RR}+\mathrm{CR}-1

Test	GPT-3.5 Turbo	GPT-4o	GPT-6.1 Sol	Sonnet 5.5	Gemini 3.8 Fl.	Qwen3.8-27B	Haiku 5.5
Sentiment	17 [14, 21]	30 [27, 33]	38 [35, 41]	29 [26, 31]	31 [28, 33]	35 [32, 38]	35 [32, 37]
Authority (on better) †	7 [2, 11]	14 [8, 20]	29 [21, 36]	31 [23, 39]	35 [27, 42]	29 [21, 37]	26 [18, 35]
Bandwagon	15 [11, 18]	3 [1, 5]	1 [0, 2] ∘	6 [4, 9]	5 [4, 7]	6 [3, 9]	3 [1, 6]
Authority	13 [8, 18]	12 [7, 18]	32 [25, 40]	14 [8, 20]	9 [2, 15]	12 [4, 19]	12 [6, 19]
Position	37 [32, 42]	21 [17, 25]	3 [1, 5]	3 [1, 6]	1 [-1, 3] ∘	6 [2, 10]	6 [2, 9]
Compassion fade	10 [7, 13]	6 [3, 9]	2 [-0, 3] ∘	2 [-0, 4] ∘	1 [-1, 3] ∘	1 [-2, 4] ∘	6 [3, 10]

Appendix

Table 11: Excess in changed verdicts per 100 beyond chance, with its 95% bootstrap interval over items, LLM judges on the same prompts. Bold : a bias at

\delta=0.05

. ∘ : not significant in the permutation test after the Benjamini–Hochberg correction (fallacy oversight has no permutation test). † Our additions. ∗ For fallacy oversight,

\mathrm{RR}+\mathrm{CR}-1

	TDMs, CALM format			TDMs, $0.05$		LLMs, $0.05$
Test	$0.02$	$0.05$	$0.10$	nat.	dec.	2026	2024
Sentiment	8	8	8	8	8	5	2
Authority (on better)	8	7	6	7	7	5	1
Bandwagon	7	5	4	7	6	0	1
Authority	8	7	4	6	5	3	2
Position	8	8	5	7	8	0	2

Appendix

Table 12: Number of judges with a bias: the TDMs (of 8) in the CALM format for the margins

\delta=0.02

, 0.05 and 0.10, the TDMs in the native (nat.) and decomposed (dec.) formats at

\delta=0.05

, and the LLM judges of 2026 (of 5) and 2024 (of 2) at

\delta=0.05

Judge	Pos. (b.)	Pos. (w.)	Ver.	Ver. (b.)	Ban.	Com.	Aut.	Aut. (b.)	Sen. (b.)	Sen. (w.)
TDMs, CALM format
Jev 1.13	$+5$	$+6$	$+0$	$-9$	$+22$	$+1$	$-13$	$-30$	$-73$	$+0$
Kev 4B	$+7$	$+16$	$-0$	$-9$	$+18$	$+13$	$-12$	$-21$	$-76$	$-1$
Tev1 4B	$+19$	$+17$	$-0$	$-3$	$+4$	$+13$	$-15$	$-24$	$-51$	$-1$
Solar Decide	$-14$	$-17$	$+3$	$-0$	$+6$	$+9$	$-10$	$-16$	$-47$	$-2$
D1	$-8$	$-11$	$-1$	$-14$	$+5$	$+2$	$-19$	$-29$	$-70$	$+0$

Appendix

Table 13: Change in the share of verdicts for the response that carries the cue, after minus before the perturbation, in percentage points. Positive: the cue pulls verdicts towards its response. Pos.: position, with the better (b.) or the worse (w.) response moved to the second place. Sen.: sentiment, with the tone on the better (b.) or the worse (w.) response. Other abbreviations as in Table 9 . Under bandwagon and compassion fade, the cue favours the worse response.

Model	Ver.	Fal.	Sen.	Pos.	Com.	Ban.	Aut.	Dst.	Div.	Aut. (b.)	Ver. (b.)	Neu.
CALM format
Jev 1.13	$-0.02$	$-0.93$	$-0.34$	$+0.04$	$+0.02$	$+0.17$	$-0.15$	$-0.06$	$-0.03$	$-0.21$	$-0.16$	$-0.03$
Kev 4B	$-0.04$	$-0.71$	$-0.28$	$+0.08$	$+0.07$	$+0.08$	$-0.08$	$-0.03$	$-0.04$	$-0.10$	$-0.08$	$-0.02$
Tev1 4B	$+0.01$	$-0.71$	$-0.22$	$+0.06$	$+0.11$	$+0.04$	$-0.09$	$-0.03$	$-0.05$	$-0.14$	$-0.06$	$-0.04$
Solar Decide	$+0.04$	$-0.84$	$-0.22$	$-0.12$	$+0.07$	$+0.06$	$-0.09$	$-0.01$	$-0.01$	$-0.15$	$-0.01$	$-0.01$
D1	$-0.04$	$-0.82$	$-0.30$	$-0.08$	$+0.01$	$+0.04$	$-0.16$	$-0.00$	$-0.01$	$-0.20$	$-0.15$	$-0.01$

Appendix

Table 14: Mean shift

\Delta p

in the probability of the targeted response per model, format and test. Positive: the cue is rewarded, negative: it is punished.

	Authority (150 items)			Sentiment (150 items)			Verbosity (500 items)
Judge	Before	On worse	On better	Before	On better	On worse	Before	On worse	On better
TDMs, CALM format
Jev 1.13	0.72	0.85	0.42	1.00	0.27	1.00	1.00	1.00	0.90
Kev 4B	0.63	0.75	0.42	0.99	0.22	1.00	0.99	0.99	0.90
Tev1 4B	0.60	0.77	0.34	0.99	0.47	1.00	0.99	0.99	0.95
Solar Decide	0.61	0.71	0.45	0.96	0.49	0.98	0.96	0.93	0.95

Appendix

Table 15: Accuracy, the share of verdicts for the better response, on the original prompts of a test’s items (Before) and after the cue is placed on the worse or on the better response: authority and authority (on better), sentiment with the tone on either response, and verbosity and verbosity (on better), the two tests on the better response being our additions. Authority pools the book, URL and quote, sentiment the four tones. The 95% bootstrap interval over items of every change is at most

\pm 0.07

wide. It excludes zero under authority, authority (on better) and sentiment on the better response, except for Clef Flash under authority (on better), and under verbosity (on better), except for Solar and Perplexity. Under sentiment on the worse response and under verbosity, accuracy hardly changes.

Figure 4: Robustness rate per version, TDMs in the CALM format and the seven LLM judges on the same prompts. (a) Bandwagon by the claimed majority, one line per TDM and per LLM judge of 2024 (black), with the median and range of the 2026 LLM judges in grey. (b) to (d) Median and range over the TDMs (blue) and the 2026 LLM judges (orange), and mean and range of the two 2024 LLM judges (green), for authority by kind of citation and response, sentiment by tone and response, and diversity by identity group, both responses pooled.

Test	Version	TDMs, $\mathrm{RR}$	LLM judges 2026, $\mathrm{RR}$	TDMs, $\Delta p$
Bandwagon	60%	0.87 [0.81, 0.98]	0.93 [0.87, 0.97]	$+0.06$
	70%	0.86 [0.74, 0.97]	0.91 [0.86, 0.97]	$+0.09$
	80%	0.83 [0.68, 0.95]	0.89 [0.84, 0.97]	$+0.11$
	90%	0.80 [0.66, 0.92]	0.89 [0.84, 0.97]	$+0.13$
Authority	Book, on worse	0.86 [0.82, 0.88]	0.79 [0.64, 0.86]	$-0.11$
	Book, on better	0.76 [0.64, 0.94]	0.60 [0.60, 0.67]	$-0.15$

Appendix

Table 16: Robustness rate per version, median [minimum, maximum] over the eight TDMs (CALM format) and over the five 2026 LLM judges on the same prompts, and the median shift of the TDMs. Diversity pools both responses.

Test	Two TDMs	Two LLM judges	TDM and LLM judge	$r$
Sentiment	74 / 11	81 / 10	75 / 12	0.97
Authority (on better)	40 / 20	60 / 23	44 / 25	0.83
Verbosity (on better)	25 / 6	46 / 10	34 / 9	0.99
Authority	31 / 13	42 / 18	33 / 16	0.89
Position	32 / 14	14 / 9	23 / 12	1.00
Bandwagon	26 / 14	16 / 9	21 / 12	0.96

Appendix

Table 17: Do two judges flip on the same prompts? For a pair of judges, the share of prompts (%) on which the verdict of the second flips, among those on which the verdict of the first flips / does not flip, averaged over the pairs of eight TDMs (CALM format), of the five 2026 LLM judges, and of a TDM and a 2026 LLM judge (both ways round). 95% bootstrap intervals over items for the mean when the first flips: 25 to 28, 28 to 32 and 25 to 28. Fallacy oversight is left out, since there a flip is the correct reaction.

r

: correlation of the prompt effects of the TDMs and the 2026 LLM judges in a logistic mixed model of the test, and of all tests in the last row (Appendix G ). A value of 1.00 is its upper limit.

Figure 5: Share of flipped verdicts by the probability that the judge gave its choice on the original prompt, CALM format, all models pooled. The bins run from 0.5 to 0.9 in steps of 0.1, then from 0.9 to 0.95, 0.95 to 0.99 and 0.99 to 1, and each point sits in the middle of its bin. The band shows the range of the eight models for the tests other than sentiment and authority (on better).

Term	Odds ratio	95% CI
Margin of the original, per 0.1	0.47	[0.45, 0.48]
Judge right on the original	1.54	[1.44, 1.65]
Log length ratio, $R_{1}$ to $R_{2}$	0.95	[0.88, 1.04]
Test: Authority	3.07	[2.64, 3.58]
Test: Authority (on better)	6.72	[5.81, 7.77]
Test: Bandwagon	4.67	[4.13, 5.29]

Appendix

Table 18: Logistic mixed model of a flipped verdict, CALM format, with a random intercept per item (100,668 prompts, 972 items, item variance 0.66 on the logit scale, intraclass correlation 0.17). References: the neutral note, Jev and Emerton-DPO. ScienceQA has no term of its own. The tests of the fact-related set use only its three sources, so the effect of ScienceQA is contained in the terms of sentiment and verbosity, and GSM8K and MATH are relative to ScienceQA.

Variance component	Shift (%)	Size of shift (%)
Residual	56.7	57.4
Test	25.2	13.8
Item $\times$ Test	8.0	5.3
Test $\times$ Model	6.0	7.3
Item $\times$ Model	2.0	9.0
Model	1.4	0.5

Appendix

Table 19: Share of the variance of the shift

\Delta p

and of its size

|\Delta p|

by crossed random effects, all formats, without fallacy oversight. The residual holds the variation between the versions and formats of a prompt that no component captures.

Test	Shift	Flip
Sentiment	0.57	0.51
Authority	0.41	0.21
Verbosity	0.35	0.13
Authority (on better)	0.35	0.21
Bandwagon	0.29	0.13
Diversity	0.28	0.13

Appendix

Table 20: Agreement between models on which items move: mean Spearman correlation, over all pairs of models, of the per-item mean shift and flip rate, CALM format.

Figure 6: Reliability diagrams on the original alignment prompts, CALM format. Each point is one bin of the confidence, the probability of the chosen response, with its share of correct verdicts. Points below the diagonal are overconfident. Marker area grows with the number of prompts in the bin.

	Alignment set (439)					Fact-related set (533)
Model	Accuracy	Confidence	ECE	Brier	Log loss	Accuracy	Confidence	ECE	Brier
Jev 1.13	0.78	0.83	0.081	0.169	0.63	1.00	0.97	0.030	0.008
Kev 4B	0.65	0.71	0.099	0.243	0.72	0.99	0.79	0.202	0.058
Tev1 4B	0.66	0.74	0.089	0.212	0.62	0.99	0.89	0.100	0.025
Solar Decide	0.67	0.90	0.250	0.282	1.12	0.96	0.95	0.026	0.034
D1	0.75	0.78	0.066	0.179	0.55	0.99	0.91	0.081	0.019

Appendix

Table 21: Calibration of the probability of the better response on the original prompts, CALM format. Confidence: mean probability of the chosen response. ECE: expected calibration error with 10 equal-width bins of the confidence ( Naeini et al., 2015 ; Guo et al., 2017 ) . Brier score ( Brier, 1950 ) and log loss of

P(R_{1})

. Lower is better for ECE, Brier and log loss. A 50/50 answer has no verdict and is left out of the accuracy, as everywhere in the paper. It enters the other measures with its probability of 0.5 and counts as half right in the ECE. On the fact-related set most models are nearly always right, so their calibration there mostly shows underconfidence.

Figure 7: Expected calibration error and Brier score of the eight TDMs on the original prompts, alignment set (filled) and fact-related set (open). Lower is better.

Judge	Format	Minutes	Parallel requests	US$	US$ per 1,000	Latency (s)
Jev 1.13	CALM	2.2	32	0.68	0.045	0.25
	native	3.5	32	0.68	0.045	0.34
	decomposed	4.6	32	1.23	0.081	0.49
Kev 4B	CALM	16.5	32	0.49	0.033	1.90
	native	20.3	32	0.48	0.032	1.86
	decomposed	23.3	32	0.89	0.059	2.76

Appendix

Table 22: Time and cost of the full design (15,195 requests per model and format). Minutes: wall-clock time of all sending passes. Parallel requests: requests in flight in the main pass. Latency: median seconds per request, the time of the answered attempt without retries, with the stated number of parallel requests. The LLM judges answer CALM’s prompt as text over the same design. Their cost is the cost OpenRouter reported in October 2026 (Claude Sonnet 5.5 at Anthropic’s list price), and their minutes are not comparable, since their runs were interrupted by credit limits.

Model	Ver.	Fal.	Sen.	Pos.	Com.	Ban.	Aut.	Dst.	Div.	Aut. (b.)	Ver. (b.)	Neu.
CALM format
Jev 1.13	1.00	0.98	0.63	0.90	0.96	0.78	0.87	0.93	0.96	0.70	0.91	0.95
Kev 4B	1.00	0.99	0.61	0.81	0.85	0.81	0.85	0.94	0.91	0.78	0.91	0.95
Tev1 4B	0.99	0.93	0.75	0.76	0.79	0.96	0.85	0.95	0.94	0.77	0.97	0.95
Solar Decide	0.93	0.92	0.75	0.76	0.83	0.87	0.81	0.88	0.89	0.78	0.95	0.88
D1	0.98	0.96	0.65	0.87	0.92	0.93	0.82	0.97	0.95	0.70	0.86	0.95

Appendix

Table 23: Robustness rate per model, format and test (fallacy oversight: share of correct reactions). Bold : a bias at

\delta=0.05

, for fallacy oversight a share of overlooked flaws beyond chance above

\delta

. Aut. (b.) and Ver. (b.): citation and padding on the better response. Neu.: neutral identity note. Abbreviations as in Table 9 .

Test	Helpfulness	Relevance	Accuracy	Depth	Creativity	Level of detail
Sentiment	$-0.259$	$-0.259$	$-0.246$	$-0.229$	$-0.144$	$-0.219$
Authority (on better)	$-0.137$	$-0.135$	$-0.143$	$-0.110$	$-0.122$	$-0.103$
Authority	$-0.093$	$-0.097$	$-0.098$	$-0.071$	$-0.084$	$-0.066$
Diversity	$-0.031$	$-0.030$	$-0.030$	$-0.024$	$-0.028$	$-0.022$
Neutral note	$-0.025$	$-0.025$	$-0.024$	$-0.018$	$-0.022$	$-0.018$
Verbosity	$-0.013$	$-0.022$	$-0.016$	$+0.021$	$+0.008$	$+0.033$

Appendix

Table 24: Decomposed format: mean shift of the probability of the targeted response per criterion, all eight models. The criteria react alike, with two exceptions: under sentiment, creativity falls less than the others, and under verbosity, depth and detail rise while accuracy and relevance fall.

Model	One	Both	Change	Agree
Jev 1.13	0.777	0.789	$+1.2$ [ $-0.6$ , $+3.2$ ]	0.90
Kev 4B	0.647	0.645	$-0.2$ [ $-3.0$ , $+2.5$ ]	0.81
Tev1 4B	0.656	0.695	$+3.8$ [ $+0.6$ , $+7.2$ ]	0.75
Solar Decide	0.667	0.690	$+2.4$ [ $-0.6$ , $+5.4$ ]	0.75
D1	0.752	0.773	$+2.2$ [ $-0.3$ , $+4.7$ ]	0.87
Perplexity Decider	0.749	0.752	$+0.2$ [ $-1.6$ , $+2.1$ ]	0.92

Appendix

Table 25: Accuracy on the 439 items of the position test when a model judges the pair in one order (One), and when the probabilities of both orders are averaged (Both), which removes position bias by construction. Change: Both minus One, in percentage points, with a 95% bootstrap interval over items. Agree: share of items with the same verdict in both orders. A 50/50 answer has no verdict and is left out.

Bias in LLM-as-a-Judge

Bias in large language model (LLM)-as-a-judge evaluation studies systematic distortions in automated assessments of model outputs.

See all 115 papers

Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines
Apr 25, 2026Sadman Kabir SoumikDebiased ML LLM-as-a-Judge
Independent Researcher
How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models
Oct 4, 2026Laurène Vaugrante, Thilo HagendorffLLM Evaluation LLM-as-a-Judge
AI Safety Research Group, University of Stuttgart
A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge
Oct 13, 2026Cesare Barbera, Luca Stradiotti, Lorenzo Cascioli +6Bias in LLM-as-a-Judge LLM-as-a-Judge
University of Pisa · University of Trento · KU Leuven +1

Apr 25, 2026J/K move · Enter open · S save

Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines

Sadman Kabir Soumik

Independent Researcher

Debiased ML LLM-as-a-Judge Language Model Generation Evaluation Language Model Bias Evaluation Bias in LLM-as-a-Judge Language Model Debiasing

Open paper AskPDF arXiv

LLM-as-a-Judge has become the dominant paradigm for evaluating language model outputs, yet LLM judges exhibit systematic biases that compromise evaluation reliability. We present a comprehensive empirical study comparing nine debiasing strategies across five judge models from four provider families (Google, Anthropic, OpenAI, Meta), three benchmarks (MT-Bench n=400, LLMBar n=200, custom n=375), and four bias types. Our headline practical finding is that a mid-tier model with the right debiasing can outperform frontier judges at a fraction of the cost: Gemini 2.5 Flash with the Combined Budget strategy reaches the highest agreement of any configuration we tested (71.0%, kappa=0.549) at ~

0.001 per evaluation, about 15x cheaper than the best frontier setup (Claude Sonnet 4, 69.5%, ~

0.015). Other key findings: (1) Style bias is the dominant bias (0.10-0.76 across models, favoring markdown over plain prose), far exceeding position bias (<=0.04), yet is rarely studied. (2) Verbosity bias is heterogeneous when measured length-aware: Pro, Flash, and Llama prefer longer answers (+0.24 to +0.44), Claude prefers concise (-0.12), and GPT-4o is neutral (-0.04); on truncation controls all models correctly prefer the complete response (0.88-1.00 accuracy). (3) Debiasing helps multiple models: Claude S8 (+11.5 pp), Flash S8 (+7.5 pp), and Claude S5 (+7.3 pp) survive Holm-Bonferroni correction, with Flash S1 (+4.7 pp) and Llama S8 (+4.5 pp) also significant. We release our evaluation framework, the 375-pair controlled dataset, and per-instance cached results for all nine strategies.

Abstract

Bias in LLM-as-a-Judge

Explore similar work

Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines

How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models

A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge