Conditional Accuracy Profiles: Diagnosing LLM Judges across Deployment Conditions

Wenqi Li; Bin Liu; Mindi Ruan; Chuanbo Hu; Minglei Yin; Xin Li

cs.LGOct 6, 2026

Conditional Accuracy Profiles: Diagnosing LLM Judges across Deployment Conditions

Authors: Wenqi Li, Bin Liu, Mindi Ruan, Chuanbo Hu, Minglei Yin, Xin Li

Organizations: University at Albany, State University of New York · West Virginia University

Abstract

LLM-as-judge is now a standard tool for scalable evaluation, but judge performance is still often summarized by a single accuracy number. This aggregate view hides the deployment conditions under which a judge succeeds or fails. We introduce \textbf{Conditional Accuracy Profiling} (CAP), a post-hoc diagnostic framework that decomposes pairwise LLM-judge accuracy into eight conditions organized into content sensitivity, robustness, and rationale quality. CAP is benchmark-agnostic: it can be applied directly when a benchmark provides the required annotations, approximately through task-subset proxies, or through controlled augmentation when perturbation pairs can be generated. We instantiate CAP on seven LLM judges across six pairwise judging benchmarks, including \textsc{judgerEva-Standard}, a controlled testbed we created to support all eight conditions. CAP exposes profile differences hidden by aggregate accuracy: on \textsc{judgerEva}'s judge-independent Hard-Constructed subset, the two judges most sensitive to omitted qualifications rank in the bottom three of seven by overall accuracy, so omission sensitivity is not predicted by aggregate accuracy. Across benchmarks, Position Robustness shows the strongest rank stability (mean Spearman $\barρ{=}0.87$ ) but is itself fragile under JudgeBench-Pro adversarial stress, showing the largest mean accuracy drop among the shared conditions, though the dominant degradation channel varies by judge. Condition-level profiles provide a more actionable basis than aggregate accuracy for selecting LLM judges.

Figures & tables

Figure 1: CAP overview. Pairwise items are evaluated in original and swapped orderings; verdicts and rationales are mapped to eight conditions in three groups (content C1–C4, robustness C5–C7, rationale C8).

#	Category	Condition	Subset	Measurement
1	Content	Factual Sensitivity	factual-fabrication items	$\Pr[v=a^{*}]$
2	Content	Logical Sensitivity	logical-reversal items	$\Pr[v=a^{*}]$
3	Content	Scope-Overclaim	overclaim items	$\Pr[v=a^{*}]$
4	Content	Scope-Omission	omission items	$\Pr[v=a^{*}]$
5	Robustness	Style Robustness	plain/polished variants	$\Pr[v(x_{1})=v(x_{2})=a^{*}]$
6	Robustness	Length Robustness	plain/padded variants	$\Pr[v(x_{1})=v(x_{2})=a^{*}]$

Table 1: The eight CAP conditions. Robustness uses a strict “both correct” criterion, distinguishing stable correctness from stable but wrong consistency.

Figure 2: judgerEva-Standard construction. Six MCQ/QA sources $\to$ GPT-4o expands Answer A and injects one of four planted errors into Answer B (subtlety-gated) $\to$ three presentation variants. The Hard subset combines a judge-error filter (Hard-Empirical) and an independent construction branch (Hard-Constructed).

#	Condition	judgerEva	JudgeBench	JB-Pro	RewardBench	MT-Bench	LLMBar
1	Factual Sens.	Direct	Approx.	Approx.	Approx.	Approx.	—
2	Logical Sens.	Direct	Approx.	Approx.	Approx.	Approx.	—
3	Scope-Overclaim	Direct	—	—	—	—	—
4	Scope-Omission	Direct	—	—	—	—	—
5	Style Rob.	Direct	Aug. †	—	—	—	Direct
6	Length Rob.	Direct	Aug. †	—	—	—	—

Table 2: Condition coverage by benchmark. Approx. = task-subset proxy; Aug. = generated perturbation variants; the exact operator behind every cell is specified in Appendix D . † Computed but omitted from main tables; see Appendix E .

Judge	C1	C2	C3	C4	C5	C6	C7	C8 †
GPT-4o	40.5 (34,46)	53.1 (38,69)	63.2 (51,75)	64.1 (53,74)	36.7 (31,42)	20.4 (16,26)	25.2 (22,28)	66.7 (33,100)
GPT-4o-mini	12.5 (8,17)	53.1 (34,72)	48.7 (41,57)	62.8 (53,74)	15.9 (11,21)	10.6 (7,15)	14.8 (12,17)	75.0 (25,100)
Claude Sonnet 4.5	92.4 (88,96)	86.7 (70,100)	89.5 (80,96)	59.2 (46,72)	84.7 (80,89)	84.7 (80,89)	86.0 (83,88)	85.7 (57,100)
Claude Haiku 4.5	82.2 (77,87)	65.6 (44,84)	96.1 (89,100)	70.5 (56,83)	71.8 (66,77)	69.0 (63,75)	73.4 (70,77)	91.7 (75,100)
Gemini 2.5 Flash	76.3 (70,82)	78.1 (59,94)	72.4 (59,83)	56.4 (44,68)	71.4 (65,77)	66.4 (61,72)	63.4 (60,67)	44.4 (11,78)
Gemini 2.5 Pro	73.2 (67,79)	78.1 (63,91)	90.8 (83,97)	61.5 (49,73)	77.5 (72,83)	70.6 (65,76)	62.3 (59,66)	83.3 (50,100)

Table 3: CAP profiles (%, raw) on the judgerEva-Standard Hard-Empirical subset (

N{=}245

). Each cell is the point estimate with 95% bootstrap CI in parentheses; bold marks the best judge per condition. C1–C8 are defined in Table 1 ; per-condition sample sizes are released in the artifact. † C8 is exploratory and evaluator-dependent: it is conditioned on correct-verdict items and measured by an LLM evaluator validated in Appendix C , so it is not directly comparable to C1–C7.

Figure 3: CAP radar profiles on Hard-Empirical (per Table 3 ). Each polygon is one judge across C1–C8; aggregate-similar judges (e.g., the two Geminis) still differ in shape. The C8 axis is exploratory and evaluator-dependent (see Table 3 ).

Judge	Overall	C1	C4	C7
Claude Sonnet 4.5	76.3 (70,82)	80.8 (69,91)	36.7 (23,52)	71.2 (63,78)
Gemini 2.5 Flash	72.3 (65,79)	71.8 (59,86)	38.3 (23,53)	66.2 (58,75)
Gemini 2.5 Pro	71.2 (64,78)	66.7 (53,78)	38.3 (22,53)	65.7 (58,74)
Claude Haiku 4.5	70.1 (63,77)	76.9 (65,88)	33.3 (18,48)	62.6 (55,71)
Llama-3.3-70B	68.3 (62,75)	56.4 (44,69)	56.7 (42,72)	58.3 (51,66)
GPT-4o	59.7 (53,67)	42.3 (28,56)	36.7 (23,52)	51.1 (42,60)

Table 4: CAP profiles (%) on judge-independent Hard-Constructed (

N{=}139

): overall accuracy and the three most diagnostic conditions from Table 3 (C1,

n{=}39

; C4,

n{=}30

; C7,

n{=}139

), each with its 95% bootstrap CI. Bold marks the top per column; C5/C6 require paired variants absent from this subset.

Judge	C1 ≈	C2 ≈	C3	C4	C5 ${}^{\text{aug}}$	C6 ${}^{\text{aug}}$	C7	C8
GPT-4o	60.7 (54,68)	62.8 (54,70)	—	—	n.r.	n.r.	51.1 (46,57)	n.r.
GPT-4o-mini	60.7 (55,67)	52.0 (47,57)	—	—	n.r.	n.r.	35.7 (31,41)	n.r.
Claude Sonnet 4.5	78.8 (74,84)	74.5 (67,82)	—	—	n.r.	n.r.	68.3 (63,73)	n.r.
Claude Haiku 4.5	73.1 (67,79)	70.4 (63,78)	—	—	n.r.	n.r.	66.4 (61,71)	n.r.
Gemini 2.5 Flash	77.6 (71,84)	92.0 (86,97)	—	—	n.r.	n.r.	81.6 (76,86)	n.r.
Gemini 2.5 Pro	84.8 (79,90)	93.5 (89,98)	—	—	n.r.	n.r.	86.8 (83,90)	n.r.

Table 5: CAP profiles (%, raw) on JudgeBench (

N{=}350

); bold marks the best judge per condition. C1 ≈ uses the Knowledge subset as proxy and C2 ≈ the Reasoning subset; C3/C4 are not measurable; C5/C6 (augmented) and C8 are computed but not aggregated here ( n.r. ; see Appendix E ).

Judge	C1 ≈	C2 ≈	C7	C8	Mean
GPT-4o	+37.8	+40.4	+40.1	n.r.	+39.4
GPT-4o-mini	+34.4	+23.3	+28.2	n.r.	+28.6
Claude Sonnet 4.5	+22.0	+35.4	+33.8	n.r.	+30.4
Claude Haiku 4.5	+43.4	+46.8	+50.8	n.r.	+47.0
Gemini 2.5 Flash	+14.4	+14.0	+18.5	n.r.	+15.6
Gemini 2.5 Pro	+17.7	+ -0.4	+15.4	n.r.	+10.9

Table 6: Stress degradation

\delta^{\mathrm{str}}_{c}(j){=}\mathrm{JB}-\mathrm{JB\text{-}Pro}

(points). Positive means worse under stress; Mean is the per-judge average over C1/C2/C7 (C8 omitted, see Appendix E ); per-condition means across judges are given in the text. C7 has the largest mean drop across judges but is the single largest drop for only two of seven judges.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Source	Items
Medical	MedQA-USMLE ( Jin et al., 2021 )	300
Legal	MMLU Professional Law ( Hendrycks et al., 2021 )	296
Math	GSM8K ( Cobbe et al., 2021 )	356
Science	MMLU College Biology/Chem./Physics	300
Logic	MMLU Logical Fallacies	300
General	TruthfulQA ( Lin et al., 2022 )	540

Appendix

Table 7: Sources contributing to judgerEva-Standard .

Domain	Fact.	Log.	Over.	Omis.	Total
Medical	75	75	75	75	300
Legal	74	74	74	74	296
Science	75	75	75	75	300
Logic	75	75	75	75	300
General	135	135	135	135	540
Math	89	89	89	89	356

Appendix

Table 8: Plain-item distribution by domain

\times

error type. Each plain item yields three presentation variants for

6{,}276

items total.

Error type	Hard-Emp.	Hard-Cons.
Factual Fabrication	152 (62.0%)	39 (28.1%)
Logical Reversal	16 (6.5%)	35 (25.2%)
Overclaim	38 (15.5%)	35 (25.2%)
Critical Omission	39 (15.9%)	30 (21.6%)
Total plain items	245	139

Appendix

Table 9: Hard subset error-type distribution.

Judge	C1	C2	C3	C4	C5	C6	C7
GPT-4o	$-0.1$	$-0.2$	$+0.3$	$-0.8$	$-1.0$	$+1.8$	$-0.8$
GPT-4o-mini	$-0.0$	$-0.4$	$+0.0$	$-1.6$	$-1.3$	$+1.5$	$-2.2$
Claude Sonnet 4.5	$-0.0$	$+0.1$	$+0.2$	$+0.0$	$+1.4$	$+1.9$	$+0.7$
Claude Haiku 4.5	$+0.0$	$-0.3$	$-0.2$	$-0.2$	$-0.6$	$+0.8$	$+0.1$
Gemini 2.5 Pro	$-0.1$	$-0.0$	$+0.3$	$-0.8$	$-0.3$	$+2.1$	$+0.3$
Gemini 2.5 Flash	$-0.1$	$+0.1$	$-0.0$	$-1.7$	$-0.3$	$+2.4$	$+0.6$

Appendix

Table 10: QC sensitivity: change in each CAP condition (points) when the

431

low_quality items and their variants are excluded. Positive means the value rises after exclusion.

By error type
Breakdown	n	Agreement
factual_fabrication	17	1.000
overclaim	37	0.946
logical_reversal	24	0.917
critical_omission	42	0.857
By judge

Appendix

Table 11: C8 evaluator vs. human-annotator agreement on the

120

-rationale stratified validation sample. The evaluator is most reliable on factual fabrication and on the two Gemini judges; the lowest agreement is on Claude Haiku 4.5 (

70.6\%

) and on critical-omission rationales (

85.7\%

), reflecting the difficulty of judging whether a rationale has identified a missing detail rather than a present error.

Benchmark	Cond.	Status	Operator
judgerEva	C1–C4	Direct	planted-error subsets ( $523$ plain items each); PA over A/B and B/A
	C5, C6	Direct	constructed polished / length-padded ( ${\sim}2\times$ ) variants; BC vs. plain
	C7	Direct	A/B and B/A orderings of every item; BC
	C8	Direct	LLM evaluator on correct-verdict rationales (Appendix C )
JudgeBench	C1	Approx.	Knowledge subset ( $n{=}154$ ); PA
	C2	Approx.	Reasoning subset ( $n{=}98$ ); PA

Appendix

Table 12: Operator behind every coverage-matrix cell. PA = position-averaged accuracy; BC = strict both-correct.

	jE	JB	JBP	RB	MT	LB
jE	—	0.71	0.71	0.89	0.71	0.86
JB		—	1.00	0.89	0.93	0.86
JBP			—	0.89	0.93	0.86
RB				—	0.93	0.96
MT					—	0.86
LB						—

Appendix

Table 13: Pairwise Spearman

\rho_{C7}

across benchmarks (lower triangle blank by symmetry). Bold values mark non-significant correlations (

p>0.05

n{=}7

judges).

	C1	C2	C3	C4	C5	C6
C2	$.82$
C3	$.64$	$.54$
C4	$-.86$	$-.79$	$-.32$
C5	$.43$	$.57$	$.54$	$-.14$
C6	$.96$	$.86$	$.79$	$-.75$	$.54$
C7	$.93$	$.89$	$.75$	$-.82$	$.39$	$.96$

Appendix

Table 14: Spearman correlation between per-condition judge rankings (full set, seven judges). Mean pairwise

|\rho|=0.69

;

7/28

pairs are negative.

	C5 Style			C6 Length			C7 Position
Judge	strict	cons.	$\Delta$	strict	cons.	$\Delta$	strict	cons.	$\Delta$
GPT-4o	90.3	95.5	5.2	62.0	66.7	4.7	72.9	87.3	14.4
GPT-4o-mini	82.8	90.6	7.8	51.0	58.6	7.6	61.0	80.9	19.9
Claude Sonnet 4.5	93.7	95.1	1.4	92.9	94.1	1.2	95.3	96.9	1.6
Claude Haiku 4.5	93.0	95.3	2.3	85.5	87.8	2.3	90.6	93.7	3.1
Gemini 2.5 Flash	93.2	95.4	2.2	82.0	84.1	2.1	86.5	92.5	6.0

Appendix

Table 15: Strict (both-correct) vs. pure-consistency robustness on the full set (%).

\Delta

= consistency

-

strict = stable-but-wrong mass.

Judge	C1	C2	C3	C4	C7
GPT-4o	$-38.0$	$-22.0$	$-17.0$	$-12.0$	$-32.0$
GPT-4o-mini	$-37.0$	$-24.0$	$-26.0$	$-5.0$	$-30.0$
Claude Sonnet 4.5	$-22.0$	$-3.0$	$-5.0$	$-6.0$	$-12.5$
Claude Haiku 4.5	$-31.0$	$-10.0$	$-9.0$	$-11.0$	$-19.1$
Gemini 2.5 Flash	$-30.0$	$-6.0$	$-5.0$	$-4.9$	$-14.2$
Gemini 2.5 Pro	$-30.0$	$-5.0$	$-7.0$	$-7.2$	$-17.2$

Appendix

Table 16: Generator replication: per-judge change (points) from the GPT-4o slice to the paired Claude-generated slice (

n{=}50

per content condition,

200

for C7). Bottom row: cross-slice Spearman of judge rankings.

Pair	jE	JB	JBP	RB	MT	LB
OpenAI (WF)	93.1	26.2	13.2	n.r.	n.r.	n.r.
Anthropic (WF)	96.4	11.7	61.5	n.r.	n.r.	n.r.
Cross-Family avg.	208.1	63.3	93.4	n.r.	n.r.	n.r.

Appendix

Table 17: Within-family vs. cross-family

L_{1}

profile distance per benchmark.

Judge	Overall	C1	C2	C3	C4
GPT-4o	$0.79\to 1.07$ † $[1.04,1.09]$	$0.80\to 1.07$ † $[1.02,1.13]$	$0.95\to 1.34$ † $[1.31,1.37]$	$0.97\to 1.37$ † $[1.33,1.40]$	$0.94\to 1.32$ † $[1.28,1.35]$
GPT-4o-mini	$0.71\to 1.14$ † $[1.10,1.18]$	$0.67\to 1.04$ † $[0.95,1.12]$	$0.88\to 1.58$ † $[1.52,1.65]$	$0.94\to 1.71$ † $[1.65,1.77]$	$0.95\to 1.74$ † $[1.68,1.81]$
Claude Sonnet 4.5	$0.97\to 1.00$ $[0.99,1.00]$	$0.97\to 1.00$ † $[0.99,1.02]$	$0.99\to 1.02$ † $[1.01,1.03]$	$0.99\to 1.02$ † $[1.01,1.03]$	$0.86\to 0.88$ $[0.85,0.91]$
Claude Haiku 4.5	$0.94\to 0.99$ $[0.98,1.00]$	$0.94\to 0.99$ $[0.97,1.01]$	$0.97\to 1.03$ † $[1.02,1.05]$	$0.99\to 1.05$ † $[1.04,1.06]$	$0.93\to 0.98$ $[0.96,1.00]$
Gemini 2.5 Flash	$0.90\to 1.02$ † $[1.00,1.03]$	$0.92\to 1.04$ † $[1.01,1.06]$	$0.98\to 1.12$ † $[1.10,1.13]$	$0.96\to 1.09$ † $[1.07,1.11]$	$0.92\to 1.04$ † $[1.01,1.07]$
Gemini 2.5 Pro	$0.91\to 1.03$ † $[1.01,1.04]$	$0.91\to 1.03$ † $[1.00,1.06]$	$0.99\to 1.12$ † $[1.10,1.14]$	$0.99\to 1.13$ † $[1.11,1.14]$	$0.94\to 1.06$ † $[1.04,1.09]$

Appendix

Table 18: Bias-corrected CAP values (raw

\to

corrected) with joint-bootstrap 95% CIs on the judgerEva-Standard full set. CIs propagate both item resampling and parametric

q_{0}/q_{1}

draws from their standard errors. † marks saturated cells (corrected point estimate

>1.0

); for

28

of the

30

saturated cells the entire CI lies above

1.0

(two lower bounds round to

1.00

in the table), indicating structural over-compensation rather than sampling noise. The correction applies only to content conditions and overall accuracy; uncorrected

C5

–

C8

values appear in the released full-set profile.

	C1	C2	C3	C4	C5	C6	C7	C8
GPT-4o	79.6	94.9	96.6	93.7	90.3	62.0	72.5	80.0
Peer mean	90.9	97.2	98.0	91.9	94.2	82.0	86.3	91.2

Appendix

Table 19: GPT-4o vs. the non-GPT peer mean (five judges) per condition, full set (%). GPT-4o exceeds the peer mean only on C4.

Judge	$n_{C1}$	$n_{C2}$	$n_{C3}$	$n_{C4}$	$n_{C5}$	$n_{C6}$	$n_{C7}$	$n_{C8}$
GPT-4o	523	523	523	523	2092	2092	6276	100
GPT-4o-mini	523	523	523	523	2092	2092	6276	100
Claude Haiku 4.5	523	523	523	523	2091	2092	6274	100
Claude Sonnet 4.5	523	523	523	523	2092	2092	6276	100
Gemini 2.5 Flash	521	521	522	521	2053	2047	6139	100
Gemini 2.5 Pro	522	523	521	523	2062	2057	6160	100

Appendix

Table 20: Per-judge per-condition sample sizes on the judgerEva-Standard full set (

N_{\text{plain}}{=}2{,}092

;

C5{-}C7

over all variants).

Explore similar work

BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories
Jun 21, 2026Shreyas KCLlm-As-A-Judge Raw Judge Outputs
Pinned and Still Unstable: Within-Judge Verdict Variance and the Noise Floor of LLM-as-Judge Leaderboards
Sep 27, 2026Krishna Chytanya AyyagariVerdict Leaderboard
Google
Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration
Sep 14, 2026Gemma Zhang, Prachi Badarayani, Asmi Kumar +2Llm-As-A-Judge Large Language Model Judges
Microsoft · Massachusetts Institute of Technology

Jun 21, 2026, cs.CLJ/K move · Enter open · S save

BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories

Shreyas KC

Llm-As-A-Judge Raw Judge Outputs Natural Language Processing

Open paper AskPDF arXiv

LLM-as-a-judge has become the dominant approach to scalable evaluation in NLP pipelines, yet judges themselves carry systematic biases that raw accuracy hides: they favor responses placed in slot A (position bias), they prefer longer responses regardless of quality (verbosity bias), and their reliability degrades sharply in lower-resource languages. We introduce BabelJudge, an open-source benchmark and reliability audit framework that measures all four failure modes -- position bias, verbosity bias, order inconsistency, and cross-lingual degradation -- on any judge model, without requiring human preference labels. The key insight is gold-labelling by degradation: starting from a high-quality reference response and applying a controlled perturbation yields a pairwise item whose gold label is known by construction, eliminating annotation cost. We evaluate Qwen2.5-7B-Instruct-4bit across English, Hindi, Arabic, and Swahili and find that our composite bias-penalised reliability score drops from 0.714 in Hindi to 0.550 in Swahili, a gap that raw accuracy (0.835 vs. 0.660) understates. Swahili order consistency collapses to 0.480, meaning judge verdicts are near-random under slot-order swaps -- a failure mode invisible to accuracy alone. We further extend the framework to agentic evaluation via nine trajectory-level perturbations (argument corruption, tool swaps, hallucinated calls, missing steps) and three new metrics: tool accuracy, hallucination detection rate, and trajectory-length bias. BabelJudge is released as a Python package supporting 11 judge backends. Code: https://github.com/Shreyaskc/BabelJudge

Abstract

Explore similar work

BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories

Pinned and Still Unstable: Within-Judge Verdict Variance and the Noise Floor of LLM-as-Judge Leaderboards

Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration