Judged in Company: In-Context Batching Makes LLM Verdicts Depend on Other Items

Peng Xie; Amr Alanwar

cs.CLOct 29, 2026

Judged in Company: In-Context Batching Makes LLM Verdicts Depend on Other Items

Authors: Peng Xie, Amr Alanwar

Organizations: TUM School of Computation, Information and Technology, Department of Computer Engineering Technical University of Munich, 74076 Heilbronn, Germany

Abstract

To save cost, practitioners often ask a language model to grade many independent items in one prompt. This assumes that verdicts are separable: that the verdict on an item does not depend on the other items in the prompt or on its position among them. Across three judgement tasks (harm in social-media comments, short-answer grading and review star ratings) and five open-weight models with 4-9B parameters, assigning the same 2,000 items to different random batches of 25 changes 11-22% of decisions on harm and grading, 6.5-13.3 points more than renaming the item keys or rewording the instruction, while AUROC against human labels differs from single-item grading only by -0.033 to +0.010; for four models with 14-32B parameters, 6-15% of decisions change, 4.2-9.1 points more. We show that flip rates decompose exactly into components of order, slot and membership, and that under a threshold model they grow with the square root of the variance that the context contributes, so that effects that are small for scores are large for decisions. Most of the latent dissimilarity between partitions comes from the order of the neighbours and the slot. The main effect of which items share the prompt, the part that voting over permutations of a batch cannot remove, accounts for 0.11-0.22 of it, and majority votes over nine permutations still leave 52-60% of the flips. Batched verdicts also differ systematically from single-item verdicts: even the majority of fifteen random batches disagrees with them on 8.4-17.8% of decisions. Editing a single generation shows that both the model's own earlier verdicts and the neighbours' texts carry the effect. On the easy task, star ratings, only 2.9-3.9% of decisions change. We recommend grading ambiguous items one per prompt, recovering most of the cost with a cached prefix, and reporting agreement with single-item grading whenever items are batched.

Figures & tables

Condition (vs. base partition)	$\omega$	$s$	$N$
Keep slot, reorder the others	✓
Reorder the whole batch	✓	✓
Keep slot, new neighbours	✓		✓
New random partition	✓	✓	✓

Table 1: The factors that each condition redraws for an item, relative to its base partition: the order

\omega

of its neighbours, its slot

s

and its membership

N

Model	new	base	excess [95% CI]	new	base	excess [95% CI]	new	base	excess [95% CI]
	Harm			Grading			Stars
Qwen3-4B	13.3	4.6	+8.7 [+6.7, +10.6]	12.9	4.7	+8.2 [+6.7, +9.7]	3.9	2.2	+1.7 [+0.6, +2.8] †
Qwen3-8B	15.5	6.2	+9.4 [+7.5, +11.4]	11.7	4.6	+7.1 [+5.6, +8.8]	3.6	1.7	+2.0 [+1.3, +2.9] †
Gemma-4	13.0	4.4	+8.6 [+6.5, +10.1]	10.7	4.2	+6.5 [+5.3, +8.1]	2.9	1.4	+1.6 [+0.8, +2.3] †
Granite	15.7	5.9	+9.8 [+7.8, +11.5]	21.7	8.4	+13.3 [+11.1, +15.5]	3.5	2.1	+1.4 [+0.7, +2.3] †
GLM-4	16.2	6.5	+9.7 [+7.5, +12.0]	15.6	8.0	+7.6 [+5.9, +9.2]	2.9	1.3	+1.6 [+0.9, +2.3] †

Table 2: Decisions that change when the same items are graded in a new random partition ( new , %: partition 0 against partition 1, the pair on which the excess and gate G1 are computed; Table 4 gives the mean over the three pairs), the largest batched baseline ( base : renamed keys, reworded instruction or one prompt at a time, %), and the excess of new over base in points with its 95% bootstrap interval, computed before rounding, so that it can differ by 0.1 from the difference of the printed columns. † : lower bound below 2 points.

Model	alone	batch	flagged	alone	batch	flagged
	Harm			Grading
Qwen3-4B	0.781	0.777	58 $\to$ 60	0.796	0.805	43 $\to$ 34
Qwen3-8B	0.784	0.763	64 $\to$ 62	0.825	0.800	37 $\to$ 35
Gemma-4	0.827	0.835	49 $\to$ 31	0.814	0.796	46 $\to$ 48
Granite	0.824	0.834	59 $\to$ 43	0.717	0.686	61 $\to$ 59
GLM-4	0.805	0.772	48 $\to$ 48	0.803	0.779	47 $\to$ 41

Table 3: AUROC of the written level against human labels when graded alone and in batches of 25 (mean of three partitions), and the share of items above the decision threshold (%, alone

\to

batched).

Figure 2: Components of the latent dissimilarity between two random partitions, as shares of

D_{\mathrm{new}}

(Proposition 2 ; 95% bootstrap intervals): the order of the neighbours

T_{\omega}

, the main effects of the slot

V_{s}

and of the membership

V_{N}

, and their interaction

V_{sN}

. The four shares add up to 1. The order is estimated from the designated slots of “keep slot, reorder the others”, weighted as in a random partition (§ 3 ).

Figure 3: Target decisions changed by editing one part of a generation, relative to the unedited forward pass: the earlier verdicts taken from a generation with other neighbours (texts kept), the neighbours’ texts taken from that generation (verdicts kept), or both.

Figure 4: Top: flips between partitions 0 and 1 for items the model is sure or unsure about when graded alone (unsure:

0.1<\zeta<0.9

in the single-item prompt, where

\zeta=P(\text{level}\geq\tau)

is read from the next-token distribution as in § 3 ); one line per model and task. Bottom: flips against single-item grading as the batch grows, harm and grading, one line per model and task.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

		single item		batched baselines			batch conditions
Model	Task	keys	reworded	keys	reworded	numeric	vs. alone	reord. others	reord. all	new neighb.	new part.
Qwen3-4B	Harm	3.0	4.0	3.5	4.6	0.9	13.7	9.3	12.3	12.4	13.1
	Grading	1.0	4.6	1.6	4.7	0.8	15.7	8.0	12.0	12.2	13.5
	Stars	0.6	1.7	1.0	2.2	0.2	5.7	2.5	3.3	3.4	3.9
Qwen3-8B	Harm	1.5	5.0	4.7	6.2	1.1	14.0	9.3	12.8	13.0	14.9
	Grading	1.5	8.3	2.0	4.6	0.5	15.0	6.4	11.5	10.1	12.1

Appendix

Table 4: Decisions changed (%) in every condition. Single-item baselines are against the item alone; batched baselines, which are on partition 0, and batch conditions are against the base partition, except “vs. alone”. Batch conditions are means over the three base partitions: reord. others is “keep slot, reorder the others”, measured on the items in the designated slots only and without the weights of § 3 , reord. all is “reorder the whole batch”, new neighb. is “keep slot, new neighbours”, and new part. is the mean over the three pairs of partitions. Numeric : the same prompts generated one at a time instead of in a GPU batch.

Model	Task	keys	reworded	numeric	reord. all	new neighb.	new part.	ratio
		batched baselines			batch conditions
Qwen3-4B	Harm	9.5	13.1	2.5	32.6	31.5	34.0	2.6
	Grading	3.2	9.2	1.8	23.5	22.5	25.1	2.7
	Stars	3.9	9.9	0.7	16.0	15.2	18.0	1.8
Qwen3-8B	Harm	10.3	16.3	2.3	30.8	28.7	34.4	2.1
	Grading	4.8	9.1	1.4	20.5	19.7	22.3	2.4

Appendix

Table 5: Written levels changed (%, any change of the digit) against partition 0: the batched baselines, the reordered and new-neighbour batches derived from partition 0, and partition 1. Ratio : new partition over reworded instruction.

		all items				designated items
Model	Task	ord.	slot	memb.	rem.	ord.	slot	memb.	rem.
Qwen3-4B	Harm	0.56	0.33	0.35	$-$ 0.24	0.53	0.21	0.48	$-$ 0.23
	Grading	0.38	0.42	0.44	$-$ 0.24	0.34	0.63	0.30	$-$ 0.27
Qwen3-8B	Harm	0.44	0.31	0.34	$-$ 0.09	0.51	0.26	0.29	$-$ 0.06
	Grading	0.28	0.63	0.41	$-$ 0.32	0.21	0.85	0.30	$-$ 0.35
Gemma-4	Harm	0.35	0.48	0.33	$-$ 0.17	0.33	0.59	0.33	$-$ 0.26

Appendix

Table 6: Shares of the new-partition dissimilarity as fixed before the runs: order

D_{\mathrm{keep}}/D_{\mathrm{new}}

, slot

(D_{\mathrm{resh}}-D_{\mathrm{keep}})/D_{\mathrm{new}}

, membership at a fixed slot

(D_{\mathrm{comp}}-D_{\mathrm{keep}})/D_{\mathrm{new}}

and the remainder (rem.)

{(D_{\mathrm{new}}-D_{\mathrm{resh}}-D_{\mathrm{comp}}+D_{\mathrm{keep}})/D_{\mathrm{new}}}

, which the analysis fixed before the runs called the interaction, with

D_{\mathrm{keep}}

on the designated items and the other dissimilarities on all items, and with all four dissimilarities on the designated items (400 per partition). If

D_{\mathrm{keep}}

were the order component at a random slot, the shares on all items would estimate

V_{s}+V_{sN}

V_{N}+V_{sN}

and

-V_{sN}

(Proposition 2 ). The designated slots give the first slot, which is less sensitive than the others, a fifth of the weight, so

D_{\mathrm{keep}}

is smaller (§ 3 ): in the columns for all items the slot and membership shares are overstated and the remainder is understated. In the columns for the designated items all four dissimilarities give the first slot a fifth of the weight, and the direction of the error is not fixed.

Edit	Q-4B	Q-8B	Gemma	Granite	GLM	Q-4B	Q-8B	Gemma	Granite	GLM
	Harm					Grading
Earlier verdicts from another batch	11	15	14	16	23	11	13	7	18	21
Neighbour texts from another batch	13	17	17	17	20	15	13	9	26	26
Both from another batch	9	10	11	16	15	12	9	9	22	14
Neighbour texts removed	15	12	41	27	26	12	16	13	26	17
Earlier verdicts from single-item grading	6	5	8	12	10	6	8	2	8	8

Appendix

Table 7: Target decisions changed (%) by each edit, relative to the unedited forward pass; 200 targets in two slots per task. Q-4B, Q-8B, Gemma and GLM are Qwen3-4B, Qwen3-8B, Gemma-4 and GLM-4. A decision is

P(\text{level}\geq\tau)\geq 0.5

in the next-token distribution, renormalised over the levels (§ 3 ). Texts removed : every neighbour’s text is replaced by a dash and its key is kept. One level lower leaves the verdicts at the lowest level unchanged, one level higher those at the highest level. Counts in the text are made before rounding.

Model	vs. alone	keys	between	flagged	AUROC
Qwen3-1.7B	24.0	24.2	25.4	59 $\to$ 55	0.77 $\to$ 0.72
Qwen3-4B	12.9	3.2	13.1	57 $\to$ 59	0.79 $\to$ 0.79
Qwen3-8B	12.8	1.1	12.9	65 $\to$ 64	0.80 $\to$ 0.77
Gemma-4	19.8	0.9	12.8	51 $\to$ 32	0.81 $\to$ 0.83
Granite	20.6	2.1	13.6	61 $\to$ 43	0.83 $\to$ 0.84
GLM-4	17.0	2.0	18.3	50 $\to$ 50	0.80 $\to$ 0.79

Appendix

Table 8: Harm on 2,000 random comments from Measuring Hate Speech,

k=25

: decisions changed (%) between batched and single-item grading, by renamed keys on single items, and between three random partitions; comments flagged (%, alone

\to

batched) and AUROC of the most likely level (alone

\to

batched).

Model	race	religion	origin	gender	sexuality
Qwen3-4B	62 $\to$ 64	55 $\to$ 58	61 $\to$ 62	58 $\to$ 59	53 $\to$ 55
Qwen3-8B	69 $\to$ 70	57 $\to$ 60	67 $\to$ 68	66 $\to$ 63	58 $\to$ 58
Gemma-4	58 $\to$ 35	44 $\to$ 30	49 $\to$ 28	50 $\to$ 33	50 $\to$ 36
Granite	66 $\to$ 48	54 $\to$ 41	60 $\to$ 44	64 $\to$ 43	54 $\to$ 40
GLM-4	49 $\to$ 55	50 $\to$ 49	48 $\to$ 49	55 $\to$ 51	47 $\to$ 47
Comments	654	367	341	609	404

Appendix

Table 9: Comments flagged (%, level 2 or above), alone

\to

batched, by the group that annotators say the comment targets; population study. The last row gives the number of comments per group (652 on race for Granite); the differences between groups are point estimates.

Model	Task	$T_{\omega}$	$V_{s}$	$V_{N}$	$V_{sN}$	$T_{\omega}$	$V_{s}$	$V_{N}$	$V_{sN}$
		latent scale, shares of $D_{\mathrm{new}}$ [95% CI]				decisions, shares of $F_{\mathrm{new}}$
Qwen3-4B	Harm	0.69 [0.44, 0.99]	0.09 [ $-$ 0.06, 0.22]	0.11 [ $-$ 0.03, 0.24]	0.11 [ $-$ 0.22, 0.44]	0.80	0.05	0.06	0.09
	Grading	0.51 [0.31, 0.75]	0.18 [0.06, 0.29]	0.20 [0.08, 0.32]	0.10 [ $-$ 0.17, 0.35]	0.69	0.09	0.11	0.11
Qwen3-8B	Harm	0.52 [0.34, 0.76]	0.23 [0.11, 0.32]	0.25 [0.14, 0.35]	0.00 [ $-$ 0.25, 0.25]	0.69	0.12	0.14	0.04
	Grading	0.42 [0.24, 0.67]	0.31 [0.19, 0.41]	0.09 [ $-$ 0.04, 0.21]	0.18 [ $-$ 0.09, 0.42]	0.62	0.17	0.05	0.16
Gemma-4	Harm	0.49 [0.31, 0.74]	0.32 [0.20, 0.41]	0.16 [0.03, 0.27]	0.03 [ $-$ 0.24, 0.30]	0.69	0.17	0.08	0.06

Appendix

Table 10: Components of the dissimilarity between two random partitions (Proposition 2 ): order

T_{\omega}

, slot

V_{s}

, membership

V_{N}

and the interaction of slot and membership

V_{sN}

. On the latent scale they are shares of

D_{\mathrm{new}}

; on the scale of decisions they are shares of the flip rate and need no threshold model. The order is estimated from the designated slots of “keep slot, reorder the others”, weighted as in a random partition (§ 3 ). Intervals are bootstraps over items, drawn once for the three base partitions; they ignore that items share generated prompts (§ 5.3 ). Harm and grading.

		not all equal		two majorities		replicated flips
Model	Task	obs.	pred.	obs.	pred.	obs.	pred.	pred., $V_{N}=0$
Qwen3-4B	Harm	30.8	30.4	8.9	8.9	3.0	2.8	2.2
	Grading	31.1	30.7	8.3	9.2	3.4	3.4	2.1
	Stars	8.6	8.6	2.4	2.6	0.9	1.1	0.6
Qwen3-8B	Harm	30.8	30.4	8.5	10.2	3.5	3.9	2.3
	Grading	28.0	28.3	8.6	8.3	2.5	2.5	2.1

Appendix

Table 11: Patterns of decisions (% of items), observed and predicted by the threshold model from the dissimilarities and the share of positive decisions (Proposition 7 ). Not all equal : the decisions in six batches, three reordered and three with new neighbours, which we treat as independent; it is predicted from the mean dissimilarity of the 15 pairs of these batches and their share of positive decisions, the other patterns from

D_{\mathrm{new}}

D_{\mathrm{resh}}

and the share of positive decisions in the nine batched conditions. Two majorities : the majority of the three reordered batches differs from that of the three batches with new neighbours. Replicated flips : two orders of one batch agree, two orders of another batch agree, and the batches differ; the last column sets the main effect of membership to zero by setting

D_{\mathrm{new}}

D_{\mathrm{resh}}

; setting

D_{\mathrm{resh}}

D_{\mathrm{new}}

instead changes this prediction by at most 0.42 points, and it stays below the observed share in ten of the ten pairs of harm and grading. The standard errors of observed minus predicted, from the item bootstrap with the prediction re-estimated in every resample, are 0.3–0.4, 0.6–0.8 and 0.3–0.4 points for the three patterns on harm and grading. Counts in the text are made before rounding.

Model	Task	$m=1$	$m=3$	$m=5$	$m=9$	$m\to\infty$
Qwen3-4B	Harm	13.1	9.5 [8.3, 10.9]	8.1 [6.7, 9.7]	6.8 [5.1, 8.7]	4.5 [0.0, 7.0]
	Grading	13.5	10.3 [9.0, 11.6]	9.1 [7.6, 10.5]	8.0 [6.4, 9.6]	6.3 [3.8, 8.2]
Qwen3-8B	Harm	14.8	11.5 [10.2, 12.8]	10.3 [8.8, 11.7]	9.3 [7.6, 10.8]	7.6 [5.4, 9.4]
	Grading	12.2	8.8 [7.7, 10.0]	7.4 [6.2, 8.8]	6.2 [4.7, 7.7]	3.8 [0.0, 6.1]
Gemma-4	Harm	13.2	9.8 [8.5, 11.1]	8.5 [7.0, 9.9]	7.4 [5.5, 8.9]	5.5 [2.3, 7.4]
	Grading	10.8	7.9 [6.7, 9.1]	6.7 [5.4, 8.1]	5.7 [4.1, 7.2]	3.8 [0.0, 5.8]

Appendix

Table 12: Decisions on which two batches with independent memberships (from two independent random partitions) would differ after majority voting over

m

permutations each (%), predicted by Proposition 4 from

D_{\mathrm{new}}

D_{\mathrm{resh}}

and the share of positive decisions, with 95% bootstrap intervals over items, drawn as for Table 10 ; they ignore that items share generated prompts. These predictions were made from the main experiment before the voting runs; Table 17 gives the measured flips, which lie inside these intervals at

m=9

in eight of the ten pairs (for Qwen3-8B on harm the measured value lies just below the lower end, and both round to the same value). Harm and grading.

Model	Task	obs.	pred.	obs.	pred.	obs.	pred.	maj. of 9	model	bound [95% CI]
		one batch		majority of 3		all three		mean bias
Qwen3-4B	Harm	13.7	13.7	11.6	12.0	5.0	4.8	10.7	10.2	2.15 [1.12, 3.18]
	Grading	15.6	15.7	14.2	14.1	6.2	6.3	13.2	12.7	3.34 [2.20, 4.51]
	Stars	5.7	5.8	5.3	5.4	3.0	3.0	5.3	5.2	2.04 [1.34, 2.93]
Qwen3-8B	Harm	14.0	14.0	11.4	11.8	4.2	4.0	10.0	9.4	1.03 [0.03, 2.05]
	Grading	15.0	15.0	13.3	13.7	6.7	6.6	13.4	12.4	3.92 [2.77, 5.12]

Appendix

Table 13: Disagreement with single-item grading (% of items). One batch : the decision in a random partition differs from the single-item decision. Majority of 3 , all three : the majority of the three partitions differs from it, or all three do; observed and under the threshold model (Proposition 8 ), which is fitted to the single-item and batched decisions in pairs. Mean bias : the share of items whose main batched decision differs from the single-item decision, as the majority of the nine batched decisions that we have per item, which are not independent, and under the model. Bound : the lower bound of Equation ( 4 ), rounded down to two decimals, with a percentile interval from 10,000 resamples of the items. Counts in the text are made before rounding. Table 18 measures the majority over fifteen independent batches.

Model	Task	$\Delta_{T}^{2}$	$\Delta_{V}^{2}$	$\Delta_{TV}^{2}$	$J^{2}$	ratio	corr.
Qwen3-4B	Harm	7.7	6.6	7.3	15.1	2.1	0.57
	Grading	10.3	7.7	11.3	17.2	1.9	0.54
Qwen3-8B	Harm	10.0	9.2	7.8	21.1	2.2	0.47
	Grading	8.5	8.3	6.2	20.9	2.5	0.62
Gemma-4	Harm	9.1	6.8	6.0	17.2	2.2	0.48
	Grading	4.5	2.2	4.8	2.3	0.7	0.26

Appendix

Table 14: Mean squares (

\times 100

) of the change of the target’s probability

\zeta

when the neighbours’ texts, the earlier verdicts or both are taken from another batch, and of the interaction contrast

J

. Ratio : the mean square of

J

over the mean of those of

\Delta_{T}

and

\Delta_{V}

, which is 0 if the channels add up and 2 for unrelated contexts. Corr. : correlation of

\Delta_{T}

and

\Delta_{V}

, which is 0.5 for unrelated contexts. Harm and grading.

Model	Task	level $\geq 1$	level $\geq 2$	level $\geq 3$	level $\geq 4$
Qwen3-4B	Harm	0.058 (8.9)	0.089 (13.1)	0.145 (14.7)	0.232 † (2.7)
	Grading	0.113 (12.9)	0.103 (13.4)	–	–
	Stars	0.024 (4.8)	0.013 (4.4)	0.008 (3.9)	0.019 (5.7)
Qwen3-8B	Harm	0.111 (13.7)	0.116 (14.9)	0.258 (16.2)	0.203 † (1.5)
	Grading	0.077 (12.1)	0.084 (12.1)	–	–
	Stars	0.043 (6.1)	0.012 (4.7)	0.008 (4.0)	0.028 (6.8)

Appendix

Table 15: Latent dissimilarity between two random partitions, with the flip rate in % in brackets, when the decision is that the index of the level (counted from 0) is at least 1, 2, 3 or 4. The primary decisions are

\geq 2

for harm and grading and

\geq 3

for stars. † Fewer than 5% of the decisions on one side; left out wherever thresholds are compared.

Model	Task	new	base	excess [95% CI]	vs. alone	alone	batched	$\Delta$ expected	alone	batched
						AUROC			flagged
Qwen3-14B	Harm	11.0	3.8	+7.3 [+5.9, +8.6]	12.0	0.793	0.804	$-$ 0.011	62	61
	Grading	9.1	2.8	+6.3 [+5.0, +7.6]	11.9	0.832	0.811	$-$ 0.023	36	44
Qwen3-32B	Harm	14.6	5.5	+9.1 [+7.0, +10.9]	17.0	0.760	0.815	+0.012	64	50
	Grading	10.3	4.7	+5.7 [+4.3, +6.8]	9.1	0.842	0.831	$-$ 0.022	34	35
Gemma-4-31B	Harm	8.4	2.6	+5.8 [+4.6, +6.9]	10.8	0.837	0.869	+0.012	54	45

Appendix

Table 16: Four larger models on harm and grading, on H200 NVL GPUs. New , base and excess as in Table 2 ; vs. alone : decisions that differ from single-item grading (%, mean of the three partitions). AUROC of the written level alone and in batches (mean of the three partitions), and the change of the AUROC of the expected level (the mean of the level distribution), batched minus alone; share of items above the decision threshold (%).

Model	Task	$n$	$m=1$	$m=3$	$m=5$	$m=9$	measured [95% CI]	in sample	main exp.
			flips after $m$ permutations				left after nine (% of $m=1$ )
Qwen3-4B	Harm	2,000	13.3	9.4	8.0	6.9	52 [44, 60]	52	52
	Grading	1,977	12.9	9.7	8.5	7.4	57 [50, 64]	60	59
Qwen3-8B	Harm	2,000	14.1	10.3	8.8	7.6	54 [48, 59]	57	63
	Grading	1,984	12.6	9.2	8.0	7.0	55 [48, 63]	58	51
Gemma-4	Harm	2,000	12.4	9.4	8.2	7.1	57 [50, 65]	58	56

Appendix

Table 17: Voting over permutations, measured: decisions on which the majority votes of partitions 0 and 1 over

m

of the nine orders of each batch differ (%), averaged over all sets of

m

orders of each partition, so that

m=1

averages over the 81 pairs of orders and differs from new in Table 2 ; and the share of the flips at

m=1

that is left at

m=9

: measured (95% normal intervals, with a variance clustered by the batches of both partitions), predicted by Proposition 4 from

D_{\mathrm{new}}

D_{\mathrm{resh}}

and the share of positive decisions of the same runs (in sample), and predicted from the main experiment before these runs (main exp., Table 12 ).

n

: items with a verdict in all eighteen orders.

Model	Task	$n$	$M=1$	$M=15$ [95% CI]	in sample	main exp.	$M\to\infty$	calibrated
					threshold model
Qwen3-4B	Harm	1,999	13.6	10.2 [8.7, 11.6]	10.1	10.7	10.2	10.1
	Grading	1,975	15.7	13.3 [11.8, 14.8]	13.2	13.0	12.7	9.6
Qwen3-8B	Harm	1,999	13.6	9.6 [8.3, 10.8]	10.1	10.0	9.4	9.9
	Grading	1,988	15.7	13.7 [12.2, 15.2]	13.3	12.7	12.4	13.5
Gemma-4	Harm	2,000	19.0	17.8 [15.4, 20.2]	18.6	19.3	19.3	8.9

Appendix

Table 18: Voting over batches: items whose majority over

M

independent random batches differs from the single-item decision of the same runs (%), measured, averaged over all sets of

M

of the fifteen partitions, with 95% intervals clustered by the batches of all fifteen partitions; under the threshold model at

M=15

, fitted to these runs (in sample) and to the main experiment (main exp.), and the limit of the latter as

M\to\infty

, which is the mean bias under the model in Table 13 ; and measured at

M=15

with the batched threshold chosen in every partition to match the single-item positive rate (calibrated).

n

: items with a verdict alone and in all fifteen partitions.

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 559 papers

How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models
Oct 4, 2026Laurène Vaugrante, Thilo HagendorffLLM Evaluation LLM-as-a-Judge
AI Safety Research Group, University of Stuttgart
GradeTrap: Authority Cues in Images Shift VLM Judgments Despite Explicit Instructions to Ignore Them
Sep 5, 2026Deep DessaiVLM Evaluation Bias in LLM-as-a-Judge
The University of Texas at Austin · Department of Computer Science The University of Texas at Austin
Opening LLM Judges: Recovering Preference Signals Beyond the Final Verdict
Sep 26, 2026Sourabrata Mukherjee, Sunayana SitaramLLM-as-a-Judge Bias in LLM-as-a-Judge
Microsoft Research

Oct 4, 2026J/K move · Enter open · S save

How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models

Laurène Vaugrante, Thilo Hagendorff

AI Safety Research Group, University of Stuttgart

LLM Evaluation LLM-as-a-Judge LLM Reliability

Open paper AskPDF arXiv

Researchers increasingly use Large Language Models as judges (LLM-as-a-judge) to evaluate model outputs. Yet there are no standards for how to design these judges. Typically, researchers choose the prompt, rating scale, and model intuitively. If these choices change the judge's verdicts, two studies can reach different conclusions about the same facts. To address this risk and to provide an empirical basis for judge designs, we evaluate 10 reasoning models across multiple designs on two tasks: a scalar rating of sentence sentiment and toxicity (over 500 items per category), as well as a binary accuracy classification of question-answer pairs (n=600). For the rating tasks, despite judges showing significant disagreements with the human ground truth, the practical size of differences is small enough to consider most judges reliable (mean absolute deviation of 0.11 points on a 1 - 7 scale); toxicity judges even outperform standard classifiers. Judges are also highly accurate on average (96.5%) for the accuracy classification task. However, design choices can produce shifts: changing the rating scale alone can shift measured bias by up to 0.93 points (rating task), and while accuracy levels are rarely impacted, design choices consistently impact judge leniency (classification task; leniency drop of 28.9 percentage points when using detailed prompts, and up to 56.1 percentage points when switching models). Counterintuitively, lower reasoning effort affects neither accuracy nor leniency. Across both tasks, model identity is the dominant source of variance. These findings suggest that while LLM judges are broadly trustworthy in aggregate, design choices can be meaningful sources of variance. Given the growing reliance on automated evaluation in LLM research, we intend this study as a methodological reference for designing more robust and replicable LLM-as-a-judge pipelines.

Abstract

LLM-as-a-Judge

Explore similar work

How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models

GradeTrap: Authority Cues in Images Shift VLM Judgments Despite Explicit Instructions to Ignore Them

Opening LLM Judges: Recovering Preference Signals Beyond the Final Verdict