JudgeProfile: Understanding and Steering Subjectivity in LLM Judges

Qi Cao; Kangning Liu; Xuan Kan; Shunwen Tan; Yang Pei; Dake Chen; Yatai Ji; Zixuan Ye

cs.AISep 29, 2026

JudgeProfile: Understanding and Steering Subjectivity in LLM Judges

Authors: Qi Cao, Kangning Liu, Xuan Kan, Shunwen Tan, Yang Pei, Dake Chen, Yatai Ji, Zixuan Ye, +5 more

Organizations: Meta · University of California San Diego · The University of Hong Kong · The Hong Kong University of Science and Technology

Abstract

LLM judges are inherently subjective, often favoring different responses in pairwise comparison when neither option is objectively wrong. To study this subjectivity, we introduce JudgeProfile, a framework that dissects LLM evaluation into perception (how a judge compares two responses across specific attributes like clarity, correctness, and detail) and prioritization (how much each attribute influences the final choice). We curate SubjectiveSet, a dataset of 50,013 response pairs from 17 public data sources, evaluated by 21 LLM judges across 87 attributes. We find a hidden consensus in perception: judges frequently agree on attribute judgments even when their overall choices diverge. Building on this separation, we first characterize each judge's prioritization using attribute weights estimated from its own overall choices. These weights differ across judges even when estimated from the same attribute judgments. We then learn new weights from reference labels to adapt their decisions to a target evaluation standard. Reweighting perceived attributes improves average held-out agreement with reference labels from 66.48% to 71.97%, outperforming fine-tuning and rubric prompting. Our findings show that understanding and steering the subjectivity of LLM judges requires attention not only to what they perceive, but also to how they prioritize it.

Figures & tables

Figure 1: Perception and prioritization underlying LLM judgment. Given a query and two responses, attribute-level comparisons

\mathbf{d}_{j}

describe how the responses differ. Prioritization assigns relative importance to these differences through weights

\mathbf{w}_{j}

, modeling an overall preference

y_{j}

Figure 2: Evaluation setup. 50,013 response pairs from 17 public distributions, 21 LLM judges ranging from 0.8B to 2.8T total parameters, and 87 response attributes. Independent evaluation of each attribute in both response orders yields 182.7 million attribute-level inference calls.

Figure 3: Judges agree on attributes despite different overall choices. Agreement for 171 pairs of 19 judges: overall choices (a), and attributes when winners are the same (b) or different (c). We include an attribute comparison only if each judge makes a definite choice of the same response in both presentation orders. Heatmaps share a color scale; inset distributions have separate vertical scales.

Figure 4: Shared attribute judgments, different winners. Judges agree that workout response B is more specific and A is more insightful, but disagree on which is better overall.

Figure 8: Reweighting improves alignment across judges and distributions. (a) Mean accuracy gains over all 17 datasets and the 11 preference-labeled datasets; judges are ordered by the latter gain. (b) Mean gains across 21 judges for each dataset.

Figure 9: Useful priorities can be learned from few labels. Held-out accuracy on three datasets. Blue curves average judges; shading shows

\pm

standard deviation over ten training-subset resamples. Dashed lines show unadapted judges; diamonds use all training pairs. Open circles mark the first budget after which every later tested doubling gains less than 0.01 accuracy.

	Qwen3.5-4B		Qwen3.5-9B		Qwen3.5-27B		Llama-4-Scout
Method	Acc.	$\Delta$	Acc.	$\Delta$	Acc.	$\Delta$	Acc.	$\Delta$
Unadapted	63.79	–	66.74	–	70.79	–	66.91	–
SFT-256	65.70	+1.91	66.43	-0.31	70.96	+0.17	64.81	-2.10
Reweighting-32	66.02	+2.23	67.21	+0.47	69.96	-0.83	66.67	-0.24
Reweighting-64	65.98	+2.18	67.83	+1.09	70.55	-0.25	68.35	+1.44
Reweighting-128	66.72	+2.93	68.24	+1.50	71.87	+1.08	68.98	+2.07

Table 2: Reweighting versus fine-tuning. Accuracy is in percent;

\Delta

is the gain in percentage points. Run details and parsing rules: Appendix F .

Judge	Method	Accuracy	$\Delta$	Weight alignment	Linear explainability
Qwen3.5-4B	Unadapted	63.79	–	58.89	77.10
	Rubric prompting (6)	66.83	+3.03	54.20	74.20
	Reweighting (6)	68.58	+4.79	57.30	–
	Reweighting (12)	69.15	+5.36	67.12	–
	Reweighting (87)	69.49	+5.69	100.00	–
Qwen3.5-9B	Unadapted	66.74	–	60.46	80.18

Table 3: Rubric prompting versus reweighting. For each judge and source, reweighting uses the

k

attributes with the largest absolute weights from the full 87-attribute training fit;

k=6

uses the same attributes as the rubric. Bold and underlined values mark the best and second-best accuracy and gains for each judge. Weight alignment is cosine similarity to the full 87-attribute target, expressed as a percentage (100 for the full fit by definition). Linear explainability measures how well fixed attribute judgments predict overall judgments. Protocol: Appendix F .

Figure 10: Different judge priorities and dataset associations. (a) Normalized fitted weights under a shared attribute representation; each dot is one judge. (b) Median agreement with reference labels across eligible readers when selecting the response with more of an attribute; each dot is one dataset. Vertical offsets only separate overlapping points. Four selected attributes are shown per panel. Full heatmaps and protocols: Appendix C.8 .

Appendix figures & tables46 assets

Supplementary material from the paper’s appendix.

Appendix

Analysis	Population	Comparison or prediction target
Heatmap consensus	19 LLMs	Full SubjectiveSet; 171 judge pairs in Figure 3
Collection and subset consensus	21 LLMs	Full collection, SubjectiveSet–LLM, and its complement; 210 judge pairs
Human cases	7 raters	SubjectiveSet–Human: 21 disagreement cases and 2 controls
Priority profiles	19 LLMs	Each judge’s own verdicts, using own or shared attributes
Attribute importance and consensus	21 LLMs	Own-verdict weights and attribute agreement; Figure 5
No-conflict comparison	21 LLMs	Overall disagreement after requiring at least 20 shared attribute directions; Figure 7

Appendix

Table 4: Each analysis addresses a different part of judgment. Judge counts and evaluation targets distinguish the study populations.

Panel of Figure 11 — Figure 11: Two types of reference labels in SubjectiveSet. Real examples from Arena and RewardBench; responses are summarized for space. Counts show how many of the 21 LLM judges identify the indicated response as higher on an attribute. Full responses, source identifiers, and judgments appear in Cases 8 and 21 in Appendix G .

Source distribution	Label type	Pairs	Train	Test
ScalerLab/JudgeBench ( Tan et al., 2024 )	Benchmark	617	463	154
IF-RewardBench ( Wen et al., 2026 )	Benchmark	807	605	202
THU-KEG/RM-Bench ( Liu et al., 2024 )	Benchmark	1,146	860	286
allenai/reward-bench-2 ( Malik et al., 2026 )	Benchmark	1,657	1,243	414
lmarena/arena-expert-5k ( LMArena, n.d.a )	Human	1,797	1,348	449
berkeley-nest/Nectar ( Zhu et al., 2023 )	AI	2,500	1,875	625

Appendix

Table 5: The 17 source distributions used in our study. “Benchmark” denotes the benchmark-provided label category, distinct from human and AI preferences. Training and test counts include only non-tie pairs with a usable

A/B

reference label.

Summarization		1,223
WebGPT		1,117
HH-RLHF		1,114
HelpSteer2		1,035
PPE–Human		992
Arena–140K		915
MT-Bench		904

Appendix

Figure 12: SubjectiveSet–LLM by source. The 11,524 selected response pairs across 17 sources. Bar lengths use a shared scale; numbers give pair counts.

Judges	Items	$n$	Overall	All	Same winner	Opp. winner	$\Delta$
				Attribute
LLMs (21)	All pairs	50,013	74.0	88.2	89.5	83.6	$+14.2$
	SubjectiveSet–LLM	11,524	49.8	84.3	85.8	82.9	$+34.5$
	Remaining pairs	38,489	81.1	89.0	89.9	84.0	$+7.9$
Humans (7)	SubjectiveSet–Human	23	62.1	92.3	92.2	89.9	$+30.2$
	Disagreement cases	21	57.6	92.8	93.3	89.9	$+35.2$

Appendix

Table 6: Full agreement breakdown by collection and winner stratum. Values are percentages;

\Delta

is all-attribute minus overall agreement. LLM rates average judge pairs; human aggregate rates pool eligible comparisons. The human disagreement and control rows partition the 23-case SubjectiveSet–Human set.

Judge	Family	Parameters	Accuracy (%)
Qwen3.5-0.8B † ( Qwen Team, 2026a )	Qwen	0.8B	47.90
Qwen3.5-2B † ( Qwen Team, 2026a )	Qwen	2B	59.30
Qwen3.5-4B ( Qwen Team, 2026a )	Qwen	4B	63.79
DeepSeek-R1-Distill-7B ( DeepSeek-AI, 2025 )	DeepSeek	7B	58.74
Qwen3.5-9B ( Qwen Team, 2026a )	Qwen	9B	66.74
DeepSeek-R1-Distill-14B ( DeepSeek-AI, 2025 )	DeepSeek	14B	64.54

Appendix

Table 7: All 21 judges in the evaluation collection. Parameter counts are total parameters. Accuracy is the unadapted judge’s reference-label accuracy (%) on the common 11,765-pair test set, with unparsed verdicts counted as errors. Models marked

\dagger

are excluded from the main 19-judge agreement analyses. The Mistral run is marked

\ddagger

because its metadata contains an extra_system_prompt flag.

Family	Count	Attributes
Alignment	6	harmfulness, moralizing tendency, neutrality, refusal tendency, safety conservatism, sycophancy
Communication	8	accessibility, clarity, coherence, fluency, notation clarity, organization, readability, terminological precision
Content	1	elaboration
Correctness	11	assumption validity, constraint satisfaction, factual accuracy, final answer correctness, groundedness, hallucination rate, instruction adherence, internal consistency, justification quality, logical validity, step correctness
Creative	9	analogy use, creativity, humor, interestingness, novelty, originality, persuasiveness, storytelling quality, vividness
Epistemic	7	acknowledgment of limitations, assertiveness, calibration, confidence, hedging, qualification, uncertainty expression

Appendix

Table 8: Complete inventory of the 87 response attributes. Spaces replace underscores in the attribute identifiers for readability. “Other” in the overview figure comprises the four families containing one attribute each.

Judges	Read	Pairs	Same agr.	Same cov.	Opp. agr.	Opp. cov.
19	stable	171	0.919	0.304	0.859	0.220
19	single	171	0.785	0.529	0.695	0.480
21	stable	210	0.895	0.260	0.836	0.190
21	single	210	0.732	0.552	0.685	0.511

Appendix

Table 9: Judge-pair strata under stable and single-read protocols. Coverage is the fraction of jointly resolved item–attribute cells. The 21-judge sensitivity analysis includes the two smallest models.

Judge	Same agr.	Same cov.	Opp. agr.	Opp. cov.
Qwen3.5-27B	0.945	0.298	0.897	0.216
V4-Flash	0.945	0.264	0.897	0.177
Qwen3.5-122B	0.935	0.322	0.889	0.245
Llama-4-Scout	0.937	0.295	0.885	0.185
Qwen3.6-27B	0.935	0.342	0.882	0.257
Gemma-4-26B	0.926	0.306	0.879	0.236

Appendix

Table 10: Stable attribute agreement and joint coverage for each judge, averaged over its 18 partners. R1 and V4 denote DeepSeek models.

Distribution	Pairs	Same agr.	Same cov.	Opp. agr.	Opp. cov.
MT-Bench	4,000	0.937	0.311	0.897	0.229
SHP	4,000	0.928	0.411	0.874	0.293
RewardBench	4,481	0.930	0.351	0.873	0.279
Arena-Human	4,000	0.918	0.325	0.869	0.248
Nectar	2,500	0.923	0.307	0.867	0.234
HH-RLHF	4,000	0.907	0.329	0.867	0.276

Appendix

Table 11: Stable conditional agreement by distribution. Counts are distinct response pairs before verdict filtering; coverage conditions on the corresponding winner stratum.

Judge	Items	Preference	All 87	Core 20	Cov. 87
R1-7B	44,590	0.665	0.833	0.872	0.211
R1-14B	45,796	0.744	0.899	0.928	0.299
R1-32B	45,220	0.767	0.906	0.934	0.314
V4-Flash	44,358	0.784	0.936	0.957	0.260
V4-Pro	46,048	0.795	0.907	0.945	0.302
Gemma-3-27B	46,970	0.755	0.913	0.943	0.341

Appendix

Table 12: Per-judge agreement on matched items under the stable protocol. Attribute agreement remains conditional on jointly resolved directions. The final row is the mean over all 171 judge pairs.

Population / protocol	Attributes	Preference	Attribute	$\Delta$
LLMs / stable	All 87	77.1	90.7	+13.6
	Core 20	77.1	93.7	+16.6
LLMs / single read	All 87	76.7	76.5	-0.2
	Core 20	76.7	81.0	+4.3
Humans / selected cases	2–3 per item	62.1	92.3	+30.2

Appendix

Table 13: Stable attributes and selected human attributes show higher agreement than overall preferences. All values are percentages;

\Delta

is attribute minus preference agreement, in percentage points. LLM comparisons use matched items within each protocol. Human agreement pools seven raters on 23 selected cases, using two or three attributes per item. All attribute rates condition on jointly resolved directions. The single-read all-attribute control does not show an advantage. Core attributes are selected for within-judge consistency and coverage.

Stratum	Stable agr.	Stable cov.	Single agr.	Observations
Both match reference	0.922	0.322	0.793	4,436,700
Both depart from reference	0.909	0.266	0.762	1,436,019
Opposite preferences	0.859	0.220	0.695	1,888,438

Appendix

Table 14: Difficulty-related strata. Counts sum item observations over judge pairs.

Subset	Items	Preference agreement	Attribute agreement
Disagreement witnesses	21	0.576 (204/354)	0.928 (942/1,015)
Controls	2	1.000 (42/42)	0.847 (61/72)
Pooled	23	0.621 (246/396)	0.923 (1,003/1,087)

Appendix

Table 15: Human pairwise agreement. Parentheses give agreeing/eligible comparisons; these comparisons are not independent samples.

Attribute	Resolve	Self	All agr.	All cov.	Opp. agr.	Opp. cov.
verbosity	0.701	0.762	0.965	0.558	0.962	0.517
moralizing tendency	0.225	0.816	0.959	0.128	0.959	0.121
technicality	0.466	0.827	0.965	0.323	0.956	0.266
evidence use	0.234	0.825	0.960	0.127	0.954	0.109
self correction	0.089	0.828	0.928	0.021	0.950	0.018
conceptual complexity ∗	0.541	0.819	0.956	0.381	0.948	0.314

Appendix

Table 16: Complete attribute summaries, part 1 of 2. All and opposite-winner (Opp.) rates use stable directions.

Attribute	Resolve	Self	All agr.	All cov.	Opp. agr.	Opp. cov.
sycophancy	0.146	0.689	0.857	0.052	0.866	0.050
confidence	0.504	0.728	0.886	0.320	0.864	0.257
personalization	0.228	0.767	0.918	0.106	0.864	0.084
insightfulness	0.476	0.777	0.924	0.301	0.861	0.225
persuasiveness	0.518	0.776	0.922	0.337	0.861	0.253
difficulty	0.483	0.735	0.861	0.307	0.860	0.261

Appendix

Table 17: Complete attribute summaries, part 2 of 2. All and opposite-winner (Opp.) rates use stable directions.

Profile / metric	Observed	Null mean	Corrected
Perception / Pearson	0.765	1.000	0.765
Prioritization / Pearson	0.609	0.944	0.645
Perception / Spearman	0.757	0.999	0.757
Prioritization / Spearman	0.462	0.847	0.545

Appendix

Table 18: Profile similarity with distinct estimation-noise references. Null-corrected values use unrounded source estimates.

Judge	Raw half r	SB	Shared SB	Own CV	Shared CV
R1-7B	0.868	0.929	0.923	0.718	0.706
R1-14B	0.905	0.950	0.871	0.805	0.798
R1-32B	0.920	0.958	0.900	0.824	0.815
V4-Flash	0.923	0.960	0.897	0.796	0.823
V4-Pro	0.924	0.960	0.872	0.836	0.845
Gemma-3-27B	0.837	0.911	0.930	0.846	0.836

Appendix

Table 19: Priority-model reproducibility and five-fold prediction accuracy. Raw half correlations and Spearman–Brown (SB) reliability are distinct quantities. CV columns predict each judge’s own preferences.

Attribute	Observed SD	Null mean SD	Null q95	Ratio
strategy quality	1.874	0.316	0.400	5.93
harmfulness	2.466	0.479	0.628	5.15
calibration	1.769	0.353	0.451	5.01
relevance	1.501	0.339	0.419	4.42
final answer correctness	1.600	0.373	0.490	4.28
assumption validity	1.243	0.299	0.414	4.15

Appendix

Table 20: Detailed null comparison for the 15 selected attributes. Ratio divides observed SD by mean null SD, not by the null 95th percentile.

Distribution	P r	P ceiling	P ratio	W r	W ceiling	W ratio
Nectar	0.772	0.998	0.773	0.305	0.525	0.581
HelpSteer2	0.770	0.999	0.771	0.320	0.599	0.534
RewardBench	0.766	0.999	0.767	0.393	0.702	0.560
MT-Bench	0.763	0.999	0.764	0.175	0.780	0.224
UltraFeedback	0.762	0.998	0.763	0.238	0.410	0.582
WebGPT	0.762	0.999	0.763	0.170	0.384	0.443

Appendix

Table 21: Distribution-specific profile correlations and split-half ceilings. Ratios are computed before rounding.

Figure 13: Shared resolution patterns and reproducible differences in priorities. (a) Raw cross-judge profile correlations for all 171 judge pairs; perception profiles contain stable attribute resolution rates. Global null-corrected Pearson correlations are 0.765 for perception and 0.645 for priorities. (b) Correlations divided by their separately estimated split-half ceilings within each of 17 distributions. This is a different correction from the global null. (c) For each judge, raw split-half priority correlation versus its mean cross-judge priority correlation. Dashed lines indicate equality. Each point represents an observed pair, distribution, or judge, respectively.

Figure 14: Shared measurements reveal different priorities. Normalized logistic coefficients for 19 judges on the same attribute representation; 13 attributes selected by combined importance and variation. Purple/teal indicate negative/positive conditional associations. Right: individual judges and median ticks. Selection, normalization, and the 21-judge check are in Appendix C.8 .

Figure 15: Attribute–label associations depend on the distribution. Each cell is median accuracy across eligible readers when choosing the response with more of one attribute (21 readers; 15 attributes, 17 distributions). Orange/blue indicate below/above 50%. The pooled column hides source-specific reversals. Sources are ordered by verbosity association, with the style-confounded RM-Bench subset ( $\dagger$ ) separated at right. Coverage and full pooled statistics: Appendix C.8 .

Figure 16: Shared-representation priorities with all 21 judges. The selection and normalization procedure matches Figure 14 . Asterisks mark the two judges with low stable coverage. Dots show individual normalized weights; black ticks show medians.

Attribute	Coverage	Accuracy	Strength	Sign
Final answer correctness	34.9	78.8	0.288	$+$
Constraint satisfaction	16.0	78.4	0.284	$+$
Step correctness	27.0	76.0	0.260	$+$
Strategy quality	54.1	75.3	0.253	$+$
Calibration	37.7	75.2	0.252	$+$
Logical validity	41.9	74.9	0.249	$+$

Appendix

Table 22: Pooled single-attribute associations. Coverage and accuracy are percentages; strength is median

|\mathrm{accuracy}-0.5|

. All values condition on a stable direction.

Attribute	Coverage	Accuracy	Strength	Sign
Assertiveness	48.7	64.9	0.149	$+$
Creativity	38.2	64.8	0.148	$+$
Depth	59.2	64.7	0.147	$+$
Breadth	56.2	64.7	0.147	$+$
Elaboration	64.9	64.7	0.147	$+$
Qualification	48.2	64.6	0.146	$+$

Appendix

Table 23: Pooled single-attribute associations (continued). Coverage and accuracy are percentages; strength is median

|\mathrm{accuracy}-0.5|

. All values condition on a stable direction.

Attribute	Own perception	Shared perception
Final answer correctness	4.29	3.26
Strategy quality	3.20	3.88
Instruction adherence	2.45	2.35
Constraint satisfaction	2.32	2.53
Calibration	1.92	3.26
Insightfulness	1.39	1.82

Appendix

Table 24: Selected coefficient summaries across 19 judges. Values are mean normalized coefficients with each judge’s own readings or a common majority-based reading. Attributes are selected to illustrate both shared positive associations and penalties.

Judge	First	Second	Third
R1-7B	Constraints (0.21)	Correctness (0.18)	step correctness (0.14)
R1-14B	Correctness (0.26)	Empathy (0.23)	Strategy (0.19)
R1-32B	Correctness (0.29)	Strategy (0.26)	Empathy (0.24)
V4-Flash	Strategy (0.36)	Constraints (0.20)	Empathy (0.20)
V4-Pro	Strategy (0.28)	Calibration (0.26)	robustness (0.23)
Gemma-3-27B	Constraints (0.31)	Strategy (0.29)	Empathy (0.29)

Appendix

Table 25: Three largest positive coefficients for each judge in the shared-perception fit. Coefficients are raw, rounded to two decimal places; tied entries are retained in source order. Correctness abbreviates final-answer correctness, constraints abbreviates constraint satisfaction, and instruction abbreviates instruction adherence. All 87 attributes were eligible for this ranking. R1 and V4 denote DeepSeek models.

Attribute group	Attributes	Agreement	Coverage
Q1 (lowest fitted importance)	22	87.1	23.6
Q2	22	82.4	20.8
Q3	21	82.4	16.9
Q4 (highest fitted importance)	22	76.8	14.7
Final-answer correctness	1	66.0	9.9
Strategy quality	1	68.4	21.6

Appendix

Table 26: Hidden consensus is uneven across attributes. Opposite-winner agreement and joint stable coverage (%) for all 21 judges. Quartiles are defined by fitted importance; each summary averages attributes within its group after averaging judge pairs equally. The selected attributes illustrate both lower-consensus qualities and high-consensus exceptions.

Family	Attributes	Importance	Agreement	Coverage
correctness	11	1.611	0.722	0.133
interpersonal	1	0.419	0.748	0.225
communication	8	0.882	0.777	0.204
relevance	5	0.979	0.779	0.231
reasoning	9	1.214	0.810	0.173
stance	1	0.428	0.819	0.170

Appendix

Table 27: Attribute consensus by family under opposite overall winners. Importance is mean normalized coefficient magnitude.

Attribute	Importance	All	Same	Opposite	Coverage
harmfulness	5.373	0.921	0.922	0.915	0.050
final answer correctness	3.944	0.881	0.919	0.660	0.099
strategy quality	2.958	0.873	0.910	0.684	0.216
constraint satisfaction	2.490	0.873	0.905	0.705	0.029
hallucination rate	2.304	0.890	0.888	0.892	0.136
instruction adherence	2.281	0.883	0.909	0.746	0.194

Appendix

Table 28: All 87 attributes, ordered by fitted importance (part 1 of 3). Agreement: all items, same winner, opposite winners; coverage: opposite-winner items.

Attribute	Importance	All	Same	Opposite	Coverage
internal consistency	0.967	0.760	0.791	0.596	0.057
respectfulness	0.949	0.865	0.893	0.722	0.104
persuasiveness	0.943	0.903	0.918	0.845	0.213
interestingness	0.905	0.901	0.909	0.875	0.323
neutrality	0.834	0.843	0.863	0.748	0.045
confidence	0.830	0.870	0.878	0.844	0.218

Appendix

Table 29: All 87 attributes, ordered by fitted importance (part 2 of 3). Agreement: all items, same winner, opposite winners; coverage: opposite-winner items.

Attribute	Importance	All	Same	Opposite	Coverage
decomposition quality	0.649	0.914	0.924	0.877	0.216
detail level	0.648	0.939	0.944	0.925	0.441
qualification	0.644	0.909	0.919	0.876	0.249
assertiveness	0.642	0.864	0.869	0.850	0.238
conversationality	0.623	0.854	0.857	0.847	0.242
novelty	0.603	0.879	0.883	0.867	0.097

Appendix

Table 30: All 87 attributes, ordered by fitted importance (part 3 of 3). Agreement: all items, same winner, opposite winners; coverage: opposite-winner items.

Judges	Condition	Items/pair	Pair mean	Observation-weighted
21	No conflict	5,245	14.5	11.5
21	At least one conflict	17,329	21.8	19.7
19	No conflict	—	$\approx 12.0$	$\approx 11.1$

Appendix

Table 31: Measured attribute consensus leaves overall disagreement. All 87 attributes, at least 20 jointly resolved directions, and definite overall verdicts. Percentages report different winners, not different attribute directions. Counts are mean items per judge pair. The 19-judge sensitivity estimates use rounded per-pair summaries.

$k$	Eligible	No conflict	Conflict	Pair mean	Obs.-weighted
5	36,403	10,948	25,454	20.8	18.2
10	30,953	8,309	22,644	18.4	15.5
20	22,575	5,245	17,329	14.5	11.5
30	15,305	3,204	12,101	10.9	8.4
40	8,728	1,748	6,979	7.5	5.7

Appendix

Table 32: Full 87-attribute inventory: coverage thresholds and remaining winner disagreement. Counts are mean items per judge pair; disagreement is in percent.

$k$	Eligible	No conflict	Conflict	Pair mean	Obs.-weighted
1	28,814	22,684	6,131	18.4	16.0
3	13,936	10,732	3,204	11.6	8.1
5	5,402	4,116	1,287	8.3	4.5
7	873	581	292	6.1	3.3

Appendix

Table 33: Top ten attributes by fitted importance: coverage thresholds and remaining winner disagreement. Counts are mean items per judge pair; disagreement is in percent.

Judge	No-conflict items/pair	Winner disagreement
Qwen3.5-0.8B	762	35.8
DeepSeek-R1-Distill-14B	3,794	15.7
DeepSeek-R1-Distill-32B	3,853	15.2
DeepSeek-R1-Distill-7B	1,965	15.1
Qwen3.5-2B	1,004	14.9
Muse-Glimmer-30B	2,257	14.8

Appendix

Table 34: No-conflict results by judge, full inventory and

k=20

. Each row averages that judge’s 20 partner pairs. Disagreement is in percent.

Source	No-conflict items/pair	Winner disagreement
IF-RewardBench	44	27.8
hh-rlhf	428	19.0
arena-140k	459	18.7
webgpt	442	17.6
RM-Bench	100	17.1
PPE-Human	443	16.6

Appendix

Table 35: No-conflict results by source, full inventory and

k=20

. Each row averages 210 judge pairs. Disagreement is in percent.

Judge	Base	Reweighted	Gain	Preference gain	Gain excl. RM
Kimi-K3	74.45	76.96	+2.51	+4.06	+1.66
Gemma-4-31B	73.25	77.18	+3.93	+4.83	+3.04
Qwen3.6-27B	73.08	76.64	+3.55	+4.05	+2.58
Qwen3.5-27B	72.16	75.18	+3.01	+4.15	+1.86
DeepSeek-V4-Pro	69.12	73.05	+3.92	+2.35	+2.11
Gemma-4-26B-A4B	70.46	75.39	+4.92	+5.88	+3.90

Appendix

Table 36: Judge-level macro results. Accuracies are percentages; gains are percentage points. Base, reweighted, and gain use all 17 datasets; preference gain uses only the 11 preference-labeled datasets. The final column uses all 16 datasets other than RM-Bench. Each included dataset receives equal weight. Full model names follow the evaluation model inventory.

Dataset	Base	Reweighted	Gain	Min / max gain
RM-Bench	65.05	94.44	+29.37	+15.4 / +64.3
MT-Human	62.06	78.54	+16.50	+3.1 / +29.4
Nectar	64.47	75.09	+10.62	+2.1 / +16.6
SHP	58.06	66.85	+8.79	+2.1 / +13.5
PPE-BoK	66.64	74.49	+7.84	+0.2 / +19.5
Summarization	60.33	66.61	+6.29	+0.6 / +27.4

Appendix

Table 37: Dataset-level macro results across 21 judges. Gain range describes between-judge variation, not estimation uncertainty.

Judge	JudgeBench	IF-RB	RM-Bench	RB2	Arena-Expert	UltraFeedback
Kimi-K3	$93.5\!\rightarrow\!80.5$	$73.8\!\rightarrow\!65.8$	$83.6\!\rightarrow\!99.7$	$94.4\!\rightarrow\!93.7$	$59.0\!\rightarrow\!62.4$	$78.6\!\rightarrow\!80.0$
Gemma-4-31B	$88.3\!\rightarrow\!77.3$	$69.3\!\rightarrow\!69.3$	$81.5\!\rightarrow\!99.7$	$94.0\!\rightarrow\!94.7$	$66.4\!\rightarrow\!63.5$	$79.0\!\rightarrow\!80.8$
Qwen3.6-27B	$90.9\!\rightarrow\!87.0$	$65.3\!\rightarrow\!62.4$	$80.1\!\rightarrow\!99.3$	$93.5\!\rightarrow\!91.5$	$63.3\!\rightarrow\!61.5$	$79.8\!\rightarrow\!81.0$
Qwen3.5-27B	$88.3\!\rightarrow\!79.2$	$66.8\!\rightarrow\!57.4$	$78.3\!\rightarrow\!99.7$	$92.0\!\rightarrow\!91.5$	$62.6\!\rightarrow\!61.9$	$79.5\!\rightarrow\!80.2$
DeepSeek-V4-Pro	$71.4\!\rightarrow\!66.9$	$53.0\!\rightarrow\!55.9$	$62.6\!\rightarrow\!95.5$	$89.4\!\rightarrow\!90.6$	$61.9\!\rightarrow\!62.4$	$77.0\!\rightarrow\!77.8$
Gemma-4-26B-A4B	$89.6\!\rightarrow\!75.3$	$57.4\!\rightarrow\!60.9$	$78.3\!\rightarrow\!99.7$	$88.9\!\rightarrow\!93.5$	$62.4\!\rightarrow\!62.1$	$76.8\!\rightarrow\!78.2$

Appendix

Table 38: Complete accuracy results, panel 1 of 3.

Judge	Nectar	PPE-BoK	HelpSteer2	WebGPT	MT-Human	SHP
Kimi-K3	$60.5\!\rightarrow\!75.2$	$80.9\!\rightarrow\!84.4$	$76.0\!\rightarrow\!77.8$	$69.0\!\rightarrow\!69.3$	$72.8\!\rightarrow\!81.2$	$59.6\!\rightarrow\!68.9$
Gemma-4-31B	$60.0\!\rightarrow\!76.6$	$79.6\!\rightarrow\!84.1$	$76.4\!\rightarrow\!78.2$	$69.3\!\rightarrow\!70.0$	$63.0\!\rightarrow\!80.3$	$58.2\!\rightarrow\!69.8$
Qwen3.6-27B	$62.6\!\rightarrow\!73.3$	$78.6\!\rightarrow\!84.8$	$76.2\!\rightarrow\!76.9$	$69.7\!\rightarrow\!70.8$	$66.8\!\rightarrow\!81.4$	$61.4\!\rightarrow\!68.8$
Qwen3.5-27B	$61.9\!\rightarrow\!75.8$	$74.3\!\rightarrow\!79.3$	$76.4\!\rightarrow\!74.6$	$70.4\!\rightarrow\!69.2$	$62.6\!\rightarrow\!81.4$	$58.9\!\rightarrow\!67.6$
DeepSeek-V4-Pro	$69.0\!\rightarrow\!74.6$	$63.0\!\rightarrow\!71.8$	$74.6\!\rightarrow\!72.4$	$69.7\!\rightarrow\!69.0$	$70.2\!\rightarrow\!79.9$	$60.2\!\rightarrow\!68.1$
Gemma-4-26B-A4B	$64.6\!\rightarrow\!74.4$	$75.9\!\rightarrow\!79.6$	$74.2\!\rightarrow\!76.4$	$67.9\!\rightarrow\!67.5$	$55.4\!\rightarrow\!80.1$	$57.7\!\rightarrow\!68.2$

Appendix

Table 39: Complete accuracy results, panel 2 of 3.

Judge	HH-RLHF	Arena-Human	Summarization	PPE-Human	RewardBench
Kimi-K3	$64.2\!\rightarrow\!65.1$	$64.0\!\rightarrow\!66.9$	$72.9\!\rightarrow\!73.5$	$69.7\!\rightarrow\!70.8$	$93.1\!\rightarrow\!93.1$
Gemma-4-31B	$63.4\!\rightarrow\!65.5$	$65.4\!\rightarrow\!66.9$	$70.5\!\rightarrow\!72.4$	$68.8\!\rightarrow\!69.4$	$92.1\!\rightarrow\!93.5$
Qwen3.6-27B	$61.6\!\rightarrow\!65.2$	$64.5\!\rightarrow\!67.2$	$69.2\!\rightarrow\!71.2$	$66.4\!\rightarrow\!68.8$	$92.4\!\rightarrow\!91.7$
Qwen3.5-27B	$62.6\!\rightarrow\!66.8$	$65.1\!\rightarrow\!65.5$	$67.2\!\rightarrow\!68.9$	$67.2\!\rightarrow\!68.3$	$92.6\!\rightarrow\!90.8$
DeepSeek-V4-Pro	$62.7\!\rightarrow\!64.3$	$63.8\!\rightarrow\!64.5$	$67.6\!\rightarrow\!69.2$	$66.7\!\rightarrow\!67.2$	$92.3\!\rightarrow\!91.7$
Gemma-4-26B-A4B	$61.7\!\rightarrow\!66.5$	$62.2\!\rightarrow\!66.1$	$65.7\!\rightarrow\!71.5$	$67.4\!\rightarrow\!69.7$	$91.7\!\rightarrow\!91.9$

Appendix

Table 40: Complete accuracy results, panel 3 of 3.

Figure 17: All 357 reported accuracy gains, in percentage points. The diverging color scale is symmetric around zero and does not clip extreme gains. Gains are rounded to one decimal place.

Judge	Smallest budget beating SFT-256	RW256 minus SFT256	RW Full minus SFT Full	RW6 minus rubric6
Qwen3.5-4B	32	+2.08	-3.45	+1.75
Qwen3.5-9B	32	+4.37	-1.92	+2.80
Qwen3.5-27B	128	+1.50	-1.36	-0.07
Llama-4-Scout	32	+5.35	+3.56	+2.62

Appendix

Table 41: Budget and prompting contrasts, in percentage points except for the budget column. Positive differences favor reweighting. The final column compares six-attribute reweighting with six-criterion rubric prompting.

Judge	$\Delta$ alignment	$\Delta$ explainability
Qwen3.5-4B	-4.69	-2.90
Qwen3.5-9B	+13.80	-3.45
Qwen3.5-27B	+20.32	-2.59
Llama-4-Scout	+1.98	-3.21

Appendix

Table 42: Changes in rubric diagnostics relative to the unadapted condition, computed from the displayed main-table entries. Both diagnostics use the original attribute judgments.

Explore similar work

SenseJudge: Human-Centric Preference-Driven Judgment Framework
Jun 2, 2026Rui Li, Junfeng Liu, Xiangwen Kong +2Llm-As-A-Judge Large Language Model Decisions
State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University · StepFun · Xi’an Jiaotong University
FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge
Feb 6, 2026Bo Yang, Lanfei Feng, Yunkui Chen +3Llm-As-A-Judge Raw Judge Outputs
College of Computer Science, Zhejiang University, Hangzhou, China · School of Software Technology, Zhejiang University, Hangzhou, China
Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration
Sep 14, 2026Gemma Zhang, Prachi Badarayani, Asmi Kumar +2Llm-As-A-Judge Large Language Model Judges
Microsoft · Massachusetts Institute of Technology

Jun 2, 2026, cs.CLJ/K move · Enter open · S save

SenseJudge: Human-Centric Preference-Driven Judgment Framework

Rui Li, Junfeng Liu, Xiangwen Kong, Linhai Xu+1

State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University · StepFun · Xi’an Jiaotong University

Llm-As-A-Judge Large Language Model Decisions Raw Judge Outputs

Open paper AskPDF arXiv

Large Language Models (LLMs) as judges across various scenarios such as assessing model responses is becoming an increasingly accepted paradigm. However, existing judgment approaches often rely on trained judgers using fixed preference data, which tend to overlook diverse user preferences and struggle to adapt to real-world human-AI dialogue scenarios. To address these limitations, we propose SenseJudge, a customizable judgment framework driven by human preferences and SenseBench, a diverse and challenging instruction-following benchmark derived from real-world multi-turn interactions. We applied the automatic judgment framework and benchmark to two tasks: (1) LLMs as personalized judges, and (2) model ranking. We conducted extensive experiments, and the results demonstrate that the SenseJudge framework surpasses other judgment methods and models in the LLMs-as-personalized-judges task and achieves model ranking that aligns with real human sense. Additionally, we conducted analyses on position bias and consistency, alongside ablation studies, which affirmed the robustness of SenseJudge.

Abstract

Explore similar work

SenseJudge: Human-Centric Preference-Driven Judgment Framework

FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge

Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration