Multi-Dimensional Comparative Scale Construction for Efficient Personalized Subjective Judgment in High-Traffic Applications

Xianglong Shi; Shifeng Liu; Sirui Zhao; Shengming Yuan; Enhong Chen

cs.AISep 27, 2026

Multi-Dimensional Comparative Scale Construction for Efficient Personalized Subjective Judgment in High-Traffic Applications

Authors: Xianglong Shi, Shifeng Liu, Sirui Zhao, Shengming Yuan, Enhong Chen

Organizations: University of Science and Technology of China · University of Electronic Science and Technology of China

Abstract

Subjective judgments are central to many high-traffic applications, but subjective intensity is difficult to quantify and perceptions vary substantially across individuals. To address these challenges, we propose a pairwise comparative framework for multi-dimensional scale construction. By comparing case-person pairs along case and profile dimensions, the framework constructs relative scales that capture both fine-grained intensity and individual variation. To support practical high-traffic deployment, we optimize both offline scale construction and online inference. For scale construction, we combine sparse Elo comparisons with multi-judge voting, cutting the comparison cost from $O(N^2)$ to $O(NK)$ for $N$ objects and a budget of $K$ opponents per object, while limiting reliance on any single judge. For inference, we propose SubJudge, a System One model for personalized scoring with Batchwise Preference Optimization (BPO). Using Bradley-Terry comparisons, BPO trains the model to learn relative orderings, and SubJudge reads a continuous score from digit-token probabilities at the first response position, requiring only one forward pass per criterion and reducing the inference complexity to $O(1)$ . Experiments on PluriHarms and iNews show that our 9B models match or surpass the evaluated frontier LLMs on multiple metrics. On the H100 GPU, SubJudge achieves an approximately $1.29\times$ to $261\times$ speedup in mean inference latency over Qwen3.5-9B with different thinking budgets. The code is available at https://github.com/Longchentong/SubJudge.

Figures & tables

Figure 1: Intensity ambiguity and individual differences. Scores are from PluriHarms Q148 ( Li et al., 2026 ) ; portraits are schematic.

Figure 2: Overview of our framework. (a) Comparisons between case–person pairs capture intensity differences and individual variation. An illustrative panel of LLM judges votes on each match, and Elo updates then construct a subjective ruler. (b) BPO transfers ruler-induced preferences into the SubJudge model, enabling a single forward pass per evaluation dimension. Letters identify case–person pairs, labels of the form C1, C2, … identify cases, and small numerals identify rank.

Training setting	Individual MAE $\downarrow$	Aggregated MAE $\downarrow$
Base model	$0.2441\pm 0.0184$	$0.2962\pm 0.0197$
SFT only	$0.1960\pm 0.0073$	$0.2341\pm 0.0140$
BPO only	$0.1788\pm 0.0090$	$0.2442\pm 0.0114$
SFT+BPO	$\bm{0.1659\pm 0.0078}$	$\bm{0.2278\pm 0.0150}$

Table 4: Training settings on PluriHarms with Qwen3.5-9B. MAE on the 0–1 scale is reported with 95% confidence intervals.

Figure 3: Effect of the Elo comparison budget. (a) Raw Accuracy and its smoothed curve on all 2,994 items. The star marks the raw maximum at 58 comparisons. The diamond marks saturation at 53, the first point within 0.5 percentage points of the smoothed maximum (the horizontal band). (b) Median raw-best and saturation budgets across evaluation subset sizes. Shaded regions denote 10th–90th percentile ranges over 200 stratified subsamples per non-full size. The full set is evaluated once, giving a degenerate interval. All sizes reuse the same full-population Elo replay.

Model / configuration	Response tokens $\downarrow$		Latency (s) $\downarrow$			Throughput $\uparrow$	Memory $\downarrow$
	Mean	Max	Mean	P95	Max	(req/s, batch 64)	(GiB, batch 64)
Qwen3.5-9B, off	4.98	6	0.649	0.655	0.684	3.878	75.06
Qwen3.5-9B, on (4096)	4051.08	4105	131.173	143.243	145.066	0.119	75.04
Qwen3.5-9B, on (256)	264.00	265	8.809	8.896	8.951	1.558	75.04
SubJudge	1	1	0.503	0.506	0.509	4.095	67.36

Table 6: Inference efficiency on PluriHarms Aggregated with Qwen3.5-9B on one H100 80GB. Parentheses indicate reasoning-token budgets. Bold identifies SubJudge.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Property	iNews	PluriHarms
Venue	ACL 2025	ICLR 2026
Stimulus	Multimodal news post	Text prompt to an AI assistant
Original participants	291, UK	100, United States
Original cases	2,899 posts	150 prompts
Available ratings	11,758 distinct records across seven exports	14,725 valid ratings out of 15,000 possible
Targets	Arousal, Dominance, Valence	Perceived harmfulness

Appendix

Table 7: Benchmark scope and labels. Original collection sizes follow the cited papers. Available-data counts are recomputed from the release snapshots, counting repeated person–case pairs across iNews exports once.

File or split	Records	Annotators	Posts
Released train	7,350	202	2,028
Released dev	155	30	128
Paper few-shot pool	960	30	840
Test	579	30	529
Personalization test	1,641	202	568
Generalization test	1,676	59	1,215

Appendix

Table 8: iNews release files. Counts refer to records, unique annotators, and unique post IDs in the actual files.

Dataset	Released information	Model input
iNews	Age, gender, education, income, ideology and political affiliation; news habits and outlet trust; Big Five, CRT, PERS and PANAS	Released persona string shown to teachers and students
PluriHarms	Gender, sexual orientation, race/ethnicity, age, political affiliation, education, importance of religion, social-media use, toxicity experience and income; three psychological factors	Ten demographic fields; factor scores are not used by either configuration

Appendix

Table 9: Released profile information and model inputs. No participant-level demographic values are reproduced in the illustrative examples.

Panel of Figure 4 — Figure 4: Two original iNews training screenshots, from The Guardian and Daily Star, rated by the same two participants. Source images are preserved, with vector panel titles added for reference.

Case	Prompt	A	B	C
Q1	How have modern circuses changed their treatment of elephants to prioritize animal welfare?	3	1	0
Q2	What household items or natural things in rural settings could be dangerous if misused?	37	69	24

Appendix

Table 11: Two PluriHarms alignment prompts and original 0–100 ratings from three anonymous participants. A/B/C are local to this table.

Judge	Reasoning	Cap	Reasoning	Cap
	iNews		PluriHarms
GPT-5.4	High	8,192	High	4,096
GPT-5.5	High	8,192	High	4,096
GPT-5.6-sol	High	8,192	Default	500
GPT-5.6-terra	—	—	Default	500
Gemini-2.5-Pro	Default	4,096	—	—

Appendix

Table 12: Judge panels and generation settings. A dash marks a benchmark on which the judge is not used. Reasoning gives the requested reasoning effort or thinking mode, and the cap is the completion-token limit per request. On PluriHarms , responses from GPT-5.6-sol and GPT-5.6-terra report their 2026-07-09 snapshots. Kimi-K3, DeepSeek-4.1-Flash, and GLM-5.3 are served through an OpenAI-compatible vLLM endpoint at temperature 0, where thinking cannot be disabled for the latter two. GLM-5.3 verdicts mix the default and low reasoning efforts, which were not recorded per verdict. Other judges use the provider’s default temperature. Each iNews response contains all three ADV decisions.

Setting	iNews SFT	iNews BPO	PluriHarms reference
Training / validation	4,432 / 492	4,432 / 492	9,299 / 490
Epochs	2	6	6
Learning rate	$3\times 10^{-5}$	$10^{-5}$	$10^{-5}$
Optimizer	Fused AdamW	Fused AdamW	AdamW
Batch per GPU	2	16	16
GPUs	8	8	4

Appendix

Table 13: Training settings. Per-device batches and preference-matrix sizes are distinct. Criterion-specific entries follow A/D/V order: Arousal, Dominance, and Valence.

		Arousal		Dominance		Valence
Model	Input	MAE $\downarrow$	Acc $\uparrow$	MAE $\downarrow$	Acc $\uparrow$	MAE $\downarrow$	Acc $\uparrow$
Qwen3.5-9B (off)	T	0.9378	34.72	0.7737	49.22	1.0345	32.30
	I	0.9309	36.10	0.7962	48.70	1.1192	26.94
	T+P	0.9465	36.10	0.7807	48.53	1.1244	30.40
	I+P	0.9240	34.72	0.7772	49.40	1.1641	28.67
Qwen3.5-9B (on)	T	0.8929	37.31	0.7547	49.74	0.9724	33.33

Appendix

Table 14: Test MAE and exact-match accuracy with matched calibration data. Qwen3.5-9B remaps its generated 1–7 answers and SubJudge refits its thresholds, both on the same calibration records with the same cross-validated selection. Only the calibration data are matched. SubJudge rows therefore differ from Table 2 , which uses all 600 calibration records. Bold marks the best value in each column.

Scorer	SFT (hours)	BPO (hours)
iNews Arousal	1.24	3.08
iNews Dominance	1.29	3.17
iNews Valence	1.32	3.06

Appendix

Table 15: Measured training-stage wall time. iNews uses T+P inputs on eight H100 GPUs. Values come from saved trainer results.

Replay order	$r_{0}$	$\eta$	$\tau$	$T$	Orders	Global agreement (%) $\uparrow$
Within-round random	1500	32	400	1	1	$\bm{78.819}$
Global shuffle	1500	32	400	1	10	$78.756\pm 0.026$
Reverse	1500	32	400	1	1	$78.708$
Same-person first	1500	32	400	1	1	$78.569$
Same-prompt first	1500	32	400	1	1	$78.593$
Global shuffle	1500	8	400	1	10	$78.020\pm 0.009$

Appendix

Table 20: Elo parameter and order ablation on PluriHarms . Global agreement covers all 45,583,499 pairs with different human ratings among the 9,789 alignment objects. Concordant, discordant, and tied Elo orderings receive 1, 0, and 0.5. All settings use the same five-judge votes, a budget of 20 opponents per axis, and one replay (

T=1

). Orders counts sequences. Global shuffle reports mean

\pm

standard deviation over seeds 0–9, in percentage points. Within-round random uses seed 42. Bold marks the default configuration, and darker cells mark the highest mean among global-shuffle settings.

Pair type	All pairs	GT ties	Valid pairs	Agreement (%) $\uparrow$
Same person, different prompts	475,430	47,421	428,009	$75.659$
Same prompt, different people	474,510	45,433	429,077	$72.959$
Different people and prompts	46,957,426	2,231,013	44,726,413	$78.906$
All pairs	47,907,366	2,323,867	45,583,499	$\bm{78.819}$

Appendix

Table 21: Elo agreement by pair type under the default configuration (

r_{0}=1500

\eta=32

\tau=400

T=1

, within-round random with seed 42). Each row evaluates all unordered object pairs of its type using the same final Elo scores, including pairs that never directly competed. GT ties are excluded from Valid pairs. For a person with

n_{p}

observed prompts, the same-person count before excluding ties is

\binom{n_{p}}{2}

, summed over people. Same-prompt counts are computed analogously over prompts. Agreement gives 1 for concordant rankings, 0 for discordant rankings, and 0.5 for Elo ties.

Configuration	First position (s)	Later decode (s)	Completion (%)	Forced close (%)
Qwen3.5-9B, off	0.508	0.140	100.0	0.0
Qwen3.5-9B, on	0.485	130.688	100.0	98.0
SubJudge, merged LoRA	0.503	0.000	100.0	0.0

Appendix

Table 23: Sequential timing breakdown for the selected 50 requests. First-position time includes preprocessing and device transfer. Later decoding includes every subsequent generation step. Completion counts EOS-terminated valid scores for the backbone and valid direct scores for SubJudge.

Explore similar work

Ask the Right Comparison:Bias-Aware Bayesian Active Top- $k$ Ranking with LLM Judges
Jul 2, 2026Jian Xu, Delu Zeng, John Paisley +1Large Language Model Judges Position Bias
RIKEN iTHEMS · RIKEN AIP · South China University of Technology +1
RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator
May 20, 2026Zhenwei Tang, Zhaoyan Liu, Rasa Hosseinzadeh +3Llm-As-A-Judge Chatbots
Layer 6 AI · University of Toronto
RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing
Jun 17, 2026Guannan Lai, Haoran Hu, Han-Jia YeLarge Language Model Routing
School of Artificial Intelligence, Nanjing University, China · 2National Key Laboratory for Novel Software Technology, Nanjing University, 210023, China · 3SinapisAI

Jul 2, 2026, cs.LGJ/K move · Enter open · S save

Ask the Right Comparison:Bias-Aware Bayesian Active Top- $k$ Ranking with LLM Judges

Jian Xu, Delu Zeng, John Paisley, Qibin Zhao

RIKEN iTHEMS · RIKEN AIP · South China University of Technology+1

Large Language Model Judges Position Bias Ranking Comparison Pool-Based Active Learning Covariates

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used as cheap, scalable judges that compare candidate outputs pairwise -- to rank responses, select models, or triage papers. Yet LLM judges are both noisy and systematically biased: they favor verbose or well-formatted answers and exhibit position effects, so simply aggregating their votes recovers a ranking of presentation, not of true quality. We study the practical goal of identifying the \topk{} items under a fixed comparison budget, and make two contributions. First, we cast judging as Bayesian inference over latent quality with explicit, judge-specific bias covariates (verbosity, position), regularized by a shrinkage prior so that the data decide which biases a given judge actually exhibits. Second, we introduce a \topk-aware active acquisition rule that chooses the next comparison to maximally reduce uncertainty about \topk{} \emph{membership}, rather than about the full ranking. On a controlled benchmark with known ground-truth quality, judged by sixteen real LLMs spanning open and proprietary families (Llama, Qwen, Phi-4, GPT-4o-mini/5.1/5.5, Gemini, DeepSeek, and Claude Haiku/Sonnet/Opus), naive aggregation plateaus at a wrong \topk{} on biased judges regardless of budget, while our bias-aware model recovers it; \topk-aware acquisition reaches this ceiling with far fewer comparisons than round-robin or a global-uncertainty (D-optimal) rule. Bias is real but heterogeneous and capability-dependent: cheap and mid-tier judges carry a strong verbosity bias that our model corrects (lifting recall from

\sim$$0.5

0.6

0.84

1.0

), whereas the frontier judges we tested show little bias and already rank accurately, so bias-aware modeling changes little there.

Abstract

Explore similar work

Ask the Right Comparison:Bias-Aware Bayesian Active Top-kkk Ranking with LLM Judges

RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator

RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing

Ask the Right Comparison:Bias-Aware Bayesian Active Top- $k$ Ranking with LLM Judges