Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation

Haocheng Yang; Licheng Pan; Yiming Dong; Xiaoxi Li; Yuan Lu; Zhenhao Chen; Jiajun Fan; Shuai Shao

cs.CLOct 14, 2026

Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation

Authors: Haocheng Yang, Licheng Pan, Yiming Dong, Xiaoxi Li, Yuan Lu, Zhenhao Chen, Jiajun Fan, Shuai Shao,

Organizations: National University of Singapore · Zhejiang University · Alibaba Group · Renmin University of China ·

Abstract

Rubric-based evaluation assesses the responses of large language models (LLMs) by aggregating the verification results of explicit criteria, yet not all criteria matter equally. However, existing methods commonly aggregate the verification results with uniform weighting, which assigns minor and overlapping criteria the same weight as essential ones and thus produces misleading evaluation scores. Although some methods prompt an LLM to directly assign weights, direct weighting requires a holistic judgment over all criteria that LLMs struggle to make reliably. To address these limitations, we propose Importance-Aware Criteria Weighting (ICW), which derives the weights of criteria from pairwise importance comparisons made by an LLM. Specifically, ICW transforms the holistic judgment over all criteria into a series of pairwise importance comparisons, checks and revises inconsistent comparisons, and derives the weights from the principal eigenvector of the comparison matrix. Extensive experiments demonstrate that ICW consistently outperforms uniform weighting and competitive weighting methods, and translates into better policy models in downstream reinforcement learning.

Figures & tables

Panel of Figure 1 — Figure 1: Case studies in which uniform weighting assigns higher evaluation scores to lower-quality responses. (a) Minor stylistic criteria outweigh the essential criterion of factual correctness. (b) Overlapping criteria give repeated credit to the same recommendation.

Figure 2: Overview of ICW. The weighting model compares the importance of every pair of criteria given the user prompt and the rubric set, and the judgments are mapped to a comparison matrix with the fixed ratio

\alpha

. If CR exceeds

\tau

, all pairs are re-compared for at most

T

rounds, and a revised matrix replaces the current matrix only if its CR is strictly lower. The weights are obtained from the principal eigenvector of the current matrix.

	RubricBench		ResearchQA		JudgeBench		RM-Chat		Chat-Hard		Precise-IF
Method	Acc	RI	Acc	RI	Acc	RI	Acc	RI	Acc	RI	Acc	RI
Uniform	0.771	–	0.583	–	0.762	–	0.740	–	0.805	–	0.720	–
Direct	0.778	0.96% $\uparrow$	0.589	1.03% $\uparrow$	0.770	1.06% $\uparrow$	0.731	1.22% $\downarrow$	0.802	0.41% $\downarrow$	0.723	0.43% $\uparrow$
AdaRubric	0.774	0.45% $\uparrow$	0.592	1.53% $\uparrow$	0.765	0.31% $\uparrow$	0.749	1.23% $\uparrow$	0.812	0.96% $\uparrow$	0.732	1.74% $\uparrow$
RaR	0.782	1.41% $\uparrow$	0.604	3.57% $\uparrow$	0.763	0.10% $\uparrow$	0.760	2.62% $\uparrow$	0.816	1.37% $\uparrow$	0.746	3.61% $\uparrow$
MAPLE	0.782	1.41% $\uparrow$	0.601	3.05% $\uparrow$	0.760	0.21% $\downarrow$	0.745	0.70% $\uparrow$	0.812	0.96% $\uparrow$	0.738	2.60% $\uparrow$

Table 1: Pairwise preference accuracy of ICW versus baselines.

Method	Pair.	Rev.	RubricBench		ResearchQA		JudgeBench		RM-Chat		Chat-Hard		Precise-IF
Method	Pair.	Rev.	Acc	RI	Acc	RI	Acc	RI	Acc	RI	Acc	RI	Acc	RI
Uniform	✗	✗	0.771	–	0.583	–	0.762	–	0.740	–	0.805	–	0.720	–
ICW †	✗	✓	0.782	1.47% $\uparrow$	0.592	1.53% $\uparrow$	0.758	0.52% $\downarrow$	0.747	0.88% $\uparrow$	0.803	0.27% $\downarrow$	0.755	4.92% $\uparrow$
ICW ‡	✓	✗	0.790	2.49% $\uparrow$	0.607	4.07% $\uparrow$	0.783	2.76% $\uparrow$	0.758	2.44% $\uparrow$	0.804	0.14% $\downarrow$	0.752	4.49% $\uparrow$
ICW	✓	✓	0.795	3.11% $\uparrow$	0.612	5.08% $\uparrow$	0.783	2.76% $\uparrow$	0.769	3.84% $\uparrow$	0.818	1.64% $\uparrow$	0.762	5.79% $\uparrow$

Table 2: Ablation study results of the two ICW components.

Table 5

Figure 3: Sensitivity to the fixed ratio

\alpha

and the consistency threshold

\tau

with GPT-4.1 as the weighting model.

Policy Model: Qwen2.5-3B-Instruct
Method	BiGGen Bench		AlpacaEval 2.0				Arena-Hard
Method	Score	RI	WR	RI	LC	RI	WR	RI	SC	RI
Base Model	59.22	–	17.25	–	16.71	–	25.93	–	27.63	–
Uniform	61.45	3.77% $\uparrow$	21.59	25.16% $\uparrow$	19.07	14.12% $\uparrow$	27.58	6.36% $\uparrow$	28.58	3.44% $\uparrow$
RACE-SC	64.95	9.68% $\uparrow$	24.32	40.99% $\uparrow$	22.74	36.09% $\uparrow$	28.67	10.57% $\uparrow$	29.61	7.17% $\uparrow$
RRD-WU	64.23	8.46% $\uparrow$	24.37	41.28% $\uparrow$	20.42	22.20% $\uparrow$	27.78	7.13% $\uparrow$	29.80	7.85% $\uparrow$

Table 5: Downstream RL results.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Table 8

Figure 4: Sensitivity to the fixed ratio

\alpha

with Gemini-3.5-Flash (left) and GLM-5.1 (right) as the weighting models.

Weighting Model	Type	RubricBench	ResearchQA	JudgeBench	RM-Chat	Chat-Hard	Precise-IF
GPT-4.1	Input	39.552	152.116	111.641	92.567	107.761	185.989
GPT-4.1	Output	1.377	2.957	1.962	1.799	1.872	2.510
Gemini-3.5-Flash	Input	49.799	190.582	127.352	106.721	122.638	231.673
Gemini-3.5-Flash	Output	18.074	41.253	22.049	17.428	17.770	29.697
GLM-5.1	Input	61.862	180.468	113.025	94.347	110.812	197.559
GLM-5.1	Output	26.279	53.043	26.773	22.474	23.030	33.580

Appendix

Table 8: Token cost of ICW with different weighting models.

Language Model Generation Evaluation

Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.

See all 395 papers

Rubric Judgments Lack Uncertainty: Conformal Weighting for LLM Evaluation
Oct 15, 2026Licheng Pan, Haocheng Yang, Haoxuan Li +7Language Model Generation Evaluation Rubric-Based Evaluation
Zhejiang University · Xiaohongshu Inc · National University of Singapore +3
Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge
Feb 2, 2026Yuzheng Xu, Tosho Hirasawa, Tadashi Kozuno +1LLM-as-a-Judge Language Model Bias Evaluation
OMRON SINIC X · NexaScience
Support Vector Rubrics: Closing the Gap Between Self-Generated and Human Rubrics
Jun 6, 2026Mengyuan Sun, Yu Li, Zhuohao Yu +2Reward Modeling Preference Modeling
National Engineering Research Center for Software Engineering, Peking University · University of Science and Technology of China

Oct 15, 2026J/K move · Enter open · S save

Rubric Judgments Lack Uncertainty: Conformal Weighting for LLM Evaluation

Licheng Pan, Haocheng Yang, Haoxuan Li, Yuzhou Chen+6

Zhejiang University · Xiaohongshu Inc · National University of Singapore+3

Language Model Generation Evaluation Rubric-Based Evaluation Rubric-Based RL LLM Uncertainty Estimation

Open paper AskPDF arXiv

Rubric-based evaluation is widely used to assess large language model (LLM) outputs by decomposing quality into multiple criteria and assigning each a binary judgment from an LLM verifier. However, judgment certainty varies substantially across criteria. Existing methods uniformly aggregate these judgments into a final score, implicitly treating every criterion as equally certain and overlooking this heterogeneity. As a result, uncertain judgments remain incorporated into reward estimation without distinction, potentially yielding misleading training signals for downstream reinforcement learning (RL). To address this limitation, we propose Uncertainty-aware Rubric Weighting (URW), a training-free framework that quantifies criterion-level judgment uncertainty via conformal calibration. Specifically, URW elicits a soft uncertainty signal from a fixed pool of paraphrased judgment templates and converts it into a per-criterion weight using split-conformal

p

-values with provable coverage guarantees. Extensive experiments demonstrate that URW consistently improves pairwise preference accuracy and delivers stronger downstream RL alignment.

Abstract

Language Model Generation Evaluation

Explore similar work

Rubric Judgments Lack Uncertainty: Conformal Weighting for LLM Evaluation

Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge

Support Vector Rubrics: Closing the Gap Between Self-Generated and Human Rubrics