Rubric Judgments Lack Uncertainty: Conformal Weighting for LLM Evaluation

Licheng Pan; Haocheng Yang; Haoxuan Li; Yuzhou Chen; Yichen Sun; Yuan Lu; Xiaoxi Li; Hao Wang

cs.AIOct 15, 2026

Rubric Judgments Lack Uncertainty: Conformal Weighting for LLM Evaluation

Authors: Licheng Pan, Haocheng Yang, Haoxuan Li, Yuzhou Chen, Yichen Sun, Yuan Lu, Xiaoxi Li, Hao Wang,

Organizations: Zhejiang University · Xiaohongshu Inc · National University of Singapore · Peking University ·

Abstract

Rubric-based evaluation is widely used to assess large language model (LLM) outputs by decomposing quality into multiple criteria and assigning each a binary judgment from an LLM verifier. However, judgment certainty varies substantially across criteria. Existing methods uniformly aggregate these judgments into a final score, implicitly treating every criterion as equally certain and overlooking this heterogeneity. As a result, uncertain judgments remain incorporated into reward estimation without distinction, potentially yielding misleading training signals for downstream reinforcement learning (RL). To address this limitation, we propose Uncertainty-aware Rubric Weighting (URW), a training-free framework that quantifies criterion-level judgment uncertainty via conformal calibration. Specifically, URW elicits a soft uncertainty signal from a fixed pool of paraphrased judgment templates and converts it into a per-criterion weight using split-conformal $p$ -values with provable coverage guarantees. Extensive experiments demonstrate that URW consistently improves pairwise preference accuracy and delivers stronger downstream RL alignment.

Figures & tables

Figure 1: A case study where a single uncertain judgment on a subjective criterion inverts the aggregated preference under uniform weighting.

Verifier: DeepSeek V4 Pro ( Xu et al., 2026b )
	RubricBench		ResearchQA		RM-Bench		RewardBench		RewardBench 2
Method	Acc	RI	Acc	RI	Acc	RI	Acc	RI	Acc	RI
Uniform	0.780	–	0.633	–	0.729	–	0.766	–	0.625	–
Beta-Binomial	0.781	0.11% $\uparrow$	0.639	0.93% $\uparrow$	0.739	1.42% $\uparrow$	0.772	0.72% $\uparrow$	0.688	10.00% $\uparrow$
Sampling Ensemble	0.781	0.11% $\uparrow$	0.669	5.61% $\uparrow$	0.711	2.48% $\downarrow$	0.785	2.43% $\uparrow$	0.731	17.00% $\uparrow$
SCP	0.780	0.00%	0.645	1.87% $\uparrow$	0.745	2.30% $\uparrow$	0.784	2.29% $\uparrow$	0.693	10.83% $\uparrow$

Table 1: Pairwise preference accuracy of URW versus baselines.

Verifier: DeepSeek V4 Pro ( Xu et al., 2026b )
Method	Multi.	Cali.	RubricBench		ResearchQA		RM-Bench		RewardBench		RewardBench 2
Method	Multi.	Cali.	Acc	RI	Acc	RI	Acc	RI	Acc	RI	Acc	RI
Uniform	✗	✗	0.780	–	0.633	–	0.729	–	0.766	–	0.625	–
URW †	✓	✗	0.779	0.11% $\downarrow$	0.645	1.87% $\uparrow$	0.736	1.06% $\uparrow$	0.781	1.86% $\uparrow$	0.682	9.17% $\uparrow$
URW ‡	✗	✓	0.791	1.45% $\uparrow$	0.648	2.34% $\uparrow$	0.758	4.08% $\uparrow$	0.797	4.01% $\uparrow$	0.716	14.50% $\uparrow$
URW	✓	✓	0.802	2.79% $\uparrow$	0.698	10.28% $\uparrow$	0.792	8.69% $\uparrow$	0.810	5.72% $\uparrow$	0.759	21.50% $\uparrow$

Table 2: Ablation study results of the two URW components.

Panel of Figure 2 — Figure 2: Generalization results of URW across verifier backbones, where “Q3” denotes the Qwen3 family and “G4” denotes the GLM4 family.

Table 5

Panel of Figure 3 — Figure 3: Sensitivity to the template count $\mathrm{N}$ .

Table 7

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Sensitivity to the template count $\mathrm{N}$ .

Prompt :	Answer the following multiple choice question. The last line of your response should be of the following format: ’Answer: $LETTER’ (without quotes) where LETTER is one of ABCD. Think step by step before answering. Sigmatropic rearrangement is a class of pericyclic reactions. In these reactions, the terminal pi bond migrates into the sigma bond. Cope and Claisen rearrangements are also sigma tropic rearrangements. These reactions are usually thermodynamically favored. Complete the following reactions. 1,1-dimethoxyethan-1-amine + but-3-en-2-ol + (H+ + Heat)$ \rightarrow $A (3R,4S)-3,4-dimethylhexa-1,5-diyne + Heat$ \rightarrow $B 2-((vinyloxy)methyl)but-1-ene + Heat$ \rightarrow$ C A) A = 6-methyl-3,4-dihydro-2H-pyran-2-amine, B = (1Z,2E)-1,2-diethylidenecyclobutane, C = 4-methylenehexanal B) A = 6-methyl-3,4-dihydro-2H-pyran-2-amine, B = (1Z,2E)-1,2-diethylidenecyclobutane, C = 4-methylenehexan-1-ol C) A = (Z)-1-(but-2-en-2-yloxy)ethen-1-amine, B = (3Z,4E)-3,4-diethylidenecyclobut-1-ene, C = 4-methylenehexan-1-ol D) A = (Z)-1-(but-2-en-2-yloxy)ethen-1-amine, B = (3Z,4E)-3,4-diethylidenecyclobut-1-ene, C = 4-methylenehexanal
True Answer :	D
Responses :
Uniform :	To determine the correct answer, we analyze each reaction step by step, focusing on the mechanisms and possible products of the sigmatropic rearrangements involved. — ### Reaction A: Starting Materials: - 1,1-dimethoxyethan-1-amine - but-3-en-2-ol (2-hydroxybut-3-ene) - Conditions: H + and heat Analysis: This reaction involves the formation of a cyclic ether, specifically a pyran ring (a six-membered ring containing one oxygen atom). The presence of two methoxy groups on the nitrogen-substituted carbon and the hydroxyl group from the alcohol facilitate the formation of a cyclic hemiaminal or pyran via an acid-catalyzed rearrangement. The product is 6-methyl-3,4-dihydro-2H-pyran-2-amine, a pyran ring with a methyl group on C6 and an amine group on C2. This matches a Claisen-type rearrangement where the hydroxyl group and the amine group participate in ring formation. — ### Reaction B: Starting Material: - (3R,4S)-3,4-dimethylhexa-1,5-diyne (a six-carbon chain with two triple bonds and methyl groups at positions 3 and 4) - Conditions: Heat Analysis: This compound undergoes a [3,3]-sigmatropic rearrangement, forming a cyclobutane ring. The two triple bonds (C1–C2 and C5–C6) become double bonds within the ring, and the methyl groups at positions 3 and 4 migrate to form the ring. The product (1Z,2E)-1,2-diethylidenecyclobutane is formed, where the ring consists of two adjacent double bonds, each with Z and E configurations, matching the stereochemistry of the starting compound. The Z and E configuration in the product aligns with the stereochemical preservation during the reaction. — ### Reaction C: Starting Material: - 2-((vinyloxy)methyl)but-1-ene - Conditions: Heat Analysis: This is a vinyloxy group (an ether group with a double bond) attached to a methyl group on carbon 2 of but-1-ene. Under heat, a [1,5]-sigmatropic rearrangement occurs, where the vinyloxy group shifts to form a new double bond. The oxygen from the vinyloxy group can act as a nucleophile, leading to the formation of an aldehyde group (as the oxygen becomes part of a carbonyl group). The product is 4-methylenehexanal, which has a methylene group at position 4 and an aldehyde group at position 1. This aligns with the Claisen-type rearrangement mechanism where the oxygen becomes part of a carbonyl. — ### Conclusion: - Reaction A forms a pyran ring, matching Option A’s product. - Reaction B forms a cyclobutane ring with ZE configurations, matching Option A’s product. - Reaction C forms an aldehyde, matching Option A’s product. All three reactions align with the product described in Option A. — Answer: A
Predicted Answer :	A
Beta-Binomial :	To determine the correct answer, we analyze each of the three reactions using the principles of sigmatropic rearrangements: — ### Reaction A: 1,1-dimethoxyethan-1-amine + but-3-en-2-ol + (H+ + Heat) → A This is a classic example of a Claisen-type rearrangement. The but-3-en-2-ol (an allylic alcohol) and the 1,1-dimethoxyethan-1-amine (likely a nucleophilic amine with two methoxy groups) undergo an acid-catalyzed, thermal rearrangement. The methoxy groups and hydroxyl group from the alcohol combine to form a six-membered cyclic ether (pyran), with an amine group at position 2 of the ring. This matches Option B’s A: 6-methyl-3,4-dihydro-2H-pyran-2-amine, which features the correct ring structure and functional group placements. — ### Reaction B: (3R,4S)-3,4-dimethylhexa-1,5-diyne + Heat → B The starting compound is a dihydrocarbon with two methyl groups on carbons 3 and 4 and triple bonds at positions 1 and 5. Heating under such conditions typically leads to a [1,5]-sigmatropic rearrangement or ring-forming reaction. Given the stereochemistry (R and S configuration on the methyl groups), and the need to form a cyclobutane ring, the product is (1Z,2E)-1,2-diethylidenecyclobutane, which aligns with the stereochemistry and geometry of the starting material. This matches Option B’s B. — ### Reaction C: 2-((vinyloxy)methyl)but-1-ene + Heat → C The starting material has a vinyloxy group attached to a methyl group on carbon 2 of but-1-ene. Upon heating, a [1,2] sigmatropic rearrangement is likely to occur, where the vinyloxy group forms a carbonyl group and extends the chain. The final product would be a hexanal with a methylene group at position 4, matching Option B’s C: 4-methylenehexan-1-ol (alcohol) or Option D’s C: 4-methylenehexanal (aldehyde). Given the oxygen from the vinyloxy group and the possibility of forming a carbonyl (aldehyde), 4-methylenehexanal is a more plausible product than an alcohol. This suggests Option D for this reaction. — ### Final Assessment: - A: Matches Option B - B: Matches Option B - C: Better aligned with Option D Thus, the correct option is: Answer: B
Predicted Answer :	B

Appendix

Table 7: Qualitative case study on GPQA-Diamond with Qwen 3 8B.

Prompt :	Answer the following multiple choice question. The last line of your response should be of the following format: ’Answer: $LETTER’ (without quotes) where LETTER is one of ABCD. Think step by step before answering. A textile dye containing an extensively conjugated pi-electrons emits light with energy of 2.3393 eV. What color of light is absorbed by the organic compound? A) Red B) Blue C) Violet D) Yellow
True Answer :	A
Responses :
Uniform :	To determine the color of light absorbed by the organic compound, we need to follow these steps: 1. Convert the energy from eV to wavelength (nm): The energy of a photon is related to its wavelength by the equation: $E=\frac{hc}{\lambda}$ where: - $E$ is the energy of the photon (in eV), - $h$ is Planck’s constant ( $4.135667696\times 10^{-15}$ eV·s), - $c$ is the speed of light ( $2.99792458\times 10^{8}$ ms), - $\lambda$ is the wavelength (in meters). Rearranging for $\lambda$ : $\lambda=\frac{hc}{E}$ Plugging in the values: $\lambda=\frac{(4.135667696\times 10^{-15}\,\text{eV\textperiodcentered s})\times(2.99792458\times 10^{8}\,\text{m\/s})}{2.3393\,\text{eV}}$ $\lambda=\frac{1.241440748\times 10^{-6}\,\text{m\textperiodcentered eV}}{2.3393\,\text{eV}}$ $\lambda=5.2975\times 10^{-7}\,\text{m}=529.75\,\text{nm}$ 2. Interpret the wavelength: - Visible light ranges from approximately 400 nm (violet) to 700 nm (red). - A wavelength of 529.75 nm falls in the green-blue region, but closer to blue. However, the options provided are: - A) Red (700 nm) - B) Blue (450-495 nm) - C) Violet (380-450 nm) - D) Yellow (570-590 nm) Since 529.75 nm is closer to blue than yellow, the closest match is B) Blue. Answer: B
Predicted Answer :	B
Beta-Binomial :	To determine the color of light absorbed by the organic compound, we need to follow these steps: 1. Convert energy from eV to wavelength: The energy of light is related to its wavelength by the equation: $E=\frac{hc}{\lambda}$ where: - $E$ is the energy in joules (J), - $h$ is Planck’s constant ( $6.626\times 10^{-34}\,\text{J}\cdot\text{s}$ ), - $c$ is the speed of light ( $3.00\times 10^{8}\,\text{m\/s}$ ), - $\lambda$ is the wavelength in meters (m). Since the energy is given in electron volts (eV), we first convert it to joules: $1\,\text{eV}=1.602\times 10^{-19}\,\text{J}$ So: $E=2.3393\,\text{eV}\times 1.602\times 10^{-19}\,\text{J\/eV}=3.746\times 10^{-19}\,\text{J}$ 2. Solve for the wavelength $\lambda$ : $\lambda=\frac{hc}{E}$ Substituting the values: $\lambda=\frac{(6.626\times 10^{-34}\,\text{J}\cdot\text{s})\times(3.00\times 10^{8}\,\text{m\/s})}{3.746\times 10^{-19}\,\text{J}}$ $\lambda=\frac{1.9878\times 10^{-25}\,\text{J}\cdot\text{m}}{3.746\times 10^{-19}\,\text{J}}$ $\lambda\approx 5.3\times 10^{-7}\,\text{m}=530\,\text{nm}$ 3. Determine the color of light absorbed: The visible light spectrum ranges from approximately 400 nm (violet) to 700 nm (red). A wavelength of 530 nm falls in the green region, but since yellow is the closest option, we choose yellow. However, the exact answer is not listed, so we need to choose the closest match. The closest color to 530 nm is yellow, so the compound absorbs yellow light. Answer: D
Predicted Answer :	D

Appendix

Table 8: Qualitative case study on GPQA-Diamond with GLM 4 9B.

Language Model Generation Evaluation

Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.

See all 395 papers

Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation
Oct 14, 2026Haocheng Yang, Licheng Pan, Yiming Dong +7Language Model Generation Evaluation Rubric-Based Evaluation
National University of Singapore · Zhejiang University · Alibaba Group +6
RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domains
May 27, 2026Haoxiang Jiang, Zihan Dong, Tianci Liu +5Reward Modeling RL for Language Models
University at Albany · 2Rutgers University · 3Purdue University +3
From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape
Jun 7, 2026Hao Chen, Ziyu Han, Yukun Yan +3LLM Evaluation Rubric-Based Evaluation
Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology · Department of Computer Science and Technology, Institute for AI, Tsinghua University

Oct 14, 2026J/K move · Enter open · S save

Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation

Haocheng Yang, Licheng Pan, Yiming Dong, Xiaoxi Li+6

National University of Singapore · Zhejiang University · Alibaba Group+6

Language Model Generation Evaluation Rubric-Based Evaluation Multi-Criteria Decision Analysis

Open paper AskPDF arXiv

Rubric-based evaluation assesses the responses of large language models (LLMs) by aggregating the verification results of explicit criteria, yet not all criteria matter equally. However, existing methods commonly aggregate the verification results with uniform weighting, which assigns minor and overlapping criteria the same weight as essential ones and thus produces misleading evaluation scores. Although some methods prompt an LLM to directly assign weights, direct weighting requires a holistic judgment over all criteria that LLMs struggle to make reliably. To address these limitations, we propose Importance-Aware Criteria Weighting (ICW), which derives the weights of criteria from pairwise importance comparisons made by an LLM. Specifically, ICW transforms the holistic judgment over all criteria into a series of pairwise importance comparisons, checks and revises inconsistent comparisons, and derives the weights from the principal eigenvector of the comparison matrix. Extensive experiments demonstrate that ICW consistently outperforms uniform weighting and competitive weighting methods, and translates into better policy models in downstream reinforcement learning.

Abstract

Language Model Generation Evaluation

Explore similar work

Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation

RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domains

From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape