Organizations: Zhejiang University · Xiaohongshu Inc · National University of Singapore · Peking University · Renmin University of China · Mohamed bin Zayed University of Artificial Intelligence
Rubric-based evaluation is widely used to assess large language model (LLM) outputs by decomposing quality into multiple criteria and assigning each a binary judgment from an LLM verifier. However, judgment certainty varies substantially across criteria. Existing methods uniformly aggregate these judgments into a final score, implicitly treating every criterion as equally certain and overlooking this heterogeneity. As a result, uncertain judgments remain incorporated into reward estimation without distinction, potentially yielding misleading training signals for downstream reinforcement learning (RL). To address this limitation, we propose Uncertainty-aware Rubric Weighting (URW), a training-free framework that quantifies criterion-level judgment uncertainty via conformal calibration. Specifically, URW elicits a soft uncertainty signal from a fixed pool of paraphrased judgment templates and converts it into a per-criterion weight using split-conformal p-values with provable coverage guarantees. Extensive experiments demonstrate that URW consistently improves pairwise preference accuracy and delivers stronger downstream RL alignment.
Figures & tables
Figure 1: A case study where a single uncertain judgment on a subjective criterion inverts the aggregated preference under uniform weighting.
RubricBench
ResearchQA
RM-Bench
RewardBench
RewardBench 2
Method
Acc
RI
Acc
RI
Acc
RI
Acc
RI
Acc
RI
Verifier: DeepSeek V4 Pro ( Xu et al., 2026b )
Uniform
0.780
–
0.633
–
0.729
–
0.766
–
0.625
–
Beta-Binomial
0.781
0.11% ↑
0.639
0.93% ↑
0.739
1.42% ↑
0.772
0.72% ↑
0.688
10.00% ↑
Sampling Ensemble
0.781
0.11% ↑
0.669
5.61% ↑
0.711
2.48% ↓
0.785
2.43% ↑
0.731
17.00% ↑
SCP
0.780
0.00%
0.645
1.87% ↑
0.745
2.30% ↑
0.784
2.29% ↑
0.693
10.83% ↑
Table 1: Pairwise preference accuracy of URW versus baselines.
Method
Multi.
Cali.
RubricBench
ResearchQA
RM-Bench
RewardBench
RewardBench 2
Acc
RI
Acc
RI
Acc
RI
Acc
RI
Acc
RI
Verifier: DeepSeek V4 Pro ( Xu et al., 2026b )
Uniform
✗
✗
0.780
–
0.633
–
0.729
–
0.766
–
0.625
–
URW †
✓
✗
0.779
0.11% ↓
0.645
1.87% ↑
0.736
1.06% ↑
0.781
1.86% ↑
0.682
9.17% ↑
URW ‡
✗
✓
0.791
1.45% ↑
0.648
2.34% ↑
0.758
4.08% ↑
0.797
4.01% ↑
0.716
14.50% ↑
URW
✓
✓
0.802
2.79% ↑
0.698
10.28% ↑
0.792
8.69% ↑
0.810
5.72% ↑
0.759
21.50% ↑
Table 2: Ablation study results of the two URW components.
Figure 2: Generalization results of URW across verifier backbones, where “Q3” denotes the Qwen3 family and “G4” denotes the GLM4 family.
Table 5
Figure 3: Sensitivity to the template count N .
Table 7
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Sensitivity to the template count N .
Prompt :
Answer the following multiple choice question. The last line of your response should be of the following format: ’Answer: LETTER’(withoutquotes)whereLETTERisoneofABCD.Thinkstepbystepbeforeanswering.Sigmatropicrearrangementisaclassofpericyclicreactions.Inthesereactions,theterminalpibondmigratesintothesigmabond.CopeandClaisenrearrangementsarealsosigmatropicrearrangements.Thesereactionsareusuallythermodynamicallyfavored.Completethefollowingreactions.1,1−dimethoxyethan−1−amine+but−3−en−2−ol+(H++Heat)\rightarrowA(3R,4S)−3,4−dimethylhexa−1,5−diyne+Heat\rightarrowB2−((vinyloxy)methyl)but−1−ene+Heat\rightarrow$ C A) A = 6-methyl-3,4-dihydro-2H-pyran-2-amine, B = (1Z,2E)-1,2-diethylidenecyclobutane, C = 4-methylenehexanal B) A = 6-methyl-3,4-dihydro-2H-pyran-2-amine, B = (1Z,2E)-1,2-diethylidenecyclobutane, C = 4-methylenehexan-1-ol C) A = (Z)-1-(but-2-en-2-yloxy)ethen-1-amine, B = (3Z,4E)-3,4-diethylidenecyclobut-1-ene, C = 4-methylenehexan-1-ol D) A = (Z)-1-(but-2-en-2-yloxy)ethen-1-amine, B = (3Z,4E)-3,4-diethylidenecyclobut-1-ene, C = 4-methylenehexanal
True Answer :
D
Responses :
Uniform :
To determine the correct answer, we analyze each reaction step by step, focusing on the mechanisms and possible products of the sigmatropic rearrangements involved. — ### Reaction A: Starting Materials: - 1,1-dimethoxyethan-1-amine - but-3-en-2-ol (2-hydroxybut-3-ene) - Conditions: H + and heat Analysis: This reaction involves the formation of a cyclic ether, specifically a pyran ring (a six-membered ring containing one oxygen atom). The presence of two methoxy groups on the nitrogen-substituted carbon and the hydroxyl group from the alcohol facilitate the formation of a cyclic hemiaminal or pyran via an acid-catalyzed rearrangement. The product is 6-methyl-3,4-dihydro-2H-pyran-2-amine, a pyran ring with a methyl group on C6 and an amine group on C2. This matches a Claisen-type rearrangement where the hydroxyl group and the amine group participate in ring formation. — ### Reaction B: Starting Material: - (3R,4S)-3,4-dimethylhexa-1,5-diyne (a six-carbon chain with two triple bonds and methyl groups at positions 3 and 4) - Conditions: Heat Analysis: This compound undergoes a [3,3]-sigmatropic rearrangement, forming a cyclobutane ring. The two triple bonds (C1–C2 and C5–C6) become double bonds within the ring, and the methyl groups at positions 3 and 4 migrate to form the ring. The product (1Z,2E)-1,2-diethylidenecyclobutane is formed, where the ring consists of two adjacent double bonds, each with Z and E configurations, matching the stereochemistry of the starting compound. The Z and E configuration in the product aligns with the stereochemical preservation during the reaction. — ### Reaction C: Starting Material: - 2-((vinyloxy)methyl)but-1-ene - Conditions: Heat Analysis: This is a vinyloxy group (an ether group with a double bond) attached to a methyl group on carbon 2 of but-1-ene. Under heat, a [1,5]-sigmatropic rearrangement occurs, where the vinyloxy group shifts to form a new double bond. The oxygen from the vinyloxy group can act as a nucleophile, leading to the formation of an aldehyde group (as the oxygen becomes part of a carbonyl group). The product is 4-methylenehexanal, which has a methylene group at position 4 and an aldehyde group at position 1. This aligns with the Claisen-type rearrangement mechanism where the oxygen becomes part of a carbonyl. — ### Conclusion: - Reaction A forms a pyran ring, matching Option A’s product. - Reaction B forms a cyclobutane ring with ZE configurations, matching Option A’s product. - Reaction C forms an aldehyde, matching Option A’s product. All three reactions align with the product described in Option A. — Answer: A
Predicted Answer :
A
Beta-Binomial :
To determine the correct answer, we analyze each of the three reactions using the principles of sigmatropic rearrangements: — ### Reaction A: 1,1-dimethoxyethan-1-amine + but-3-en-2-ol + (H+ + Heat) → A This is a classic example of a Claisen-type rearrangement. The but-3-en-2-ol (an allylic alcohol) and the 1,1-dimethoxyethan-1-amine (likely a nucleophilic amine with two methoxy groups) undergo an acid-catalyzed, thermal rearrangement. The methoxy groups and hydroxyl group from the alcohol combine to form a six-membered cyclic ether (pyran), with an amine group at position 2 of the ring. This matches Option B’s A: 6-methyl-3,4-dihydro-2H-pyran-2-amine, which features the correct ring structure and functional group placements. — ### Reaction B: (3R,4S)-3,4-dimethylhexa-1,5-diyne + Heat → B The starting compound is a dihydrocarbon with two methyl groups on carbons 3 and 4 and triple bonds at positions 1 and 5. Heating under such conditions typically leads to a [1,5]-sigmatropic rearrangement or ring-forming reaction. Given the stereochemistry (R and S configuration on the methyl groups), and the need to form a cyclobutane ring, the product is (1Z,2E)-1,2-diethylidenecyclobutane, which aligns with the stereochemistry and geometry of the starting material. This matches Option B’s B. — ### Reaction C: 2-((vinyloxy)methyl)but-1-ene + Heat → C The starting material has a vinyloxy group attached to a methyl group on carbon 2 of but-1-ene. Upon heating, a [1,2] sigmatropic rearrangement is likely to occur, where the vinyloxy group forms a carbonyl group and extends the chain. The final product would be a hexanal with a methylene group at position 4, matching Option B’s C: 4-methylenehexan-1-ol (alcohol) or Option D’s C: 4-methylenehexanal (aldehyde). Given the oxygen from the vinyloxy group and the possibility of forming a carbonyl (aldehyde), 4-methylenehexanal is a more plausible product than an alcohol. This suggests Option D for this reaction. — ### Final Assessment: - A: Matches Option B - B: Matches Option B - C: Better aligned with Option D Thus, the correct option is: Answer: B
Predicted Answer :
B
Appendix
Table 7: Qualitative case study on GPQA-Diamond with Qwen 3 8B.
Prompt :
Answer the following multiple choice question. The last line of your response should be of the following format: ’Answer: $LETTER’ (without quotes) where LETTER is one of ABCD. Think step by step before answering. A textile dye containing an extensively conjugated pi-electrons emits light with energy of 2.3393 eV. What color of light is absorbed by the organic compound? A) Red B) Blue C) Violet D) Yellow
True Answer :
A
Responses :
Uniform :
To determine the color of light absorbed by the organic compound, we need to follow these steps: 1. Convert the energy from eV to wavelength (nm): The energy of a photon is related to its wavelength by the equation: E=λhc where: - E is the energy of the photon (in eV), - h is Planck’s constant ( 4.135667696×10−15 eV·s), - c is the speed of light ( 2.99792458×108 ms), - λ is the wavelength (in meters). Rearranging for λ : λ=Ehc Plugging in the values: λ=2.3393eV(4.135667696×10−15eV\textperiodcentereds)×(2.99792458×108m\/s)λ=2.3393eV1.241440748×10−6m\textperiodcenteredeVλ=5.2975×10−7m=529.75nm 2. Interpret the wavelength: - Visible light ranges from approximately 400 nm (violet) to 700 nm (red). - A wavelength of 529.75 nm falls in the green-blue region, but closer to blue. However, the options provided are: - A) Red (700 nm) - B) Blue (450-495 nm) - C) Violet (380-450 nm) - D) Yellow (570-590 nm) Since 529.75 nm is closer to blue than yellow, the closest match is B) Blue. Answer: B
Predicted Answer :
B
Beta-Binomial :
To determine the color of light absorbed by the organic compound, we need to follow these steps: 1. Convert energy from eV to wavelength: The energy of light is related to its wavelength by the equation: E=λhc where: - E is the energy in joules (J), - h is Planck’s constant ( 6.626×10−34J⋅s ), - c is the speed of light ( 3.00×108m\/s ), - λ is the wavelength in meters (m). Since the energy is given in electron volts (eV), we first convert it to joules: 1eV=1.602×10−19J So: E=2.3393eV×1.602×10−19J\/eV=3.746×10−19J 2. Solve for the wavelength λ : λ=Ehc Substituting the values: λ=3.746×10−19J(6.626×10−34J⋅s)×(3.00×108m\/s)λ=3.746×10−19J1.9878×10−25J⋅mλ≈5.3×10−7m=530nm 3. Determine the color of light absorbed: The visible light spectrum ranges from approximately 400 nm (violet) to 700 nm (red). A wavelength of 530 nm falls in the green region, but since yellow is the closest option, we choose yellow. However, the exact answer is not listed, so we need to choose the closest match. The closest color to 530 nm is yellow, so the compound absorbs yellow light. Answer: D
Predicted Answer :
D
Appendix
Table 8: Qualitative case study on GPQA-Diamond with GLM 4 9B.
Rubric-based evaluation assesses the responses of large language models (LLMs) by aggregating the verification results of explicit criteria, yet not all criteria matter equally. However, existing methods commonly aggregate the verification results with uniform weighting, which assigns minor and overlapping criteria the same weight as essential ones and thus produces misleading evaluation scores. Although some methods prompt an LLM to directly assign weights, direct weighting requires a holistic judgment over all criteria that LLMs struggle to make reliably. To address these limitations, we propose Importance-Aware Criteria Weighting (ICW), which derives the weights of criteria from pairwise importance comparisons made by an LLM. Specifically, ICW transforms the holistic judgment over all criteria into a series of pairwise importance comparisons, checks and revises inconsistent comparisons, and derives the weights from the principal eigenvector of the comparison matrix. Extensive experiments demonstrate that ICW consistently outperforms uniform weighting and competitive weighting methods, and translates into better policy models in downstream reinforcement learning.
Haocheng Yang, Licheng Pan, Yiming Dong +7
National University of Singapore · Zhejiang University · Alibaba Group +6
Pointwise reward modeling offers critical signals for LLM post-training, yet struggles with absolute scoring in subjective, non-verifiable settings. Rubric-based methods address this by decomposing evaluation into explicit criteria, but existing approaches typically depend on frontier LLMs and suffer from ties caused by hard Boolean aggregation. We present RUBRIC-ARROW, an alternating framework that jointly trains a rubric generator and a rubric-conditioned judge, with its RL stage using only pairwise preference data. Our method couples a probability-based scoring rule that reduces ties with phase-specific preference-based rewards and an alternating GRPO scheme that together train the pointwise evaluator. Extensive experiments show that RUBRIC-ARROW achieves competitive reward-modeling accuracy and yields consistent gains for downstream policy post-training.
Haoxiang Jiang, Zihan Dong, Tianci Liu +5
University at Albany · 2Rutgers University · 3Purdue University +3
As Large Language Models (LLMs) advance toward open-ended autonomous agents, the mechanisms used to evaluate and guide their behavior must evolve accordingly. This work introduces the rubric as a unifying framework capturing this evolution, characterizing rubrics as a dynamic response to successive LLM paradigm shifts that recurs across otherwise independent efforts in evaluation, reinforcement learning, and safety alignment. We define rubrics as explicit criteria sets that transform complex quality judgments into structured and actionable standards, and demonstrate that their recurrence across these research threads is not coincidental. We systematically organize existing rubric designs, examine their construction and optimization, and analyze their role across evaluation and training. Rubrics manifest at three progressively deeper levels: at the evaluative level, they decompose holistic judgments into verifiable dimensions; at the training level, they serve as dense feedback signals providing process-level guidance where scalar rewards fall short; at the intrinsic level, they emerge dynamically from model behaviors, driving self-improvement. We further assess rubric reliability across generation quality, execution fidelity, theoretical constraints, and security threats, before surveying rubric-based benchmarks across diverse domains. By rendering assessment transparent and decomposable, rubrics translate human value expectations into machine-learnable signals, serving as the enduring bridge between human intentions and machine behavior.
Hao Chen, Ziyu Han, Yukun Yan +3
Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology · Department of Computer Science and Technology, Institute for AI, Tsinghua University