RubricOpt: Self-Revision of Semantic Rubrics for Preference Optimization

Wei Chen; Yubing Wu; Shigui Li; Delu Zeng; Jingfeng Liu; Junmei Yang; Zhiheng Zhou; Bo Li

cs.AIOct 21, 2026

RubricOpt: Self-Revision of Semantic Rubrics for Preference Optimization

Authors: Wei Chen, Yubing Wu, Shigui Li, Delu Zeng, Jingfeng Liu, Junmei Yang, Zhiheng Zhou, Bo Li,

Organizations: South China University of Technology · LinkSprite · University of Waterloo

Abstract

A model's failures can reveal gaps in the behavioral criteria guiding its training. We introduce RubricOpt, a framework that lets a policy revise these criteria within preference optimization. The policy diagnoses its own rubric-free outputs, retains requirements already covered by the rubric, and edits clauses that leave recurring failures unaddressed. A shared semantic representation connects task demands, preference-data contrasts, and rubric clauses, while a deterministic gate checks evidence references and fixed semantic anchors. The rubric then guides Direct Preference Optimization (DPO) during training, with preference pairs, labels, and the objective held fixed, and is removed at evaluation. Our analysis relates the local effect on rubric-free success to semantic alignment and a residual beyond the measured factors. Experiments on capability and open-ended generation benchmarks show that rubric conditioning can improve outcomes over ordinary DPO, with gains and regressions varying across models and datasets. A matched continuation isolates the contribution of revision from additional training and finds mixed incremental effects for the tested edit, whose feedback overlaps the evaluation set. RubricOpt thus brings the behavioral specification itself into the policy's feedback loop.

Figures & tables

Figure 1: The RubricOpt loop. (a) Task demand and preference-data supply share eight factor axes. (b) Matched Prefer/Avoid clauses condition both responses during DPO. (c) Rubric-free evaluation supplies feedback for policy self-revision and the next refine / keep decision. Dashed frames group stages. Green boxes denote computation.

Figure 2: Policy-driven rubric refinement (appendix C.2 ). Prompt stages 1–5 diagnose failures, attribute gaps, propose edits, and audit the candidate. The audit is advisory; step 6 decides acceptance. Covered requirements and rejected edits retain the original clause. The excerpt shows the factual-grounding edit. The dashed frame groups prompt stages. Green boxes denote computation.

Figure 3: Diagnosis on shared semantic factors. (a) Contrast coverage (fraction of pairs) on both corpora using the same frozen judge; the dashed line marks the

0.25

sparsity threshold. (b) Qwen3-4B demand, averaged equally over four reasoning or three knowledge benchmarks. Demand is the success–failure gap on the

0

–

4

judge scale. Factors follow appendix A.2 .

Figure 4: Semantic demand in Qwen3-4B and Qwen3-8B. Lines connect the same factor; error bars show recorded

95\%

confidence intervals.

r

is Pearson correlation across the eight factors.

Qwen3-4B
	Tulu3 comprehensive capability (%)								HelpSteer3 open-ended generation
Method	GSM8K	MATH- Hard	BBH	MuSR	ARC	Truthful QA	MMLU- Pro	Avg.	MT- Bench	Arena- Hard (%)	Wild- Bench	Avg. $\Delta$ (%)
Instruct	$85.90$	$\mathbf{62.24}$	$12.82$	$\mathbf{43.39}$	$\mathbf{62.80}$	$52.27$	$41.98$	$\mathbf{51.63}$	$6.320$	$49.0$	$\mathbf{3.730}$	$-2.96$
Ordinary DPO	$84.53$	$39.15$	$15.43$	$42.59$	$61.09$	$52.71$	$41.88$	$48.20$	$6.890$	$50.9$	$3.617$	$0.00$
RubricOpt -R1	$\mathbf{86.81}$	$43.35$	$19.64$	$43.12$	$61.35$	$53.74$	$\mathbf{42.10}$	$50.02$	$\mathbf{6.930}$	$\mathbf{53.0}$	$3.727$	$\mathbf{+2.58}$
RubricOpt -R2	$80.74$	$37.12$	$\mathbf{22.73}$	$42.06$	$59.47$	$\mathbf{55.16}$	$40.43$	$48.24$	—

Table 1: Rubric-free Qwen3 results on both benchmark suites, each from its own dataset. Capability scores and their unweighted seven-task mean are percentages from the Tulu3 runs. The open-ended protocols come from the HelpSteer3 runs: MT-Bench and WildBench use native scores and Arena-Hard is the win rate against GPT-4-0314 (%). Open-ended

\mathrm{Avg.}\,\Delta=\frac{100}{3}\sum_{b}(s_{b}/s_{b}^{\mathrm{DPO}}-1)

averages relative changes across all three benchmarks against the same block’s DPO control. Bold marks maxima within each model block (including ties). Shading marks rubric-trained rows.

	Tulu3 comprehensive capability (%)								HelpSteer3 open-ended generation
Method	GSM8K	MATH- Hard	BBH	MuSR	ARC	Truthful QA	MMLU- Pro	Avg.	MT- Bench	Arena- Hard (%)	Wild- Bench	Avg. $\Delta$ (%)
Instruct	$45.75$	$19.65$	$36.95$	$\mathbf{52.45}$	$\mathbf{79.45}$	$\mathbf{58.75}$	$30.05$	$\mathbf{46.15}$	$5.090$	$18.1$	$2.336$	$-8.44$
Ordinary DPO	$44.45$	$20.90$	$\mathbf{40.40}$	$50.95$	$76.50$	$53.90$	$30.00$	$45.30$	$4.850$	$24.0$	$2.477$	$0.00$
RubricOpt -R1	$\mathbf{47.90}$	$\mathbf{21.50}$	$32.45$	$51.85$	$75.60$	$55.40$	$\mathbf{32.15}$	$45.26$	$\mathbf{5.140}$	$\mathbf{25.8}$	$\mathbf{2.480}$	$\mathbf{+4.53}$

Table 2: Mistral-7B results on both benchmark suites, each from its own dataset. Capability scores and their unweighted seven-task mean are percentages. Open-ended Avg.

\Delta

follows the definition in Tab. 1 . Bold marks maxima within each dataset block; shading marks rubric-trained rows.

Method	BBH	MATH-Hard	MT-Bench	Arena-Hard	WildBench	Avg. $\Delta$ (%)
RubricOpt -R1	$14.96$	$\mathbf{64.18}$	$\mathbf{6.93}$	$53.0$	$3.727$	$+2.58$
RubricOpt -R2 ( keep )	$\mathbf{18.05}$	$63.82$	$6.419$	$\mathbf{54.6}$	$3.781$	$+1.66$
RubricOpt -R2 ( refine )	$17.80$	$63.72$	$6.825$	$54.5$	$\mathbf{3.836}$	$\mathbf{+4.06}$

Table 3: Qwen3-4B matched continuation on HelpSteer3. Both branches start from RubricOpt -R1 and differ only in rubric text. Evaluation is rubric-free. Open-ended Avg.

\Delta

follows the relative-DPO definition in Tab. 1 . Bold marks column maxima.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Factor	Decisive	Agree	Decisive-only rate
$f_{1}$ task comprehension	$11$	$10$	$0.9091$
$f_{2}$ reasoning validity	$52$	$41$	$0.7885$
$f_{3}$ verification and calibration	$232$	$143$	$0.6164$
$f_{4}$ factual and evidential grounding	$7$	$6$	$0.8571$
$f_{5}$ completeness	$105$	$80$	$0.7619$
$f_{6}$ instruction adherence	$15$	$12$	$0.8000$

Appendix

Table 4: Blind pairwise agreement across four benchmarks. Pairwise verdicts are compared with the direction of independent per-response scores. Decisive excludes ties in either judgment; agreement rates use this subset.

Benchmark	Instruct	Ordinary DPO	$f_{2}$ Criterion	$f_{7}$ Criterion
GSM8K accuracy	$0.8945$	$0.8955$	$\mathbf{0.9015}$	$0.8910$
GSM8K pass@4	$0.928$	$0.928$	$\mathbf{0.932}$	$0.926$
MATH-Hard accuracy	$0.9070$	$0.9105$	$\mathbf{0.9125}$	$0.9095$
MATH-Hard pass@4	$0.964$	$0.964$	$\mathbf{0.966}$	$0.962$
MuSR accuracy	$\mathbf{0.5870}$	$0.5825$	$0.5795$	$0.5840$
MuSR pass@4	$0.706$	$\mathbf{0.708}$	$\mathbf{0.708}$	$0.686$

Appendix

Table 5: Pilot accuracy and pass@4. Instruct is the untrained starting model. Trained conditions share pairs, labels, objective, data order, and LoRA configuration, differing only in criterion text. Bold marks row maxima, including ties.

Condition	GSM8K	MMLU-Pro	TruthfulQA	ARC	BBH	MuSR
OFF	$0.8995$	$\mathbf{0.6390}$	$\mathbf{0.6955}$	$\mathbf{0.9070}$	$0.4310$	$0.5755$
$R_{1}$ ON	$0.8960$	$0.6205$	$0.6895$	$0.8945$	$0.4185$	$0.5640$
$R_{2}$ ON	$\mathbf{0.9055}$	$0.6285$	$0.6905$	$0.9000$	$\mathbf{0.4680}$	$\mathbf{0.5775}$

Appendix

Table 6: Rubric reinsertion on the same checkpoint and matched sampled items (accuracy fractions).

Mean factor-score gap
Factor	GSM8K	MATH-Hard	BBH	MuSR	ARC	TruthfulQA	MMLU-Pro
$f_{1}$	$+0.080$	$+0.039$	$+0.023$	$+0.005$	$+0.110$	$+0.100$	$+0.069$
$f_{2}$	$\mathbf{+0.170}$	$\mathbf{+0.246}$	$+0.066$	$+0.047$	$\mathbf{+0.248}$	$\mathbf{+0.157}$	$\mathbf{+0.235}$
$f_{3}$	$-0.051$	$+0.090$	$+0.018$	$+0.025$	$+0.105$	$+0.145$	$+0.142$
$f_{4}$	$+0.036$	$+0.054$	$+0.028$	$+0.006$	$+0.093$	$\mathbf{+0.150}$	$+0.091$
$f_{5}$	$\mathbf{+0.158}$	$\mathbf{+0.486}$	$\mathbf{+0.232}$	$+0.038$	$+0.148$	$+0.034$	$\mathbf{+0.399}$

Appendix

Table 7: Qwen3-4B diagnostic matrix: mean success–failure gaps on the

0

–

4

judge scale, followed by Cohen’s

d

. Bold shaded gaps are at least

0.15

Factor	$\mu_{k}$	$C_{k}$	$P_{k}$	Tie rate	Reduction
$f_{1}$ task comprehension	$0.1042$	$0.3379$	$0.2118$	$0.6621$	—
$f_{2}$ reasoning validity	$0.0908$	$\mathbf{0.1463}$	$0.1032$	$\mathbf{0.8537}$	$13\%$
$f_{3}$ verification and calibration	$0.0818$	$0.2202$	$0.1506$	$0.7798$	—
$f_{4}$ factual and evidential grounding	$0.0866$	$\mathbf{0.1493}$	$0.0992$	$0.8507$	—
$f_{5}$ completeness	$0.1556$	$\mathbf{0.4743}$	$0.3037$	$0.5257$	$38\%$
$f_{6}$ instruction adherence	$0.0988$	$0.3405$	$0.2046$	$0.6595$	—

Appendix

Table 8: HelpSteer3 supply statistics over all assessable pairs: mean signed contrast

\mu_{k}

, contrast coverage

C_{k}=\Pr(|\delta|\geq 1)

, and positive fraction

P_{k}=\Pr(\delta>0)

. Shading marks

C_{k}<0.25

. Reduction is the relative percentage drop in mean contrast after length matching, where reported.

Figure 5: Length sensitivity of HelpSteer3 supply estimates. Points show mean chosen–rejected gaps for four factors on the

0

–

4

judge scale, using all

38{,}173

pairs or

9{,}551

length-matched pairs (

|\Delta L|\leq 164

characters).

\Delta

is matched minus all.

Cell	Comparison	$\Delta$	Reading
4B $\times$ HS3	Common $-$ Ordinary DPO	$+0.0090$	positive
	Common $+$ Specific $-$ Ordinary DPO	$+0.0047$	positive
	Common $+$ Low $-$ Ordinary DPO	$+0.0040$	positive
	RubricOpt -R1 $-$ Ordinary DPO	$+0.0023$	null
	RubricOpt -R2 ( keep ) $-$ RubricOpt -R1	$+0.0309$	positive
	RubricOpt -R2 ( refine ) $-$ RubricOpt -R2 ( keep )	$-0.0025$	null

Appendix

Table 9: Full-subtask BBH differences within each model–dataset cell (accuracy fractions). Shading marks the matched refine -minus- keep comparison.

Qwen3-4B
	HelpSteer3 comprehensive capability (%)								HelpSteer3 open-ended generation
Method	GSM8K	MATH- Hard	BBH	MuSR	ARC	Truthful QA	MMLU- Pro	Avg.	MT- Bench	Arena- Hard	Wild- Bench	Avg. $\Delta$
Instruct	$85.90$	$62.24$	$12.82$	$\mathbf{43.39}$	$62.80$	$52.27$	$41.98$	$51.63$	$6.320$	$49.0\%$	$3.730$	$-2.96$
Ordinary DPO	$\mathbf{87.57}$	$63.64$	$14.73$	$43.12$	$62.63$	$\mathbf{53.91}$	$\mathbf{42.30}$	$52.56$	$6.890$	$50.9\%$	$3.617$	$0.00$
Common	$87.04$	$63.48$	$\mathbf{15.64}$	$43.12$	$63.23$	$53.72$	$42.23$	$52.64$	$6.720$	$52.3\%$	$3.711$	$+0.96$
Common $+$ Specific	$87.04$	$63.74$	$15.21$	$42.99$	$\mathbf{63.40}$	$53.71$	$42.25$	$52.62$	$6.610$	$\mathbf{53.1\%}$	$\mathbf{3.840}$	$+2.14$

Appendix

Table 10: Per-benchmark factor-set results on Qwen3-4B with HelpSteer3. Accuracy and Arena-Hard win rate are percentages. Avg. is the unweighted seven-task accuracy mean. Open-ended protocols use native scores.

\mathrm{Avg.}\,\Delta

is the relative change against ordinary DPO across MT-Bench, Arena-Hard, and WildBench. Each rubric includes Common, with Specific and/or Low-demand factors added. Bold marks column maxima. Shading marks rubric-trained rows.

Benchmark	$f_{1}$	$f_{2}$	$f_{3}$	$f_{4}$	$f_{5}$	$f_{6}$	$f_{7}$	$f_{8}$
GSM8K	$+0.113$	$+0.130$	$-0.096$	$+0.014$	$\mathbf{+0.189}$	$+0.058$	$-0.065$	$\mathbf{+0.171}$
MATH-Hard	$+0.011$	$+0.079$	$\mathbf{+0.182}$	$+0.013$	$+0.144$	$+0.058$	$\mathbf{+0.163}$	$+0.080$
BBH	$-0.038$	$-0.007$	$+0.142$	$+0.001$	$-0.100$	$-0.028$	$\mathbf{-0.171}$	$+0.073$
MuSR	$+0.022$	$+0.042$	$+0.039$	$+0.012$	$+0.042$	$+0.022$	$+0.038$	$+0.024$
ARC	$+0.072$	$\mathbf{+0.167}$	$+0.109$	$+0.089$	$+0.141$	$+0.033$	$-0.006$	$-0.035$
TruthfulQA	$+0.016$	$+0.053$	$+0.078$	$+0.100$	$-0.017$	$-0.031$	$-0.131$	$+0.039$

Appendix

Table 11: Qwen3-8B success–failure gaps on the

0

–

4

judge scale, pooling diagnostic and held-out halves within the model. Bold shaded cells have

|\mathrm{gap}|\geq 0.15

. The Qwen3-4B matrix is in Tab. 7 .

Rubric-Guided Generation

Rubric-guided generation uses explicit, task-specific criteria to steer, score, and improve model outputs when quality is hard to measure automatically.

See all 42 papers

Rubrics from the Task, Verdicts from Evidence: Pointwise Rubric Rewards with Tool-Augmented Judges
Oct 13, 2026Tianci Liu, Haoyu WangReward Modeling Rubric-Based Evaluation
University of Tennessee · University at Albany
Visual Preference Optimization with Rubric Rewards
Apr 14, 2026Ya-Qi Yu, Fangyu Hong, Xiangyang Qu +15Preference Optimization Multimodal Reward Modeling
Huawei Technologies Co., Ltd.
Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR
May 19, 2026Utkarsh Tyagi, Xingang Guo, MohammadHossein Rezaei +5Reward Modeling Rubric-Based RL
Scale AI · Persona

Oct 13, 2026J/K move · Enter open · S save

Rubrics from the Task, Verdicts from Evidence: Pointwise Rubric Rewards with Tool-Augmented Judges

Tianci Liu, Haoyu Wang

University of Tennessee · University at Albany

Reward Modeling Rubric-Based Evaluation LLM-as-a-Judge

Open paper AskPDF arXiv

Rubric-based reward models (RMs) evaluate responses against explicit criteria. However, existing methods often derive criteria from observed response candidates, introducing response-induced bias that may limit rubric generalizability. Their judges also struggle with mechanical constraints when relying on reading alone, and commonly used pairwise judges provide only relative preferences, complicating their direct use for policy optimization. We propose ReVeR, a pointwise rubric RM that derives hard rules and graded principles by reasoning about the instruction alone and evaluates responses independently through tool-augmented judging. Both components are trained through label-blind distillation, in which preference labels only filter a teacher's outputs, yielding ReVeR-Traces, a large-scale dataset of rubric-reasoning and judging trajectories. Experiments demonstrate that ReVeR outperforms pairwise rubric RMs in average preference accuracy, most clearly on instruction following, as well as in test-time scaling and downstream policy optimization. We further find that generator RL improves preference accuracy while reducing rubric fidelity, and that higher preference accuracy does not necessarily translate into better downstream policies. These findings highlight that rubric fidelity, preference accuracy, and downstream utility are distinct properties, shedding light on future RM research.

Abstract

Rubric-Guided Generation

Explore similar work

Rubrics from the Task, Verdicts from Evidence: Pointwise Rubric Rewards with Tool-Augmented Judges

Visual Preference Optimization with Rubric Rewards

Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR