Rubric-based Reinforcement Learning (RL) has emerged as a promising approach for aligning Large Language Models (LLMs) with complex, open-domain instruction following tasks. However, existing methods predominantly rely on response-level rewards, introducing severe reward sparsity and reward ambiguity problems. To address these issues, we propose Rubrics to Tokens (RTT), a novel rubric-based RL framework that bridges coarse response-level scores and fine-grained token-level credit assignment. RTT introduces a Token-Level Relevance Discriminator to predict which tokens in the response are responsible for a specific constraint, and optimizes the policy model via RTT-GRPO, which integrates response-level and token-level advantages within a unified framework. Furthermore, when transitioning from one-dimensional, outcome-level reward to three-dimensional reward space in the token-level rubric-based RL, we propose a novel group normalization method, called Intra-sample Token Group Normalization, to accommodate this shift. Extensive experiments and benchmarks demonstrate that RTT consistently outperforms other baselines in both instruction- and rubric-level accuracy across different models.
Figures & tables
Figure 1: Architecture of the RTT framework. The policy model first generates a response group for instructions. The top computes A^res(i,t) by normalizing rewards across the sampled responses. The bottom uses a token-level discriminator to map rewards to specific tokens based on rubrics, applying Intra-Sample Group Normalization to calculate A^tok(i,t) . These two components are merged into the final aggregated advantage A^sum(i,t) to drive the policy update.
Figure 2: Reward structure under standard GRPO and RTT-GRPO. In GRPO (left), each of G sampled completions receives one outcome-level reward, advantages are normalized along the sample axis only. The RTT-GRPO (right) assigns rewards over three dimensions, motivating two group normalization strategies, inter-sample and intra-sample group normalization.
IFEval
IFBench
MulDimIF
AdvancedIF
Model
Prompt
Instruct
Prompt
Instruct
Acc.
Overall
Rubric
Qwen3-4B-Instruct †
83.40 ↑ 0.0
88.13 ↑ 0.0
30.95 ↑ 0.0
32.83 ↑ 0.0
56.08 ↑ 0.0
44.31 ↑ 0.0
79.17 ↑ 0.0
+ SFT
83.73 ↑ 0.3
88.85 ↑ 0.7
29.59 ↓ 1.4
31.64 ↓ 1.2
56.50 ↑ 0.4
45.07 ↑ 0.8
79.23 ↑ 0.1
+ DPO
82.62 ↓ 0.8
87.77 ↓ 0.4
29.93 ↓ 1.0
32.84 ↑ 0.0
55.83 ↓ 0.3
45.71 ↑ 1.4
79.33 ↑ 0.2
+ RL-CSR
84.29 ↑ 0.9
89.21 ↑ 1.1
32.31 ↑ 1.4
33.43 ↑ 0.6
74.38 ↑ 18.3
46.02 ↑ 1.7
79.81 ↑ 0.6
+ RL-AON
82.99 ↓ 0.4
88.61 ↑ 0.5
32.99 ↑ 2.0
35.52 ↑ 2.7
74.25 ↑ 18.2
47.29 ↑ 3.0
79.69 ↑ 0.5
Table 1: Main results on instruction-following benchmarks. Fine-tuned method rows show scores with delta ( ↑ / ↓ ) relative to the base model. Highlighted rows are our proposed RTT variants. Best results per model group are bolded and second-best are underlined . † We use discriminator initialized from Qwen3-4B-Base, ⋆ initialized from Llama3.2-3B-Base.
Model
Math-500
GPQA
MMLU-Pro
Qwen3-4B-Instruct
87.20
51.79
69.60
+ RTT-AON (Ours)
91.80 ↑ 4.6
52.01 ↑ 0.2
66.76 ↓ 2.8
Qwen2.5-7B-Instruct
74.80
30.80
56.43
+ RTT-AON (Ours)
73.80 ↓ 1.0
31.25 ↑ 0.5
56.12 ↓ 0.3
Llama3.2-3B-Instruct
36.60
25.89
38.96
+ RTT-AON (Ours)
40.00 ↑ 3.4
26.12 ↑ 0.2
39.04 ↑ 0.1
Table 2: RTT on out-of-domain benchmarks.
IFEval
IFBench
MulDimIF
AdvancedIF
Model / Strategy
Prompt
Instruct
Prompt
Instruct
Acc.
Overall
Rubric
Qwen3-4B-Instruct
w/ Inter-sample group normalization
82.99
88.49
30.95
32.84
75.08
44.67
78.73
w/ Intra-sample group normalization
85.21
88.73
34.69
37.61
76.75
47.48
80.79
Qwen2.5-7B-Instruct
w/ Inter-sample group normalization
82.07
86.81
28.57
29.25
69.75
31.61
72.04
Table 3: Ablation on group partitioning strategies across all model configurations. We compare Inter-sample vs. Intra-sample Token Group Normalization for advantage estimation. Best results per model are bolded .
IFEval
IFBench
MulDimIF
AdvancedIF
Strategy
Prompt
Instruct
Prompt
Instruct
Acc.
Overall
Rubric
Base Model: Qwen3-4B-Instruct-2507
Random
70.43
79.50
32.99
35.52
61.75
46.50
78.39
Uniform (RL-AON)
82.99
88.61
32.99
35.52
74.25
47.29
79.69
Greedy
85.03
88.49
30.95
32.84
75.08
46.20
79.82
Ours (RTT-AON)
85.21
88.73
34.69
37.61
76.75
47.48
80.79
Table 4: Ablation on token weighting strategies for RL training.
Figure 3: Sensitivity of RTT to β . Baseline β=0 is shown as a horizontal line.
Figure 4: Training dynamics of RTT-CSR vs. RL-CSR baseline. Left side shows model performance on IFEval, IFBench, and MulDimIF benchmarks, while right side shows response entropy.
Figure 5: Training stability comparison between RTT-CSR and RL-CSR on Llama3.2-3B-Instruct. We report rollout accuracy, entropy, KL loss, and clip fraction over training. RL-CSR exhibits severe instability, while RTT-CSR remains stable.
Training Phase
GPU hours
Fraction
Rollout Generation
1,160.7
76.8%
Policy Gradient Update
131.6
8.7%
Others (e.g., Communication)
101.1
6.7%
Discriminator Forward
117.9
7.8%
RTT Total
1,511.3
100%
RL Baseline (w/o Discriminator)
1,393.4
—
Table 5: GPU-hour breakdown for 500-step RL training with Qwen3-4B-Instruct on A100-80GB GPUs. The first three rows constitute the shared RL pipeline; the discriminator forward pass (below the rule) is the only additional cost introduced by RTT.
Constraint
Token Attribution
Make sure your response in all capital letters.
Explanation: Since the constraint applies globally, the discriminator assigns high relevance to every token in both the satisfying and violating responses. Even when the violating response switches to lowercase mid-sentence, the dense attribution reflects that the entire output is implicated in the constraint.
Your response should start with “My Answer:”
Explanation: The discriminator concentrates high relevance on the opening tokens “My” and “Answer:” in the satisfying response. In the violating response, which begins with “Answer:” (missing “My”), relevance scores collapse across the entire output, as the pattern match fails from the first token.
Do not include word ‘ride’ in your response.
Explanation: All tokens receive near-zero relevance in the first response, confirming the constraint is met. In the violating response, relevance spikes sharply at the forbidden word “ride” and its neighbors, localizing the violation site.
State that recent global warming is primarily caused by human greenhouse gas emissions.
Explanation: The discriminator also captures semantic content. In the satisfying response, it assigns high relevance to the statement directly expressing the correct claim. In the violating response, relevance concentrates on the misattributing statement, so the corresponding tokens can receive negative reward signals.
Table 6: Token attribution visualization for different constraint types. Darker shading indicates higher discriminator relevance score.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
CoT prompt
MATH-500
Question: {}\nPlease reason step by step, and put your final answer within \boxed{}.
GPQA & MMLU-Pro
Question: {}\nAnswer the multiple choice question. The last line of your response should be of the following format: ’Answer: $LETTER’ (without quotes) where LETTER is one of choices. Think step by step before answering.
Appendix
Table 7: CoT evaluation prompts for out-of-domain benchmarks.
Rubric-based reinforcement learning enriches language model training by evaluating model outputs against explicit criteria. Yet in GRPO-style pipelines, these structured judgments are reduced to a scalar response-level reward and converted into a response-level advantage, which is broadcast uniformly to all generated tokens. This leaves no explicit mechanism for allocating credit within a response, even when different criteria are grounded in different spans, formatting decisions, or semantic choices. We propose CoRT, a token-level credit weighting method for rubric-conditioned GRPO. Instead of training an auxiliary token scoring model, CoRT uses counterfactual replay to rescore the same sampled response under the original rubric-conditioned prompt and a matched criteria-free prompt. The resulting tokenwise log-likelihood contrasts serve as a proxy for dependence on the rubric context. CoRT maps these contrasts to bounded, response-normalized weights and uses them to redistribute the signed GRPO advantage across tokens, without introducing an auxiliary scorer or changing the response-level reward. Experiments across instruction-tuned models and reward granularities show that CoRT improves over matched response-level GRPO in the vast majority of comparisons, with an average gain of 4.4 percentage points. The method remains competitive with learned token-level credit baselines while avoiding a separate relevance-learning stage. These results suggest that policy-internal counterfactual likelihood contrasts provide an effective training signal for within-response credit allocation while retaining the simplicity and stability of GRPO.
Bo-Wen Zhang, Junwei He, Wen Wang +5
State Key Laboratory of Novel Software Technology, Nanjing University · School of Intelligence Science and Technology, Nanjing University · ByteDance +1
Rubric-based reinforcement learning (RL) evaluates responses against explicit criteria to construct rewards, offering a viable way to train large language models on open-ended tasks. However, standard methods compress criterion-level feedback into scalar rewards and assign uniform token advantages, leaving structured rubric supervision underutilized. Recent work seeks finer-grained guidance by treating rubrics as privileged information to derive self-distillation targets, but the resulting learning signal is not grounded in criterion outcomes and can lead to reward degradation. This motivates us to reframe the problem as criterion-aware credit assignment, where privileged rubric information serves as a relevance cue for credit allocation rather than a source of imitation targets. Accordingly, we propose CoCA, which derives criterion-level credits from group-relative outcomes and allocates them across tokens based on criterion-induced distribution shifts, yielding both outcome-grounded and fine-grained learning signals. Empirically, CoCA achieves an average gain of +4.2% over standard rubric-based RL across medicine and science benchmarks, demonstrating more effective learning on open-ended tasks.
Reinforcement learning from human feedback (RLHF) has become the dominant paradigm for aligning large language models (LLMs) with human preferences. However, traditional RLHF relies on scalar reward signals that lack interpretability and fail to capture the multifaceted nature of response quality. Rubric-guided reinforcement learning addresses these limitations by introducing structured, interpretable evaluation criteria, or rubrics, as the backbone of reward design, feedback generation, and policy optimization. In this survey, we introduce a Bayesian framework that defines constitutions as prior distributions P(R) over evaluation criteria and rubrics as conditional instantiations Rx∼P(R∣x). Under this unified view, we present a taxonomy of rubric-guided RL along the prior-posterior axis, covering constitutional AI, instance-specific rubrics, process-level supervision, self-evolving rubrics, and their agentic and multimodal extensions. Furthermore, as rubrics are natural-language artifacts, we present a linguistic analysis of how granularity trade-offs, semantic drift, and linguistic reward hacking impact alignment reliability, identifying key open problems for future research.