Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks

Tianze Xu; Yanzhao Zheng; Pengrui Lu; Lyumanshan Ye; Yong Wu; Zhentao Zhang; Yuanqiang Yu; Chao Ma

cs.CLApr 3, 2026

Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks

Authors: Tianze Xu, Yanzhao Zheng, Pengrui Lu, Lyumanshan Ye, Yong Wu, Zhentao Zhang, Yuanqiang Yu, Chao Ma,

Organizations: Shanghai Jiao Tong University · GAIR · Alibaba Group · Shanghai Innovation Institute ·

Abstract

Rubric-based Reinforcement Learning (RL) has emerged as a promising approach for aligning Large Language Models (LLMs) with complex, open-domain instruction following tasks. However, existing methods predominantly rely on response-level rewards, introducing severe reward sparsity and reward ambiguity problems. To address these issues, we propose Rubrics to Tokens (RTT), a novel rubric-based RL framework that bridges coarse response-level scores and fine-grained token-level credit assignment. RTT introduces a Token-Level Relevance Discriminator to predict which tokens in the response are responsible for a specific constraint, and optimizes the policy model via RTT-GRPO, which integrates response-level and token-level advantages within a unified framework. Furthermore, when transitioning from one-dimensional, outcome-level reward to three-dimensional reward space in the token-level rubric-based RL, we propose a novel group normalization method, called Intra-sample Token Group Normalization, to accommodate this shift. Extensive experiments and benchmarks demonstrate that RTT consistently outperforms other baselines in both instruction- and rubric-level accuracy across different models.

Figures & tables

Figure 1: Architecture of the RTT framework. The policy model first generates a response group for instructions. The top computes $\smash{\hat{A}_{res}^{(i,t)}}$ by normalizing rewards across the sampled responses. The bottom uses a token-level discriminator to map rewards to specific tokens based on rubrics, applying Intra-Sample Group Normalization to calculate $\smash{\hat{A}_{tok}^{(i,t)}}$ . These two components are merged into the final aggregated advantage $\smash{\hat{A}_{sum}^{(i,t)}}$ to drive the policy update.

Figure 2: Reward structure under standard GRPO and RTT-GRPO. In GRPO (left), each of $G$ sampled completions receives one outcome-level reward, advantages are normalized along the sample axis only. The RTT-GRPO (right) assigns rewards over three dimensions, motivating two group normalization strategies, inter-sample and intra-sample group normalization.

	IFEval		IFBench		MulDimIF	AdvancedIF
Model	Prompt	Instruct	Prompt	Instruct	Acc.	Overall	Rubric
Qwen3-4B-Instruct †	83.40 $\uparrow$ 0.0	88.13 $\uparrow$ 0.0	30.95 $\uparrow$ 0.0	32.83 $\uparrow$ 0.0	56.08 $\uparrow$ 0.0	44.31 $\uparrow$ 0.0	79.17 $\uparrow$ 0.0
+ SFT	83.73 $\uparrow$ 0.3	88.85 $\uparrow$ 0.7	29.59 $\downarrow$ 1.4	31.64 $\downarrow$ 1.2	56.50 $\uparrow$ 0.4	45.07 $\uparrow$ 0.8	79.23 $\uparrow$ 0.1
+ DPO	82.62 $\downarrow$ 0.8	87.77 $\downarrow$ 0.4	29.93 $\downarrow$ 1.0	32.84 $\uparrow$ 0.0	55.83 $\downarrow$ 0.3	45.71 $\uparrow$ 1.4	79.33 $\uparrow$ 0.2
+ RL-CSR	84.29 $\uparrow$ 0.9	89.21 $\uparrow$ 1.1	32.31 $\uparrow$ 1.4	33.43 $\uparrow$ 0.6	74.38 $\uparrow$ 18.3	46.02 $\uparrow$ 1.7	79.81 $\uparrow$ 0.6
+ RL-AON	82.99 $\downarrow$ 0.4	88.61 $\uparrow$ 0.5	32.99 $\uparrow$ 2.0	35.52 $\uparrow$ 2.7	74.25 $\uparrow$ 18.2	47.29 $\uparrow$ 3.0	79.69 $\uparrow$ 0.5

Table 1: Main results on instruction-following benchmarks. Fine-tuned method rows show scores with delta (

\uparrow

\downarrow

) relative to the base model. Highlighted rows are our proposed RTT variants. Best results per model group are bolded and second-best are underlined . † We use discriminator initialized from Qwen3-4B-Base, ⋆ initialized from Llama3.2-3B-Base.

Model	Math-500	GPQA	MMLU-Pro
Qwen3-4B-Instruct	87.20	51.79	69.60
+ RTT-AON (Ours)	91.80 $\uparrow$ 4.6	52.01 $\uparrow$ 0.2	66.76 $\downarrow$ 2.8
Qwen2.5-7B-Instruct	74.80	30.80	56.43
+ RTT-AON (Ours)	73.80 $\downarrow$ 1.0	31.25 $\uparrow$ 0.5	56.12 $\downarrow$ 0.3
Llama3.2-3B-Instruct	36.60	25.89	38.96
+ RTT-AON (Ours)	40.00 $\uparrow$ 3.4	26.12 $\uparrow$ 0.2	39.04 $\uparrow$ 0.1

Table 2: RTT on out-of-domain benchmarks.

	IFEval		IFBench		MulDimIF	AdvancedIF
Model / Strategy	Prompt	Instruct	Prompt	Instruct	Acc.	Overall	Rubric
Qwen3-4B-Instruct
w/ Inter-sample group normalization	82.99	88.49	30.95	32.84	75.08	44.67	78.73
w/ Intra-sample group normalization	85.21	88.73	34.69	37.61	76.75	47.48	80.79
Qwen2.5-7B-Instruct
w/ Inter-sample group normalization	82.07	86.81	28.57	29.25	69.75	31.61	72.04

Table 3: Ablation on group partitioning strategies across all model configurations. We compare Inter-sample vs. Intra-sample Token Group Normalization for advantage estimation. Best results per model are bolded .

	IFEval		IFBench		MulDimIF	AdvancedIF
Strategy	Prompt	Instruct	Prompt	Instruct	Acc.	Overall	Rubric
Base Model: Qwen3-4B-Instruct-2507
Random	70.43	79.50	32.99	35.52	61.75	46.50	78.39
Uniform (RL-AON)	82.99	88.61	32.99	35.52	74.25	47.29	79.69
Greedy	85.03	88.49	30.95	32.84	75.08	46.20	79.82
Ours (RTT-AON)	85.21	88.73	34.69	37.61	76.75	47.48	80.79

Table 4: Ablation on token weighting strategies for RL training.

Figure 3: Sensitivity of RTT to

\beta

. Baseline

\beta=0

is shown as a horizontal line.

Figure 4: Training dynamics of RTT-CSR vs. RL-CSR baseline. Left side shows model performance on IFEval, IFBench, and MulDimIF benchmarks, while right side shows response entropy.

Figure 5: Training stability comparison between RTT-CSR and RL-CSR on Llama3.2-3B-Instruct. We report rollout accuracy, entropy, KL loss, and clip fraction over training. RL-CSR exhibits severe instability, while RTT-CSR remains stable.

Training Phase	GPU hours	Fraction
Rollout Generation	1,160.7	76.8%
Policy Gradient Update	131.6	8.7%
Others (e.g., Communication)	101.1	6.7%
Discriminator Forward	117.9	7.8%
RTT Total	1,511.3	100%
RL Baseline (w/o Discriminator)	1,393.4	—

Table 5: GPU-hour breakdown for 500-step RL training with Qwen3-4B-Instruct on A100-80GB GPUs. The first three rows constitute the shared RL pipeline; the discriminator forward pass (below the rule) is the only additional cost introduced by RTT.

Constraint	Token Attribution
Make sure your response in all capital letters.	Explanation: Since the constraint applies globally, the discriminator assigns high relevance to every token in both the satisfying and violating responses. Even when the violating response switches to lowercase mid-sentence, the dense attribution reflects that the entire output is implicated in the constraint.
Your response should start with “My Answer:”	Explanation: The discriminator concentrates high relevance on the opening tokens “My” and “Answer:” in the satisfying response. In the violating response, which begins with “Answer:” (missing “My”), relevance scores collapse across the entire output, as the pattern match fails from the first token.
Do not include word ‘ride’ in your response.	Explanation: All tokens receive near-zero relevance in the first response, confirming the constraint is met. In the violating response, relevance spikes sharply at the forbidden word “ride” and its neighbors, localizing the violation site.
State that recent global warming is primarily caused by human greenhouse gas emissions.	Explanation: The discriminator also captures semantic content. In the satisfying response, it assigns high relevance to the statement directly expressing the correct claim. In the violating response, relevance concentrates on the misattributing statement, so the corresponding tokens can receive negative reward signals.

Table 6: Token attribution visualization for different constraint types. Darker shading indicates higher discriminator relevance score.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	CoT prompt
MATH-500	Question: {}\nPlease reason step by step, and put your final answer within \boxed{}.
GPQA & MMLU-Pro	Question: {}\nAnswer the multiple choice question. The last line of your response should be of the following format: ’Answer: $LETTER’ (without quotes) where LETTER is one of choices. Think step by step before answering.

Appendix

Table 7: CoT evaluation prompts for out-of-domain benchmarks.

(a) Training configuration of SFT and DPO
Method	SFT & DPO
Training	per_device_train_batch_size = 4 & 2, gradient_accumulation_steps = 64 & 128, learning_rate = $10^{-6}$ , lr_scheduler_type = constant, cutoff_len = 2048, warmup_steps = 10, epochs = 4
Optimizations	deepspeed: z2 & z3, bf16

Appendix

Table 8: Training configurations across methods and model backbones.

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 433 papers

CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization
Jul 28, 2026Bo-Wen Zhang, Junwei He, Wen Wang +5Token-Level Credit Assignment Rubric-Based RL
State Key Laboratory of Novel Software Technology, Nanjing University · School of Intelligence Science and Technology, Nanjing University · ByteDance +1
CoCA: Criterion-Aware Credit Assignment for Rubric-Based Reinforcement Learning
Oct 13, 2026Guangcheng Zhu, Yingfan Ma, Xing Zheng +7RL for Language Models Rubric-Based RL
Zhejiang University · Ant Group
A Survey on Rubric-Guided Reinforcement Learning for Language Models
Aug 27, 2026Zifei Shan, Fangning ShaoRL for Language Models LLM Alignment
WeChat, Tencent · Independent Researcher

Jul 28, 2026J/K move · Enter open · S save

CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization

Bo-Wen Zhang, Junwei He, Wen Wang, Song-Lin Lv+4

State Key Laboratory of Novel Software Technology, Nanjing University · School of Intelligence Science and Technology, Nanjing University · ByteDance+1

Token-Level Credit Assignment Rubric-Based RL Policy Gradient Methods Credit Assignment Counterfactual Credit Assignment

Open paper AskPDF arXiv

Rubric-based reinforcement learning enriches language model training by evaluating model outputs against explicit criteria. Yet in GRPO-style pipelines, these structured judgments are reduced to a scalar response-level reward and converted into a response-level advantage, which is broadcast uniformly to all generated tokens. This leaves no explicit mechanism for allocating credit within a response, even when different criteria are grounded in different spans, formatting decisions, or semantic choices. We propose CoRT, a token-level credit weighting method for rubric-conditioned GRPO. Instead of training an auxiliary token scoring model, CoRT uses counterfactual replay to rescore the same sampled response under the original rubric-conditioned prompt and a matched criteria-free prompt. The resulting tokenwise log-likelihood contrasts serve as a proxy for dependence on the rubric context. CoRT maps these contrasts to bounded, response-normalized weights and uses them to redistribute the signed GRPO advantage across tokens, without introducing an auxiliary scorer or changing the response-level reward. Experiments across instruction-tuned models and reward granularities show that CoRT improves over matched response-level GRPO in the vast majority of comparisons, with an average gain of 4.4 percentage points. The method remains competitive with learned token-level credit baselines while avoiding a separate relevance-learning stage. These results suggest that policy-internal counterfactual likelihood contrasts provide an effective training signal for within-response credit allocation while retaining the simplicity and stability of GRPO.

Abstract

RL for Language Models

Explore similar work

CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization

CoCA: Criterion-Aware Credit Assignment for Rubric-Based Reinforcement Learning

A Survey on Rubric-Guided Reinforcement Learning for Language Models