Are Dense Rewards Worth Their Tokens in GRPO?
Organizations: Department of Computer and Data Sciences, Case Western Reserve University, Cleveland, OH, USA.
Abstract
RLVR trains language models to reason by rewarding correct final answers. A widely used algorithm, group-relative policy optimization (GRPO), gives every token of a solution the same credit, so dense process rewards have been proposed to credit intermediate steps, for example with a Monte-Carlo reward that scores each step by sampling continuations from it. Comparing such a reward with GRPO per training update, or vetting it with a gradient test on a single batch, leaves out what it costs: in our setting, the continuations alone take 91% of all generated tokens. We introduce a matched-compute protocol that gives every training intervention the same budget, measured in generated tokens and training GPU-hours, and tests it in preregistered studies with controls for side effects. Across 1,668 runs on six models, matching compute reverses the ranking. On Qwen2.5-1.5B, the Monte-Carlo reward leads standard GRPO per update, yet ends 14.3 percentage points lower at equal tokens and 7.8 points lower at equal training time, and none of four dense-reward variants outperforms GRPO even with as much training time. A per-token gradient test orders these rewards as training does but overestimates each by one to two points, as much as the gains it is meant to detect. In contrast, training on the prompts whose predicted success is nearest 1/2 improves accuracy at matched compute, by 1.7 points on Qwen3-0.6B and 0.8 on Qwen2.5-1.5B. These gains replicate on fresh seeds and unseen problems, and a simple rule, that the candidate prompts are predicted to fail more often than not, predicted where they appear in five of six preregistered tests. Both gains come early in training, and standard GRPO with twice the initial learning rate comes within half a point of them. We argue that training interventions should be validated at matched compute, which overturns some apparent gains and confirms others.
Figures & tables
| Setting | Cand. | Predicted | Result |
|---|---|---|---|
| Qwen3-1.7B, L5 | gain ✓ | ||
| Qwen3-0.6B, L1–2 | none ✓ | ||
| Llama-3.2-3B | none ✓ | ||
| Llama-3.2-1B | gain | ||
| Qwen3-0.6B, | loss ✓ | ||
| Qwen2.5-1.5B, | loss ✓ |
Appendix figures & tables38 assets
Supplementary material from the paper’s appendix.
Appendix
| Qwen | Llama-3.2 | |||||
| Intervention | 2.5-1.5B | 3-0.6B | 3-1.7B | 3-4B | 1B | 3B |
| Completion limit | ||||||
| Monte-Carlo reward | – | – | ||||
| Learned-critic reward | – | – | ||||
| Rollout reuse, VinePPO-style | – | – | – | – | – | |
| Continuation reuse | – | – | – | – | – | |
| Method | Model, training pool | accuracy CI | Test | ||
| Dense rewards within a completion | −0.10 0 +0.10 | ||||
| Monte-Carlo reward, | Qwen2.5-1.5B, L4–5 | Holm | |||
| same, at matched updates | Qwen2.5-1.5B, L4–5 | exploratory | |||
| fresh seeds | Qwen2.5-1.5B, L4–5 | secondary | |||
| Monte-Carlo reward, | Qwen2.5-1.5B, L4–5 | secondary | |||
| Monte-Carlo reward, | Qwen2.5-1.5B, L4–5 | Holm | |||
| Method | Model, training pool | accuracy CI | Test | ||
| Critic filter at the other four models | −0.04 0 +0.04 | ||||
| Critic filter | Qwen3-1.7B, L3–5 | estimate, TOST | |||
| fresh seeds of one study | Qwen3-1.7B, L3–5 | secondary, TOST | |||
| Critic filter | Qwen3-1.7B, level 5 | Holm | |||
| Critic filter | Qwen3-4B, L4–5 | Holm | |||
| Critic filter | Qwen3-4B, L3–5 | TOST | |||
| Study | Artifact | Question | Model, pool | Where | |
|---|---|---|---|---|---|
| S1 | P3 | Does the Monte-Carlo reward pay for its continuations? | Qwen2.5-1.5B, L4–5; Qwen3, three sizes; GSM8K | ; each | C.3 |
| S2 | P4 | Longer completions; gradient efficiency per update and per token | Qwen2.5-1.5B, L4–5 | C.3 | |
| S3 | P5 | Pilot: reusing samples | Qwen2.5-1.5B, L4–5 | E | |
| S4 | P6 | Learned critic; pilot of other estimators | Qwen2.5-1.5B, L4–5 | ; | C.3 |
| S5 | P9 | Learned critic on other models | Qwen3, three sizes, L4–5 | C.3 | |
| S6 | P10 | Does the critic filter help? | Qwen3 0.6B+1.7B, L4–5 | C.4 |
| Comparison | Model, pool | Estimate | Test | ||
|---|---|---|---|---|---|
| S1 | Monte-Carlo reward, | Qwen2.5-1.5B, L4–5 | Holm | ||
| at matched updates ( vs. ) | exploratory | ||||
| centered, max-normalized rule minus it | TOST | ||||
| Monte-Carlo reward, | Qwen3-0.6B, L4–5 | ||||
| Qwen3-1.7B, L4–5 | |||||
| Qwen3-4B, L4–5 |
| Configuration | accuracy | wrong | correct | mixed | prompts |
|---|---|---|---|---|---|
| Outcome only | |||||
| Random choice | |||||
| Filter, | |||||
| Filter, | |||||
| Filter, |
| training reward | |||||||
| Setting | candidates | filter | outcome-only | iteration 0 | start | filter outcome | |
| Qwen3-0.6B, L4–5 | (0.32–0.35) | (0.30–0.31) | |||||
| Qwen3-1.7B, level 5 | (0.38–0.41) | (0.34–0.36) | |||||
| Qwen2.5-1.5B, L3–5 | (0.38–0.41) | (0.35–0.38) | |||||
| Qwen3-0.6B, L3–5 | (0.40–0.43) | (0.37–0.40) | |||||
| Llama-3.2-1B, L3–5 | (0.39–0.44) | (0.36–0.42) | |||||
| Model | configuration | pred. | real. | real. pred. | ||
|---|---|---|---|---|---|---|
| Qwen3-0.6B | random | |||||
| filter | ||||||
| Qwen3-1.7B | random | |||||
| filter | ||||||
| Qwen2.5-1.5B | random | |||||
| filter |
| Model | Configuration ( ) | pass@1 | pass@2 | pass@4 | pass@8 | base |
|---|---|---|---|---|---|---|
| Qwen3-0.6B | filter ( ) | |||||
| random ( ) | ||||||
| filter, default reward, ( ) | ||||||
| filter, strict reward ( ) | ||||||
| same, ( ) | ||||||
| Qwen2.5-1.5B | filter ( ) |
| Setting ( ) | target | tokens | training time |
|---|---|---|---|
| Qwen3-0.6B, L3–5 ( ) | |||
| strict reward ( ) | – | ||
| strict reward, target at ( ) | – | ||
| Qwen2.5-1.5B, L3–5 ( ) | |||
| history filter ( ) | |||
| Qwen3-1.7B, level 5 ( ) |
| Study | Model, pool, horizon | outcome-only | critic filter | random choice |
| S14 | Qwen3-4B, level 5, | ( ) | ( ) | – |
| S18 | same, strict training reward | ( ) | – | – |
| S16 | Qwen3-0.6B, L3–5, | ( ) | ( ) | – |
| S17 | same, strict training reward | ( ) | ( ) | – |
| S23 | Qwen3-0.6B, L3–5, , learning rate | ( ) | ( ) | – |
| S34 | Qwen3-0.6B, L3–5, , fresh seeds | ( ) | ( ) | – |
| Configuration | Base | Equal tokens | Equal training time | Hours to (configuration / base) | |
|---|---|---|---|---|---|
| Monte-Carlo, | outcome-only | / | |||
| Monte-Carlo, | outcome-only | / | |||
| Monte-Carlo, | outcome-only | / | |||
| Critic, TD form | outcome-only | / | |||
| Critic, baseline form | outcome-only | / | |||
| Monte-Carlo, , tokens | outcome-only | / |
| Configuration | Select | Generate | Credit | Update | Total |
|---|---|---|---|---|---|
| Outcome-only (L4–5) | |||||
| Monte-Carlo, | |||||
| Monte-Carlo, | |||||
| Monte-Carlo, | |||||
| Critic, TD form | |||||
| Critic, baseline form |
| Equal training time (primary) | Equal tokens | Earlier time at | Hours | ||||
|---|---|---|---|---|---|---|---|
| Configuration | CI | Holm | TOST | Lower | CI | CI | to |
| Outcome-only GRPO | – | – | – | ||||
| Monte-Carlo, | |||||||
| Monte-Carlo, | |||||||
| VinePPO-style, steps per batch | |||||||
| Continuation reuse, | |||||||
| VinePPO | Our VinePPO-style variant | SPO-tree; TreeRL | Our continuation reuse | |
|---|---|---|---|---|
| Credit | for each reasoning step; advantages normalized over the batch | at a segment’s end minus at its start, with the leave-one-out group mean and ; not normalized | SPO-tree: a node’s value minus its siblings’ mean, on tokens with probability below . TreeRL: , over | Rollouts: group advantage plus segment credit (the reward). Continuations: their outcome minus the other draw’s at their cut |
| Branch points | Every reasoning step (newline and punctuation boundaries) | fixed offsets ( , , …, tokens) | SPO-tree: fixed token intervals, levels. TreeRL: the most uncertain tokens in each of trees | The same offsets; one level |
| Samples per point | rollouts | continuations | SPO-tree: children per node ( paths). TreeRL: branches per fork ( responses) | continuations |
| Trains on continuations | No | No | SPO-tree: segments with nonzero credit. TreeRL: every root-to-leaf path | Every continuation, with the prompt and prefix as context |
| Shared prefixes | – | – | SPO-tree: one training segment per node. TreeRL: in every path through them, downweighted | Trained once, in the rollout |
| Update | Clipped ( ), epochs, minibatch , episodes per step; KL penalty | Clipped ( ), epochs minibatches of , completions per iteration; no KL penalty | SPO-tree: clipped, epoch, KL . TreeRL: policy gradient, one step per prompts, KL | One on-policy step over rollouts and continuations, the loss divided by their number; no clipping or KL penalty |
| Training reward | gain (final start) | vs. outcome | regime | ||
|---|---|---|---|---|---|
| GRPO (outcome) | – | – | baseline | ||
| Monte-Carlo, | equivalent | ||||
| Monte-Carlo, | equivalent | ||||
| Monte-Carlo, | equivalent | ||||
| Monte-Carlo, | harm (bootstrap only) | ||||
| Monte-Carlo, | harm (all tests) |
| Dense reward | vs. outcome ( ) | vs. ungated, same | ||
| Sign-opposition gate, -problem evaluation: | ||||
| sign-opposition, expanded | ( ) | – | ||
| sign-opposition, exploratory | ( ) | – | ||
| Disagreement gate and ungated reward, -problem evaluation: | ||||
| rollout-disagreement | ( ) | |||
| ungated (ungated) | ( ) | – | ||
| Setting | Value | Setting | Value |
|---|---|---|---|
| policy | Qwen2.5-1.5B-Instruct | optimizer | Adam, |
| trained parameters | all ( B, full FT) | grad-norm clip | |
| iterations | inner epochs per iteration | (single step) | |
| prompts per iteration | microbatch (completions) | ||
| group size | ratio clipping | none (on-policy) | |
| max new tokens | (full FT) |
| Dense reward | |||
|---|---|---|---|
| implicit reward | |||
| MC-within | |||
| MC-shuffled control |
| (MC shuf) | family | seeds/ | ||||
|---|---|---|---|---|---|---|
| Model | (residual) | MC | shuf | [ CI] | [ CI] | rows |
| Qwen2.5-1.5B | ||||||
| Qwen2.5-Math-1.5B | ||||||
| SmolLM2-1.7B | ||||||
| Phi-3.5-mini-3.8B | ||||||
| Qwen2.5-Math-7B | ||||||
| Training reward | gain (final start) | vs. outcome ( ) | regime | |
| MC-within, | ( ) | transition | ||
| MC-within, | ( ) | transition | ||
| MC-shuffled control, | ( ) | uncertain | ||
| MC-shuffled control, | ( ) | uncertain | ||
| MC-shuffled control, | ( ) | uncertain | ||
| Co-equal standardization (same MC reward, , segments, seeds – ): | ||||
| Training reward | gain (final start) | vs. outcome ( ) | collapse events | |
|---|---|---|---|---|
| GRPO (outcome) | – | |||
| MC-within, | ( ) | |||
| MC-within, | ( ) | |||
| MC-within, | ( ) | |||
| MC-within, | ( ) | |||
| MC-within, | ( ) |
| Family / reward | gain | CI | vs. outcome | regime | |
|---|---|---|---|---|---|
| SmolLM3-3B, outcome | – | baseline | |||
| inconclusive | |||||
| deficit (desc.) | |||||
| deficit (desc.) |
| Training reward | gain (final start) | vs. outcome ( ) | regime | |
|---|---|---|---|---|
| GRPO (outcome) | – | baseline | ||
| PRM, | ( ) | inconclusive | ||
| PRM, | ( ) | deficit (desc.) | ||
| PRM, | ( ) | deficit (desc.) |
| Dataset / reward | gain | CI | vs. outcome | regime | |
|---|---|---|---|---|---|
| NuminaMath, outcome | – | baseline | |||
| gain (desc.) | |||||
| deficit (desc.) |
| Symbol | Meaning |
|---|---|
| Group size, completion length and parameter dimension | |
| Group-relative completion coefficient on the sampled batch equation 1 | |
| Token score and completion score sum | |
| Sampled score matrix, completion-sum operator and score Gram matrix | |
| Conditional score contribution of token coefficients | |
| Completion-constant and within-completion zero-sum parts of |
RL for Language Models
Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.
See all 460 papers