Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization

Shiping Gao; Hongzhan Chen; Xiaojun Quan; Qifan Wang; Lifu Huang

cs.CLApr 14, 2026

Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization

Authors: Shiping Gao, Hongzhan Chen, Xiaojun Quan, Qifan Wang, Lifu Huang

Organizations: School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, Guangdong, China · Shenzhen Loop Area Institute, Shenzhen, Guangdong, China · Meta AI, Menlo Park, California, USA · Computer Science Department, University of California, Davis, Davis, California, USA

Abstract

Process reward models (PRMs) provide fine-grained supervision for reasoning, but reliable PRMs often require step annotations or heavy verification pipelines, making them costly to scale and refresh during online RL. Implicit PRMs reduce this cost by training log-likelihood-ratio rewards from trajectory-level outcome labels. However, the log-ratio is constrained only as a sequence-level aggregate during training, while inference decomposes it into token- or step-level scores for partial prefixes. This train-inference mismatch leaves local credits weakly identified, so distribution-wide scoring can amplify misleading advantages. We propose Implicit Prefix-Value Reward Model (IPVRM), which directly learns the probability of eventual correctness for each prefix from outcome labels. Step signals are then obtained as temporal-difference (TD) differences between consecutive prefix values, aligning the training target with inference-time use. IPVRM markedly improves step-verification F1 on ProcessBench. To exploit these prefix values during policy optimization, we further introduce Distribution-Level RL (DistRL), which applies TD advantages to both sampled tokens and high-probability candidate tokens, providing dense counterfactual updates without additional rollouts. Experiments show that DistRL brings limited gains with unreliable implicit rewards, but consistently improves downstream reasoning when paired with IPVRM. The implementation of our method is available at https://github.com/gaoshiping/IPVRM .

Figures & tables

Figure 1: Paradigm comparison and qualitative trade-offs. Existing paradigms either incur high annotation/inference costs (Explicit/Generative PRMs) or suffer from a train–inference objective mismatch (Implicit PRMs). Our proposed IPVRM resolves these limitations by directly learning prefix-conditioned values $V_{\phi}(s_{t})$ , achieving both low-cost efficiency and training consistency (”what we train is what we use”) to support Distribution-Level RL.

Figure 2: Performance comparison on Qwen3-0.6B. IPVRM improves both step-level error localization on ProcessBench (Left) and sequence-level reranking in Best-of- $N$ selection (Middle) over prior implicit reward models. Leveraging these better signals, DistRL further outperforms standard GRPO in downstream policy performance (AVG@8) (Right).

Figure 3: Overview of the IPVRM+DistRL framework.

Figure 4: Illustration of DistRL. Unlike standard RL that optimizes solely on the sampled trajectory (red bar), DistRL introduces a dual-branch mechanism. It simultaneously updates high-probability candidate tokens (blue bars) using dense one-step TD advantages, while refining the sampled token (red bar) using a hybrid of GAE and TD advantages derived from IPVRM estimates.

Figure 5: Schematic of Adaptive Difficulty Boundary (ADB) and Dynamic Loss Weighting (DLW). As illustrated, ADB shifts the sigmoid decision boundary by $V(x)$ (the expected accuracy), while DLW reweights samples. This focuses learning on hard negatives for easy prompts (left) and hard positives for hard prompts (right), mitigating label imbalance.

Base Model	RM Method	MATH-500			Minerva-Math			Avg.
Base Model	RM Method	@4	@16	@64	@4	@16	@64	Avg.
Qwen3-0.6B Base	Q-RM	41.4	44.4	44.6	9.2	12.1	10.7	27.1
	EndoRM	32.0	33.8	38.8	6.3	8.8	7.7	21.2
	DPO-RM	41.6	42.6	42.2	10.7	9.9	11.0	26.3
	Implicit PRM	39.6	39.6	38.6	10.3	11.4	11.0	25.1
	IPVRM	42.0	46.0	47.6	10.3	11.0	11.4	28.1

Table 1: Different reward models’ best-of-N sampling performance on MATH-500 and Minerva-Math with SFT models (Avg is computed by averaging over the two datasets, where each dataset score is the mean over @4/@16/@64).

Base Model	RM Method	GSM8K	MATH	Olympiad Bench	Omni Math	Avg.
Qwen3-0.6B base	Q-RM	47.2	38.9	31.7	35.0	38.2
	EndoRM	33.7	30.1	26.9	31.5	30.6
	DPO-RM	32.1	32.7	23.5	22.9	27.8
	Implicit PRM	30.3	31.0	21.5	18.8	25.4
	IPVRM	52.8	42.1	30.0	31.3	39.1
Qwen3-8B base	Q-RM	61.0	45.2	39.5	35.3	45.3

Table 2: Performance (

F_{1}

score) of different reward modeling methods on ProcessBench .

Base Model	RM Method	MATH-500			Avg.
Base Model	RM Method	@4	@16	@64	Avg.
Qwen3-0.6B base	IPVRM ${}_{\text{late}}$	42.4	47.0	50.0	46.5
	IPVRM	42.0	46.0	47.6	45.2
	IPVRM ${}_{\text{early}}$	41.0	44.2	44.2	43.1
Llama3.2-1B base	IPVRM ${}_{\text{late}}$	14.4	14.6	17.0	15.3
	IPVRM	13.9	14.2	14.8	14.3

Table 3: Variants of IPVRM: best-of-N sampling performance on MATH-500 with SFT model.

Base Model	RM Method	GSM8K	MATH	Olympiad Bench	Omni Math	Avg.
Qwen3-0.6B base	IPVRM ${}_{\text{late}}$	50.8	40.3	28.4	29.2	37.2
	IPVRM	52.8	42.1	30.0	31.3	39.1
	IPVRM ${}_{\text{early}}$	49.9	43.4	32.2	34.8	40.1
Llama3.2-1B base	IPVRM ${}_{\text{late}}$	41.1	33.8	27.1	27.6	32.4
	IPVRM	42.7	35.2	28.7	29.2	34.0
	IPVRM ${}_{\text{early}}$	45.0	34.8	30.1	31.4	35.3

Table 4: Performance (

F_{1}

score) of different IPVRM variants on ProcessBench .

Base Model	Method	AIME	MATH 500	Minerva Math	Olympiad Bench	AMC	Avg.
Qwen3-0.6B base	SFT	0.4	33.2	7.3	7.6	9.3	11.6
	PRIME	1.1	39.6	11.4	11.4	12.3	15.2
	SPRO	1.7	38.6	10.7	11.6	13.7	15.3
	Reinforce w/ Q-RM	1.1	36.4	9.2	11.0	10.8	13.7
	GRPO w/ VR	0.8	38.3	10.5	10.5	12.5	14.5
	GRPO w/ DPO-RM	0.8	38.4	9.8	10.0	11.1	14.0

Table 5: Main results for policy model (AVG@8)

Base Model	Method	AIME	MATH 500	Minerva Math	Olympiad Bench	AMC	Avg.
Qwen3-0.6B base	GRPO w/ DPO-RM	0.8	38.4	9.8	10.0	11.1	14.0
	DistRL w/ DPO-RM	0.8	37.2	9.4	10.1	13.0	14.1
	GRPO w/ Implicit PRM	1.2	34.9	10.2	9.5	10.8	13.3
	DistRL w/ Implicit PRM	0.8	37.1	10.9	10.5	14.2	14.7
	GRPO w/ IPVRM	1.2	41.5	12.1	12.9	13.9	16.3
	DistRL w/ IPVRM	1.5	42.5	13.3	12.1	14.0	16.7

Table 6: Impact of reward models and DistRL across mathematical benchmarks (AVG@8).

	RM score	Verifier acc.
Qwen3-0.6B	$-0.140$ / $-0.181$	$23.5$ / $23.0$
Qwen3-8B	$-0.178$ / $-0.202$	$41.6$ / $41.4$
Llama3.2-1B	$-0.040$ / $-0.050$	$7.7$ / $7.4$
Llama3.2-3B	$-0.024$ / $-0.025$	$15.9$ / $15.5$

Table 7: Training-time analysis of TD. DistRL achieves higher RM score and verifier accuracy than GRPO across all backbones. Each entry is reported as DistRL / GRPO .

Figure 6: Ablation of online IPVRM update strategies on Llama3.2-1B. Left: outcome training reward during RL. Right: RM accuracy across training samples (

\times 16384

Method	AIME	MATH500	Minerva	Olympiad	AMC	Avg
Implicit PRM w/o ADB/DLW	0.8	37.1	10.9	10.5	14.2	14.7
IPVRM w/o ADB/DLW	1.0	40.8	11.2	12.2	12.0	15.4
IPVRM w/ ADB/DLW	1.5	42.5	13.3	12.1	14.0	16.7

Table 8: Policy performance under different online RM update settings on Qwen3-0.6B with DistRL.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Base Model	Reference Choice	AIME	MATH 500	Minerva Math	Olympiad Bench	AMC	Avg.
Qwen3-0.6B base	SFT Model	1.4	39.8	12.1	11.3	11.3	15.2
Qwen3-0.6B base	Behavior Policy	1.5	42.5	13.3	12.1	14.0	16.7
Llama3.2-1B base	SFT Model	0.3	10.3	2.9	3.5	2.1	3.8
Llama3.2-1B base	Behavior Policy	0.1	13.2	3.4	3.1	4.2	4.8

Appendix

Table 9: Impact of reference choice on distribution-level RL across mathematical benchmarks (AVG@8).

$m$	BoN-AVG	PB-AVG
0	26.1	37.0
5	28.1	39.1
10	27.5	39.1

Appendix

Table 10: Sensitivity to the margin

m

on Qwen3-0.6B.

$P_{\min}$	Avg Candidate Size	Prob. Mass Coverage
0.05	1.54	97.57%
0.10	1.33	96.11%
0.20	1.16	93.58%

Appendix

Table 11: Sensitivity to the candidate threshold

P_{\min}

on Qwen3-0.6B.

Table 12: Best-of-

N

sampling accuracy (%) on AMC-23 with SFT models.

Model	Method	Peak Memory (GB)	Avg Time / Step (s)
Qwen3-0.6B	GRPO	30.20	487.5 $\pm$ 27.1
Qwen3-0.6B	DistRL	30.88	458.7 $\pm$ 33.2
Llama3.2-1B	GRPO	31.30	349.8 $\pm$ 12.6
Llama3.2-1B	DistRL	31.52	319.6 $\pm$ 13.0

Appendix

Table 13: Peak GPU memory usage and average wall-clock time per training step for IPVRM with DistRL and GRPO.

Protocol	Method	GSM8K	MATH	Olympiad	OmniMath	Avg
Prefix-based	DPO-RM	27.0	27.4	19.8	18.4	23.2
	Implicit PRM	30.3	28.2	19.9	16.9	23.8
	IPVRM (Ours)	42.6	36.4	24.3	25.2	32.1
Process-level	DPO-RM	32.1	32.7	23.5	22.9	27.8
	Implicit PRM	30.3	31.0	21.5	18.8	25.4
	IPVRM (Ours)	52.8	42.1	30.0	31.3	39.1

Appendix

Table 14: ProcessBench

F_{1}

under different evaluation protocols.

Process Reward Models

Process reward models assess intermediate reasoning steps and actions to guide learning, search, and evaluation.

See all 140 papers

Learning Process Rewards via Reasoning State Propagation
Sep 30, 2026Kai Gan, Zi-Hao Zhou, Bo Ye +3Process Reward Models RL for Language Model Reasoning
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China · Zhongguancun Academy +1
Unsupervised Process Reward Models
May 11, 2026Artyom Gadetsky, Maxim Kodryan, Siba Smarak Panigrahi +2Reward Modeling Process Reward Models
Swiss Federal Institute of Technology (EPFL)
The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning
Jun 8, 2026Aakriti Agrawal, Souradip Chakraborty, Armin Saghafian +6Process Reward Models RL for Language Model Reasoning
University of Maryland, College Park · Capital One · University of Central Florida

Sep 30, 2026J/K move · Enter open · S save

Learning Process Rewards via Reasoning State Propagation

Kai Gan, Zi-Hao Zhou, Bo Ye, Jian Zhao+2

School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China · Zhongguancun Academy+1

Process Reward Models RL for Language Model Reasoning Temporal Credit Assignment

Open paper AskPDF arXiv

Process reward models (PRMs) have demonstrated notable effectiveness in test-time scaling and reinforcement learning by providing fine-grained signals for evaluating intermediate reasoning states, but their training relies heavily on costly process annotations. A natural way to alleviate this dependence is to complement limited process supervision with scalable outcome supervision. However, existing PRMs often model reasoning prefixes independently, providing no explicit mechanism for effectively using final outcome to guide the learning of intermediate reasoning states. We introduce Reasoning State Propagation (RSP), which represents each reasoning prefix with a binary validity state and models transitions between successive states across the reasoning trajectory. Specifically, RSP predicts a break probability that a valid state becomes invalid and a repair probability that an invalid state returns to valid. By propagating these transitions, RSP connects intermediate states to the final state, allowing process annotations to supervise intermediate states while outcome labels supervise the final state and can provide learning signals to preceding steps. Across reasoning search, response selection, and reinforcement learning, RSP consistently outperforms representative PRM baselines, with average improvements over Qwen2.5-Math-PRM of 5.6% in beam search and 2.1% in reinforcement learning.

Abstract

Process Reward Models

Explore similar work

Learning Process Rewards via Reasoning State Propagation

Unsupervised Process Reward Models

The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning