Inducing Process Supervision from Outcome-Only Reinforcement Learning

Shengda Fan; Xin Cong; Zhong Zhang; Haotian Chen; Yankai Lin

cs.LGSep 29, 2026

Inducing Process Supervision from Outcome-Only Reinforcement Learning

Authors: Shengda Fan, Xin Cong, Zhong Zhang, Haotian Chen, Yankai Lin

Organizations: Renmin University of China · Tsinghua University · University of Electronic Science and Technology of China · Shanghai Jiao Tong University

Abstract

Process reward models (PRMs) have become a key component for LLMs, as their step-level feedback supports both post-training and test-time reasoning. However, training strong PRMs remains costly: human step annotation is difficult to scale, while Monte Carlo estimation is computationally expensive and can drift from the intrinsic correctness of steps. To get effective PRMs at low cost, we introduce TIPS (Thinking-Induced Process Supervision), an outcome-only reinforcement learning (RL) framework for training generative PRMs. In TIPS, the model generates a chain-of-thought (CoT) followed by step-level labels and an outcome label. The reward depends solely on whether the predicted outcome matches the ground truth, and the resulting group-relative advantage is used to optimize the entire generated response. Intuitively, when checking intermediate steps helps determine the outcome, more accurate checks can lead to better outcome judgments and higher rewards. Outcome-only RL can therefore reinforce step-level verification without explicit process supervision. We validate the effectiveness of TIPS across math and agent benchmarks and four backbone families. Notably, TIPS-Qwen3-4B-Thinking-2507 reaches 85.2 F1 on ProcessBench with only 3.2K outcome-labeled trajectories, surpassing all evaluated trained PRMs and strong prompt-only judges such as GPT-5.4-Instruct and Claude-4.7-Opus, while still trailing o1-mini. Code and data are available at https://github.com/RUCBM/TIPS.

Figures & tables

Figure 1: ProcessBench performance across model scales. Tips achieves strong performance using a 4B-parameter model trained on only 3.2K publicly available, outcome-labeled reasoning traces.

Figure 2: Process verification emerges in generated thoughts.

Figure 3: Overview of Tips . Given an input trajectory, the model first generates a thinking chain and then predicts both process labels and an outcome label. During RL, only the outcome-level prediction is used to compute the reward, while the generated process labels receive no direct supervision.

LLM-as-a-Judge
Model	Supervision	GSM8K	MATH	Olympiad Bench	OmniMath	Avg.
Qwen3-4B-Instruct-2507	N/A	64.8	68.7	63.6	58.1	63.8
Qwen3-4B-Thinking-2507	N/A	52.1	70.1	59.5	59.4	60.3
GPT-5.4-Instruct	N/A	89.8	84.6	74.1	69.1	79.4
Claude-4.7-Opus	N/A	85.3	82.9	75.7	73.4	79.3
o1-mini	N/A	93.2	88.9	87.2	82.4	87.9

Table 1: Evaluation results of F1 on ProcessBench (%). Human : human-annotated step labels; MC : pseudo process labels obtained via Monte Carlo estimation; Distill : stronger-model-derived supervision; Outcome : trajectory-level outcome labels only.

Model	HotPotQA		GAIA		BFCL		$\tau^{2}$ -Bench		Avg.
Qwen3-4B-Instruct-2507	57.2		35.6		34.8		42.4		42.5
w/ Tips	58.0	(+0.8)	38.0	(+2.4)	35.6	(+0.8)	48.4	(+6.0)	45.0	(+2.5)
Qwen3-4B-Thinking-2507	60.4		33.6		37.2		43.6		43.7
w/ Tips	61.2	(+0.8)	41.2	(+7.6)	43.6	(+6.4)	50.4	(+6.8)	49.1	(+5.4)
Qwen3-8B-Thinking	56.8		31.6		30.8		46.8		41.5
w/ Tips	53.6	(-3.2)	48.8	(+17.2)	32.4	(+1.6)	49.2	(+2.4)	46.0	(+4.5)

Table 2: Comparison of FirstErrAcc on AgentProcessBench (%).

Baselines and Upper Bound
Model	Supervision	GSM8K	MATH	College Math	Olympiad Bench	Minerva	Avg.
Majority Vote@8	N/A	93.8	80.1	66.4	45.9	52.6	67.8
Pass@8 (Upper Bound)	Oracle	97.2	89.0	74.5	62.7	66.9	78.1
LLM-as-a-Judge
Qwen3-4B-Instruct-2507	N/A	95.1	85.2	69.1	52.6	56.3	71.6
Qwen3-4B-Thinking-2507	N/A	95.0	87.5	69.5	56.2	55.9	72.8

Table 3: Best-of-8 reranking evaluation with Qwen2.5-7B-Instruct as the policy model.

Figure 4: Tips improves ProcessBench F1 across diverse backbones.

Figure 5: Outcome–process coupling during outcome-supervised RL. Solid lines show the absolute improvement in ORM accuracy, while dashed lines show the absolute improvement in the corresponding PRM metric. All values are measured relative to the base model. The annotated gain ratio is defined as

\Delta_{\mathrm{process}}/\Delta_{\mathrm{ORM}}

, where each

\Delta

is the endpoint improvement. Pearson

r

denotes the correlation between ORM accuracy and the corresponding process metrics.

Figure 6: Model priors affect outcome-to-process transfer. Qwen2.5-7B-Instruct improves ORM rapidly but shows degraded ProcessBench F1 at later checkpoints, while the DeepSeek-R1-distilled model maintains coupled ORM and PRM improvements.

Metric	CoT	Label-only
Base F1	63.8	47.5
PRM gain	13.9	0.7
ORM gain	17.6	-5.6
Entropy	0.31	0.002
Avg. length	1828.7	47.9
Max KL loss	$1{\times}10^{-3}$	$1.5{\times}10^{-5}$

Table 4: Prompting ablation on Qwen3-4B-Instruct-2507. Values are measured at the end of the first epoch.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	AIME 2026	HMMT Feb. 2026	ProcessBench
Qwen3-4B-Instruct-2507	53.3	36.4	63.8
+ Solver RLVR	55.8	33.3	62.8
+ TIPS	53.3	31.8	77.7
GooseReason-4B-Instruct	62.5	33.3	72.7
Qwen3-4B-Thinking-2507	80.0	53.0	60.3
+ TIPS	80.8	54.6	85.2

Appendix

Table 5: Direct problem solving and process verification. AIME 2026 and HMMT February 2026 scores are mean@4 accuracy (%). ProcessBench scores are evaluated in F1. The solver RLVR baseline is initialized from Qwen3-4B-Instruct-2507 and trained on DAPO-Math-17K.

Explore similar work

Jun 26, 2026cs.LG

The Weakest Link Tells It All: Outcome-Supervised Process Reward Modeling via Learnable Credit Assignment

Process reward models (PRMs) enhance the reasoning capabilities of large language models (LLMs) by providing fine-grained feedback, yet training PRMs typically requires expensive stepwise annotations. Outcome-supervised PRMs offer a scalable alternative by learning from final-answer correctness alone, but this introduces a fundamental credit assignment challenge, i.e., attributing outcomes to responsible reasoning steps. Existing approaches rely on either uniform or causal assignment, both of which fail to anchor credit in step correctness and thus hinder process error identification. In this work, we propose Outcome-Supervised Process Reward Modeling via Learnable Credit Assignment (LCA), an outcome-supervised PRM framework that jointly learns credit assignment and reward modeling under the principle of Weakest Link Assignment: a reasoning chain is as strong as its weakest link. To address mutual dependence between credit assignment and reward modeling, we formalize outcome-supervised PRM as a Multiple Instance Learning (MIL) problem and introduce Softmax-Weighted-Sum (SWS) pooling, an MIL pooling technique tailored for strong dependence and redundancy among reasoning states. We prove Bayes consistency of our algorithm under mild assumptions. Extensive experiments demonstrate that LCA consistently outperforms state-of-the-art outcome-supervised PRMs across multiple tasks and backbones. Code is available at https://anonymous.4open.science/r/LCA.

Tianyu Jia, Yue Fang, Hongxin Ding +6

National Engineering Research Center of Software Engineering, Peking University, Beijing, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing, China +3

Process Reward Model Credit Assignment Process-Level Supervision Reasoning Chain

Ask PDF

Dec 2, 2025cs.LG

ScalePRM: Training Process Reward Models by Scaling Verification Compute Without Ground Truth

Training process reward models (PRMs) requires step-level correctness labels, obtained either through expensive human annotation or by relying on ground-truth answers, limiting the ability to scale process-level supervision. We propose ScalePRM, which scales verification compute as an alternative: given a problem and a candidate solution, we generate multiple independent verifications of each reasoning step and aggregate their judgments to produce synthetic step-level labels without ground truth. We explore two representative inference-time scaling strategies, parallel scaling through self-consistency and sequential scaling through meta-critique, and train generative PRMs on the resulting synthetic data. On ProcessBench, a benchmark for identifying erroneous steps in mathematical reasoning, PRMs trained on step-level self-consistency data achieve 67.5 F1, surpassing reference-guided training with ground-truth access (66.4 F1) and GPT-4o as a critic (61.9 F1). When deployed as reward signals in RL training with Qwen2.5-Math-7B, our best PRM achieves 47.4% average accuracy across six mathematical reasoning benchmarks, outperforming ground-truth-based RLVR (43.9%). We also identify and address reward exploitation patterns unique to generative PRM-based RL. Our results demonstrate that scaling verification compute is a viable alternative to ground-truth supervision for training process reward models.

Salman Rahman, Sruthi Gorantla, Arpit Gupta +3

2UCLA · Work done while as an intern at Amazon AGI. · 1Amazon AGI

Process Reward Model Mathematical Reasoning Benchmarks Reinforcement Learning With Verifiable Reward Ground Truth Scaling Self-Consistency

Ask PDF

May 11, 2026cs.LG

Unsupervised Process Reward Models

Process Reward Models (PRMs) are a powerful mechanism for steering large language model reasoning by providing fine-grained, step-level supervision. However, this effectiveness comes at a significant cost: PRMs require expert annotations for every reasoning step, making them costly and difficult to scale. Here, we propose a method for training unsupervised PRMs (uPRM) that requires no human supervision, neither at the level of step-by-step annotations nor through ground-truth verification of final answers. The key idea behind our approach is to define a scoring function, derived from LLM next-token probabilities, that jointly assesses candidate positions of first erroneous steps across a batch of reasoning trajectories. We demonstrate the effectiveness of uPRM across diverse scenarios: (i) uPRM achieves up to 15% absolute accuracy improvements over the LLM-as-a-Judge in identifying first erroneous steps on the ProcessBench dataset; (ii) as a verifier for test-time scaling, uPRM performs comparably to supervised PRMs and outperforms the majority voting baseline by up to 6.9%, and (iii) when used as a reward signal in reinforcement learning, uPRM enables more robust policy optimization throughout training compared to a supervised PRM trained using ground-truth labels. Overall, our results open a path toward scalable reward modeling for complex reasoning tasks.

Artyom Gadetsky, Maxim Kodryan, Siba Smarak Panigrahi +2

Swiss Federal Institute of Technology (EPFL)

Process Reward Model LLM Reasoning Strategies Benchmark Datasets

Ask PDF