Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models

Kyuyoung Kim; Kevin Wang; Yunfei Xie; Peiyang Xu; Peiyao Sheng; Chen Wei; Zhangyang Wang; Jinwoo Shin

cs.CLApr 3, 2026

Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models

Authors: Kyuyoung Kim, Kevin Wang, Yunfei Xie, Peiyang Xu, Peiyao Sheng, Chen Wei, Zhangyang Wang, Jinwoo Shin,

Organizations: KAIST AI · University of Texas, Austin · Rice University · Princeton University ·

Abstract

Training language models to produce both correct answers and sound reasoning remains an open challenge. Reinforcement learning with verifiable rewards typically optimizes only final outcomes, which can improve task accuracy at the expense of reasoning quality, producing inaccurate, incomplete, or inconsistent traces. We propose verifiable process supervision (VPS), a post-training framework that jointly optimizes prediction accuracy and reasoning quality by supervising structured intermediate claims. We first apply supervised fine-tuning to induce a structured reasoning format, enabling deterministic extraction and verification of intermediate claims for process-level rewards. To address the heterogeneous difficulty of reasoning subtasks, we introduce adaptive weighting that prioritizes components with the largest remaining errors, creating an implicit curriculum. We evaluate VPS on chess as a controlled testbed where reasoning steps can be deterministically verified against engine signals. While outcome-only RL improves move accuracy, it sharply degrades reasoning quality, increasing win-rate error by up to 112% and reducing internal consistency by up to 69%. In contrast, VPS preserves accuracy while significantly improving reasoning quality, reducing win-rate error by up to 30% and restoring consistency to near saturation. A reasoning-space analysis further shows that, without a structured prior, outcome-only RL converges to budget-dependent shortcuts rather than sound multi-step reasoning. Beyond chess, we observe the same phenomenon on math reasoning, where outcome-only RL improves accuracy while degrading step-level arithmetic and consistency, whereas VPS maintains both. These results show that VPS enables language models to reason both accurately and reliably in verifiable domains.

Figures & tables

Figure 1: Overview of VPS. (1) A structured reasoning prior is induced via supervised fine-tuning, enabling syntactic extraction of intermediate claims. (2) During RL, these claims are verified against ground-truth signals to produce process-level rewards. (3) The rewards are adaptively weighted based on subtask performance, focusing learning on components with the largest remaining errors and inducing a curriculum over reasoning subskills.

Figure 2: Example synthetic reasoning trace. Colors indicate individual verifiable claims.

Model	Prediction Accuracy			Reasoning Quality
Model	Top-1 Acc. $\uparrow$	Elo $\uparrow$	Coverage $\uparrow$	WR MAE $\downarrow$	PV Overlap $\uparrow$	Consist. $\uparrow$
Qwen3-8B
SFT only	0.212	1160	0.207	0.310	0.323	0.961
SFT + GRPO	0.534	1670	0.302	0.452	0.183	0.721
VPS	0.536	1706	0.441	0.218	0.376	0.978
R1-Distill-Llama-8B

Table 1: Accuracy and reasoning quality comparison. Prediction accuracy and reasoning quality metrics for Qwen3-8B and DeepSeek-R1-Distill-Llama-8B. Reasoning quality is evaluated using a deterministic verifier on held-out positions; lower win-rate MAE and higher values on other metrics indicate better reasoning.

Model	Acc. $\uparrow$	Relev. $\uparrow$	Compl. $\uparrow$	Clarity $\uparrow$	Fluency $\uparrow$	Overall $\uparrow$
Qwen3-8B
GRPO only (best acc.)	0.507	2.501	2.296	2.429	3.102	2.582
VPS (matched acc.)	0.505	2.856	3.239	3.751	3.833	3.420
GRPO only (same step)	0.500	2.484	2.292	2.390	3.275	2.610
R1-Distill-Llama-8B
GRPO only (best acc.)	0.539	2.057	1.862	1.985	3.164	2.267

Table 2: Accuracy-controlled reasoning comparison. Reasoning quality is evaluated using three LLM judges (GPT-4o, Claude Opus 4.6, Gemini 3 Flash) along four dimensions (1–5 scale). We compare the highest-accuracy GRPO-only checkpoint with a VPS checkpoint at matched accuracy, and include the GRPO-only checkpoint at the same training step.

Method	Acc.	Step Arith.	Consist.
SFT only	$0.446$	$0.638$	$0.821$
GRPO-only	$0.760$	$0.681$	$0.647$
SFT + GRPO	$0.760$	$0.658$	$0.621$
VPS	$\bf{0.763}$	$\bf{0.905}$	$\bf{0.889}$

Table 3: GSM8K results. Outcome-only RL improves accuracy but degrades reasoning quality, whereas VPS preserves both.

Figure 3: Reward weighting across SFT scales. Adaptive weighting improves the hardest reasoning subtasks, particularly when models start with stronger domain knowledge.

Panel of Figure 4 — Figure 4: Qualitative reasoning comparison. Outcome-only optimization produces flawed or inconsistent reasoning, while VPS yields concise and factually grounded explanations.

Figure 5: Reasoning space. Outcome-only RL produces variable reasoning patterns without a structured prior.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

	Hyperparameter	Value
Training	Epochs	2
	Batch size	256
	Gradient clipping	1.0
	Precision	bf16
Optimization	Learning rate	1e-5
	LR scheduler	cosine

Appendix

Table 4: Supervised fine-tuning hyperparameters.

	Hyperparameter	Value
Training	Training steps	300
	Learning rate	1e-6
	PPO epochs per rollout	1
	KL coefficient	0.001
	Discount factor ( $\gamma$ )	1.0
Data	Prompts per batch	128

Appendix

Table 5: Reinforcement learning hyperparameters.

Model	Relev. $\uparrow$	Compl. $\uparrow$	Clarity $\uparrow$	Fluency $\uparrow$	Overall $\uparrow$
Qwen3-8B
GRPO only (best acc.)	2.370	2.053	2.773	3.657	2.713
VPS (matched acc.)	2.658	2.727	4.379	3.903	3.417
GRPO only (same step)	2.384	2.048	2.773	3.911	2.779
R1-Distill-Llama-8B
GRPO only (best acc.)	1.946	1.728	2.302	3.698	2.418

Appendix

Table 6: Reasoning comparison with GPT-4o. Accuracy-controlled evaluation of reasoning quality using GPT-4o as the judge.

Model	Relev. $\uparrow$	Compl. $\uparrow$	Clarity $\uparrow$	Fluency $\uparrow$	Overall $\uparrow$
Qwen3-8B
GRPO only (best acc.)	2.669	2.230	2.124	2.267	2.323
VPS (matched acc.)	3.257	3.381	3.581	3.146	3.341
GRPO only (same step)	2.618	2.229	2.057	2.325	2.307
R1-Distill-Llama-8B
GRPO only (best acc.)	2.300	1.870	1.813	2.277	2.065

Appendix

Table 7: Reasoning comparison with Claude Opus 4.6. Accuracy-controlled evaluation of reasoning quality using Claude Opus 4.6 as the judge.

Model	Relev. $\uparrow$	Compl. $\uparrow$	Clarity $\uparrow$	Fluency $\uparrow$	Overall $\uparrow$
Qwen3-8B
GRPO only (best acc.)	2.464	2.604	2.390	3.382	2.710
VPS (matched acc.)	2.652	3.610	3.293	4.449	3.501
GRPO only (same step)	2.451	2.600	2.339	3.588	2.745
R1-Distill-Llama-8B
GRPO only (best acc.)	1.925	1.990	1.840	3.517	2.318

Appendix

Table 8: Reasoning comparison with Gemini 3 Flash. Accuracy-controlled evaluation of reasoning quality using Gemini 3 Flash as the judge.

Model	Relev. $\uparrow$	Compl. $\uparrow$	Clarity $\uparrow$	Fluency $\uparrow$	Overall $\uparrow$
GPT-4o judge
LLM reward	2.129	1.860	2.186	2.816	2.248
VPS	2.658	2.727	4.379	3.903	3.417
Gemini 3 Flash judge
LLM reward	2.461	2.453	2.104	3.579	2.649
VPS	2.652	3.610	3.293	4.449	3.501

Appendix

Table 9: Comparison with LLM-based process supervision. Reasoning quality of Qwen3-8B trained using GPT-4o-mini as the process reward model compared with VPS, evaluated using two independent LLM judges.

Model	Reasoning Quality
Model	Capture $\uparrow$	Check $\uparrow$	Move Div. $\uparrow$	Form. Err. $\downarrow$
Qwen3-8B
SFT only	0.399	0.651	2.193	0.003
SFT + GRPO	0.215	0.936	1.647	0.003
VPS	0.615	0.988	3.098	0.002
R1-Distill-Llama-8B

Appendix

Table 10: Accuracy and reasoning quality comparison. Additional reasoning quality metrics for Qwen3-8B and DeepSeek-R1-Distill-Llama-8B.

Figure 6: Adaptive subtask weighting dynamics. ( Left ) Running EMA-based estimates of per-subtask model performance. ( Center ) Corresponding weights allocated to each subtask that are dynamically adjusted. ( Right ) Subtask rewards on held-out positions.

Model	Prediction Accuracy			Reasoning Quality
Model	Top-1 Acc. $\uparrow$	Elo $\uparrow$	Coverage $\uparrow$	WR MAE $\downarrow$	PV Overlap $\uparrow$	Consist. $\uparrow$
Qwen3-8B
GRPO only	0.507	1550	–	–	–	–
SFT only	0.212	1160	0.207	0.310	0.323	0.961
SFT + GRPO	0.534	1670	0.302	0.452	0.183	0.721
VPS	0.536	1706	0.441	0.218	0.376	0.978

Appendix

Table 11: Model scale comparison. Prediction accuracy and reasoning quality metrics for Qwen3-8B, Qwen3-4B, and Qwen3-1.7B.

Model	Reasoning Quality
Model	Capture $\uparrow$	Check $\uparrow$	Move Div. $\uparrow$	Form. Err. $\downarrow$
Qwen3-8B
SFT only	0.399	0.651	2.193	0.003
SFT + GRPO	0.215	0.936	1.647	0.003
VPS	0.615	0.988	3.098	0.002
Qwen3-4B

Appendix

Table 12: Model scale comparison. Additional reasoning quality metrics for Qwen3-8B, Qwen3-4B, and Qwen3-1.7B.

Panel of Figure 7 — Figure 7: Accuracy vs. reasoning length under different budgets. Short traces achieve high accuracy, while longer traces correlate with substantially lower accuracy. Distributions show that larger budgets induce longer, more dispersed reasoning traces.

Table 13: Faithfulness and robustness of learned reasoning. ( Left ) Sensitivity of model predictions to perturbations of intermediate reasoning. ( Right ) Prediction accuracy under alternative prompting schemes. Results are averaged over three random seeds.

Panel of Figure 8 — Figure 8: Qualitative reasoning comparison. GRPO exhibits repetition and non-termination, while SFT + GRPO produces superficial, overconfident reasoning. VPS yields grounded, calibrated analysis with correct move selection.

Panel of Figure 9 — Figure 9: Qualitative reasoning comparison. Baselines fixate on Qxf6+ and produce superficial or templated analysis, while VPS correctly prioritizes Rg8+ based on its forcing continuation, yielding more grounded reasoning.

Panel of Figure 10 — Figure 10: Qualitative reasoning comparison. Baseline models perform minimal analysis or rely on shortcut heuristics, repeating checking moves without position-specific evaluation.

Method	Accuracy $\uparrow$	Step Arith. $\uparrow$	Ans. Cons. $\uparrow$
SFT baseline	$0.446$	$0.638$	$0.821$
Step arith. only	0.753	0.906	$0.817$
Ans. cons. only	$0.752$	$0.863$	0.865
VPS	0.763	0.905	0.889

Appendix

Table 14: Contribution of individual verifier signals on GSM8K. Each variant is trained with a single verifier signal or the full objective. Results are averaged over three seeds.

Method	Accuracy $\uparrow$	Step Arith. $\uparrow$	Ans. Cons. $\uparrow$
SFT baseline	0.528	0.669	0.851
SFT + GRPO	0.721	0.618	0.731
VPS	0.722	0.974	0.958

Appendix

Table 15: Generalization to a non-reasoning language model. Results on GSM8K using Qwen3-0.6B-Base, the non-reasoning counterpart of Qwen3-0.6B. The qualitative findings mirror those observed for reasoning models.

Process Supervision

Process supervision trains and evaluates AI systems on intermediate reasoning steps, not just final answers, to improve reliability and control.

See all 74 papers

GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning
Apr 22, 2026Jingyi Wang, Lei Zhu, Tengjin Weng +8RL for Language Model Reasoning Group Relative Policy Optimization
Tsinghua University · Huawei Noah’s Ark Lab · Shenzhen University
Learning to Generate Formally Verifiable Step-by-Step Logic Reasoning via Structured Formal Intermediaries
Mar 31, 2026Luoxin Chen, Yichi Zhou, Huishuai ZhangLLM Reasoning LLM Reliability
Wangxuan Institue of Computer Technology, Peking University. · ByteDance.
Answer-Conditioned Chains of Thought Degrade Verifiable-Reasoning Distillation in Large Language Models
Jul 16, 2026Jungseob Lee, Seungyoon Lee, Suhyune Son +4CoT Distillation Language Model Distillation
1Korea University · 2Zoom Communications · 3Yonsei University

Apr 22, 2026J/K move · Enter open · S save

GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning

Jingyi Wang, Lei Zhu, Tengjin Weng, Song-Li Wu+7

Tsinghua University · Huawei Noah’s Ark Lab · Shenzhen University

RL for Language Model Reasoning Group Relative Policy Optimization Reinforcement Learning with Verifiable Rewards Process Supervision

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) has advanced the reasoning capabilities of Large Language Models (LLMs) by leveraging direct outcome verification instead of learned reward models. Building on this paradigm, Group Relative Policy Optimization (GRPO) eliminates the need for critic models but suffers from indiscriminate credit assignment for intermediate steps, which limits its ability to identify effective reasoning strategies and incurs overthinking. In this work, we introduce a model-free and verifiable process supervision via probing the model's belief in the correct answer throughout its reasoning trajectory. By segmenting the generation into discrete steps and tracking the conditional probability of the correct answer appended at each segment boundary, we efficiently compute interpretable segment-wise progress measurements to refine GRPO's trajectory-level feedback. This approach enables more targeted and sample-efficient policy updates, while avoiding the need for intermediate supervision derived from costly Monte Carlo rollouts or auxiliary models. Experiments on mathematical and general-domain benchmarks show consistent gains over GRPO across diverse models: up to 2.6-point accuracy improvements and 13.7% reasoning-length reductions on math tasks, and up to 2.4 points and 4% on general-domain tasks, demonstrating strong generalization.

Abstract

Process Supervision

Explore similar work

GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning

Learning to Generate Formally Verifiable Step-by-Step Logic Reasoning via Structured Formal Intermediaries

Answer-Conditioned Chains of Thought Degrade Verifiable-Reasoning Distillation in Large Language Models