Training language models to produce both correct answers and sound reasoning remains an open challenge. Reinforcement learning with verifiable rewards typically optimizes only final outcomes, which can improve task accuracy at the expense of reasoning quality, producing inaccurate, incomplete, or inconsistent traces. We propose verifiable process supervision (VPS), a post-training framework that jointly optimizes prediction accuracy and reasoning quality by supervising structured intermediate claims. We first apply supervised fine-tuning to induce a structured reasoning format, enabling deterministic extraction and verification of intermediate claims for process-level rewards. To address the heterogeneous difficulty of reasoning subtasks, we introduce adaptive weighting that prioritizes components with the largest remaining errors, creating an implicit curriculum. We evaluate VPS on chess as a controlled testbed where reasoning steps can be deterministically verified against engine signals. While outcome-only RL improves move accuracy, it sharply degrades reasoning quality, increasing win-rate error by up to 112% and reducing internal consistency by up to 69%. In contrast, VPS preserves accuracy while significantly improving reasoning quality, reducing win-rate error by up to 30% and restoring consistency to near saturation. A reasoning-space analysis further shows that, without a structured prior, outcome-only RL converges to budget-dependent shortcuts rather than sound multi-step reasoning. Beyond chess, we observe the same phenomenon on math reasoning, where outcome-only RL improves accuracy while degrading step-level arithmetic and consistency, whereas VPS maintains both. These results show that VPS enables language models to reason both accurately and reliably in verifiable domains.
Figures & tables
Figure 1: Overview of VPS. (1) A structured reasoning prior is induced via supervised fine-tuning, enabling syntactic extraction of intermediate claims. (2) During RL, these claims are verified against ground-truth signals to produce process-level rewards. (3) The rewards are adaptively weighted based on subtask performance, focusing learning on components with the largest remaining errors and inducing a curriculum over reasoning subskills.
Table 1: Accuracy and reasoning quality comparison. Prediction accuracy and reasoning quality metrics for Qwen3-8B and DeepSeek-R1-Distill-Llama-8B. Reasoning quality is evaluated using a deterministic verifier on held-out positions; lower win-rate MAE and higher values on other metrics indicate better reasoning.
Model
Acc. ↑
Relev. ↑
Compl. ↑
Clarity ↑
Fluency ↑
Overall ↑
Qwen3-8B
GRPO only (best acc.)
0.507
2.501
2.296
2.429
3.102
2.582
VPS (matched acc.)
0.505
2.856
3.239
3.751
3.833
3.420
GRPO only (same step)
0.500
2.484
2.292
2.390
3.275
2.610
R1-Distill-Llama-8B
GRPO only (best acc.)
0.539
2.057
1.862
1.985
3.164
2.267
Table 2: Accuracy-controlled reasoning comparison. Reasoning quality is evaluated using three LLM judges (GPT-4o, Claude Opus 4.6, Gemini 3 Flash) along four dimensions (1–5 scale). We compare the highest-accuracy GRPO-only checkpoint with a VPS checkpoint at matched accuracy, and include the GRPO-only checkpoint at the same training step.
Figure 3: Reward weighting across SFT scales. Adaptive weighting improves the hardest reasoning subtasks, particularly when models start with stronger domain knowledge.
Figure 4: Qualitative reasoning comparison. Outcome-only optimization produces flawed or inconsistent reasoning, while VPS yields concise and factually grounded explanations.
Figure 5: Reasoning space. Outcome-only RL produces variable reasoning patterns without a structured prior.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Training
Epochs
2
Batch size
256
Gradient clipping
1.0
Precision
bf16
Optimization
Learning rate
1e-5
LR scheduler
cosine
Appendix
Table 4: Supervised fine-tuning hyperparameters.
Hyperparameter
Value
Training
Training steps
300
Learning rate
1e-6
PPO epochs per rollout
1
KL coefficient
0.001
Discount factor ( γ )
1.0
Data
Prompts per batch
128
Appendix
Table 5: Reinforcement learning hyperparameters.
Model
Relev. ↑
Compl. ↑
Clarity ↑
Fluency ↑
Overall ↑
Qwen3-8B
GRPO only (best acc.)
2.370
2.053
2.773
3.657
2.713
VPS (matched acc.)
2.658
2.727
4.379
3.903
3.417
GRPO only (same step)
2.384
2.048
2.773
3.911
2.779
R1-Distill-Llama-8B
GRPO only (best acc.)
1.946
1.728
2.302
3.698
2.418
Appendix
Table 6: Reasoning comparison with GPT-4o. Accuracy-controlled evaluation of reasoning quality using GPT-4o as the judge.
Model
Relev. ↑
Compl. ↑
Clarity ↑
Fluency ↑
Overall ↑
Qwen3-8B
GRPO only (best acc.)
2.669
2.230
2.124
2.267
2.323
VPS (matched acc.)
3.257
3.381
3.581
3.146
3.341
GRPO only (same step)
2.618
2.229
2.057
2.325
2.307
R1-Distill-Llama-8B
GRPO only (best acc.)
2.300
1.870
1.813
2.277
2.065
Appendix
Table 7: Reasoning comparison with Claude Opus 4.6. Accuracy-controlled evaluation of reasoning quality using Claude Opus 4.6 as the judge.
Model
Relev. ↑
Compl. ↑
Clarity ↑
Fluency ↑
Overall ↑
Qwen3-8B
GRPO only (best acc.)
2.464
2.604
2.390
3.382
2.710
VPS (matched acc.)
2.652
3.610
3.293
4.449
3.501
GRPO only (same step)
2.451
2.600
2.339
3.588
2.745
R1-Distill-Llama-8B
GRPO only (best acc.)
1.925
1.990
1.840
3.517
2.318
Appendix
Table 8: Reasoning comparison with Gemini 3 Flash. Accuracy-controlled evaluation of reasoning quality using Gemini 3 Flash as the judge.
Model
Relev. ↑
Compl. ↑
Clarity ↑
Fluency ↑
Overall ↑
GPT-4o judge
LLM reward
2.129
1.860
2.186
2.816
2.248
VPS
2.658
2.727
4.379
3.903
3.417
Gemini 3 Flash judge
LLM reward
2.461
2.453
2.104
3.579
2.649
VPS
2.652
3.610
3.293
4.449
3.501
Appendix
Table 9: Comparison with LLM-based process supervision. Reasoning quality of Qwen3-8B trained using GPT-4o-mini as the process reward model compared with VPS, evaluated using two independent LLM judges.
Model
Reasoning Quality
Capture ↑
Check ↑
Move Div. ↑
Form. Err. ↓
Qwen3-8B
SFT only
0.399
0.651
2.193
0.003
SFT + GRPO
0.215
0.936
1.647
0.003
VPS
0.615
0.988
3.098
0.002
R1-Distill-Llama-8B
Appendix
Table 10: Accuracy and reasoning quality comparison. Additional reasoning quality metrics for Qwen3-8B and DeepSeek-R1-Distill-Llama-8B.
Figure 6: Adaptive subtask weighting dynamics. ( Left ) Running EMA-based estimates of per-subtask model performance. ( Center ) Corresponding weights allocated to each subtask that are dynamically adjusted. ( Right ) Subtask rewards on held-out positions.
Model
Prediction Accuracy
Reasoning Quality
Top-1 Acc. ↑
Elo ↑
Coverage ↑
WR MAE ↓
PV Overlap ↑
Consist. ↑
Qwen3-8B
GRPO only
0.507
1550
–
–
–
–
SFT only
0.212
1160
0.207
0.310
0.323
0.961
SFT + GRPO
0.534
1670
0.302
0.452
0.183
0.721
VPS
0.536
1706
0.441
0.218
0.376
0.978
Appendix
Table 11: Model scale comparison. Prediction accuracy and reasoning quality metrics for Qwen3-8B, Qwen3-4B, and Qwen3-1.7B.
Model
Reasoning Quality
Capture ↑
Check ↑
Move Div. ↑
Form. Err. ↓
Qwen3-8B
SFT only
0.399
0.651
2.193
0.003
SFT + GRPO
0.215
0.936
1.647
0.003
VPS
0.615
0.988
3.098
0.002
Qwen3-4B
Appendix
Table 12: Model scale comparison. Additional reasoning quality metrics for Qwen3-8B, Qwen3-4B, and Qwen3-1.7B.
Figure 7: Accuracy vs. reasoning length under different budgets. Short traces achieve high accuracy, while longer traces correlate with substantially lower accuracy. Distributions show that larger budgets induce longer, more dispersed reasoning traces.
Table 13: Faithfulness and robustness of learned reasoning. ( Left ) Sensitivity of model predictions to perturbations of intermediate reasoning. ( Right ) Prediction accuracy under alternative prompting schemes. Results are averaged over three random seeds.
Figure 8: Qualitative reasoning comparison. GRPO exhibits repetition and non-termination, while SFT + GRPO produces superficial, overconfident reasoning. VPS yields grounded, calibrated analysis with correct move selection.
Figure 9: Qualitative reasoning comparison. Baselines fixate on Qxf6+ and produce superficial or templated analysis, while VPS correctly prioritizes Rg8+ based on its forcing continuation, yielding more grounded reasoning.
Figure 10: Qualitative reasoning comparison. Baseline models perform minimal analysis or rely on shortcut heuristics, repeating checking moves without position-specific evaluation.
Method
Accuracy ↑
Step Arith. ↑
Ans. Cons. ↑
SFT baseline
0.446
0.638
0.821
Step arith. only
0.753
0.906
0.817
Ans. cons. only
0.752
0.863
0.865
VPS
0.763
0.905
0.889
Appendix
Table 14: Contribution of individual verifier signals on GSM8K. Each variant is trained with a single verifier signal or the full objective. Results are averaged over three seeds.
Method
Accuracy ↑
Step Arith. ↑
Ans. Cons. ↑
SFT baseline
0.528
0.669
0.851
SFT + GRPO
0.721
0.618
0.731
VPS
0.722
0.974
0.958
Appendix
Table 15: Generalization to a non-reasoning language model. Results on GSM8K using Qwen3-0.6B-Base, the non-reasoning counterpart of Qwen3-0.6B. The qualitative findings mirror those observed for reasoning models.
Reinforcement Learning with Verifiable Rewards (RLVR) has advanced the reasoning capabilities of Large Language Models (LLMs) by leveraging direct outcome verification instead of learned reward models. Building on this paradigm, Group Relative Policy Optimization (GRPO) eliminates the need for critic models but suffers from indiscriminate credit assignment for intermediate steps, which limits its ability to identify effective reasoning strategies and incurs overthinking. In this work, we introduce a model-free and verifiable process supervision via probing the model's belief in the correct answer throughout its reasoning trajectory. By segmenting the generation into discrete steps and tracking the conditional probability of the correct answer appended at each segment boundary, we efficiently compute interpretable segment-wise progress measurements to refine GRPO's trajectory-level feedback. This approach enables more targeted and sample-efficient policy updates, while avoiding the need for intermediate supervision derived from costly Monte Carlo rollouts or auxiliary models. Experiments on mathematical and general-domain benchmarks show consistent gains over GRPO across diverse models: up to 2.6-point accuracy improvements and 13.7% reasoning-length reductions on math tasks, and up to 2.4 points and 4% on general-domain tasks, demonstrating strong generalization.
Jingyi Wang, Lei Zhu, Tengjin Weng +8
Tsinghua University · Huawei Noah’s Ark Lab · Shenzhen University
Large language models (LLMs) have recently demonstrated impressive performance on complex, multi-step reasoning tasks, especially when post-trained with outcome-rewarded reinforcement learning Guo et al. 2025. However, it has been observed that outcome rewards often overlook flawed intermediate steps, leading to unreliable reasoning steps even when final answers are correct. To address this unreliable reasoning, we propose PRoSFI (Process Reward over Structured Formal Intermediates), a novel reward method that enhances reasoning reliability without compromising accuracy. Instead of generating formal proofs directly, which is rarely accomplishable for a modest-sized (7B) model, the model outputs structured intermediate steps aligned with its natural language reasoning. Each step is then verified by a formal prover. Only fully validated reasoning chains receive high rewards. The integration of formal verification guides the model towards generating step-by-step machine-checkable proofs, thereby yielding more credible final answers. PRoSFI offers a simple and effective approach to training trustworthy reasoning models.
Luoxin Chen, Yichi Zhou, Huishuai Zhang
Wangxuan Institue of Computer Technology, Peking University. · ByteDance.
A standard recipe for distilling the reasoning ability of large language models (LLMs) is to sample chains of thought from the model, keep those that reach the correct final answer, and fine-tune on the survivors. When sampling fails, a common fix shows the generator the gold answer and asks it to write a chain that reaches that answer. We show that this second step degrades the training data in a way that correctness filtering cannot catch. We run a controlled experiment that fixes the generator, the problem set, and the correctness filter, and varies only whether the chain is generated under answer-conditioning, the gold answer shown with a request to reach it. Training a strong instruction-tuned reasoning model on its own answer-conditioned chains sharply lowers its verifiable-reasoning accuracy. The loss grows with difficulty, reaching as much as about 27 points on the hardest competition problems. The mechanism is legible in the chains themselves, which rationalize backward from the shown answer instead of deriving it, with the early final-answer statement as the measurable symptom. The harm is a property of the data rather than the generator, read off unlabeled generations before any fine-tuning, ordering the penalty across eight thinking models from four families, and transferring across teacher families. A prompt ablation localizes it to the rationalize-toward instruction rather than the answer's bare visibility. The practical takeaway is to generate answer-blind, because no correctness filter can see this damage in the data.
Jungseob Lee, Seungyoon Lee, Suhyune Son +4
1Korea University · 2Zoom Communications · 3Yonsei University