TPPO: Tracking-Value PPO For Language Model Reasoning

Futing Wang; Chenxi Li; Jianhao Yan; Yun Luo; Zhi Wang; Yizhuo Li; Runzhe Zhan; Yu Cheng

cs.LGOct 12, 2026

TPPO: Tracking-Value PPO For Language Model Reasoning

Authors: Futing Wang, Chenxi Li, Jianhao Yan, Yun Luo, Zhi Wang, Yizhuo Li, Runzhe Zhan, Yu Cheng,

Organizations: Zhejiang University · Westlake University · Shanghai AI Laboratory · The Hong Kong University of Science and Technology ·

Abstract

Proximal Policy Optimization (PPO) is widely used in reinforcement learning with verifiable rewards for language-model reasoning. While value evolution along trajectory characterizes relative token-level credit, yet standard critics often fail to track it under sparse terminal rewards. To model this evolution, we analyze the relation between successive prefix values and show that they admit a Bayesian decomposition with additive updates in log-odds space. We use this additive structure to construct Tracking-Value PPO (\method), a cumulative value parameterization that sums position-wise scalar outputs into value logits. These outputs are learned through terminal-outcome supervision on the reconstructed values. \method improves aggregate mathematical reasoning scores over PPO by 7.0 and 8.4 points on Qwen3-4B-Base and Qwen3-8B-Base, respectively, while improving out-of-distribution generalization. Gains extend to Qwen3-4B, Qwen2.5-Math-1.5B, Qwen3.5-9B and to countdown, and scientific coding tasks. Further analysis shows more accurate tracking of value evolution and more reliable advantage estimates. Our code is publicly available in https://github.com/LINs-lab/TPPO.

Figures & tables

Figure 1: Critics mispredict how value evolves along reasoning. (a) Representative responses showing that different steps within a response can cause different MC-estimated value changes; the PPO critic predicts changes with opposite signs in these examples. (b) The sign accuracy of these predictions decreases during training. MC evaluation details are in Appendix D.3 .

Figure 2: Difference between standard and TPPO critic parameterizations. (a) A standard critic directly predicts a value for each prefix. (b) TPPO accumulates token-wise critic outputs into value logits, then applies a sigmoid to obtain prefix values.

Critic + loss	Sign accuracy ( $\tau=0$ )	Sign accuracy ( $\tau=1$ )
Standard critic + MSE	50.51 [49.15, 51.79]	51.19 [49.57, 52.69]
Standard critic + BCE	50.70 [49.29, 52.13]	51.14 [49.54, 52.74]
TPPO critic + BCE	52.71 [51.67, 53.77]	53.94 [52.59, 55.33]

Table 1: Directional accuracy of local value evolution under a fixed actor. Entries are accuracies (%) with 95% confidence intervals in brackets, computed on intervals satisfying

\mathrm{SNR}>\tau

. The metric compares the signs of critic-predicted and MC-estimated local value evolution between evaluation anchors (Appendix D.3 ).

Method	In-Domain Performance						Out-of-Domain Performance
	AIME24/25/26	AMC	MATH500	Minerva	Olympiad	Avg.	ARC-C	AGI-Eval	BBH	OOD Avg.
Qwen3-4B-Base
Base	4.3/3.3/3.4	27.3	43.2	22.1	22.3	18.0	34.8	24.8	12.8	24.1
GRPO	12.1/11.4/8.1	51.9	76.9	47.6	40.1	35.4	90.2	63.7	42.9	65.6
PPO	13.5 /9.5/ 9.3	53.8	76.1	41.2	39.1	34.6	88.7	59.0	40.7	62.8
VAPO	8.3/8.3/6.7	48.8	72.2	44.9	34.9	32.0	88.7	58.1	41.2	62.7

Table 2: In-domain and out-of-domain results with

n=8

. All entries are accuracies (%). Orange parenthetical values show the absolute improvement over PPO in percentage points. Bold and underlined values indicate the best and second-best results within each model group, respectively.

Figure 3: Training dynamics comparing PPO and TPPO on Qwen3-8B-Base (

n=8

). We track actor gradient norm, policy entropy, validation accuracy, and Pass@4 over training. TPPO establishes an early lead over PPO in both performance metrics and retains it through the final checkpoint.

Method	In-Domain Performance						Out-of-Domain Performance
	AIME24/25/26	AMC	MATH500	Minerva	Olympiad	Avg.	ARC-C	AGI-Eval	BBH	OOD Avg.
Qwen3-8B-Base
Base	8.2/6.8/6.4	33.8	54.3	24.9	27.2	23.1	62.4	44.2	27.8	44.8
PPO	11.7/ 15.0 / 15.0	56.9	77.9	45.9	39.2	37.4	91.4	67.1	42.0	66.8
SPPO	15.8 /10.8/ 15.0	56.3	78.2	43.4	41.3	37.3	91.8	66.0	40.4	66.1
TPPO	23.3 / 20.0 / 20.8	68.1	84.9	46.3	50.9	44.9 (+7.5)	93.0	71.8	46.8	70.5

Table 3: In-domain and out-of-domain results with

n=1

Figure 4: Countdown task on Qwen2.5-Math-1.5B. Trained on problems with three or four input numbers.

Benchmark	Base	PPO	TPPO
SciCode	29.3	32.6	35.0 (+2.4)
HumanEval	94.7	93.5	96.5 (+3.0)
HumanEvalFix	58.3	70.1	75.5 (+5.4)
LCB Code Gen.v6	61.9	57.6	73.1 (+15.5)
APPS Introductory	69.1	72.3	80.8 (+8.5)

Table 4: Coding results on Qwen3.5-9B. Accuracy (%) averaged over three evaluation runs. Orange parenthetical values show the improvement over PPO in percentage points. HumanEvalFix averages Python, Rust, and JavaScript.

Figure 7: Value estimates on test data. Error and accuracy against MC estimates at different reasoning positions.

Figure 8: Critic ablations and actor GAE coefficients. Validation accuracy for (a) critic parameterization and loss variants, and (b) PPO and TPPO with

\lambda_{\mathrm{actor}}\in\{1,0.95,0\}

. Settings for (b) are given in Appendix E.2 .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Math	Scientific coding
Maximum response length	8,192	20,000
Temperature	1.0	1.0
Top- $p$	0.95	1.0
Batch size	64 prompts (Base) / 128 prompts (4B)	32 main problems $\times$ 8 trajectories
Minibatch size	32 prompts (Base) / 64 prompts (4B)	256 responses
Actor learning rate	$10^{-6}$	$10^{-6}$

Appendix

Table 5: Training, rollout, and optimization settings. Shared settings apply to both domains unless a domain-specific value is shown.

Figure 9: Additional results on Llama3.2-3B-Instruct. Overall scores for the base model, PPO, GRPO, and TPPO (%).

Figure 10: Value-change sign accuracy by rollout source. Rows distinguish PPO-actor and TPPO-actor rollouts; columns show the SNR thresholds

\tau=0

and

\tau=1

	$\tau=0$		$\tau=1$
Actor update	PPO	TPPO	PPO	TPPO
100	2,765 (67.5%)	2,843 (69.4%)	1,655 (40.4%)	1,972 (48.1%)
200	2,403 (58.7%)	2,968 (72.5%)	1,586 (38.7%)	1,930 (47.1%)
300	2,186 (53.4%)	2,793 (68.2%)	1,468 (35.8%)	1,860 (45.4%)
400	1,944 (47.5%)	2,755 (67.3%)	1,323 (32.3%)	1,852 (45.2%)
500	1,991 (48.6%)	2,767 (67.6%)	1,339 (32.7%)	1,830 (44.7%)

Appendix

Table 6: Intervals retained by SNR filtering. Counts per method and checkpoint, with percentages of the 4,096 candidate intervals in parentheses.

	Advantage		Value-change sign accuracy
Actor update	Cosine	Sign accuracy	$\tau=0$	$\tau=1$
100	$-0.0228$	$+1.76$	$-0.12$	$+0.06$
200	$+0.1189$	$+15.34$	$+3.90$	$+4.94$
300	$+0.0574$	$+16.60$	$+6.35$	$+7.40$
400	$+0.1089$	$+19.92$	$+4.62$	$+5.29$
500	$+0.0485$	$+19.44$	$+6.76$	$+6.23$

Appendix

Table 7: Diagnostic gains of TPPO over PPO. Sign-accuracy gains are in percentage points; cosine gains are absolute differences. Rows correspond to actor update steps.

	Initial logit		Increment
Head	Mean	SD	Mean	SD
Shared	$-0.0273$	$0.0052$	$-0.00054$	$0.00210$
Separate	$-0.8657$	$0.3016$	$-0.00132$	$0.00257$

Appendix

Table 8: Output statistics for shared and separate heads. The separate head has substantially larger initial-logit magnitude and variance, while increment statistics remain comparable.

Figure 11: Shared and separate output heads. Validation accuracy, response length, and actor gradient norm for PPO, the shared-head TPPO critic, and the two separate-head variants shown in the legend.

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 430 papers

ReDiPPO: Reference-Guided Value Calibration and Discrepancy-Aware Token Reweighting for Mathematical Reasoning
Jul 30, 2026Zhenrong Zhang, Fei Wu, Jun Du +2RL for Language Model Reasoning Credit Assignment in RL
1IFLYTEK Research · University of Science and Technology of China
SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks
Apr 10, 2026Tianyi Wang, Yixia Li, Long Li +6RL for Language Model Reasoning Proximal Policy Optimization
Southern University of Science and Technology · Beijing University of Posts and Telecommunications · INFLY TECH +3
OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoning
May 21, 2026Yu Li, Rui Miao, Tian Lan +1Value Function Estimation Token-Level Credit Assignment
1George Washington University · 2The University of Texas at Dallas

Jul 30, 2026J/K move · Enter open · S save

ReDiPPO: Reference-Guided Value Calibration and Discrepancy-Aware Token Reweighting for Mathematical Reasoning

Zhenrong Zhang, Fei Wu, Jun Du, Jianshu Zhang+1

1IFLYTEK Research · University of Science and Technology of China

RL for Language Model Reasoning Credit Assignment in RL Proximal Policy Optimization Sparse-Reward RL LLM Mathematical Reasoning

Open paper AskPDF arXiv

Reinforcement learning has emerged as an effective paradigm for enhancing the mathematical reasoning capabilities of large language models. Among existing policy optimization methods, Proximal Policy Optimization (PPO) remains particularly appealing because its learned critic can, in principle, provide token-level credit assignment. However, in mathematical reasoning tasks characterized by long reasoning horizons and sparse outcome rewards, reliable token-level credit assignment remains challenging. The standard critic often fails to accurately evaluate intermediate reasoning states, resulting in noisy advantage estimates and suboptimal policy updates. In this paper, we propose ReDiPPO, a Reference-guided and Discrepancy-aware PPO framework for mathematical reasoning. ReDiPPO introduces a reference-guided critic that uses reference answers as training-time privileged signals to provide more accurate value estimation. Meanwhile, it retains a standard critic and quantifies the token-level reference-standard discrepancy between the standard value estimate and the reference-guided value estimate. This discrepancy serves as an indicator of difficult reasoning states and is used to reweight the corresponding token-level advantages during PPO optimization. Extensive experiments on diverse mathematical reasoning benchmarks demonstrate that ReDiPPO improves value-estimation accuracy and consistently outperforms strong policy optimization baselines, including PPO, DAPO, and GSPO, in final reasoning performance. Our code is available on https://github.com/cii030/ReDiPPO.

Abstract

RL for Language Models

Explore similar work

ReDiPPO: Reference-Guided Value Calibration and Discrepancy-Aware Token Reweighting for Mathematical Reasoning

SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks

OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoning