Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Sequence-Level Likelihood

Xingyu Lin; Yilin Wen; Du Su; Jinchang Hou; En Wang; Wenbin Liu; Chenfu Bao; Zhonghou Lv

cs.CLApr 14, 2026

Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Sequence-Level Likelihood

Authors: Xingyu Lin, Yilin Wen, Du Su, Jinchang Hou, En Wang, Wenbin Liu, Chenfu Bao, Zhonghou Lv

Organizations: College of Computer Science and Technology, Jilin University · Key Laboratory of Symbolic Computation and Knowledge Engineering of MOE, Jilin University · Baidu Inc. · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · Tsinghua University.

Abstract

Group Relative Policy Optimization (GRPO) has significantly advanced the reasoning ability of large language models (LLMs), particularly in their mathemat ical reasoning performance. However, GRPO and related entropy regularization methods still struggle with token-level sparse-rewards, which is an inherent chal lenge in chain-of-thought (CoT) reasoning. These approaches often rely on undifferen tiated token-level entropy regularization, which easily leads to entropy collapse or model degradation under sparse token rewards. In this work, we propose TEPO, a novel token-level framework that (1) leverages sequence-level likelihood to link group-level rewards with individual tokens via token-level aggregation, and (2) introduces a token-level KL-Divergence mask constraint that targets tokens with positive advantages and decreasing entropy to mitigate abrupt policy updates. Experiments demonstrate that TEPO not only achieves state-of-the-art performance on mathematical reasoning benchmarks but also markedly enhances training stability, reducing convergence time by 50% compared with GRPO/DAPO.

Figures & tables

Figure 1: Overview of the TEPO Framework TEPO (1) replaces baselines’ noisy, sparse token-level credit assignment with sequence-level likelihood, using soft aggregation to broadcast group rewards to tokens and stabilize training. (2) A selective KL mask curbs abrupt updates exclusively for tokens with positive advantage and decreasing entropy, balancing entropy reduction and stability.

	$\displaystyle J(\theta)={\color[rgb]{0,0,1}\frac{1}{\sum_{i=1}^{G}\\|o_{i}\\|}\sum_{i=1}^{G}\sum_{t=1}^{\\|o_{i}\\|}}\Bigg[$	$\displaystyle\min\bigg({\color[rgb]{0,1,0}w_{i}(\theta)},\operatorname{clip}\big({\color[rgb]{0,1,0}w_{i}(\theta)},1-\epsilon,1+\epsilon\big)\bigg)\cdot A_{i,t}-\beta\cdot\text{KL}\big(\pi_{\theta}\parallel\pi_{\theta_{\text{old}}}\big)\cdot{\color[rgb]{1,0,0}M}\Bigg],$		(4)
	$\displaystyle M$	$\displaystyle=\begin{cases}1,&\text{if }A_{i,t}>0\land\Delta\mathcal{H}_{i,t}<0,\\ 0,&\text{otherwise}.\end{cases}$		(4)

Table 2

Panel of Figure 2 — Figure 2: Lower Gradient Bias and Faster Reasoning Efficiency with Markov Likelihood: The left panel shows that our method with a lower clip ratio effectively mitigates gradient bias, while the right panel indicates that it reduces the generation of redundant reasoning steps. Specifically, the average reasoning time of TEPO is 338 seconds per step , which is lower than that of DAPO/GRPO (357).

Panel of Figure 3 — Figure 3: Markov Likelihood enhanced performance (We transform the raw data into percentage-based values.): The left panel shows that our method achieves steadier and higher rewards across training steps, demonstrating more efficient learning dynamics. The right panel indicates that our method exhibits consistently higher gradient norms, reflecting more active and effective parameter reasoning.

Method	AIME24	AIME25	AMC	MATH-500	OMNI-MATH	OlympiadBench	Minerva	Avg.
Qwen2.5-7B	$0.94$	$0.94$	$14.34$	$43.20$	$13.73$	$16.74$	$21.69$	$13.30$
w. GRPO/DAPO	$11.25$	$5.00$	$42.80$	$76.60$	$25.00$	$37.92$	$33.08$	$30.85$
w. ClLIP-Cov	$10.83$	$7.40$	$42.84$	$75.60$	$26.11$	$39.40$	37.86	$31.64$
w. KL-Cov	$12.29$	$8.23$	$42.77$	$75.60$	$25.64$	$39.11$	$34.92$	$31.60$
w. Entropy-based Term	$11.35$	$6.15$	$43.18$	$74.80$	$26.14$	$40.14$	$36.02$	$31.62$
w. GPG	13.54	$6.04$	$43.29$	$75.00$	$26.28$	40.44	$34.55$	$31.91$

Table 1: Performance Comparison of 7B and 14B Models on Mathematical Reasoning Benchmarks. Notations: 1-(w/o. GRPO/DAPO) indicates models not applying GRPO/DAPO, while (w. GRPO/DAPO) indicates models applying GRPO/DAPO; 2-higher values indicate better performance; 3-best results are marked in bold ; 4-

\Delta

denotes the performance difference between models without and with our method; .

	AIME24	AIME25	AMC	MATH-500	OMNI-MATH	OlympiadBench	Minerva	Avg.
DeepSeek-R1-Distill-Llama-8B	$12.70$	$13.33$	$46.49$	$70.40$	$25.50$	$36.88$	$23.52$	$32.46$
w. GRPO/DAPO	27.50	$18.43$	61.89	$78.40$	$32.97$	$41.62$	$26.10$	$42.23$
TEPO	$25.72$	18.85	$60.84$	79.60	33.11	43.55	29.77	42.76
Mistral-7B-Instruct-v0.2	-	-	$2.1830$	$9.800$	$3.9840$	$1.1850$	$5.8820$	$2.7700$
w.GRPO/DAPO	-	-	$3.1250$	$10.8000$	$4.2680$	2.3700	8.0880	$3.3690$
TEPO	-	-	3.5760	12.8000	4.5890	$1.9250$	8.0880	3.6520

Table 2: Performance Comparison of Various Models on Mathematical Reasoning Benchmarks

Method	AIME24	AIME25	AMC	MATH-500	OMNI-MATH	OlympiadBench	Minerva	Avg.
TEPO -w/o. KL	$12.7000$	$6.3540$	43.5600	$75.4000$	$27.0000$	$39.5500$	37.5000	$32.21$
Panel A: TEPO w. Undifferentiated KL-Divergence
w. $\beta=1$ (model collapse in 24 steps)	$0$	$0$	$0$	$0$	$0$	$0$	$0$	$0$
w. $\beta=0.1$ (leads to Avg drop of 4.6%.)	$5.5200$	$3.9580$	$42.8400$	$75.6000$	$22.4100$	$32.7400$	$30.8800$	$26.2500$
w. $\beta=0.01$	$11.45$	$7.395$	$42.2400$	$76.2000$	$26.3200$	$40.5900$	$33.8200$	$31.6400$
w. $\beta=0.001$	$12.29$	$8.958$	$42.2400$	$77.2000$	$25.8900$	$40.1100$	$36.3900$	$31.8100$

Table 3: Ablation Studies of Key Components. All panels are compared with TEPO -w/o. KL: Panel A demonstrates the fragility of undifferentiated KL-Divergence and its performance degradation effect; Panel B shows that undifferentiated entropy regularization yields marginal improvements to TEPO -w/o-KL; Panel C validates TEPO ’s effectiveness across different importance sampling strategies and corroborates that GRPO exhibits sparse token-level rewards; Panel D confirms the superiority of token-mean aggregation over alternative strategies; Panel E verifies that TEPO achieves optimal control of token-level KL-Divergence.

Method	AIME24	AIME25	AMC	MATH-500	OMNI-MATH	OlympiadBench	Minerva	Avg.
GRPO/DAPO (72 steps, seed1)	$11.56$	$6.25$	$40.47$	$72.40$	$26.00$	$37.77$	$36.76$	$30.30$
GRPO/DAPO (72 steps, seed2)	$11.25$	$5.00$	$42.80$	$76.60$	$25.00$	$37.92$	$33.08$	$30.85$
GRPO/DAPO (132 steps, seed1)	$12.08$	$7.08$	$42.65$	$76.20$	$26.07$	$39.85$	$37.86$	$31.73$
GRPO/DAPO (132 steps, seed2)	$12.08$	$6.98$	$41.67$	$74.20$	$27.42$	$39.85$	$36.76$	$31.69$
TEPO (72 steps, seed1)	$12.50$	$7.60$	$45.18$	$77.80$	$26.78$	$41.03$	$38.97$	$32.62$
TEPO (72 steps, seed2)	$12.60$	$8.75$	$43.48$	$77.40$	$27.17$	$40.44$	$34.92$	$32.59$

Table 4: Ablation Studies of Random Seeds. We report performance on seven mathematical reasoning benchmarks using two different random seeds for each method. “Mean” denotes the average performance over two seeds. TEPO consistently outperforms baseline methods across all settings.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Notation	Definition
$\{(x_{i},y_{i}/o_{i})\}_{i=1}^{G}$	Batch of prompt-response pairs, $x$ denotes the prompt, and $y/o_{i}$ denotes the response generated by LLMs
$y_{i,j\leq t}$	Prefix of the $i$ -th sentence up to the $t$ -th token
$G$	Number of responses generated by LLMs per prompt
$\\|{o_{i}}\\|$	Length of $y_{i}/o_{i}$
$\theta(a\|s)$	Score assigned by large language models (LLMs)
$\tau$	A complete solution trajectory for a single math problem.

Appendix

Table 5: Core Notations for Entropy Derivation

Explore similar work

ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models
Mar 30, 2026Song Yu, Li Li, Wenwen Zhao +1RL for Language Model Reasoning RL for Language Models
School of Computer and Information Science, Southwest University, Chongqing, 400715, China
Beyond Token-Level Policy Gradients for Complex Reasoning with Large Language Models
Feb 16, 2026Mufan Xu, Kehai Chen, Xuefeng Bai +4RL for Language Model Reasoning Policy Optimization
School of Computer Science and Technology, Harbin Institute of Technology, China · Baidu Inc., Beijing, China
EP-GRPO: Entropy-Progress Aligned Group Relative Policy Optimization with Implicit Process Guidance
May 6, 2026Song Yu, Li Li, Wenwen Zhao +1RL for Language Model Reasoning Group Relative Policy Optimization
School of Computer and Information Science, Southwest University, Chongqing 400715, China

Mar 30, 2026J/K move · Enter open · S save

ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models

Song Yu, Li Li, Wenwen Zhao, Zhisheng Yang

School of Computer and Information Science, Southwest University, Chongqing, 400715, China

RL for Language Model Reasoning RL for Language Models LLM Reasoning Large Reasoning Models

Open paper AskPDF arXiv

Reinforcement learning from verifiable rewards has significantly advanced the reasoning capabilities of large language models. However, Group Relative Policy Optimization (GRPO) typically assigns a uniform, sequence-level advantage to all tokens, thereby overlooking the intrinsic information heterogeneity along reasoning chains. We show that this coarse-grained credit assignment leads to premature entropy collapse and encourages the model to generate redundant, low-quality reasoning paths. Through systematic empirical analysis, we identify Critical Decision Pivots (CDPs): transient high-entropy states where the policy's trajectory is most sensitive to perturbations. These pivots represent the "forks in the road" where effective multi-path exploration is most crucial yet often suppressed by uniform advantage signals. Building on these insights, we propose Entropy-Regulated Policy Optimization (ERPO), which transitions the optimization focus from coarse sequences to fine-grained token dynamics. ERPO introduces three synergistic components: (i) Entropy-aware Gating, which adaptively amplifies exploration at CDPs to facilitate diverse path discovery; (ii) Bucket-based Implicit Normalization, which mitigates difficulty bias by aligning token progress windows; and (iii) Result-anchored Advantage Synthesis, which re-weights token-level signals via outcome-driven anchors. Extensive experiments on competitive mathematical benchmarks demonstrate that ERPO significantly outperforms GRPO. Notably, ERPO not only boosts reasoning accuracy but also yields significantly more concise and robust derivation paths, while achieving performance comparable to large models with orders of magnitude more parameters.

Abstract

Explore similar work

ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models

Beyond Token-Level Policy Gradients for Complex Reasoning with Large Language Models

EP-GRPO: Entropy-Progress Aligned Group Relative Policy Optimization with Implicit Process Guidance