STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens

Shiqi Liu; Zeyu He; Guojian Zhan; Letian Tao; Zhilong Zheng; Jiang Wu; Yinuo Wang; Yang Guan

cs.CLFeb 17, 2026

STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens

Authors: Shiqi Liu, Zeyu He, Guojian Zhan, Letian Tao, Zhilong Zheng, Jiang Wu, Yinuo Wang, Yang Guan, +5 more

Organizations: School of Vehicle and Mobility & College of AI, Tsinghua University · Didi Voyager Labs, DiDi Autonomous Driving

Abstract

Reinforcement Learning (RL) has significantly improved large language model reasoning, but existing RL fine-tuning methods rely heavily on heuristic techniques such as entropy regularization and reweighting to maintain stability. In practice, they often suffer from late-stage performance collapse, leading to degraded reasoning quality and unstable training. We identify a key factor behind this instability: a small fraction of tokens, termed spurious tokens (around 0.01%), which contribute little to the reasoning outcome but receive disproportionately amplified gradient updates due to inheriting the full sequence-level reward. We present a unified framework for evaluating token-level optimization impacts across spurious risk, gradient norms, and entropy changes. Building on the analysis of token characteristics that severely disrupt optimization, we propose the Silencing Spurious Tokens (S2T) mechanism to efficiently suppress their gradient perturbations. Incorporating this mechanism into a group-based objective, we propose Spurious-Token-Aware Policy Optimization (STAPO), which promotes stable and effective large-scale model refinement. Across six mathematical reasoning benchmarks using Qwen 1.7B, 8B, and 14B base models, STAPO consistently demonstrates superior entropy stability and achieves an average performance improvement of 11.49% ( $ρ_{\mathrm{T}}$ =1.0, top-p=1.0) and 3.73% ( $ρ_{\mathrm{T}}$ =0.7, top-p=0.9) over GRPO, 20-Entropy, and JustRL.

Figures & tables

Figure 1: Core Idea. (a) In RLVR training, we argue that spurious tokens, which are rare and uninformative tokens within otherwise correct responses, can harm training stability, analogous to a dissonant vocalist disrupting the harmony of a performance. (b) By masking this negligible fraction (near $0.01\%$ ) of spurious tokens during the RL process of Qwen3-8B-Base, STAPO approaches the Pareto frontier of AIME24 Acc and entropy stability, compared to GRPO, 20-Entropy, and JustRL.

Panel of Figure 2 — Figure 2: Comprehensive Analysis of Spurious Tokens.

Figure 3: Training curves of entropy and AIME24 accuracy. JustRL-FullMask denotes a variant of JustRL that masks all low-probability tokens within correct responses.

Token Properties			Influence on Optimization
Advantage	Token Prob.	Entropy	Spurious Risk	Gradient Norm	Entropy Change [ 17 ]
$\hat{A}>0$	$\pi_{t}\geq\tau_{p}$	$\mathcal{H}\geq\tau_{h}$	Low	Low	$\downarrow$
	$\pi_{t}\geq\tau_{p}$	$\mathcal{H}<\tau_{h}$	Low	Medium	$\downarrow$
	$\pi_{t}<\tau_{p}$	$\mathcal{H}\geq\tau_{h}$	Medium	Medium	$\uparrow$
	$\pi_{t}<\tau_{p}$	$\mathcal{H}<\tau_{h}$	High	High	$\bm{\uparrow}$

Table 1: Taxonomy of Token-Level Optimization Mechanisms. Left: Token properties. Right: Influence on optimization. Green indicates beneficial updates, yellow indicates intermediate updates, and red indicates detrimental updates.

\tau_{p}

and

\tau_{h}

represent the thresholds for token probability and entropy, respectively.

Figure 4: Illustration of the Entropy Threshold.

Baseline	AIME24	AIME25	AMC23	MATH500	Minerva	Olympiad	Avg
RL from the Qwen3-1.7B Base Model
GRPO	7.60 (10.42)	3.33 (3.33)	37.81 (39.84)	65.90 (66.40)	33.73 ( 34.83)	31.16 (31.42)	29.92 (31.04)
20-Entropy	11.98 (17.40)	6.67 (11.67)	42.66 (47.58)	63.70 (64.90)	21.23 (24.45)	31.34 (32.31)	29.60 (33.05)
JustRL	9.17 (15.73)	6.67 (9.28)	43.20 (48.28)	66.25 (70.10)	30.61 (32.08)	30.71 (32.90)	31.10 (34.73)
STAPO	18.44 ( 19.27)	10.00 ( 13.33)	54.22 ( 53.12)	73.10 ( 71.80)	28.40 (26.10)	36.94 ( 36.80)	36.85 ( 36.74)
RL from the Qwen3-8B Base Model

Table 2: Main Results on Six Benchmarks across Three Models. Each cell reports performance under the training-aligned configuration (temperature

\rho_{\mathrm{T}}

=1.0, top-p=1.0) and the JustRL [ 16 ] evaluation setting (temperature

\rho_{\mathrm{T}}

=0.7, top-p=0.9, shown in gray). For each configuration, the best result is highlighted in bold (training-aligned setting) and in dark gray (JustRL setting), respectively.

Figure 5: Training Curves on the Qwen3-1.7B Base Model. Notably, STAPO achieves superior performance while maintaining stable policy entropy. Training dynamics for larger models (8B and 14B) exhibit consistent trends and are provided in Appendix 11.1 .

Figure 6: Spurious Token Ratio Curves. After warm-up, the ratio of spurious tokens stays below 0.01% across all models.

Hyperparameter	Value
Advantage Estimator	GRPO
Reward Function	DAPO
Probability Threshold $\tau_{p}$	0.002
Entropy Percentile $q$	75
Use KL Loss	No
Use Entropy Regularization	No

Table 3: Full Training Hyperparameters

Panel of Figure 10 — Figure 10: Word clouds. The font size of each token is proportional to its occurrence frequency. To facilitate a clearer analysis, common articles and prepositions are filtered out.

Case 1

Context: The length of

A

1981

, and the length of

B

1014

. Since the length of

U

2012

, for the minimum length of

A\cap B

B

should start immediately from the right of

A

. The length of

A\cap B

is the sum of the lengths of

A

and

B

minus the length of

U

, so its minimum value is

1981+1014-2012=983

. Here is a Python calculation to verify the above reasoning…

Spurious Token Prob (

P

) Adv Top-5 Distribution calculation 0.0115% 1.62 code(51.90%) | program(31.48%) | solution(11.58%) | script(3.32%) | function(0.58%)

Case 2

Context: We are given

n

islands with ferry lines between each pair of distinct islands, which is essentially a complete graph

K_{n}

. Each edge of the graph is colored by one of

k

colors, corresponding to the companies operating the ferry lines. For any closure of one company, all its lines must be broken (the corresponding edges ignored) such that the remaining graph can no longer support a Hamiltonian Cycle, meaning it’s no longer possible to visit all the islands exactly once without returning to the starting point. According to the condition, every color, therefore, must serve to connect the graph in such a way that the removal of edges of any single color makes the graph Hamiltonian non-existent. In a complete graph

K_{n}

, we can color the edges in such a way that removing any single color results in a graph that lacks a Hamiltonian Cycle. For this, we can use a result from graph theory, informally known as "edge coloring minimum spanning trees": for

n

nodes, we can divide the set of nodes into

2\lfloor{\frac{n}{2}}\rfloor

color classes (as far as coloring spans, and where

\lfloor{x}\rfloor

means "the greatest integer less than

x

"). So when we have

n=2m

(even) nodes, the maximal number of colors

k

we can use is

m-1

, since we can divide our

n

nodes into

m

colors and deleting any one will still result in a complete graph

K_{2m-1}

which remains Hamiltonian (since

K_{2m-1}

is always Hamiltonian for

m\geq 1

). For

n=2m+1

(odd) nodes, we can divide our

n

nodes into

m

classes, so the discard of any one of these

m

classes will disintegrate the graph such that a Hamiltonian Cycle will become impossible because it contains a vertex of degree 1 or a subdivision thereof.

Spurious Token Prob (

P

Case 3

Table 4: Category I: Uncommon Syntax.

Case 1

Context:

A=\frac{1}{2}\left|\text{x}_{1}(\text{y}_{2}-\text{y}_{3})+\text{x}_{2}(\text{y}_{3}-\text{y}_{{\color[rgb]{1,0,0}\textbf{2}}})+\text{x}_{3}(\text{y}_{1}-\text{y}_{2})\right|

Here,

x

s and

y

s are the coordinates of the points.

Spurious Token Prob (

P

) Adv Top-5 Distribution 2 0.0142% 0.54 1(99.95%) | 2(0.01%) | 3(0.01%) | 0(0.00%) | }_(0.00%)

s 0.0117% 0.54 _(94.78%) | 1(1.05%) | =(0.82%) | _{(0.82%) |

(0.50%)

Case 2

Context:

(a-10,b-10)=(193,11)

→

a=203

b=21{\color[rgb]{1,0,0}\textbf{3}}

Spurious Token Prob (

P

) Adv Top-5 Distribution 3 0.0017% 0.94 \ (99.98%) | \n (0.01%) | 1(0.01%) | \n (0.00%) | 3(0.00%)

Case 3

Table 5: Category II: Hallucinations and Math Errors.

Case 1

Context: To solve this problem, let’s start by using the given information about the function

f(x)=ax^{2}+bx+c

: 1. Since

f(1)=0

, we know that:

a+b+c=0

This means

c=-a-b

. Now , substitute

c=-a-b

into the quadratic function:

f(x)=ax^{2}+bx-(a+b)

Spurious Token Prob (

P

) Adv Top-5 Distribution Now 0.0552% 0.35 2(99.74%) | Next(0.12%) | Now(0.06%) | So(0.03%) | Sub(0.02%)

Case 2

Context: \ boxed{5 \ text{ agony}}

Spurious Token Prob (

P

) Adv Top-5 Distribution \ 0.0015% 0.35 }\n (99.57%) | }(0.25%) | } \n\n (0.12%) | } \ (0.03%) | }. \n (0.01%)

Case 3

Table 6: Category III: Formatting Errors.

Explore similar work

STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability
Jun 17, 2026Haipeng Luo, Qingfeng Sun, Songli Wu +4RL for Language Model Reasoning Maximum Entropy RL
Shenzhen International Graduate School, Tsinghua University · Tencent Hunyuan
Not All Tokens Learn Alike: Attention Entropy Reveals Heterogeneous Signals in RL Reasoning
May 8, 2026Gengyang Li, Zheng-Fan Wu, Siqi Bao +1Token-Level Credit Assignment RL for Language Model Reasoning
1Baidu · School of Software and Microelectronics, Peking University · School of Computer Science, Peking University
Beyond Entropy: Learning from Token-Level Distributional Deviations for LLM Reasoning
Jun 18, 2026Xuanzhi Feng, Zhengyang Li, Zeyu Liu +6Reinforcement Learning RL for Language Model Reasoning
The Hong Kong University of Science and Technology, Hong Kong, China · Sichuan University, Chengdu, China · The Hong Kong Polytechnic University, Hong Kong, China

Jun 17, 2026J/K move · Enter open · S save

STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability

Haipeng Luo, Qingfeng Sun, Songli Wu, Can Xu+3

Shenzhen International Graduate School, Tsinghua University · Tencent Hunyuan

RL for Language Model Reasoning Maximum Entropy RL Reinforcement Learning with Verifiable Rewards Policy Gradient Methods

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards algorithms like GRPO have emerged as the dominant post-training paradigm for complex reasoning in LLMs, yet commonly suffer from policy entropy collapse during training. We conduct a first-order gradient analysis of token-level entropy dynamics under GRPO and identify a token-level credit assignment mismatch: the per-token entropy variation decomposes into the product of the trajectory-level advantage and an entropy sensitivity function over the next-token distribution, yielding an advantage-surprisal four-quadrant structure and a near-criticality property. Motivated by it, we propose STARE (Surprisal-guided Token-level Advantage Reweighting for policy Entropy stability), which identifies entropy-critical token subsets via batch-internal surprisal quantiles, selectively reweights their effective advantages, and incorporates a target-entropy closed-loop gate for stable entropy regulation. Across model scales from 1.5B to 32B and three task families (Short CoT, Long CoT, and Multi-Turn Tool Use), STARE sustains stable RL training over thousands of steps while maintaining policy entropy within the target band. On AIME24 and AIME25, STARE outperforms DAPO and other competitive baselines by 4%-8% in average accuracy, with reflection tokens and response length growing in tandem, indicating sustained exploration-exploitation balance that further unlocks RL training potential.Code is available at https://github.com/hp-luo/STARE.

Abstract

Explore similar work

STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability

Not All Tokens Learn Alike: Attention Entropy Reveals Heterogeneous Signals in RL Reasoning

Beyond Entropy: Learning from Token-Level Distributional Deviations for LLM Reasoning