cs.AIOct 19, 2026

Sparse RL Relaxation for LLM Reasoning

Authors: Jiabin Fan, Matthias Lindemann, Zita Marinho, André FT Martins, Lili Mou

Organizations: University of Alberta · Alberta Machine Intelligence Institute (Amii) · Instituto de Telecomunicações · Instituto Superior Técnico, Universidade de Lisboa · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Recent work has explored continuous representations in place of discrete tokens during Chain-of-Thought (CoT) reasoning, often referred to as soft CoT. This is motivated by the idea that a mixture of token embeddings can represent a superposition of multiple reasoning possibilities within a single step. However, it remains challenging to train large language models to realize this potential. Existing work has explored reinforcement learning (RL)-style training for soft CoT. However, these approaches treat soft tokens as actions in a Markov decision process rather than as differentiable intermediate computation steps. Thus, they do not propagate gradients across the soft tokens and miss out on the signal of how the choice of one soft token affects the next. In this paper, we propose a sparse relaxation method for RL (called SpaRLax), which leverages the differentiability of soft-CoT reasoning over time steps and makes training more stable and effective. We adopt an entmax sparsity method that adaptively determines the degree of softness, which truncates long-tail noise at confident steps while retaining multiple plausible candidates when uncertain. Across nine reasoning benchmarks, our method achieves the highest average performance compared with baseline methods. Further analyses validate the effective use of soft-token superposition in our approach, and also demonstrate the contribution of adaptive sparsity in our approach.

Figures & tables

Figure 1: Overview of our approach. Black arrows indicate forward propagation, and blue arrows indicate backpropagation. Different from Soft-GRPO ( Zheng and Lee, 2025 ) , we only apply policy gradient to output steps, but backpropagate cross-step gradients for reasoning steps.
 Math ReasoningKnowledgeCode 
MethodGSM8KMATHAIME24AIME25AMC23Olym.GPQAHEvalMBPPW.Avg.
Hard CoT Inference
Initial80.08 ±\pm 0.7375.18 ±\pm 0.9615.94 ±\pm 4.1220.94 ±\pm 2.9059.69 ±\pm 4.9526.24 ±\pm 0.7520.55 ±\pm 2.6155.11 ±\pm 2.1958.55 ±\pm 1.8759.58 ±\pm 0.48
Discrete-GRPO84.42 ±\pm 0.7275.97 ±\pm 0.9516.98 ±\pm 4.2621.25 ±\pm 2.8462.11 ±\pm 5.8227.31 ±\pm 0.7421.26 ±\pm 2.2356.23 ±\pm 2.3159.45 ±\pm 2.1461.90 ±\pm 0.38
Soft-GRPO84.29 ±\pm 0.5575.48 ±\pm 1.3815.94 ±\pm 4.4620.94 ±\pm 3.5158.91 ±\pm 5.3526.25 ±\pm 0.8720.15 ±\pm 1.9055.22 ±\pm 2.8457.66 ±\pm 1.8861.24 ±\pm 0.47
STHT80.35 ±\pm 0.5876.49 ±\pm 1.0213.85 ±\pm 4.6522.92 ±\pm 3.2862.11 ±\pm 5.6527.36 ±\pm 0.7522.77 ±\pm 2.3055.16 ±\pm 2.0058.18 ±\pm 2.0460.28 ±\pm 0.43
Table 1: Pass@1 performance (%) under hard and soft chain-of-thought (CoT) inference. Results are reported as mean ±\pm standard deviation. Bold and underlined values indicate the best and second-best mean performance within each inference setting, respectively.
 Math ReasoningKnowledgeCode 
MethodGSM8KMATHAIME24AIME25AMC23Olym.GPQAHEvalMBPPW.Avg.
Hard CoT Inference
Initial97.5091.6050.0055.0087.5042.0782.3283.5486.7781.22
Discrete-GRPO ( Guo et al., 2025 )97.5091.8046.6758.3395.0042.9682.3284.7687.1681.65
SofT-GRPO ( Zheng and Lee, 2025 )97.3591.8040.0053.3390.0041.1974.7581.7184.0580.14
STHT ( Butt et al., 2026 )97.1291.6046.6756.6797.5042.3785.3583.5484.8281.28
Table 2: Pass@32 performance (%) under hard and soft chain-of-thought (CoT) inference. Bold and underlined values denote the best and second-best results within each inference setting, respectively.
Figure 2: Comparing the hidden states when using the soft token embedding and replacing it with top-1 token embedding. Higher KL and lower cosine indicate a stronger utilization of superposition.
Figure 3: Effect of candidate-set size on softmax and entmax training. Softmax becomes increasingly sensitive to larger kk , whereas entmax remains robust as kk increases.
ParameterDiscrete GRPO ( Guo et al., 2025 )SofT-GRPO ( Zheng and Lee, 2025 )SpaRLax (ours)STHT ( Butt et al., 2026 )LEPO ( Zhou et al., 2026 )
Batch configuration
Fine-tuningFull-parameter fine-tuning
Filter overlong promptsEnabledEnabledEnabledEnabledEnabled
Prompt batch size3232323216
Responses per prompt (group size)88888
Global rollout batch (trajectories)256256256256128
Table 3: Training hyperparameters. The first four columns retain the supplied GSM8K settings; LEPO reports launcher defaults, which use DeepScaleR training data. Batch sizes distinguish prompt groups from generated trajectories.
More in this topic

CoT Reasoning

Chain-of-thought (CoT) reasoning studies how language models use intermediate steps, visible or latent, to solve multi-step tasks.

See all 228 papers

Explore similar work

CardsList
Apr 9, 2026

SeLaR: Selective Latent Reasoning in Large Language Models

Chain-of-Thought (CoT) has become a cornerstone of reasoning in large language models, yet its effectiveness is constrained by the limited expressiveness of discrete token sampling. Recent latent reasoning approaches attempt to alleviate this limitation by replacing discrete tokens with soft embeddings (probability-weighted mixtures of token embeddings) or hidden states, but they commonly suffer from two issues: (1) global activation injects perturbations into high-confidence steps, impairing reasoning stability; and (2) soft embeddings quickly collapse toward the highest-probability token, limiting exploration of alternative trajectories. To address these challenges, we propose SeLaR (Selective Latent Reasoning), a lightweight and training-free framework. SeLaR introduces an entropy-gated mechanism that activates soft embeddings only at low-confidence steps, while preserving discrete decoding at high-confidence steps. Additionally, we propose an entropy-aware contrastive regularization that pushes soft embeddings away from the dominant (highest-probability) token's direction, encouraging sustained exploration of multiple latent reasoning paths. Experiments on five reasoning benchmarks demonstrate that SeLaR consistently outperforms standard CoT and state-of-the-art training-free methods.

Renyu Fu, Guibo Luo

Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology Shenzhen Graduate School, Peking University

CoT ReasoningContinuous Latent ReasoningReasoning Consistency in Language Models
AskPDF
May 10, 2026

RuPLaR : Efficient Latent Compression of LLM Reasoning Chains with Rule-Based Priors From Multi-Step to One-Step

The Chain-of-Thought (CoT) paradigm, while enhancing the interpretability of Large Language Models (LLMs), is constrained by the inefficiencies and expressive limits of natural language. Latent Chain-of-Thought (latent CoT) reasoning, which operates in a continuous latent space, offers a promising alternative but faces challenges from structural complexities in existing multi-step or multi-model paradigms, such as error propagation and coordination overhead. In this paper, we introduce One-Model One-Step, a novel compression framework for Latent Reasoning with Rule-Based Priors(RuPLaR) to address this challenge. Our method trains an LLM to autonomously generate latent reasoning tokens in a single training stage, guided by rule-based prior probability distributions, thereby eliminating cascaded processes and inter-model dependencies. To ensure reasoning quality, we design a joint training objective that enforces answer consistency via cross-entropy, aligns soft token distributions with rule-based priors via KL divergence (the Soft Thinking constraint), and adds a problem-thought semantic alignment constraint in the representation space. Extensive experiments show that our compression framework not only improves accuracy by 11.03% over existing latent CoT methods on GSM8k-Aug but also achieves this with minimal token usage, underscoring its effectiveness and extensibility. Code: https://github.com/xiaocen-luo/RuPLaR.

Xiaocheng Luo, Kang Wang, Zaifu Zhan +2

School of Computer Science and Technology, Soochow University, Suzhou, 215006, Jiangsu, PR China · Department of Electrical and Computer Engineering, University of Minnesota, Minneapolis, 55455, MN, USA

LLM CompressionCoT CompressionLLM ReasoningLatent CoT Reasoning
AskPDF
Dec 25, 2025

Do Latent Tokens Think? A Causal and Adversarial Analysis of Chain-of-Continuous-Thought

Latent tokens are gaining attention for enhancing reasoning in large language models (LLMs), yet their internal mechanisms remain unclear. This paper examines the problem from a reliability perspective, uncovering fundamental weaknesses: latent tokens function as uninterpretable placeholders rather than encoding faithful reasoning. While resistant to perturbation, they promote shortcut usage over genuine reasoning. We focus on Chain-of-Continuous-Thought (COCONUT), which claims better efficiency and stability than explicit Chain-of-Thought (CoT) while maintaining performance. We investigate this through two complementary approaches. First, steering experiments perturb specific token subsets, namely COCONUT and explicit CoT. Unlike CoT tokens, COCONUT tokens show minimal sensitivity to steering and lack reasoning-critical information. Second, shortcut experiments evaluate models under biased and out-of-distribution settings. Results on MMLU and HotpotQA demonstrate that COCONUT consistently exploits dataset artifacts, inflating benchmark performance without true reasoning. These findings reposition COCONUT as a pseudo-reasoning mechanism: it generates plausible traces that conceal shortcut dependence rather than faithfully representing reasoning processes.

Yuyi Zhang, Boyu Tang, Tianjie Ju +2

Shanghai Jiao Tong University

Faithfulness of Language Model ReasoningImplicit Reasoning in Language ModelsContinuous Latent ReasoningLanguage Model Reasoning EvaluationShortcut Learning in Language Models
AskPDF