When and Why Does Unsupervised RL Succeed in Mathematical Reasoning? A Manifold Envelopment Perspective

Zelin Zhang; Fei Cheng; Chenhui Chu

cs.LGMar 17, 2026

When and Why Does Unsupervised RL Succeed in Mathematical Reasoning? A Manifold Envelopment Perspective

Authors: Zelin Zhang, Fei Cheng, Chenhui Chu

Organizations: Kyoto University

Abstract

Although outcome-based reinforcement learning (RL) significantly advances the mathematical reasoning capabilities of Large Language Models (LLMs), its reliance on computationally expensive ground-truth annotations imposes a severe scalability bottleneck. Unsupervised RL guided by intrinsic rewards offers a scalable alternative, yet it suffers from opaque training dynamics and catastrophic instability, such as policy collapse and reward hacking. In this paper, we first design and evaluate a suite of intrinsic rewards that explicitly enforce concise and certain generation. Second, to discover the boundaries of this approach, we test base models across a spectrum of intrinsic reasoning capabilities, revealing how a model's foundational logical prior dictates its success or failure. Finally, to demystify why certain configurations stabilize while others collapse, we introduce a novel geometric diagnostic lens, showing that successful cases are enveloped by manifolds. Ultimately, our work goes beyond merely demonstrating that enforcing concise and certain responses successfully boosts mathematical reasoning; we reveal when this unsupervised approach breaks down and geometrically diagnose why.

Figures & tables

Figure 1: Overview of Research Pipeline. (a) Design Rewards: enforce concise and certain. (b) Find Boundary Conditions: evaluate across models with different reasoning abilities. (c) Find Diagnostic Lens: DTW clustering to 3D phase space.

Method	Uncertainty	Length
Ent	Penalty	Strong penalty
AvgEnt	Penalty	None
LP ∗	None	Strong penalty
CH2 ∗	Penalty	Weak penalty
CP ∗	Penalty	Encourage (Confusing)

Table 1: Formulation of the unsupervised rewards. We decompose each reward into two orthogonal dimensions: uncertainty penalization and length penalization. Methods marked with an asterisk ( ∗ ) are newly designed formulations introduced in this work to complete the design matrix across both dimensions. Ent Agarwal et al. (2025) and AvgEnt Prabhudesai et al. (2025) are previous methods.

Model	Method	MATH-500			Olympiad			Minerva			AIME 24			AMC 23			AIME 26			AVG
Model	Method	P@1	P@5	P@10	P@1	P@5	P@10	P@1	P@5	P@10	P@1	P@5	P@10	P@1	P@5	P@10	P@1	P@5	P@10	P@1	P@5	P@10
Qwen3-1.7B	Base	70.2	81.4	84.9	30.1	41.6	46.9	30.6	42.0	46.0	6.0	14.2	19.1	47.2	61.4	68.0	7.1	15.5	20.0	42.7	54.0	58.5
	S-RL	78.8	89.5	91.8	39.5	52.0	56.0	35.4	49.2	53.9	17.9	31.0	37.8	56.6	76.0	82.1	12.7	26.0	31.1	51.0	63.3	67.0
	Ent	79.8	90.4	92.4	41.4	54.9	58.8	34.4	48.3	52.9	22.5	42.6	49.3	61.3	81.8	88.1	14.0	27.9	31.6	52.2	65.1	68.7
	AvgEnt	78.2	89.7	91.9	39.8	52.8	56.7	36.1	50.0	54.5	20.2	38.8	49.1	58.3	80.3	87.9	16.0	34.7	42.0	51.2	64.3	68.1
	LP	80.0	90.4	92.4	42.6	55.3	59.1	34.4	47.9	52.4	20.0	34.2	39.3	57.8	79.7	86.8	14.4	28.0	33.6	52.7	65.0	68.5

Table 2: Main Results on Mathematical Reasoning Benchmarks. Performance comparison across models, reward formulations, and training settings. All training is conducted on the DAPO-Math-17K dataset. S-RL refers to Supervised RL. Bold denotes the overall best performance, and underlined indicates the best result among unsupervised methods within each model. P@k means Pass@k. † indicates a semi-collapse phenomenon. For the DeepSeek-Distill model, these unsupervised methods experience a very brief initial period of improvement where the peak performance slightly exceeds the base model, but the training rapidly collapses shortly afterward.

Entropy Cluster	Semantic	Representative Tokens (Top Frequency)	Characteristics
Low (Execution)	Arithmetic	Qwen : 1 , 2 , 3 , 0 , + , - DeepSeek : + , 2 , 4 , - , 3 , 1 Llama : 4 , 2 , = , + , 1 , 3	Operators and concrete numerical digits.
Low (Execution)	Syntax	Qwen : by , ’t , to , be , ( , ) DeepSeek : ’t , ) , be , by , of , to Llama : of , { , }{ , \ , frac , $	Structural formatting and LaTeX syntax.
Medium (Logic)	Formulation	Qwen : Let , The , this , it DeepSeek : Let , this , here , each Llama : need , let , given , find , solve	Establishing variables and conditions.
Medium (Logic)	Transitions	Qwen : Wait , But , ? , Therefore , So DeepSeek : ? , but , should , So , Wait Llama : However , To , must , also , Now	Step-by-step reasoning deductions.
High (Thinking)	Exploration	Qwen : perhaps , maybe , suppose , Hmm DeepSeek : maybe , perhaps , wait , Hmm , think Llama : First , start , break , analyze	Macro-planning and expressing uncertainty.
	Verification	Qwen : confirm , verify , actually , If , assume DeepSeek : consider , double , Because , Or Llama : denote , determine , considering	Halting to verify or initiating branch paths.

Table 3: Semantic Interpretation of Entropy Clusters Across Models. We extract the representative tokens for the Qwen, DeepSeek-Distill, and Llama. To ensure objective categorization, these tokens are rigorously filtered by retaining only the core samples (the 50% closest to the respective cluster centroids in the DTW space) and ranked strictly by their occurrence frequency. Across all three model families, these high-frequency tokens consistently align with the semantic functions of Execution, Logic, and Thinking.

Panel of Figure 2 — Figure 2: Visualization of 3D Semantic Manifold Boundaries. The three axes define the phase space corresponding to the mean entropy of the Thinking , Logic , and Execution semantic clusters, as established in Section 5 . (a-c) Exploration Manifolds: The translucent polygons denote the 3D convex hulls encompassing all aggregated trajectory points for specific training methods. The legend identifies the training method and provides the calculated volume of its corresponding hull. DS-Distill denotes DeepSeek-R1-Distill-Llama-8B.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Method	MATH-500			Olympiad			Minerva			AIME 24			AMC 23			AIME 26			AVG
Model	Method	P@1	P@5	P@10	P@1	P@5	P@10	P@1	P@5	P@10	P@1	P@5	P@10	P@1	P@5	P@10	P@1	P@5	P@10	P@1	P@5	P@10
Qwen3-8B (8K)	Base	85.1	91.9	93.7	46.9	56.7	60.8	42.2	51.4	54.4	38.3	53.2	59.0	70.5	85.9	90.4	28.1	42.0	46.9	58.5	67.5	70.8
	S-RL	90.9	94.8	95.7	56.4	64.9	68.0	47.7	56.1	58.7	50.2	66.5	71.8	86.1	93.1	94.6	43.8	58.3	63.6	66.4	73.6	76.0
	Ent	91.7	95.7	96.5	58.3	67.7	70.8	47.7	55.9	58.1	53.5	76.5	82.1	86.4	96.4	97.5	49.4	66.4	70.4	67.7	75.6	77.8
	AvgEnt	89.9	94.4	95.5	54.3	62.3	65.3	46.8	56.0	58.9	44.4	61.7	69.6	82.8	93.4	94.7	44.4	60.9	66.7	64.8	72.3	74.8
	LP	91.9	95.8	96.6	58.4	67.3	70.2	47.5	56.2	59.4	53.1	75.3	82.8	87.8	94.5	96.2	50.0	65.4	70.0	67.8	75.4	77.7

Appendix

Table 4: Boundary Conditions Results. We expand our evaluation across two dimensions: (1) 8K training and evaluations results; and (2) cross-dataset generalization tests utilizing DeepMath. Notions follow Table 2 .

Panel of Figure 3 — Figure 3: Token Entropy Trajectories on Qwen3-1.7B (Part 1 of 2). Clustering results evaluated on the AIME 24 dataset across six different reward formulations. The consistent emergence of Execution, Logic, and Thinking clusters is observed despite the smaller parameter scale. The text boxes within each subplot display the top frequency tokens corresponding to that cluster.

Panel of Figure 4 — Figure 4: Token Entropy Trajectories on Qwen3-1.7B (Part 2 of 2). (Continued from previous page.)

Panel of Figure 5 — Figure 5: Token Entropy Trajectories on Qwen3-8B (Standard Setup) (Part 1 of 2). Visualizations of the optimization dynamics under standard context length on our primary testbed model. The text boxes within each subplot display the top frequency tokens corresponding to that cluster.

Panel of Figure 6 — Figure 6: Token Entropy Trajectories on Qwen3-8B (Standard Setup) (Part 2 of 2). (Continued from previous page.)

Panel of Figure 7 — Figure 7: Token Entropy Trajectories on Qwen3-8B (8K Context) (Part 1 of 2). These visualizations demonstrate that the emergence of reasoning clusters remains consistent when scaling the context window to 8K. The text boxes within each subplot display the top frequency tokens corresponding to that cluster.

Panel of Figure 8 — Figure 8: Token Entropy Trajectories on Qwen3-8B (8K Context) (Part 2 of 2). (Continued from previous page.)

Panel of Figure 9 — Figure 9: Token Entropy Trajectories on Qwen3-8B (DeepMath Dataset) (Part 1 of 2). Trajectories analyzed under deeper RL interventions, illustrating the robustness of the intrinsic phase separation. The text boxes within each subplot display the top frequency tokens corresponding to that cluster.

Panel of Figure 10 — Figure 10: Token Entropy Trajectories on Qwen3-8B (DeepMath Dataset) (Part 2 of 2). (Continued from previous page.)

Panel of Figure 11 — Figure 11: Token Entropy Trajectories on DeepSeek-R1-Distill-Llama-8B (Part 1 of 2). Analysis conducted on an architecture with a highly-aligned initial state, further isolating the impact of the reward signal. The text boxes within each subplot display the top frequency tokens corresponding to that cluster.

Panel of Figure 12 — Figure 12: Token Entropy Trajectories on DeepSeek-R1-Distill-Llama-8B (Part 2 of 2). (Continued from previous page.)

Panel of Figure 13 — Figure 13: Token Entropy Trajectories on Llama3.1-8B (Part 1 of 2). Clustering evaluation on the Llama base architecture to verify cross-model generalizability of the mechanistic interpretations. The text boxes within each subplot display the top frequency tokens corresponding to that cluster.

Panel of Figure 14 — Figure 14: Token Entropy Trajectories on Llama3.1-8B (Part 2 of 2). (Continued from previous page.)

Panel of Figure 15 — Figure 15: 3D Exploration Boundaries of Semantic Phases (Qwen Models). The convex hulls illustrate the volume of the entropy phase space explored by the models under six different reward formulations.

Panel of Figure 16 — Figure 16: 3D Exploration Boundaries of Semantic Phases (Other Architectures). The convex hulls illustrate the volume of the entropy phase space explored by the models under six different reward formulations.

Optimization
Hyperparameter	Value
Actor Learning Rate	$1\times 10^{-6}$
Learning Rate Warmup Steps	10
Weight Decay	0.1
Mini-batch Size	24
Prompt Batch Size	8

Appendix

Table 5: Hyperparameters for the RL trainings.

Unsupervised RL

See all 13 papers

Free Energy-Driven Reinforcement Learning with Adaptive Advantage Shaping for Unsupervised Reasoning in LLMs
Apr 11, 2026Yiming Huang, Zhenbo Shi, Xin-Cheng Wen +4Reinforcement Learning RL for Language Model Reasoning
Harbin Institute of Technology, Shenzhen · The Chinese University of Hong Kong · Peng Cheng Laboratory
Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning
Jun 2, 2026Ziyue Wang, Aomufei Yuan, Yongfu Zhu +10LLM Alignment RL for Language Model Reasoning
1Peking University · 2JINGDONG · 3Shanghai Innovation Institute +2
DenoiseRL: Bootstrapping Reasoning Models to Recover from Noisy Prefixes
May 27, 2026Caijun Xu, Changyi Xiao, Zhongyuan Peng +1LLM Self-Correction RL for Language Model Reasoning
1Fudan University · 2Shanghai Innovation Institute

Apr 11, 2026J/K move · Enter open · S save

Free Energy-Driven Reinforcement Learning with Adaptive Advantage Shaping for Unsupervised Reasoning in LLMs

Yiming Huang, Zhenbo Shi, Xin-Cheng Wen, Jichuan Zeng+3

Harbin Institute of Technology, Shenzhen · The Chinese University of Hong Kong · Peng Cheng Laboratory

Reinforcement Learning RL for Language Model Reasoning LLM Reasoning Reward Shaping

Open paper AskPDF arXiv

Unsupervised reinforcement learning (RL) has emerged as a promising paradigm for enabling self-improvement in large language models (LLMs). However, existing unsupervised RL-based methods often lack the capacity to adapt to the model's evolving reasoning capabilities during training. Therefore, these methods can misdirect policy optimization in the absence of ground-truth supervision. To address this issue, we introduce FREIA, a novel RL-based algorithm built on two key innovations: (1) Free Energy-Driven Reward (FER) adapts rewards to balance consensus and exploration based on the Free Energy Principle. (2) Adaptive Advantage Shaping (AAS) adaptively adjusts learning signals based on the statistical characteristics of sampled rewards. Empirical evaluations on nine datasets across three reasoning tasks showcase that FREIA outperforms other unsupervised RL-based baselines. Notably, in mathematical reasoning tasks, FREIA surpasses other methods by an average of 0.5 to 3.5 points in Pass@1 using the DeepSeek-R1-Distill-Qwen-1.5B model.

Abstract

Unsupervised RL

Explore similar work

Free Energy-Driven Reinforcement Learning with Adaptive Advantage Shaping for Unsupervised Reasoning in LLMs

Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning

DenoiseRL: Bootstrapping Reasoning Models to Recover from Noisy Prefixes