PowerFlow: Unlocking the Dual Nature of LLMs via Principled Distribution Matching

Ruishuo Chen; Yu Chen; Zhuoran Li; Longbo Huang

cs.CLMar 19, 2026

PowerFlow: Unlocking the Dual Nature of LLMs via Principled Distribution Matching

Authors: Ruishuo Chen, Yu Chen, Zhuoran Li, Longbo Huang

Organizations: Institute for Interdisciplinary Information Sciences, Tsinghua University, Beijing, China

Abstract

Unsupervised Reinforcement Learning from Internal Feedback (RLIF) has emerged as a promising paradigm for eliciting the latent capabilities of Large Language Models (LLMs) without external supervision. However, current methods rely on heuristic intrinsic rewards, which often lack a well-defined theoretical optimization target and are prone to degenerative biases. In this work, we introduce PowerFlow, a principled framework that reformulates unsupervised fine-tuning as a distribution matching problem. By casting GFlowNet as an amortized variational sampler for unnormalized densities, we propose a length-aware Trajectory-Balance objective that explicitly neutralizes the structural length biases inherent in autoregressive generation. By targeting $α$ -power distributions, PowerFlow enables the directional elicitation of the dual nature of LLMs: sharpening the distribution ( $α> 1$ ) to intensify logical reasoning, or flattening it ( $α< 1$ ) to unlock expressive creativity. Extensive experiments demonstrate that PowerFlow consistently outperforms existing RLIF methods, matching or even exceeding supervised GRPO. Furthermore, by mitigating over-sharpening in aligned models, our approach achieves simultaneous gains in diversity and quality, shifting the Pareto frontier in creative tasks.

Figures & tables

Figure 1: Illustration of the PowerFlow framework for directional capability elicitation. By matching the length-aware $\alpha$ -power distribution, PowerFlow can either sharpen the distribution ( $\alpha>1$ ) to enhance logical reasoning or flatten it ( $\alpha<1$ ) to restore latent creativity. The right panels illustrate significant performance gains and a clear Pareto improvement over existing baselines.

Figure 2: The PowerFlow framework. During training (top), the policy $\pi_{\theta}$ and $\log Z^{\prime}_{\phi}$ module are optimized via the LA-TB objective to match the $\alpha$ -power distribution of the base model while neutralizing length bias. The control knob $\alpha$ enables directional elicitation: sharpening ( $\alpha>1$ ) for reasoning or flattening ( $\alpha<1$ ) for creativity. The inference pipeline (bottom) remains standard.

Figure 3: Stability analysis of distribution matching strategies. Matching the trajectory-level

\alpha

-power distribution via standard TB or RL objectives ( -traj ) leads to rapid length collapse. Token-level normalization ( -token ) initially improves performance but eventually decays due to the exploitation of repetitive tokens. PowerFlow maintains both stable response length and superior reasoning accuracy (pass@1 on MATH) throughout training.

	MATH500	Olympiad	AIME24	AIME25	AMC23	GPQA	Average
Qwen2.5-1.5B
Base	6.20	1.90	0.00	0.00	1.40	25.80	5.88
Low-temp	18.60	4.90	0.80	0.40	7.50	26.60	9.80
Intuitor	47.40	15.30	1.50	0.80	22.30	26.40	18.95
PowerFlow (Ours)	49.30	16.00	0.80	1.50	23.80	27.70	19.85
Instruct	34.90	10.00	0.80	0.00	13.60	28.00	14.55

Table 1: Model Performance Comparison (avg@16) across Benchmarks. Within each model block, horizontal lines separate unsupervised methods (top: PowerFlow and RLIF baselines) from reference models and supervised counterparts (bottom: Instruct and GRPO ).

Figure 4: Comparison of solution diversity scores on AIME24/25. PowerFlow maintains superior strategy variety.

Figure 5: Quality vs. Semantic Diversity on creative writing tasks. The shaded region indicates the area of Pareto improvement relative to the Instruct baseline. PowerFlow (stars) consistently shifts the Pareto frontier outward across all model scales.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Performance comparison of Qwen2.5-Math-1.5B with varying

\alpha

. The results demonstrate that

\alpha=4

provides the optimal balance for distribution sharpening.

Figure 7: Pass@

n

comparison on OlympiadBench. As

n

increases, the performance gap between fine-tuned models and the Base model narrows, suggesting that PowerFlow primarily improves elicitation efficiency.

Configuration	Avg	SEM avg
Dist. matching only (= TB-traj)	$\approx 0$	—
+ format penalty $\psi$ only	14.30	0.27
+ length correction only (no $\psi$ )	33.48	0.36
PowerFlow (full)	34.30	0.30

Appendix

Table 2: Component ablation of PowerFlow on Qwen2.5-Math-1.5B. “Dist. matching only” is exactly the trajectory-balance variant TB-traj reported in Figure 3 , which suffers length collapse during training; “+ format” and “+ length correction” isolate the effect of each individual ingredient.

Figure 8: Quality-Lexical Diversity Pareto frontier across four model families. Shaded regions indicate areas of Pareto dominance over the Instruct baseline. PowerFlow (stars) uniquely improves both metrics simultaneously.

	Poem			Joke			Story			Overall Average
Model	Div. $\uparrow$	R-L $\downarrow$	Qual. $\uparrow$	Div. $\uparrow$	R-L $\downarrow$	Qual. $\uparrow$	Div. $\uparrow$	R-L $\downarrow$	Qual. $\uparrow$	Div. $\uparrow$	R-L $\downarrow$	Qual. $\uparrow$
Qwen2.5-1.5B-Instruct
Instruct	0.1623	0.1882	0.4056	0.2981	0.4391	0.7077	0.2253	0.1816	0.3726	0.2286	0.2696	0.4953
High-temp	0.2313	0.1227	0.4051	0.3620	0.3141	0.6854	0.2863	0.1402	0.3624	0.2932	0.1923	0.4843
Base	0.1895	0.1720	0.2947	0.3795	0.1804	0.6725	0.2055	0.1693	0.2894	0.2582	0.1739	0.4189
VS-Standard	0.2404	0.2087	0.3203	0.3900	0.2742	0.7181	0.2817	0.2063	0.2670	0.3040	0.2298	0.4351

Appendix

Table 3: Comprehensive Performance Comparison across Creative Writing Tasks. Div. denotes Semantic Diversity, R-L denotes ROUGE-L (Lexical Redundancy), and Qual. denotes LLM-judged Quality.

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 435 papers

Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening
Jan 29, 2026Xiaotong Ji, Rasul Tutunov, Matthieu Zimmer +1Efficient Language Model Inference RL for Language Model Reasoning
Huawei Noah’s Ark Lab · UCL Centre for Artificial Intelligence
Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillation
May 6, 2026Akiyoshi Tomihari, Issei SatoReinforcement Learning KL-Regularized RL
Department of Computer Science, The University of Tokyo
A Critical Look at Power Sampling when Test Time Sharpening LLMs
Oct 9, 2026Abiel Malkani Talwar, Christopher Nemeth, Henry MossRL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Lancaster University Lancaster, United Kingdom

Jan 29, 2026J/K move · Enter open · S save

Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening

Xiaotong Ji, Rasul Tutunov, Matthieu Zimmer, Haitham Bou Ammar

Huawei Noah’s Ark Lab · UCL Centre for Artificial Intelligence

Efficient Language Model Inference RL for Language Model Reasoning

Open paper AskPDF arXiv

Reinforcement learning (RL) post-training is a dominant approach for improving the reasoning performance of large language models (LLMs), yet growing evidence suggests that its gains arise primarily from distribution sharpening rather than the acquisition of new capabilities. Recent work has shown that sampling from the power distribution of LLMs using Markov chain Monte Carlo (MCMC) can recover performance comparable to RL post-training without relying on external rewards; however, the high computational cost of MCMC makes such approaches impractical for widespread adoption. In this work, we propose a theoretically grounded alternative that eliminates the need for iterative MCMC. We derive a token-level decomposition of the autoregressive power distribution, showing that the global power distribution can be approximated by a token-level scaled low-temperature one, where the scaling factor captures future trajectory quality. Leveraging this insight, we introduce a training-free and verifier-free algorithm that sharpens the base model's generative distribution autoregressively. Empirically, we evaluate our method on math, QA, and code tasks, and show that our method matches or surpasses one-shot GRPO without relying on any external rewards, while reducing inference latency by over

10\times

compared to MCMC-based sampling.

Abstract

RL for Language Models

Explore similar work

Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening

Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillation

A Critical Look at Power Sampling when Test Time Sharpening LLMs