Free Energy-Driven Reinforcement Learning with Adaptive Advantage Shaping for Unsupervised Reasoning in LLMs

Yiming Huang; Zhenbo Shi; Xin-Cheng Wen; Jichuan Zeng; Cuiyun Gao; Peiyi Han; Chuanyi Liu

cs.CLApr 11, 2026

Free Energy-Driven Reinforcement Learning with Adaptive Advantage Shaping for Unsupervised Reasoning in LLMs

Authors: Yiming Huang, Zhenbo Shi, Xin-Cheng Wen, Jichuan Zeng, Cuiyun Gao, Peiyi Han, Chuanyi Liu

Organizations: Harbin Institute of Technology, Shenzhen · The Chinese University of Hong Kong · Peng Cheng Laboratory

Abstract

Unsupervised reinforcement learning (RL) has emerged as a promising paradigm for enabling self-improvement in large language models (LLMs). However, existing unsupervised RL-based methods often lack the capacity to adapt to the model's evolving reasoning capabilities during training. Therefore, these methods can misdirect policy optimization in the absence of ground-truth supervision. To address this issue, we introduce FREIA, a novel RL-based algorithm built on two key innovations: (1) Free Energy-Driven Reward (FER) adapts rewards to balance consensus and exploration based on the Free Energy Principle. (2) Adaptive Advantage Shaping (AAS) adaptively adjusts learning signals based on the statistical characteristics of sampled rewards. Empirical evaluations on nine datasets across three reasoning tasks showcase that FREIA outperforms other unsupervised RL-based baselines. Notably, in mathematical reasoning tasks, FREIA surpasses other methods by an average of 0.5 to 3.5 points in Pass@1 using the DeepSeek-R1-Distill-Qwen-1.5B model.

Figures & tables

Figure 1: An analysis of reward signals for a math problem, where the correct answer is “13”. Note that

p

indicates the frequency of sampled answers.

Figure 2: An analysis of standard advantage shaping. (Top: Weak Consensus) The high-reward answer appears infrequently. Standard advantage assigns excessive values to these rare answers. (Bottom: Strong Consensus) The majority answer dominates the population. Standard advantage excessively penalizes the occasional deviations.

Figure 3: The overall framework of FREIA, including Free Energy-Driven Reward (FER) and Adaptive Advantage Shaping (AAS).

Figure 4: Visualization of FER. Specifically, the left part depicts the Exploration term when $C_{G}$ is low, while the right part shows the Consensus term under high $C_{G}$ .

Dataset	Base	GRPO (Supervised)	TTRL	Entropy	Intuitor	FREIA
Qwen2.5-Math-1.5B-Instruct
MATH500	74.2	75.2 ±0.7	75.0 ±0.4	74.6 ±0.8	74.4 ±0.8	75.4 ±0.2
AIME24	10.0	13.3 ±1.9	13.3 ±0.0	10.0 ±0.0	10.0 ±0.0	13.3 ±0.0
AIME25	3.3	16.7 ±0.0	16.7 ±0.0	13.3 ±0.0	10.0 ±0.0	16.7 ±0.0
AMC23	47.5	52.5 ±1.4	52.5 ±0.0	47.5 ±0.0	47.5 ±0.0	52.5 ±0.0
Minerva	28.7	31.3 ±0.9	30.9 ±0.5	29.4 ±0.8	28.7 ±0.9	32.0 ±0.4

Table 1: Experimental results on multiple mathematical reasoning benchmarks. The results are reported as mean and standard deviation across 3 random seeds ( i.e. , Mean

{}_{\pm\text{Std}}

). The best results are highlighted in bold.

Figure 5: Experimental results on SQL generation and multi-modal reasoning.

Figure 6: Comparison of total training wall-clock time (minutes) using various methods.

Figure 7: Training dynamics of FREIA. (a) Policy entropy using DeepSeek-R1-Distill-Qwen-1.5B; (b) Group confidence score (

C_{G}

); (c) Exploration component of the reward; (d) Consensus component of the reward.

Figure 8: Ablation study and hyperparameter sensitivity analysis of FREIA. (a) Average Pass@1 of the full FREIA compared to its ablated variants. (b) Average Pass@1 under varying values of the hyperparameter

\alpha

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Name	Access Link	# Samples
Models
Qwen2.5-Math-1.5B-Instruct	https://huggingface.co/Qwen/Qwen2.5-Math-1.5B-Instruct	-
Qwen2.5-3B-Instruct	https://huggingface.co/Qwen/Qwen2.5-3B-Instruct	-
DeepSeek-R1-Distill-Qwen-1.5B	https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B	-
Qwen2.5-Coder-3B-Instruct	https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct	-
Qwen2.5-VL-3B-Instruct	https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct	-

Appendix

Table 2: Details of models and benchmarks utilized in this study.

Setup: Sample Size

G=8

. Scenario: A high-ambiguity task with a weak false consensus.

Generated Answers:

A_{1}\sim A_{5}

are incorrect (Weak Majority).

A_{6}\sim A_{8}

are correct (Strong Minority).

Empirical Frequencies:

f_{wrong}=5/8=0.625

f_{correct}=3/8=0.375

1. Belief Formation (Step 1, with

\alpha=2.0

- Raw Proportions (

f

\{0.625,0.375\}

- Squared Proportions (

f^{\alpha}

\{0.625^{2}\approx 0.39,\ 0.375^{2}\approx 0.14\}

. Sum

\Sigma\approx 0.53

- Belief Weights (

w=f^{\alpha}/\Sigma

Appendix

Table 3: A case study on False Consensus. FREIA detects the high uncertainty and inverts the reward signal, prioritizing the correct minority over the incorrect majority.

Figure 9: The evolution of the reward skewness throughout training.

Method	ACS ( $\downarrow$ )	Self-BLEU ( $\downarrow$ )	GPT-Diversity ( $\uparrow$ )
GRPO	0.88	0.79	1.05
TTRL	0.76	0.68	1.23
FREIA	0.62	0.51	1.56

Appendix

Table 4: Diversity metrics on the MATH500 dataset using Qwen2.5-Math-1.5B-Instruct.

\downarrow

indicates lower is better;

\uparrow

indicates higher is better.

Method	Geometry3K	Spider	BIRD	Avg.
Base	25.7	70.2	43.6	46.5
GRPO	35.6 ±1.1	73.8 ±0.5	55.7 ±0.9	55.0 ±0.8
TTRL	35.3 ±0.8	74.0 ±0.4	56.0 ±0.2	55.1 ±0.4
Intuitor	34.8 ±0.9	72.8 ±0.6	54.5 ±0.8	54.0 ±0.8
Entropy	34.4 ±0.7	72.5 ±0.5	54.0 ±0.7	53.6 ±0.6
FREIA	36.1 ±0.5	74.4 ±0.2	56.4 ±0.3	55.6 ±0.3

Appendix

Table 5: Generalization performance on SQL generation (Spider, BIRD) and multi-modal reasoning (Geometry3K). The results are reported as mean and standard deviation across 3 random seeds (format:

\text{Mean}_{\pm\text{Std}}

Model	Method	MATH500	AIME24	AIME25	AMC23	Minerva	Olympiad	Avg.
Qwen2.5-Math-1.5B	w/o Consensus	74.4	10.0	6.7	47.5	29.4	38.9	34.5
	w/o Exploration	74.8	13.3	10.0	50.0	30.1	39.5	36.3
	w/o AAS	75.2	13.3	13.3	50.0	31.3	40.5	37.3
	FREIA	75.4	13.3	16.7	52.5	32.0	41.2	38.5
Qwen2.5-3B-Instruct	w/o Consensus	63.2	3.3	3.3	35.0	24.6	29.5	26.5
	w/o Exploration	64.2	6.7	6.7	37.5	25.4	30.7	28.5

Appendix

Table 6: Detailed Ablation Study (RQ3). We compare FREIA against removing key components: w/o Consensus, w/o Exploration, and w/o AAS.

Model	Coefficient	MATH500	AIME24	AIME25	AMC23	Minerva	Olympiad	Avg.
Qwen2.5-Math-1.5B	$\alpha=0.5$	74.8	13.3	16.7	50.0	31.3	40.1	37.7
	$\alpha=1.0$	75.2	13.3	16.7	52.5	31.6	40.8	38.4
	$\alpha=2.0$	75.4	13.3	16.7	52.5	32.0	41.2	38.5
	$\alpha=3.0$	75.0	13.3	16.7	50.0	32.0	40.9	38.0
	$\alpha=4.0$	74.8	13.3	13.3	50.0	31.3	40.2	37.2
Qwen2.5-3B-Instruct	$\alpha=0.5$	64.6	6.7	6.7	37.5	25.4	30.6	28.6

Appendix

Table 7: Hyperparameter Sensitivity Analysis (RQ4). Impact of the coefficient

\alpha

on model performance across diverse benchmarks.

Setting	Mixing Strategy	MATH500	AIME24	AIME25	AMC23	Minerva	Olympiad	Avg.
Fixed Mixing	$\lambda=0.2$	74.6	10.0	13.3	48.5	30.5	39.8	36.1
Fixed Mixing	$\lambda=0.5$	74.8	13.3	13.3	50.0	30.8	40.2	37.1
Fixed Mixing	$\lambda=0.8$	75.0	13.3	16.7	50.0	31.3	40.7	37.8
FREIA	Dynamic $C_{G}$	75.4	13.3	16.7	52.5	32.0	41.2	38.5

Appendix

Table 8: Ablation study on the mixing coefficient

\lambda

using Qwen2.5-Math-1.5B-Instruct. Baselines with fixed

\lambda

represent a static trade-off between Consensus (

\lambda

) and Surprisal (

1-\lambda

). FREIA uses Group Confidence (

C_{G}

) to adaptively modulate this trade-off per instance.

Setting	MATH500	AIME24	AIME25	AMC23	Minerva	Olympiad	Avg.	Growth Rate
FREIA ( $G=4$ )	74.6	13.3	13.3	47.5	30.5	40.1	36.6	-
FREIA ( $G=6$ )	75.2	13.3	16.7	50.0	31.6	41.1	38.0	+1.4%
FREIA ( $G=8$ )	75.4	13.3	16.7	52.5	32.0	41.2	38.5	+0.5%

Appendix

Table 9: Ablation study on Group Size

G

using Qwen2.5-Math-1.5B-Instruct. We report the Pass@1 score across all datasets. Growth Rate indicates the performance gain relative to the previous group size setting.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,661 papers

Adapt to Thrive! Adaptive Power-Mean Policy Optimization for Improved LLM Reasoning
Apr 11, 2026Yiming Huang, Zhenbo Shi, Shuzheng Gao +3RL for Language Model Reasoning Policy Optimization
Harbin Institute of Technology, Shenzhen · The Chinese University of Hong Kong · Peng Cheng Laboratory
When and Why Does Unsupervised RL Succeed in Mathematical Reasoning? A Manifold Envelopment Perspective
Mar 17, 2026Zelin Zhang, Fei Cheng, Chenhui ChuUnsupervised RL Reinforcement Learning
Kyoto University
DenoiseRL: Bootstrapping Reasoning Models to Recover from Noisy Prefixes
May 27, 2026Caijun Xu, Changyi Xiao, Zhongyuan Peng +1LLM Self-Correction RL for Language Model Reasoning
1Fudan University · 2Shanghai Innovation Institute

Apr 11, 2026J/K move · Enter open · S save

Adapt to Thrive! Adaptive Power-Mean Policy Optimization for Improved LLM Reasoning

Yiming Huang, Zhenbo Shi, Shuzheng Gao, Cuiyun Gao+2

Harbin Institute of Technology, Shenzhen · The Chinese University of Hong Kong · Peng Cheng Laboratory

RL for Language Model Reasoning Policy Optimization LLM Reasoning Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) is an essential paradigm that enhances the reasoning capabilities of Large Language Models (LLMs). However, existing methods typically rely on static policy optimization schemes that misalign with the model's evolving reasoning capabilities. To address this issue, we propose Adaptive Power-Mean Policy Optimization (APMPO), which comprises two main innovations: Power-Mean Policy Optimization (PMPO) and Feedback-Adaptive Clipping (FAC). Specifically, PMPO introduces a generalized power-mean objective. This enables the model to adaptively transition from the signal-amplifying behavior of the arithmetic mean to the consistency-enforcing behavior of the geometric mean. FAC adaptively adjusts clipping bounds based on real-time reward statistics to overcome the limitations of static mechanisms. Capitalizing on these innovations, APMPO improves learning dynamics and reasoning performance. Extensive experiments on nine datasets across three reasoning tasks showcase the superiority of APMPO over state-of-the-art RLVR-based baselines. For instance, APMPO boosts the average Pass@1 score on mathematical reasoning benchmarks by 3.0 points compared to GRPO when using Qwen2.5-3B-Instruct.

Abstract

Reinforcement Learning

Explore similar work

Adapt to Thrive! Adaptive Power-Mean Policy Optimization for Improved LLM Reasoning

When and Why Does Unsupervised RL Succeed in Mathematical Reasoning? A Manifold Envelopment Perspective

DenoiseRL: Bootstrapping Reasoning Models to Recover from Noisy Prefixes