CFG-Policy: Unified Diffusion Policy Learning Framework via Classifier-free Guidance

Shutong Ding; Zhaoyang Liu; Yimiao Zhou; Jingya Wang; Jiayuan Gu; Ye Shi

cs.ROOct 10, 2026

CFG-Policy: Unified Diffusion Policy Learning Framework via Classifier-free Guidance

Authors: Shutong Ding, Zhaoyang Liu, Yimiao Zhou, Jingya Wang, Jiayuan Gu, Ye Shi

Organizations: ShanghaiTech University, Shanghai, China · InstAdapt, Shanghai, China

Abstract

Diffusion policies are increasingly used in robotic control and manipulation, vision-language-action models, and world-model-based agents. Their ability to represent multimodal action distributions offers rich opportunities for exploration in reinforcement learning (RL). However, optimizing diffusion policies with RL can reduce the behavioral diversity and further limit their exploration capability. In RL post-training, directly fine-tuning a pretrained policy may also disrupt its acquired capabilities, while action-space residuals may lack the expressive power for multimodal exploration. Together, these issues reveal a common challenge across online and offline-to-online diffusion RL: improving task performance while retaining the behavioral diversity and capabilities needed for continued adaptation. We introduce CFG-Policy, a unified, plug-and-play framework that separates exploration and exploitation into two generative policies and combines their score or velocity fields during sampling through classifier-free guidance. For online RL, a frozen exploration branch sustains behavioral diversity while an exploitation branch learns from task rewards; for offline-to-online RL, a pretrained exploitation branch remains frozen while a trainable branch steers the composed behavior. CFG-Policy preserves the underlying RL objectives and improves performance with QSM and QVPO on 5 MuJoCo tasks and with Flow-SDE+PPO on 4 RoboTwin manipulation tasks. Project page is at https://zhy-liu.github.io/cfg-policy-page.

Figures & tables

Figure 1: Exploration–exploitation challenges in diffusion RL. In online diffusion RL, optimizing a single generative policy can progressively reduce behavioral diversity and miss better modes. In VLA RL post-training, directly fine-tuning the pretrained policy may disrupt pretrained capabilities, while residual policies can provide limited exploration.

Setting	Family	Method	Exploration	Adaptive E-E	E-E Decoupling	Capability Preserv.
Generative RL	Q-weighted	QVPO	✓	$\triangle$	✗	✗
	Q-weighted	SDAC	✓	✓	✗	✗
	Critic-Guided	DIPO	$\triangle$	✗	✗	✗
		QSM	$\triangle$	✗	✗	✗
		DACER	✓	✓	✗	✗
		DIME	✓	✓	✗	✗

Table 1: Comparison of representative generative-policy RL methods. CFG-Policy is a plug-and-play module applicable to compatible generative RL backbones listed above. ✓: explicit support;

\triangle

: partial or implicit support; ✗: unsupported. E-E : Exploration-Exploitation

Figure 2: The sampling process of CFG-Policy. It composes two generative policies with different behavioral preferences in score space with guidance weight $w$ .

Method	HalfCheetah-v4	Walker2d-v4	Ant-v4	Humanoid-v4	Hopper-v4
QSM Original	$5020\pm 455$	$2041\pm 477$	$-24\pm 3$	$5922\pm 262$	$2753\pm 307$
QSM CFG-Policy	$6218\pm 533$	$2349\pm 583$	$1425\pm 308$	$6094\pm 169$	$3101\pm 461$
Improvement	$+23.9\%$	$+15.1\%$	$+6141.3\%$	$+2.90\%$	$+12.7\%$
QVPO Original	$11166\pm 5076$	$5220\pm 689$	$7129\pm 61$	$6612\pm 214$	$3727\pm 176$
QVPO CFG-Policy	$\mathbf{12990\pm 1993}$	$\mathbf{5692\pm 413}$	$\mathbf{7374\pm 45}$	$\mathbf{7041\pm 385}$	$\mathbf{3871\pm 192}$
Improvement	$+16.3\%$	$+9.0\%$	$+3.4\%$	$+6.49\%$	$+3.8\%$

Table 2: Results across five environments. Values are mean

\pm

sample standard deviation over 5 runs. The maximum value for each task is shown in bold.

Figure 3: Learning curve of QSM, QVPO before and after using CFG-Policy on five MuJoCo tasks. The x-axis is the environment steps and the y-axis shows average episodic return with one standard deviation shaded.

Model	Method	Adjust Bottle	Pick Dual Bottles	Rotate QR Code	Grab Roller
$\pi_{0}$	FlowSDE+PPO	82.81%	43.75%	34.38%	92.19%
$\pi_{0}$	CFG-Policy	98.44%	57.81%	42.19%	93.75%
$\pi_{0.5}$	FlowSDE+PPO	89.06%	12.50%	7.81%	100.00%
$\pi_{0.5}$	CFG-Policy	92.19%	15.62%	10.94%	100.00%

Table 3: Success rates of the original Flow-SDE + PPO and CFG-Policy.

Figure 4: Comparison of adaptive guidance with two fixed guidance coefficients. The dashed line corresponds to QVPO and the solid line corresponds to QSM.

Blend coefficient $a$	Return	Change vs. original QVPO
$0$	$-599\pm 171$	$-105.44\%$
$0.2$	$-362\pm 127$	$-103.29\%$
$0.5$	$883\pm 608$	$-91.98\%$
$0.8$	$4078\pm 1132$	$-62.96\%$
$0.98$	$10642\pm 4787$	$-3.33\%$
$1.0$	$11166\pm 5076$	–

Table 4: Action-space blending ablation on HalfCheetah-v4. Results are the mean and sample standard deviation across five seeds, with ten evaluation episodes per seed.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Evaluation environments. (a) HalfCheetah-v4, (b) Walker2d-v4, (c) Ant-v4, (d) Humanoid-v4, and (e) Hopper-v4 are MuJoCo locomotion tasks. (f) Adjust Bottle, (g) Pick Dual Bottles, (h) Rotate QR Code, and (i) Grab Roller are RoboTwin manipulation tasks.

Parameter	QVPO	QSM
Policy hidden layers	$3\times 256$ , Mish	$2\times 512$ , Mish
Time embedding	32-dim sinusoidal embedding	64-dimensional learnable Fourier embedding with a $2\times 128$ conditioning MLP
Critic	$3\times 256$ hidden layers and 25 quantiles	$2\times 512$ hidden layers
RND hidden dimensions	$128/128$	$128/128$
RND pre-training steps	$100$ k	$100$ k
Policy learning rate	$1\times 10^{-4}$	$3\times 10^{-4}$ initially, with cosine decay

Appendix

Table 5: Network, optimization, and diffusion settings for Online RL.

Network and optimization
Parameter	$\pi_{0}$	$\pi_{0.5}$
Visual inputs	3 images	3 images
Action dimension	14	14
Trainable actor	copied action expert	copied action expert
Critic input	action features (1024-D)	VLM features (2048-D)
Critic hidden widths	512, 256, 128	512, 256, 128

Appendix

Table 6: Network and flow-sampling hyperparameters for CFG post-training.

Algorithm	Policy	Training (s)	GPU memory (MiB)
QSM	Original	2.42	507
QSM	CFG-Policy	2.94	577
QVPO	Original	8.87	831
QVPO	CFG-Policy	9.37	867

Appendix

Table 7: Runtime comparison on HalfCheetah-v4.

Diffusion Policy

Diffusion policies generate multimodal action sequences by iteratively denoising noise, enabling imitation, planning, and control in robotics.

See all 207 papers

Sample-Efficient Diffusion-based Reinforcement Learning with Critic Guidance
May 28, 2026Shutong Ding, Zejia Zhong, Zhongyi Wang +4Diffusion Guidance Robotic RL
1ShanghaiTech University.
PPGuide: Steering Diffusion Policies with Performance Predictive Guidance
Mar 11, 2026Zixing Wang, Devesh K. Jha, Ahmed H. Qureshi +1Diffusion Guidance Diffusion Policy
Department of Computer Science at Purdue University, IN 47907, USA. · Contribution was conducted while the author was at Mitsubishi Electric Research Laboratories. · Mitsubishi Electric Research Laboratories, Cambridge, MA 02139 USA.
From Noise to Control: Parameterized Diffusion Policies
May 29, 2026Renhao Zhang, Haotian Fu, Mingxi Jia +3Robot Policy Learning Robot Policy Adaptation
University of Massachusetts · 2Brown University · 3Harvard

May 28, 2026J/K move · Enter open · S save

Sample-Efficient Diffusion-based Reinforcement Learning with Critic Guidance

Shutong Ding, Zejia Zhong, Zhongyi Wang, Ke Hu+3

1ShanghaiTech University.

Diffusion Guidance Robotic RL Diffusion Policy RL for Diffusion Models Sample-Efficient RL

Open paper AskPDF arXiv

Recent advances in reinforcement learning (RL) have achieved great successes by leveraging the multimodality and exploration capability of diffusion policies. Among these approaches, one representative branch focuses on the sampling-based policy optimization. This design enables better exploration capability of the diffusion model, particularly at the beginning of training, but suffer from low exploitation in Q-value information, resulting in a slow policy convergence. Another branch pays attention to gradient-based policy optimization, which sufficiently exploits the gradient of the Q function yet tends to collapse into a unimodal policy with low diversity. To address this issue, we propose CGPO, Critic-Guided diffusion Policy Optimization, which effectively balances exploration and exploitation with the training-free guidance technique integrated into the denoising process of diffusion policy. Concretely, CGPO steers action generation toward high-value regions defined by the critic network and uses the guided actions as regression objectives. In this manner, CGPO reduces the time required to obtain high-quality actions and improves final performance with better balance between the exploration-exploitation tradeoff. We validate the effectiveness of CGPO on 5 MuJoCo locomotion tasks, and CGPO achieves state-of-the-art performance compared with existing diffusion-based RL methods. Notably, CGPO is the first success to incorporate diffusion policy into real-world RL, with its superior performance on Franka robot arm grasping tasks. Our official page is released at https://dingsht.tech/cgpo-webpage.

Abstract

Diffusion Policy

Explore similar work

Sample-Efficient Diffusion-based Reinforcement Learning with Critic Guidance

PPGuide: Steering Diffusion Policies with Performance Predictive Guidance

From Noise to Control: Parameterized Diffusion Policies