DR-APO: Decoupled-Representation Advantage-Guided Policy Optimization from Continuous Control to Embodied Navigation

Runhua Zhang; Yingjie Niu; Chao Li

cs.LGOct 12, 2026

DR-APO: Decoupled-Representation Advantage-Guided Policy Optimization from Continuous Control to Embodied Navigation

Authors: Runhua Zhang, Yingjie Niu, Chao Li

Organizations: DEEP Robotics, Hangzhou, China · Zhejiang University, Hangzhou, China

Abstract

Policy representations in reinforcement learning have expanded beyond Gaussian policies to diffusion and flow-matching based generative policies. However, existing RL methods remain representation-specific, lacking a unified optimization interface for emerging generative policies. In particular, extending likelihood-based objectives to multi-step generative policies couples optimization to their internal generation process, where accumulated likelihood ratios can weaken the update signal. Meanwhile, generative-policy-specific methods often rely on multi-candidate generation and critic-based selection, improving overall return while providing limited improvement to the policy itself. We introduce DR-APO (Decoupled-Representation Advantage-Guided Policy Optimization), which decouples policy improvement from policy representation through advantage-guided target improvement and representation-compatible direct action fitting, providing a shared optimization principle for Gaussian, diffusion, and flow policies. Across five MuJoCo-v5 tasks, DR-APO remains competitive with SAC for Gaussian policies and achieves the best aggregate policy-only performance for both diffusion and flow-matching policies. NavDP post-training improves the Success Rate (SR) from 70.33% to 89.55% in simulation and from 25% to 65% in real-world navigation. These results demonstrate a unified policy-improvement principle across Gaussian, diffusion, and flow policies, from continuous-control learning to embodied-policy post-training.

Figures & tables

Fig. 1: Overview of DR-APO. Advantage-guided target improvement and representation-compatible direct action fitting enable unified optimization of Gaussian, diffusion, and flow policies across continuous control and embodied visual navigation.

Fig. 2: Overview of DR-APO. Policy improvement is decoupled from policy representation through advantage-guided target improvement and representation-compatible direct action fitting across Gaussian, diffusion, and flow policies.

Fig. 3: Learning curves of Gaussian, diffusion, and flow policies on five MuJoCo-v5 tasks. Curves and shaded regions denote the mean and standard deviation.

Policy	Method	HalfCheetah		Hopper		Walker2d		Swimmer		Humanoid		Avg. rank $\downarrow$
Policy	Method	Final Return	AUC@1M	Final Return	AUC@1M	Final Return	AUC@1M	Final Return	AUC@1M	Final Return	AUC@1M	Final	AUC
Gaussian	PPO	3250.1	3057.5	3022.9	2484.4	3652.1	2547.9	68.4	53.4	541.0	493.2	3.00	2.80
	SAC	6538.1	5621.8	3158.3	2420.7	4343.8	3181.8	69.8	58.0	5137.5	3951.5	1.40	1.80
	DR-APO	5154.5	4574.1	3288.9	2692.3	4243.2	3202.6	128.7	106.5	2363.9	1945.0	1.60	1.40
Diffusion	AWR	135.5	205.7	144.9	143.9	220.2	228.1	19.2	17.3	640.6	509.2	3.60	3.60
	FPO	-587.5	-344.9	40.7	119.4	-2.0	-0.2	2.8	11.1	201.9	197.1	5.00	5.00

TABLE I: Policy-only results on five MuJoCo-v5 tasks. We report Final Return, AUC@1M, and within-family average ranks.

	Diffusion Policy			Flow Policy
Env.	QVPO [-1.5pt] P $\rightarrow$ B64	LSEVPO [-1.5pt] P $\rightarrow$ B64	DR-APO [-1.5pt] Direct	RFM [-1.5pt] P $\rightarrow$ B32	DR-APO [-1.5pt] Direct
HC	2981.3 $\!\rightarrow\!$ 8047.0	1924.6 $\!\rightarrow\!$ 8591.7	4572.7	46.0 $\!\rightarrow\!$ 9862.7	4299.2
Hop	324.4 $\!\rightarrow\!$ 2792.6	497.8 $\!\rightarrow\!$ 2506.8	2631.5	806.1 $\!\rightarrow\!$ 1839.8	3016.4
W2	3913.3 $\!\rightarrow\!$ 3130.7	2521.9 $\!\rightarrow\!$ 3316.4	3853.6	48.4 $\!\rightarrow\!$ 3211.1	3650.0
Swi	88.9 $\!\rightarrow\!$ 113.3	83.0 $\!\rightarrow\!$ 112.1	124.9	72.4 $\!\rightarrow\!$ 109.4	124.2
Hum	292.1 $\!\rightarrow\!$ 2191.2	439.4 $\!\rightarrow\!$ 1797.9	1462.0	104.1 $\!\rightarrow\!$ 1470.7	1535.4

TABLE II: Direct policy execution versus Best-of-

N

critic selection.

(a) Target Improvement
Policy	Variant	HC	Hop	W2	Swi	Hum
Gaussian	Full	5155 $\pm$ 317	3289 $\pm$ 263	4243 $\pm$ 361	128.7 $\pm$ 8.7	2364 $\pm$ 423
	w/o Ref.	1531 $\pm$ 263	1071 $\pm$ 127	918 $\pm$ 93	61.7 $\pm$ 7.8	887 $\pm$ 157
	w/o Weight.	3707 $\pm$ 271	2361 $\pm$ 216	2238 $\pm$ 319	90.1 $\pm$ 9.7	1857 $\pm$ 383
Diffusion	Full	4573 $\pm$ 391	2632 $\pm$ 337	3854 $\pm$ 431	124.9 $\pm$ 9.8	1462 $\pm$ 263
	w/o Ref.	1784 $\pm$ 227	1289 $\pm$ 301	1930 $\pm$ 349	52.1 $\pm$ 7.7	673 $\pm$ 91

TABLE III: Ablation studies on target improvement and direct action fitting. Final Return is reported as mean

\pm

SD over five seeds.

Method	Overall		Commercial		Home
	SR	SPL	SR	SPL	SR	SPL
iPlanner	45.28	43.54	51.75	50.02	38.80	37.05
ViPlanner	45.08	44.00	55.40	53.94	34.75	34.05
NavDP	70.33	66.85	72.55	69.74	68.10	63.96
NavOL	67.00	63.00	75.20	71.10	58.80	54.90
SIDP	77.43	69.32	82.15	74.16	72.70	64.48

TABLE IV: Closed-loop wheeled-robot navigation performance on InternScenes. SR denotes Success Rate and SPL denotes Success weighted by Path Length.

Fig. 4: Qualitative comparison of pretrained NavDP and DR-APO post-training in simulated and real-world navigation.

Method	NavDP (pretrained)	DR-APO
SR (%)	25	65

TABLE V: Real-world navigation success rate (SR) across 20 scenarios.

Policy Optimization

Policy optimization develops algorithms for improving decision rules in reinforcement learning, language models, and sequential decision systems.

See all 221 papers

Doob-Based Generative Policy Optimization
Oct 20, 2026Maojiang Su, Qijie Zhu, Yu Xiao +4Policy Optimization RL for Diffusion Models
Department of Computer Science, Northwestern University · Center for Foundation Models and Generative AI, Northwestern University · Department of Statistics and Data Science, Northwestern University +1
Drift-Based Policy Optimization: Native One-Step Policy Learning for Online Robot Control
Apr 4, 2026Yuxuan Gao, Yedong Shen, Shiqi Zhang +6Diffusion Policy Robotic RL
University of Science and Technology of China, Hefei 230026, China. · LYNSENSE, Shanghai, China. · iFLYTEK, Hefei 230088, China.
Mean Flow Policy Optimization
Apr 16, 2026Xiaoyi Dong, Xi Sheryl Zhang, Jian ChengMeanFlow Mean Flow
C2DL, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · AiRiA +1

Oct 20, 2026J/K move · Enter open · S save

Doob-Based Generative Policy Optimization

Maojiang Su, Qijie Zhu, Yu Xiao, Shuyang Yu+3

Department of Computer Science, Northwestern University · Center for Foundation Models and Generative AI, Northwestern University · Department of Statistics and Data Science, Northwestern University+1

Policy Optimization RL for Diffusion Models RL for Flow Matching Models Robotic Control Doob H-Transform

Open paper AskPDF arXiv

Diffusion and flow models provide expressive policy classes for robotic and sequential decision-making systems. However, their action likelihoods are generally intractable, limiting the direct use of widely adopted policy optimization methods such as PPO and GRPO. We propose Doob Policy Optimization (DoobPO), a framework that leverages Doob's

h

-transform for likelihood-free reinforcement learning fine-tuning of generative policies. DoobPO separates policy improvement from generative policy learning. It first learns a lightweight ratio model that represents the desired policy update, and then distills the resulting ratio-tilted target policy into the generative policy via conditional drift matching. This formulation applies uniformly to both diffusion and flow policies and supports a broad family of policy improvement rules, including PPO and GRPO. We prove that the practical conditional drift matching objective is gradient-equivalent to the ideal Doob drift matching objective. We further show that the induced diffusion process solves an endpoint-constrained stochastic optimal control problem. This establishes that it achieves the minimum-energy modification required to realize the desired policy tilt. Across MuJoCo continuous-control benchmarks, DoobPO achieves strong performance against competitive diffusion- and flow-policy baselines, showing a principled and practical route to RL fine-tuning for generative policies.

Abstract

Policy Optimization

Explore similar work

Doob-Based Generative Policy Optimization

Drift-Based Policy Optimization: Native One-Step Policy Learning for Online Robot Control

Mean Flow Policy Optimization