Posterior Optimization with Clipped Objective for Bridging Efficiency and Stability in Generative Policy Learning

Yuhui Chen; Haoran Li; Zhennan Jiang; Yuxing Qin; Yuxuan Wan; Weiheng Liu; Dongbin Zhao

cs.ROApr 2, 2026

Posterior Optimization with Clipped Objective for Bridging Efficiency and Stability in Generative Policy Learning

Authors: Yuhui Chen, Haoran Li, Zhennan Jiang, Yuxing Qin, Yuxuan Wan, Weiheng Liu, Dongbin Zhao

Organizations: Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · CFCS, School of Computer Science, Peking University, Beijing 100871

Abstract

Expressive generative models have advanced robotic manipulation by capturing complex, multi-modal action distributions over temporally extended trajectories. However, fine-tuning these policies via RL remains challenging due to instability and sample inefficiency. We introduce Posterior Optimization with Clipped Objective (POCO), a principled RL framework that formulates policy improvement as a posterior inference problem tailored for temporal action chunks. Through an Expectation-Maximization procedure, POCO distills a reward-weighted implicit posterior into the policy without likelihood estimation. Furthermore, POCO adopts an offline-to-online paradigm that anchors online exploration to pre-trained priors, and its model-agnostic design scales to fine-tune large VLA models without architectural modifications. Evaluations across 7 simulation benchmarks and 4 contact-rich real-world tasks demonstrate that POCO prevents catastrophic policy collapse, outperforms SOTA baselines, and achieves a 96.7% success rate on real-world tasks. Videos are available at our project website https://cccedric.github.io/poco/.

Figures & tables

Fig. 1: Conceptual overview comparing typical RL paradigms against POCO. (Left) Off-policy methods enable efficient data reuse, while on-policy methods ensure stability. (Right) Schematic performance curves show that POCO is designed for sample-efficient, stable improvement.

Fig. 2: Overview of our proposed framework. The learning paradigm consists of two main stages. (Left) First, the policy is pre-trained through supervised learning using expert demonstrations collected via tele-operation. (Right) Second, during online fine-tuning, the policy interacts with the environment and improves through an iterative E-M procedure. In the Implicit E-step, multiple candidate actions are sampled from the current actor and evaluated by the critic to obtain importance weights. In the M-step, these weighted samples are utilized to compute the POCO loss, updating the parameters to obtain the new actor.

Fig. 3: Visualization of all simulation tasks. The simulation tasks includes: a) scene , b) puzzle-3x3 , c) cube-double , d) cube-triple from OGBench [ 50 ] and e) lift , f) can , g) square from RoboMimic [ 51 ] .

Tasks		Num. of Trajs.	Num. of Trans.	Action Dimension
Simulation	scene	1,000	1,000,000	5
	puzzle-3x3	1,000	1,000,000	5
	cube-double	1,000	1,000,000	5
	cube-triple	3,000	3,000,000	5
	lift	300	31,127	7
	can	300	62,756	7

TABLE I: Metadata for offline datasets in simulation and real-world tasks.

Fig. 4: Visualization of the real-world environment platform. During execution, an operator presses the green button to assign a sparse reward upon success, or the red button to immediately terminate dangerous trajectories for hardware safety.

Fig. 5: Visualization of all real-world tasks. The real-world tasks includes: a) Pick Cube , b) Route Cable , c) Insert USB , d) Assemble SSD , e) Pick Pen . The camera perspectives used in the experiments differ from those used for visualization.

Fig. 6: Learning curves during online training for 7 simulation tasks. Results are averaged over 50 trials per task at each evaluation step with 5 random seeds.

Parameter	Scene	Puzzle 3x3	Cube-double	Cube-triple	Lift	Can	Square
Batch size	256	256	256	256	256	256	256
Learning rate	3e-4	3e-4	3e-4	3e-4	3e-4	3e-4	3e-4
$\gamma$	0.99	0.99	0.99	0.99	0.99	0.99	0.99
$T$	5	5	5	5	5	5	5
Critic warmup step	5,000	5,000	5,000	5,000	5,000	5,000	5,000
$N$	32	32	32	32	32	32	32

TABLE II: Policy training details for online simulation experiments.

Fig. 7: Learning curves during offline-to-online training for 4 simulation tasks. Results are averaged over 50 trials per task at each evaluation step with 5 random seeds.

	Simulation Tasks				Real-world Tasks
Parameter	Scene	Puzzle 3x3	Can	Square	Pick Cube	Route Cable	Insert USB	Assemble SSD
Batch size	256	256	256	256	256	256	256	256
Learning rate	3e-4	3e-4	3e-4	3e-4	3e-4	3e-4	3e-4	3e-4
$\gamma$	0.99	0.99	0.99	0.99	0.99	0.99	0.99	0.99
$H$	750	500	300	400	200	150	200	180
$T$	5	5	5	5	4	4	4	4

TABLE III: Policy training details for offline-to-online experiments.

Fig. 8: Learning curves during offline-to-online training for 4 real-world tasks. Results are displayed as a running average over 20 episodes.

Task	BC	QC	DSRL	POCO
Pick Cube	63.3%	66.7%	93.3%	100.0%
Route Cable	73.3%	70.0%	80.0%	100.0%
Insert USB	46.7%	70.0%	76.7%	90.0%
Assemble SSD	26.7%	36.7%	73.3%	96.7%
Average	52.5%	60.9%	80.8%	96.7%

TABLE IV: Success Rates of offline-to-online training for real-world tasks. Results are reported over 30 trials per task.

Fig. 9: Learning curves during offline-to-online training with different $\zeta$ for 2 simulation tasks. Results are averaged over 50 trials per task at each evaluation step with 5 random seeds.

Fig. 10: Learning curves during offline-to-online training with different $\beta$ for 2 simulation tasks. Results are averaged over 50 trials per task at each evaluation step with 5 random seeds.

Parameter	Pick Pen	Hang Keychain
Batch size	64	64
Learning rate for actor	5e-5	5e-5
Learning rate for critic	3e-4	3e-4
$\gamma$	0.99	0.99
$H$	200	200
$T$	10	10

TABLE V: Policy training details for VLA scaling real-world experiments.

Base Model	$\pi_{0.5}$		GR00T N1.6
Task	SFT	POCO 1	SFT	POCO 1
Pick Pen	76.7%	93.3%	63.3%	86.7%
Hang Keychain	60.0%	86.7%	53.3%	83.3%

TABLE VI: Success Rates of offline-to-online training for 2 real-world tasks with VLA models. Results are reported over 30 trials per task.

Policy Optimization

Policy optimization develops algorithms for improving decision rules in reinforcement learning, language models, and sequential decision systems.

See all 224 papers

OGPO: Sample Efficient Full-Finetuning of Generative Control Policies
May 4, 2026Sarvesh Patil, Mitsuhiko Nakamoto, Manan Agarwal +14RL for Robotics RL for Generative Models
Carnegie Mellon University · University of California, Berkeley · University of Washington +1
RoboFFT: Finetuning generative robot policy via online reinforcement learning with forward process
Sep 26, 2026Yu Li, Shenghe Hu, Yuhan Wang +4Robot Policy Learning RL for Robotics
Institute for Artificial Intelligence, Peking University · Zhongguancun Academy · PKU-PsiBot Joint Lab +1
GenPO++: Generative Policy Optimization with Jacobian-free Likelihood Ratios
Jun 5, 2026Ke Hu, Shutong Ding, Panxin Tao +2RL Control Policy Gradient Methods
ShanghaiTech University

May 4, 2026J/K move · Enter open · S save

OGPO: Sample Efficient Full-Finetuning of Generative Control Policies

Sarvesh Patil, Mitsuhiko Nakamoto, Manan Agarwal, Shashwat Saxena+13

Carnegie Mellon University · University of California, Berkeley · University of Washington+1

RL for Robotics RL for Generative Models Residual Policy Learning RL Fine-Tuning Off-Policy RL

Open paper AskPDF arXiv

Generative control policies (GCPs), such as diffusion- and flow-based control policies, have emerged as effective parameterizations for robot learning. This work introduces Off-policy Generative Policy Optimization (OGPO), a sample-efficient algorithm for finetuning GCPs that maintains off-policy critic networks to maximize data reuse and propagate policy gradients through the full generative process of the policy via a modified PPO objective, using critics as the terminal reward. OGPO achieves state-of-the-art performance on manipulation tasks spanning multi-task settings, high-precision insertion, and dexterous control. To our knowledge, it is also the only method that can fine-tune poorly-initialized behavior cloning policies to near full task-success with no expert data in the online replay buffer, and does so with few task-specific hyperparameter tuning. Through extensive empirical investigations, we demonstrate that OGPO drastically outperforms methods alternatives on policy steering and learning residual corrections, and identify the key mechanisms behind its performance. We further introduce practical stabilization tricks, including success-buffer regularization, two-sided conservative advantages, and Q-variance reduction, to mitigate critic over-exploitation across state- and pixel-based settings. Beyond proposing OGPO, we conduct a systematic empirical study of GCP finetuning, identifying the stabilizing mechanisms and failure modes that govern successful off-policy full-policy improvement.

Abstract

Policy Optimization

Explore similar work

OGPO: Sample Efficient Full-Finetuning of Generative Control Policies

RoboFFT: Finetuning generative robot policy via online reinforcement learning with forward process

GenPO++: Generative Policy Optimization with Jacobian-free Likelihood Ratios