REFINEPPO: Learning Continuous Control Policies by Iterative Action Refinement

Sachini Weerasekara; Sagar Kamarthi; Jacqueline Isaacs

cs.LGSep 17, 2026

REFINEPPO: Learning Continuous Control Policies by Iterative Action Refinement

Authors: Sachini Weerasekara, Sagar Kamarthi, Jacqueline Isaacs

Organizations: Northeastern University, Boston, MA, USA.

Abstract

Deep reinforcement learning (DRL) has achieved strong performance across a wide range of continuous-control problems. These continuous-control policies, however, are often defined as direct mappings from an observed state to an action or action distribution, requiring a single feed-forward network to construct an optimal control decision in one pass. While effective, this formulation leaves little opportunity for the policy to reconsider or progressively improve an action once an initial prediction has been formed. In this work, we explore an alternative approach: rather than learning only to directly predict an action, can a policy learn to iteratively improve one, and can this iterative process provide advantages during policy learning? We introduce Iterative Action Refinement (IAR), an iterative action-construction method that constructs control actions through a sequence of learned residual corrections. Starting from an initial proposal, a shared refinement network repeatedly conditions on the observed state and the current action proposal, allowing each refinement step to revise the action constructed by preceding steps. The final refined proposal is then used to determine the action executed by the agent. We integrate this iterative action-construction mechanism with Proximal Policy Optimization (PPO), yielding REFINEPPO. We evaluate REFINEPPO across 14 benchmark control tasks, complemented by controlled ablations of refinement depth and update schedules and analyses aimed at understanding why iterative refinement is effective. Across these environments, REFINEPPO matches or exceeds the performance of standard PPO while demonstrating faster convergence on several tasks.

Figures & tables

Fig. 1 : Overview of RefinePPO . Given the current environment state $s_{t}$ , the policy initializes an action proposal $m_{0}=\mathbf{0}$ and progressively refines it for $K$ steps using the state-conditioned residual network $f_{\theta}$ . The state $s_{t}$ remains fixed throughout refinement, while each update conditions on the current action proposal $m_{k}$ . The final proposal $m_{K}$ defines the mean of the Gaussian policy, from which a single action $a_{t}$ is sampled and executed in the environment, producing the next state $s_{t+1}$ . Intermediate proposals are latent computations and are never executed in the environment.

Environment	Dim.	PPO	RefinePPO	Improv. (%)	Environment	Dim.	PPO	RefinePPO	Improv. (%)
Ant-v5	8	$4101\pm 681$	$\mathbf{4370\pm 472}$	$\mathbf{+6.6\%}$	Pendulum-v1	1	$\mathbf{-770\pm 445}$	$-833\pm 73$	$-8.2\%$
HalfCheetah-v5	6	$3034\pm 1803$	$\mathbf{5686\pm 2640}$	$\mathbf{+87.4\%}$	MountainCarContinuous-v0	1	$93\pm 0.2$	$\mathbf{94\pm 0.1}$	$\mathbf{+1.1\%}$
Hopper-v5	3	$2720\pm 826$	$\mathbf{3120\pm 287}$	$\mathbf{+14.7\%}$	LunarLanderContinuous-v3	2	$269\pm 20$	$\mathbf{281\pm 8}$	$\mathbf{+4.5\%}$
Walker2d-v5	6	$5042\pm 929$	$\mathbf{5810\pm 354}$	$\mathbf{+15.2\%}$	BipedalWalker-v3	4	$289\pm 4.2$	$\mathbf{291\pm 4}$	$\mathbf{+0.7\%}$
Swimmer-v5	2	$123\pm 8$	$\mathbf{197\pm 134}$	$\mathbf{+60.2\%}$	BipedalWalkerHardcore-v3	4	$-28\pm 47$	$\mathbf{-4\pm 59}$	$\mathbf{+85.7\%}$
Pusher-v5	7	$-44.0\pm 10$	$\mathbf{-37\pm 9}$	$\mathbf{+15.9\%}$	HumanoidStandup-v5	17	$148902\pm 7799$	$\mathbf{165067\pm 32322}$	$\mathbf{+10.9\%}$

TABLE I : Evaluation performance of PPO and RefinePPO across 14 continuous-control environments. Results report the mean return

\pm

standard deviation for the best-performing hyperparameter configuration of each method.

Panel of Figure 3 — Fig. 2 : Evaluation returns throughout training for PPO and RefinePPO across six representative environments. Solid lines show the mean evaluation return across five independent seeds, and shaded regions indicate variability across seeds.

Panel of Figure 4 — Fig. 3 : Empirical evidence for progressive action construction in a trained IAR policy on Humanoid Standup. (a) Some action components require substantially less subsequent refinement than others. (b) Intermediate values of earlier-established components provide information about subsequent corrections to components that continue to be refined. Together, the results are consistent with progressive, cross-component action refinement.

Fig. 4 : Refinement depth

K

effect on normalized return across environments. Error bars indicate

\pm 1

standard deviation.

Environment	Average	Uniform	Inverse
Ant-v5	$\mathbf{4370.4}$	$3298.8$	$2892.9$
HalfCheetah-v5	$\mathbf{5685.9}$	$1361.2$	$1713.1$
Hopper-v5	$\mathbf{3120.4}$	$2897.0$	$2708.5$
Humanoid-v5	$\mathbf{3060.5}$	$1345.5$	$1024.5$
HumanoidStandup-v5	$\mathbf{165066.3}$	$149432.5$	$147769.1$
Pusher-v5	$\mathbf{-37.8}$	$-58.4$	$-44.5$

TABLE II : Effect of refinement schedule on RefinePPO performance across MuJoCo continuous-control environments. Results report the mean evaluation return for the Average, Uniform, and Inverse refinement schedules.

RL Control

Reinforcement learning control uses reward-driven learning to make decisions in dynamic systems, from robots and vehicles to energy and infrastructure.

See all 276 papers

Aug 5, 2026

Learning When to Stop: Prefix-Optimal Dynamic Diffusion Policies for Continuous Control

Diffusion policies are a powerful policy class for continuous control, but their iterative denoising process creates a substantial computational bottleneck. Reducing this cost requires adapting the number of denoising steps to the difficulty of each action while preserving task performance. We introduce Prefix-Optimal Generative Policies (POGP), a framework that learns a prefix value function at every intermediate denoising step through a Bellman-style recursion over the denoising chain. The prefix value function serves two purposes: it provides an auxiliary training objective that encourages intermediate outputs to become high-quality actions, and it enables a test-time stopping rule that terminates denoising when additional steps are unlikely to produce meaningful improvement. Across four MuJoCo environments and comparisons with 12 baselines, POGP reduces the required number of denoising iterations by approximately 2.7-fold while retaining near-full task performance. Compared with state-of-the-art dynamic diffusion baselines, prefix training also improves final task performance by approximately 3.5%. These results indicate that supervising intermediate denoising steps is useful not only for adaptive early stopping, but also as an auxiliary objective that improves the learned policy.

Rohit Kumar Salla, Manoj Saravanan, Simon Stepputtis

Virginia Polytechnic Institute and State University (Virginia Tech), Blacksburg, Virginia, USA

Diffusion Policy Continuous Control RL for Diffusion Models Policy Learning Diffusion Models

Ask PDF

Apr 4, 2026

Drift-Based Policy Optimization: Native One-Step Policy Learning for Online Robot Control

Diffusion policies effectively model multimodal action distributions for robotic manipulation, but their iterative denoising requires tens to hundreds of network function evaluations (NFEs) for each control prediction, limiting their applicability to high-frequency closed-loop control and online reinforcement learning (RL). We present a two-stage framework for native one-step generative policies that transfers iterative refinement from inference to training. First, Drift-Based Policy (DBP) uses a fixed-point drifting objective to internalize corrective dynamics into the model parameters, producing multimodal action chunks with a single network evaluation by design. Second, Drift-Based Policy Optimization (DBPO) augments the pretrained backbone with a stochastic interface that provides exact conditional rollout likelihoods for PPO-style on-policy updates while preserving 1-NFE deployment. On the 12-task Diffusion Policy suite, DBP improves the average success rate from 0.79 to 0.83 while reducing inference from 100 NFEs to 1. Across 37 point-cloud manipulation tasks, DBP achieves an average success rate of 88.4%, surpassing the leading 1-NFE baseline OMP at 82.3%. DBPO further improves pretrained one-step policies through stable online fine-tuning on RoboMimic and D4RL. On a physical dual-arm UR5 platform, DBP achieves 123/150 successes (82%) with an average end-to-end latency of 9.5 ms, compared with MP1's 89/150 successes (59%) under the same setup. Code is available at https://github.com/YuxuanGao0822/DBPO.

Yuxuan Gao, Yedong Shen, Shiqi Zhang +6

University of Science and Technology of China, Hefei 230026, China. · LYNSENSE, Shanghai, China. · iFLYTEK, Hefei 230088, China.

Diffusion Policy Robotic RL Robotic Manipulation

Ask PDF

Mar 2, 2026

State-Action Inpainting Diffuser for Continuous Control with Delay

Signal delay poses a fundamental challenge in continuous control and reinforcement learning (RL) by introducing a temporal gap between interaction and perception. Current solutions have largely evolved along two distinct paradigms: model-free approaches which utilize state augmentation to preserve Markovian properties, and model-based methods which focus on inferring latent beliefs via dynamics modeling. In this paper, we bridge these perspectives by introducing State-Action Inpainting Diffuser (SAID), a framework that integrates the inductive bias of dynamics learning with the direct decision-making capability of policy optimization. By formulating the problem as a joint sequence inpainting task, SAID implicitly captures environmental dynamics while directly generating consistent plans, effectively operating at the intersection of model-based and model-free paradigms. Crucially, this generative formulation allows SAID to be seamlessly applied to both online and offline RL. Extensive experiments on delayed continuous control benchmarks demonstrate that SAID achieves state-of-the-art and robust performance. Our study suggests a new methodology to advance the field of RL with delay.

Dongqi Han, Wei Wang, Enze Zhang +1

Microsoft Research Asia · University of Pennsylvania · Shanghai Jiaotong University

Delayed RL Reinforcement Learning RL Control Model-Based RL Model-Free RL

Ask PDF

Abstract

RL Control

Explore similar work

Learning When to Stop: Prefix-Optimal Dynamic Diffusion Policies for Continuous Control

Drift-Based Policy Optimization: Native One-Step Policy Learning for Online Robot Control

State-Action Inpainting Diffuser for Continuous Control with Delay