CANDO: Conservative Action-Noise Directional Optimization for Residual Reinforcement Learning
Authors: Zeyu Gao, Chengkai Hou, Chaoyue Li, Zhuangzhe Wu, Jiaming Liu, Yao Mu, Peng Jia, Li Zhang, Shanghang Zhang
Organizations: School of Data Science, Fudan University, Shanghai 200433, China · Shanghai Innovation Institute, Shanghai 200233, China · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University, Beijing 100871, China · Simplexity Robotics, Beijing 100080, China · School of Computer Science, Shanghai Jiao Tong University, Shanghai 200240, China
Pretrained generative policies provide strong behavior priors for robot control, but adapting them efficiently with online reinforcement learning remains difficult. A central challenge is how to improve a frozen generative policy without destroying the structure learned from offline demonstrations. Noise-space residual methods address this by modifying the policy's input noise, but existing noise-space methods typically regulate adaptation through fixed latent feasible regions, which may exclude valuable noise edits. In contrast, action-space residual methods are more sample-efficient, but their efficiency can still be further improved. We propose Conservative Noise Directional Optimization (CNDO), a noise-space residual RL method that improves a frozen generative policy by learning state- and noise-conditioned directions around Gaussian latent samples. CNDO removes predefined latent feasible regions and instead regulates noise-space adaptation through the soft conservative constraint, which consists of conservative value estimation and gated execution. We further introduce Conservative Action-Noise Directional Optimization (CANDO), which adds an action-space residual, combining structured latent-space improvement with local action correction. Experiments show that, once CNDO narrows the gap between noise-space and action-space residual learning, CANDO further improves over single-space optimization, achieving 76.1 NAUC and a 91.4% final success rate, surpassing the best single-space results by 6.6 NAUC and 4.1% success rate. More demos and info are available on https://cando-rl.github.io/.
Figures & tables
Fig. 1: Motivation for residual learning in noise and action spaces. (a) Noise-space residual RL edits the input noise before frozen-policy decoding, while action-space residual RL corrects decoded actions. (b) Existing noise-space methods show a clear sample-efficiency gap relative to DICE. (c) Naively combining DSRL and DICE provides no additional improvement.
Fig. 2: Comparison of noise-space regulation strategies. (a) DSRL bounds latent variables with tanh squashing. (b) LPS projects latents onto the Gaussian hypersphere. (c) LP-DS constrains residual noise with clipping and a trust region. (d) CNDO removes fixed geometric constraints and instead uses conservative value estimation and gated execution.
Fig. 3: Overview of CANDO. During online inference, a noise direction modifies the frozen-policy output, the gate selects whether to use the edit, and an action residual further refines the action. During off-policy learning, the critic is trained conservatively, the gate is supervised by the target critic, and both residual branches are jointly optimized. The pretrained policy remains frozen throughout.
Fig. 4: Noise-space policy improvement on five tasks. Lines and shaded regions show the mean and sample standard deviation over three seeds; the gray dashed line is the frozen base policy.
Fig. 5: CNDO component analysis on five tasks. CNDO “w/o gated execution” always executes the full latent edit, and “w/o conservative critic” sets the behavior-support conservative loss weight to zero. Lines and shaded regions denote the mean and sample standard deviation over three seeds; the gray dashed line is the frozen base policy.
Method
Space
Can
Square
Transport
PegInsertionSide
StackCube
Average
NAUC
Final SR
NAUC
Final SR
NAUC
Final SR
NAUC
Final SR
NAUC
Final SR
NAUC
Final SR
Base policy
Base
/
87.4
/
57.8
/
54.0
/
49.7
/
40.0
/
57.8
DSRL [ 5 ]
Noise
90.6
95.8
55.7
66.1
58.5
75.7
46.4
47.9
38.5
45.4
57.9
66.2
LPS [ 8 ]
Noise
91.4
90.0
58.0
69.4
22.5
55.4
9.1
1.4
27.3
34.8
41.7
50.2
LP-DS [ 9 ]
Noise
93.0
94.4
57.5
59.4
55.2
55.0
52.5
51.8
41.2
41.9
59.9
60.5
CNDO
Noise
91.6
96.0
68.1
80.6
71.2
87.7
65.8
83.1
50.9
60.1
69.5
81.5
TABLE I: Main comparison results. For each task, NAUC and Final SR are reported in separate columns, in percent, averaged over three seeds. Bold and underlined values denote the best and second-best results, respectively.
Fig. 6: Joint noise-action residual learning on five tasks. Lines and shaded regions denote the mean and sample standard deviation over three seeds; the gray dashed line is the frozen base policy.
Fig. 7: Real-world snapshots of our platform and the execution process of tasks.
Fig. 8: Success rates on real-world tasks. We report the success rates measured over 20 evaluation trials for each task.
Pretrained generative robot policies learn expressive action priors from demonstrations. However, existing reinforcement learning methods only steer the noisy space but fail to modulate intermediate action representations during the generation process, resulting in performance degradation and inefficiency. To address this limitation, we propose a novel Dual-Latent Space Reinforcement Learning (DLSRL) framework, which complements initial-noise steering with representation-level control inside the frozen generator. Specifically, our actor network predicts two distinct latent variables: an initial-noise latent variable that steers behavior generation, and an action-representation latent variable for intermediate feature modulation. Moreover, this representation latent variable is mapped to adapter features and ingeniously injected into the hidden states of intermediate action tokens via residual connections. Our dual-control design enables direct adjustment of action representations without updating the base policy. Experiments across generative policy architectures and robotic manipulation tasks show that DLSRL effectively accelerates online robot policy adaptation and achieves competitive performance. Our code is available at https://github.com/xianchaoxiu/DLSRL.
Pengfei Zhang, Teng Sun, Xianchao Xiu
School of Mechatronic Engineering and Automation, Shanghai University, Shanghai 200444, China
Pretrained vision-language-action (VLA) models encode broad manipulation priors, yet adapting them to specific tasks requires balancing limited robot interaction, computational resources, and safety constraints. Noise space learning (NSL) adapts flow-matching-based VLAs by learning a context-conditioned distribution over noise while keeping the pretrained VLA fixed. Building on prior noise-steering methods, we formulate NSL as an interface for VLA adaptation that accommodates different learning algorithms and instantiate it with PPO, SAC, and AWR. Theoretically, we provide an information-theoretic analysis of how NSL enables efficient adaptation. Empirically, we study these NSL algorithms on LIBERO, RoboTwin, and four real-robot manipulation tasks under interaction, computation, and execution constraints. Across these settings, we develop a suite of effective NSL variants and derive practical recommendations for choosing among them under different resource and deployment regimes.
Junjie Lu, Chuheng Zhang, Yucen Wang +4
University of Technology Sydney · Microsoft Research · Nanjing University +1
Diffusion policies offer a powerful and expressive parameterization for continuous control. Yet, their integration with reinforcement learning remains conceptually and algorithmically challenging. In this work, we address this gap by introducing a noisy-space action-value (Q-)function that assigns values to diffusion latents through the distribution of executed actions induced by the denoising process. We show that this construction admits a precise semantic interpretation and derive a noisy-space policy gradient (NSPG) that optimizes noisy latents using only clean action-space value estimates. Building on this result, we formulate a KL-regularized policy improvement over noisy latents and show that the resulting objective admits a diffusion-compatible regression form, avoiding backpropagation through the denoising process. Empirical results on state-based D4RL benchmarks and vision-based OGBench tasks demonstrate that the proposed noisy-space objective provides a principled and effective basis for training diffusion policies in offline reinforcement learning. Project webpage: https://mahmoud-selim.github.io/NSPG/
Mahmoud Selim, Cristina Cipriani, Karl H. Johansson
TRATON CV AB, Stockholm, Sweden · KTH Royal Institute of Technology, Stockholm, Sweden