Continual adaptation to new tasks is essential for robots deployed in the real world, but adapting pre-trained policies remains challenging due to parameter growth and the need for continual demonstration collection. Parameter-Efficient Continual Fine-Tuning (PECFT) isolates task-specific updates, but adapter-per-task designs require demonstrations and grow with the task set. We propose ResMerge, a two-stage framework that learns mergeable residual policies atop a frozen pre-trained policy: residual reinforcement learning (RL) acquires task-specific corrections without additional expert demonstrations during adaptation, and parameter-space merging consolidates compatible residuals to reduce stored parameters. We introduce Residual-Field Consistency (RFC), a simple, bounded, and interpretable metric that assesses compatibility in residual action-field space by comparing correction direction and magnitude. Experiments on LIBERO across two backbones and on a real-world Franka robot show improved continual learning performance over strong PECFT baselines, while selective merging reduces the residual-parameter footprint. Project website: https://resmerge.github.io/.
Figures & tables
Fig. 2: ResMerge framework. In the Learn stage, a LoRA-parameterized residual policy Rcand is trained on the new task via residual RL with πbase frozen. In the Merge stage, Rcand is merged into the most compatible existing expert when their Residual-Field Consistency (RFC) meets or exceeds a compatibility threshold; otherwise it is added as a new expert.
Fig. 3: Overview of the experimental platforms and tasks. A Franka Panda with a Leap hand performs four sequential pinch- and power-grasp tasks; three frames illustrate each rollout. The bottom strip shows representative LIBERO-Spatial and LIBERO-Object tasks used for simulation comparisons.
Base Policy
LIBERO-Spatial
LIBERO-Object
Architecture
CL Method
MPE ↑
Demonstrations ↓
FWT ↑
NBT ↓
AUC ↑
FWT ↑
NBT ↓
AUC ↑
ViT-T
Seq. SFT
—
50 demos/task
63±2
76±1
16±1
76±3
73±3
27±2
ER
—
50 demos/task
63±1
29±2
50±2
70±2
28±1
57±1
PackNet
—
50 demos/task
57±4
15±1
59±3
58±3
18±2
56±4
TAIL
0.70
50 demos/task
51±2
56±1
60±3
58±2
50±2
70±1
DMPEL
0.69
50 demos/task
22±2
28±3
29±2
14±3
25±1
27±3
TABLE I: Continual learning on LIBERO. Results are mean ± standard deviation over five seeds. Demonstrations are counted only during adaptation.
Fig. 4: Sequential RFC analysis on the real robot. Streamlines show the compared correction fields; background color indicates local directional agreement (green) or opposition (red). Statistics use full observations and actions; the plotted projection is only a visualization. (a) RFC accepts (R1,R2) , producing RAB . (b) RFC rejects (RAB,R3) and retains R3 . (c) RFC accepts (R3,R4) , producing RCD . Two consolidated experts serve all four tasks.
(a) Continual learning metrics
Method
FWT (%)
NBT (pp)
AUC (%)
DMPEL
24
23
54
ResMerge
50
7
75
(b) Average success and residual-expert count
Method
Avg. SR (%)
Experts
TAIL
52.5
–
TABLE II: Real-world evaluation. We report continual learning metrics in (a) and average success with residual-expert counts in (b).
Strategy
Learn-SR
Merge-SR
FWT
NBT
AUC
No Merge
36
–
27
13
54
Merge-All
36
21
23
22
37
ResMerge
36
78
49
−8
79
TABLE III: Effect of merging strategy. Learn-SR and Merge-SR are mean success rates across all ten LIBERO-Object tasks with ViT-T. Merge-SR includes the RL refinement that follows consolidation.
Residual Architecture
Learn-SR (%)
Merge-SR (%)
MLP Residual Policy
36
14
LoRA Residual Policy (ours)
36
78
TABLE IV: Effect of residual architecture. LoRA and MLP residuals are compared before and after merging and refinement.
Parameter
OpenVLA-OFT
Real robot
Actor learning rate
3×10−4
10−5
Critic learning rate
3×10−4
3×10−4
Residual scaling factor α
0.05
0.05
TABLE V: Implementation settings by experimental platform. Real-robot architecture and evaluation details are specified separately in the text.
Threshold τ
FWT
NBT
AUC
0.1
23
22
37
0.3
49
−2
71
0.5
49
−8
79
0.7
27
22
47
TABLE VI: RFC threshold sensitivity. We vary τ on LIBERO-Object with ViT-T, keeping it fixed throughout each task sequence.
Metric
Pre-SR
Post-SR
FWT
NBT
AUC
Task Similarity
36.0
40.0
26.0
11.0
46.0
RFC
36.0
78.0
49.0
−8.0
78.7
TABLE VII: Comparison of merging metrics. Results are on LIBERO-Object with ViT-T. Post-merge success includes RL refinement.
Stage and residual pool
T1
T2
T3
T4
T1 : {R1}
85
35
55
55
T2 : {RAB}
75
60
55
50
T3 : {RAB,R3}
75
60
85
50
T4 : {RAB,RCD}
75
60
75
80
TABLE VIII: Real-robot success across the task sequence. Rows indicate the completed training stage and residual pool; columns indicate the evaluated task. Each cell is evaluated over 20 trials. Success rates are percentages.
Pretrained imitation policies have become a strong foundation for robot manipulation, but online adaptation can further improve policy performance within their pretrained behavioral support. A central question is therefore how reinforcement learning (RL) should adapt policies after offline pretraining. Existing lightweight methods commonly apply residual corrections directly in action space. We investigate how the steering interface affects online RL. In this work, we propose Z-Perturbation Reinforcement Learning (ZPRL), an approach that steers pretrained policies through a compact bottleneck latent rather than through policy weights or output actions. Building on an offline-learned bottleneck policy interface introduced by previous work, ZPRL turns this representation into an online RL control space through reward-driven residual perturbations while freezing the pretrained policy. We instantiate ZPRL on flow-matching policies and evaluate it on eight simulation tasks and four real-world tasks. Across diverse manipulation settings, ZPRL improves sample efficiency with competitive or better final performance. In the real world, ZPRL improves the average success rate on four tasks by 33.7 percentage points over imitation base policies. Our qualitative and quantitative results also indicate that ZPRL combines task success with smooth actions without explicit temporal filtering or smoothness regularization. These results suggest that a compact, task-aligned bottleneck latent provides an effective interface for online RL adaptation. Videos and code can be found at https://manutdmoon.github.io/ZPRL/.
Dongjie Yu, Kun Lei, Zhennan Jiang +2
School of Computing and Data Science, The University of Hong Kong, Hong Kong SAR. · Shanghai Qizhi Institute, Shanghai, China. · Shanghai Jiao Tong University, Shanghai, China. +2
Reinforcement learning (RL) post-training improves robot foundation models (RFMs) beyond their demonstrations, but real-world deployment demands more: learning new tasks efficiently while retaining previously acquired behaviors. Existing continual RL methods on top of VLAs mitigate forgetting but rely on on-policy updates with limited forward transfer, whereas off-policy methods improve single-task sample efficiency without addressing learner persistence across tasks. We propose ACCRUE, a continual off-policy RL framework that carries lightweight actor and critic networks across tasks, allowing each stage to reuse behavioral and value knowledge acquired through prior RL. The inherited critic adapts only on current-task transitions and guides current-task actor learning, while prior behavior is preserved by distilling the actor toward cached targets on a small number of retained episodes. Fixed visual features keep the inherited learner and its replay compatible with base-policy updates. Across contact-rich assembly sequences in simulation and on real-robot hardware, ACCRUE accelerates subsequent-task learning with near-zero forgetting and significantly reduces real-robot training time by 31.0% to 68.5%.
Robots deployed in the real world rarely operate under a single fixed dynamics model: wind changes, payloads vary, batteries drain, contacts shift, and hardware wears. Yet most learning-based controllers are trained once and deployed as if learning were complete. This prevents the robot from using deployment experience to further improve task performance. In this work, we propose a continual learning framework that uses real-world experience to improve robot policies under hidden and recurring dynamics. Our method learns a condition-aware dynamics model from real state-action trajectories by combining an analytical physics prior with a neural residual for unmodeled effects. A recurrent encoder infers the current hidden condition from recent interaction, and this estimate conditions both the residual model and the policy. Policy learning is performed via differentiable simulation using diverse learned dynamics sampled from the latent model. At deployment, these sampled conditions are replaced by conditions inferred online from recent real interaction, allowing the policy to recover recurring dynamics by recognition rather than residual re-fitting. Through extensive simulation studies and real-world experiments, we demonstrate that the framework improves policy performance under diverse unobserved disturbances. On real quadrotor trajectory tracking under changing wind, the policy recovers from recurring disturbances in roughly 1s, about 5x faster than online residual re-fitting. It also reduces large-disturbance hover and tracking errors by 65.7% and 53.3% over the state-of-the-art online adaptation approaches
Jiaxu Xing, Zhiyuan Zhu, Yunfan Ren +4
Robotics and Perception Group, University of Zurich