Hierarchical Residual Policy Learning for Real-World Mobile Manipulation with Sparse Human Guidance

Zhefei Gong; Hefeifei Jiang; Zehao Wang; Jiachen Li

cs.ROOct 14, 2026

Hierarchical Residual Policy Learning for Real-World Mobile Manipulation with Sparse Human Guidance

Authors: Zhefei Gong, Hefeifei Jiang, Zehao Wang, Jiachen Li

Organizations: Northeastern University · University of California, Riverside · Georgia Institute of Technology

Abstract

Mobile Manipulation (MoMa) is a cornerstone of autonomous robotic systems, yet real-world fine-tuning remains a formidable challenge because of high-dimensional action spaces and intricate coupling between heterogeneous modules. In this paper, we propose HiRe-MoMa (Hierarchical Residual Mobile Manipulation), a systematic framework for stable and sample-efficient real-world fine-tuning of complex MoMa systems. At its core, HiRe-MoMa adopts a Hierarchical Residual Architecture that fine-tunes a frozen base policy. A Policy-Agnostic Residual learns additive corrections atop the base policy, while Hierarchical Decoupling splits the residual into mobile-base and manipulator-arm heads with unidirectional conditioning, mitigating gradient interference and aligning corrections with the platform's physical hierarchy. To improve sample efficiency in data-expensive real-world settings, we propose Expert-Guided Value Alignment, which leverages sparse human interventions with under 20% training coverage to reshape the critic's value landscape, effectively guiding autonomous exploration through expert priors. Extensive real-world experiments on the TIAGo++ platform demonstrate that HiRe-MoMa consistently enhances task performance with only around 30 minutes of online interaction, regardless of the base policy architecture. To our knowledge, this is among the first frameworks to enable efficient on-robot fine-tuning for mobile manipulation in the real world, providing a practical paradigm for deploying complex robotic systems in dynamic, human-centric environments. Further details can be found on our project website: https://hire-moma.github.io/

Figures & tables

Figure 1: HiRe-MoMa System Overview. (a) Training Paradigm: A real-world dual-unit configuration consisting of an onboard controller for real-time interaction and a computing unit for policy optimization and action prediction. (b) Coordinated Execution: The red dashed lines illustrate the stringent spatiotemporal precision required for base-arm synchronization. In mobile manipulation, marginal tracking deviations are often magnified by kinematic coupling, causing even routine grasping maneuvers to fail.

Figure 2: HiRe-MoMa Framework enables policy-agnostic real-world fine-tuning of mobile manipulators through an asynchronous residual reinforcement learning paradigm. (Right) Shared-Control Rollout: an onboard rollout worker augments a frozen base policy with learnable residuals; sparse human interventions during critical or hazardous phases are stored in an expert buffer. (Left) Asynchronous Central Learner: the residual policy is optimized along two pillars: a) a Hierarchical Residual Architecture ( Sec. 4.1 ) that decouples the residual into mobile and manipulator heads with unidirectional conditioning, embedding the platform’s physical hierarchy directly into the policy; and b) an Expert-Guided Value Alignment loss ( Sec. 4.2 ) that fuses sparse environment rewards with sparse human corrections to reshape the critic toward expert-preferred behaviors.

	Training Time (mins)	Success Rate (%)				Episode length (steps)
Task	Training Time (mins)	Base (DP)	+HG-DAgger [ 26 ]	+Flat Residual [ 35 ]	HiRe-MoMa (Full)	Base (DP)	+HG-DAgger [ 26 ]	+Flat Residual [ 35 ]	HiRe-MoMa (Full)
Place Only	25	50.0	10.0 (-80%)	60.0 (+20%)	95.0 (+90%)	342	420 (0.81x)	329 (1.03x)	235 (1.46x)
Sweep Litter	30	45.0	5.0 (-89%)	40.0 (-11%)	90.0 (+100%)	359	478 (0.75x)	394 (0.91x)	271 (1.32x)
Pick Place	40	22.5	5.0 (-78%)	27.5 (+22%)	40.0 (+78%)	664	742 (0.89x)	649 (1.02x)	552 (1.20x)
Average	31.7	39.2	6.7 (-83%)	42.5 (+8%)	75.0 (+91%)	455.0	546.7 (0.82x)	457.3 (0.99x)	352.7 (1.29x)

Table 1: Comparison with Fine-Tuning Baselines. All methods use a frozen DP as the base policy. +HG-DAgger [ 26 ] overfits to sparse human corrections, and +Flat Residual (equivalent to HIL-SERL [ 35 ] Residual) is limited by inter-module gradient interference in its monolithic action space. The full HiRe-MoMa consistently outperforms both in success rate and execution efficiency. Metrics are averaged over 20 trials per task.

	Training Time (mins)	Success Rate (%)			Episode length (steps)
Task	Training Time (mins)	+Flat Residual	+Hierarchical Decoupling	+Alignment (Full)	+Flat Residual	+Hierarchical Decoupling	+Alignment (Full)
Place Only	25	50 $\rightarrow$ 60	50 $\rightarrow$ 85	50 $\rightarrow$ 95	342 $\rightarrow$ 329	342 $\rightarrow$ 283	342 $\rightarrow$ 235
Sweep Litter	30	45 $\rightarrow$ 40	45 $\rightarrow$ 85	45 $\rightarrow$ 90	359 $\rightarrow$ 394	359 $\rightarrow$ 304	359 $\rightarrow$ 271
Average	27.5	47.5 $\rightarrow$ 50.0	47.5 $\rightarrow$ 85.0	47.5 $\rightarrow$ 92.5	350.5 $\rightarrow$ 361.5	350.5 $\rightarrow$ 293.5	350.5 $\rightarrow$ 253.0

Table 2: Build-Up Ablation of HiRe-MoMa Components. Each column adds one method component on top of the previous: +Flat Residual is a single residual head over the joint action space (equivalent to HIL-SERL [ 35 ] Residual); +Hierarchical Decoupling splits the residual into base/arm heads with unidirectional conditioning ( Sec. 4.1 ); +Alignment is the full HiRe-MoMa with Expert-Guided Value Alignment ( Sec. 4.2 ). Results show base policy

\rightarrow

fine-tuned performance on tasks Place Only and Sweep Litter .

Figure 3: Real-World Experimental Tasks. Our evaluation encompasses three tasks, Place Only , Sweep Litter , and Pick Place , that demand varying levels of inter-module coordination. These tasks are particularly challenging for mobile manipulators as they require the simultaneous optimization of base positioning and arm reaching, where minor misalignments in the platform’s pose propagate as significant errors at the end-effector.

	Training Time (mins)	Success Rate (%)		Episode length (steps)
Task	Training Time (mins)	Base (ACT)	HiRe-MoMa (Full)	Base (ACT)	HiRe-MoMa (Full)
Place Only	25	40	80 (+100%)	381	303 (1.26x)
Sweep Litter	30	40	75 (+88%)	362	297 (1.22x)
Average	27.5	40.0	77.5 (+94%)	371.5	300.0 (1.24x)

Table 3: Policy-Agnostic Generalization. Our HiRe-MoMa yields consistent gains when applied to ACT [ 62 ] , demonstrating architecture-independent efficacy. While the performance ceiling is inherently tied to the base policy’s quality, the results confirm HiRe-MoMa’s capacity to fine-tune diverse base policies.

Figure 4: Training Dynamics. Comparison across methods over training time, where we construct the figure along the actual training time. (a) Success rate: HiRe-MoMa(full) attains a consistently higher success rate throughout training, validating the effectiveness of our design combining hierarchical residual architecture and alignment. (b) Human intervention steps: HiRe-MoMa requires fewer human interventions and converges faster than the baselines, and the comparison with and without alignment ( +Hierarchical Decoupling ) demonstrates the effectiveness of the extra alignment loss.

Figure 5: Disentangling Decomposition and Conditioning. Training-time success-rate curves for +Flat Residual , +Decomposition-Only , and +Hierarchical Decoupling on Place Only and Sweep Litter ; see App. A.1 .

Figure 6: Residual Correction Visualization. Base longitudinal velocity and manipulator vertical position throughout the Place Only task. HiRe-MoMa applies coherent corrections at critical trajectory phases, most notably during base acceleration and the manipulator’s lay-down motion.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

	Success Rate (%)			Episode Length (steps)
Task	+Flat Residual	+Decomposition Only	+Hierarchical Decoupling	+Flat Residual	+Decomposition Only	+Hierarchical Decoupling
Place Only	50 $\rightarrow$ 60	50 $\rightarrow$ 85	50 $\rightarrow$ 85	342 $\rightarrow$ 329	342 $\rightarrow$ 291	342 $\rightarrow$ 283
Sweep Litter	45 $\rightarrow$ 40	45 $\rightarrow$ 75	45 $\rightarrow$ 85	359 $\rightarrow$ 394	359 $\rightarrow$ 322	359 $\rightarrow$ 304
Average	47.5 $\rightarrow$ 50.0	47.5 $\rightarrow$ 80.0	47.5 $\rightarrow$ 85.0	350.5 $\rightarrow$ 361.5	350.5 $\rightarrow$ 306.5	350.5 $\rightarrow$ 293.5

Appendix

Table 4: Decomposition vs. Conditioning within Hierarchical Decoupling. Decomposition alone provides the dominant gain by mitigating gradient interference (

+30

pt average success rate). Conditioning adds a targeted improvement (

+5

pt on average) that manifests on the coupled dual-arm task ( Sweep Litter ).

Figure 7: Training-Step Progression vs. Wall-Clock Time. HG-DAgger advances through training steps faster than the SAC-based residual methods, but its higher step throughput does not translate into higher success rates ( Fig. 4 (a)). HiRe-MoMa adds only a modest per-step overhead over +Flat Residual (

1

–

2\times

) while delivering clear gains in success rate and episode length.

Explore similar work

MobiAgent: Dual-Loop Recursive Policy Self-Improvement for Long-Horizon Mobile Manipulation
Oct 2, 2026Chenzhi Liu, Yue Zhang, Jiehong Lin +4Long-Horizon Robotic Manipulation Mobile Manipulation
The University of Hong Kong · Astribot · Southern University of Science and Technology
MoRI: Mixture of RL and IL Experts for Long-Horizon Manipulation Tasks
Apr 11, 2026Yaohang Xu, Lianjie Ma, Gewei Zuo +3Robotic RL Long-Horizon Robotic Manipulation
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China. · DRAGON Lab, The University of Tokyo, Tokyo, Japan. · School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan, China.
Whole-Body Mobile Manipulation using Offline Reinforcement Learning on Sub-optimal Controllers
Apr 14, 2026Snehal Jauhri, Vignesh Prasad, Georgia ChalvatzakiMobile Manipulation Whole-Body Robotic Manipulation
PEARL Lab, Computer Science Department, Technische Universität Darmstadt, Germany · Hessian.AI, Darmstadt, Germany · Robotics Institute, Germany

Oct 2, 2026J/K move · Enter open · S save

MobiAgent: Dual-Loop Recursive Policy Self-Improvement for Long-Horizon Mobile Manipulation

Chenzhi Liu, Yue Zhang, Jiehong Lin, Jianan Wang+3

The University of Hong Kong · Astribot · Southern University of Science and Technology

Long-Horizon Robotic Manipulation Mobile Manipulation Robot Learning Continual Robot Learning Large Language Model-Based Robot Planning VLMs for Robotics

Open paper AskPDF arXiv

Long-horizon mobile manipulation presents significant challenges due to compounding execution errors and capacity interference between locomotion and arm control. While recent Vision-Language-Action models excel at short-horizon tasks, they lack the hierarchical reasoning required for multi-stage objectives. Furthermore, existing hierarchical agents suffer from rigid sub-task mapping, inflexible replanning, and a lack of continuous learning. To address these limitations, we introduce MobiAgent, a dual-loop agentic framework that bridges robust deployment execution and recursive policy self-improvement. During deployment, the Inner Loop decouples high-level reasoning from low-level control through highly composable atomic skills. It employs Vision-Language models for receding-horizon planning and visual reflection, dynamically composing skills to ensure robust error recovery. These skills are executed by specialized flow-matching experts that share a unified VLM backbone, maximizing reusability while mitigating capacity interference. Concurrently, the Outer Loop drives automated lifelong learning by autonomously segmenting and verifying deployment rollouts, clustering them to discover atomic skills, and continuously fine-tuning the skill library without human annotations. Evaluations on RoboCasa, BEHAVIOR-1K, and real-world tasks demonstrate the effectiveness of MobiAgent. It outperforms

π_{0.5}

-TA by 22.5 percentage points on BEHAVIOR-1K and enables robust recovery from execution failures. Through autonomous data recycling, success improves from 7.50% to 27.50% on RoboCasa and from 32.5% to 57.5% on Astribot S1.

Abstract

Explore similar work

MobiAgent: Dual-Loop Recursive Policy Self-Improvement for Long-Horizon Mobile Manipulation

MoRI: Mixture of RL and IL Experts for Long-Horizon Manipulation Tasks

Whole-Body Mobile Manipulation using Offline Reinforcement Learning on Sub-optimal Controllers