Organizations: Robotics and Autonomous Systems Thrust, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · School of Artificial Intelligence, Sun Yat-sen University, Zhuhai, China · Department of Mechanical and Energy Engineering, Southern University of Science and Technology, Shenzhen, China
Humans seamlessly fuse anticipatory planning with immediate feedback to perform successive mobile manipulation tasks without stopping, achieving both high efficiency and reliability. Replicating this fluid and reliable behavior in robots remains fundamentally challenging, not only due to conflicts between long-horizon planning and real-time reactivity, but also because excessively pursuing efficiency undermines reliability in uncertain environments: it impairs stable perception and the potential for compensation, while also increasing the risk of unintended contact. In this work, we present a unified framework that synergizes efficiency and reliability for continuous mobile manipulation. It features a reliability-aware trajectory planner that embeds essential elements for reliable execution into spatiotemporal optimization, generating efficient and reliability-promising global trajectories. It is coupled with a phase-dependent switching controller that seamlessly transitions between global trajectory tracking for efficiency and task-error compensation for reliability. We also investigate a hierarchical initialization that facilitates online replanning despite the complexity of long-horizon planning problems. Real-world evaluations demonstrate that our approach enables efficient and reliable completion of successive tasks under uncertainty (e.g., dynamic disturbances, perception and control errors). Moreover, the framework generalizes to tasks with diverse end-effector constraints. Compared with state-of-the-art baselines, our method consistently achieves the highest efficiency while improving the task success rate by 26.67%--81.67%. Comprehensive ablation studies further validate the contribution of each component. The source code will be released.
Figures & tables
Figure 1: Teaser. Continuous on-the-move mobile manipulation for tightly arranged task sequences, synergizing efficiency and reliability under real-world uncertainty using onboard sensing.
Figure 2: Overview of challenges and solutions for efficient and reliable mobile manipulation. ( A ) (i) A mobile manipulator robot capable of coupled locomotion, manipulation, and perception, and (ii) a composite top-view of the execution trajectory for a mobile manipulator performing continuous tasks, where colored segments correspond to specific challenges in (C)–(G). ( B ) Example task set. ( C )–( G ) illustrate five challenges while pursuing both efficiency and reliability (rows i–ii) and our corresponding solutions (row iii): ( C ) To bridge long-horizon planning and real-time reactivity, we propose a high-frequency controller that bridges whole-body trajectory tracking with real-time task-error compensation using a phase-dependent switching scheme. ( D ) To prevent insufficient observation windows due to fast motion, we introduce Time-assured Active Perception to ensure sufficient observation time. ( E ) To avoid limited compensation margins while pursuing efficiency, we present Compensation Margin Zone. ( F ) To resolve unsafe end-effector motions, we propose Efficient Safe Interaction for pre-grasp and post-place phases. ( G ) To handle conflicts between task and safety requirements, we propose Elastic Collision Spheres.
Figure 3: Illustration of Task Model.
Figure 4: System overview. Starting from a coarse prior environment map and a coarse task set, the planner computes a time-efficient whole-body trajectory that is explicitly shaped to support reliable task execution. During execution, the planner continuously replans using the latest obstacle, robot state, and task information updated online via onboard perception. Based on both the planned trajectory and the latest online information, the controller then smoothly switches between global trajectory tracking and task-error compensation while preserving the safe interaction structure and the overall efficiency of the planned motion. This planning–control loop runs continuously until all tasks are completed.
Figure 5: Overview of reliability-aware whole-body trajectory optimization. It generates a time-parameterized whole-body trajectory while enforcing (i) time-assured active perception, (ii) compensation margin zone constraints, (iii) efficient safe interaction constraints, (iv) elastic collision sphere safety constraints, (v) task satisfaction constraints, (vi) dynamic feasibility, and (vii) collision avoidance, yielding (viii) the optimized whole-body trajectory.
Figure 6: Illustration of fd,ray : smooth transition from perpendicular distance to Euclidean distance.
Figure 7: Illustration of the elastic radius relastic as a function of the distance x=∣pt−p∣2 from the target position.
Figure 8: Overview of hierarchical multi-task whole-body path planner consisting of ( A ) Sequential-Progress Hybrid A* for the mobile base, and ( B ) a layered-graph shortest-path search for the manipulator.
Figure 9: Safe-warping-based phase-dependent controller. ( A ) a cascaded model predictive controller, ( B ) end-effector trajectory warping incorporates online task pose estimation into a safe compensation motion, and ( C ) smooth switching weights transition between whole-body trajectory tracking and task-error compensation.
Figure 10: Mobile manipulation in constrained and dynamic environments. ( A ) Constrained indoor environment. (i) Overview of the office lounge with the executed trajectory of the mobile manipulator and five task locations (a-e). (ii) Point cloud of the environment with the executed mobile base trajectory color-coded by the measured maximum wheel angular velocity of the two driven wheels. (iii)–(v) Representative motion segments: (iii) grasping a bottle while the mobile base executes a tight U-turn, (iv) closing an open drawer while moving toward the opposite side of the lounge, and (v) grasping a bottle randomly placed on the table. ( B ) Dynamic environment. (i) and (iii) Overviews of the environment and the mobile manipulator trajectories during two phases with six task locations (a-f). (ii) and (iv) Point clouds of the environment and dynamic obstacles with mobile base trajectories, color-coded by the measured maximum wheel angular velocity during two phases. (v)–(x) Representative events: (v) grasping a bottle on a reciprocating turntable, (vi) lowering the manipulator to pass under a table, (vii) raising the manipulator to avoid a pedestrian, (viii) a person moving a chair to block the path under the table, (ix) the mobile base detouring to avoid a pedestrian, and (x) grasping a bottle that is displaced by a person just before grasping.
Figure 11: Long-horizon tasks under task pose uncertainty. ( A ) Tasks overview: the MM repeatedly ( 10× ) employs active perception to estimate domino poses, then grasps and arranges them into a straight line. ( B ) Final placement comparison: the line of dominoes arranged by a human (left) versus that produced by the MM (right). ( C ) Snapshots captured at the grasping instant for the ten consecutive grasping tasks (i)–(x).
Figure 12: Top-down comparison of the coarse initial pose and the final refined estimate (last perception estimate before grasping) of the domino for the ten grasping tasks.
Figure 13: Time evolution of pose estimation deviation (mean and min-max range over the ten trials) relative to the final refined estimate.
Figure 14: Comparison of computation time and trajectory duration.
Figure 15: Distribution of pose errors between the planned and target manipulation poses.
Figure 16: Position and orientation errors with respect to the final refined estimate, including trajectory error (deviation of the planned trajectory) and post-compensation error (final deviation after task-error compensation).
Figure 17: Base-arm coordination for complex tasks. For each of six mobile manipulation tasks, the top row shows representative snapshots and the bottom row plots the left and right wheel angular velocities over time. In each plot, four shaded time intervals indicate the trajectory segments corresponding to the four snapshots above and highlight how the wheel angular velocities evolve across representative phases of execution. The tasks include ( A ) watering a row of flowers, ( B ) opening a cabinet door along a circular arc, ( C ) pulling a curtain open along a straight line, ( D ) inserting an umbrella into an umbrella stand, ( E ) turning off a stove knob, and ( F ) turning a large valve while moving around it. Across all tasks, the wheel velocities remain non-zero and are automatically modulated to accommodate the end-effector–constrained phases while maintaining continuous whole-body motion.
Figure 18: Overview of four benchmarking scenarios (office, apartment, café, and simple scenario) and mobile base trajectory for our method versus baselines. Each scenario comprises six pick-and-place/drop missions with object displacements of 0, 0.05, or 0.10 m.
Scenarios
Methods
Mission Success Rate (%) ↑
SSCT ↑
Avg Vel (m/s) ↑
dσ=0.0
dσ=0.05
dσ=0.1
dσ=0.0
dσ=0.05
dσ=0.1
Mean
Std
Mean
Std
Mean
Std
Office
Thakar
30.00
15.00
3.33
0.0852
0.0561
0.0374
0.0385
0.0171
0.0343
0.1335
Reister
93.33
68.33
65.00
0.2628
0.0151
0.1905
0.0801
0.1979
0.0802
0.1118
Burgess
26.67
18.33
21.67
0.1674
0.0712
0.1001
0.0845
0.1217
0.0859
0.2362
Ours
100.0
100.0
100.0
0.6125
0.0168
0.6143
0.0118
0.6230
0.0193
0.2462
Table 1: Simulation benchmark results across four scenarios under different levels of task-pose displacement dσ . For each scenario, we report the mission success rate, scenario success weighted by cycle time (SSCT), and average base linear velocity (Avg Vel) over 60 missions; for SSCT, both the mean and standard deviation (Std) are provided.
Figure 19: Scenario Success-weighted Cycle Time (SSCT) for the benchmarked methods; bar heights denote the mean, and error bars the standard deviation, across 10 trials for each scenario–displacement combination.
Figure 20: Distributions of mobile base linear velocity before collision or getting stuck for the four methods in each scenario, computed from 30 trials per scenario (10 trials for each displacement). White circles and capped vertical bars represent the median and interquartile range (IQR), respectively.
Figure 21: Distribution of operation time across four scenarios for our method and two ablations.
Variants
SR (%)
Operation Time (s)
w/o TAP(D1 ∗ )
90.42
119.87
Ours(D1 ∗ )
99.58
123.24
w/o TAP(D2 ∗ )
76.67
120.23
Ours(D2 ∗ )
99.17
122.91
w/o CMZ
88.75
119.97
Ours(N ∗ )
100.0
123.56
Table 2: Ablation results on compensation ability. For each variant, we report the mission success rate (SR) across 240 missions and the mean operation time averaged over 40 runs (10 per scenario).
Figure 22: Distribution of (a) effective observation time and (b) time from first pose estimate to manipulation, comparing our method and the variant without time-assured active perception (TAP), under perception initialization delays of 1 s (D1) and 2 s (D2). D1 and D2 denote the perception-module initialization delay of 1 s and 2 s, respectively.
Variants
Succ
Cnt
Teff(s)
Tmani(s)
Median
Min
Median
Min
w/o TAP(D1 ∗ )
✓
217
3.28
0.00
2.40
0.00
✗
23
1.03
0.00
0.00
0.00
✓
239
5.42
3.25
4.23
1.83
Ours(D1 ∗ )
✗
1
4.38
4.38
3.25
3.25
w/o TAP(D2 ∗ )
✓
184
3.47
1.73
1.50
0.25
Table 3: Ablation study of time-assured active perception (TAP) under different perception initialization delays. For each variant, results are grouped by mission outcome (Succ: success or failure), and we report the number of successful or failed missions (Cnt), together with the median and minimum (Min) values of effective observation time ( Teff ) and time from first pose estimate to manipulation ( Tmani ).
Figure 23: Scatter plots comparing position error against joint limit distance (left two) and manipulability (right two), evaluated along the global trajectory (“Trajectory”) and after compensation (“Post-compensation”) at the manipulation instant, comparing our method and the variant without compensation margin zone (CMZ).
Figure 24: Stacked bar chart showing success count with failure modes across 240 missions for our method and three ablation variants.
Figure 25: Distribution of operation time for four method configurations across four scenarios.
Figure 26: Distribution of end-effector position (left) and angle (right) errors between the planned and target manipulation pose, comparing our method with a variant without elastic collision spheres.
Recent advances in open-vocabulary mobile manipulation have brought robots into real domestic environments. In such settings, reliable long-horizon execution under open-set object references and frequent disturbances becomes essential. However, many failures persist. These are not caused by semantic misunderstanding but by inconsistencies between symbolic plans and the evolving physical world, manifested as three recurring limitations: (i) existing systems often rely on pre-scanned semantic maps that become inconsistent after scene changes and disturbances; (ii) they select navigation endpoints without considering downstream manipulation feasibility, causing the "arrived but inoperable" problem; and (iii) they handle anomalies through undifferentiated global replanning, which often fails to contain local errors. To address this execution inconsistency, we present ANCHOR, a physically grounded closed-loop framework that aligns symbolic reasoning with verifiable physical state during execution. ANCHOR integrates three mechanisms: (i) physically anchored task planning, which binds symbolic predicates to observable geometric anchors and re-validates them after each action; (ii) operability-aware base alignment, which ensures that navigation endpoints satisfy kinematic reachability and local collision feasibility; and (iii) minimum-responsible-layer hierarchical recovery, which localizes failures across perception, base-arm coordination, and execution layers to prevent cascading retries. Across 60 real-robot trials in previously unseen environments, ANCHOR improves task success from 53.3% to 71.7% and achieves a 71.4% recovery rate under perturbations, demonstrating that explicit physical grounding and structured failure containment are critical for robust mobile manipulation. Our project page is available at https://anchor9178.github.io/ANCHOR/ .
The autonomous operation of tracked mobile manipulators in rescue missions requires not only ensuring the reachability and safety of robot motion but also maintaining stable end-effector manipulation under diverse task demands. However, existing studies have overlooked many end-effector motion properties at both the planning and control levels. This paper presents a motion generation framework for tracked mobile manipulators to achieve stable end-effector operation in complex rescue scenarios. The framework formulates a coordinated path optimization model that couples end-effector and mobile base states and designs compact cost/constraint representations to mitigate nonlinearities and reduce computational complexity. Furthermore, an isolated control scheme with feedforward compensation and feedback regulation is developed to enable coordinated path tracking for the robot. Extensive simulated and real-world experiments on rescue scenarios demonstrate that the proposed framework consistently outperforms SOTA methods across key metrics, including task success rate and end-effector motion stability, validating its effectiveness and robustness in complex mobile manipulation tasks.
Although robotic manipulation has made significant progress, reliable execution remains challenging because task failures are inevitable in dynamic and unstructured environments. To handle such failures, existing frameworks typically follow a stepwise detect-reason-recover pipeline, which often incurs high latency and limited robustness due to delayed reasoning and reactive planning. Inspired by the human capability to anticipate and proactively plan for potential failures, we introduce AgentChord, an agentic system that models a manipulation task as a directed task graph. Before execution, this graph is enriched with anticipatory recovery branches that specify context-aware corrective behaviors, enabling immediate and targeted responses when failures occur. Specifically, AgentChord operates through a choreography of specialized agents: a composer that structures the nominal task graph, an arranger that augments the graph with anticipatory recovery branches, and a conductor that compiles and coordinates executable transitions using low-latency monitors to detect deviations and trigger pre-compiled recoveries without re-planning. Empirical studies on diverse long-horizon bimanual manipulation tasks demonstrate that AgentChord substantially improves success rates and execution efficiency, advancing the reliability and autonomy of real-world robotic systems. The project page is available at: https://shengxu.net/AgentChord/.
Sheng Xu, Ruixing Jin, Huayi Zhou +6
School of Data Science, The Chinese University of Hong Kong, Shenzhen · DexForce Technology, 3Shenzhen Loop Area Institute.