Object- and Task-Aware Active Perception for Safe Navigation of Mobile Manipulators in Semi-Static Environments

Jingxing Qian; Xintong Du; Jun Yang; Siqi Zhou; Steven L. Waslander; Angela P. Schoellig

cs.ROOct 21, 2026

Object- and Task-Aware Active Perception for Safe Navigation of Mobile Manipulators in Semi-Static Environments

Authors: Jingxing Qian, Xintong Du, Jun Yang, Siqi Zhou, Steven L. Waslander, Angela P. Schoellig

Organizations: University of Toronto Institute for Aerospace Studies (UTIAS) and the Vector Institute for Artificial Intelligence, Canada. · Epson Canada Ltd., Toronto, ON L3R 6G3, Canada. · Learning Systems and Robotics Lab at the Technical University of Munich and the Munich Institute for Robotics and Machine Intelligence (MIRMI), Germany. · School of Computing Science, Faculty of Applied Sciences, Simon Fraser University, Burnaby, BC, Canada. ·

Abstract

Mobile manipulators must navigate safely in cluttered, partially unknown environments across logistics, manufacturing, and service applications. Semi-static changes between robot visits can lead to outdated scene knowledge and increased collision risk. We propose an active perception framework that enables a high degree-of-freedom (DoF) mobile manipulator to inspect uncertain or safety-critical regions with its hand camera during navigation. A probabilistic, object- and task-aware next-best-view (NBV) formulation prioritizes observations near the desired path and around uncertain or potentially changing objects. The perception objective is integrated into hierarchical-task model predictive control (HTMPC), exploiting kinematic redundancy for viewpoint adjustment while preserving the higher priority of base-trajectory tracking. The system unifies navigation, map maintenance, and active perception in a closed loop. Real-world tests show improved map consistency, reduced task-dependent map entropy, and lower navigation risk than passive or non-task-aware baselines. Additional simulation and real-world sequences demonstrate responses to controlled semi-static scene changes.

Figures & tables

Fig. 1 : A snapshot of the proposed active perception and navigation framework for high-degree-of-freedom (DoF) mobile manipulators in action. Equipped with a hand camera, the robot operates in cluttered, semi-static environments. It uses its kinematic redundancy to reposition the camera around obstacles and observe uncertain or occluded regions near its navigation path. See the multimedia attachment or online videos 6 6 6 Video demonstrations: https://tinyurl.com/nbv-for-mm-nav . for demonstrations.

Fig. 2 : Qualitative demonstration of the proposed active perception and navigation framework for mobile manipulation in action (left) compared with a baseline method that randomly points the hand camera (right). Red curves represent the planned base paths. Shaded blue regions represent the hand camera’s FoV. Yellow exclamation marks highlight hidden hazards such as a blind corner (top) and a hidden pole (bottom).

Fig. 3 : An overview of the proposed active perception and navigation framework for high-DoF mobile manipulators.

Fig. 4 : Our mobile manipulator setup. The system consists of a Universal Robots UR10 arm mounted on a Clearpath Robotics Ridgeback omnidirectional base. Two synchronized Orbbec Femto Bolt RGB-D cameras are installed, with one mounted on the base and the other mounted on the hand. An onboard laptop runs all the perception, planning and control algorithms.

Fig. 5 : Detailed schematic of the proposed active perception module.

Fig. 6 : Real-world static-scene benchmark. Left: Scene 1 Easy Loops. Middle: Scene 2 Hard L-Shaped Walls. Right: Scene 3 Hard Office Scene. The scenes progress from open loops to sharper turns and cluttered occlusions, where hidden hazards near the waypoints make active map maintenance important.

Fig. 7 : Bird’s-eye-view 3D reconstruction results in the real-world static-scene benchmark. Rows correspond to Scene 1–3. Green, yellow, and red indicate true positives, false negatives, and false positives, respectively. The proposed method recovers more occupied voxels in blind and path-critical regions.

Scene 1 Easy Loops	Precision $\uparrow$	Recall (TPR) $\uparrow$	FPR $\downarrow$	Outcome
Fixed	81.0	56.1	2.2	Success
Lookahead	81.7	59.6	2.3	Success
NBV-Map	81.8	61.9	2.2	Success
Random	77.2	56.2	2.8	Success
NBV-Map-Task (ours)	87.4	70.9	1.7	Success
Signed improvement	5.6	9.0	0.5

TABLE I : Quantitative mapping results from the real-world static-scene benchmark. The signed improvement compares ours with the best baseline: ours minus baseline for precision and recall, and baseline minus ours for FPR; positive values favor ours.

Fig. 8 : Task-dependent map entropy along executed paths in the real-world static-scene benchmark. Left: boxplots over the three loops in each scene. Right: representative top-down views with trajectories colored by the local entropy. Lower values indicate lower entropy in the regions traversed by the robot.

Fig. 9 : Safety margin in the real-world static-scene benchmark. Boxplots show the point-cloud clearance proxy over all loop executions. The dashed red line marks zero clearance under this proxy, corresponding to a near-contact configuration. Larger positive values indicate greater estimated clearance, especially in the hard turning scenes.

Method	Success Rate $\uparrow$
Fixed	33.3%
Lookahead	66.7%
NBV-Map	33.3%
Random	33.3%
NBV-Map-Task (ours)	100.0%

TABLE II : Completion percentage across nine sequential loop executions per method: three loops in each of three scenes.

Fig. 10 : Controlled Isaac Sim scene-change experiment. The red curves are the executed base trajectories, and the blue arrows show the sampled hand camera headings. After one row of boxes is shifted, the hand camera redirects toward that side to acquire additional observations.

Panel of Figure 13 — Fig. 11 : Consistency belief and task-dependent map entropy for two representative box regions in the controlled scene-change simulation shown in Fig. 10 . Vertical dashed lines mark loop boundaries, and shaded regions indicate intervals when the robot is away from the boxes and receives no observations of them. Each box changes primarily when its side of the corridor is displaced.

Fig. 12 : Real-world long-term scene-change experiment over four visits. In each visit, the robot re-enters the room with the maintained object map, acquires new observations near the shelf, updates the map, and replans according to the current shelf position, switching between the outside and inner routes.

Safe Robot Navigation

Safe robot navigation develops methods for reaching goals while avoiding collisions and managing uncertainty, dynamic obstacles, and environmental hazards.

See all 139 papers

Perceptive Hierarchical-Task MPC for Sequential Mobile Manipulation in Unstructured Semi-Static Environments
Mar 10, 2026Xintong Du, Jingxing Qian, Siqi Zhou +13D Scene Change Detection Mobile Manipulation
University of Toronto Institute for Aerospace Studies (UTIAS) and the Vector Institute for Artificial Intelligence, Canada. · School of Computing Science, Faculty of Applied Sciences, Simon Fraser University, Burnaby, BC, Canada. · Learning Systems and Robotics lab at the Technical University of Munich and the Munich Institute for Robotics and Machine Intelligence (MIRMI), Germany.
MAAP: Multi-Agent Active Perception for Collaborative Manipulation
Sep 18, 2026Bruno N. Y. Chen, Li Kang, Heng Zhou +4Active Vision for Robot Manipulation Robotic Manipulation
Carnegie Mellon University · Shanghai Jiao Tong University · University of Science and Technology of China +3
LEAP: Learning Emergent Active Perception for Quadruped Navigation
Sep 15, 2026Ü. Bora Gökbakan, Stéphane Caron, Philippe SouèresRobot Navigation Quadruped Robot Navigation
WILLOW · ISIR · LAAS-GEPETTO

Mar 10, 2026J/K move · Enter open · S save

Perceptive Hierarchical-Task MPC for Sequential Mobile Manipulation in Unstructured Semi-Static Environments

Xintong Du, Jingxing Qian, Siqi Zhou, Angela P. Schoellig

University of Toronto Institute for Aerospace Studies (UTIAS) and the Vector Institute for Artificial Intelligence, Canada. · School of Computing Science, Faculty of Applied Sciences, Simon Fraser University, Burnaby, BC, Canada. · Learning Systems and Robotics lab at the Technical University of Munich and the Munich Institute for Robotics and Machine Intelligence (MIRMI), Germany.

3D Scene Change Detection Mobile Manipulation Robot Task Planning Long-Horizon Robotic Manipulation Dynamic Robotic Manipulation Model Predictive Control

Open paper AskPDF arXiv

As compared to typical mobile manipulation tasks, sequential mobile manipulation poses a unique challenge -- as the robot operates over extended periods, successful task completion is not solely dependent on consistent motion generation but also on the robot's awareness and adaptivity to changes in the operating environment. While existing motion planners can generate whole-body trajectories to complete sequential tasks, they typically assume that the environment remains static and rely on precomputed maps. This assumption often breaks down during long-term operations, where semi-static changes such as object removal, introduction, or shifts are common. In this work, we propose a novel perceptive hierarchical-task model predictive control (HTMPC) framework for efficient sequential mobile manipulation in unstructured, changing environments. To tackle the challenge, we leverage a Bayesian inference framework to explicitly model object-level changes and thereby maintain a temporally accurate representation of the 3D environment; this up-to-date representation is embedded in a lexicographic optimization framework to enable efficient execution of sequential tasks. We validate our perceptive HTMPC approach through both simulated and real-robot experiments. In contrast to baseline methods, our approach systematically accounts for moved and phantom obstacles, successfully completing sequential tasks with higher efficiency and reactivity, without relying on prior maps or external infrastructure.

Abstract

Safe Robot Navigation

Explore similar work

Perceptive Hierarchical-Task MPC for Sequential Mobile Manipulation in Unstructured Semi-Static Environments

MAAP: Multi-Agent Active Perception for Collaborative Manipulation

LEAP: Learning Emergent Active Perception for Quadruped Navigation