Object- and Task-Aware Active Perception for Safe Navigation of Mobile Manipulators in Semi-Static Environments
Authors: Jingxing Qian, Xintong Du, Jun Yang, Siqi Zhou, Steven L. Waslander, Angela P. Schoellig
Organizations: University of Toronto Institute for Aerospace Studies (UTIAS) and the Vector Institute for Artificial Intelligence, Canada. · Epson Canada Ltd., Toronto, ON L3R 6G3, Canada. · Learning Systems and Robotics Lab at the Technical University of Munich and the Munich Institute for Robotics and Machine Intelligence (MIRMI), Germany. · School of Computing Science, Faculty of Applied Sciences, Simon Fraser University, Burnaby, BC, Canada. · Institute for Aerospace Studies (UTIAS) and Robotics Institute, University of Toronto, Canada.
Mobile manipulators must navigate safely in cluttered, partially unknown environments across logistics, manufacturing, and service applications. Semi-static changes between robot visits can lead to outdated scene knowledge and increased collision risk. We propose an active perception framework that enables a high degree-of-freedom (DoF) mobile manipulator to inspect uncertain or safety-critical regions with its hand camera during navigation. A probabilistic, object- and task-aware next-best-view (NBV) formulation prioritizes observations near the desired path and around uncertain or potentially changing objects. The perception objective is integrated into hierarchical-task model predictive control (HTMPC), exploiting kinematic redundancy for viewpoint adjustment while preserving the higher priority of base-trajectory tracking. The system unifies navigation, map maintenance, and active perception in a closed loop. Real-world tests show improved map consistency, reduced task-dependent map entropy, and lower navigation risk than passive or non-task-aware baselines. Additional simulation and real-world sequences demonstrate responses to controlled semi-static scene changes.
Figures & tables
Fig. 1 : A snapshot of the proposed active perception and navigation framework for high-degree-of-freedom (DoF) mobile manipulators in action. Equipped with a hand camera, the robot operates in cluttered, semi-static environments. It uses its kinematic redundancy to reposition the camera around obstacles and observe uncertain or occluded regions near its navigation path. See the multimedia attachment or online videos 6 6 6 Video demonstrations: https://tinyurl.com/nbv-for-mm-nav . for demonstrations.
Fig. 2 : Qualitative demonstration of the proposed active perception and navigation framework for mobile manipulation in action (left) compared with a baseline method that randomly points the hand camera (right). Red curves represent the planned base paths. Shaded blue regions represent the hand camera’s FoV. Yellow exclamation marks highlight hidden hazards such as a blind corner (top) and a hidden pole (bottom).
Fig. 3 : An overview of the proposed active perception and navigation framework for high-DoF mobile manipulators.
Fig. 4 : Our mobile manipulator setup. The system consists of a Universal Robots UR10 arm mounted on a Clearpath Robotics Ridgeback omnidirectional base. Two synchronized Orbbec Femto Bolt RGB-D cameras are installed, with one mounted on the base and the other mounted on the hand. An onboard laptop runs all the perception, planning and control algorithms.
Fig. 5 : Detailed schematic of the proposed active perception module.
Fig. 6 : Real-world static-scene benchmark. Left: Scene 1 Easy Loops. Middle: Scene 2 Hard L-Shaped Walls. Right: Scene 3 Hard Office Scene. The scenes progress from open loops to sharper turns and cluttered occlusions, where hidden hazards near the waypoints make active map maintenance important.
Fig. 7 : Bird’s-eye-view 3D reconstruction results in the real-world static-scene benchmark. Rows correspond to Scene 1–3. Green, yellow, and red indicate true positives, false negatives, and false positives, respectively. The proposed method recovers more occupied voxels in blind and path-critical regions.
Scene 1 Easy Loops
Precision ↑
Recall (TPR) ↑
FPR ↓
Outcome
Fixed
81.0
56.1
2.2
Success
Lookahead
81.7
59.6
2.3
Success
NBV-Map
81.8
61.9
2.2
Success
Random
77.2
56.2
2.8
Success
NBV-Map-Task (ours)
87.4
70.9
1.7
Success
Signed improvement
5.6
9.0
0.5
TABLE I : Quantitative mapping results from the real-world static-scene benchmark. The signed improvement compares ours with the best baseline: ours minus baseline for precision and recall, and baseline minus ours for FPR; positive values favor ours.
Fig. 8 : Task-dependent map entropy along executed paths in the real-world static-scene benchmark. Left: boxplots over the three loops in each scene. Right: representative top-down views with trajectories colored by the local entropy. Lower values indicate lower entropy in the regions traversed by the robot.
Fig. 9 : Safety margin in the real-world static-scene benchmark. Boxplots show the point-cloud clearance proxy over all loop executions. The dashed red line marks zero clearance under this proxy, corresponding to a near-contact configuration. Larger positive values indicate greater estimated clearance, especially in the hard turning scenes.
Method
Success Rate ↑
Fixed
33.3%
Lookahead
66.7%
NBV-Map
33.3%
Random
33.3%
NBV-Map-Task (ours)
100.0%
TABLE II : Completion percentage across nine sequential loop executions per method: three loops in each of three scenes.
Fig. 10 : Controlled Isaac Sim scene-change experiment. The red curves are the executed base trajectories, and the blue arrows show the sampled hand camera headings. After one row of boxes is shifted, the hand camera redirects toward that side to acquire additional observations.
Fig. 11 : Consistency belief and task-dependent map entropy for two representative box regions in the controlled scene-change simulation shown in Fig. 10 . Vertical dashed lines mark loop boundaries, and shaded regions indicate intervals when the robot is away from the boxes and receives no observations of them. Each box changes primarily when its side of the corridor is displaced.
Fig. 12 : Real-world long-term scene-change experiment over four visits. In each visit, the robot re-enters the room with the maintained object map, acquires new observations near the shelf, updates the map, and replans according to the current shelf position, switching between the outside and inner routes.
Safe robot navigation develops methods for reaching goals while avoiding collisions and managing uncertainty, dynamic obstacles, and environmental hazards.
As compared to typical mobile manipulation tasks, sequential mobile manipulation poses a unique challenge -- as the robot operates over extended periods, successful task completion is not solely dependent on consistent motion generation but also on the robot's awareness and adaptivity to changes in the operating environment. While existing motion planners can generate whole-body trajectories to complete sequential tasks, they typically assume that the environment remains static and rely on precomputed maps. This assumption often breaks down during long-term operations, where semi-static changes such as object removal, introduction, or shifts are common. In this work, we propose a novel perceptive hierarchical-task model predictive control (HTMPC) framework for efficient sequential mobile manipulation in unstructured, changing environments. To tackle the challenge, we leverage a Bayesian inference framework to explicitly model object-level changes and thereby maintain a temporally accurate representation of the 3D environment; this up-to-date representation is embedded in a lexicographic optimization framework to enable efficient execution of sequential tasks. We validate our perceptive HTMPC approach through both simulated and real-robot experiments. In contrast to baseline methods, our approach systematically accounts for moved and phantom obstacles, successfully completing sequential tasks with higher efficiency and reactivity, without relying on prior maps or external infrastructure.
Xintong Du, Jingxing Qian, Siqi Zhou +1
University of Toronto Institute for Aerospace Studies (UTIAS) and the Vector Institute for Artificial Intelligence, Canada. · School of Computing Science, Faculty of Applied Sciences, Simon Fraser University, Burnaby, BC, Canada. · Learning Systems and Robotics lab at the Technical University of Munich and the Munich Institute for Robotics and Machine Intelligence (MIRMI), Germany.
Multi-agent manipulation naturally produces multiple task-driven viewpoints: every arm carries a wrist camera and moves through the scene while acting. Yet these observations are typically underutilized, and active perception in manipulation is still often treated as requiring a dedicated sensing agent. We introduce MAAP (Multi-Agent Active Perception), in which every arm is dual-purpose: it executes manipulation actions and, through the wrist camera it carries, simultaneously serves as a moving viewpoint for the team. We pair this with RAIL (Role-Aware Imitation Learning), a controller that predicts each arm's current role alongside its action chunk and conditions action generation on it, representing role-dependent actions within one network. Across four simulated tasks, widening the perception regime lifts average success from 56.5% with a fixed camera to 62.5% with one active wrist view and 70.0% with all of them, while MAAP+RAIL reaches 79.2%. RAIL's additional gain is concentrated on the three-arm Microwave task, where success rises from 47% to 82% on identical multi-wrist inputs. On a dual-arm platform, MAAP+RAIL succeeds in 14 of 20 placement trials compared with 0 of 20 for fixed-view ACT. Collaborative manipulation can thus serve as an active perception mechanism in its own right.
Bruno N. Y. Chen, Li Kang, Heng Zhou +4
Carnegie Mellon University · Shanghai Jiao Tong University · University of Science and Technology of China +3
Active perception allows autonomous agents to select their viewpoints rather than passively process the viewpoints given to them, enabling them to target where to reduce uncertainty about their environment. Learned systems typically encourage this behavior with hand-designed proxy objectives, such as coverage or curiosity bonuses, that may conflict with the task. In this work, we propose a method to learn emergent active perception (LEAP) without augmentation of the task objective. We formulate the problem of goal-oriented navigation over hazardous terrains with goals that must be discovered visually. We then propose an architecture for navigation policies with active perception, and train them on a terrain curriculum where task pressure alone leads to the emergence of gaze control. Key to this emergence, LEAP works on a gaze-invariant representation that integrates depth images into egocentric belief maps. We validate its performance in held-out evaluation scenarios, where it achieves a 92.7% success rate, compared to 74.2% for scripted or 34.5% for passive perception, and comes within 4.6 points of a privileged oracle. We validate that LEAP navigation policies, unchanged, can be directly applied to steering quadrupedal locomotion policies in physics simulation.
Ü. Bora Gökbakan, Stéphane Caron, Philippe Souères