HERON: Real-Time Obstacle-Aware Reachability Maps for Mobile Manipulators

G. Dupoiron; F. Michaud; J. P. Roberge

cs.ROOct 28, 2026

HERON: Real-Time Obstacle-Aware Reachability Maps for Mobile Manipulators

Authors: G. Dupoiron, F. Michaud, J. P. Roberge

Organizations: Command and Robotics Laboratory, ´Ecole de technologie sup´erieure, Montreal, H3C 1K3, Quebec, Canada. · Interdisciplinary Institute for Technological Innovation (3IT), Department of Electrical Engineering and Computer Engineering, Universit´e de Sherbrooke, Sherbrooke, J1K 0A5, Quebec, Canada.

Abstract

Mobile manipulator robots rarely reposition their base during manipulation, because doing so requires real-time knowledge of how well the arm can reach its surroundings among obstacles. Reachability maps (RMs) encode this knowledge, but existing methods are slow, coarse (typically 5 cm), or ignore real obstacles. This article asks whether a neural network can predict an obstacle-aware RM at 2 cm resolution from a live depth camera, on embedded hardware, fast enough for online use. We present HERON, a 3D U-Net trained on RMs computed with the GPU-accelerated inverse kinematics of cuRobo, first on synthetic scenes and then on real depth data, and deployed on a Jetson AGX Orin with TensorRT and INT8 quantization. HERON predicts the RM of a 2.56 m cube around the arm in 115 ms on average, for a measured system rate of 8.57 Hz, close to the 10 Hz update rate of the obstacle map, whereas computing the same map with cuRobo takes about 90 min. Against the cuRobo ground truth, 95% of the voxels lie within 1 class out of 51 with the FP32 model and within 3 classes with the INT8 model, with a Pearson correlation of 0.99 and a quadratic weighted kappa above 0.9. Accuracy degrades mainly at the workspace boundaries.

Figures & tables

Figure 1: Experimental setup. HERON runs on a Jetson AGX Orin onboard an MM composed of a Clearpath Ridgeback omnidirectional base, a Doosan M1013 arm, and an Azure Kinect depth camera. A vertical slice of the generated RM is overlaid on the photograph, aligned with an ARTag marker.

Figure 2: Ground-truth RMs of four arms computed with cuRobo , without obstacles, on a $152^{3}$ grid at 2 cm resolution (vertical slice in the plane $x=0$ ). From left to right: Doosan M1013, UR5e, Jaco7 and iiwa. Color gives the reachability score $R_{i}$ , from 0 (no reachable orientation) to 1 (all 50 orientations reachable).

Panel of Figure 3 — Figure 3: HERON’s architecture, a 3D U-Net. A $128^{3}$ obstacle map is downsampled by four max-pooling stages (16 to 256 feature maps), then upsampled by transposed convolutions to an RM of the same size. Skip connections (green) preserve fine spatial detail; each block applies convolution, normalization, and activation (purple arrows).

Figure 4: Visualization of the evaluation regions on an example generated with cuRobo : input obstacle map (top-left), ground-truth RM (top-right), reachable region (bottom-left, $R_{i}\geq\frac{1}{50}$ ), and ridge region (bottom-right, top 1% of voxels by gradient magnitude).

Metric		Global	Reachable	Ridge
MAE	Mean	0.00657	0.0148	0.0207
MAE	Std	0.00108	0.00251	0.00747
RMSE	Mean	0.0154	0.0231	0.0393
RMSE	Std	0.00403	0.00605	0.0186
Pearson $r$	Mean	0.999	0.997	0.980
Pearson $r$	Std	0.000585	0.00165	0.0216

Table 1: Result of regression metrics (MAE, RMSE, Pearson

r

) and ordinal metric (QWK) obtained with the FP32 3D U-Net model, across evaluated workspace regions.

Figure 5: Fraction of voxels whose predicted class lies within $\pm k$ classes of the ground truth, for the global, reachable and ridge regions (FP32 model). The dashed line marks 95%; dotted lines mark the tolerance at which each region reaches it.

	FP32	FP16	FP16 TRT	INT8 TRT
Prediction time $t_{p}$ , mean (ms)	872.2	575.5	146.1	115.0
Prediction time $t_{p}$ , std (ms)	160.3	82.7	20.8	34.2
System rate $f_{s}$ (Hz)	1.18	1.71	6.46	8.57
Tolerance for 95% accuracy (classes)	$\pm 1$	$\pm 1$	$\pm 1$	$\pm 3$
MAE	0.006566	0.006573	0.006573	0.014251
RMSE	0.015390	0.015389	0.015389	0.031833

Table 2: Prediction time

t_{p}

(network inference only), system rate

f_{s}

(whole processing chain, measured on the output ROS topic) and accuracy on the Jetson AGX Orin for each inference optimization level.

Mobile Manipulation

Mobile manipulation studies robots that navigate through environments while using arms or other effectors to interact with objects.

See all 93 papers

RAM: Reachability Across Morphologies
Jun 8, 2026Tim Walter, Xinyu Chen, Jonathan Külz +1Reachability Analysis Implicit Neural Representations
Department of Computer Engineering, Technical University Munich, Germany · German Electron Synchrotron, University of Hamburg, Germany
Real-Time Human Trajectory Prediction from RGB-D for Mobile Robots Using Bi-Mamba
Oct 24, 2026Basheer Al-Tawil, Magnus Jung, Ayoub Al-HamadiHuman Trajectory Prediction State Space Models
Neuro Information Technology, Otto von Guericke University, 39106 Magdeburg, Germany
Time-to-Collision Based Dynamic Obstacle Avoidance Using Pretrained Vision Models for Robots in Unstructured Environments
Jul 8, 2026Erik Jagnandan, Mulugeta Haile, Gregory Barber +1Collision Avoidance Dynamic Obstacle Avoidance
GRASP Laboratory, University of Pennsylvania, PA · U.S. Army Research Laboratory, Aberdeen Proving Ground, MD

Jun 8, 2026J/K move · Enter open · S save

RAM: Reachability Across Morphologies

Tim Walter, Xinyu Chen, Jonathan Külz, Matthias Althoff

Department of Computer Engineering, Technical University Munich, Germany · German Electron Synchrotron, University of Hamburg, Germany

Reachability Analysis Implicit Neural Representations Robotic Optimization Robot Motion Planning Robot Kinematics

Open paper AskPDF arXiv

Many stages of the robotic lifecycle, from morphology synthesis to operation, rely fundamentally on the reachable workspace. However, current methods for approximating workspaces are slow, imprecise, or tied to a single morphology. We introduce Reachability Across Morphologies (RAM): a morphology-conditioned, implicit neural representation that acts as a fast, differentiable surrogate for pose reachability, generalising to unseen morphologies while inherently accounting for self-collisions. RAM is trained on a large-scale dataset of

3\cdot10^{10}

samples generated solely from forward kinematics, which we publish alongside the model. Experiments show that RAM reaches 84% balanced accuracy at nanosecond inference, 15 percentage points higher than a generative inverse kinematics baseline that also generalises across morphologies, while running three orders of magnitude faster. This speed carries over to downstream tasks, accelerating morphology and trajectory optimisation by one to two orders of magnitude.

Abstract

Mobile Manipulation

Explore similar work

RAM: Reachability Across Morphologies

Real-Time Human Trajectory Prediction from RGB-D for Mobile Robots Using Bi-Mamba

Time-to-Collision Based Dynamic Obstacle Avoidance Using Pretrained Vision Models for Robots in Unstructured Environments