HERON: Real-Time Obstacle-Aware Reachability Maps for Mobile Manipulators

Many stages of the robotic lifecycle, from morphology synthesis to operation, rely fundamentally on the reachable workspace. However, current methods for approximating workspaces are slow, imprecise, or tied to a single morphology. We introduce Reachability Across Morphologies (RAM): a morphology-conditioned, implicit neural representation that acts as a fast, differentiable surrogate for pose reachability, generalising to unseen morphologies while inherently accounting for self-collisions. RAM is trained on a large-scale dataset of $3\cdot10^{10}$ samples generated solely from forward kinematics, which we publish alongside the model. Experiments show that RAM reaches 84% balanced accuracy at nanosecond inference, 15 percentage points higher than a generative inverse kinematics baseline that also generalises across morphologies, while running three orders of magnitude faster. This speed carries over to downstream tasks, accelerating morphology and trajectory optimisation by one to two orders of magnitude.

Predicting the short-term future trajectory of a person from RGB-D cameras is a key capability for mobile robots navigating in dynamic and human populated environment. The key challenge to deploying learned predictors on real robots is the train-deploy pipeline gap. Training data are often prepared with an idealized, smoothed ground truth, whereas at inference, the model is fed noisy estimates from pose detectors and depth sensors. The model is therefore evaluated on an input distribution it never encountered during training. Most existing models rely on recurrent neural networks (RNNs) or transformers, which either struggle to capture short-term temporal dependencies or suffer from computational cost. We present TrajMamba, a lightweight, bidirectional state space model for short-term human trajectory prediction using an RGB-D camera. The model uses a single shared processing path for both training and deployment, aligning human pose extraction, depth back projection, and smoothing processes. It encodes a short window of persons's hip center velocities with stacked bidirectional Mamba blocks and decodes future velocities with a GRU head. On the ETH/UCY benchmark, TrajMamba reaches an average ADE of 0.29 m FDE of 0.51 m. With 262K parameters, it runs in real time on modest hardware and is deployed on a TIAGo robot with an onboard NVIDIA Jetson Orin.

Dynamic obstacle avoidance in unstructured outdoor environments remains a critical challenge for autonomous mobile robots, particularly when large-scale robot-specific training data and simulation-based policies are impractical. We present a data-efficient, interpretable method for vision-based dynamic obstacle avoidance that operates entirely on real-world data, avoiding the sim-to-real transfer problem inherent in simulation-trained policies. Our approach leverages UniDepth, a large pretrained monocular depth estimation model, to produce dense depth maps from RGB video without requiring stereo cameras or LiDAR at inference time. Dynamic obstacle avoidance is achieved by extending the SuperPoint and SuperGlue feature correspondence pipeline to track keypoints across long frame sequences, projecting their 2D pixel-space positions into 3D using camera intrinsics and predicted depth, running bundle adjustment initialized from these 3D keypoints, and computing per-keypoint time-to-collision (TTC). A 2D motion primitive in the ground plane is then selected to move the robot away from the closest point of approach of the minimum-TTC keypoint. Evaluated on real-world data from the M3ED dataset, our pipeline achieves a precision of 0.49 and a recall of 0.38 in identifying frames with a ground truth TTC below 1 second, and correctly generates the evasive motion direction in 84% of true positive detections. Crucially, it detects at least one frame with TTC less than 1 second for 20 out of 22 unique physical obstacles present in our test sequences. Unlike end-to-end learned methods that demand thousands of hours of robot-specific training data, our approach eliminates model training entirely, requiring only 74 seconds of data for hyperparameter tuning. This demonstrates exceptional data efficiency while preserving interpretable and generalizable behavior across diverse obstacle types.