HINT-VLM: Humanoid Interactive Navigation over Rough Terrain with Embodiment-Grounded Vision-Language Model
Authors: Ziwon Yoon, Seok Hwan Kim, Yiyu Chen, Quan Nguyen, Lu Gan, Ye Zhao
Organizations: Institute for Robotics and Intelligent Machines, Georgia Institute of Technology, Atlanta, GA 30332, USA · VinMotion · Department of Aerospace and Mechanical Engineering, University of Southern California, Los Angeles, CA 90089, USA
Humanoid robots have achieved significant progress in locomotion and manipulation, enabling traversal over rough terrain and richer interaction with the environment, such as moving obstacles. However, most traditional navigation methods assume non-interactive environments and focus only on traversal. While Navigation Among Movable Obstacles (NAMO) approaches relax this assumption, prior efforts have largely focused on flat environments and reactive interaction, rather than predicting object movability and planning ahead. These limitations become critical in outdoor rough-terrain scenarios, where robots must jointly reason about terrain traversability and object movability. In this work, we address NAMO for humanoid robots in cluttered outdoor rough terrain by proposing a hybrid planning framework that combines semantic priors from a Vision-Language Model (VLM) with embodied robot interaction data through a Gaussian process (GP) residual model. The GP corrects the VLM-derived movability prior to produce embodiment-grounded probabilistic movability estimates, which are then incorporated into hierarchical uncertainty-aware planning. The proposed approach allows humanoids to jointly leverage locomotion and manipulation for efficient and safe navigation, representing the first solution to NAMO in rough terrain through predictive movability reasoning and probabilistic VLM integration.
Figures & tables
Figure 1: Overview of HINT-VLM. The proposed framework uses an embodiment-grounded VLM-GP to predict object movability from visual observations, maps objects and reconstructs the underlying terrain, and incorporates the resulting information into an uncertainty-aware global planner. A finite-state machine switches among path tracking, interaction, and failure recovery.
Figure 2: Overview of the proposed VLM-GP training and inference. The residual GP is trained on simulated robot interaction outcomes and ground-truth physical attributes, while perceived attributes are used at inference for embodiment-grounded movability estimation.
Figure 3: Spatial grounding and terrain reconstruction for a candidate object. GSAM uses the semantic label to obtain the object mask, which is applied to the aligned depth image. The masked region is then inpainted with LaMa, and both the object mask and reconstructed region are lifted into 3D to obtain the object footprint and reconstructed terrain for planning.
Figure 4: Illustration of uncertainty-aware path planning. The solid green path leads to the object, while the dotted and dashed green paths represent the routes following successful and failed interactions, respectively. The orange path denotes a non-interactive detour.
Model
Real
Sim
All
Tried
OOD
AUROC ↑
VLM-GP
0.868
0.770
1.000
0.702
LoRA fine-tuned
0.817
0.740
0.960
0.712
Vanilla VLM
0.827
0.681
1.000
0.662
DINOv3
0.736
0.695
0.880
0.726
Table I: Object movability prediction performance on the simulation validation set (Sim) and real-world test set (Real), where Real is further divided into the Tried and Out-of-Domain (OOD) subsets. Brier scores are computed from raw predictions for Sim and from calibrated predictions for Real.
Figure 5: Overview of the IsaacSim benchmark environments. (a) Garden : The robot starts on the white tiles and reaches the goal through the left opening or right dirt path, both blocked by movable obstacles. (b) Construction : The wooden plank bridge offers a shorter route across uneven terrain than the detour along the concrete boundary. (c) Park : The shorter route is blocked by an object on highly uneven terrain, where pushing it may cause the robot to trip.
Garden
Construction
Park
Planner
N-SR
T (Mean, s)
T (CVaR 0.9 , s)
I-SR
N-SR
T (Mean, s)
T (CVaR 0.9 , s)
I-SR
N-SR
T (Mean, s)
T (CVaR 0.9 , s)
I-SR
Proposed
95.5%
14.6 (+16%)
20.0 (+13%)
91.1%
97.7%
20.3 (+5.7%)
24.9 (+0%)
100%
100%
17.8 (+0%)
19.0 (+0%)
-
Pro-NoRecon
95.5%
14.4 (+14%)
18.7 (+5.6%)
93.5%
93.2%
21.2 (+10%)
33.0 (+33%)
96.0%
86.4%
18.9 (+6.5%)
28.6 (+51%)
0.0%
Pro-VanillaVLM
93.2%
16.5 (+31%)
30.4 (+72%)
84.0%
97.7%
22.4 (+17%)
25.2 (+1.4%)
90.0%
93.2%
17.8 (+0.3%)
19.7 (+3.7%)
-
Pro-NoUnc (Best w )
95.5%
14.4 (+14%)
21.0 (+19%)
100%
97.7%
20.0 (+4.0%)
32.6 (+31%)
84.6%
100%
18.0 (+1.1%)
20.9 (+10%)
0.0%
STATE-NAV
0% †
-
-
-
100%
24.6 (+28%)
25.0 (+0.5%)
-
97.7%
17.8 (+0%)
19.3 (+1.7%)
-
Table II: Navigation performance in Garden , Construction , and Park . N-SR denotes goal-reaching success with body tilt at most 45∘ . Mean T and T -CVaR 0.9 are computed over successful runs, with the latter averaging the slowest 10%. I-SR denotes the success rate of attempted interactions. For Pro-NoUnc, w is tuned independently for each environment based on N-SR. Bold indicates the best value in each column, and parentheses show relative degradation computed from unrounded values. † STATE-NAV cannot find a path because obstacles block all routes. A dash indicates unavailable metrics, including I-SR when no interaction is attempted.
Figure 6: Illustration of the real-world testing environment with path plan candidates. The green dashed line represents a route to the goal that involves interaction with a black wagon, while the orange line represents an alternative detouring route.
Figure 7: Outdoor testing environment (22 m × 41 m) with a shortcut path blocked by a chair. The green path indicates the route selected by the robot, which clears the chair to take the shortcut. The inset on the lower right shows a top-down view of the entire test site, including the selected shortcut path and the alternative orange detour path.
Figure 8: Real-world terrain reconstruction on rough terrain. (a) Egocentric view of a chair resting on an artificial rough-terrain setup constructed in the indoor environment shown in Fig. 6 . (b) BEV elevation map measured with the chair present. (c) Terrain map reconstructed using the method in Section IV . (d) BEV elevation map measured after the chair is removed.
Figure 9: Sensitivity to w across three environments. The top and bottom rows show navigation failure rate ( 1−N-SR ) and T -CVaR 0.9 , respectively. Lower values indicate better performance for both metrics. Dashed lines represent Pro-NoUnc with varying w , while solid lines represent the proposed method without weight tuning.
Planner
N-SR
T (Mean, s)
Distance (Mean, m)
Proposed
100%
23.2 (+0%)
10.3 (+0%)
Pro-VanillaVLM
100%
29.2 (+26%)
16.7 (+62%)
STATE-NAV
100%
27.6 (+19%)
16.8 (+61%)
VAMOS
0% †
-
-
Pro-NoRecon
0% †
-
-
Table III: Real-world navigation performance. N-SR denotes goal-reaching success without safety intervention by a human operator. Mean navigation time ( T ) and distance are computed over successful runs. Bold indicates the best value in each column, and parentheses show relative degradation from the best value. † VAMOS and Pro-NoRecon are evaluated only on the chair and wagon. All attempted trials require early termination due to imminent instability, yielding 0% N-SR on this limited subset. Further testing is discontinued for hardware safety. A dash indicates unavailable metrics or no interaction attempts.
Terrain assessment is a critical capability for off-road mobile robots, enabling safe and reliable navigation through unstructured and geometrically complex environments. Conventional geometry-based terrain assessment is fast to compute but often overly conservative in unstructured environments. We present PIVOT: a Physically Informed Vision-Language Off-Road Traversability navigation system that augments conventional geometry-based planning with vision-language-model (VLM)-based semantic reasoning for field robots. To physically ground this assessment, we quantify how strongly the VLM's predicted traversal energy cost, robot vibration, and wheel slip correlate with real-world measurements and introduce a unified traversability score that weights each modality by its prediction-measurement correlation. For efficiency, we design a two-level navigation architecture that retains geometry-based planning as the nominal mode and invokes semantic replanning only when that mode fails to find a path. Across five repeated closed-loop trials on a mixed-terrain route totalling around 6.4 km, the proposed system increases overall autonomy from 59.6% to 97.0%, reduces human interventions from 11 to 3, and increases the mean distance between interventions (MDBI) from 69.2 m to 412.9 m compared with geometry-only navigation. These results demonstrate that physically grounded VLM-based terrain assessment can substantially extend autonomous navigation beyond the limitations of geometry alone, while preserving efficient geometric planning as the nominal mode.
Real-world environments are messy, with paths often blocked by everyday objects like chairs and toys, requiring robots to perform interactive navigation by manipulating obstacles to clear a path. Existing methods rely on extensive simulation-based learning or per-scene instructions specifying which objects to avoid or manipulate. They struggle to generalize across novel objects, diverse environments, and varied contexts such as a cart of fragile glassware that should be avoided while an empty one can be pushed. We formalize this problem as open-world interactive navigation and present MessyNav, a zero-shot framework based on vision-language models (VLMs). At its core, MessyNav performs explicit VLM-based affordance reasoning before task planning to decide whether to swipe, pick up, or avoid an object and produce a high-level plan. Evaluated on a mobile manipulator across real-world indoor and outdoor environments, MessyNav improves success rate by 27 percentage points (pp) over the strongest baseline in real-robot trials and achieves statistically significant gains in a larger-scale offline evaluation, without task-specific training.
This paper presents TRACE, a traversability-aware reactive navigation framework designed for the autonomous operation of ground robots in unstructured forest environments. Unlike traditional approaches, TRACE leverages VLM-based confidence maps from positive and negative natural-language prompt sets and combines them through evidential fusion to produce a dense traversability map that prioritizes the terrain based on signs of previously traversed paths. Traversability-annotated points are then extracted and used to derive the left and right boundaries of the traversable path via percentile binning followed by spline or regularized polynomial fitting. These boundaries are then incorporated as soft barrier constraints within an NMPC formulation, enabling safe and adaptive path following in complex natural environments. Rigorous experimental trials carried out in real-life forest environments demonstrate TRACE ability for smooth navigation in cluttered, deformable, and visually ambiguous terrains.
Mario A. V. Saucedo, Midhun Edatharam Kunnath, Elias Small +4
Robotics and AI Team, Lule˚a University of Technology, Lule˚a, SE-97187, Sweden. · Korea Advanced Institute of Science & Technology (KAIST), Daejeon, 34141, South Korea.