What the Elevation Map Cannot See: Semantic-Aware Locomotion and Execution-Aware Navigation for Humanoid Robot

Shunyu Yao; Songyang Liu; Dinghao Chen; Yuanyuan Lei; Shuai Li

cs.ROOct 5, 2026

What the Elevation Map Cannot See: Semantic-Aware Locomotion and Execution-Aware Navigation for Humanoid Robot

Authors: Shunyu Yao, Songyang Liu, Dinghao Chen, Yuanyuan Lei, Shuai Li

Organizations: Engineering School of Sustainable Infrastructure & Environment, University of Florida, Gainesville, FL 32611, USA · Independent researcher · Department of Computer & Information Science & Engineering, University of Florida, Gainesville, FL 32611, USA

Abstract

Navigation for humanoid robots is critical, yet large-scale evaluation on physical hardware is often impractical due to cost and safety concerns, making simulation benchmarks essential. Existing VLN benchmarks achieve physically executable navigation, but still assume (1) all hazards are observable from elevation maps; (2) realized motions closely match desired motions. In real environments, however, fallen bottles may be ambiguous in elevation maps, while phones and water spills may be difficult to differentiate; hazard avoidance by the locomotion policy can cause the robot's actual trajectory to deviate from the path intended by the VLN policy. Such command-execution mismatch can accumulate and lead the robot toward unintended locations. To expose these failure modes, we introduce a benchmark that models both elevation-subtle hazards and execution deviations, together with a closed-loop VLN + locomotion control framework that continuously realigns high-level navigation with the robot's actual state. We evaluate navigation in simulation and further validate the locomotion policy on a physical Unitree G1 humanoid robot. Results show that semantic input reduces contact with hazards poorly represented in elevation maps, while anti-deviation improves navigation success. These findings highlight the need to evaluate humanoid navigation jointly in terms of route completion and hazard avoidance.

Figures & tables

Figure 2: Method overview. (a) A semantic-aware teacher learns locomotion from ground-truth elevation and semantic maps. (b) DAgger distills the teacher into an RGB-D student with elevation and semantic branches. (c) We use the high-level VLN module from NaVILA [ 3 ] to generate relative target poses, which the anti-deviation module helps track using localization feedback.

Benchmark	Simulator	VLN	Physics	Humanoid	Geometric obstacles on path	Semantic hazards on path	Dynamics variations & external disturbances
VLN-CE [ 10 ]	Habitat	✓	✗	✗	✗	✗	✗
HumanoidBench [ 18 ]	MuJoCo	✗	✓	✓	✗	✗	✗
VLN-PE [ 19 ]	Isaac Sim	✓	✓	✓	✗	✗	✗
VLN-CE-Isaac [ 3 ]	Isaac Sim	✓	✓	✓	✗	✗	✗
Ours	Isaac Sim	✓	✓	✓	✓	✓	✓

Table 1: Comparison of Navigation Benchmarks

Figure 3: Testing environments. (a) Navigation in simulation; (b) Locomotion in simulation; (c) Real-world with external payload/push and lighting conditions.

Figure 4: Training curriculum for semantic-aware locomotion. An elevation-based policy provides a warm start for learning hazard avoidance using ground-truth elevation and semantic maps on challenging terrains with obstacles. The teacher is then fine-tuned under domain randomization and external disturbances to improve robustness. Finally, teacher-student distillation transfers this capability to a deployable policy using waist-mounted RGB-D observations. Colored markers indicate the observations and training conditions used at each stage.

Figure 5: (a) Terrain mesh from detailed offline handheld scanning, (b) reconstructed elevation map, and (c) RGB observation.

Figure 6: Selected Grad-CAM visualizations conditioned on right-leg actions. (a) Depth observation during stake traversal. (b) Corresponding Grad-CAM visualization. (c) RGB observation while stepping over semantic obstacles. (d) Corresponding Grad-CAM visualization.

Figure 7: Two complementary reward terms for stepping over hazards. Top: rewarding sufficient foot clearance during horizontal swing motion. Bottom: penalizing premature descent while the swing foot still overlaps a hazard.

Figure 8: Command-execution mismatch. When no safe foothold is available along the direction toward the VLN-commanded waypoint (red), the robot detours around hazards, producing a deviated trajectory (green). The anti-deviation module detects this mismatch through localization feedback and adjusts the velocity commands to steer the robot back (blue) toward the intended target pose.

Group	Terrain; increasing difficulty (L1 $\rightarrow$ L3)
Uneven Ground	Ground with obstacles, hazards, pits, and bumps; more/larger obstacles, greater height variation.
Stairs	Descent/ascent from elevated/recessed starts; taller steps.
Stakes	Isolated stake-top footholds over deep gaps; smaller footholds, wider gaps.
Stonebridge	Narrow stepping stones without lateral support; narrower/shorter slabs, wider gaps.

Table 2: Terrain Groups and Difficulty Levels

Category	Magnitudes at scale $1.0$
Dynamics	Friction: $0.4$ – $1.0$ ; $k_{p}/k_{d}$ : $\times 0.85$ – $1.15$ ; motor: $\times 0.9$ – $1.1$
Payload	Mass offset: $\pm 2$ kg; center-of-mass offset: $\pm 5$ cm
Disturbances	Push: $\pm 0.8$ m/s every $3$ – $7$ s; sustained force/torque: $\pm 5$ N / $\pm 1.67$ N $\cdot$ m
Proprio. noise	Angular vel.: $\pm 0.5$ rad/s; gravity proj.: $\pm 0.05$ ; joint pos./vel.: $\pm 0.05$ rad / $\pm 2$ rad/s
Image noise	Depth: $\sigma=2$ cm, $2\%$ dropout; RGB gain: $0.65$ – $1.35$ ; RGB noise: $\sigma=0.02$
Lighting	Dome/sun intensity: $450$ – $1250$ / $0$ – $1200$ ; color temp.: $3000$ – $8000$ K

Table 3: Locomotion Perturbations at Scale

1.0

Elevation Visibility	Hazard	Height (m)	Footprint (m)
Visible	Crate	$0.06$ – $0.24$	$0.25$ – $0.50\times 0.30$ – $0.60$
Visible	Box	$0.05$ – $0.18$	$0.15$ – $0.35\times 0.15$ – $0.35$
Ambiguous	Fallen bottle	$0.071$	$0.24\times 0.071$
Invisible	Phone	$0.014$	$0.163\times 0.079$
Invisible	Water/oil stain	$0$	$0.10$ – $0.45\times 0.09$ – $0.40$

Table 4: Hazard Types and Elevation Visibility

Method

Diff.

Clean

Perturbation

0.5\times

Perturbation

1.0\times

Perturbation

1.5\times

Success rate (SR)

\uparrow

Progress

\uparrow

Velocity tracking error (VTE)

\downarrow

Success rate (SR)

\uparrow

Progress

\uparrow

Velocity tracking error (VTE)

\downarrow

Success rate (SR)

\uparrow

Progress

\uparrow

Velocity tracking error (VTE)

\downarrow

Success rate (SR)

\uparrow

Progress

\uparrow

Velocity tracking error (VTE)

\downarrow

Oracle

99.0\pm 1.3

99.7\pm 0.4

0.103\pm 0.001

100.0\pm 0.0

100.0\pm 0.0

0.105\pm 0.001

98.8\pm 0.8

99.5\pm 0.3

0.116\pm 0.002

91.0\pm 3.1

97.3\pm 1.4

0.144\pm 0.004

95.6\pm 4.0

98.2\pm 1.6

0.108\pm 0.002

96.5\pm 1.1

98.1\pm 0.9

0.111\pm 0.001

93.3\pm 0.8

97.3\pm 0.8

0.127\pm 0.003

86.2\pm 2.6

94.6\pm 2.0

0.149\pm 0.005

95.0\pm 0.8

98.1\pm 0.6

0.113\pm 0.001

91.2\pm 1.1

96.7\pm 0.5

0.117\pm 0.002

86.2\pm 2.5

95.0\pm 1.6

0.132\pm 0.002

79.8\pm 0.5

93.0\pm 0.8

0.156\pm 0.003

Elevation-only

47.9\pm 4.0

77.7\pm 2.3

0.137\pm 0.004

49.2\pm 2.9

77.4\pm 2.7

0.144\pm 0.003

40.6\pm 3.4

72.4\pm 2.8

0.167\pm 0.004

31.5\pm 4.3

65.2\pm 3.1

0.203\pm 0.005

32.7\pm 5.3

68.7\pm 3.5

0.155\pm 0.006

27.5\pm 3.4

66.5\pm 3.1

0.159\pm 0.003

20.4\pm 4.1

59.6\pm 2.2

0.182\pm 0.004

17.1\pm 3.6

54.9\pm 2.6

0.219\pm 0.006

Table 5: Simulated Locomotion Across Difficulty and Perturbation Levels

Method	Success rate (SR) $\uparrow$	Hazard contact rate (HCR) $\downarrow$			Fall rate (FR) $\downarrow$	Method	Success rate (SR) $\uparrow$	Hazard contact rate (HCR) $\downarrow$			Fall rate (FR) $\downarrow$
	Success rate (SR) $\uparrow$	Visible	Ambiguous	Invisible	Fall rate (FR) $\downarrow$		Success rate (SR) $\uparrow$	Visible	Ambiguous	Invisible	Fall rate (FR) $\downarrow$
Clean						Perturbation $0.5\times$
Oracle	$93.6\pm 3.0$	$2.2\pm 3.1$	$1.7\pm 2.7$	$1.9\pm 2.1$	$0.6\pm 0.7$	Oracle	$92.2\pm 3.4$	$2.5\pm 3.1$	$0.8\pm 0.7$	$4.4\pm 2.0$	$0.0\pm 0.0$
Elevation-only	$9.4\pm 2.4$	$27.8\pm 7.7$	$41.4\pm 13.2$	$21.1\pm 11.6$	$0.3\pm 0.6$	Elevation-only	$12.5\pm 2.9$	$29.4\pm 9.6$	$37.2\pm 7.6$	$20.3\pm 5.5$	$0.6\pm 0.7$
Ours	$\mathbf{50.6\pm 5.6}$	$24.7\pm 6.5$	$17.2\pm 5.8$	$6.1\pm 4.7$	$1.4\pm 1.2$	Ours	$\mathbf{52.8\pm 6.0}$	$23.1\pm 3.4$	$16.4\pm 5.9$	$7.2\pm 1.8$	$0.6\pm 0.7$
Perturbation $1.0\times$						Perturbation $1.5\times$

Table 6: Success and Failure Rates on Uneven Ground

Method	Clean SR $\uparrow$	Out of corridor $\downarrow$	Hazard Contact Rate(HCR) $\downarrow$
			Visible	Ambiguous	Invisible
Plastic flooring (dark-colored, very slippery)
Elevation-only	0	20	20	80	100
Sonic	20	80	0	40	40
Ours	40	20	0	20	20
Wooden boards (light brown, rough, narrow)

Table 7: Real-World Locomotion on Three Surfaces

Condition	Clean SR $\uparrow$
No disturbance	60
Flashlight (2000 lm)	40
Added payload (1Kg)	40
Pool-noodle strikes	60

Table 8: Robustness to Physical Disturbances

Method

Perturbation scale

5 obstacles

10 obstacles

Safe SR

\uparrow

Avg. Progress

\uparrow

Collision Rate

\downarrow

Hazard Intrusion

\downarrow

\downarrow

\uparrow

SPL

\uparrow

Final goal distance (m)

\downarrow

VLM early stop

\downarrow

Timeout

\downarrow

Safe SR

\uparrow

Avg. Progress

\uparrow

Collision Rate

\downarrow

Hazard Intrusion

\downarrow

\downarrow

\uparrow

SPL

\uparrow

Final goal distance (m)

\downarrow

VLM early stop

\downarrow

Timeout

\downarrow

Ours full

0.5

35.38\%

82.46\%

15.04\%

11.98\%

16.16\%

40.95\%

0.385

1.67

0.56\%

27.30\%

17.55\%

43.34\%

20.89\%

18.11\%

11.98\%

24.23\%

0.212

5.58

24.79\%

18.11\%

Elevation-only

0.5

23.12\%

72.13\%

22.84\%

20.61\%

25.07\%

30.08\%

0.250

3.05

3.62\%

18.38\%

13.65\%

48.04\%

29.25\%

27.86\%

21.73\%

21.17\%

0.155

5.60

16.71\%

11.14\%

Ours full

1.0

23.68\%

43.20\%

12.26\%

12.81\%

10.86\%

29.80\%

0.256

5.38

29.24\%

17.83\%

13.09\%

49.78\%

32.59\%

22.28\%

14.21\%

20.06\%

0.178

4.55

14.48\%

18.66\%

Elevation-only

1.0

17.27\%

39.89\%

20.89\%

22.84\%

19.50\%

23.96\%

0.185

6.20

25.07\%

10.58\%

9.75\%

42.00\%

33.43\%

33.43\%

23.68\%

16.16\%

0.125

6.00

13.37\%

13.37\%

Table 9: Semantic-Branch Ablation in Simulated Navigation

Method	Safe SR $\uparrow$	Avg. Progress $\uparrow$	Collision Rate $\downarrow$	Hazard Intrusion $\downarrow$	FR $\downarrow$	SR $\uparrow$	SPL $\uparrow$	Final goal distance (m) $\downarrow$	VLM early stop $\downarrow$	Timeout $\downarrow$
Ours	$23.68\%$	$43.20\%$	$12.26\%$	$12.81\%$	$10.86\%$	$29.80\%$	$0.256$	$5.38$	$29.24\%$	$17.83\%$
Ours w/o anti-deviation	$11.98\%$	$46.12\%$	$35.93\%$	$18.38\%$	$8.64\%$	$19.50\%$	$0.169$	$5.16$	$18.38\%$	$17.55\%$

Table 10: Anti-Deviation Ablation in Simulated Navigation

Explore similar work

HumanoidVLN: A Physics-Grounded Simulator and Benchmark for Vision-Language Navigation Across Diverse Humanoid Embodiments
Aug 13, 2026Quan-Dung Pham, Anh Dao, The-Anh Nguyen +8Vision-Language Navigation Humanoid
VinMotion, Inc., Vietnam · University of Southern California, USA
Learning Safe Humanoid Navigation from Reduced Order Models
Sep 16, 2026William D. Compton, Zachary Olkin, Ryan Bena +1Robot Navigation Autonomous Navigation
Department of Computing and Mathematical Sciences, California Institute of Technology, Pasadena, CA · Amazon Safe Autonomy Frontiers (SAF) Lab
NavVerse: Benchmarking Indoor-to-Outdoor Embodied Navigation in Continuous Robot Simulation
Jul 22, 2026Junzhe Wu, Yue Hu, Zeyu Han +4Indoor Environments Robot Systems
University of Michigan, Ann Arbor · Tsinghua University

Aug 13, 2026, cs.ROJ/K move · Enter open · S save

HumanoidVLN: A Physics-Grounded Simulator and Benchmark for Vision-Language Navigation Across Diverse Humanoid Embodiments

Quan-Dung Pham, Anh Dao, The-Anh Nguyen, Minh Nguyen-Dinh+7

VinMotion, Inc., Vietnam · University of Southern California, USA

Vision-Language Navigation Humanoid Physics Simulation

Open paper AskPDF arXiv

Vision-Language Navigation (VLN) for humanoid robots poses challenges existing benchmarks fail to address: bipedal locomotion imposes physical constraints absent from wheeled agents, humanoid morphologies vary across platforms, and egocentric observations are distorted by locomotion-induced camera dynamics. We present HumanoidVLN, a physics-grounded simulator and benchmark for VLN across diverse humanoid embodiments. Built on NVIDIA Isaac Sim, our platform supports an extensible set of humanoid configurations, demonstrated on four robots (Unitree G1, Unitree H1, Internal-A, Internal-B) spanning 10-12 lower-body DoF and heights from 1.17m to 1.80m, via a hierarchical control stack combining a reinforcement learning locomotion policy with interchangeable PD or MPC path trackers. New robots and VLN models integrate with minimal effort; we demonstrate compatibility with NaVILA, DualVLN, StreamVLN, and JanusVLN. Environments are drawn from artist-designed scenes and 3D Gaussian Splatting reconstructions, filtered for navigable areas exceeding 100 square meters. Instructions are generated by a dual generator-reviewer plus paraphraser multi-agent pipeline with human-in-the-loop verification, yielding 933 collision-aware reference episodes, each paired with one fine-grained instruction and three coarse-grained stylistic variants (formal, natural, casual). Across four models and four embodiments, JanusVLN achieves the highest mean success rate of 43.55% and nDTW of 48.38. In a 20-episode sim-to-real pilot with DualVLN and the Unitree G1, navigation errors correlate strongly (r=0.935), with a mean absolute difference of 0.68m and mean trajectory similarity of 0.782 (+/-0.188) nDTW. These results highlight the interaction between VLN models, controllers, and humanoid embodiments under physical execution. Code, benchmark, and data will be released upon acceptance at https://humanoid-vln.github.io/.

Abstract

Explore similar work

HumanoidVLN: A Physics-Grounded Simulator and Benchmark for Vision-Language Navigation Across Diverse Humanoid Embodiments

Learning Safe Humanoid Navigation from Reduced Order Models

NavVerse: Benchmarking Indoor-to-Outdoor Embodied Navigation in Continuous Robot Simulation