Watch Your Step: Learning Semantically-Guided Locomotion in Cluttered Environment

Denan Liang; Yuan Zhu; Ruimeng Liu; Thien-Minh Nguyen; Shenghai Yuan; Lihua Xie

cs.ROMar 3, 2026

Watch Your Step: Learning Semantically-Guided Locomotion in Cluttered Environment

Authors: Denan Liang, Yuan Zhu, Ruimeng Liu, Thien-Minh Nguyen, Shenghai Yuan, Lihua Xie

Abstract

Although legged robots demonstrate impressive mobility on rough terrain, using them safely in cluttered environments remains a challenge. A key issue is their inability to avoid stepping on low-lying objects, such as high-cost small devices or cables on flat ground. This limitation arises from a disconnection between high-level semantic understanding and low-level control, combined with errors in elevation maps during real-world operation. To address this, we introduce SemLoco, a Reinforcement Learning (RL) framework designed to avoid obstacles precisely in densely cluttered environments. SemLoco uses a two-stage RL approach that combines both soft and hard constraints. It performs pixel-wise foothold safety inference, which enables more accurate foot placement. Additionally, SemLoco integrates semantic map, allowing it to assign traversability costs instead of relying only on geometric data. SemLoco greatly reduces collisions and improves safety around sensitive objects, enabling reliable navigation in situations where traditional controllers would likely cause damage. Experimental results further show that SemLoco can be effectively applied to more complex, unstructured real-world environments. A demo video can be view at https://youtu.be/FSq-RSmIxOM.

Figures & tables

Fig. 2: Framework of SemLoco . Sub-modules have different styles based on their functions. Among them, the red trapezoid represents the neural network, the blue rectangle represents unprocessed raw data, and the green rectangle represents processed ready-to-use data. (a) Training in the simulator: In stage 1, we use virtual obstacles (Highly yellow spheres). Although the robot walks on flat ground, it receives a virtual perception map containing height and semantic information to simulate scenarios with real obstacles. In stage 2, we use rigid obstacles. The robot receives corresponding real perception information, and fine-tunes the policy to improve task performance. (b) Deployment in the real world: Exteroception information is obtained by Odin1 and fed into the semantic graph algorithm.

Fig. 3: Footswing tracking for each leg.

Panel of Figure 3 — Fig. 4: Examples of obstacle curricula.

Policy	10 obstacles $\mathrm{m}^{-2}$			15 obstacles $\mathrm{m}^{-2}$			20 obstacles $\mathrm{m}^{-2}$			25 obstacles $\mathrm{m}^{-2}$
Policy	$D\uparrow$	$S\uparrow$	$C\downarrow$	$D\uparrow$	$S\uparrow$	$C\downarrow$	$D\uparrow$	$S\uparrow$	$C\downarrow$	$D\uparrow$	$S\uparrow$	$C\downarrow$
	(m)	(%)	(%)	(m)	(%)	(%)	(m)	(%)	(%)	(m)	(%)	(%)
Ours	9.67	97.00	2.37	9.51	92.60	8.78	9.29	87.40	13.45	8.94	82.20	24.33
Blind	5.53	28.40	51.11	5.38	23.20	75.56	5.08	15.60	86.67	4.62	9.40	93.33
Ours w/o Virtual Obstacle Stage	6.19	51.60	13.18	5.88	45.80	20.88	5.42	39.40	29.67	5.06	30.80	36.15
Ours w/o ReLU Clearance Penalty	8.22	75.60	16.48	7.71	62.80	23.07	7.25	53.60	31.87	6.97	46.20	39.56

TABLE I: Ablation experiment under different obstacle densities.

Fig. 5: Qualitative comparison of obstacle avoidance in a real cluttered environment.We compare our semantic-aware locomotion policy (Bottom) against an elevation-only baseline (Top). Elevation maps completely fail to capture small, low-profile objects (Fig 1 ). Consequently, Ours w/o Semantic Map degrades into a nearly blind policy, failing to adjust foot placements to avoid obstacles. As highlighted by the colored circles, the robot frequently steps on or kicks the scattered objects; the resulting environmental disruption is highly evident when comparing the Pre-Traversal and Post-Traversal states (the first and last columns). In contrast, our full framework leverages explicit semantic cues to achieve precise obstacle avoidance. The environment remains almost entirely undisturbed, with the exception of a negligible displacement of a power bank caused by an incidental grazing contact during the swing leg’s descent.

Legged Robotics

Legged robotics studies how robots with legs move, perceive their surroundings, and perform useful tasks across varied terrain.

See all 126 papers

Learning Agile Navigation in Crowded Environments for Quadruped Robots
Jul 16, 2026Shuyu Wu, Zeyu Liu, Tianbao Zhang +6Robot Policy Learning Robot Navigation
Shanghai Jiao Tong University, Shanghai 200240, China · State Key Laboratory of High-end Heavy-load Robots, Midea Group, Foshan 528300, China
Learning Scene-Aware Humanoid Locomotion through 3D Clutter from Immersive Human Demonstrations
Sep 17, 2026Beichen Wang, Tong Xu, Daniel Kosukhin +3Humanoid Robot Locomotion Robot Imitation Learning
George Mason University · New York University
SemGeoNav:A Safety-Guided Visual Navigation Approach with Semantic Reasoning and Geometric Planning
Jun 15, 2026Yu Liu, Zongyang Chen, Yan Guo +2Robot Navigation Visual Robot Navigation
College of Intelligence Science and Technology, National University of Defense Technology, Changsha 410073, China.

Jul 16, 2026J/K move · Enter open · S save

Learning Agile Navigation in Crowded Environments for Quadruped Robots

Shuyu Wu, Zeyu Liu, Tianbao Zhang, Fanxing Li+5

Shanghai Jiao Tong University, Shanghai 200240, China · State Key Laboratory of High-end Heavy-load Robots, Midea Group, Foshan 528300, China

Robot Policy Learning Robot Navigation Dynamic Obstacle Avoidance Quadruped Robot Navigation

Open paper AskPDF arXiv

Navigating dynamic and crowded environments presents significant challenges for quadruped robots due to severe sensor occlusion and unpredictable human motion. Existing approaches face a trade-off: model-based methods, such as Velocity Obstacles (VO), theoretically guarantee safety but rely on accurate obstacle motion estimates that often fail in dense crowds, while end-to-end learning methods offer robustness but lack motion prediction capability of obstacles, leading to collisions or conservative behaviors. To solve this, we propose VOP-Nav, a novel navigation system that combines the geometric safety of VO with the agile adaptability of end-to-end learning. Using only local onboard observations, our system avoids explicit obstacle detection and tracking pipelines. The VOP-Net processes multi-frame LiDAR data to implicitly encode dynamic constraints and predict a safe velocity region derived from Velocity Obstacle theory. Importantly, the VO predictions serve a dual role: they are used as input to the navigation policy during inference and as a reward signal during training to encourage safe motion. Evaluations in Isaac Gym demonstrate that VOP-Nav achieves higher success rates than all baselines while balancing locomotion speed and collision avoidance. Real-world deployment on a Unitree Go2 quadruped robot further validates the system's robustness and efficiency in complex indoor and outdoor dynamic environments.

Abstract

Legged Robotics

Explore similar work

Learning Agile Navigation in Crowded Environments for Quadruped Robots

Learning Scene-Aware Humanoid Locomotion through 3D Clutter from Immersive Human Demonstrations

SemGeoNav:A Safety-Guided Visual Navigation Approach with Semantic Reasoning and Geometric Planning