Organizations: Interdisciplinary Program in Artificial Intelligence, Seoul National University · Department of Electrical and Computer Engineering, Seoul National University
Safe household navigation requires robots to regulate clearance and speed according to what an obstacle is and how close it is, without unnecessary conservatism. Yet existing evaluations do not test whether semantic safety distinctions are reflected in continuous motion. To that end, we introduce OSMA-Bench, an Obstacle-Conditioned Safe Motion Adaptation Benchmark built on spatially matched scenario sets that differ only in obstacle type. We also propose the Semantic Sensitivity Index (SSI), which measures whether clearance and distance-dependent speed and smoothness follow the caution ordering of obstacles. Evaluating VLM-based and vision-language-action (VLA) policies on the benchmark, we find that neither consistently adapts its motion to obstacle safety, even with explicit safety prompts. Our analysis shows that the policies can correctly distinguish obstacles by safety relevance but fail to reflect this distinction in the motion parameters or actions they produce. For VLM-based policies, which recognize the obstacle and then generate a navigation program, adding an explicit parameter-inference step between these two stages improves adaptation. These findings expose a gap in semantic-to-motion grounding, highlighting the need to translate semantic caution into appropriately safe motion.
Figures & tables
Figure 1: Obstacle-conditioned safe motion adaptation goes beyond task outcomes. The robot passes a trash bin (left, low caution) or a baby (right, high-caution) for a given route. Both reach the goal without collision, yet an obstacle-conditioned safe policy should exhibit more conservative motion near the baby. SSI (Section 3.2 ) measures whether minimum clearance and near-obstacle speed, acceleration, and jerk follow this caution ordering.
Factor
Values
Count
Layout variant
Layout 0–4 ∗
5
Route geometry
Route A–G ∗
7
Obstacle entities
6 per tier
18
Obstacle Caution Tier
High / Moderate / Low
High (Living)
Human, Boy, Girl, Baby, Dog, Cat
6
Moderate (Fragile)
Wine, Water, Hot chocolate, Vase, Floor pot, Table lamp
6
Table 1: OSMA-Bench task composition. The benchmark combines 18 obstacle entities grouped into three caution tiers with 7 routes and 5 kitchen-layout variants.
Figure 2: Illustrative example of proximity-normalized speed.
Figure 3: Main results. Left: Task-level safety ( CFSR/TSR ) and semantic sensitivity ( SSI ) across policy groups. The dashed line marks the tier-indifferent reference, SSI=0 . Right: (b) Representative trajectories around a boy (high caution tier), using π0.5 -OpenTau (VLA), Gemma-4-12B-it (PIVOT), and Qwen3.6-35B-A3B (VoxPoser), from left to right. (c) VoxPoser with Qwen3.6-35B-A3B evaluated across obstacle caution tiers under a fixed route and layout. S and G indicate the start and goal positions, respectively.
Safe robot navigation develops methods for reaching goals while avoiding collisions and managing uncertainty, dynamic obstacles, and environmental hazards.
Safe autonomous navigation requires not only geometric obstacle avoidance but also semantic awareness. Different object categories demand different safety clearances, and the set of relevant categories may expand as the robot explores novel environments. Vision-language models (VLMs) offer rich semantic representations for building maps of unknown scenes, yet their confidence scores are poorly calibrated, making raw predictions unreliable for safety-critical planning. Conformal prediction (CP) can provide coverage guarantees, but existing methods assume a fixed label set and stationary data, limiting their use in open-world settings. We propose Conformal Semantic Mapping for Open-World Safe Robot Navigation (CoSMO), a semantic safety mapping framework that transforms VLM embeddings into calibrated coverage sets and safety-aware obstacle dilations over a dynamically expanding hazard vocabulary, even under distribution shift. Experiments on the Matterport3D (MP3D) dataset show that CoSMO reduces semantic collisions by 61.6% relative to a geometry-only baseline, while maintaining formal coverage guarantees even under delayed expert feedback. Real-world experiments on a quadruped robot further demonstrate generalization by achieving long-run empirical coverage of 90.73% against the 90% target. Project website: https://yangming911.github.io/cosmo-page/.
Shuqi Wang, Yiwei Zheng, Pingrui Zhang +2
Shanghai Jiao Tong University · Shanghai Innovation Institute · Fudan University
Vision-Language Models (VLMs) have shown remarkable progress in Vision-Language Navigation (VLN), offering new possibilities for navigation decision-making that could benefit both robotic platforms and human users. However, real-world navigation is inherently conditioned by the agent's mobility constraints. For example, a sweeping robot cannot traverse stairs, while a quadruped can. We introduce Capability-Conditioned Navigation (CapNav), a benchmark designed to evaluate how well VLMs can navigate complex indoor spaces given an agent's specific physical and operational capabilities. CapNav defines five representative human and robot agents, each described with physical dimensions, mobility capabilities, and environmental interaction abilities. CapNav provides 45 real-world indoor scenes, 473 navigation tasks, and 2365 QA pairs to test if VLMs can traverse indoor environments based on agent capabilities. We evaluate 13 modern VLMs and find that current VLM's navigation performance drops sharply as mobility constraints tighten, and that even state-of-the-art models struggle with obstacle types that require reasoning on spatial dimensions. We conclude by discussing the implications for capability-aware navigation and the opportunities for advancing embodied spatial reasoning in future VLMs. The benchmark is available at https://github.com/makeabilitylab/CapNav
Xia Su, Ruiqi Chen, Benlin Liu +4
University of Washington Seattle, WA, USA · University of California, Santa Cruz Santa Cruz, CA, USA
Vision-language-action (VLA) benchmarks measure whether a policy completes a requested manipulation task, but binary success can hide safety violations along the trajectory: a policy may reach the goal while applying excessive contact, disturbing bystander objects, destabilizing a held object, or entering robot self-contact. We present SafeVLA-Bench, a post-hoc safety-evaluation framework for existing simulator-based VLA benchmarks that reveals violations missed by success-only evaluation. It encodes task-aware safety requirements as Signal Temporal Logic (STL) invariants with quantitative robustness semantics. Alongside native success, it reports the safety rate and the success-but-unsafe rate (SBU) used in prior safety evaluations, and introduces the Violation Severity Index (VSI), a bounded worst-violation depth score. We instantiate SafeVLA-Bench on LIBERO and RoboCasa-365, evaluating twenty-seven policy-benchmark entries across tabletop and kitchen manipulation tasks. High task success does not imply safe execution: the fifteen tabletop policies above 90% mean success still have 18-28% unsafe-episode rates, and 38-56% of successful RoboCasa-365 rollouts violate at least one active safety clause. A post-training case study further shows that SafeVLA-Bench can be used to improve policy safety. Project page: https://safevla.org
Jialiang Fan, Weizhe Xu, Zijun Wang +3
University of Notre Dame · University of Pennsylvania