Affordance-Conditioned Decision Making: Bridging the Semantic-Spatial Gap in Zero-Shot Cross-Floor Vision-and-Language Navigation

Xuekang Yang; Lu Chen; Shuang Luo; Jialing Zhu; Qi Zhang; Yue Gao; Xiang Zhang

cs.ROSep 26, 2026

Affordance-Conditioned Decision Making: Bridging the Semantic-Spatial Gap in Zero-Shot Cross-Floor Vision-and-Language Navigation

Authors: Xuekang Yang, Lu Chen, Shuang Luo, Jialing Zhu, Qi Zhang, Yue Gao, Xiang Zhang

Organizations: School of Computer Science, Shanghai Jiao Tong University · School of Automation and Intelligent Sensing, Shanghai Jiao Tong University · Defense Innovation Institute, Academy of Military Sciences · MoE Key Laboratory of Artificial Intelligence and AI Institute, Shanghai Jiao Tong University

Abstract

Vision-and-language navigation increasingly relies on general-purpose semantic planners, yet translating correct high-level intent into reliable physical execution remains difficult in spatially constrained transitions. Reaching a staircase, doorway, or narrow passage does not ensure traversal; the agent must identify an executable affordance pose and recover from accumulated action errors. We propose PACE (Preference-refined Affordance-Conditioned Execution), a supervised local execution module that augments frozen zero-shot semantic planners for reliable cross-floor navigation. PACE grounds transition-related semantics into a long-horizon, agent-centric traversable affordance pose and conditions short-horizon action generation on this spatial target, thereby aligning semantic goals with physical execution. We further post-train PACE through failure-aware preference refinement using rollout-derived pairs that contrast normal or recovery behaviors with deviation-amplifying behaviors, thereby improving closed-loop correction. We integrate PACE into six open-source zero-shot VLN navigators and demonstrate consistent improvements on the cross-floor subsets of R2R-CE and RxR-CE, increasing the average success rate from 16.35% to 27.65% and from 4.76% to 12.06%, respectively. Real-world experiments further demonstrate PACE's applicability in unseen environments, highlighting the potential of traversable affordances to bridge semantic intent and reliable embodied behavior.

Figures & tables

Figure 1 : Overview of PACE. PACE bridges the semantic-spatial gap through traversable affordance estimation, affordance-conditioned action generation, and rollout-derived preference refinement.

Panel of Figure 3 — Figure 3 : Standalone PACE predictions on R2R-CE-CF val-unseen. In (a), the center denotes zero positional error; radius and angle encode its magnitude and direction, while arrow deviation from the inward direction represents heading error.

Figure 4 : Real-world two-floor stair traversal with VLN-Zero and PACE. Blue and orange points denote VLM- and PACE-controlled trajectory segments, respectively.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Method	R2R-CE-CF																RxR-CE-CF
Method	$\Delta$ SR (pp)				$\Delta$ OSR (pp)				$\Delta$ SPL (pp)				$\Delta$ NDTW				$\Delta$ SR (pp)				$\Delta$ OSR (pp)				$\Delta$ SPL (pp)				$\Delta$ NDTW
VLN-Zero	16.86				19.92				12.46				0.219				4.15				4.15				2.73				0.035
VLN-Zero	[	11.88,	22.22	]	[	14.56,	25.67	]	[	8.13,	16.88	]	[	0.184,	0.255	]	[	0.92,	7.37	]	[	0.92,	7.37	]	[	0.66,	5.10	]	[	0.010,	0.062	]
3-step-Nav	12.64				24.52				6.58				0.074				4.15				14.29				-1.16				-0.051
3-step-Nav	[	7.28,	18.39	]	[	18.01,	31.03	]	[	2.40,	10.76	]	[	0.041,	0.108	]	[	0.46,	7.83	]	[	9.22,	19.35	]	[	-3.87,	1.38	]	[	-0.081,	-0.022	]
CA-Nav	10.73				28.74				4.69				0.060				7.83				13.36				3.41				0.033

Appendix

Table 6 : Observed differences

\Delta=+\mathrm{PACE}-\mathrm{Alone}

under the main-paper protocol. The first and second lines for each method report

\Delta

and its percentile paired-bootstrap 95% CI, respectively, using 10,000 episode-aligned resamples.

Panel of Figure 5 — Figure 5 : Additional real-world navigation examples. (a) “Walk forward to the end, turn right, and continue straight until you reach the trash bin.” (b) “Walk forward through the first doorway, turn right at the second doorway, and stop in front of the elevator.” The point-cloud maps visualize the executed trajectories and are not used for planning or as input to PACE.

Figure 6 : Failure-rate decomposition across six navigation baselines on the 478 episode-aligned val-unseen episodes. Segment lengths are normalized by all episodes, such that each bar represents the overall failure rate. Blue highlights stair-transition failures.

Figure 7 : Representative successful navigation cases with PACE. The top row compares 3-step-Nav, Open-Nav, and STRIDER in the same scene, showing that PACE supports navigators with different internal mechanisms. The bottom row presents additional cases across CA-Nav, VLN-Zero, and HSGM. In particular, PACE completes the traversal when activated either on the staircase (VLN-Zero) or near the stair entrance (HSGM). Blue and orange denote the PACE-executed and expert trajectories, respectively.

Figure 8 : Representative failure cases across the six navigators. Blue and orange denote the executed and expert trajectories, respectively. Failures arise from both PACE-specific execution errors and limitations of the underlying navigators.

Explore similar work

TravExplorer: Cross-Floor Embodied Exploration via Traversability-Aware 3-D Planning
May 19, 2026Han Zheng, Zhe Chen, Yudong Huang +4Dynamic Environments Exploration
Shanghai Jiao Tong University, Shanghai, China.
SpaceVLN: A Zero-Shot Vision-and-Language Navigation Agent with Online Spatial Cognitive Memory and Reasoning
Jun 8, 2026Yucheng Deng, Pingrui Lai, Xinhai Li +5Vision-Language Navigation Spatial Memory
School of Information Science and Electronic Engineering & School of Integrated Circuits, Shanghai Jiao Tong University · Institute of Artificial Intelligence, China Telecom · 3Central South University +1
P2DNav: Panorama-to-Downview Reasoning for Zero-shot Vision-and-Language Navigation
May 19, 2026Kai Sheng, Liuyi Wang, Haojie Dai +5Vision-Language Navigation Menu
Department of Control Science and Engineering, Tongji University, Shanghai 210804, China

May 19, 2026, cs.ROJ/K move · Enter open · S save

TravExplorer: Cross-Floor Embodied Exploration via Traversability-Aware 3-D Planning

Han Zheng, Zhe Chen, Yudong Huang, Haoran Liu+3

Shanghai Jiao Tong University, Shanghai, China.

Dynamic Environments Exploration Planning

Open paper AskPDF arXiv

Zero-shot Object Navigation (ZSON) has shown promise for open-vocabulary target search in unseen environments, yet most existing systems remain tied to planar representations and single-floor assumptions. These assumptions become inadequate in real buildings, where navigation involves floors, stairs, landings, and vertically overlapping spaces. This article presents TravExplorer, a cross-floor embodied exploration framework that couples zero-shot semantic guidance with traversability-aware 3-D planning. TravExplorer maintains a unified volumetric map that distinguishes occupied structures from robot-reachable support surfaces and extracts traversable frontiers from connected support surfaces, including floors, stairs, and landings. A FOV-aware active perception strategy further resolves incomplete observations during cross-floor traversal. To reduce semantic-reasoning latency, a lightweight guidance module aligns a probabilistic instance map from online open-vocabulary segmentation with a spatial value map from fast image-to-text matching. Based on these geometric and semantic memories, a hierarchical planner performs target-aware frontier touring over object hypotheses, traversable frontiers, and stair landmarks, and generates executable cross-floor motions through foothold-guided 3-D search and vertically constrained local trajectory optimization. Experiments over 4,195 simulated episodes on HM3D and MP3D demonstrate consistent advantages over representative ObjectNav baselines. Fifty real-world trials on a Unitree Go2 further validate open-vocabulary target search across single-floor and cross-floor indoor environments without prior maps or human intervention. The code will be released at https://github.com/wuyi2121/TravExplorer.

Abstract

Explore similar work

TravExplorer: Cross-Floor Embodied Exploration via Traversability-Aware 3-D Planning

SpaceVLN: A Zero-Shot Vision-and-Language Navigation Agent with Online Spatial Cognitive Memory and Reasoning

P2DNav: Panorama-to-Downview Reasoning for Zero-shot Vision-and-Language Navigation