Decide Before You Move: Diffusion-Generated Search Reference for Zero-Shot Open-Vocabulary Object Navigation in Semi-Static Environments
Authors: Yuxuan Feng, Fujing Xie, Sören Schwertfeger, Hermann Blum
Organizations: School of Information Science and Technology, ShanghaiTech University, Shanghai, China. · the University of Bonn and the Lamarr Institute, Bonn, Germany.
Open-world navigation requires a robot to locate a user-specified target in an indoor scene under open-vocabulary queries. Realistic indoor environments are semi-static: walls and furniture largely persist while smaller objects move, relocate, or vanish between visits. Prior-map systems answer queries from object poses recorded at reconstruction time, which expire as soon as objects move, whereas online methods re-derive where to look from current observations every few steps, often through expensive calls to a vision--language model. Assuming that walls and large furniture do not move, we keep their map to plan paths, but use a target--furniture co-occurrence prior to decide where to search. We present PreNav, an ObjectNav framework on semi-static environments. Rather than invoking a vision-language model at each step, PreNav generates a single global search reference before the robot moves: an LLM converts target--furniture co-occurrence into a spatial prior, a conditional diffusion model turns that prior and the persistent structure into an ordered centerline with a band around it, and the robot approaches the target once it is detected online. Experiments on a Gibson ObjectNav benchmark show that PreNav outperforms a strong online baseline under matched structural priors, using a generator trained only on room-level ScanNet++ scenes and applied to floor-level Gibson without fine-tuning. Real-world experiments under object relocation and open-vocabulary queries further validate the approach.
Figures & tables
Fig. 1: Overview of PreNav. (a) For a query q , an LLM scores how useful each furniture category c is for finding the target; the scores rc(q) are computed once and cached. (b) The normalized weights αc(q) , projected onto the mapped furniture, form the spatial prior Dq , which is stacked with the wall raster W and the start map S(s0) . (c) Conditioned on these three channels, a diffusion model denoises random points into the search reference. (d) The reference on the real lab map: a centerline that fixes the visiting order, and a band (teal) that widens over high-scoring furniture (outlined). (e) Real-robot execution: viewpoints are sampled near the centerline to cover where the band overlaps furniture whose weight αc(q) clears a threshold—the regions most worth searching, shown in violet. The robot visits them in order, detects the target with an open-vocabulary detector, and approaches it. Steps (a)–(d) run once, before the robot moves.
Table 2
Category
Mask R-CNN
SAM 3
3DSG oracle
chair
70
73
98
oven
79
91
96
potted plant
80
54
81
refrigerator
41
71
75
tv
54
57
74
vase
33
34
66
TABLE III: Comparison of per-category success rates (%) across different detectors
Fig. 2: The 3DSG oracle and the success criterion read different annotations ( Collierville ). (a) RGB view of a shelf, with each 3DSG object boxed. (b) The 3DSG oracle labels two bottles and a cup. (c) The top-down GT semantic map used to judge success marks the cup (grid) but has no bottle anywhere in the scene; 3DSG objects are projected onto it for comparison, and the wedge marks the camera’s field of view.
Fig. 3: Real-world test scene, a laboratory of over 200 m 2 . (a) The Fetch robot that used for real-world experiments, photographed in the test scene. (b) The room’s osmAG map [ 9 ] with the 21 annotated furniture boxes, the only semantic input, colored by category. (c) A 3D scan of the same room with the osmAG walls overlaid.
Ref. gen. (s)
Mask R-CNN
3DSG oracle
Method
Mean ↓
P95 ↓
SR ↑
SPL ↑
SR ↑
SPL ↑
TSP
25.705
169.893
71.0
39.6
82.7
44.0
Diffusion
0.995
1.029
70.3
39.8
82.5
46.6
TABLE IV: Search-reference generation cost and navigation performance comparison
Fig. 4: Search references of diffusion and its TSP teacher on the 600 six-category episodes. (a,b) Search references of diffusion and TSP for one episode in the Gibson validation scene Darden , with the query potted plant . The native band is the region swept by each generator’s own half-widths wj along its centerline; the purple GT targets are the potted plants in the scene, overlaid for evaluation only. (c) Planning: share of episodes whose band, swept along the reference, contains the target, against reference length; each curve ends with that generator’s longest reference. (d) Execution: share of episodes that succeed within a distance traveled, under Mask R-CNN.
Fig. 5: Target recognition and navigation outcomes across 600 episodes. Node labels give counts and percentages; ribbon widths represent episode counts.
Open-world navigation requires robots to make decisions in complex everyday environments while adapting to flexible task requirements. Conventional navigation approaches often rely on dense 3D reconstruction and hand-crafted goal metrics, which limits their generalization across tasks and environments. Recent advances in vision-language navigation (VLN) and vision-language-action (VLA) models enable end-to-end policies conditioned on natural language, but typically require interactive training, large-scale data collection, or task-specific fine-tuning with a mobile agent. We formulate navigation as a sparse subgoal identification and reaching problem and observe that providing visual anchoring targets for high-level semantic priors enables highly efficient goal-conditioned navigation. Based on this insight, we select visual frontiers as semantic anchors and propose OpenFrontier, a navigation framework that requires no task-specific training or fine-tuning and seamlessly integrates diverse vision-language prior models. OpenFrontier enables efficient navigation with a lightweight system design, without dense 3D semantic mapping, task-specific policy training, or model fine-tuning. We evaluate OpenFrontier across multiple navigation benchmarks and demonstrate strong zero-shot performance, as well as effective real-world deployment on a mobile robot.
Esteban Padilla-Cerdio, Boyang Sun, Marc Pollefeys +1
1ETH Zurich · 2Microsoft Spatial AI Lab · University of Bonn
Zero-shot ObjectNav methods increasingly use vision-language priors, but direct object-object similarity in the latent space is often a weak proxy for spatial co-occurrence. We present an analytical, training-free semantic navigation pipeline that mediates object relationships through a compact room lexicon. Each object label is mapped to a CLIP-derived Room Probability Vector (RPV), and object-target co-occurrence is computed from RPV distribution overlap. These scores are projected onto a value map using geodesic flood-fill propagation (Fast Marching Method), with adaptive signal decay, and are used to rank frontiers by semantic score for navigation. Together, these components form an integrated, training-free, object-centric pipeline for open-vocabulary zero-shot navigation. Results show that our object-centric approach improves Success Rate (SR) and Success by weighted inverse Path Length (SPL) by a relative 3% and 1.3%, respectively, compared to image-holistic baselines on the HM3D dataset validation split, while preserving interpretability and open-vocabulary flexibility. Code is available at: uts-ri.github.io/RPV-SemNav.
Adam Scicluna, Gavin Paul, Alen Alempijevic
Robotics Institute, Faculty of Engineering and Information Technology, University of Technology Sydney (UTS), Sydney, Australia
Object-oriented embodied navigation tasks require agents to locate specific objects, either defined by category or images, in unseen environments. While recent methods have made progress in extending closed-set models to open-vocabulary scenarios with foundation models, they typically rely on training-free large language models (LLMs) or finetuning with end-to-end reinforcement learning (RL). However, they face challenges in efficiency (e.g., the overhead and cost of LLM inference) and limited generalization from intensive RL training. In this paper, we propose OVExp, a training-efficient framework for open-vocabulary exploration. We make the first effort to demonstrate the generalization capabilities of semantic map-based goal prediction networks using Dense CLIP models. A major challenge is that preserving both precise point-wise object locations and generalizable visual representations in the semantic map leads to unaffordable training costs. To address this, we design a Cross-Modal Transfer on Semantic Mapping strategy which adapts an intriguing text-only training and transfer to multi-model semantic mapping and goals in test-time. Despite relying on text-based spatial layouts with limited objects, OVExp demonstrates robust generalization to unseentargets on established ObjectNav benchmarks.
Meng Wei, Chenyang Wan, Tai Wang +6
Shanghai AI Lab · Zhejiang University · The University of Hong Kong