Real-world environments are messy, with paths often blocked by everyday objects like chairs and toys, requiring robots to perform interactive navigation by manipulating obstacles to clear a path. Existing methods rely on extensive simulation-based learning or per-scene instructions specifying which objects to avoid or manipulate. They struggle to generalize across novel objects, diverse environments, and varied contexts such as a cart of fragile glassware that should be avoided while an empty one can be pushed. We formalize this problem as open-world interactive navigation and present MessyNav, a zero-shot framework based on vision-language models (VLMs). At its core, MessyNav performs explicit VLM-based affordance reasoning before task planning to decide whether to swipe, pick up, or avoid an object and produce a high-level plan. Evaluated on a mobile manipulator across real-world indoor and outdoor environments, MessyNav improves success rate by 27 percentage points (pp) over the strongest baseline in real-robot trials and achieves statistically significant gains in a larger-scale offline evaluation, without task-specific training.
Figures & tables
Figure 1: MessyNav enables zero-shot open-world interactive navigation. [Top] MessyNav reasons zero-shot about object affordances using fine-grained semantic and contextual cues, identifying the utility cart carrying a plastic bin as interactive (swipeable) and the otherwise movable crate carrying unstable stacked blocks as non-interactive. The robot then swipes the cart aside to clear a path and reach the goal without task-specific training or system modifications. [Bottom] MessyNav deployed across diverse real-world indoor and outdoor environments.
Figure 2: MessyNav : A framework for zero-shot open-world interactive navigation. MessyNav consists of an Affordance Reasoner that infers object interactability from an RGB image using a VLM to evaluate both robot-centric physical feasibility and object-centric interaction safety. For example, it will label a cart with a laptop as non-interactive while labeling a wheeled crate with a bin as interactive. Given the object affordance labels, a Task Planner then sequences manipulation and navigation actions from an open-language skill library containing a simple set of motion primitives ( pick_up , swipe_left , swipe_right ) and a LiDAR-based navigation stack ( move_to ) that we find can tackle a wide range of interactive navigation scenarios. Finally, the generated plan is executed by grounding object references in 3D coordinates using VLM keypoint identification, semantic segmentation, and depth-based deprojection.
Figure 3: MessyNav reasons about context, not just object category. In this cluttered scene, closed-set-oracle assigns affordances by category alone, misclassifying the rolling cart as interactive (ignoring the glassware it carries) and the suitcase as interactive (ignoring its fragile sticker). In contrast, MessyNav reasons at the instance level. It treats the cart and the glass stemware as a single fragile unit and reads the sticker on the suitcase as a contextual cue, correctly labeling both as non-interactive.
Table 4
Figure 4: Explicit affordance reasoning improves context-dependent task planning. In this scene, a suitcase supporting a stack of books is inherently unstable, while the black cart holding a single book can be safely cleared. [Left] MessyNav reasons about affordances before planning, correctly labeling the suitcase as non-interactive and the cart as interactive, and swiping the cart aside to clear the path. [Middle] vlm-planner-only plans directly from the image without an explicit affordance stage and incorrectly decides to swipe the unstable suitcase. [Right] closed-set-planner assigns fixed category-level affordances that misclassify the suitcase as interactive, leading to an unsafe plan.
Figure 5: Qualitative Results across Environments. We deploy MessyNav in varied settings to test its zero-shot capability. The robot successfully reasons about affordances and executes interactions across different lighting conditions, environments, and object types.
Overall
By Visual Condition
Method
Success (%)
Outdoor Day
Indoor Lit
Night
MessyNav
62.4
78.3
62.8
29.2
vlm-planner-only
55.4
60.8
55.5
44.2
Table 3: Scaled-Up Offline Planning Results. We evaluate high-level planning performance across 55 real-world scenes in 5 environments, with 15 repeated trials per scene, assuming perfect low-level execution. MessyNav outperforms the strongest baseline from the real-world evaluation overall and under well-lit conditions, while nighttime remains challenging for both methods.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Hardware: We deploy on the TidyBot++ holonomic mobile manipulator [ 18 ] with two cameras and a LiDAR for perception. An onboard laptop runs the autonomy stack.
Figure 7: Example Open-Vocabulary Keypoint Detection. We use gemini-2.5-pro [ 36 ] to detect keypoints of objects specified by the task planner through language queries.
Parameter
Value
Map resolution
0.05 m/pixel
Map size
30m × 30m
Max laser range
10m
Appendix
Table 4: Navigation Costmap Parameters.
Figure 8: Sample Images from the Affordance Reasoning Evaluation Set. To evaluate open-world interactive navigation, we collect images spanning three types of diversity: objects, environments, and contextual variations.
Attribute
Composition
Scene type
Indoor (69%); outdoor (31%)
Lighting
Bright artificial (66%); natural (28%);
dim (6%)
Object affordance
(I) 94 / 217 (43.3%);
(N) 123 / 217 (56.7%)
Clutter
1–4 foreground objects per scene
Appendix
Table 5: Composition of the Affordance Reasoning Benchmark. Our benchmark spans diverse environments, lighting, clutter, and obstacle sizes.
Table 6: Example Context Variations in Affordance Reasoning Evaluation Set. The same object type is marked as interactive (I) or non-interactive (N) depending on context.
Simple
Complex
Total
closed-set
30 / 112
23 / 105
53 / 217
closed-set-oracle
106 / 112
38 / 105
144 / 217
height-only
57 / 112
73 / 105
130 / 217
MessyNav (ours)
101 / 112
85 / 105
186 / 217
Appendix
Table 7: Affordance Reasoning Comparison . Number of objects with correctly classified interactivity. Simple cases contain objects in their default state, while Complex cases involve context-dependent configurations (e.g., cart with fragile items, table with vase, suitcase with caution tape).
Environment
Additional Objects in Scenario
Interaction Needed
Messy Bedroom
Cart w/ broken wheels (N) , Cart (I)
Swipe cart
Computer Box (N) , Clear Box (I)
Swipe clear box
Suitcase balancing books (N) , Cart w/ one book (I)
Swipe cart
Crate w/ one book (I) , Cart w/ loose golf balls (N)
Swipe crate
Small full box (N) , Large clear box (I)
Swipe clear box
Wheeled drawer w/ one toy block (I) , Cart w/ toy castle (N)
Swipe wheeled drawer
Appendix
Table 8: Real-World Navigation Scenarios. To evaluate open-world interactive navigation, we test in a bedroom and conference room environments while deliberately varying objects, contexts, and required interactions across trials. The scenes also contain furniture (e.g., couch, bed, drawers, meeting tables) as fixed non-interactive obstacles. Additional objects are listed and marked as interactive (I) or non-interactive (N) .
Figure 9: MessyNav ’s failure breakdown compared to the strongest baseline. Sankey diagrams of trial outcomes for MessyNav (left) and vlm-planner-only (right) over 15 real-world trials. MessyNav succeeds on 10/15 trials versus 6/15 for vlm-planner-only . While most of vlm-planner-only ’s failures are planning failures in which vlm-planner-only decides to interact with non-interactive objects (7 of 9), MessyNav reduces these to 2, showcasing the importance of explicit affordance reasoning.
Mobile manipulation requires a robot to navigate to a target object or receptacle and then perform intended manipulation. However, reaching the vicinity of the target does not guarantee a manipulation-ready base pose, a problem known as last-mile navigation. Prior methods for last-mile navigation either rely on manual pose annotation or task-specific training, limiting their scalability to open-vocabulary settings with fine-grained spatial constraints. We propose UniLM-Nav, a unified framework for zero-shot open-vocabulary last-mile navigation. UniLM-Nav decomposes last-mile navigation into view selection, task-conditioned affordance grounding, and geometry-aware base-pose reasoning, all resolved with a shared multimodal large language model (MLLM) backend. Specifically, UniLM-Nav first selects a reference view that best captures the target object or receptacle from recently collected observations. It then grounds task-relevant affordance point in the selected view and lifts the result into the robot-centric coordinate frame. Finally, conditioned on the grounded affordance, task context, and robot geometry, it infers a manipulation-ready base pose for the robot. We evaluate UniLM-Nav on the OVMM benchmark, where it outperforms the previous state-of-the-art method, MoTo, by 3.13 percentage points. Analyses show that the components of our method are crucial to final performance, and that the choice of MLLM also has a substantial effect. We further deploy UniLM-Nav on a Unitree B2 quadruped robot with a 6-DoF Unitree Z1 manipulator, validating its applicability to real-world mobile manipulation tasks.
Zhuofan Zhang, Tianxu Wang, Guoxi Zhang +6
1Tsinghua University · 2State Key Laboratory of General Artificial Intelligence, BIGAI · 3Harbin Institute of Technology +1
Open-world navigation requires robots to make decisions in complex everyday environments while adapting to flexible task requirements. Conventional navigation approaches often rely on dense 3D reconstruction and hand-crafted goal metrics, which limits their generalization across tasks and environments. Recent advances in vision-language navigation (VLN) and vision-language-action (VLA) models enable end-to-end policies conditioned on natural language, but typically require interactive training, large-scale data collection, or task-specific fine-tuning with a mobile agent. We formulate navigation as a sparse subgoal identification and reaching problem and observe that providing visual anchoring targets for high-level semantic priors enables highly efficient goal-conditioned navigation. Based on this insight, we select visual frontiers as semantic anchors and propose OpenFrontier, a navigation framework that requires no task-specific training or fine-tuning and seamlessly integrates diverse vision-language prior models. OpenFrontier enables efficient navigation with a lightweight system design, without dense 3D semantic mapping, task-specific policy training, or model fine-tuning. We evaluate OpenFrontier across multiple navigation benchmarks and demonstrate strong zero-shot performance, as well as effective real-world deployment on a mobile robot.
Esteban Padilla-Cerdio, Boyang Sun, Marc Pollefeys +1
1ETH Zurich · 2Microsoft Spatial AI Lab · University of Bonn
Zero-shot object navigation requires agents to locate unseen targets in unfamiliar environments without prior maps or task-specific training. Despite the commonsense reasoning ability of vision-language models (VLMs), existing mapless navigators often suffer from repeated exploration and premature stopping caused by limited historical context and false-positive target predictions. We propose ReMemNav, a training-free framework that combines lightweight semantic grounding, memory-based decision correction, and target verification. RAM-derived semantic priors guide panoramic direction selection, while a geometry-triggered correction mechanism retrieves historical scene descriptions when the selected direction points toward previously visited regions. When a target is predicted to be visible, ReMemNav verifies the corresponding single-view observation before the final approach. Depth-based action sampling then converts the high-level decision into collision-free motion. Experiments on HM3D and MP3D show that ReMemNav achieves higher success rates and path efficiency than existing training-free zero-shot baselines. Specifically, ReMemNav achieves absolute SR/SPL gains of 1.7%/5.3% on HM3D v0.1, 18.2%/11.1% on HM3D v0.2, and 8.7%/6.9% on MP3D.