cs.RONov 5, 2026
SaveMessyNav: Zero-Shot Navigation via Manipulation in Messy Environments
Organizations: Stanford University · UC Berkeley
Abstract
Real-world environments are messy, with paths often blocked by everyday objects like chairs and toys, requiring robots to perform interactive navigation by manipulating obstacles to clear a path. Existing methods rely on extensive simulation-based learning or per-scene instructions specifying which objects to avoid or manipulate. They struggle to generalize across novel objects, diverse environments, and varied contexts such as a cart of fragile glassware that should be avoided while an empty one can be pushed. We formalize this problem as open-world interactive navigation and present MessyNav, a zero-shot framework based on vision-language models (VLMs). At its core, MessyNav performs explicit VLM-based affordance reasoning before task planning to decide whether to swipe, pick up, or avoid an object and produce a high-level plan. Evaluated on a mobile manipulator across real-world indoor and outdoor environments, MessyNav improves success rate by 27 percentage points (pp) over the strongest baseline in real-robot trials and achieves statistically significant gains in a larger-scale offline evaluation, without task-specific training.
Figures & tables
Figure 1: MessyNav enables zero-shot open-world interactive navigation. [Top] MessyNav reasons zero-shot about object affordances using fine-grained semantic and contextual cues, identifying the utility cart carrying a plastic bin as interactive (swipeable) and the otherwise movable crate carrying unstable stacked blocks as non-interactive. The robot then swipes the cart aside to clear a path and reach the goal without task-specific training or system modifications. [Bottom] MessyNav deployed across diverse real-world indoor and outdoor environments.
Figure 2: MessyNav : A framework for zero-shot open-world interactive navigation. MessyNav consists of an Affordance Reasoner that infers object interactability from an RGB image using a VLM to evaluate both robot-centric physical feasibility and object-centric interaction safety. For example, it will label a cart with a laptop as non-interactive while labeling a wheeled crate with a bin as interactive. Given the object affordance labels, a Task Planner then sequences manipulation and navigation actions from an open-language skill library containing a simple set of motion primitives ( pick_up , swipe_left , swipe_right ) and a LiDAR-based navigation stack ( move_to ) that we find can tackle a wide range of interactive navigation scenarios. Finally, the generated plan is executed by grounding object references in 3D coordinates using VLM keypoint identification, semantic segmentation, and depth-based deprojection.
Figure 3: MessyNav reasons about context, not just object category. In this cluttered scene, closed-set-oracle assigns affordances by category alone, misclassifying the rolling cart as interactive (ignoring the glassware it carries) and the suitcase as interactive (ignoring its fragile sticker). In contrast, MessyNav reasons at the instance level. It treats the cart and the glass stemware as a single fragile unit and reads the sticker on the suitcase as a contextual cue, correctly labeling both as non-interactive.
Figure 4: Explicit affordance reasoning improves context-dependent task planning. In this scene, a suitcase supporting a stack of books is inherently unstable, while the black cart holding a single book can be safely cleared. [Left] MessyNav reasons about affordances before planning, correctly labeling the suitcase as non-interactive and the cart as interactive, and swiping the cart aside to clear the path. [Middle] vlm-planner-only plans directly from the image without an explicit affordance stage and incorrectly decides to swipe the unstable suitcase. [Right] closed-set-planner assigns fixed category-level affordances that misclassify the suitcase as interactive, leading to an unsafe plan.
Figure 5: Qualitative Results across Environments. We deploy MessyNav in varied settings to test its zero-shot capability. The robot successfully reasons about affordances and executes interactions across different lighting conditions, environments, and object types.
| Overall | By Visual Condition | |||
|---|---|---|---|---|
| Method | Success (%) | Outdoor Day | Indoor Lit | Night |
| MessyNav | 62.4 | 78.3 | 62.8 | 29.2 |
| vlm-planner-only | 55.4 | 60.8 | 55.5 | 44.2 |
Table 3: Scaled-Up Offline Planning Results. We evaluate high-level planning performance across 55 real-world scenes in 5 environments, with 15 repeated trials per scene, assuming perfect low-level execution. MessyNav outperforms the strongest baseline from the real-world evaluation overall and under well-lit conditions, while nighttime remains challenging for both methods.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Hardware: We deploy on the TidyBot++ holonomic mobile manipulator [ 18 ] with two cameras and a LiDAR for perception. An onboard laptop runs the autonomy stack.
Figure 7: Example Open-Vocabulary Keypoint Detection. We use gemini-2.5-pro [ 36 ] to detect keypoints of objects specified by the task planner through language queries.
| Parameter | Value |
|---|---|
| Map resolution | 0.05 m/pixel |
| Map size | 30m 30m |
| Max laser range | 10m |
Table 4: Navigation Costmap Parameters.
Figure 8: Sample Images from the Affordance Reasoning Evaluation Set. To evaluate open-world interactive navigation, we collect images spanning three types of diversity: objects, environments, and contextual variations.
| Attribute | Composition |
|---|---|
| Scene type | Indoor (69%); outdoor (31%) |
| Lighting | Bright artificial (66%); natural (28%); |
| dim (6%) | |
| Object affordance | (I) 94 / 217 (43.3%); |
| (N) 123 / 217 (56.7%) | |
| Clutter | 1–4 foreground objects per scene |
Table 5: Composition of the Affordance Reasoning Benchmark. Our benchmark spans diverse environments, lighting, clutter, and obstacle sizes.
| Object | Contexts |
|---|---|
| Rolling cart | Default (I) ; w/ glassware (N) ; |
| w/ glasses (N) ; w/ stuffed toys (I) ; | |
| w/ toys + “DO NOT TOUCH” sign (N) ; | |
| w/ heavy robot arm (N) ; | |
| w/ snack boxes (I) | |
| Wheeled crate | Default (I) ; w/ toaster box precariously on edge (N) ; |
Table 6: Example Context Variations in Affordance Reasoning Evaluation Set. The same object type is marked as interactive (I) or non-interactive (N) depending on context.
| Simple | Complex | Total | |
|---|---|---|---|
| closed-set | 30 / 112 | 23 / 105 | 53 / 217 |
| closed-set-oracle | 106 / 112 | 38 / 105 | 144 / 217 |
| height-only | 57 / 112 | 73 / 105 | 130 / 217 |
| MessyNav (ours) | 101 / 112 | 85 / 105 | 186 / 217 |
Table 7: Affordance Reasoning Comparison . Number of objects with correctly classified interactivity. Simple cases contain objects in their default state, while Complex cases involve context-dependent configurations (e.g., cart with fragile items, table with vase, suitcase with caution tape).
| Environment | Additional Objects in Scenario | Interaction Needed |
|---|---|---|
| Messy Bedroom | Cart w/ broken wheels (N) , Cart (I) | Swipe cart |
| Computer Box (N) , Clear Box (I) | Swipe clear box | |
| Suitcase balancing books (N) , Cart w/ one book (I) | Swipe cart | |
| Crate w/ one book (I) , Cart w/ loose golf balls (N) | Swipe crate | |
| Small full box (N) , Large clear box (I) | Swipe clear box | |
| Wheeled drawer w/ one toy block (I) , Cart w/ toy castle (N) | Swipe wheeled drawer |
Table 8: Real-World Navigation Scenarios. To evaluate open-world interactive navigation, we test in a bedroom and conference room environments while deliberately varying objects, contexts, and required interactions across trials. The scenes also contain furniture (e.g., couch, bed, drawers, meeting tables) as fixed non-interactive obstacles. Additional objects are listed and marked as interactive (I) or non-interactive (N) .
Figure 9: MessyNav ’s failure breakdown compared to the strongest baseline. Sankey diagrams of trial outcomes for MessyNav (left) and vlm-planner-only (right) over 15 real-world trials. MessyNav succeeds on 10/15 trials versus 6/15 for vlm-planner-only . While most of vlm-planner-only ’s failures are planning failures in which vlm-planner-only decides to interact with non-interactive objects (7 of 9), MessyNav reduces these to 2, showcasing the importance of explicit affordance reasoning.
More in this topic
Robot Navigation
Robot navigation studies how robots perceive, localize, plan, and move safely toward goals in static or changing environments.
See all 570 papers