Selecting a next motion in a populated scene is not just a matter of avoiding what is currently in the way: an action can be locally feasible yet drift onto the road instead of the crosswalk, cut into a stream of crossing pedestrians, or bottleneck a narrow passage. These failure modes are invisible from the current state alone, and acting on them requires reasoning about what each candidate action will cause. World models offer exactly this, namely action-conditioned predictions of the future, but rolling them out at every decision step is a cost deployable navigation systems cannot pay, and the field has largely traded foresight for speed. We argue this is a false dichotomy. The natural place for a world model in human-aware navigation is at training time, as a preference oracle that distills foresight into a fast policy, rather than at inference time, as an online planner. We pair a discrete visual action prior with two task-aligned world models, one for nearby social dynamics and one for future scene semantics, that score candidate actions by their predicted consequences and refine the policy through group-relative preference distillation. At deployment, both world models are removed and the policy acts in a single forward pass. Across five held-out evaluation scenarios, this yields navigation that is simultaneously safer, more socially and group-coherent, and more goal-directed than strong reactive and learned baselines, with naive observers rating the resulting motion as more human-like. Taken together, our findings point to a different approach for bringing world-model foresight into deployable agents: not accelerating rollouts, but amortizing them into the policy so that the cost of looking ahead is confined to training. Project page: https://seoneun.github.io/-NeurIPS-2026-Seeing-Beyond-the-Next-Step/
Figures & tables
Figure 1: Overview of the proposed training pipeline. In Stage 1, we adapt a pretrained VLM into a VLA policy through supervised learning, grounding agent-centric visual observations and structured navigation context to executable motion-skill actions. In Stage 2, we further refine the policy with dual world models, which teaches the agent action–outcome knowledge by comparing plausible candidate actions according to their predicted social, semantic, and goal-directed consequences.
Figure 2: Overview of the training and evaluation scenes. We use six scenes in our experiments: Students003, ETH_Hotel, Zara01, ETH, ETH_Hallway, and ETH_Intersection.
Scene
Method
GSR ↑
SRVR ↓
GD ↓
Zara01
GBM [ 12 ]
55.6
21.0
0.6
CCP [ 30 ]
33.3
16.0
2.6
CEDRL [ 29 ]
50.0
15.6
1.3
Ours
72.2
7.0
0.6
Zara01_Group
GBM [ 12 ]
38.9
17.5
1.9
CCP [ 30 ]
27.8
19.5
4.8
Table 1: Closed-loop results on five held-out evaluation scenarios. Higher is better for GSR, while lower is better for SRVR and GD.
Figure 3: Crosswalk navigation comparison. We compare CCP, CEDRL, GBM, and our method in a road-crossing scene with a marked pedestrian crosswalk. Red curves show each agent’s trajectory over the previous 60 frames. Blue boxes mark frames where the baselines leave the crosswalk and traverse road regions directly, indicating limited use of semantic scene cues. Our method, by contrast, remains on the crosswalk throughout the crossing, producing semantically compliant and pedestrian-like behavior. All trajectories are generated by the respective policies. In the second frame of ours, the agent moves along the crosswalk edge to avoid an oncoming pedestrian while staying on the crosswalk.
Scene
Variant
GSR ↑
SRVR ↓
GD ↓
Zara01
Ours
72.2
7.0
0.6
w/o Dual World Models
25.9
12.6
0.7
w/o State Transition World Model
44.4
10.7
0.6
w/o Semantic Transition World Model
44.4
9.3
0.6
w/o VLM
48.1
10.9
0.9
Top-10 Candidates
44.4
18.2
0.7
Table 2: Ablation study on five held-out evaluation scenarios. Higher is better for GSR, while lower is better for SRVR and GD.
Method
GSR ↑
SRVR ↓
GD ↓
OnlineWM
80.6
3.2
0.9
Constant Velocity [ 36 ]
56.1
15.6
2.2
Social-LSTM [ 1 ]
51.9
15.0
2.1
Social-GAN [ 15 ]
53.2
15.0
2.2
Trajectron++ [ 35 ]
54.4
17.0
2.1
Gen-Safe-Nav [ 45 ]
42.9
14.1
2.3
Table 3: Comparison with simpler pipelines and an online world-model planner. Trajectory predictors are combined with an MPC planner [ 33 ] . Results are averaged over the five held-out evaluation scenarios; per-scene results are in Appendix M . Higher is better for GSR, while lower is better for SRVR and GD. OnlineWM runs both world models at every decision; bold marks the best result among the remaining methods.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Semantic tag
Navigation meaning
Hex
Role in semantic score
sidewalk
Pedestrian sidewalk or primary outdoor walking surface.
#804080
Positive. A high sidewalk ratio indicates that the candidate anchor remains on a pedestrian-compatible walking region.
crosswalk
Marked pedestrian crossing region.
#FFFFFF
Positive. Treated as a valid walking region and grouped with sidewalk for walkability scoring.
road
Vehicle-oriented road surface.
#808080
Negative. A large road ratio indicates unsafe or semantically inappropriate pedestrian motion unless the region is explicitly labeled as crosswalk.
obstacle
Static obstacle such as a pole, barrier, fence, furniture, or other non-traversable object.
#FF8C00
Strongly negative. Footprint overlap indicates physically invalid or unsafe navigation.
building
Building, wall, or other non-traversable static structure.
#464646
Strongly negative. Overlap indicates an invalid trajectory through static structure.
vehicle
Moving or parked vehicle.
#00008E
Strongly negative. Even a small vehicle overlap indicates high collision or scene-validity risk.
Appendix
Table 4: Segmentation classes used by the Semantic Transition World Model. Each class is identified by a semantic tag and a fixed Hex color code. The score role describes how the class is treated by the semantic-validity score in Appendix B .
Figure 4: Real trajectory clips and nearest-anchor reconstructions. We randomly sample 20 trajectory clips and visualize each real trajectory against its nearest anchor reconstruction from the K=100 vocabulary. Blue solid lines denote real trajectory clips, and red dashed lines denote anchor reconstructions. The examples show that the 100-anchor vocabulary preserves the main direction, scale, and endpoint tendency of short-horizon pedestrian motion, supporting its use as a compact discrete action space.
Method
Pixel Acc. ↑
mIoU ↑
Dice/F1 ↑
FWIoU ↑
EDM
99.2848%
87.0020%
89.0814%
98.6099%
DDPM
97.7863%
86.5421%
88.2132%
95.4289%
Appendix
Table 5: Semantic prediction performance. Class-aware test metrics for the Semantic Transition World Model.
Pos. ADE ↓
Pos. RMSE ↓
Vel. MAE ↓
Vel. RMSE ↓
Speed MAE ↓
Head. Err. ↓
0.1846
0.2481
0.3354
0.4414
0.2096
20.77
Appendix
Table 6: State prediction performance. Test metrics for future relative position and velocity prediction.
Figure 5: Segmentation image sequence from Zara01. Columns are ordered in time. Rows show the third-person RGB input, the ground-truth segmentation, and the segmentation predicted by the Semantic Transition World Model. The red silhouette marks the controlled agent; other pedestrians are excluded from the segmentation target.
Figure 6: Segmentation image sequence from ETH_Hotel. Columns are ordered in time; rows are as in Figure 5 .
Scene
Method
ADE ↓
FDE ↓
Zara01
GBM [ 12 ]
0.3057±0.0345
0.5203±0.0509
CCP [ 30 ]
0.2406±0.0005
0.4524±0.0005
CEDRL [ 29 ]
0.3629±0.0071
0.6817±0.0134
Ours
0.2064±0.0069
0.3897±0.0141
Students003
GBM [ 12 ]
0.2147±0.0026
0.4194±0.0056
CCP [ 30 ]
0.2053±0.0031
0.3873±0.0060
Appendix
Table 7: Motion fidelity on held-out trajectories. ADE and FDE (in meters) compare generated trajectories against reference human trajectories. Lower is better for both metrics; bold and underline mark the best and second-best results. Values are reported with 95% confidence intervals.
Figure 7: User study results for motion quality, naturalness, and smoothness. Bars show participant-level mean ratings. Brackets mark Holm-corrected paired Wilcoxon signed-rank tests comparing our method against each baseline (* p<.05 , ** p<.01 ).
Figure 8: Segmentation image sequence from ETH. Columns are ordered in time; rows are as in Figure 5 .
Figure 9: Failure case of a segmentation image sequence from ETH. Rows are as in Figure 5 . White boxes mark regions where the predicted segmentation deviates from the ground truth and loses temporal consistency.
Table 8: World model and VLA compute and runtime. Runtime is measured per candidate for the world models and per decision for the deployed policy.
Table 9: Anchor-selection prompt template. The prompt asks the policy to choose the safest and most natural anchor for crowd simulation based on the current image, ego state, goal, nearby agents, and nearby dynamic objects.
Figure 10: Additional qualitative comparison on group-coherent navigation. We compare CCP, CEDRL, GBM, and our method in a representative group-navigation scene. Each row shows a later timestep in the same rollout, and trajectory traces indicate recent motion over the previous 60 frames. The highlighted regions show representative failure modes of the baseline methods, including group separation, collision with a vehicle, and excessive group spread that causes group members to move into road regions. By contrast, our method maintains a more compact and coherent relation among group members while avoiding vehicles and road regions, producing trajectories that better preserve group-level motion and scene compliance.
Figure 11: Qualitative ablation on the dual world models. We compare our full method with the variant without the dual world models in a representative closed-loop navigation scene; time runs from left to right. Without the dual world models, the agent shows typical failure cases such as less anticipatory interaction behavior, unstable trajectory adjustment, or reduced compliance with scene semantics. In contrast, the full model, refined with world-model-predicted consequences, produces more stable, safe, and scene-aware trajectories.
Figure 12: Narrow-alley interaction with an approaching vehicle. Blue highlights indicate collision-prone trajectories produced by existing methods and the ablated variant, while green highlights show the collision-free trajectory produced by our method.
Figure 13: Screenshot of the Google Form used for the user study. The study was administered through Google Forms. Participants watched anonymized rollout videos and answered three 7-point Likert-scale questions evaluating motion quality, naturalness, and smoothness.
Scene
Method
GSR ↑
SRVR ↓
GD ↓
Zara01
OnlineWM
78.9
5.7
0.6
Constant Velocity [ 36 ]
51.8
15.1
0.7
Social-LSTM [ 1 ]
44.4
14.1
0.7
Social-GAN [ 15 ]
44.4
15.0
1.0
Trajectron++ [ 35 ]
48.2
13.8
0.8
Gen-Safe-Nav [ 45 ]
37.0
19.1
1.3
Appendix
Table 10: Per-scene comparison with simpler pipelines and an online world-model planner. Trajectory predictors are combined with an MPC planner [ 33 ] . Higher is better for GSR, while lower is better for SRVR and GD. Bold marks the best result in each scene among methods other than OnlineWM.
Scene
Variant
GSR ↑
SRVR ↓
GD ↓
Zara01
State only
44.4
13.1
0.6
Vision only
61.2
10.9
0.8
Ours
72.2
7.0
0.6
Zara01_Group
State only
44.4
12.3
2.2
Vision only
44.4
6.4
2.3
Ours
77.8
1.2
2.1
Appendix
Table 11: Input ablation on five held-out evaluation scenarios. State only removes the visual input; Vision only removes the hand-crafted state features except the goal. Higher is better for GSR, while lower is better for SRVR and GD.
Most-likely
Best-of-20
Dataset
Social-LSTM [ 1 ]
Trajectron++ [ 35 ]
Ours
Social-GAN [ 15 ]
Trajectron++ [ 35 ]
Ours
ETH
1.09/2.35
1.02/2.00
1.05/2.08
0.81/1.52
0.67/1.18
0.65/1.08
Hotel
0.79/1.76
0.33/0.62
0.37/0.68
0.72/1.61
0.18/0.28
0.25/0.30
Univ
0.67/1.40
0.53/1.19
0.40/1.02
0.60/1.26
0.30/0.54
0.23/0.50
Zara1
0.47/1.00
0.44/0.99
0.38/0.90
0.34/0.69
0.25/0.41
0.22/0.40
Zara2
0.56/1.17
0.32/0.73
0.35/0.79
0.42/0.84
0.18/0.32
0.20/0.38
Appendix
Table 12: ETH/UCY forecasting benchmark (ADE/FDE in meters, lower is better). Trajectron++ numbers are those after the evaluation bug fix [ 43 ] .
Social navigation requires robots to act safely in dynamic human environments. Effective behavior demands thinking ahead: reasoning about how the scene and pedestrians evolve under different robot actions rather than reacting to current observations alone. This creates a coupled prediction-planning challenge, where robot actions and human motion mutually influence each other. To address this challenge, we propose NavThinker, a future-aware framework that couples an action-conditioned world model with on-policy reinforcement learning. The world model operates in the Depth Anything V2 patch feature space and performs autoregressive prediction of future scene geometry and human motion; multi-head decoders then produce future depth maps and human trajectories, yielding a future-aware state aligned with traversability and interaction risk. Crucially, we train the policy with DD-PPO while injecting world-model think-ahead signals via: (i) action-conditioned future features fused into the current observation embedding and (ii) social reward shaping from predicted human trajectories. Experiments on single- and multi-robot Social-HM3D show state-of-the-art navigation success, with zero-shot transfer to Social-MP3D and real-world deployment on a Unitree Go2, validating generalization and practical applicability. Webpage: https://hutslib.github.io/NavThinker.
Tianshuai Hu, Zeying Gong, Lingdong Kong +7
The Hong Kong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou) · National University of Singapore +1
Enabling embodied agents to imagine future states is essential for robust and generalizable visual navigation. Yet, state-of-the-art systems typically rely on modular designs that decouple navigation planning from visual world modeling, which often induces state-action misalignment and weak adaptability in novel or dynamic scenarios. We propose UniWM, a unified, memory-augmented world model that integrates egocentric visual foresight and planning within a single multimodal autoregressive backbone. UniWM explicitly grounds action selection in visually imagined outcomes, tightly aligning prediction with control. Meanwhile, a hierarchical memory mechanism fuses short-term perceptual cues with longer-term trajectory context, supporting stable and coherent reasoning over extended horizons. Extensive experiments on four challenging benchmarks (Go Stanford, ReCon, SCAND, HuRoN) and the 1X Humanoid Dataset show that UniWM improves navigation success rates by up to 30%, substantially reduces trajectory errors against strong baselines, generalizes zero-shot to the unseen TartanDrive dataset, and scales naturally to high-dimensional humanoid navigation. These results position UniWM as a principled step toward unified, imagination-driven embodied navigation. The code and models are available at https://github.com/UWMILab/UniWM.
Yifei Dong, Fengyi Wu, Guangyu Chen +9
University of Washington · National University of Singapore · Apple +2
World-model-based visual navigation enables action planning by predicting future states under candidate trajectories. However, long-horizon rollout remains challenging: models are often trained with ground-truth visual histories but deployed in closed-loop prediction, and evaluating many candidates with step-by-step rollout is computationally expensive. We propose SkipWorld, a latent skip planning framework for efficient visual navigation. SkipWorld introduces three components: (i) Latent AutoRoll, which trains a world model to recursively roll out future states in latent space, reducing the mismatch between training and closed-loop inference; (ii) SkipDistill, which distills the latent autoregressive model into a skip world model that directly predicts long-stride future latent states; and (iii) Skip-then-Refine Planning, a dual-model coarse-to-fine planner that first filters candidate trajectories with the skip model and then reranks selected candidates with the stronger autoregressive model. This design preserves accurate trajectory evaluation while reducing redundant rollout computation. Experiments on RECON, HuRoN, SCAND, and TartanDrive show that, compared with NWM, SkipWorld reduces the average ATE by 4.2% and RPE by 2.7%, while decreasing planning time by 62.4% with a 2.7-fold speedup.
Yifei Yang, Zehua Fan, Zike Yan +2
Institute for AI Industry Research (AIR), Tsinghua University · Hong Kong Embodied AI Lab · The Chinese University of Hong Kong +1