NavThinker: Action-Conditioned World Models for Coupled Prediction and Planning in Social Navigation

Tianshuai Hu; Zeying Gong; Lingdong Kong; XiaoDong Mei; Yiyi Ding; Qi Zeng; Ao Liang; Rong Li

cs.ROMar 16, 2026

NavThinker: Action-Conditioned World Models for Coupled Prediction and Planning in Social Navigation

Authors: Tianshuai Hu, Zeying Gong, Lingdong Kong, XiaoDong Mei, Yiyi Ding, Qi Zeng, Ao Liang, Rong Li,

Organizations: The Hong Kong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou) · National University of Singapore · University of Chinese Academy of Sciences

Abstract

Social navigation requires robots to act safely in dynamic human environments. Effective behavior demands thinking ahead: reasoning about how the scene and pedestrians evolve under different robot actions rather than reacting to current observations alone. This creates a coupled prediction-planning challenge, where robot actions and human motion mutually influence each other. To address this challenge, we propose NavThinker, a future-aware framework that couples an action-conditioned world model with on-policy reinforcement learning. The world model operates in the Depth Anything V2 patch feature space and performs autoregressive prediction of future scene geometry and human motion; multi-head decoders then produce future depth maps and human trajectories, yielding a future-aware state aligned with traversability and interaction risk. Crucially, we train the policy with DD-PPO while injecting world-model think-ahead signals via: (i) action-conditioned future features fused into the current observation embedding and (ii) social reward shaping from predicted human trajectories. Experiments on single- and multi-robot Social-HM3D show state-of-the-art navigation success, with zero-shot transfer to Social-MP3D and real-world deployment on a Unitree Go2, validating generalization and practical applicability. Webpage: https://hutslib.github.io/NavThinker.

Figures & tables

Fig. 1: NavThinker: future-aware social navigation. Social navigation requires the robot to reach a goal in environments shared with dynamic pedestrians. Red regions highlight interaction zones where the robot and pedestrians may conflict. Top: egocentric depth observations. Bottom: the world model imagines action-conditioned futures for the corresponding observations. By imagining how the scene evolves under different candidate actions, NavThinker anticipates potential social conflicts and selects safe, socially compliant actions.

Fig. 2: Overview of the NavThinker framework. Our framework consists of two modules: a world model that learns action-conditioned scene dynamics (top), and an imagination-augmented planner policy trained with DD-PPO (bottom). During World Model Learning , depth observations, actions, and robot states are extracted from the Replay Buffer. A frozen DA-V2 ViT encoder extracts patch embeddings, and a Causal Attention Transformer autoregressively predicts future latent features. A Depth Decoder and a Human Trajectory Decoder are trained alongside a latent consistency loss to anchor representations to scene geometry and human motion. During Policy Learning , the Observation Encoder produces the current embedding from depth and robot states. The Imagination module queries the world model under each candidate action, generating action-conditioned future features that are fused with the current embedding via Feature Fusion. The fused representation feeds into the DD-PPO Actor-Critic network. Predicted human trajectories additionally provide Reward Shaping, coupling prediction with planning in an imagine-then-act loop.

Methods	Social-HM3D				Social-MP3D
	SR $\uparrow$	SPL $\uparrow$	PSC $\uparrow$	H-Coll $\downarrow$	SR $\uparrow$	SPL $\uparrow$	PSC $\uparrow$	H-Coll $\downarrow$
Rule-based
A* [ 34 ]	44.81	43.99	90.38	54.80	45.67	44.69	91.97	54.00
ORCA [ 35 ]	37.44	32.91	92.23	39.77	38.81	34.65	94.03	39.86
Reinforcement Learning-based
Habitat-official [ 36 ]	38.99	33.53	90.37	55.48	37.00	31.76	92.03	52.33

TABLE I: Single-Robot Social Navigation Results on Social-HM3D and Social-MP3D (zero-shot transfer from Social-HM3D). We report SR/SPL (success/efficiency), PSC (social compliance), and H-Coll (human collisions). Best and 2nd-best are in bold and underline .

Method	SR $\uparrow$	SPL $\uparrow$	PSC $\uparrow$	H-Coll $\downarrow$	T-SR $\uparrow$	T-SPL $\uparrow$
Rule-Based
A* [ 34 ]	26.06	25.70	95.20	35.68	14.76	14.51
ORCA [ 35 ]	24.13	22.48	95.53	35.13	15.09	13.84
Reinforcement Learning
Habitat-official [ 36 ]	26.78	24.41	95.47	31.39	14.98	13.68
Falcon [ 22 ]	28.63	26.48	94.98	28.63	16.08	15.12

TABLE II: Multi-Robot Social Navigation Results on Social-HM3D. Robots navigate to individual goals without communication. We report SR/SPL (success/efficiency), PSC (social compliance), H-Coll (human collisions), and team-level T-SR/T-SPL. Best and second-best are highlighted in bold and underline , respectively.

Fig. 3: World model imagination quality. Left: current depth observation with RGB, third-person view, and top-down map for reference. Right: after executing the action at the previous timestep, we show (from top to bottom) the ground-truth depth, the world model’s predicted depth, the PCA visualization of ground-truth DA-V2 features, and the PCA visualization of predicted features. The world model produces faithful depth predictions and latent features that closely match the ground truth across different actions.

LookH.	TrajR.	SR $\uparrow$	SPL $\uparrow$	PSC $\uparrow$	H-Coll $\downarrow$
$\times$	$\times$	43.36	36.27	90.81	47.33
$\checkmark$	$\times$	55.87	49.36	90.58	39.19
$\checkmark$	$\checkmark$	59.46	55.00	89.91	39.09

TABLE III: Ablation Study of NavThinker planning policy components on the single-robot Social-HM3D benchmark. LookH.: world-model lookahead imagination; TrajR.: trajectory-based social reward shaping. Metrics include SR/SPL (success/efficiency), PSC (social compliance), and H-Coll (human collisions).

LookH.	TrajR.	SR $\uparrow$	SPL $\uparrow$	PSC $\uparrow$	H-Coll $\downarrow$	T-SR $\uparrow$	T-SPL $\uparrow$
$\times$	$\times$	13.64	11.27	93.85	59.25	13.44	11.13
$\checkmark$	$\times$	28.52	26.52	95.47	25.88	14.76	13.73
$\checkmark$	$\checkmark$	30.04	28.14	95.55	25.55	16.30	15.22

TABLE IV: Ablation Study of planner policy components on multi-robot Social-HM3D. LookH.: world-model lookahead imagination; TrajR.: trajectory-based social reward shaping. Metrics include SR/SPL (success/efficiency), PSC (social compliance), H-Coll (human collisions), and T-SR/T-SPL (team-level success/efficiency).

$\mathcal{L}_{d}$	$\mathcal{L}_{\xi}$	CosSim $\uparrow$	Depth RMSE $\downarrow$	Traj ADE $\downarrow$	Traj FDE $\downarrow$
$\times$	$\times$	0.9278	0.0406	—	—
$\checkmark$	$\times$	0.9342	0.0389	—	—
$\checkmark$	$\checkmark$	0.9383	0.0371	1.74	1.93

TABLE V: Ablation Study on auxiliary losses for World Model training. Evaluation on 200 episodes from the held-out replay buffer.

\mathcal{L}_{d}

is the depth reconstruction loss and

\mathcal{L}_{\xi}

is the human trajectory prediction loss. CosSim is the cosine similarity between predicted and target latent features. Depth RMSE is the root mean squared error of predicted depth images. Traj ADE/FDE are the average and final displacement errors of predicted human trajectories.

Fig. 4: Qualitative comparisons on Social-HM3D. Navthinker produces safe actions under different social situations. Rows 1–2 (Blind corner): A* collides when turning, while ours avoids the pedestrian. Rows 3–4 (Intersection): Falcon collides at the intersection, while ours safely yields and passes. Rows 5–6 (Front approach): Habitat-official collides in a head-on encounter, while ours gives way for the pedestrian. Red boxes mark baseline failures; green checkmarks indicate safe behaviors.

Fig. 5: Real-world deployment on a Unitree Go2 robot. In this scenario, the robot navigates around a group of people and reaches its goal while maintaining safe distances.

Social Robot Navigation

Social robot navigation studies how robots move safely and comfortably around people while respecting human behavior, preferences, and shared-space norms.

See all 81 papers

HumAIN: Human-Aware Implicit Social Robot Navigation
Jul 8, 2026Daeun Song, Nhat Le, Jeffrey Chen +7Social Robot Navigation Robot Motion Planning
Ewha Womans University, Korea. · George Mason University, USA. · University of Virginia, USA. +3
Social-WM: Safety-Aware Latent World Models for Robot Social Navigation
Sep 30, 2026Zhihao Zheng, Mooi Choo ChuahWorld Models for Robotics World Model-Based Planning
Computer Science and Engineering department, P.C. Rossin College of Engineering and Applied Science, Lehigh University, Bethlehem, PA 18015, USA.
TRACER: Adaptive Multi-Robot Social Navigation via Joint Human-Response Prediction and Interaction-Aware Replanning
Sep 16, 2026Lan Hu, Minghui Liwang, Wenbo Zhu +4Multi-Robot Motion Planning Multi-Robot Systems
Guohao School, Tongji University, Shanghai, China · Shanghai Research Institute for Intelligent Autonomous Systems, the State Key Laboratory of Autonomous Intelligent Unmanned Systems, Department of Control Science and Engineering, Tongji University, Shanghai, China

Jul 8, 2026J/K move · Enter open · S save

HumAIN: Human-Aware Implicit Social Robot Navigation

Daeun Song, Nhat Le, Jeffrey Chen, Mohammad Nazeri+6

Ewha Womans University, Korea. · George Mason University, USA. · University of Virginia, USA.+3

Social Robot Navigation Robot Motion Planning Social Robotics

Open paper AskPDF arXiv

Effective social robot navigation requires sensitivity to human behavior, often revealed through subtle skeletal cues like gait and orientation. We present Human-Aware Implicit Social Robot Navigation (HumAIN), a novel framework that fuses implicit social cues directly into the planning loop via knowledge distillation. We first employ a transformer-based teacher model that fuses rich multi-modal inputs, including historic images, skeletal keypoints, robot state, and a robot's target goal, to learn robust, human-aware representations for the robot's future trajectory planning. To enable real-time deployment, we then distill this knowledge into a lightweight student model. By optimizing for both trajectory reconstruction and latent feature alignment with the teacher, the student learns to infer complex social dynamics from minimal inputs. Bridging the prediction-planning gap with an efficient distilled architecture, our method enables robots to reason about human behavior in a manner that is adaptive, robust, and socially compliant. We validate HumAIN through extensive experiments, where it improves trajectory prediction metrics by an average of 29.8% across all metrics compared to state-of-the-art baselines. These results highlight the benefit of using implicit, whole-body cues to achieve human-like navigation awareness on resource-constrained platforms.

Abstract

Social Robot Navigation

Explore similar work

HumAIN: Human-Aware Implicit Social Robot Navigation

Social-WM: Safety-Aware Latent World Models for Robot Social Navigation

TRACER: Adaptive Multi-Robot Social Navigation via Joint Human-Response Prediction and Interaction-Aware Replanning