STARS: From Spatiotemporal Dynamics to Social Representations in Human-Robot Interaction

Nathan Tsoi; Michael J. Munje; Tejas Oberoi; Rishab Maheshwari; Pengen Zheng; Tanush Chauhan; Peter Stone; Joydeep Biswas

cs.ROSep 30, 2026

STARS: From Spatiotemporal Dynamics to Social Representations in Human-Robot Interaction

Authors: Nathan Tsoi, Michael J. Munje, Tejas Oberoi, Rishab Maheshwari, Pengen Zheng, Tanush Chauhan, Peter Stone, Joydeep Biswas

Organizations: Department of Computer Science, The University of Texas at Austin

Abstract

Robot navigation in dynamic, human-centered environments requires socially-compliant decisions grounded in robust scene understanding. Recent Vision-Language Models (VLMs) exhibit promising capabilities such as object recognition, common-sense reasoning, and contextual understanding, capabilities that align with the nuanced requirements of social robot navigation. However, it remains unclear whether VLMs can accurately understand complex social navigation scenes (e.g., inferring the spatial-temporal relations among agents and human intentions), which is essential for safe and socially compliant robot navigation. While some recent works have explored the use of VLMs in social robot navigation, no existing work systematically evaluates their ability to meet these necessary conditions. In this paper, we introduce the Social Navigation Scene Understanding Benchmark (SocialNav-SUB), a Visual Question Answering (VQA) dataset and benchmark designed to evaluate VLMs for scene understanding in real-world social robot navigation scenarios. SocialNav-SUB provides a unified framework for evaluating VLMs against human and rule-based baselines across VQA tasks requiring spatial, spatiotemporal, and social reasoning in social robot navigation. Through experiments with state-of-the-art VLMs, we find that while the best-performing VLM achieves an encouraging probability of agreeing with human answers, it still underperforms simpler rule-based approach and human consensus baselines, indicating critical gaps in social scene understanding of current VLMs. Our benchmark sets the stage for further research on foundation models for social robot navigation, offering a framework to explore how VLMs can be tailored to meet real-world social robot navigation needs. An overview of this paper along with the code and data can be found at https://larg.github.io/stars.

Figures & tables

Figure 1: STARS three stages: ( A ) construct spatiotemporal graphs from social navigation datasets, ( B ) utilize an unsupervised message-passing graph autoencoder to compress physical dynamics into latent representations, and ( C ) adapt frozen latents to downstream tasks using lightweight task-specific prediction heads. See Sec. 3 for details.

Figure 2: (A) Dataset Overview. We utilize two human-robot interaction datasets: SEAN-T [ 47 ] for evaluating subjective human ratings of robot navigation and SNS [ 31 ] for downstream relational action classification (e.g., avoid , follow ). (B) Latent Space Analysis (RQ3). 2D Projections of the SNS Human Node Action Latent Space for the avoid and follow actions. We visualize the 128-dimensional representations of nodes extracted from the STARS-trained homogeneous GNN-VAE encoder using linear PCA and non-linear t-SNE . Each point represents a node embedding, color-coded by action label. See Sec. 4.2 for details.

	Method	1%	5%	10%	25%	50%	100%
SEAN-T Competence	MLP	$0.450\pm 0.22$	$0.546\pm 0.19$	$0.664\pm 0.15$	$0.761\pm 0.03$	$0.778\pm 0.02$	$0.797\pm 0.02$
	Autoencoder	$\mathbf{0.663\pm 0.08}$	$0.678\pm 0.05$	$0.706\pm 0.05$	$0.734\pm 0.02$	$0.747\pm 0.02$	$0.742\pm 0.02$
	Random Forest	$0.557\pm 0.20$	$0.641\pm 0.12$	$0.680\pm 0.10$	$0.704\pm 0.09$	$0.715\pm 0.09$	$0.718\pm 0.08$
	STARS-He (Ours)	$0.635\pm 0.10$	$0.673\pm 0.07$	$0.726\pm 0.04$	$0.762\pm 0.03$	$0.781\pm 0.03$	$0.794\pm 0.02$
	STARS-Ho (Ours)	$0.612\pm 0.14$	$\mathbf{0.702\pm 0.08}$	$\mathbf{0.738\pm 0.05}$	$\mathbf{0.765\pm 0.04}$	$\mathbf{0.787\pm 0.03}$	$\mathbf{0.799\pm 0.02}$
SEAN-T Surprise	MLP	$0.217\pm 0.22$	$0.250\pm 0.14$	$0.423\pm 0.27$	$0.450\pm 0.21$	$0.630\pm 0.23$	$0.720\pm 0.09$

Table 1: Downstream Performance (RQ1) and Data Efficiency (RQ2) on SEAN-T and SNS. Results showcase

F_{1}

-Scores (

\mu\pm\sigma

) across progressively scaled fractions of labeled data. We compare unstructured baseline methods against our proposed STARS variants ( STARS-He : Heterogeneous, STARS-Ho : Homogeneous). To rigorously evaluate both representation quality and data efficiency, STARS utilizes a frozen pretrained encoder paired with a lightweight linear probe.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	# Scenarios	Sample Freq.	Window Size $T$	Window Duration	Max Nodes $N$	Max Edges
SEAN-T	2969	5 Hz	40	8s	16	240
SocialNav-SUB	3052	8 Hz	20	2.5s	31	930

Appendix

Table 2: Dataset and graph construction details. The temporal window is fixed within each dataset. The maximum number of edges follows from the maximum number of nodes under the fully connected graph construction.

Model	Dataset	Robot features	Human features
STARS-He	SEAN-T	592D map and goal features	8D learned embedding + 8D pooled edge features
STARS-He	SNS	8D learned embedding + 8D pooled edge features	8D learned embedding + 8D pooled edge features
STARS-Ho	SEAN-T	592D map and goal features + type embedding	592D zero vector + type embedding
STARS-Ho	SNS	592D zero vector + type embedding	592D zero vector + type embedding

Appendix

Table 3: Node features by architecture and dataset. All configurations use 8D edge features comprising relative position, relative heading (sine and cosine), and their temporal differences. Both architectures use reconstruction pretraining followed by linear probing with a frozen encoder.

Component	Homogeneous	Heterogeneous
Node feature projection	shared	per dataset and node type
Temporal edge encoder	shared ( $T{=}40$ )	per dataset ( $40$ , $20$ )
Message-passing layers	shared	shared
Edge-type embedding	not used	shared
Latent heads $f_{\mu},f_{\sigma}$	shared	shared
Edge decoder	shared ( $T{=}40$ )	per dataset

Appendix

Table 4: Components shared across datasets in each variant.

Panel of Figure 3 — Figure 3: STARS Heterogeneous architecture overview. Raw inputs (robot occupancy + goal, learned human embeddings, 8D temporal edge features) are projected to a 256D shared hidden space via type-specific preprocessing (MLPs for nodes, a Transformer for edges). The encoder performs HEAT-style message passing (MP) over the fully connected heterogeneous graph (2 layers), with attention and messages parameterized over the concatenated edge-conditioned representation $\mathbf{u}_{ij}$ . The resulting 256D node states are projected through shared linear heads to per-node Gaussian posteriors and sampled to 128D latents $\mathbf{z}_{i}$ via the reparameterization trick. The decoder reconstructs edge features (and robot features on SEAN) from latent pairs, trained jointly with a KL divergence term against $\mathcal{N}(0,I)$ . Pretrained encoders are frozen for downstream linear probing (SLA).

Model / Component	Stage	Trainable Params.	Frozen Params.	Notes
STARS-He Encoder/Decoder	Pretraining	6,420,810	N/A	Heterogeneous graph autoencoder
STARS-Ho Encoder/Decoder	Pretraining	2,936,848	N/A	Homogeneous graph autoencoder
Linear Prediction Head	Downstream adaptation	1,285 (SNS), 771 (SEAN)	6,420,810	Used with frozen STARS encoder

Appendix

Table 5: Model capacity comparison. For STARS, parameter counts are separated into the pretrained representation model and the downstream task-specific prediction head.

Model / Pipeline	Dataset / Task	Pretraining Time	Downstream Training Time	Inference Time / Scenario	Peak GPU Memory
STARS-He + Linear Head	SEAN-T + SNS	$\sim$ 2.5 hrs / run	$<$ 1 sec	$\sim$ 12 ms	$\sim$ 13 GB
STARS-Ho + Linear Head	SEAN-T + SNS	$\sim$ 3 hrs / run	$\sim$ 12 sec	$\sim$ 30 ms	$\sim$ 17 GB

Appendix

Table 6: Runtime and memory usage for each experimental pipeline. For STARS-based pipelines, pretraining is performed once and the frozen encoder is reused for downstream tasks. Downstream training time refers to fitting the task-specific prediction head. Inference runtime is reported per interaction scenario.

Task	Method	1%	5%	10%	25%	50%	100%
Competence	STARS-He (Ours)	$0.635\pm 0.10$	$0.673\pm 0.07$	$0.726\pm 0.04$	$0.762\pm 0.03$	$0.781\pm 0.03$	$0.794\pm 0.02$
Competence	STARS-Ho (Ours)	$0.612\pm 0.14$	$0.702\pm 0.08$	$0.738\pm 0.05$	$0.765\pm 0.04$	$0.787\pm 0.03$	$0.799\pm 0.02$
Surprise	STARS-He (Ours)	$0.455\pm 0.17$	$0.481\pm 0.17$	$0.526\pm 0.15$	$0.603\pm 0.12$	$0.655\pm 0.08$	$0.709\pm 0.04$
Surprise	STARS-Ho (Ours)	$0.440\pm 0.23$	$0.541\pm 0.21$	$0.548\pm 0.20$	$0.610\pm 0.16$	$0.713\pm 0.06$	$0.741\pm 0.03$
Intention	STARS-He (Ours)	$0.576\pm 0.08$	$0.619\pm 0.05$	$0.671\pm 0.04$	$0.694\pm 0.03$	$0.705\pm 0.03$	$0.721\pm 0.03$
Intention	STARS-Ho (Ours)	$0.603\pm 0.10$	$0.671\pm 0.07$	$0.701\pm 0.06$	$0.715\pm 0.04$	$0.723\pm 0.03$	$0.734\pm 0.02$

Appendix

Table 7: Extended STARS results on SEAN-T and SNS across labeled-data fractions.

Task	Model	1%	5%	10%	25%	50%	100%
Competence	STARS-He	$0.568\pm 0.10$	$0.636\pm 0.07$	$0.686\pm 0.06$	$0.733\pm 0.03$	$0.773\pm 0.03$	$0.792\pm 0.02$
Surprise	STARS-He	$0.455\pm 0.14$	$0.453\pm 0.16$	$0.480\pm 0.22$	$0.55\pm 0.15$	$0.653\pm 0.09$	$0.716\pm 0.05$
Intention	STARS-He	$0.528\pm 0.09$	$0.601\pm 0.05$	$0.644\pm 0.05$	$0.683\pm 0.03$	$0.705\pm 0.03$	$0.721\pm 0.02$

Appendix

Table 8: SEAN-T downstream performance after SEAN-T-only pretraining. Results report

F_{1}

-Score (

\mu\pm\sigma

) across labeled-data fractions.

Model	Pretraining	1%	5%	10%	25%	50%	100%
STARS-He	SEAN-T only	$0.355\pm 0.06$	$0.442\pm 0.05$	$0.463\pm 0.04$	$0.489\pm 0.03$	$0.493\pm 0.03$	$0.497\pm 0.03$
STARS-He	SEAN-T+SNS	$0.362\pm 0.06$	$0.447\pm 0.05$	$0.470\pm 0.04$	$0.493\pm 0.03$	$0.501\pm 0.03$	$0.503\pm 0.02$
STARS-Ho	SEAN-T only	$0.274\pm 0.05$	$0.332\pm 0.04$	$0.351\pm 0.03$	$0.365\pm 0.03$	$0.407\pm 0.02$	$0.419\pm 0.01$
STARS-Ho	SEAN-T+SNS	$0.344\pm 0.05$	$0.431\pm 0.05$	$0.453\pm 0.04$	$0.468\pm 0.03$	$0.471\pm 0.02$	$0.475\pm 0.02$

Appendix

Table 9: Cross-dataset transfer to SNS. Models are pretrained either on SEAN-T only or jointly on SEAN-T and SNS and evaluated on SNS using progressively larger fractions of labeled training data. Results report macro

F_{1}

-Score (

\mu\pm\sigma

Model	Pretraining	Task	1%	5%	10%	25%	50%	100%
STARS-He	SNS only	Competence	$0.615\pm 0.10$	$0.674\pm 0.08$	$0.731\pm 0.05$	$0.772\pm 0.03$	$0.786\pm 0.03$	$0.801\pm 0.01$
STARS-He	SNS only	Surprise	$0.409\pm 0.20$	$0.446\pm 0.18$	$0.493\pm 0.17$	$0.595\pm 0.11$	$0.652\pm 0.08$	$0.710\pm 0.05$
STARS-He	SNS only	Intention	$0.571\pm 0.09$	$0.619\pm 0.08$	$0.668\pm 0.04$	$0.700\pm 0.03$	$0.717\pm 0.03$	$0.729\pm 0.03$
STARS-He	SEAN-T+SNS	Competence	$0.635\pm 0.10$	$0.673\pm 0.07$	$0.726\pm 0.04$	$0.762\pm 0.03$	$0.781\pm 0.03$	$0.794\pm 0.02$
STARS-He	SEAN-T+SNS	Surprise	$0.455\pm 0.17$	$0.481\pm 0.17$	$0.526\pm 0.15$	$0.603\pm 0.12$	$0.655\pm 0.08$	$0.709\pm 0.04$
STARS-He	SEAN-T+SNS	Intention	$0.576\pm 0.08$	$0.619\pm 0.05$	$0.671\pm 0.04$	$0.694\pm 0.03$	$0.705\pm 0.03$	$0.721\pm 0.03$

Appendix

Table 10: Cross-dataset transfer to SEAN-T. Models are pretrained either on SNS only or jointly on SEAN-T and SNS and evaluated on the three SEAN-T perception tasks using progressively larger fractions of labeled training data. Results report

F_{1}

-Score (

\mu\pm\sigma

Task	Method	1%	5%	10%	25%	50%	100%
Avoid	Hetero Self-Sup	$0.098\pm 0.09$	$0.173\pm 0.11$	$0.190\pm 0.10$	$0.219\pm 0.10$	$0.230\pm 0.09$	$0.242\pm 0.08$
Avoid	Hetero Sup	$0.178\pm 0.07$	$0.234\pm 0.05$	$0.258\pm 0.04$	$0.292\pm 0.03$	$0.297\pm 0.02$	$0.284\pm 0.02$
Follow	Hetero Self-Sup	$0.155\pm 0.15$	$0.327\pm 0.12$	$0.374\pm 0.08$	$0.404\pm 0.04$	$0.412\pm 0.03$	$0.405\pm 0.03$
Follow	Hetero Sup	$0.112\pm 0.11$	$0.287\pm 0.07$	$0.305\pm 0.06$	$0.353\pm 0.04$	$0.373\pm 0.04$	$0.384\pm 0.04$
Not Consider	Hetero Self-Sup	$0.834\pm 0.06$	$0.840\pm 0.04$	$0.846\pm 0.03$	$0.855\pm 0.02$	$0.861\pm 0.02$	$0.862\pm 0.02$
Not Consider	Hetero Sup	$0.218\pm 0.25$	$0.351\pm 0.22$	$0.429\pm 0.23$	$0.604\pm 0.13$	$0.662\pm 0.08$	$0.667\pm 0.07$

Appendix

Table 11: SNS Action Classification: Self-Supervised vs. Supervised GNN-VAE. We compare STARS’ frozen self-supervised encoder with a linear probe against an identical architecture trained end-to-end with supervised classification. Both models use the heterogeneous graph formulation. Results report F1-Score (

\mu\pm\sigma

) across label fractions.

		Architecture	1%	5%	10%	25%	50%	100%
Random encoder:	Competence	Heterogeneous	$0.530\pm 0.22$	$0.596\pm 0.16$	$0.635\pm 0.12$	$0.677\pm 0.09$	$0.691\pm 0.08$	$0.707\pm 0.07$
	Surprise	Heterogeneous	$0.251\pm 0.20$	$0.296\pm 0.21$	$0.372\pm 0.17$	$0.424\pm 0.13$	$0.467\pm 0.12$	$0.480\pm 0.14$
	Intention	Heterogeneous	$0.606\pm 0.20$	$0.648\pm 0.14$	$0.677\pm 0.10$	$0.700\pm 0.08$	$0.704\pm 0.08$	$0.713\pm 0.09$
	$F_{1}$ Avoid	Heterogeneous	$0.070\pm 0.11$	$0.251\pm 0.08$	$0.301\pm 0.03$	$0.304\pm 0.02$	$0.305\pm 0.02$	$0.302\pm 0.02$
	$F_{1}$ Follow	Heterogeneous	$0.070\pm 0.12$	$0.065\pm 0.11$	$0.101\pm 0.12$	$0.101\pm 0.11$	$0.082\pm 0.10$	$0.095\pm 0.10$
	$F_{1}$ Not Consider	Heterogeneous	$0.865\pm 0.03$	$0.849\pm 0.02$	$0.833\pm 0.02$	$0.839\pm 0.01$	$0.839\pm 0.01$	$0.839\pm 0.00$

Appendix

Table 12: Random encoder ablation results.

		Architecture	1%	5%	10%	25%	50%	100%
128D robot-z only	Competence	Heterogeneous	$0.592\pm 0.13$	$0.627\pm 0.10$	$0.663\pm 0.08$	$0.695\pm 0.07$	$0.707\pm 0.07$	$0.711\pm 0.08$
	Surprise	Heterogeneous	$0.373\pm 0.12$	$0.400\pm 0.12$	$0.410\pm 0.11$	$0.432\pm 0.12$	$0.454\pm 0.12$	$0.479\pm 0.14$
	Intention	Heterogeneous	$0.623\pm 0.12$	$0.662\pm 0.08$	$0.696\pm 0.08$	$0.716\pm 0.07$	$0.730\pm 0.07$	$0.730\pm 0.07$
	$F_{1}$ Avoid	Heterogeneous	$0.017\pm 0.05$	$0.004\pm 0.02$	$0.001\pm 0.01$	$0.000\pm 0.00$	$0.000\pm 0.00$	$0.000\pm 0.00$
	$F_{1}$ Follow	Heterogeneous	$0.004\pm 0.02$	$0.008\pm 0.03$	$0.007\pm 0.02$	$0.003\pm 0.01$	$0.000\pm 0.00$	$0.000\pm 0.00$
	$F_{1}$ Not Consider	Heterogeneous	$0.875\pm 0.02$	$0.881\pm 0.01$	$0.883\pm 0.00$	$0.883\pm 0.00$	$0.883\pm 0.00$	$0.883\pm 0.00$

Appendix

Table 13: Robot-latent only (128D) ablation results.

		Architecture	1%	5%	10%	25%	50%	100%
128D byst only	Competence	Heterogeneous	$0.551\pm 0.17$	$0.533\pm 0.15$	$0.556\pm 0.15$	$0.621\pm 0.10$	$0.642\pm 0.09$	$0.650\pm 0.07$
	Surprise	Heterogeneous	$0.323\pm 0.15$	$0.337\pm 0.13$	$0.302\pm 0.13$	$0.232\pm 0.14$	$0.161\pm 0.15$	$0.149\pm 0.14$
	Intention	Heterogeneous	$0.574\pm 0.17$	$0.598\pm 0.15$	$0.634\pm 0.13$	$0.693\pm 0.09$	$0.713\pm 0.09$	$0.717\pm 0.09$
	$F_{1}$ Avoid	Heterogeneous	$0.134\pm 0.10$	$0.154\pm 0.08$	$0.167\pm 0.08$	$0.149\pm 0.09$	$0.148\pm 0.08$	$0.153\pm 0.09$
	$F_{1}$ Follow	Heterogeneous	$0.223\pm 0.12$	$0.317\pm 0.06$	$0.341\pm 0.04$	$0.344\pm 0.03$	$0.344\pm 0.03$	$0.343\pm 0.03$
	$F_{1}$ Not Consider	Heterogeneous	$0.661\pm 0.08$	$0.756\pm 0.06$	$0.786\pm 0.04$	$0.807\pm 0.03$	$0.807\pm 0.03$	$0.811\pm 0.02$

Appendix

Table 14: 128D byst only ablation results.

	Architecture	1%	5%	10%	25%	50%	100%
Competence	Heterogeneous	$0.541\pm 0.11$	$0.641\pm 0.07$	$0.673\pm 0.07$	$0.714\pm 0.05$	$0.724\pm 0.04$	$0.746\pm 0.03$
Surprise	Heterogeneous	$0.528\pm 0.10$	$0.591\pm 0.09$	$0.610\pm 0.08$	$0.652\pm 0.05$	$0.682\pm 0.04$	$0.700\pm 0.04$
Intention	Heterogeneous	$0.498\pm 0.09$	$0.580\pm 0.06$	$0.590\pm 0.05$	$0.632\pm 0.05$	$0.649\pm 0.05$	$0.688\pm 0.05$

Appendix

Table 15: 592D robot node only (SEAN-ablation compared with combined model).

		Architecture	1%	5%	10%	25%	50%	100%
WRB:	Competence	Heterogeneous	$0.524\pm 0.14$	$0.528\pm 0.12$	$0.517\pm 0.10$	$0.517\pm 0.10$	$0.530\pm 0.11$	$0.513\pm 0.10$
	Surprise	Heterogeneous	$0.345\pm 0.13$	$0.366\pm 0.12$	$0.384\pm 0.10$	$0.360\pm 0.08$	$0.368\pm 0.08$	$0.360\pm 0.09$
	Intention	Heterogeneous	$0.563\pm 0.11$	$0.574\pm 0.09$	$0.575\pm 0.09$	$0.578\pm 0.08$	$0.575\pm 0.08$	$0.563\pm 0.08$
	$F_{1}$ Avoid	Heterogeneous	$0.090\pm 0.07$	$0.104\pm 0.05$	$0.104\pm 0.04$	$0.108\pm 0.04$	$0.105\pm 0.05$	$0.103\pm 0.05$
	$F_{1}$ Follow	Heterogeneous	$0.046\pm 0.06$	$0.062\pm 0.05$	$0.069\pm 0.05$	$0.083\pm 0.04$	$0.070\pm 0.05$	$0.083\pm 0.05$
	$F_{1}$ Not Consider	Heterogeneous	$0.812\pm 0.05$	$0.806\pm 0.03$	$0.800\pm 0.02$	$0.811\pm 0.02$	$0.813\pm 0.02$	$0.809\pm 0.01$

Appendix

Table 16: WRB ablation results.

Model	$F_{1}$ Avoid	$F_{1}$ Follow	$F_{1}$ Overtake	$F_{1}$ Yield	$F_{1}$ NC	PA
Gemini	$0.223$	$0.000$	$0.000$	$0.222$	$0.745$	$0.493$
4o	$0.241$	$0.176$	$0.000$	$0.000$	$0.521$	$0.344$
o4-mini	$0.222$	$0.363$	$0.000$	$0.111$	$0.852$	$0.582$

Appendix

Table 17: SocialNav-SUB VLM baseline results on the held-out SNS action classification task [ 31 ] . Models are evaluated on the full downstream test set. Metrics report per-class

F_{1}

-scores alongside Probability of Agreement ( PA ).

Explore similar work

Vision-Language Models for Deployable Social Robot Navigation: Bridging Semantic Reasoning and Low-Level Control
Jun 27, 2026Runji Cai, Toshihiko Yamasaki, Ling XiaoVision-Language Navigation Robot Navigation
Graduate School of Information Science and Technology, Hokkaido University, Sapporo, 060-0814, Japan. · Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, 113-8656, Japan.
Think When It Matters: Conditional VLM Reasoning for Social Navigation with RL Policies
Jul 13, 2026Ali Ahmadi, Hamed Rahimi, Adrien Jacquet Cretides +3Robot Navigation Safe Navigation
Institut des Syst`emes Intelligents et de Robotique (ISIR) Sorbonne Universit´e France
G2-Nav: Grounded and Guarded Vision-Language Costmaps for Robot Social Navigation
Jul 18, 2026Yuwen Liao, Yihang Lan, Yizhuo Yang +4Robot Navigation Autonomous Navigation
School of Electrical and Electronic Engineering Nanyang Technological University Singapore

Jun 27, 2026, cs.ROJ/K move · Enter open · S save

Vision-Language Models for Deployable Social Robot Navigation: Bridging Semantic Reasoning and Low-Level Control

Runji Cai, Toshihiko Yamasaki, Ling Xiao

Graduate School of Information Science and Technology, Hokkaido University, Sapporo, 060-0814, Japan. · Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, 113-8656, Japan.

Vision-Language Navigation Robot Navigation Object-Level Control

Open paper AskPDF arXiv

Social robot navigation (SRN) requires more than geometric path planning; it demands understanding human intentions, social norms, and contextual cues to generate socially compliant behaviors. Although classical navigation methods provide reliable metric planning and collision avoidance, they often lack the semantic reasoning capabilities necessary for operation in complex human-centered environments. Recent advances in Vision-Language Models (VLMs) have opened new opportunities for SRN by enabling high-level VLM understanding, commonsense reasoning, and natural language interaction. However, a fundamental challenge remains: how to integrate VLMs into real-time, safety-critical navigation systems and reliably translate their high-level reasoning into grounded navigation actions. In this survey, we present a unified perspective of VLM-based SRN and organize existing approaches into three interconnected components: high-level VLM reasoning, low-level planning and control, and intermediate mechanisms that bridge reasoning and action. Based on this perspective, we propose a structured roadmap for coupling VLMs with navigation systems, covering semantic reasoning, evaluators, spatial grounding, intermediate representations, and control modules. The roadmap highlights both the strengths of VLMs and the necessity of hybrid architectures for practical deployment. We further review representative datasets and evaluation platforms developed for SRN. Finally, we discuss key open challenges. This survey aims to provide a foundation for building reliable, socially compliant, and deployable VLM-enabled navigation systems.

Abstract

Explore similar work

Vision-Language Models for Deployable Social Robot Navigation: Bridging Semantic Reasoning and Low-Level Control

Think When It Matters: Conditional VLM Reasoning for Social Navigation with RL Policies

G2-Nav: Grounded and Guarded Vision-Language Costmaps for Robot Social Navigation