Humans build shared spatial understanding by communicating partial, viewpoint-dependent observations. We ask whether Multimodal Large Language Models (MLLMs) can do the same, aligning distinct egocentric views through dialogue to form a coherent, allocentric mental model of a shared environment. To study this systematically, we introduce COSMIC, a benchmark for Collaborative Spatial Communication. In this setting, two static MLLM agents observe a 3D indoor environment from different viewpoints and exchange natural-language messages to solve spatial queries. COSMIC contains 899 diverse scenes and 1250 question-answer pairs spanning five tasks. We find a capability hierarchy, MLLMs are most reliable at identifying shared anchor objects across views, perform worse on relational reasoning, and largely fail at building globally consistent maps, performing near chance, even for frontier models. Moreover, we find thinking capability yields gains in anchor grounding, but is insufficient for higher-level spatial communication. To contextualize model behavior, we collect 250 human-human dialogues. Humans achieve 95% aggregate accuracy, while the best model, Gemini-3-Pro-Thinking, reaches 72%, leaving substantial room for improvement. Moreover, human conversations grow more precise as partners align on a shared spatial understanding, whereas MLLMs keep exploring without converging, suggesting limited capacity to form and sustain a robust shared mental model throughout the dialogue. Our code and data is available at https://github.com/ankursikarwar/Cosmic.
Figures & tables
Figure 1: Left: MLLM agents attempt to communicate and build a spatial mental model for answering questions in COSMIC . Right: Answerer and Helper agents integrate distinct egocentric views via communication to answer the question. Humans demonstrate efficient and precise strategies while MLLM agents are more verbose, inefficient, and fail to build and maintain a robust shared mental model.
Figure 2: Overview of the COSMIC benchmark. Each task pair shows the Answerer’s view (left) and the Helper’s view (right), along with the question and options posed to the Answerer.
Figure 3: Benchmark curation pipeline. Our pipeline involves scene generation, sampling complementary agent viewpoints, generating questions using templates and unique object description followed by paraphrasing the questions.
Figure 4: COSMIC benchmark composition. Left: Scenes from our benchmark. Center Top: Distribution of room types. Center Bottom: Distribution of scene clutter (number of object instances per scene). Right Top: Object-category frequencies across the benchmark. Right Bottom: Word cloud representing the most frequent spatial and object-related terms in the dataset.
Figure 5: Top: Evaluation on COSMIC . Error bars denote 90% confidence intervals computed via bootstrap resampling. Dashed lines indicate chance levels (25% for 4-choice MCQ, 50% for binary map tasks and 30% for overall). Bottom: Evaluation on COSMIC-Human .
Figure 6: Left: Performance Hierarchy in MLLMs across Object, Relation and Map-level. Right: Two-Agent Communication vs Single Agent.
Figure 7: Failure mode analysis on COSMIC . Qualitative examples illustrate the three failure categories, Perceptual Failure (Object Recognition Failure), Cross-View Grounding Failure (Instance Merging), and Geometric and Relational Failure (Layout Understanding Failure). The bar chart shows the distribution of failure modes across tasks, and the donut chart shows the overall breakdown.
Figure 8: Communication Efficiency & Information Dynamics on Cosmic-Human . Top: Communication Efficiency (Verbosity vs. Accuracy) . Bottom: Information Dynamics (Unique Objects Mentioned vs. Conversation Turn Number) . Shaded regions denotes variance across conversations.
Figure 9: Dialogue Repair behavior in Gemini-3-Pro-Thinking.
Task
Template
Anchor Recognition
Which of the following objects is visible in both your and your partner’s views of the room?
Global Counting
What is the total number of < object category > in the room?
Relative Direction
From your perspective, in which direction is < object description > ?
Relative Distance
Which of the following objects is closest to / farthest from the < object description > ?
Cognitive Mapping
Is this top-down map of the room correct?
Table 1: Templates used for question generation. Below, object category is a placeholder for an object category, for example sofa , while object description is a placeholder for the unique description of a given object instance, for example green sofa near a white door .
Before Paraphrasing
After Paraphrasing
Question: Which of the following objects is visible in both your and your partner’s views of the room? Options: (A) floor purple Lamp (B) black shutter Window (C) green Plant Container next to a black Desk (D) Lamp near a black shutter Window
Question: Which of the following objects is present in both your and your partner’s views of the room? Options: (A) floor purple lamp (B) window with black shutters (C) green plant container located next to a black desk (D) lamp near a window with black shutters
Question: What is the total number of Monitor in the room? Options: (A) 1 (B) 2 (C) 3 (D) 4
Question: What is the total number of computer monitors in the room? Options: (A) 1 (B) 2 (C) 3 (D) 4
Question: Which of the following objects is closest to the floor purple Lamp? Options: (A) beige curtain Window (B) Monitor near a green Plant Container (C) brown curtain Window (D) green Plant Container next to a black Desk
Question: Which of the following objects is nearest to the floor lamp with purple color? Options: (A) window with beige curtains (B) computer monitor located near a green plant container (C) window with brown curtains (D) green plant container located next to a black desk
Question: From your perspective, in which direction is Window near a Monitor? Options: (A) behind (B) front (C) left (D) right
Question: From your perspective, in which direction is the window that is near a computer monitor? Options: (A) behind (B) front (C) left (D) right
Question: Is this top-down map of the room correct?
Question: Is this top-down layout of the room accurate?
Table 2: Examples of template-generated questions and their corresponding paraphrased versions.
Figure 10: Failure Mode Analysis. Qualitative samples illustrating different failure categories, Perceptual Failure (Attribute Mislabelling), Cross-View Grounding Failure (Referential Ambiguity and Instance Duplication), and Geometric & Relational Failure (Perspective-Taking Failure). Green ticks denote the ground truth answer.
Figure 11: Conversation between agents (Gemini-3-Pro-Thinking). Although the agents demonstrate successful behaviors on some atomic steps, errors in the dialogue negate these successes, leading them to select the incorrect answer (option A).
Figure 12: Samples from Cosmic Benchmark (Anchor Recognition).
Figure 13: Samples from Cosmic Benchmark (Global Counting).
Figure 14: Samples from Cosmic Benchmark (Relative Distance).
Figure 15: Samples from Cosmic Benchmark (Relative Direction).
Figure 16: Samples from Cosmic Benchmark (Cognitive Mapping).
Figure 17: Representative dialogues from Gemini-3-Pro-Thinking on Cosmic . The tick mark denotes the ground truth answer and the red-highlighted option indicates the answerer’s prediction. From top to bottom, Anchor Recognition (top-left), Global Counting (top-right), Relative Distance (middle-left), Relative Direction (middle-right), and Cognitive Mapping (bottom). Each panel displays the Answerer’s and Helper’s egocentric views, the multiple-choice options, and the multi-turn dialogue exchanged between the two agents.
Figure 18: Representative dialogues from Humans on Cosmic . The tick mark denotes the ground truth answer and the red-highlighted option indicates the answerer’s prediction. From top to bottom, Anchor Recognition (top-left), Global Counting (top-right), Relative Distance (middle-left), Relative Direction (middle-right), and Cognitive Mapping (bottom). Each panel displays the Answerer’s and Helper’s egocentric views, the multiple-choice options, and the multi-turn dialogue exchanged between the two agents.
Figure 19: Instruction page for the human dialogue data collection interface (Answerer)
Figure 21: Human Dialogue Data Collection Interface (Answerer)
Spatial intelligence is a key frontier for multimodal large language models (MLLMs), enabling them to reason about the physical world from visual experience. Inspired by human spatial cognition, recent approaches construct grid-based cognitive maps from multi-frame visual inputs to maintain coherent spatial representations over time. However, limited context lengths still challenge spatial understanding, while existing methods, such as long-context modeling and external memory, often require architectural changes, memory modules, or finetuning, limiting their applicability to off-the-shelf pretrained MLLMs. This motivates a lightweight, model-agnostic method for preserving spatial information beyond the native context window. To this end, we propose a plug-and-play multi-agent framework that collaboratively constructs cognitive maps as structured spatial memory, enhancing the spatial understanding of arbitrary pretrained MLLMs without architectural modification or additional training. Our framework features local-global agent coordination, cognitive map construction with atomic commits, and cross-agent verification. Extensive experiments demonstrate that our method achieves superior performance on spatial understanding tasks while remaining fully training-free. Code will be released.
Yiming Zhang, Ruoxuan Cao, Zhihang Zhong
2Cornell University · 1Shanghai Jiao Tong University
Multimodal large language models (MLLMs) are increasingly being applied to spatial cognition tasks, where they are expected to understand and interact with complex environments. Most existing works improve spatial reasoning by introducing 3D priors or geometric supervision, which enhances performance but incurs substantial data preparation and alignment costs. In contrast, purely 2D approaches often struggle with multi-frame spatial reasoning due to their limited ability to capture cross-frame spatial relationships. To address these limitations, we propose EgoMind, a Chain-of-Thought framework that enables geometry-free spatial reasoning through Role-Play Caption, which jointly constructs a coherent linguistic scene graph across frames, and Progressive Spatial Analysis, which progressively reasons toward task-specific questions. With only 5K auto-generated SFT samples and 20K RL samples, EgoMind achieves competitive results on VSI-Bench, SPAR-Bench, SITE-Bench, and SPBench, demonstrating its effectiveness in strengthening the spatial reasoning capabilities of MLLMs and highlighting the potential of linguistic reasoning for spatial cognition. Code and data are released at https://github.com/Hyggge/EgoMind.
Zhenghao Chen, Huiqun Wang, Di Huang
State Key Laboratory of Complex and Critical Software Environment, Beihang University · School of Computer Science and Engineering, Beihang University
Multimodal language models (MLLMs) exhibit promising spatial reasoning capabilities, yet the internal mechanisms supporting them remain poorly understood. We uncover S-Space, a linear representation subspace in which object-token representations encode continuous coordinates along horizontal, vertical, and distance axes. Although we identify these axes by differentiating spatial-answer scores with respect to object-token activations, S-Space is intrinsic rather than induced by specific data or prompts, remaining readable under non-spatial questions. We show that S-Space functions as an internal spatial workspace of MLLMs. Perception encodes object locations into it, forming spatial representations that can be directly read out to support accurate spatial judgments. Internal reasoning transforms it for changing viewpoints. Verbal responses selectively access it, evidenced by causal interventions that alter spatial judgments while largely preserving non-spatial responses. Case studies further show that S-Space assigns locations to unseen entities and the observer, and connects physical directions with abstract linguistic meanings, suggesting both the capabilities and limitations of large-scale language training. Finally, we use S-Space as a diagnostic tool for spatial reasoning failures. Direction-word readouts reveal systematic coupling between cardinal and image-relative reference frames. More importantly, a perception-computation decomposition exposes a key bottleneck: MLLMs can form capable spatial representations, yet fall short in the spatial transformations required to reason reliably across viewpoint changes.