cs.CVMar 28, 2026
SaveCommunicating about Space: Language-Mediated Spatial Integration Across Partial Views
Organizations: Mila – Quebec AI Institute · Université de Montréal · IIIT Hyderabad · Canada CIFAR AI Chair
Abstract
Humans build shared spatial understanding by communicating partial, viewpoint-dependent observations. We ask whether Multimodal Large Language Models (MLLMs) can do the same, aligning distinct egocentric views through dialogue to form a coherent, allocentric mental model of a shared environment. To study this systematically, we introduce COSMIC, a benchmark for Collaborative Spatial Communication. In this setting, two static MLLM agents observe a 3D indoor environment from different viewpoints and exchange natural-language messages to solve spatial queries. COSMIC contains 899 diverse scenes and 1250 question-answer pairs spanning five tasks. We find a capability hierarchy, MLLMs are most reliable at identifying shared anchor objects across views, perform worse on relational reasoning, and largely fail at building globally consistent maps, performing near chance, even for frontier models. Moreover, we find thinking capability yields gains in anchor grounding, but is insufficient for higher-level spatial communication. To contextualize model behavior, we collect 250 human-human dialogues. Humans achieve 95% aggregate accuracy, while the best model, Gemini-3-Pro-Thinking, reaches 72%, leaving substantial room for improvement. Moreover, human conversations grow more precise as partners align on a shared spatial understanding, whereas MLLMs keep exploring without converging, suggesting limited capacity to form and sustain a robust shared mental model throughout the dialogue. Our code and data is available at https://github.com/ankursikarwar/Cosmic.
Figures & tables
Figure 1: Left: MLLM agents attempt to communicate and build a spatial mental model for answering questions in COSMIC . Right: Answerer and Helper agents integrate distinct egocentric views via communication to answer the question. Humans demonstrate efficient and precise strategies while MLLM agents are more verbose, inefficient, and fail to build and maintain a robust shared mental model.
Figure 2: Overview of the COSMIC benchmark. Each task pair shows the Answerer’s view (left) and the Helper’s view (right), along with the question and options posed to the Answerer.
Figure 3: Benchmark curation pipeline. Our pipeline involves scene generation, sampling complementary agent viewpoints, generating questions using templates and unique object description followed by paraphrasing the questions.
Figure 4: COSMIC benchmark composition. Left: Scenes from our benchmark. Center Top: Distribution of room types. Center Bottom: Distribution of scene clutter (number of object instances per scene). Right Top: Object-category frequencies across the benchmark. Right Bottom: Word cloud representing the most frequent spatial and object-related terms in the dataset.
Figure 5: Top: Evaluation on COSMIC . Error bars denote 90% confidence intervals computed via bootstrap resampling. Dashed lines indicate chance levels (25% for 4-choice MCQ, 50% for binary map tasks and 30% for overall). Bottom: Evaluation on COSMIC-Human .
Figure 6: Left: Performance Hierarchy in MLLMs across Object, Relation and Map-level. Right: Two-Agent Communication vs Single Agent.
Figure 7: Failure mode analysis on COSMIC . Qualitative examples illustrate the three failure categories, Perceptual Failure (Object Recognition Failure), Cross-View Grounding Failure (Instance Merging), and Geometric and Relational Failure (Layout Understanding Failure). The bar chart shows the distribution of failure modes across tasks, and the donut chart shows the overall breakdown.
Figure 8: Communication Efficiency & Information Dynamics on Cosmic-Human . Top: Communication Efficiency (Verbosity vs. Accuracy) . Bottom: Information Dynamics (Unique Objects Mentioned vs. Conversation Turn Number) . Shaded regions denotes variance across conversations.
Figure 9: Dialogue Repair behavior in Gemini-3-Pro-Thinking.
| Task | Template |
| Anchor Recognition | Which of the following objects is visible in both your and your partner’s views of the room? |
| Global Counting | What is the total number of object category in the room? |
| Relative Direction | From your perspective, in which direction is object description ? |
| Relative Distance | Which of the following objects is closest to / farthest from the object description ? |
| Cognitive Mapping | Is this top-down map of the room correct? |
Table 1: Templates used for question generation. Below, object category is a placeholder for an object category, for example sofa , while object description is a placeholder for the unique description of a given object instance, for example green sofa near a white door .
| Before Paraphrasing | After Paraphrasing |
| Question: Which of the following objects is visible in both your and your partner’s views of the room? Options: (A) floor purple Lamp (B) black shutter Window (C) green Plant Container next to a black Desk (D) Lamp near a black shutter Window | Question: Which of the following objects is present in both your and your partner’s views of the room? Options: (A) floor purple lamp (B) window with black shutters (C) green plant container located next to a black desk (D) lamp near a window with black shutters |
| Question: What is the total number of Monitor in the room? Options: (A) 1 (B) 2 (C) 3 (D) 4 | Question: What is the total number of computer monitors in the room? Options: (A) 1 (B) 2 (C) 3 (D) 4 |
| Question: Which of the following objects is closest to the floor purple Lamp? Options: (A) beige curtain Window (B) Monitor near a green Plant Container (C) brown curtain Window (D) green Plant Container next to a black Desk | Question: Which of the following objects is nearest to the floor lamp with purple color? Options: (A) window with beige curtains (B) computer monitor located near a green plant container (C) window with brown curtains (D) green plant container located next to a black desk |
| Question: From your perspective, in which direction is Window near a Monitor? Options: (A) behind (B) front (C) left (D) right | Question: From your perspective, in which direction is the window that is near a computer monitor? Options: (A) behind (B) front (C) left (D) right |
| Question: Is this top-down map of the room correct? | Question: Is this top-down layout of the room accurate? |
Table 2: Examples of template-generated questions and their corresponding paraphrased versions.
Figure 10: Failure Mode Analysis. Qualitative samples illustrating different failure categories, Perceptual Failure (Attribute Mislabelling), Cross-View Grounding Failure (Referential Ambiguity and Instance Duplication), and Geometric & Relational Failure (Perspective-Taking Failure). Green ticks denote the ground truth answer.
Figure 11: Conversation between agents (Gemini-3-Pro-Thinking). Although the agents demonstrate successful behaviors on some atomic steps, errors in the dialogue negate these successes, leading them to select the incorrect answer (option A).
Figure 12: Samples from Cosmic Benchmark (Anchor Recognition).
Figure 13: Samples from Cosmic Benchmark (Global Counting).
Figure 14: Samples from Cosmic Benchmark (Relative Distance).
Figure 15: Samples from Cosmic Benchmark (Relative Direction).
Figure 16: Samples from Cosmic Benchmark (Cognitive Mapping).
Figure 17: Representative dialogues from Gemini-3-Pro-Thinking on Cosmic . The tick mark denotes the ground truth answer and the red-highlighted option indicates the answerer’s prediction. From top to bottom, Anchor Recognition (top-left), Global Counting (top-right), Relative Distance (middle-left), Relative Direction (middle-right), and Cognitive Mapping (bottom). Each panel displays the Answerer’s and Helper’s egocentric views, the multiple-choice options, and the multi-turn dialogue exchanged between the two agents.
Figure 18: Representative dialogues from Humans on Cosmic . The tick mark denotes the ground truth answer and the red-highlighted option indicates the answerer’s prediction. From top to bottom, Anchor Recognition (top-left), Global Counting (top-right), Relative Distance (middle-left), Relative Direction (middle-right), and Cognitive Mapping (bottom). Each panel displays the Answerer’s and Helper’s egocentric views, the multiple-choice options, and the multi-turn dialogue exchanged between the two agents.
Figure 19: Instruction page for the human dialogue data collection interface (Answerer)
Figure 21: Human Dialogue Data Collection Interface (Answerer)