Ego to World: Collaborative Spatial Reasoning in Embodied Systems via Reinforcement Learning
Authors: Heng Zhou, Li Kang, Yiran Qin, Xiufeng Song, Ao Yu, Zilu Zhang, Haoming Song, Kaixin Xu, Yuchen Fan, Dongzhan Zhou, Xiaohong Liu, Ruimao Zhang, Philip Torr, Lei Bai, Zhenfei Yin
Organizations: University of Science and Technology of China · Shanghai AI Laboratory · Shanghai Jiao Tong University · Beijing University of Posts and Telecommunications · Fudan University · Sun Yat-sen University · University of Oxford
Understanding the world from distributed, partial viewpoints is a fundamental challenge for embodied multi-agent systems. Each agent perceives the environment through an ego-centric view that is often limited by occlusion and ambiguity. To study this problem, we introduce the Ego-to-World (E2W) benchmark, which evaluates a vision-language model's ability to fuse heterogeneous viewpoints across three tasks: (i) global counting, (ii) relational location reasoning, and (iii) action-oriented grasping that requires predicting view-specific image coordinates. To address this setting, we propose CoRL, a two-stage framework that combines Chain-of-Thought supervised fine-tuning with reinforcement learning using Group-Relative Policy Optimization. Its core component, the Cross-View Spatial Reward (CVSR), provides dense task-aligned feedback by linking reasoning steps to visual evidence, ensuring coherent cross-view entity resolution, and guiding the model toward correct final predictions. Experiments on E2W show that CoRL consistently surpasses strong proprietary and open-source baselines on both reasoning and perception-grounding metrics, while ablations further confirm the necessity of each CVSR component. Beyond that, CoRL generalizes to external spatial reasoning benchmarks and enables effective real-world multi-robot manipulation with calibrated multi-camera rigs, demonstrating cross-view localization and successful grasp-and-place execution. Together, E2W and CoRL provide a principled foundation for learning world-centric scene understanding from distributed, ego-centric observations, advancing collaborative embodied AI.
Figures & tables
Figure 1: An illustration of collaborative spatial reasoning in embodied systems. Reasoning from a single viewpoint fails due to occlusions or a limited field of view. In contrast, cross-view compositional reasoning integrates multiple perspectives to correctly localize and grasp the target object—the blue block farthest from the strawberry.
Figure 2: Overview of the Ego-to-World (E2W) Benchmark. Top: Multiple agents (Robot A, B, C) each provide partial ego-centric views of a shared scene. The vision language model trained with our CoRL framework integrate these complementary perspectives to solve three tasks: Counting (E2W-1), Location Reasoning (E2W-2), and Grasping (E2W-3). Bottom: The benchmark combines diverse real and simulated data and organizes them into varying complexity levels.
Figure 3: CoRL framework. The model is first initialized via supervised fine-tuning (SFT) on Chain-of-Thought annotations, then refined with reinforcement learning (RL). During RL, the policy is optimized with an format reward and the Cross-View Spatial Reward (CVSR), which supplies dense feedback on cross-view fusion and spatial consistency, guiding robust collaborative reasoning.
Model
Reasoning (Acc)
Perception (Score)
E2W-1
E2W-2(S)
E2W-2(R)
Avg.
E2W-3(S)
E2W-3(R)
Avg.
Closed Source Models
GPT-5
42.5
48.5
72.5
54.50
50.43
12.02
31.23
Doubao-Seed-1.6
35.0
40.0
46.0
40.33
16.60
5.26
10.93
Gemini-2.5-Pro
32.5
42.5
50.0
41.67
35.98
10.15
23.07
Open Source Models
Table 1: Performance on E2W-Bench. Tasks are grouped into Reasoning (accuracy, %) and Perception (score, %). (S) = simulation, (R) = real-world. Best in bold, second best underlined.
Figure 4: Illustrative demonstrations of our model in real-world robotic manipulation tasks involving pick-and-place with cross-view spatial reasoning.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Simulated Data
Real-World Data
Total Data
Task 1 (Counting)
30k
0
30k
Task 2 (Location Reasoning)
35k
25k
60k
Task 3 (Grasping)
35k
35k
70k
Appendix
Table 5: E2W Benchmark Dataset Distribution for Each Task
Figure 5: Example of an E2W-1 sample (Case 1). Two ego-centric views of a kitchen scene are provided, and the model must count the total number of pizzas by resolving cross-view object correspondences.
Figure 6: Example of an E2W-2 sample (Case 2). The model receives two views of a table scene and must identify the third item from the left by integrating spatial information across views.
Figure 7: Example of an E2W-3 sample. The model receives two ego-centric views from different robot perspectives and must localize the target object—in this case, the pizza—and output the precise grasp location. Solving the task requires cross-view object correspondence, reasoning about spatial ordering, identifying the visible target in each view, and determining the final grasp coordinate in the main perspective.
Spatial reasoning studies how systems represent geometry, relations, viewpoints, and motion to understand scenes, plan actions, and solve spatial tasks.
Vision-language models (VLMs) have achieved impressive results on single-view vision tasks, but lack the multi-view spatial reasoning capabilities essential for embodied AI systems to understand 3D environments and manipulate objects across different viewpoints. In this work, we introduce Cross-View Relations (XVR), a large-scale dataset designed to teach VLMs spatial reasoning across multiple views. XVR comprises 100K vision-question-answer samples derived from 18K diverse 3D scenes and 70K robotic manipulation trajectories, spanning three fundamental spatial reasoning tasks: Correspondence (matching objects across views), Verification (validating spatial relationships), and Localization (identifying object positions). VLMs fine-tuned on XVR achieve substantial improvements on established multi-view and robotic spatial reasoning benchmarks (MindCube and RoboSpatial). When integrated as backbones in Vision-Language-Action models, XVR-trained representations improve success rates on RoboCasa. Our results demonstrate that explicit training on cross-view spatial relations significantly enhances multi-view reasoning and transfers effectively to real-world robotic manipulation.
Multimodal Large Language Models (MLLMs) have made substantial progress in egocentric video understanding, but their ability to reason cooperatively from multiple embodied viewpoints remains largely unexplored. We study this problem through multi-robot cooperative dynamic spatial reasoning, where a model must answer spatial, temporal, visibility, and coordination questions by integrating synchronized egocentric videos from a team of moving robots. To support this setting, we introduce CoopSR, the first benchmark for this task, together with EgoTeam, a multi-robot egocentric QA dataset. EgoTeam contains 114,227 QA pairs spanning 19 question types, four difficulty tiers, and three team sizes in Habitat and iGibson, along with a real-world test set of around 2,326 QAs collected using two quadruped robots. We further propose SG-CoR (Spectral and Geometry-Aware Cooperative Reasoner), an MLLM framework for fine-grained cooperative spatial reasoning. SG-CoR combines dynamics-aware multi-robot frame sampling, spectral- and geometric-guided view fusion, and geometry- and motion-informed prompt distillation, enabling the model to benefit from privileged robot-pose supervision during training while requiring only egocentric videos at test time. Across 22 MLLM baselines, SG-CoR consistently improves cooperative reasoning, outperforming the strongest fine-tuned baseline by +3.87% on Habitat and +7.12% on iGibson. It also shows stronger generalization to unseen team sizes and real-world robot tests. Code can be found at https://github.com/KPeng9510/seeing-together.git.
Kunyu Peng, Zhikun Zhou, Kailun Yang +9
Karlsruhe Institute of Technology · INSAIT, Sofia University “St. Kliment Ohridski” · Hunan University +4
Egocentric world models predict first-person observations conditioned on an agent's actions, but most focus on a single agent. Real embodied settings often involve multiple agents that act and interact within a shared environment. Existing multi-agent world models rely on coarse actions like locomotion, camera control, or discrete commands, leaving fine-grained embodied interactions underexplored. We formulate multi-agent egocentric world modeling as synchronized ego-stream generation for multiple agents interacting through fine-grained actions in a shared world. This requires cross-view action consistency, shared-environment consistency, and consistent propagation of interaction-induced state updates. We propose Multi-agent Egocentric World Model (ME-World), which jointly denoises multiple ego streams in a shared token sequence, conditions each stream on all agents' target-view poses, and grounds generation with shared environment memory. We train and evaluate on real and synthetic multi-agent data and introduce shared-world consistency metrics for environment, update, and identity consistency. Experiments show ME-World improves shared-world consistency, action control, identity preservation, and video quality over existing methods.