Multimodal Large Language Models (MLLMs) show strong visual perception capabilities, but existing evaluations largely focus on single-view or multi-view settings with shared camera parameters, leaving the challenge of multi-agent collaborative scenarios underexplored. In such settings, multiple agents observe a shared environment from independent viewpoints, and reliable decisions require reasoning across these independent observations of the same environment. To study this problem, we introduce MaRiO, a benchmark for evaluating multi-agent collaborative reasoning, and evaluate 23 state-of-the-art MLLMs on it. Our analysis reveals three key findings: (1) tasks governed by static perception are generally easier than tasks that require transformational event sequences; (2) most models struggle to resolve cross-agent correspondences, with errors increasing with scene complexity; and (3) models show limited ability to leverage geometric cues present in the scene. As an initial step toward addressing these challenges, we explore geometry-guided synthetic view augmentation that generates intermediate transition frames between agent observations, providing auxiliary spatial context that improves performance in certain categories. Overall, our results highlight that multi-agent collaborative reasoning remains an open challenge and an important direction for multimodal embodied systems. The code is available on the project page: https://omarftt.github.io/MaRiO-page/.
Figures & tables
Figure 1: Multi-agent collaborative reasoning in a shared environment, where multiple agents observe the same environment from independent coordinate frames, requiring cross-agent correspondence before inferring spatial and geometric relations.
Benchmark
Setup
Reasoning Requirement
Reasoning Challenge
MV
MA
Perc.
Comp.
Cam. Geom.
Aux. Info.
VSR [ 48 ]
×
×
✓
×
-
None
Spatial relation
Winoground [ 10 ]
×
×
✓
×
-
Attribute tag
Compositionality
GeoMeter [ 11 ]
×
×
×
✓
-
None
Depth perception
OmniSpatial [ 22 ]
×
×
✓
✓
-
None
Cognitive spatial reasoning
ScanQA [ 52 ]
×
×
✓
×
-
Point Clouds
3D scene understanding
Table 1: Comparison of MaRiO with existing visual reasoning benchmarks. MV and MA respectively denote Multi-view and Multi-agent. Perc. and Comp. respectively refer to static perception and compositional reasoning over geometric transformations. Cam. Geom. refers to the provision of explicit camera geometry, and Aux. Info. refers to auxiliary information beyond RGB frames.
Figure 2: Overview of MaRiO. The benchmark consists of five spatial reasoning tasks: Occlusion, Egocentric Motion, Relative Position, Navigation, and Distance Awareness; evaluated in a multiple-choice question (MCQ) format. Each question is constructed using observations from multiple agents viewing the same environment, requiring reasoning across the images to determine the correct answer, without access to auxiliary information such as camera parameters or geometric metadata.
Model
Comp.
Static Perc.
Overall
Avg.
Ego.
Nav.
Dist.
Occ.
Rel.
Human Evaluators
64.7
68.8
86.7
77.8
68.8
73.2
73.3
Random Choice
21.7
24.9
22.1
26.5
25.3
23.9
24.1
Proprietary Models
GPT 4o [ 70 ]
26.4
41.8
43.7
51.1
34.1
38.3
39.4
GPT 5.1 [ 71 ]
32.3
43.9
47.4
53.6
55.9
45.2
46.6
Table 2: Zero-shot performance of multi-agent collaborative reasoning on MaRiO. Dist.: Distance awareness, Ego.: Egocentric motion, Nav.: Navigation, Occ.: Occlusion, and Rel.: Relative position. Model type-wise best and second-best scores are highlighted, with overall best and second-best additionally emphasized. Comp. and Static Perc. respectively refer to Compositional Reasoning over Geometric Transformations and Static Perception.
Figure 3: Relationship between spatial perception and geometric reasoning. Left : Hexbin density of static perception vs. compositional accuracy pooled across models, with linear regression fit. Right : Per-model regressions of compositional accuracy on static perception accuracy, with right 1 showing regression slope (geometric scaling strength), and right 2 reporting regression R2 values (compositional variance over accuracy explained by static perception accuracy).
Figure 4: Impact of removal of observation on reasoning performance. Removing agents’ observations simulates loss of shared visual context.
Figure 5: Impact of increasing relational complexity. Performance degrades from two to three agents, indicating weaker cross-agent alignment under higher relational complexity.
Figure 6: Orientation consistency analysis under horizontal flipping. Left: Question outcomes classified by correctness across original and flipped orientations. Here, Flip-fragile: correct only in original; reverse-fragile: correct only when flipped. Bias ratio = flip-fragile / (flip-fragile + reverse-fragile); 0.50 indicates symmetric fragility. Right Accuracy change under reflection by reasoning category; dashed lines denote mean shifts across models.
Table 3: Human error analysis. (a) Audit of incorrect human responses across 21 questions. (b) Cross-tabulation of human and Gemini-2.5-Pro correctness on the same 81 questions.
Figure 7: Impact of depth cue complexity. (a) Accuracy change ( Δ ) between high and low depth edge density in distinct depth layers.
Table 11
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Bias towards navigational primitives. Heatmap displaying MLLM accuracy for navigation QA subsets where ’Translation only’ refers to navigation QA samples whose answer choices consist only of translation primitives, ’Rotation only’ refers to navigation QA samples whose answer choices consist only of rotation primitives, and ’Compositions’ refers to navigation QA samples that have at least one answer choice with a combination of translation and rotation primitives
Figure 10: Egocentric directional bias across models. Predicted cardinal frequencies normalized by the ground-truth prior are shown under the original binding ( left ) and a 90° clockwise remapped binding ( right ). Values >1 indicate systematic overprediction relative to task frequency. In the original mapping, north , east , south , and west correspond to forward, rightward, backward, and leftward motion, respectively.
Figure 11: Relationship between spatial perception and geometric reasoning. Per-model regressions of geometric accuracy on spatial accuracy, with left showing regression slope (geometric scaling strength), and right reporting regression R2 values (geometric variance explained by spatial perception).
Figure 12: Impact of removal of observation on reasoning performance. Here, removing agents’ observations simulates loss of shared visual context.
Figure 13: Examples of shared anchor density across agent viewpoints . Each image pair corresponds to two synchronized agent perspectives observing the same environment. Objects visible in both views are marked as shared anchors using consistent color-coded bounding boxes and indices across the pair. Below each pair is their respective anchor-density bin as used in our analysis: [0,2] , [3,5] , [6,9] , and 10+ shared anchors.
Figure 14: Effect of varying density of shared environmental landmarks. Accuracy follows a non-monotonic pattern: it improves as anchor density grows from very sparse scenes, then degrades in more cluttered environments. Navigation and Egocentric Motion are most sensitive to this effect.
Figure 15: Impact of shared environmental landmarks on number of agents . We observe a performance divergence between 2-agent and 3-agent configurations as anchor density increases. While 3-agent scenarios exhibit a higher peak in moderate density ( [6,9] bin), they undergo a sharper degradation under extreme structural complexity.
Figure 16: Orientation consistency under horizontal reflection for 15 open-source models. Left: Question outcomes under original and flipped orientations. Bias ratio = flip-fragile / (flip-fragile + reverse-fragile); 0.50 indicates symmetric fragility. Right: Accuracy drop Δ by reasoning sub-category. Dashed lines indicate cross-model means.
Figure 17: Scene complexity across depth layer strata. Each row shows two examples from a stratum. Columns: original RGB image; monocular depth map; depth layer segmentation by histogram peak assignment (distinct colors indicate distinct depth planes); detected depth edges overlaid in red on the RGB image.
Figure 18: Depth edge density effect across 20 models. Each panel shows the accuracy difference ( Δ ) between high and low depth edge density samples within a complexity stratum. Only Gemini-2.5-Pro in the Simple stratum shows significant effect after Bonferroni correction ( α=0.0025 ).
Model
RGB only
+Depth
Δ
p
Gemini-2.5-Pro
61.9
48.3
−13.6
<10−5
Qwen2.5-VL-7B
53.2
39.8
−13.3
<10−5
LLaVA-OneVision-7B
49.7
47.1
−2.6
0.25
LLaVA-NeXT-Video-7B
46.2
46.8
+0.6
0.82
Appendix
Table 5: Distance Awareness accuracy (%) without and with metric depth maps. Scores use questions with valid responses in both conditions: 470 for Gemini-2.5-Pro, 467 for LLaVA-OneVision-7B, and 472 for the other models. Δ is the change in percentage points, computed before rounding; p is from a paired McNemar test.
Model
Input
Ego.
Nav.
Dist.
Occ.
Rel.
Overall
LLaVA-OneVision-7B
RGB
30.4
36.4
49.8
47.8
43.2
40.5
Gemini-2.5-Flash
RGB
31.3
44.6
51.5
50.3
46.3
44.1
SpatialBot
RGB + depth
27.1
24.6
37.1
42.0
41.3
33.2
Appendix
Table 6: Accuracy on the two-agent subset (2,590 questions). SpatialBot receives RGB views and depth maps; the other models receive RGB views only.
Figure 19: Ablation of generated synthetic views. Left: effect of the number of synthetic frames sampled from the generated video. Right: effect of interpolation order (Shuff. − BL) and image augmentation with cropped frames (Crop. − BL), both reported as accuracy deltas relative to the default baseline.
Model
Comp.
Static Perception
Overall
Avg.
Ego.
Nav.
Dist.
Occ.
Rel.
General Models
Phi-3.5-Vision-Instruct-4B
27.3
31.9
36.7
37.3
31.4
32.3
32.9
+ Syn. View
28.4
34.4
38.7
39.2
34.1
34.3
35.0
+1.1
+2.5
+2.0
+1.9
+2.7
+2.0
+2.1
VILA1.5-7B
28.1
32.5
41.3
39.8
37.5
34.9
35.8
Appendix
Table 7: Performance of synthetic view augmentations. Improvements after adding synthetic views and best overall gain are highlighted. † denotes the corrected average performance; the corresponding value in Table 2 contained a minor typographical error.
Figure 20: MaRiO construction pipeline. Image Generation (top) combines real-world and synthetic sources into multi-agent observation sets via agent configuration. QA Generation (bottom) produces verified 4-choice questions through task specification, expert annotation, joint verification, and MCQ formatting.
Figure 21: Multi-agent observations covering all categories of reasoning tasks of MaRiO.
Figure 22: Qualitative analysis of MaRiO. Here, ground truth option is highlighted in green box, followed by predictions of several models.
Type
Category
2-Agents
3-Agents
Total
Comp.
Egocentric Motion
576
189
765
Navigation
659
214
873
Static Perc.
Distance Awareness
472
149
621
Occlusion
366
41
407
Relative Position
517
72
589
All
2,590
665
3,255
Appendix
Table 8: Category-wise distribution of MaRiO. Comp. and Static Perc. respectively denote Compositions of Geometric Transformations and Static Spatial Perception.
Criterion
A1
A2
A3
Pooled
(i) Category-style fit
99.2
92.8
89.6
93.9
(ii) Answer-option consistency
100.0
92.8
99.2
97.3
(iii) Entity concreteness
98.4
96.8
96.0
97.1
(iv) Overall style match
97.6
89.6
86.4
91.2
Overall
98.8
93.0
92.8
94.9
Appendix
Table 9: Blind audit of QA-style consistency across the three annotators (A1–A3), reported as the percentage of sampled items passing each criterion (125 items per annotator).
Task
4 Choices
6 Choices
8 Choices
Distance Awareness
35.90% (14/39)
38.46% (15/39)
33.33% (13/39)
Egocentric Motion
40.57% (43/106)
23.58% (25/106)
20.75% (22/106)
Navigation
39.11% (79/202)
31.68% (64/202)
28.22% (57/202)
Occlusion
44.44% (48/108)
32.41% (35/108)
25.00% (27/108)
Relative Position
44.14% (64/145)
30.34% (44/145)
19.31% (28/145)
Overall
41.33% (248/600)
30.50% (183/600)
24.50% (147/600)
Appendix
Table 10: Gemini-2.5-Flash accuracy across answer-choice settings.
Task
4 Choices
6 Choices
8 Choices
Distance Awareness
20.51% (8/39)
28.21% (11/39)
17.95% (7/39)
Egocentric Motion
30.19% (32/106)
16.98% (18/106)
20.75% (22/106)
Navigation
36.14% (73/202)
29.70% (60/202)
19.80% (40/202)
Occlusion
51.85% (56/108)
34.26% (37/108)
26.85% (29/108)
Relative Position
50.34% (73/145)
24.83% (36/145)
20.69% (30/145)
Overall
40.33% (242/600)
27.00% (162/600)
21.33% (128/600)
Appendix
Table 11: LLaVA-OneVision-7B accuracy across answer-choice settings.
Task
4 Choices
6 Choices
8 Choices
Distance Awareness
35.90% (14/39)
41.03% (16/39)
41.03% (16/39)
Egocentric Motion
33.96% (36/106)
26.42% (28/106)
21.70% (23/106)
Navigation
33.17% (67/202)
26.24% (53/202)
25.74% (52/202)
Occlusion
50.93% (55/108)
31.48% (34/108)
39.81% (43/108)
Relative Position
45.52% (66/145)
19.31% (28/145)
15.17% (22/145)
Overall
39.67% (238/600)
26.50% (159/600)
26.00% (156/600)
Appendix
Table 12: Qwen2.5-VL-7B accuracy across answer-choice settings.
Compositional
Static Perception
Ego.
Nav.
Dist.
Occ.
Rel.
Ego.
1.000
0.407
0.399
0.530
0.634
Nav.
0.407
1.000
0.426
0.612
0.542
Dist.
0.399
0.426
1.000
0.933
0.855
Occ.
0.530
0.612
0.933
1.000
0.856
Rel.
0.634
0.542
0.855
0.856
1.000
Appendix
Table 13: Spearman rank correlation of model accuracy across task categories, computed over the 19 open-source models on the full benchmark.
Model
Synthetic
Real-World
LLaVA-NeXT-Video-7B-HF
31.97
32.61
mPLUG-Owl3-7B-240728
34.77
37.90
Qwen2-VL-7B-Instruct
39.86
42.97
Qwen2.5-VL-3B-Instruct
40.76
40.81
Qwen2.5-VL-7B-Instruct
37.26
39.71
Appendix
Table 14: Accuracy (%) on synthetic and real-world image sets.