cs.CVMar 5, 2026
SaveMultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents
Organizations: INSAIT, Sofia University “St. Kliment Ohridski” · Lanzhou University · King Abdullah University of Science and Technology · Heriot-Watt University · Monash University · University College Dublin
Abstract
Multimodal large language models (MLLMs) achieve strong performance on benchmarks that evaluate text, image, or video understanding separately. However, these settings do not assess a critical real-world requirement, which involves retrieving relevant evidence from large, heterogeneous multimodal corpora prior to reasoning. Most existing benchmarks restrict retrieval to small, single-modality candidate sets, substantially simplifying the search space and overstating end-to-end reliability. To address this gap, we introduce MultiHaystack, the first benchmark designed to evaluate both retrieval and reasoning under large-scale, cross-modal conditions. MultiHaystack comprises over 46,000 multimodal retrieval candidates across documents, images, and videos, along with 747 open yet verifiable questions. Each question is grounded in a unique validated evidence item within the retrieval pool, requiring evidence localization across modalities and fine-grained reasoning. In our study, we find that models perform competitively when provided with the corresponding evidence, but their performance drops sharply when required to retrieve that evidence from the full corpus. Additionally, even the strongest retriever, E5-V, achieves only 40.8% Recall@1, while state-of-the-art MLLMs such as GPT-5 experience a significant drop in reasoning accuracy from 80.86% when provided with the corresponding evidence to 51.4% under top-5 retrieval. These results indicate that multimodal retrieval over heterogeneous pools remains a primary bottleneck for MLLMs, positioning MultiHaystack as a valuable testbed that highlights underlying limitations obscured by small-scale evaluations and promotes retrieval-centric advances in multimodal systems.
Figures & tables
Figure 1: Comparison with existing visual question answering benchmarks. Existing benchmarks often suffer from three key limitations: ( i ) ambiguous evidence that leads to multiple possible answers, ( ii ) retrieval restricted to a single modality, and ( iii ) small candidate pools (often limited to a single image, document, or video). In contrast, MultiHaystack provides questions grounded in uniquely verifiable evidence over a large-scale multimodal corpus of 46K+ items spanning documents, images, and videos, requiring both modality selection and fine-grained reasoning.
Figure 2: Performance on MultiHaystack. “Gold in Top-1/5” directly provides answer-containing files; “Single-Modality” and “Cross-Modality” require retrieval within one or across multiple modalities.
| Benchmark | Modality | Retrieval Candidates per QA | Data Types | Unique Evidence | Task Types |
| WebQA [ 4 ] | 1–5 | Web images | ✓ | 1 | |
| RetVQA [ 35 ] | 20–30 | Natural images | ✓ | 2 | |
| MM-NIAH [ 46 ] | 10–70+ | Mixed text-image | ✗ | 3 | |
| MMNeedle [ 44 ] | 10–160 | Image patch | ✓ | 1 | |
| DocHaystack [ 5 ] | 100–1000 | Document images | ✓ | 2 | |
| MultiHaystack | Multimedia items | ✓ | 6 |
Table 1: Comparison of Benchmarks. MultiHaystack is a large-scale multimodal benchmark with 46K+ items, supporting multimodal retrieval, unique evidence, and six task types, while prior benchmarks remain limited.
Figure 3: Examples of six tasks in MultiHaystack: Visual Parsing & Positioning (spatial layouts), Contextual Understanding (embedded text), Video Temporal Reasoning (motion/order), Statistical Reasoning (charts/tables), Metadata Identification (affiliations/timestamps), and Factual Knowledge Retrieval (corpus-grounded facts).
Figure 4: Benchmark construction pipeline. MultiHaystack is built in four stages: collecting diverse multimodal sources, generating specific QA pairs, filtering for unique and grounded answers, and enriching with data. This design ensures coverage across six tasks ( Fig. 3 ) and overcomes the unimodal, small-scale, or ambiguous limitations of prior benchmarks.
| Model | Video | Image | Document | Overall | ||||||||
| R@1 | R@3 | R@5 | R@1 | R@3 | R@5 | R@1 | R@3 | R@5 | R@1 | R@3 | R@5 | |
| CLIP | 26.67 (56.19) | 40.00 (78.10) | 51.43 (80.00) | 21.71 (30.25) | 31.64 (40.88) | 34.87 (44.34) | 34.93 (38.76) | 46.89 (51.67) | 48.80 (53.59) | 26.10 (36.28) | 37.08 (49.13) | 41.10 (51.94) |
| SigLIP2 | 40.00 (63.81) | 60.00 (83.81) | 74.29 (91.43) | 32.10 (44.11) | 40.88 (53.12) | 45.27 (58.66) | 59.81 (61.72) | 68.42 (70.33) | 72.73 (75.12) | 40.96 (51.81) | 51.27 (62.25) | 57.03 (67.87) |
| OpenCLIP | 38.10 (60.00) | 56.19 (74.29) | 62.86 (78.10) | 19.40 (25.40) | 27.94 (35.80) | 32.33 (42.26) | 28.71 (32.06) | 36.84 (42.58) | 43.06 (47.85) | 24.63 (32.13) | 34.40 (43.11) | 39.63 (48.86) |
| Jina-Clip-V1 | 21.90 (42.86) | 38.10 (59.05) | 47.62 (67.62) | 7.39 (13.39) | 10.16 (19.17) | 12.93 (22.40) | 16.75 (17.70) | 21.05 (22.01) | 22.49 (22.97) | 12.05 (18.74) | 17.14 (25.57) | 20.48 (28.92) |
| Jina-Clip-V2 | 20.00 (36.19) | 30.48 (56.19) | 35.24 (76.19) | 11.78 (27.25) | 21.02 (42.73) | 25.17 (48.04) | 40.67 (41.63) | 51.67 (52.63) | 55.98 (56.46) | 21.02 (32.53) | 30.92 (47.39) | 35.21 (54.35) |
Table 2: Retrieval performance in cross-modality vs. single-modality settings. Cross-modality results are shown in black, while single-modality results are shown in gray for comparison. Best values per column are highlighted in bold.
| Model | VPP | CU | VTR | FKR | SR | MI | ||||||||||||
| R@1 | R@3 | R@5 | R@1 | R@3 | R@5 | R@1 | R@3 | R@5 | R@1 | R@3 | R@5 | R@1 | R@3 | R@5 | R@1 | R@3 | R@5 | |
| CLIP | 33.33 | 39.39 | 42.42 | 16.67 | 30.00 | 43.33 | 29.55 | 40.91 | 50.00 | 20.59 | 23.53 | 29.41 | 25.23 | 35.51 | 38.63 | 27.37 | 40.35 | 43.51 |
| SigLIP2 | 42.42 | 66.67 | 72.73 | 53.33 | 66.67 | 80.00 | 29.55 | 52.27 | 65.91 | 26.47 | 32.35 | 47.06 | 38.01 | 45.48 | 50.47 | 46.32 | 56.49 | 60.00 |
| OpenCLIP | 36.36 | 51.52 | 54.55 | 30.00 | 46.67 | 50.00 | 38.64 | 61.36 | 70.45 | 17.65 | 20.59 | 20.59 | 22.74 | 31.78 | 37.69 | 23.51 | 31.58 | 36.49 |
| Jina-Clip-V1 | 21.21 | 27.27 | 39.39 | 13.33 | 13.33 | 20.00 | 29.55 | 59.09 | 70.45 | 2.94 | 8.82 | 8.82 | 9.03 | 13.08 | 15.26 | 12.63 | 15.44 | 17.89 |
| Jina-Clip-V2 | 33.33 | 42.42 | 48.48 | 10.00 | 16.67 | 23.33 | 11.36 | 20.45 | 25.00 | 11.76 | 20.59 | 23.53 | 17.13 | 28.35 | 31.78 | 27.72 | 36.84 | 41.75 |
Table 3: Retrieval results across six tasks. Recall@1/3/5 of different vision-language retrieval models on MultiHaystack across six distinct tasks.
| Model | Video | Image | Document | Overall |
| Ola | 14.29 (22.86) | 20.09 (31.41) | 36.36 (44.98) | 23.83 (34.00) |
| InternVL-3 | 17.14 (20.95) | 29.33 (38.80) | 49.28 (51.67) | 33.29 (39.89) |
| Qwen2-VL | 16.19 (18.10) | 16.86 (24.94) | 19.62 (22.49) | 17.54 (23.29) |
| Gemini-2.5-Flash | 52.38 (61.90) | 35.10 (44.57) | 56.94 (58.37) | 43.64 (50.87) |
| GPT-5 | 60.00 (67.62) | 43.19 (52.66) | 64.11 (70.81) | 51.41 (59.84) |
Table 4: Multimodal reasoning performance. Each model answers questions using top- items retrieved by E5-V from cross-modality inputs; gray numbers show single-modality Recall@5 for reference.
Figure 5: Top- ablation analysis for MLLMs integrated with E5-V.
| Model | VPP | CU | VTR | FKR | SR | MI |
| Ola | 27.27 | 30.00 | 18.18 | 20.59 | 26.17 | 21.40 |
| InternVL-3 | 42.42 | 23.33 | 11.36 | 23.53 | 29.91 | 41.40 |
| Qwen2-VL | 18.18 | 23.33 | 6.82 | 14.71 | 18.38 | 17.89 |
| Gemini-2.5-Flash | 54.55 | 46.67 | 56.82 | 32.35 | 35.51 | 50.53 |
| GPT-5 | 57.58 | 56.67 | 52.27 | 50.00 | 43.61 | 58.95 |
Table 5: Comparison of MLLMs’ reasoning performance integrated with E5-V across six tasks.
Figure 6: Comparison in three distinct modalities. (a) represents the video modality, (b) represents the image modality, and (c) represents the document modality.
| Model | Cohen’s | 95% CI | Accuracy | Pearson |
| Ola | 0.918 | [0.710, 1.000] | 0.967 | 0.921 |
| InternVL-3 | 0.865 | [0.667, 1.000] | 0.933 | 0.873 |
| Qwen2-VL | 1.000 | [1.000, 1.000] | 1.000 | 1.000 |
| Gemini-2.5-Flash | 0.932 | [0.772, 1.000] | 0.967 | 0.934 |
| GPT-5 | 1.000 | [1.000, 1.000] | 1.000 | 1.000 |
Table 6: Reliability Matrix for LLM-as-Judge
Figure 7: Pool-size controlled comparison. Recall of MM-Embed under single-modality retrieval and mixed-modality retrieval with an identical total pool size. Performance remains substantially lower in the mixed-modality condition, indicating that the cross-modal gap arises from modality heterogeneity rather than pool size.
Figure 8: Effect of data enrichment under varying candidate pool sizes, showing that recall consistently drops as the pool expands.
Figure 9: Error cases illustrating retrieval errors, including modality bias (retrieving images instead of video evidence) and semantic drift (violating temporal constraints), and reasoning errors, including visual numeracy (misreading numbers in charts) and layout-aware multi-step reasoning (failing to integrate structured cues across layouts).
| Methods | Video | Image | Document | Overall | ||||||||
| R@1 | R@3 | R@5 | R@1 | R@3 | R@5 | R@1 | R@3 | R@5 | R@1 | R@3 | R@5 | |
| Single-Modality | ||||||||||||
| E5-V | 62.86 | 81.90 | 83.81 | 43.19 | 68.36 | 73.44 | 60.77 | 71.29 | 76.08 | 50.87 | 71.08 | 75.64 |
| Cross-Modality | ||||||||||||
| E5-V | 34.29 | 51.43 | 60.95 | 33.49 | 55.20 | 62.82 | 59.33 | 70.33 | 75.12 | 40.83 | 58.90 | 66.00 |
| E5-V + Refined Query | 41.90 | 59.05 | 64.76 | 36.26 | 56.58 | 66.74 | 60.29 | 70.81 | 75.12 | 43.78 | 60.91 | 68.81 |
Table 7: Preliminary studies with advanced retrieval pipelines.
Appendix figures & tables36 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1 : Video–Document Distribution Overview. Distributions of video duration (left) and document page count (right), with red dashed lines indicating means and green dashed lines indicating medians.
Figure B.1 : Video Example 1.
Figure B.2 : Video Example 2.
Figure B.3 : Video Example 3.
Figure B.4 : Image Example 1.
Figure B.5 : Image Example 2.
Figure B.6 : Image Example 3.
Figure B.7 : Document Example 1.
Figure B.8 : Document Example 2.
Figure B.9 : Document Example 3.
Figure B.10 : Data Enrichment Example 1.
Figure B.11 : Data Enrichment Example 2.
Figure B.12 : Data Enrichment Example 3.
| Model | Overall |
| Ola | 0.54 |
| InternVL-3 | 0.80 |
| Qwen2-VL | 0.67 |
| Gemini-2.5-Flash | 1.07 |
| GPT-5 | 4.28 |
Table F.1 : VQA performance in zero context.
Figure F.1 : Distribution of cosine similarity scores.
| Model | Video | Image | Document | Overall |
| InternVL-3.5 | 20.95 (26.67) | 30.95 (39.72) | 51.20 (54.55) | 35.21 (42.03) |
| Qwen2.5-VL | 19.05 (21.90) | 22.17 (28.41) | 31.10 (34.45) | 24.23 (29.18) |
Table F.2 : VQA performance. Results are conditioned on top-5 cross-modal retrieval context. Baseline single-modality Recall@5 scores are indicated in gray .
Figure G.1 : Distribution of error types. Panels: (a) retrieval error distribution and (b) reasoning error distribution. Retrieval errors are quantified by Recall@5 with the strongest retriever (E5-V), while reasoning errors are evaluated using VQA with the strongest reasoning model (GPT-5). Retrieval errors dominate across tasks, though reasoning errors remain substantial.
Figure G.2 : Contextual Understanding representative error cases.
Figure G.3 : Factual Knowledge Retrieval representative error cases.
Figure G.4 : Metadata Identification representative error cases.
Figure G.5 : Statistical Reasoning representative error cases.
Figure G.6 : Video Temporal Reasoning representative error cases.
Figure G.7 : Visual Parsing and Positioning representative error cases.
Figure H.1 : Contextual Understanding Retrieval.
Figure H.2 : Contextual Understanding VQA.
Figure H.3 : Factual Knowledge Retrieval.
Figure H.4 : Factual Knowledge VQA.
Figure H.5 : Metadata Identification Retrieval.
Figure H.6 : Metadata Identification VQA.
Figure H.7 : Statistical Reasoning Retrieval.
Figure H.8 : Statistical Reasoning VQA.
Figure H.9 : Video Temporal Reasoning Retrieval.
Figure H.10 : Video Temporal Reasoning VQA.
Figure H.11 : Visual Parsing and Positioning Retrieval.
Figure H.12 : Visual Parsing and Positioning VQA.
| Data Sources | URL | License |
| VideoVista | Link | Apache-2.0 |
| MMBench-Video | Link | CC BY 4.0 |
| FineVideo | Link | CC BY 4.0 |
| MVBench | Link | MIT |
| DocHaystack | Link | MIT |
| MMIU | Link | CC BY 4.0 |
Table H.1 : License information for the scientific artifacts.