Motivated by the underspecified, multi-hop nature of search queries and the multimodal, heterogeneous, and often conflicting nature of real-world web results, we introduce MERRIN (Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments), a human-annotated benchmark for evaluating search-augmented agents. MERRIN measures AI agents' ability to identify relevant modalities, retrieve multimodal evidence, and perform multi-hop reasoning over noisy web sources. It differs from prior work in three important aspects: (1) using natural language queries without explicit modality cues, (2) incorporating underexplored modalities such as video and audio, and (3) requiring the retrieval of complex, often noisy or conflicting multimodal evidence during web search. We evaluate diverse search agents powered by ten models, including strong closed-source models (e.g., GPT-5.4-mini, Gemini 3/3.1 Flash/Pro) and open-weight models (Qwen3-4B/30B/235B), across three search settings (no search, native search, and agentic search). Our results show that MERRIN is highly challenging: the average accuracy across all agents is 22.3%, with the best-performing agent reaching only 40.1%. We further observe that while stronger agents like Gemini Deep Research achieve higher performance, gains are modest due to over-exploration; they take more steps and use more tools, but are often distracted by conflicting or partially relevant web content, leading to incorrect answers. Compared to humans, these agents consume more resources yet achieve lower accuracy, largely due to inefficient source selection and an overreliance on text modalities. These findings highlight the need for search agents capable of robust search and reasoning across diverse modalities in noisy web environments, making MERRIN a valuable testbed for evaluating such capabilities.
Figures & tables
Figure 1: Overview of MERRIN . Given a query, the agent must identify the appropriate modality, retrieve relevant evidence, and perform multi-hop reasoning over noisy, conflicting, and incomplete web sources. The green path shows the ideal case: the agent selects the correct modality and source, arriving at the correct answer. The remaining paths illustrate three failure modes: Reasoning Error ( blue )—correct source retrieved but incorrect grounding to the evidence; Modality Error ( red )—agent relies on text when asked about visual information; Retrieval Error ( purple )—correct modality but misleading source selected.
Figure 2: MERRIN composition. (a) Gold source resources by modality. (b) Questions by the role of visual content. (c) Questions by reasoning type.
No Search
Native Search
Agentic Multimodal Search
Model
Acc
Acc
# Search Qs
# Pages
Acc
# Search Qs
# Pages
Qwen3-4B
10.3±0.4
-
-
-
10.5±1.6
1.7±0.0
0.2±0.1
Qwen3-30B
8.0±0.6
-
-
-
16.1±0.6
2.0±0.1
0.5±0.0
Qwen3-235B
12.1±0.9
-
-
-
23.3±1.3
3.0±0.2
1.0±0.1
GPT-5.4-nano
9.9±2.7
12.6±1.3
37.7±2.4
5.9±0.4
31.9±3.0
11.6±0.4
7.5±0.4
GPT-5.4-mini
14.0±0.4
15.6±0.4
38.6±3.7
5.3±0.2
31.1±3.1
9.2±0.3
3.4±0.2
Table 2: Performance of search agents powered by different models on MERRIN . Acc denotes average accuracy over three runs with standard deviation, # Search Qs is the average number of search queries issued per question, and # Pages is the average number of webpages explicitly visited and read per question. No Search has no search module; thus, both # Search Qs and # Pages are 0. Gemini 3.1 Lite refers to Gemini 3.1 Flash Lite, and Gemini Research refers to the Gemini Deep Research Agent. For Qwen models, Native Search is not applicable since they do not have an internal search agent. Gemini Research only supports using its built-in search system ( Native Search ) and detailed outputs are unavailable, so # of Search Qs and # Pages are omitted.
Figure 4Figure 5
Acc.
Acc@5min
Time
Search Effort
Modality (%)
URL Overlap w/ Golden
System
(%)
(%)
(min)
Searches
Visits
URLs
Text
Video
Image
Prec.
Rec.
F1
Human
71.4
59.2
4.1
2.9
2.9
—
53.2
28.2
18.5
38.1
48.9
42.8
Native
30.9
29.6
2.3
9.8
0.1
34.9
96.2
0.0
3.8
—
—
—
Agentic
40.1
34.0
4.0
9.1
3.5
63.6
87.0
4.4
8.5
1.8
61.4
3.6
Table 3: Performance across human annotators, Native Search (Native), and Agentic Multimodal Search (Agentic) with Gemini 3.1 Pro. Acc@5min : accuracy under a 5-minute budget, where any question taking more than 5 minutes is counted as incorrect. Time : average completion time in minutes. Search Effort : average number of search queries issued (Searches), webpages visited (Visits), and unique URLs encountered (URLs) per question. Modality : distribution of resource modalities among accessed content. URL Overlap w/ Golden : precision, recall, and F1 of the system’s visited URLs against the golden reference URLs.
Setting
Web Search
Gold Sources
Agent Tools
Acc.
No Search
✗
✗
✗
24.7±1.6
Native Search
✓
✗
✗
29.0±1.1
Agentic Multimodal Search
✓
✗
✓
40.1±2.8
+ Gold Sources Injection
✓
✓
✓
43.4±3.8
+ Gold Sources Only
✗
✓
✓
45.5±2.3
Gold Sources Prompting
✗
✓
✗
47.7±2.0
Table 4: Isolating search vs. reasoning limitations for Gemini-3.1-Pro on MERRIN . Web Search : whether the agent can search the open web. Gold Sources : whether gold sources are provided. Agent Tools : whether the agent can use custom tools ( visit_webpage , watch_video ) to process evidence.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Source
# Q
Avg Q Len
Avg # Res
Existing Benchmarks
42
23.4
2.1
From Scratch
120
25.2
1.9
Total
162
24.7
2.0
Appendix
Table 5: Dataset statistics for MERRIN . # Q : number of questions; Avg Q Len : average question length in words; Avg # Res : average number of gold resources per question.
Figure 7: Temporal characteristics of MERRIN . (a) Distribution by Effective Year — the year in which the answer first became correct. Sparse years before 2020 are collapsed into Pre-2010, 2010–14, and 2015–19 buckets, while recent years are shown individually. (b) Distribution by Freshness — how time-sensitive the ground-truth answer is.
Figure 8: Instruction for Human Annotation
Figure 9: Autorater prompt used for grading responses. Placeholders {question} , {response} , and {correct_answer} are filled at evaluation time.
Input Query
Built-in Search
Model
Context (In/Out)
Text
Image
Video
Audio
Text
Image
Video
Audio
GPT
400k/128k
✓
✓
✗
✗
✓
✓
✗
✗
Gemini
1M/64k
✓
✓
✓
✓
✓
✓
✗
✗
Qwen3
230k/32k
✓
✗
✗
✗
-
-
-
-
Appendix
Table 6: Model context window sizes and modality support. Context (In/Out) : maximum input/output token limits. Input Query : modalities the model can accept as direct input. Built-in Search : modalities the model can process when using its built-in search tool under Native Search .
Figure 10: Instruction for Human Evaluation
Model
No Search
Native Search
Native Search + Video Tool
Agentic Multimodal Search
Gemini 3 Flash
19.1±3.2
31.7±3.8
36.8±2.6
32.9±0.9
Gemini 3 Pro
23.5±1.1
28.8±1.4
35.6±0.4
39.9±1.6
Gemini 3.1 Lite
12.8±2.3
20.6±2.2
21.6±1.2
26.3±1.9
Gemini 3.1 Pro
24.7±1.6
29.0±1.1
37.5±2.0
40.1±2.8
Appendix
Table 7: Impact of adding a video processing tool to Native Search . Accuracy (%) across four Gemini models under four settings: No Search , Native Search , Native Search + Video Tool (adding video processing tool), and Agentic Multimodal Search (full multimodal agent).
With the increasing demand for step-wise, cross-modal, and knowledge-grounded reasoning, multimodal large language models (MLLMs) are evolving beyond the traditional fixed retrieve-then-generate paradigm toward more sophisticated agentic multimodal retrieval-augmented generation (MM-RAG). Existing benchmarks, however, mainly focus on simplified QA with short retrieval chains, leaving adaptive planning and multimodal reasoning underexplored. We present MC-Search, the first benchmark for agentic MM-RAG with long, step-wise annotated reasoning chains spanning five representative reasoning structures. Each example specifies sub-questions, retrieval modalities, supporting facts, and intermediate answers, with fidelity ensured by HAVE (Hop-wise Attribution and Verification of Evidence), resulting in 3,333 high-quality examples averaging 3.7 hops. Beyond answer accuracy, MC-Search introduces new process-level metrics for reasoning quality, stepwise retrieval and planning accuracy. By developing a unified agentic MM-RAG pipeline, we benchmark six leading MLLMs and reveal systematic issues such as over- and under-retrieval and modality-misaligned planning. Finally, we introduce Search-Align, a process-supervised fine-tuning framework leveraging verified reasoning chains, showing that our data not only enables faithful evaluation but also improves planning and retrieval fidelity in open-source MLLMs.
Xuying Ning, Dongqi Fu, Tianxin Wei +7
University of Illinois Urbana-Champaign · Meta · IBM Research +1
AI agents with advanced reasoning and tool-use capabilities have demonstrated impressive performance in web browsing for deep search. However, existing benchmarks such as BrowseComp primarily focus on textual content, overlooking the prevalence of multimodal content. To bridge this gap, we introduce MM-BrowseComp, a novel benchmark comprising 400 challenging, hand-crafted questions designed to evaluate multimodal retrieval and reasoning capabilities. Unlike prior work, MM-BrowseComp incorporates visual prompts and necessitates the extraction of key evidence from web images and videos to complete questions, rendering text-only approaches insufficient. Additionally, we provide a verified checklist for each question, enabling fine-grained analysis of multimodal dependencies and reasoning paths. Our comprehensive evaluation of 27 state-of-the-art models reveals that even leading models like GPT-5-High with tools achieve only 24.25% accuracy, highlighting the suboptimal multimodal browsing capabilities, establishing MM-BrowseComp as a rigorous new standard for the field.
Shilong Li, Xingyuan Bu, Wenjie Wang +22
1ByteDance · 2Nanjing University · 5Zhejiang University +2
Large models are increasingly becoming autonomous agents that interact with real-world environments and use external tools to augment their static capabilities. However, most recent progress has focused on text-only large language models, which are limited to a single modality and therefore have narrower application scenarios. On the other hand, multimodal large models, while offering stronger perceptual capabilities, remain limited to static knowledge and lack the ability to access and leverage up-to-date web information. In this paper, we propose VSearcher, turning static multimodal model into multimodal search agent capable of long-horizon, multi-turn tool use in real-world web environments, including text search, image search, and web browsing, via reinforcement learning. Specifically, we introduce Iterative Injection Data Synthesis pipeline to generate large-scale, complex multimodal QA questions, which are further filtered with comprehensive metrics to ensure high quality and sufficient difficulty. We then adopt an SFT-then-RL training pipeline to turn base multimodal models to agent capable of multi-turn tool calling in real-world web environments. Besides, we propose a multimodal search benchmark MM-SearchExam dedicated to evaluating search capabilities of multimodal search agents, which proves highly challenging for recent proprietary models. Extensive evaluations across multiple multimodal search benchmarks reveal effectiveness of our method. VSearcher achieves superior performance compared to recent multimodal search agents and even surpasses several proprietary models on multimodal web search tasks.