MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains

Xuying Ning; Dongqi Fu; Tianxin Wei; Mengting Ai; Jiaru Zou; Ting-Wei Li; Hanghang Tong; Yada Zhu

cs.AIMar 1, 2026

MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains

Authors: Xuying Ning, Dongqi Fu, Tianxin Wei, Mengting Ai, Jiaru Zou, Ting-Wei Li, Hanghang Tong, Yada Zhu, +2 more

Organizations: University of Illinois Urbana-Champaign · Meta · IBM Research · Stony Brook University

Abstract

With the increasing demand for step-wise, cross-modal, and knowledge-grounded reasoning, multimodal large language models (MLLMs) are evolving beyond the traditional fixed retrieve-then-generate paradigm toward more sophisticated agentic multimodal retrieval-augmented generation (MM-RAG). Existing benchmarks, however, mainly focus on simplified QA with short retrieval chains, leaving adaptive planning and multimodal reasoning underexplored. We present MC-Search, the first benchmark for agentic MM-RAG with long, step-wise annotated reasoning chains spanning five representative reasoning structures. Each example specifies sub-questions, retrieval modalities, supporting facts, and intermediate answers, with fidelity ensured by HAVE (Hop-wise Attribution and Verification of Evidence), resulting in 3,333 high-quality examples averaging 3.7 hops. Beyond answer accuracy, MC-Search introduces new process-level metrics for reasoning quality, stepwise retrieval and planning accuracy. By developing a unified agentic MM-RAG pipeline, we benchmark six leading MLLMs and reveal systematic issues such as over- and under-retrieval and modality-misaligned planning. Finally, we introduce Search-Align, a process-supervised fine-tuning framework leveraging verified reasoning chains, showing that our data not only enables faithful evaluation but also improves planning and retrieval fidelity in open-source MLLMs.

Figures & tables

Figure 1: Example query requiring multimodal agentic search with a long reasoning chain.

Benchmark	Knowledge Modality	Typical Hops	Long Chain ( $\geq 4$ Hops)	Stepwise Annotation	Reasoning Topology
OK-VQA ( Marino et al., 2019 )	Text	1 Hop	✗	✗	✗
ViQuAE ( Lerner et al., 2022 )	Text	1 Hop	✗	✗	✗
WebQA ( Chang et al., 2022 )	Text/Caption	$\leq$ 2 Hops	✗	✗	✗
InfoSeek ( Chen et al., 2023 )	Text	1 Hop	✗	✗	✗
MMSearch ( Jiang et al., 2024 )	Text/Image	1 Hop	✗	✗	✗
Dyn-VQA ( Li et al., 2025b )	Text/Image	$\leq$ 2 Hops	✗	✗	✗

Table 1: Left : Comparison of existing multimodal retrieval-augmented QA datasets. Right : Distribution of the five reasoning topologies in MC-Search , with outer rings illustrating hop diversity (2–5 hops).

Figure 2: Overview of MC-Search benchmark and evaluation. Left : Benchmark covering five reasoning topologies, filtered via the hop-wise attribution and verification of evidence (HAVE) process. Right : Multimodal agentic RAG pipeline, where an MLLM iteratively generates sub-queries and actions, retrieves multimodal evidence, reasons over the retrieved information, and integrates it to produce the final answer. Our framework further aligns predicted reasoning chains with golden trajectories to assess chain-level retrieval and planning.

Statistic	Number
Parallel Image-Text Fork	680
Image–Initiated Chain	1,306
Text Chain	945
Text–Initiated Chain	169
Multi–Images Fork	233
Total Samples	3,333

Table 2: Benchmark statistics.

Panel of Table 3 — Table 3: Evaluation of MLLMs on the MC-Search benchmark under the agentic MM-RAG pipeline, reported across five reasoning graphs. Best backbone results are shown in bold , second-best are underlined , and improvements from Search-Align on open-source models are highlighted in bold red .

Type	Modality	Gemini-2.5-Pro		InternVL-3.5-8B
Type	Modality	Covered/Gold	Cov. (%)	Covered/Gold	Cov. (%)
Query w/ Image	Image	1,339/1,533	87.35	977/1,533	63.84
Query w/ Image	Text	3,230/4,109	78.61	3,395/4,109	82.67
Query w/o Image	Image	300/1,017	29.50	7/1,017	0.66
Query w/o Image	Text	2,097/2,510	83.55	2,251/2,510	89.78
Overall	Image	1,639/2,550	64.27	861/2,550	33.69

Table 4: Stepwise retrieval modality coverage (Cov.). Red = high coverage; Blue = low coverage.

Figure 5: Eight-way error taxonomy proportions (higher is worse), annotated by Gemini-2.5-Pro.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Reason Graphs	Model	Answer Accuracy			Chain Alignment		Golden F1 ( $\uparrow$ )
Reason Graphs	Model	F1 ( $\uparrow$ )	$\Delta$ F1 ( $\uparrow$ )	LJ ( $\uparrow$ )	HPS ( $\uparrow$ )	RD ( $\downarrow$ )	Golden F1 ( $\uparrow$ )
Text-Only Chain	GPT-4o-Mini	27.50	27.48	1.69	19.22	3.00	61.90
	Gemini-2.5-Flash	10.88	10.81	0.75			67.72
	Gemini-2.5-Pro	11.11	11.10	0.70			62.42
	Claude-3.7-Sonnet	22.71	22.09	1.69			66.38
	InternVL3.5-8B	30.98	24.00	1.47			65.72

Appendix

Table 5: Evaluation of various models on the MC-Search benchmark under single-step RAG .

Reason Graphs	Model	Answer Accuracy			Chain Alignment		Golden F1 ( $\uparrow$ )
Reason Graphs	Model	F1 ( $\uparrow$ )	$\Delta$ F1 ( $\uparrow$ )	LLJ ( $\uparrow$ )	HPS ( $\uparrow$ )	RD ( $\downarrow$ )	Golden F1 ( $\uparrow$ )
Image Initiated Chain	GPT-4o-Mini	35.61	33.30	1.58	21.11	2.42	68.29
	Gemini-2.5-Flash	7.42	0.70	0.32			72.39
	Gemini-2.5-Pro	6.06	1.21	0.25			69.83
	Claude-3.7-Sonnet	21.72	21.10	1.72			72.62
	InternVL3.5-8B	31.26	21.64	1.24			63.86

Appendix

Table 6: Evaluation of various models on the MC-Search benchmark under traditional MM-RAG .

Statistic	Number
Parallel Visual-Textual Fork Samples	680
Image-Initiated Chain Samples	1,306
Text Chain Samples	945
Text-Initiated Chain Samples	169
Multi-Images Fork Samples	233
Total Samples	3,333

Appendix

Table 7: Statistics of MC-Search benchmark.

Dimension	Average Score
Factual correctness	4.93
Step necessity	4.70
Clarity	4.88
Multimodal alignment	4.95
Overall (mean)	4.87

Appendix

Table 8: Average Gemini-2.5-Pro evaluation scores.

Evaluator Pair	Pearson	Spearman	Mean Bias	LoA
Gemini / GPT-5	0.9330 (0.00076)	0.8720 (0.00120)	+0.0288 (0.00051)	0.8438 (0.07937)
Gemini / Qwen	0.9147 (0.00116)	0.8145 (0.00360)	$-0.0083$ (0.00082)	0.9295 (0.08834)
GPT-5 / Qwen	0.9147 (0.00131)	0.8259 (0.00374)	$-0.0371$ (0.00212)	0.9341 (0.09565)

Appendix

Table 9: Cross-model LLJ consistency scores, with mean (variance) reported. High correlations and small mean biases indicate strong agreement across evaluator families.

Figure 6: Hexbin plots of Gemini-2.5-Pro vs. GPT-5 scores across four LLJ dimensions.

Dimension	Pearson	Spearman	Mean Bias	LoA
Accuracy	0.9632	0.9556	+0.0800	0.6606
Entity Coverage	0.9375	0.8691	+0.1200	0.9309
Coherence	0.6061	0.8200	$-0.0200$	0.6778
Alignment	0.9275	0.7925	+0.2750	0.9982
Stacked Score	0.9323	0.8675	+0.1138	0.8562

Appendix

Table 10: Human–LLJ agreement scores.

Graph Type	Model	F1 Score ( $\uparrow$ )		LLJ Score ( $\uparrow$ )
Graph Type	Model	GPT	Gemini	GPT	Gemini
Text Chain	Gemini-Pro	32.52	34.42	3.44	3.46
Text Chain	GPT-4o-mini	26.05	26.98	3.01	3.05
Image-Initiated Chain	Gemini-Pro	41.37	46.57	3.82	3.81
Image-Initiated Chain	GPT-4o-mini	35.01	38.96	3.24	3.20
Multi-Images Fork	Gemini-Pro	35.49	38.04	3.48	3.66

Appendix

Table 11: Performance of Gemini-2.5-Pro and GPT-4o-mini when evaluated against ground-truth reasoning chains generated by GPT-5 (GPT columns) or by the original Gemini-based construction (Gemini columns).

Figure 7: Error proportions before and after SEARCH-ALIGN across four process-level categories.

Hyper-parameter	InternVL3.5-8B	Qwen2.5-VL-7B-Instruct
Optimizer	AdamW	AdamW
Learning Rate	1e-5	1e-4
Training Epochs	1	2
Warm-up Ratio	0.1	0.1
Scheduler Type	Cosine	Cosine
Train Batch (per device)	1	1

Appendix

Table 12: Fine-tuning hyper-parameters for Search-Align alignment on InternVL3.5-8B and Qwen/Qwen2.5-VL-7B-Instruct. Both models are trained on 4

\times

A100 GPUs.

Graph Type	Gemini-2.5-Pro Soft-HPS				Gemini-Flash Soft-HPS
Graph Type	$\tau$ =1.0	$\tau$ =0.95	$\tau$ =0.90	$\tau$ =0.85	$\tau$ =1.0	$\tau$ =0.95	$\tau$ =0.90	$\tau$ =0.85
Text Chain	21.59	35.59	38.05	41.26	13.33	26.86	28.78	30.97
Image-Initiated Chain	25.90	38.74	40.48	42.87	25.03	43.19	45.04	47.42
Multi-Images Fork	18.68	25.58	26.49	28.18	18.53	24.51	25.35	27.20
Parallel Image-Text Fork	16.74	26.25	27.68	30.22	20.73	25.68	27.12	29.69
Text-Initiated Chain	19.51	26.31	27.81	29.88	21.67	29.94	30.55	32.03

Appendix

Table 13: Soft HPS scores for Gemini-2.5-Pro and Gemini-Flash under different similarity thresholds

\tau

Figure 8:

\Delta

F1 across

\Delta

Step values for Gemini-2.5-Flash under top-

k

retrieval (

k=1,3,5

Graph Type	Model	Retrieval@1				Retrieval@3				Retrieval@5
Graph Type	Model	F1	$\Delta$ F1	HPS	RD	F1	$\Delta$ F1	HPS	RD	F1	$\Delta$ F1	HPS	RD
Image-Initiated Chain	Qwen2.5-VL-7B	26.30	8.65	16.51	4.04	29.21	11.56	16.17	3.79	29.35	11.70	16.95	3.79
Image-Initiated Chain	+SEARCH-ALIGN	45.70	28.05	33.59	0.70	47.19	29.54	39.45	0.75	47.29	29.64	41.79	0.79
Multi-Images Fork	Qwen2.5-VL-7B	26.13	7.53	12.58	3.80	30.41	11.81	12.80	3.35	30.38	11.78	14.15	3.23
Multi-Images Fork	+SEARCH-ALIGN	38.01	19.41	41.20	1.16	40.00	21.40	46.82	1.21	39.80	21.20	48.21	1.28
Parallel Image-Text Fork	Qwen2.5-VL-7B	22.94	14.32	10.09	3.93	26.28	17.66	12.62	3.39	26.41	17.79	15.24	3.26

Appendix

Table 14: Performance of Qwen2.5-VL-7B and +SEARCH-ALIGN under Retrieval@1, @3, and @5. We report F1,

\Delta

F1, Hit per Step (HPS), and Rollout Deviation (RD). Best values per topology and metric are highlighted in bold font.

Figure 9: Success case. The reasoning chain produced by the agent aligned with the golden reasoning chain and the final answer includes all key knowledge entities.

Figure 10: Failure case. While the agent successfully retrieves the first and the last hop information, it fails to retrieve the second and third hop information. Thus, the final answer does not include all key knowledge entities.

Explore similar work

DR-MMSearchAgent: Deepening Reasoning in Multimodal Search Agents
Apr 21, 2026Shengqin Wang, Wentao Yan, Huichi Zhou +4Reward Shaping RL for Language Model Reasoning
East China Normal University · University College London · Huawei Noah’s Ark Lab +2
MTA-Agent: An Open Recipe for Multimodal Deep Search Agents
Apr 7, 2026Xiangyu Peng, Can Qin, An Yan +4Multimodal Search Agents Multimodal QA
Salesforce AI Research
MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baseline
Mar 1, 2026Huanjin Yao, Qixiang Yin, Min Yang +5Multimodal Search Agents Multimodal QA
ByteDance · Zhongguancun Academy · Nanyang Technological University +2

Apr 21, 2026J/K move · Enter open · S save

DR-MMSearchAgent: Deepening Reasoning in Multimodal Search Agents

Shengqin Wang, Wentao Yan, Huichi Zhou, Yihang Chen+3

East China Normal University · University College London · Huawei Noah’s Ark Lab+2

Reward Shaping RL for Language Model Reasoning Multimodal QA Multimodal Search Agents Multimodal Agents RL for LLM Agents

Open paper AskPDF arXiv

Agentic multimodal models have garnered significant attention for their ability to leverage external tools to tackle complex tasks. However, it is observed that such agents often meet premature interaction collapse, caused by two primary reasons: 1) the terminal reward often appending on the last token prevents the advantage from distinguishing trajectories with exploratory behavior; 2) excessively redundant context hinders the agent from absorbing useful feedback. To address these issues, we propose the Deepening Reasoning MMSearchAgent, the framework leverages the structural proximity to derive advantage signals from the whole rollout trajectories in an entire batch, such that trajectories of different lengths are further encouraged to be generated, even when containing the same correct answer. Additionally, differentiated gaussian rewards are employed to dynamically calibrate interaction tolerance, thereby ensuring information reliability and reduce redundancy. To support multi-turn interaction training, we have constructed a multi-step deep-reasoning dataset including 3602 high-quality QA pair with at least 3 reasonning steps. Extensive experiments demonstrate that our method achieves state-of-the-art performance, outperforming the MMSearch-R1 by 8.4

\%

on FVQA-test.

Abstract

Explore similar work

DR-MMSearchAgent: Deepening Reasoning in Multimodal Search Agents

MTA-Agent: An Open Recipe for Multimodal Deep Search Agents

MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baseline