MTA-Agent: An Open Recipe for Multimodal Deep Search Agents

Xiangyu Peng; Can Qin; An Yan; Xinyi Yang; Zeyuan Chen; Ran Xu; Chien-Sheng Wu

cs.CVApr 7, 2026

MTA-Agent: An Open Recipe for Multimodal Deep Search Agents

Authors: Xiangyu Peng, Can Qin, An Yan, Xinyi Yang, Zeyuan Chen, Ran Xu, Chien-Sheng Wu

Organizations: Salesforce AI Research

Abstract

Multimodal large language models (MLLMs) have demonstrated strong capabilities in visual understanding, yet they remain limited in complex, multi-step reasoning that requires deep searching and integrating visual evidence with external knowledge. In this work, we address this challenge by constructing high-quality, verified multi-hop vision-language training data for multimodal deep-search agents. We propose a Multi-hop Tool-Augmented Agent for Evidence-based QA Synthesis (MTA-Agent), which automatically selects tools and their parameters to retrieve and validate evidence from both visual and textual sources and generates structured multi-hop question-answer trajectories. Starting from diverse VQA seed datasets, our pipeline produces a large-scale training dataset, MTA-Vision-DeepSearch, containing 21K high-quality multi-hop examples. The data is filtered through a multi-stage verification process to ensure factual consistency and answer uniqueness. Using MTA-Vision-DeepSearch, a 32B open-source multimodal search agent achieves state-of-the-art performance, reaching an average of 54.63% across six challenging benchmarks, outperforming GPT-5 (51.86%), Gemini-2.5-Pro (50.98%), and Gemini-3-Pro (54.46%) under the same tool settings. We further show that training on our data improves both reasoning depth and tool-use behavior, increasing the average number of steps from 2.27 to 4.28, and leading to more systematic and persistent search strategies. Additionally, we demonstrate that training can be performed without real-time tool calls by replaying cached interactions, significantly reducing training cost. Importantly, we present MTA-Agent as a fully open recipe for multimodal deep search: we release the entire dataset, training trajectories, and implementation details to enable reproducibility and future research on open multimodal search agents.

Figures & tables

Figure 1: Data creation pipeline. Starting from filtered seed VQA data, the QA agent iteratively selects tools to collect sufficient feedback for generating single-hop QA candidates. These candidates are verified, and the most diverse valid candidate is appended to the multi-hop reasoning chain. If no candidate passes verification, the agent continues to gather additional evidence by selecting tools. Detailed examples are shown in Appendix D .

Seed Dataset	Original	2-hop	3-hop	4-hop	Total	Test	Cost	Tools
FVQA	4,750	1,155	1,153	1,113	8,171	33	$ 0.30	9.8
Infoseek	1,942	1,135	1,130	934	5,141	51	$ 0.25	8.4
InfoVQA	217	483	481	400	1,581	38	$ 0.28	9.8
News	918	240	238	236	1,632	13	$ 0.27	8.6
OK-VQA	629	1,418	1,415	1,168	4,630	43	$ 0.31	10.1
Total	8,456	4,431	4,417	3,851	21,155	178	$ 0.28	9.3

Table 1: Number of scenarios per hop in our training and test data. "Cost" and “Tools" indicate the average LLM cost and number of tool calls per sample for generation.

Direct Answer
Model	MMSrch-Plus	HR-MMSrch	BC-VL	MMSrch	FVQA	MTA-test	Avg
GPT-5	13.18	16.39	41.60	36.47	50.83	21.35	29.97
Gemini-2.5-pro	12.22	15.41	39.85	41.76	54.50	21.91	30.94
Gemini-3-pro	26.37 ♠	23.61	41.35	65.88 ♠	58.94	26.97	40.52
Qwen3-VL-8B-Inst.	1.29	4.59	20.05	12.35	26.94	6.18	11.90
Qwen3-VL-32B-Inst.	2.25	4.59	24.81	17.65	24.81	11.24	14.22

Table 2: Accuracy (%) on deep-search benchmarks. Best results are in bold , second-best are underlined ,

{\dagger}

denotes results reported in the original papers, and

\spadesuit

indicates potential dataset contamination.

Data Choice	MMSrch-Plus	BC-VL	MMSrch	HR-MMSrch	Avg.
Qwen3-VL-8B-Inst.	18.38	35.89	57.06	32.13	35.87
w/ F+L	20.96	43.22	73.46	43.28	45.23
w/ F + L + 4-hops (F+L)	20.32	43.35	74.12	43.60	45.35
w/ F + L + 2/3/4 hops (F + L)	20.96	44.11	75.29	44.61	46.24
w/ F + L + 2/3/4 hops (Full)	26.77 26.77	44.36 44.36	79.41 79.41	47.54 47.54	49.52 49.52

Table 3: Impact of different training data choices for Qwen3-VL-8B-Inst. as the search agent backbone, using the same settings (100 steps, batch size = 64, rollout = 8).

Model	MMSrch-Plus	HR-MMSrch	BC-VL	MMSrch	FVQA	MTA-test	Avg
Qwen3-VL-8B-Inst.	18.38	32.13	35.89	57.06	64.22	11.80	36.58
+ FVQA & LiveVQA	20.96	43.28	43.22	73.46	71.06	12.36	44.06
+ MTA-DS (replay)	27.62	43.61	43.86	76.47	71.33	15.17	46.34
+ MTA-DS (tool-call)	26.77	47.54	44.36	79.41	73.06	20.79	48.66

Table 4: Accuracy (%) on search-oriented benchmarks with and without external tool calls.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Example entities
People	Barack Obama, Marie Curie, Lionel Messi
Organizations	Google, UNICEF, Manchester United
Locations	Tokyo, Eiffel Tower, Yellowstone
Products	iPhone 15, Tesla Model 3, Photoshop
Events	FIFA World Cup, Diwali, TED Conference
Creative Works	Inception, Mona Lisa, Harry Potter

Appendix

Table 5: Entity categories used for filtering.

Stage	Check	Model	Reject if…
1	Vision requirement	GPT-5-mini	Question answerable without image
2	Q&A rewriting	GPT-5	Question cannot be free-form; conversion fails
3	Entity classification	GPT-5	Answer is not a named proper noun entity
4	Entity validity	GPT-5-mini	Answer is a generic or vague term
5	Visual verification	GPT-5 (vision + web)	Answer is factually incorrect or uncertain

Appendix

Table 6: Summary of the VQA filtering pipeline.

Tool Name	XML Tag	Purpose
web_text_to_text_search	<text_search_text>	Text-to-text web search; returns snippets and URLs
web_text_to_img_search	<text_search_image>	Text-to-image web search; returns image descriptions
web_url_reader	<web_read>	Full-page content extraction from a URL
web_image_to_text	<image_search_text>	Google Lens reverse-image search; identifies objects
ocr_tool	<ocr_tool>	OCR extraction of all visible text from an image
ipython_code	<python>	IPython code execution with persistent state

Appendix

Table 7: Tool registry available to the multimodal search agent.

Explore similar work

May 6, 2026

OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents

Deep search has become a crucial capability for frontier multimodal agents, enabling models to solve complex questions through active search, evidence verification, and multi-step reasoning. Despite rapid progress, top-tier multimodal search agents remain difficult to reproduce, largely due to the absence of open high-quality training data, transparent trajectory synthesis pipelines, or detailed training recipes. To this end, we introduce OpenSearch-VL, a fully open-source recipe for training frontier multimodal deep search agents with agentic reinforcement learning. First, we curated a dedicated pipeline to construct high-quality training data through Wikipedia path sampling, fuzzy entity rewriting, and source-anchor visual grounding, which jointly reduce shortcuts and one-step retrieval collapse. Based on this pipeline, we curate two training datasets, SearchVL-SFT-36k for SFT and SearchVL-RL-8k for RL. Besides, we design a diverse tool environment that unifies text search, image search, OCR, cropping, sharpening, super-resolution, and perspective correction, enabling agents to combine active perception with external knowledge acquisition. Finally, we propose a multi-turn fatal-aware GRPO training algorithm that handles cascading tool failures by masking post-failure tokens while preserving useful pre-failure reasoning through one-sided advantage clamping. Built on this recipe, OpenSearch-VL delivers substantial performance gains, with over 10-point average improvements across seven benchmarks, and achieves results comparable to proprietary commercial models on several tasks. We will release all data, code, and models to support open research on multimodal deep search agents.

Shuang Chen, Kaituo Feng, Hangting Chen +7

Tencent Hunyuan · University of California, Los Angeles · The Chinese University of Hong Kong +1

Multimodal IR Multimodal Search Agents Multimodal Agents RL for LLM Agents Multimodal Tool Use

Ask PDF

Mar 1, 2026

MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baseline

We aim to develop a multimodal research agent capable of explicit reasoning and planning, multi-tool invocation, and cross-modal information synthesis, enabling it to conduct deep research tasks. However, we observe three main challenges in developing such agents: (1) scarcity of search-intensive multimodal QA data, (2) lack of effective search trajectories, and (3) prohibitive cost of training with online search APIs. To tackle them, we first propose Hyper-Search, a hypergraph-based QA generation method that models and connects visual and textual nodes within and across modalities, enabling to generate search-intensive multimodal QA pairs that require invoking various search tools to solve. Second, we introduce DR-TTS, which first decomposes search-involved tasks into several categories according to search tool types, and respectively optimize specialized search tool experts for each tool. It then recomposes tool experts to jointly explore search trajectories via tree search, producing trajectories that successfully solve complex tasks using various search tools. Third, we build an offline search engine supporting multiple search tools, enabling agentic reinforcement learning without using costly online search APIs. With the three designs, we develop MM-DeepResearch, a powerful multimodal deep research agent, and extensive results shows its superiority across benchmarks. Code is available at https://github.com/HJYao00/MM-DeepResearch

Huanjin Yao, Qixiang Yin, Min Yang +5

ByteDance · Zhongguancun Academy · Nanyang Technological University +2

Multimodal Search Agents Multimodal QA

Ask PDF

Jun 13, 2026

Visual-Seeker: Towards Visual-Native Multimodal Agentic Search via Active Visual Reasoning

Multimodal large language models (MLLMs) have demonstrated impressive capabilities in many visual tasks, but they often struggle with factual grounding when confronted with complex, open-world scenarios. While recent multimodal deep search agents attempt to address this issue by utilizing external tools, the visual-native search paradigm remains underexplored. Existing methods primarily rely on simple images with explicit semantics and text-only evidence trajectories, limiting the agent's ability to perform multi-hop, cross-modal reasoning and search. To address these limitations, we propose Visual-Seeker, a visual-native multimodal deep search agent via active visual reasoning. Rather than treating vision as a static input, our agent actively attends to fine-grained visual details, dynamically harvests visual evidence throughout the search process. To unlock its visual-native potential, we design an active visual reasoning data pipeline and synthesize 5K high-quality multimodal trajectories for model training. Extensive experiments demonstrate the state-of-the-art performance across five challenging multimodal search benchmarks, even surpassing several proprietary models, validating robust visual-native reasoning and search in real-world web environments. The code and data can be accessed at: https://github.com/ZhengboZhang/Visual-Seeker.

Zhengbo Zhang, Changtao Miao, Jinbo Su +10

School of Artificial Intelligence, UCAS · Institute of Automation, CAS · Ant Digital Technologies, Ant Group +2

Multimodal IR Multimodal Grounding Multimodal Search Agents Tool-Using Vision-Language Agents Visual Search Multimodal Agents+1

Ask PDF