Organizations: Peking University · Southwest University · Tsinghua University · Independent Researcher · National Engineering Research Center of New Electronic Publishing Technologies
Static and incomplete parametric knowledge limits the performance of multimodal large language models on novel-knowledge and knowledge-intensive tasks, especially those requiring multi-hop evidence acquisition. Existing multimodal agentic search methods often fix visual evidence at the input or a predefined trajectory position, making it difficult for agents to select search modalities as their information needs evolve and to handle complex real-world queries. We define this capability as autonomous cross-modal search: an agent chooses among direct reasoning, text-to-text search, text-to-image search, and reverse-image search, actively introduces visual evidence when appropriate, and switches between and integrates modalities across multiple hops. We propose ModalSearch-Pipeline, a scalable procedure for constructing an interleaved multimodal knowledge graph. It samples paths with diverse depths and modality compositions and uses backward question construction to obscure entity names and image targets that would expose reasoning shortcuts while retaining necessary disambiguating clues, yielding text-only, image-initialized, and arbitrarily interleaved multi-hop problems. We train the ModalSearcher Agent with supervised fine-tuning and reinforcement learning in a live Web environment, and construct ModalSearch-Bench. Experiments show that ModalSearcher-8B improves average accuracy by 16.2 points over Qwen3-VL-8B under the same agentic workflow across five multimodal knowledge-intensive QA benchmarks, with consistent gains on independently constructed active visual-search benchmarks.
Figures & tables
Figure 1: Comparison of multimodal search task formulations: input-image understanding, explicitly specified visual-search targets, and autonomous acquisition of visual evidence during reasoning.
Figure 2: Overview of the proposed data-synthesis pipeline. The pipeline constructs an interleaved multimodal knowledge graph, samples modality-diverse reasoning paths, and synthesizes questions that obscure shortcuts and visual targets while retaining necessary disambiguating clues and avoiding explicit modality-switch instructions.
Model
VDR
MMSearch
LiveVQA
BC-VL
FVQA
Avg
Direct Reasoning
GPT-5 [ 29 ]
9.8
35.1
44.4
48.6
54.4
38.5
Gemini-2.5-Pro [ 27 ]
8.0
39.8
60.3
43.1
60.7
42.4
Claude-4-Sonnet
2.0
18.7
38.5
29.3
35.3
24.8
Qwen3-VL-8B [ 3 ]
2.4
15.0
41.7
21.8
27.9
21.8
Qwen3-VL-32B [ 3 ]
4.0
23.7
49.7
28.3
35.6
28.3
Table 1: Performance on multimodal knowledge-intensive QA and web-search benchmarks, including MMSearch [ 17 ] , BrowseComp-VL [ 8 ] , VDR-Bench [ 38 ] , FVQA [ 30 ] , and LiveVQA [ 7 ] . Bold and underline mark the best and second-best scores in each column within each group, respectively.
Model
VisBrowse
BrowseComp -V3
ModalSearch
Avg.
IO
SR
Avg.
GPT-5 [ 29 ]
30.2
13.8
66.0
37.8
47.2
30.4
Gemini-2.5-Pro [ 27 ]
23.4
13.5
53.5
29.5
37.5
24.8
GPT-5 (w/ tools)
51.5
26.2
73.0
54.8
60.9
46.2
Gemini-2.5-Pro (w/ tools)
38.5
25.3
59.4
47.3
51.3
38.4
OpenSearch-VL-8B [ 5 ]
11.2
–
15.5
10.9
12.4
–
Table 2: Performance on autonomous cross-modal search benchmarks, including VisBrowse [ 42 ] and BrowseComp-V3 [ 39 ] . The first two rows use direct answering; “w/ tools” denotes tool-augmented evaluation. IO and SR denote Input-only (100 questions) and Search-required (200 questions). Bold and underlined scores mark the best and second-best open-source results. We report the mean accuracy over 5 runs.
Training data
MMSearch
BC-VL
VisBrowse
LiveVQA
ModalSearch
Avg.
Full (16K)
64.3
44.9
28.9
68.3
39.3
49.1
Full (7K subset)
63.0
41.9
26.6
69.1
37.3
47.6
Input-only (7K)
63.6
39.8
21.3
67.4
33.0
45.0
Text-only (7K)
58.7
37.9
19.5
60.3
28.7
41.0
Table 3: Ablation of cross-modal data generation. All 7K variants use only pipeline-generated data; Full (16K) additionally includes FVQA and LiveVQA training examples.
Figure 3: Search-tool usage and agentic framework ablation. (a) Search-tool proportions, with raw invocation counts annotated. Red bars have a fixed width; each blue bar’s width is scaled by the ModalSearcher-to-Qwen total-call ratio within that benchmark. Only t2t_search , t2i_search , and i2i_search are counted; read_url is excluded. (b) Benchmark accuracy under progressively restricted tool settings, including no-tool answering; each legend item pairs the Qwen3-VL-8B (red) and ModalSearcher-8B (blue) colors.
Table 4: Visual-search behavior before and after training on 200 search-required questions with 235 non-initial reference image nodes. All rates are percentages; Δ reports point differences computed before rounding. Alignment and coverage concern search targets rather than returned-image correctness.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Comparison of the data-construction strategies used by our pipeline and prior multimodal search settings. The comparison highlights differences in modality transitions, visual-evidence use, and the degree to which the search target is exposed in the question.
Metric
Training graph
Benchmark graph
Text nodes
35,000
10,000
Image nodes
35,048
10,035
Total nodes
70,048
20,035
Text-to-text edges
53,811
20,386
Text-to-image edges
19,546
5,915
Image-to-text edges
71,154
13,741
Appendix
Table 6: Statistics of the source graphs for training-data and benchmark construction. Unique images combine the original unique and semi-unique categories; non-unique images combine no-unique , wiki_inline , and previously unclassified images.
Stage
Configuration
Value / Setting
Graph construction
Relation-aware textual expansion model
Qwen3-VL-32B
Visual expansion, entity recognition, and linking model
Gemini-2.5-Pro
Image referential-stability assessment model
Gemini-3.5-Flash
Textual neighbor candidate budget Kcand
50
Maximum visual proposals per text node
5
Retrieved images per query Kimg
3
Appendix
Table 7: Models and parameters used in ModalSearch-Pipeline. Answer verification is distinct from benchmark answer judging. Visual verification uses three trials per condition, with and without the reference image.
Hops
SFT
RL
Total
Share
1
4,493
1,412
5,905
30.45%
2
4,721
2,001
6,722
34.67%
3
3,545
1,760
5,305
27.36%
4
761
698
1,459
7.52%
Total
13,520
5,871
19,391
100%
Appendix
Table 8: Reasoning-path length distribution of pipeline-generated training examples. The separately sampled Text-only ablation data and the FVQA and LiveVQA examples added to SFT are excluded. Share is computed over the combined SFT and RL total.
Figure 6: Overview of the pipeline-generated training data. Left: counts and proportions of Input-only examples, whose reference-path visual evidence is provided initially, and Search-required examples, whose reference solutions call for actively acquiring new visual information during reasoning. Right: a word cloud of 70 randomly sampled entities, illustrating the breadth of covered domains. Auxiliary FVQA and LiveVQA training examples and the separately sampled Text-only ablation data are excluded.
Table 13
Figure 7: RL training dynamics over 110 training steps: (a) answer accuracy on the training set, (b) the fraction of rollouts truncated at the maximum interaction-turn limit, and (c) the average number of repeated-search events per rollout. All panels show the same training run (Experiment 2). A repeated search is counted when two queries have cosine similarity above 0.9 and at least three of their five returned results overlap.
Parameter
Value
Parameter
Value
greedy
false
top_p
0.8
top_k
20
temperature
0
repetition_penalty
1.0
presence_penalty
1.5
max-turns
46
max-seq-lens
48,000
Appendix
Table 11: Sampling parameters used for inference.
Model
Accuracy
Positive class
Negative class
Macro-F1
Confusion matrix
Precision
Recall
F1
Precision
Recall
F1
TP
FP
FN
TN
Gemini-3.5-Flash
88.79
82.89
100.00
90.65
100.00
75.47
86.02
88.33
63
13
0
40
GPT-5.4
86.21
84.06
92.06
87.88
89.36
79.25
84.00
85.94
58
11
5
42
Gemini-2.5-Pro
83.62
76.83
100.00
86.90
100.00
64.15
78.16
82.53
63
19
0
34
Claude-Sonnet-4.6
81.03
74.12
100.00
85.14
100.00
58.49
73.81
79.47
63
22
0
31
Appendix
Table 12: Visual-retrievability judgments against human annotations on 116 examples: 63 stably retrievable (positive) and 53 retrieval-ambiguous (negative). Metric scores are percentages; TP, FP, FN, and TN are confusion-matrix counts. Bold marks the best metric score in each column, including ties.
Figure 8: A Search-required example generated by ModalSearch-Pipeline, with an initial image and additional visual evidence to be retrieved.
Figure 9: An Input-only example generated by ModalSearch-Pipeline, with visual evidence confined to the initial input.
Figure 10: A Search-required example generated by ModalSearch-Pipeline, with interleaved textual and visual evidence.
Figure 11: A Search-required example generated by ModalSearch-Pipeline, with no initial image and new visual evidence to be retrieved during reasoning.
Figure 12: System prompt used by the multimodal research agent during inference.
Figure 14: A real ModalSearcher-8B inference trajectory on the Search-required category of ModalSearch-Bench, involving reverse-image search, text retrieval, and visual verification of a historical painting.
Figure 15: A real ModalSearcher-8B inference trajectory on the Search-required category of ModalSearch-Bench, combining textual entity tracing with retrieval and inspection of new visual evidence.
Multimodal large language models (MLLMs) have demonstrated impressive capabilities in many visual tasks, but they often struggle with factual grounding when confronted with complex, open-world scenarios. While recent multimodal deep search agents attempt to address this issue by utilizing external tools, the visual-native search paradigm remains underexplored. Existing methods primarily rely on simple images with explicit semantics and text-only evidence trajectories, limiting the agent's ability to perform multi-hop, cross-modal reasoning and search. To address these limitations, we propose Visual-Seeker, a visual-native multimodal deep search agent via active visual reasoning. Rather than treating vision as a static input, our agent actively attends to fine-grained visual details, dynamically harvests visual evidence throughout the search process. To unlock its visual-native potential, we design an active visual reasoning data pipeline and synthesize 5K high-quality multimodal trajectories for model training. Extensive experiments demonstrate the state-of-the-art performance across five challenging multimodal search benchmarks, even surpassing several proprietary models, validating robust visual-native reasoning and search in real-world web environments. The code and data can be accessed at: https://github.com/ZhengboZhang/Visual-Seeker.
Zhengbo Zhang, Changtao Miao, Jinbo Su +10
School of Artificial Intelligence, UCAS · Institute of Automation, CAS · Ant Digital Technologies, Ant Group +2
Multimodal large language models (MLLMs) have demonstrated strong capabilities in visual understanding, yet they remain limited in complex, multi-step reasoning that requires deep searching and integrating visual evidence with external knowledge. In this work, we address this challenge by constructing high-quality, verified multi-hop vision-language training data for multimodal deep-search agents. We propose a Multi-hop Tool-Augmented Agent for Evidence-based QA Synthesis (MTA-Agent), which automatically selects tools and their parameters to retrieve and validate evidence from both visual and textual sources and generates structured multi-hop question-answer trajectories. Starting from diverse VQA seed datasets, our pipeline produces a large-scale training dataset, MTA-Vision-DeepSearch, containing 21K high-quality multi-hop examples. The data is filtered through a multi-stage verification process to ensure factual consistency and answer uniqueness. Using MTA-Vision-DeepSearch, a 32B open-source multimodal search agent achieves state-of-the-art performance, reaching an average of 54.63% across six challenging benchmarks, outperforming GPT-5 (51.86%), Gemini-2.5-Pro (50.98%), and Gemini-3-Pro (54.46%) under the same tool settings. We further show that training on our data improves both reasoning depth and tool-use behavior, increasing the average number of steps from 2.27 to 4.28, and leading to more systematic and persistent search strategies. Additionally, we demonstrate that training can be performed without real-time tool calls by replaying cached interactions, significantly reducing training cost. Importantly, we present MTA-Agent as a fully open recipe for multimodal deep search: we release the entire dataset, training trajectories, and implementation details to enable reproducibility and future research on open multimodal search agents.
Large models are increasingly becoming autonomous agents that interact with real-world environments and use external tools to augment their static capabilities. However, most recent progress has focused on text-only large language models, which are limited to a single modality and therefore have narrower application scenarios. On the other hand, multimodal large models, while offering stronger perceptual capabilities, remain limited to static knowledge and lack the ability to access and leverage up-to-date web information. In this paper, we propose VSearcher, turning static multimodal model into multimodal search agent capable of long-horizon, multi-turn tool use in real-world web environments, including text search, image search, and web browsing, via reinforcement learning. Specifically, we introduce Iterative Injection Data Synthesis pipeline to generate large-scale, complex multimodal QA questions, which are further filtered with comprehensive metrics to ensure high quality and sufficient difficulty. We then adopt an SFT-then-RL training pipeline to turn base multimodal models to agent capable of multi-turn tool calling in real-world web environments. Besides, we propose a multimodal search benchmark MM-SearchExam dedicated to evaluating search capabilities of multimodal search agents, which proves highly challenging for recent proprietary models. Extensive evaluations across multiple multimodal search benchmarks reveal effectiveness of our method. VSearcher achieves superior performance compared to recent multimodal search agents and even surpasses several proprietary models on multimodal web search tasks.