Organizations: Peking University · Southwest University · Tsinghua University · Independent Researcher · National Engineering Research Center of New Electronic Publishing Technologies
Static and incomplete parametric knowledge limits the performance of multimodal large language models on novel-knowledge and knowledge-intensive tasks, especially those requiring multi-hop evidence acquisition. Existing multimodal agentic search methods often fix visual evidence at the input or a predefined trajectory position, making it difficult for agents to select search modalities as their information needs evolve and to handle complex real-world queries. We define this capability as autonomous cross-modal search: an agent chooses among direct reasoning, text-to-text search, text-to-image search, and reverse-image search, actively introduces visual evidence when appropriate, and switches between and integrates modalities across multiple hops. We propose ModalSearch-Pipeline, a scalable procedure for constructing an interleaved multimodal knowledge graph. It samples paths with diverse depths and modality compositions and uses backward question construction to obscure entity names and image targets that would expose reasoning shortcuts while retaining necessary disambiguating clues, yielding text-only, image-initialized, and arbitrarily interleaved multi-hop problems. We train the ModalSearcher Agent with supervised fine-tuning and reinforcement learning in a live Web environment, and construct ModalSearch-Bench. Experiments show that ModalSearcher-8B improves average accuracy by 16.2 points over Qwen3-VL-8B under the same agentic workflow across five multimodal knowledge-intensive QA benchmarks, with consistent gains on independently constructed active visual-search benchmarks.
Figures & tables
Figure 1: Comparison of multimodal search task formulations: input-image understanding, explicitly specified visual-search targets, and autonomous acquisition of visual evidence during reasoning.
Figure 2: Overview of the proposed data-synthesis pipeline. The pipeline constructs an interleaved multimodal knowledge graph, samples modality-diverse reasoning paths, and synthesizes questions that obscure shortcuts and visual targets while retaining necessary disambiguating clues and avoiding explicit modality-switch instructions.
Model
VDR
MMSearch
LiveVQA
BC-VL
FVQA
Avg
Direct Reasoning
GPT-5 [ 29 ]
9.8
35.1
44.4
48.6
54.4
38.5
Gemini-2.5-Pro [ 27 ]
8.0
39.8
60.3
43.1
60.7
42.4
Claude-4-Sonnet
2.0
18.7
38.5
29.3
35.3
24.8
Qwen3-VL-8B [ 3 ]
2.4
15.0
41.7
21.8
27.9
21.8
Qwen3-VL-32B [ 3 ]
4.0
23.7
49.7
28.3
35.6
28.3
Table 1: Performance on multimodal knowledge-intensive QA and web-search benchmarks, including MMSearch [ 17 ] , BrowseComp-VL [ 8 ] , VDR-Bench [ 38 ] , FVQA [ 30 ] , and LiveVQA [ 7 ] . Bold and underline mark the best and second-best scores in each column within each group, respectively.
Model
VisBrowse
BrowseComp -V3
ModalSearch
Avg.
IO
SR
Avg.
GPT-5 [ 29 ]
30.2
13.8
66.0
37.8
47.2
30.4
Gemini-2.5-Pro [ 27 ]
23.4
13.5
53.5
29.5
37.5
24.8
GPT-5 (w/ tools)
51.5
26.2
73.0
54.8
60.9
46.2
Gemini-2.5-Pro (w/ tools)
38.5
25.3
59.4
47.3
51.3
38.4
OpenSearch-VL-8B [ 5 ]
11.2
–
15.5
10.9
12.4
–
Table 2: Performance on autonomous cross-modal search benchmarks, including VisBrowse [ 42 ] and BrowseComp-V3 [ 39 ] . The first two rows use direct answering; “w/ tools” denotes tool-augmented evaluation. IO and SR denote Input-only (100 questions) and Search-required (200 questions). Bold and underlined scores mark the best and second-best open-source results. We report the mean accuracy over 5 runs.
Training data
MMSearch
BC-VL
VisBrowse
LiveVQA
ModalSearch
Avg.
Full (16K)
64.3
44.9
28.9
68.3
39.3
49.1
Full (7K subset)
63.0
41.9
26.6
69.1
37.3
47.6
Input-only (7K)
63.6
39.8
21.3
67.4
33.0
45.0
Text-only (7K)
58.7
37.9
19.5
60.3
28.7
41.0
Table 3: Ablation of cross-modal data generation. All 7K variants use only pipeline-generated data; Full (16K) additionally includes FVQA and LiveVQA training examples.
Figure 3: Search-tool usage and agentic framework ablation. (a) Search-tool proportions, with raw invocation counts annotated. Red bars have a fixed width; each blue bar’s width is scaled by the ModalSearcher-to-Qwen total-call ratio within that benchmark. Only t2t_search , t2i_search , and i2i_search are counted; read_url is excluded. (b) Benchmark accuracy under progressively restricted tool settings, including no-tool answering; each legend item pairs the Qwen3-VL-8B (red) and ModalSearcher-8B (blue) colors.
Table 4: Visual-search behavior before and after training on 200 search-required questions with 235 non-initial reference image nodes. All rates are percentages; Δ reports point differences computed before rounding. Alignment and coverage concern search targets rather than returned-image correctness.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Comparison of the data-construction strategies used by our pipeline and prior multimodal search settings. The comparison highlights differences in modality transitions, visual-evidence use, and the degree to which the search target is exposed in the question.
Metric
Training graph
Benchmark graph
Text nodes
35,000
10,000
Image nodes
35,048
10,035
Total nodes
70,048
20,035
Text-to-text edges
53,811
20,386
Text-to-image edges
19,546
5,915
Image-to-text edges
71,154
13,741
Appendix
Table 6: Statistics of the source graphs for training-data and benchmark construction. Unique images combine the original unique and semi-unique categories; non-unique images combine no-unique , wiki_inline , and previously unclassified images.
Stage
Configuration
Value / Setting
Graph construction
Relation-aware textual expansion model
Qwen3-VL-32B
Visual expansion, entity recognition, and linking model
Gemini-2.5-Pro
Image referential-stability assessment model
Gemini-3.5-Flash
Textual neighbor candidate budget Kcand
50
Maximum visual proposals per text node
5
Retrieved images per query Kimg
3
Appendix
Table 7: Models and parameters used in ModalSearch-Pipeline. Answer verification is distinct from benchmark answer judging. Visual verification uses three trials per condition, with and without the reference image.
Hops
SFT
RL
Total
Share
1
4,493
1,412
5,905
30.45%
2
4,721
2,001
6,722
34.67%
3
3,545
1,760
5,305
27.36%
4
761
698
1,459
7.52%
Total
13,520
5,871
19,391
100%
Appendix
Table 8: Reasoning-path length distribution of pipeline-generated training examples. The separately sampled Text-only ablation data and the FVQA and LiveVQA examples added to SFT are excluded. Share is computed over the combined SFT and RL total.
Figure 6: Overview of the pipeline-generated training data. Left: counts and proportions of Input-only examples, whose reference-path visual evidence is provided initially, and Search-required examples, whose reference solutions call for actively acquiring new visual information during reasoning. Right: a word cloud of 70 randomly sampled entities, illustrating the breadth of covered domains. Auxiliary FVQA and LiveVQA training examples and the separately sampled Text-only ablation data are excluded.
Table 13
Figure 7: RL training dynamics over 110 training steps: (a) answer accuracy on the training set, (b) the fraction of rollouts truncated at the maximum interaction-turn limit, and (c) the average number of repeated-search events per rollout. All panels show the same training run (Experiment 2). A repeated search is counted when two queries have cosine similarity above 0.9 and at least three of their five returned results overlap.
Parameter
Value
Parameter
Value
greedy
false
top_p
0.8
top_k
20
temperature
0
repetition_penalty
1.0
presence_penalty
1.5
max-turns
46
max-seq-lens
48,000
Appendix
Table 11: Sampling parameters used for inference.
Model
Accuracy
Positive class
Negative class
Macro-F1
Confusion matrix
Precision
Recall
F1
Precision
Recall
F1
TP
FP
FN
TN
Gemini-3.5-Flash
88.79
82.89
100.00
90.65
100.00
75.47
86.02
88.33
63
13
0
40
GPT-5.4
86.21
84.06
92.06
87.88
89.36
79.25
84.00
85.94
58
11
5
42
Gemini-2.5-Pro
83.62
76.83
100.00
86.90
100.00
64.15
78.16
82.53
63
19
0
34
Claude-Sonnet-4.6
81.03
74.12
100.00
85.14
100.00
58.49
73.81
79.47
63
22
0
31
Appendix
Table 12: Visual-retrievability judgments against human annotations on 116 examples: 63 stably retrievable (positive) and 53 retrieval-ambiguous (negative). Metric scores are percentages; TP, FP, FN, and TN are confusion-matrix counts. Bold marks the best metric score in each column, including ties.
Figure 8: A Search-required example generated by ModalSearch-Pipeline, with an initial image and additional visual evidence to be retrieved.
Figure 9: An Input-only example generated by ModalSearch-Pipeline, with visual evidence confined to the initial input.
Figure 10: A Search-required example generated by ModalSearch-Pipeline, with interleaved textual and visual evidence.
Figure 11: A Search-required example generated by ModalSearch-Pipeline, with no initial image and new visual evidence to be retrieved during reasoning.
Figure 12: System prompt used by the multimodal research agent during inference.
Figure 14: A real ModalSearcher-8B inference trajectory on the Search-required category of ModalSearch-Bench, involving reverse-image search, text retrieval, and visual verification of a historical painting.
Figure 15: A real ModalSearcher-8B inference trajectory on the Search-required category of ModalSearch-Bench, combining textual entity tracing with retrieval and inspection of new visual evidence.