cs.AINov 4, 2026

ModalSearcher: Weaving Visual Evidence into Autonomous Cross-Modal Deep Search

Authors: Zhihan Yin, Ruyi Li, Chengyu Fang, Yifeng Yao, Nonghai Zhang, Pengfei Xiong, Terry Zhang, Huishuai Zhang, 

Organizations: Peking University · Southwest University · Tsinghua University · Independent Researcher · 

Abstract

Static and incomplete parametric knowledge limits the performance of multimodal large language models on novel-knowledge and knowledge-intensive tasks, especially those requiring multi-hop evidence acquisition. Existing multimodal agentic search methods often fix visual evidence at the input or a predefined trajectory position, making it difficult for agents to select search modalities as their information needs evolve and to handle complex real-world queries. We define this capability as autonomous cross-modal search: an agent chooses among direct reasoning, text-to-text search, text-to-image search, and reverse-image search, actively introduces visual evidence when appropriate, and switches between and integrates modalities across multiple hops. We propose ModalSearch-Pipeline, a scalable procedure for constructing an interleaved multimodal knowledge graph. It samples paths with diverse depths and modality compositions and uses backward question construction to obscure entity names and image targets that would expose reasoning shortcuts while retaining necessary disambiguating clues, yielding text-only, image-initialized, and arbitrarily interleaved multi-hop problems. We train the ModalSearcher Agent with supervised fine-tuning and reinforcement learning in a live Web environment, and construct ModalSearch-Bench. Experiments show that ModalSearcher-8B improves average accuracy by 16.2 points over Qwen3-VL-8B under the same agentic workflow across five multimodal knowledge-intensive QA benchmarks, with consistent gains on independently constructed active visual-search benchmarks.

Figures & tables

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Visual-Seeker: Towards Visual-Native Multimodal Agentic Search via Active Visual Reasoning

    Jun 13, 2026Zhengbo Zhang, Changtao Miao, Jinbo Su +10Multimodal IRMultimodal Grounding

  2. MTA-Agent: An Open Recipe for Multimodal Deep Search Agents

    Apr 7, 2026Xiangyu Peng, Can Qin, An Yan +4Multimodal Search AgentsMultimodal QA

  3. VSearcher: Long-Horizon Multimodal Search Agent via Reinforcement Learning

    Mar 3, 2026Ruiyang Zhang, Qianguo Sun, Chao Song +2Multimodal Search AgentsMultimodal Tool Use