RACE: Learning Multimodal Search through Referential Abstraction and Compositional Evidence

Eric Tao; Minhao Sun; Liyan Liu; Zhijing Wu; Xiaohan Yu; Lang Mei; Li Huang; Chao Feng

cs.CVOct 20, 2026

RACE: Learning Multimodal Search through Referential Abstraction and Compositional Evidence

Authors: Eric Tao, Minhao Sun, Liyan Liu, Zhijing Wu, Xiaohan Yu, Lang Mei, Li Huang, Chao Feng,

Organizations: Huawei · Beijing Institute of Technology

Abstract

Training multimodal search agents requires tasks that make visually grounded search necessary for determining the answer. Textual identity clues and unused target evidence allow shortcuts in synthetic questions, weakening supervision for visual search and evidence integration. We propose RACE, an image-conditioned task synthesis framework that constructs dependencies from images to targets and from target-specific evidence to answers. Referential Abstraction (RA) replaces identity-revealing text with specific visual references or ambiguous relational references resolved by searching among depicted candidates. Compositional Evidence (CE) links retrieved facts about multiple entities in one image or entities from different images into joint questions whose answers require every designated contribution. Both operations also apply to images found during search, allowing new visual observations to determine subsequent search targets or supply answer evidence. Fine-tuning Qwen3.5-27B on only 12K teacher-generated trajectories from RACE raises average accuracy across ten multimodal benchmarks from 53.5 to 56.0, a relative improvement of approximately 4.7% over the matched base agent. Our code, data, and model will be publicly available.

Figures & tables

Figure 1: Making visually grounded search necessary. (a) Textual identity clues allow the target to be identified without the image. (b) The question uses the gun-holder’s evidence while leaving the knife-holder’s evidence unused. (c) RA resolves the relational reference and assigns the two candidates’ years to $u$ and $v$ ; CE combines both years to answer the directed comparison $u-v$ . Arrows denote information dependencies.

Figure 2: RACE task synthesis and training. (a) Source annotations or visual analysis and retrieval establish target identities and provenance. (b) RA expresses targets through specific visual or ambiguous relational references. CE combines evidence from multiple targets within or across images. The example compares the years of the actors’ first credited feature-film appearances, yielding $|t_{A}-t_{B}|$ . Evidence exploration and question-only search feedback refine the task. (c) The teacher receives $(\mathcal{I},q)$ and uses tools to solve the task; trajectories with correct final answers are retained for SFT.

Frontier API and large-backbone references
	Multimodal search						Factual generalization
Model / setting	MM- BC	MS+	BC-V 3	VisB	BC-VL	VDR	MM Search	Simple VQA	Live VQA	FVQA	Avg.
Qwen3.5-397B-A17B	46.0	42.1	31.0	52.7	49.4	—	78.3	76.4	85.2	77.1	—
Qwen3.7-Plus	37.1	46.3	42.3	59.8	55.6	30.2	80.7	77.7	84.6	79.7	59.4
Qwen3.8-Max	53.2	59.1	56.3	66.9	—	33.2	84.0	83.8	88.8	85.6	—
Gemini-3.1-Pro	48.7	51.5	50.0	62.1	65.9	40.8	86.1	81.0	87.3	84.0	65.7

Table 1: Search accuracy and factual generalization (%,

\uparrow

). Results cover four primary search benchmarks and six supplementary benchmarks. Models are grouped by role. The final block compares Qwen3.5-27B before and after SFT under matched settings. Avg. is the unweighted mean over all ten benchmarks.

Figure 4: Training trajectory distributions and behavior. (a) Visual calls per trajectory. (b) Proportion of trajectories containing a visual-web-visual pattern: a visual call, web search or page fetching, and another visual call in serialized order, with intervening calls allowed. (c) The proportion of trajectories with visual calls after the first tool round and mean number of distinct tool types. Colors are standardized within each metric, and the blue outline marks Ours. Abbreviations: OS-VL, OpenSearch-VL; VDR, Vision-DeepResearch; RS-MM, REDSearcher-MM.

Figure 5: Evaluation-time search behavior before and after SFT. Gray and blue bars give base-agent and SFT-agent point estimates, respectively. (a) Mean tool calls per trajectory. (b) Mean within-trajectory fraction of tool calls that use visual tools. (c) Fraction of trajectories that use visual tools after the first tool round. Panels (b) and (c) show percentages. Abbreviations: MM-BC, MM-BrowseComp; VisB, VisBrowse-Bench; BC-V3, BrowseComp-V 3 ; MS+, MMSearch-Plus.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Field	Specification
Teacher and student	Qwen3.7-Plus and Qwen3.5-27B
Main training data	12K synthesized trajectories only
Training stage and supervision	One SFT stage over assistant tool calls and responses
Trajectory-run decoding settings	temperature = 0.7 , top-p = 0.9
Trajectory-run iteration limits	MM-BC (100), MS+ (20), BC-V 3 (20), VisB (15)
Primary benchmark questions	MM-BC (224), MS+ (311), BC-V 3 (300), VisB (169), using full sets

Appendix

Table 3: Training setup and archived evaluation settings. Decoding settings and iteration limits refer to the archived runs used for behavioral analysis.

Table 8

	Post-first-round visual use		Visual-search -fetch
Input images	Base	Trained	Base	Trained
1	22.5	34.2	42.3	83.3
2	27.8	51.9	44.4	98.1
3+	31.4	37.1	34.3	88.6

Appendix

Table 6: MMSearch-Plus search behavior (%). Base versus trained by input-image count.

Figure 6: Paired differences across task strata. SFT minus base-agent trajectory proportions, in points, for the question pairs in Tables 5 - 6 . Error bars are pointwise paired-bootstrap 95% intervals. The horizontal line marks zero.

Figure 7: Continued visual use across interaction lengths. Bars give the proportion of trajectories with a visual-tool call after the first round, within each round-count range. All panels share a scale. Blue bars denote Ours, and blue annotations give the difference from the highest baseline rate.

Figure 8: Supplementary visual-span statistics for training trajectories. (a) Proportion of trajectories whose normalized visual span exceeds each threshold

x

; higher curves indicate more trajectories with widely separated visual calls. (b) Mean normalized visual span for each source. Both panels summarize trajectories with at least one visual call; a single visual call has span zero. Baseline names follow Table 1 .

Construction	Target configuration	Required behavior
Single-target abstraction	One target among the candidates in an image	Resolve a specific visual or ambiguous relational reference, then retrieve the requested information.
Within-image composition	Multiple required targets in one image	Retrieve evidence for each target and derive an answer requiring all designated contributions.
Cross-image composition	One required target per image across multiple images	Resolve each image’s target and combine its evidence with the other images’ contributions.

Appendix

Table 7: Task configurations. Referential Abstraction constructs target references, and Compositional Evidence adds joint information requirements. The same configurations apply to targets in retrieved images.

Multimodal Search Agents

Multimodal search agents retrieve and reason across text, images, video, and other sources to answer complex questions.

See all 47 papers

MTA-Agent: An Open Recipe for Multimodal Deep Search Agents
Apr 7, 2026Xiangyu Peng, Can Qin, An Yan +4Multimodal Search Agents Multimodal QA
Salesforce AI Research
ProMMSearchAgent: A Generalizable Multimodal Search Agent Trained with Process-Oriented Rewards
Apr 22, 2026Wentao Yan, Shengqin Wang, Huichi Zhou +4Multimodal IR Process Reward Models
East China Normal University · Huawei Noah’s Ark Lab · University College London +2
Visual-Seeker: Towards Visual-Native Multimodal Agentic Search via Active Visual Reasoning
Jun 13, 2026Zhengbo Zhang, Changtao Miao, Jinbo Su +10Multimodal IR Multimodal Grounding
School of Artificial Intelligence, UCAS · Institute of Automation, CAS · Ant Digital Technologies, Ant Group +2

Apr 7, 2026J/K move · Enter open · S save

MTA-Agent: An Open Recipe for Multimodal Deep Search Agents

Xiangyu Peng, Can Qin, An Yan, Xinyi Yang+3

Salesforce AI Research

Multimodal Search Agents Multimodal QA Multimodal Reasoning Multi-Hop QA Tool Use in VLMs

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) have demonstrated strong capabilities in visual understanding, yet they remain limited in complex, multi-step reasoning that requires deep searching and integrating visual evidence with external knowledge. In this work, we address this challenge by constructing high-quality, verified multi-hop vision-language training data for multimodal deep-search agents. We propose a Multi-hop Tool-Augmented Agent for Evidence-based QA Synthesis (MTA-Agent), which automatically selects tools and their parameters to retrieve and validate evidence from both visual and textual sources and generates structured multi-hop question-answer trajectories. Starting from diverse VQA seed datasets, our pipeline produces a large-scale training dataset, MTA-Vision-DeepSearch, containing 21K high-quality multi-hop examples. The data is filtered through a multi-stage verification process to ensure factual consistency and answer uniqueness. Using MTA-Vision-DeepSearch, a 32B open-source multimodal search agent achieves state-of-the-art performance, reaching an average of 54.63% across six challenging benchmarks, outperforming GPT-5 (51.86%), Gemini-2.5-Pro (50.98%), and Gemini-3-Pro (54.46%) under the same tool settings. We further show that training on our data improves both reasoning depth and tool-use behavior, increasing the average number of steps from 2.27 to 4.28, and leading to more systematic and persistent search strategies. Additionally, we demonstrate that training can be performed without real-time tool calls by replaying cached interactions, significantly reducing training cost. Importantly, we present MTA-Agent as a fully open recipe for multimodal deep search: we release the entire dataset, training trajectories, and implementation details to enable reproducibility and future research on open multimodal search agents.

Abstract

Multimodal Search Agents

Explore similar work

MTA-Agent: An Open Recipe for Multimodal Deep Search Agents

ProMMSearchAgent: A Generalizable Multimodal Search Agent Trained with Process-Oriented Rewards

Visual-Seeker: Towards Visual-Native Multimodal Agentic Search via Active Visual Reasoning