LiteSearch-VL: Small Multimodal Search Agents via Trajectory Distillation and Synthetic Step-DPO

Saeed Khaki; Nima Safaei; Kamal Ginotra

cs.AIAug 29, 2026

LiteSearch-VL: Small Multimodal Search Agents via Trajectory Distillation and Synthetic Step-DPO

Authors: Saeed Khaki, Nima Safaei, Kamal Ginotra

Organizations: Microsoft AI · Ohio State University

Abstract

Multimodal search agents answer visual questions by interleaving image understanding, web retrieval, tool use, and evidence synthesis. Strong systems exist, but in two expensive regimes: proprietary frontier models such as GPT-5 and Gemini, or large open vision-language backbones trained with substantial agentic data and reinforcement learning. We ask a different question: when released agent trajectories are distilled into much smaller backbones under a single-node budget, what is actually transferred? We study this with LiteSearch-VL, a low-compute recipe for Qwen3-VL-2B and Qwen3-VL-4B that uses only released OpenSearch-VL trajectories, parameter-efficient LoRA adapters, and synthetic step-level preferences: DPO on GPT-5-generated hard negatives targeting five local failure modes (premature answer, wrong tool, weak query, repeated query, ignored image). Across 12,400 GPT-5-judged rollouts on SimpleVQA, FVQA, LiveVQA, and VDR-Bench-testmini, the dominant effect is behavioral rather than a uniform accuracy lift: full-trajectory supervised fine-tuning transfers the agent contract, taking the 2B model from almost never emitting a usable answer (1,237/1,240 no_answer rollouts) to 28.4% macro Pass@1, matching or slightly exceeding the off-the-shelf 4B base (25.6%). Synthetic preference learning and compact tool distillation act as refinements rather than phase transitions (best 4B configuration: 30.8% macro Pass@1). Finally, a controlled VDR step-budget ablation shows that extra search turns convert abstentions into wrong_entity errors rather than correct answers, identifying answer verification, not search depth, as the next bottleneck for small multimodal agents.

Figures & tables

Panel of Table 1 — Table 1: Pass@1 (%) across the four short-answer benchmarks for ten model cells. Bold : best in each column; underline : best at 2B and 4B separately; Avg : macro-average over the four benchmarks. Cells are exact $n_{\text{correct}}/n$ counts from the GPT-5 judge (Section 3 ); $n=$ 500, 59, 181, 500 for the four columns.

Model	Pass@1 (%)		Partial-credit (%)		no_answer		$\Delta$ Pass@1
Model	5 steps	8 steps	5 steps	8 steps	5 steps	8 steps	$\Delta$ Pass@1
2B SFT	2.0	2.4	5.2	7.5	304	171	+0.4
2B SFT + DPO	1.0	1.8	5.6	8.0	297	148	+0.8
4B SFT	2.6	2.4	6.4	8.3	351	236	$-$ 0.2
4B SFT + DPO	2.4	3.4	7.0	8.9	327	213	+1.0

Table 2: VDR step-budget ablation (

n=500

each). Raising MAX_STEPS from 5 to 8 cuts no_answer by 33–50% on every model; net Pass@1 moves within

\pm 1

point, and both DPO checkpoints improve (2B +0.8, 4B +1.0).

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Overview of LiteSearch-VL. The pipeline reuses the released SearchVL-SFT-36k corpus to drive three model families (full-trajectory SFT, 3-tool distilled SFT, and SFT+DPO with mode-conditioned synthetic negatives). Every trained checkpoint is then evaluated against the same seven-tool agent environment, with GPT-5 adjudicating each rollout under a fixed five-field rubric.

Domain	Records	Train	Val	Step examples	Median turns
livevqa	13,326	13,145	181	47,892	3
wiki_art	5,093	5,023	70	19,027	4
fvqa	4,413	4,353	60	14,628	3
webqa	3,804	3,752	52	13,845	3
wiki_en	3,503	3,455	48	13,208	4
wiki_zh	3,480	3,432	48	13,210	4

Appendix

Table 3: Composition of the SearchVL-SFT-36k corpus after preprocessing. Median turns is the per-trajectory median across the train split.

Figure 3: Distribution of validated synthetic preference pairs across the five hard-negative failure modes. Each step record can contribute up to one pair per applicable mode.

Recipe	Backbones	Training stages	Reported training allocation
OpenSearch-VL	Qwen3-VL-8B / 30B-A3B / 32B	Agentic SFT + multi-turn fatal-aware GRPO	SFT: 256 H20 GPUs for 2–4 days; GRPO: 64 H20 GPUs for 10 days
LiteSearch-VL	Qwen3-VL-2B / 4B	LoRA SFT + synthetic step-level DPO; no on-policy RL	One eight-GPU node for each training stage

Appendix

Table 4: Compute footprint relative to the OpenSearch-VL training recipe. OpenSearch-VL allocations are those reported in [ 3 ] ; LiteSearch-VL reports the maximum concurrent training allocation used by our runs. The comparison is intended to clarify regime, not to normalize for hardware throughput.

Model	SimpleVQA	FVQA	LiveVQA	VDR	Avg
2B Base	0.2	0.0	0.0	0.0	0.1
2B SFT	33.7	78.0	24.6	5.2	35.4
2B Step-SFT	26.4	72.9	24.6	3.8	31.9
2B 3-tool SFT	20.5	54.2	16.6	3.3	23.7
2B SFT + DPO	32.7	78.0	25.4	5.6	35.4
4B Base	26.2	64.4	20.7	3.1	28.6

Appendix

Table 5: Partial-credit rate (%), computed as

\text{Pass@1}+0.5\cdot\text{partial}

, captures “right entity, wrong sub-attribute” answers that strict Pass@1 misses. The relative ranking of models is similar, but the 4B+DPO checkpoint is the clear winner on the macro average. Note: the implied partial-response counts from this table (via

2\times(\text{partial-credit}-\text{Pass@1})\times n

) can differ from the discrete error_mode=partial tally in Table 6 by exactly one row for two cells (2B 3-tool SFT: 68 vs. 67; 4B SFT: 105 vs. 104). This traces to judged rows where GPT-5 set partially_correct=1 but labeled error_mode as something else (e.g. hallucinated , wrong_entity ), breaking the intended mutual exclusivity between the two signals: two of the 12,400 main-evaluation rows (one per affected cell), plus a third such row in the VDR step-budget ablation of Section 4 (3 of all 14,400 judgments). It does not affect Pass@1, partial-credit rates, or any reported ranking.

Model	none $\uparrow$	wrong_entity	no_answer	partial	other
2B Base	1	2	1,237	0	0
2B SFT	232	383	514	106	5
2B Step-SFT	202	184	780	73	1
2B 3-tool SFT	147	450	572	67	4
2B SFT + DPO	231	394	505	105	5
4B Base	191	343	632	62	12

Appendix

Table 6: Error-mode distribution per model. Counts are over all

n=1{,}240

judged questions (sum of the four benchmarks). none denotes the correct bucket, partial is the partial-credit bucket, and the other modes are mutually exclusive failure types from the GPT-5 rubric (Section 3 ). The 2B base model produces almost no final answers, which is the failure pattern SFT primarily addresses. Note: the partial column counts rows with the discrete label error_mode=partial , whereas Table 5 is computed from the partially_correct flag; for 2B 3-tool SFT and 4B SFT one judged row each sets partially_correct=1 while labeling error_mode differently (two of the 12,400 main-evaluation rows; a third such row occurs in the VDR step-budget ablation), so those two rows sum to one less than Table 5 implies (67 vs. 68 and 104 vs. 105, respectively). See the note on Table 5 .

Model	Avg Pass@1 (%)	95% CI (%)
2B Base	0.1	[0.0, 0.1]
2B SFT	28.4	[24.9, 31.9]
2B Step-SFT	26.9	[23.3, 30.5]
2B 3-tool SFT	19.5	[16.0, 23.0]
2B SFT + DPO	29.4	[26.0, 32.8]
4B Base	25.6	[22.1, 29.0]

Appendix

Table 7: Macro Pass@1 uncertainty from the existing question-level GPT-5 judgments. For each model, we bootstrap questions within each benchmark independently and recompute the four-benchmark macro average; intervals are percentile 95% CIs over 10,000 bootstrap replicates.

Panel of Figure 4 — Figure 4: Two views of the main result: the absolute gain attributable to each training stage (left), and the average Pass@1 across regimes for the 2B and 4B backbones (right).

Figure 5: Stacked error-mode distribution per model. The dominant qualitative effect of SFT is collapsing the no_answer pile of the base model into a roughly even split of correct and wrong_entity . DPO further reduces no_answer on 4B (587 $\to$ 511) and increases correct (248 $\to$ 254).

Figure 6: VDR step-budget ablation. The extra step budget reduces no_answer by roughly one third to one half on every model (right axis, solid lines), but the additional rollouts mostly produce wrong_entity , not correct . Net Pass@1 movement is within $\pm 1$ point on all four models.

Stage	Data	Adapter	Precision	$L_{\max}$	LR	$B_{\text{eff}}$	Steps
2B SFT	$\sim$ 36K traj.	LoRA-16	fp16	8,192	$1\!\times\!10^{-4}$	64	564
4B SFT	$\sim$ 36K traj.	LoRA-16	fp16	8,192	$1\!\times\!10^{-4}$	64	564
2B/4B Step-SFT	$\sim$ 36K steps	LoRA-16	fp16	8,192	$1\!\times\!10^{-4}$	64	564
2B/4B 3-tool SFT	$\sim$ 33K traj.	LoRA-16	fp16	6,144	$1\!\times\!10^{-4}$	64	519
2B/4B DPO	$\sim$ 234K pairs	LoRA-16	bf16	5,120	$1\!\times\!10^{-6}$	64	3,663

Appendix

Table 8: Training hyperparameters. “

L_{\max}

” is the model context length in tokens, “

B_{\text{eff}}

” is the effective batch size, and “Steps” is the total number of optimizer steps in one epoch.

Tool	Backend	Observation format
text_search	SerpAPI google	Passage blocks (title / URL / summary)
web_search	alias of text_search	same as text_search
image_search	SerpAPI google_lens	Python-repr list of {title, source}
crop	Pillow	New image id + storage URL
layout_parsing	docTR OCR	Text blocks + recognized text
sharpen	Pillow unsharp mask	Same template as crop

Appendix

Table 9: The agent’s tool environment at inference time (seven distinct tools; web_search is an alias of text_search ). The 3-tool distilled variant (Section 2 ) restricts the action space to image_search , text_search , and crop .

Domain	Val records	Used as benchmark?
livevqa	181	yes (LiveVQA track)
wiki_art	70	no
fvqa	60	yes (FVQA track)
webqa	52	no
wiki_en	48	no
wiki_zh	48	no

Appendix

Table 10: Per-domain composition of the held-out 500-record validation split. The fvqa and livevqa subsets are used as the two corresponding benchmark tracks in the main evaluation; the other five domains are kept for ablation but not reported in the main table. The FVQA track evaluates 59 of the 60 fvqa records: one record is excluded because its image lacks a public URL required by image_search .

Figure 7: Image for Case K.1 (FVQA fvqa_002606 ).

Figure 8: Image for Case K.2 (FVQA fvqa_004100 ).

Figure 9: Image for Case K.3 (SimpleVQA simplevqa_1057 ).

Figure 10: Image for Case K.4 (FVQA fvqa_004018 ).

Figure 11: Image for Case K.5 (VDR vdr_base_Nature_833 ).

Multimodal Search Agents

Multimodal search agents retrieve and reason across text, images, video, and other sources to answer complex questions.

See all 47 papers

Jun 30, 2026

SimpleSearch-VL: A Simple Recipe for Multimodal Agentic Deep Search

We present SimpleSearch-VL, an efficient, reliable, and practical framework for multimodal agentic search. Its core idea is to improve the agent's own search-and-verification process rather than scaling data, tools, or auxiliary model components. For efficiency, Factorized Adaptive Rollout (FAR) improves sampling efficiency by forming more informative training groups while using redundant samples to mitigate long-tail latency and expose hard samples. For reliability, SimpleSearch-VL performs evidence-verified reasoning, explicitly using chain-of-thought verification to assess the relevance of retrieved visual and textual cues to the original context. For practicality, SimpleSearch-VL keeps a lightweight tool interface and performs webpage self-summary within the agent, requiring no additional external dependencies. With only 5K supervised tool-interleaved trajectories and 2K RL data, SimpleSearch-VL improves Qwen3-VL agentic baselines by 15.8 and 16.0 average points for the 8B and 30B-A3B variants, respectively. The SimpleSearch-VL-30B-A3B model further achieves performance competitive with agentic Gemini-3-Pro.

Ming Dai, Zhihong Lu, Jinjie Gu +5

Southeast University · Ant Group

Tool-Augmented Language Model Agents Agentic Search Multimodal Search Agents Agentic Retrieval Tool-Using Vision-Language Agents VLM Reasoning+1

Ask PDF

May 6, 2026

OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents

Deep search has become a crucial capability for frontier multimodal agents, enabling models to solve complex questions through active search, evidence verification, and multi-step reasoning. Despite rapid progress, top-tier multimodal search agents remain difficult to reproduce, largely due to the absence of open high-quality training data, transparent trajectory synthesis pipelines, or detailed training recipes. To this end, we introduce OpenSearch-VL, a fully open-source recipe for training frontier multimodal deep search agents with agentic reinforcement learning. First, we curated a dedicated pipeline to construct high-quality training data through Wikipedia path sampling, fuzzy entity rewriting, and source-anchor visual grounding, which jointly reduce shortcuts and one-step retrieval collapse. Based on this pipeline, we curate two training datasets, SearchVL-SFT-36k for SFT and SearchVL-RL-8k for RL. Besides, we design a diverse tool environment that unifies text search, image search, OCR, cropping, sharpening, super-resolution, and perspective correction, enabling agents to combine active perception with external knowledge acquisition. Finally, we propose a multi-turn fatal-aware GRPO training algorithm that handles cascading tool failures by masking post-failure tokens while preserving useful pre-failure reasoning through one-sided advantage clamping. Built on this recipe, OpenSearch-VL delivers substantial performance gains, with over 10-point average improvements across seven benchmarks, and achieves results comparable to proprietary commercial models on several tasks. We will release all data, code, and models to support open research on multimodal deep search agents.

Shuang Chen, Kaituo Feng, Hangting Chen +7

Tencent Hunyuan · University of California, Los Angeles · The Chinese University of Hong Kong +1

Multimodal IR Multimodal Search Agents Multimodal Agents RL for LLM Agents Multimodal Tool Use

Ask PDF

Apr 7, 2026

MTA-Agent: An Open Recipe for Multimodal Deep Search Agents

Multimodal large language models (MLLMs) have demonstrated strong capabilities in visual understanding, yet they remain limited in complex, multi-step reasoning that requires deep searching and integrating visual evidence with external knowledge. In this work, we address this challenge by constructing high-quality, verified multi-hop vision-language training data for multimodal deep-search agents. We propose a Multi-hop Tool-Augmented Agent for Evidence-based QA Synthesis (MTA-Agent), which automatically selects tools and their parameters to retrieve and validate evidence from both visual and textual sources and generates structured multi-hop question-answer trajectories. Starting from diverse VQA seed datasets, our pipeline produces a large-scale training dataset, MTA-Vision-DeepSearch, containing 21K high-quality multi-hop examples. The data is filtered through a multi-stage verification process to ensure factual consistency and answer uniqueness. Using MTA-Vision-DeepSearch, a 32B open-source multimodal search agent achieves state-of-the-art performance, reaching an average of 54.63% across six challenging benchmarks, outperforming GPT-5 (51.86%), Gemini-2.5-Pro (50.98%), and Gemini-3-Pro (54.46%) under the same tool settings. We further show that training on our data improves both reasoning depth and tool-use behavior, increasing the average number of steps from 2.27 to 4.28, and leading to more systematic and persistent search strategies. Additionally, we demonstrate that training can be performed without real-time tool calls by replaying cached interactions, significantly reducing training cost. Importantly, we present MTA-Agent as a fully open recipe for multimodal deep search: we release the entire dataset, training trajectories, and implementation details to enable reproducibility and future research on open multimodal search agents.

Xiangyu Peng, Can Qin, An Yan +4

Salesforce AI Research

Multimodal Search Agents Multimodal QA Multimodal Reasoning Multi-Hop QA Tool Use in VLMs

Ask PDF

Abstract

Multimodal Search Agents

Explore similar work

SimpleSearch-VL: A Simple Recipe for Multimodal Agentic Deep Search

OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents

MTA-Agent: An Open Recipe for Multimodal Deep Search Agents