cs.CVApr 14, 2026
SaveTowards Long-horizon Agentic Multimodal Search
Organizations: Gaoling School of Artificial Intelligence, Renmin University of China. · City University of Hong Kong.
Abstract
Multimodal deep search agents have shown great potential in solving complex tasks by iteratively collecting textual and visual evidence. However, managing the heterogeneous information and high token costs associated with multimodal inputs over long horizons remains a critical challenge, as existing methods often suffer from context explosion or the loss of crucial visual signals. To address this, we propose a novel Long-horizon MultiModal deep search framework, named LMM-Searcher, centered on a file-based visual representation mechanism. By offloading visual assets to an external file system and mapping them to lightweight textual identifiers (UIDs), our approach mitigates context overhead while preserving multimodal information for future access. We equip the agent with a tailored fetch-image tool, enabling a progressive, on-demand visual loading strategy for active perception. Furthermore, we introduce a data synthesis pipeline designed to generate queries requiring complex cross-modal multi-hop reasoning. Using this pipeline, we distill 12K high-quality trajectories to fine-tune Qwen3-VL-Thinking-30A3B into a specialized multimodal deep search agent. Extensive experiments across four benchmarks demonstrate that our method successfully scales to 100-turn search horizons, achieving state-of-the-art performance among open-source models on challenging long-horizon benchmarks like MM-BrowseComp and MMSearch-Plus, while also exhibiting strong generalizability across different base models. Our code will be released in https://github.com/RUCAIBox/LMM-Searcher.
Figures & tables
Figure 1: An illustration of LMM-Searcher. For simplicity, we employ simple strings as uids in this figure. In real implementation, we use URLs. This figure serves solely as a functional demonstration and does not represent any actual search results.
Figure 2: The webpage returned to the agent. Its content is reorganized into a structured representation, where textual information is summarized into key bullet points and visual elements are converted into image–caption pairs, where the images are replaced with their URLs.
Figure 3: Overview of automated Visual Question-Answer (VQA) data synthesis pipeline. The pipeline constructs multimodal deep search data by synthesizing VQA pairs based on multimodal webpages and subsequently extending the reasoning chain.
| Dataset | FVQA | LiveVQA | REDSearcher-MM | REDSearcher-Text | Ours |
| Num. of Samples | 2301 | 1672 | 3366 | 3808 | 1589 |
| Avg. Turns | 5.63 | 7.16 | 13.21 | 21.97 | 17.26 |
Table 1: Dataset statistics, including the number of samples and the average number of tool-use turns per sample.
| Model | MMBC | MMSearch+ | VisBrowse | MMSearch | Avg. |
| Direct Answer | |||||
| GPT-5 | 10.3 | 19.1 | 26.0 | 33.3 | 22.2 |
| Seed-1.8 | 13.0 | 8.6 | 18.9 | 31.0 | 17.9 |
| Kimi-K2.5 | 2.7 | 12.6 | 18.3 | 47.0 | 20.2 |
| Gemini-2.5-Pro | 10.3 | 14.5 | 27.2 | 39.8 | 23.0 |
| Gemini-2.5-Flash | 5.4 | 8.1 | 16.0 | 30.4 | 15.0 |
Table 2: The performance comparison between our model and baseline methods. The performance marked with ∗ is evaluated with 100 turns and the context management technique. MMBC shorts for MM-BrowseComp, and MMSearch+ shorts for MMSearch-Plus.
| Model | Evaluation Method | MMBC | MMSearch+ | VisBrowse | MMSearch |
| GPT-5 | Direct Answer | 10.3 | 19.1 | 26.0 | 33.3 |
| w. Previous Framework | - | 17.2 | - | 63.7 | |
| w. Our Framework | 36.5 | 34.8 | 35.5 | 72.2 | |
| Seed-1.8 | Direct Answer | 13.0 | 8.6 | 18.9 | 31.0 |
| w. Previous Framework | 21.4 | 11.0 | - | 69.7 | |
| w. Our Framework | 35.1 | 46.7 | 58.0 | 73.2 |
Table 3: Performance comparison among different frameworks.
Figure 4: Tool call distribution of the training data. Our synthesized data require diverse types of tool calls and a higher proportion of “visual search” and “fetch image”.
Figure 5: Interactive scaling results. We evaluate different models with our context management strategy and force the agent to stop at 100 turns. For each interaction-turn threshold on the x-axis, we count a sample as successful if the agent has autonomously terminated and produced the correct answer within that number of turns, and compute the corresponding accuracy accordingly.
| MMBC | MMSearch+ | VisBrowse | MMSearch | Avg. | |
| Qwen3-VL-30B-A3B-Thinking | 9.8 | 14.4 | 16.0 | 62.0 | 25.6 |
| + Open-source Visual Query | 20.7 | 33.8 | 39.5 | 70.4 | 41.1 |
| + Open-source Textual Query | 21.6 | 32.4 | 39.1 | 70.3 | 40.9 |
| + Our Synthesized Query | 22.3 | 32.9 | 42.0 | 71.0 | 42.1 |
Table 4: Data ablation results. Each dataset is incrementally added on top of the previous training set.
| MMBC | MMSearch+ | VisBrowse | MMSearch | Avg. | |
| Seed-1.8 | 9.8 | 14.4 | 16.0 | 62.0 | 25.6 |
| w/ fetch-image tool | 35.1 | 46.7 | 58.0 | 73.2 | 53.3 |
| wo/ fetch-image tool | 29.5 | 43.7 | 48.5 | 71.0 | 48.2 |
| -5.6 | -3.0 | -9.5 | -2.2 | -5.1 |
Table 5: Tool ablation results. “w/ fetch-image” represents equipping the agent with a full tool interface, while “wo/ fetch-image tool” represents removing the fetch-image tool.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Detailed illustration of key steps within the model’s search trajectory.
| Examples of our LMM-Searcher on VisBrowse-Bench (Part 1/4) Data Source: VisBrowse-Bench Problem: The image on the beverage is the logo of a food company that acquired a newspaper company in 2009. How many trees are there in the newspaper’s Wikipedia poster? Image URL: [http://rand-oss//2F%2F9bXW.jpg] Description: [Original input image] Ground Truth: 5 Iteration 1 < think > First step is to identify the food company logo in the beverage image. Let’s use visual search on the beverage (the blue carton with the cartoon character). < /think > < tool_call > {"name": "tool-google-search-visual_search", "arguments": {"image_url": "http://rand-oss//2FP2F9bXW"}} < /tool_call > < tool_response > 1. | Title: Giay tu cong bo san pham FROZEN COLA | Image URL: https://wantwant.vn/vnt..| Source: https://wantwant.vn/cn/gi.. 2. | Title: Facts For Kids- France | Image URL: https://www.oocities.org/… | Source: https://www.oocities.org/… < /tool_response > Iteration 2 < think > The visual search didn’t directly identify the logo. Let’s try zooming in on the blue beverage carton part of the image, since that’s the logo area. Let’s find the coordinates of the blue carton. Looking at the image, the blue carton is on the right side. Let’s define the region: x around 670, y around 250, width around 300, height around 400. Let’s zoom in on that area. < /think > < tool_call > { "name": "tool-image-processing-zoom_in", "arguments": {"image_url": "http://rand-oss/…"} } < /tool_call > < tool_response > [{"error": "Your cropped region extends beyond image bounds (image size: 900x900, your cropped region: (670+300)x(250+400))"}] < /tool_response > Iteration 3 < think > I need to adjust the zoom region to fit within the 900x900 image. Let’s calculate: max x + width <=900, max y + height <=900. So if x=670, width can be up to 230 (900-670). If y=250, height can be up to 650 (900-250). Let’s use x=670, y=250, width=220, height=600. That should fit. < /think > < tool_call > {"name": "tool-image-processing-zoom_in", "arguments": {"image_url": "http://rand-oss/…"}} < /tool_call > < tool_response > [SUCCESS]: Zoomed in on region (670, 250, 220, 600). Image dimensions: 220x600. Image URL: http://rand-oss/…, Description: [Zoomed region: (670, 250, 220, 600) from original image] < /tool_response > |
Table 6: A case study illustrating the complete search trajectory of the model.
| Examples of our LMM-Searcher on VisBrowse-Bench (Part 2/4) Iteration 4 < think > Now I can see the logo clearly: it’s Want Want’s mascot (the boy with the red cheeks). Want Want is a food company. The question says this company acquired a newspaper company in 2009. Let me confirm: Want Want acquired a newspaper? Let me search for "Want Want acquired newspaper 2009". < /think > < tool_call > {"name": "tool-google-search-google_search", "arguments": "{"q": "Want Want acquired newspaper 2009"}"} < /tool_call > < tool_response > 1. "title": "China Times Group is sold to Want Want", "link": "https://www.taipeitimes.com/News/biz/…", "snippet": "It owns the Chinese-language newspapers China Times…" 2. "title": "China Times Joins a Snack-food Empire|Industry|2009-03-05", "link": "https://english.cw.com.tw/article/…", "snippet": "A month after Want Want Holdings chairman Tsai Eng-meng …" …… < /tool_response > Iteration 5 < think > So the newspaper company is China Times Group ( 中时集团), acquired by Want Want in 2008 (the deal closed in 2009? The news says completed in March 2009). Now, the question asks: "How many trees are there in the newspaper’s Wikipedia poster?" … Let me search for the Wikipedia page of the newspaper acquired, then check the image (poster) for tree count. First, let’s get the Wikipedia page for China Times Group. Let me search for "China Times Group Wikipedia" or " 旺旺中时媒体集团Wikipedia" (since it’s in Chinese). < /think > < tool_call > { "name": "tool-google-search-google_search", "arguments": "{"q": "Want Want acquired newspaper 2009"}" } < /tool_call > < tool_response > 1. "title": " 旺旺中时媒体集团 - 维基百科", "link": "https://zh.wikipedia.org/zh-cn/…" 2. "title": " 可靠来源/常见/旺旺中时媒体集团 - 维基百科", "link": "https://zh.wikipedia.org/zh-cn/…" …… < /tool_response > Iteration 6 < think > Now, let’s open the Wikipedia page for 旺旺中时媒体集团(https://zh.wikipedia.org/zh-cn/ 旺旺中时媒体集团). We need to find the "poster" (infobox image) and count the number of trees in it. Let’s use the scrape_website tool to get the content of this page, … Let’s call the scrape_and_extract_info tool with the URL and ask to count the trees in the Wikipedia poster (the infobox image). < /think > < tool_call > { "name": "jina_scrape_llm_summary-scrape_and_extract_info", "arguments": "{"url": "https://zh.wikipedia.org/zh-cn/…", "info_to_extract": "Count the number of trees in the Wikipedia poster of 旺旺中时媒体集团"}" } < /tool_call > < tool_response > {"success": false, "url": "https://zh.wikipedia.org/zh-cn/ 旺旺中时媒体集团", "extracted_info": "", "error": "Scraping failed (both Jina and Python): Python Scrape: Unexpected error occurred: Client error ’403 Forbidden’ for url ’https://zh.wikipedia.org/zh-cn/…"} < /tool_response > Iteration 7 < think > 403 Forbidden error. Let’s try a different approach. Let’s first find the infobox image URL for this Wikipedia page. … Let’s search for " 旺旺中时媒体集团 维基百科 图片". < /think > < tool_call > { "name": "tool-google-search-image_search", "arguments": "{"q": " 旺旺中时媒体集团 维基百科 图片"}" } < /tool_call > < tool_response > 1. | Title: 旺旺中时媒体集团- 维基百科，自由的百科全书| Image URL: https://upload.wikimedia.org/wikipedia/zh/thumb/0/05/… | Source: https://zh.wikipedia.org/zh-tw/… 2. | Title: 旺旺集团- 维基百科，自由的百科全书| Image URL: https://upload.wikimedia.org/wikipedia/commons/7/74/… | Source: https://zh.wikipedia.org/zh-tw/… < /tool_response > |
| Examples of our LMM-Searcher on VisBrowse-Bench (Part 3/4) Iteration 8 < think > Wait, the first image is the logo of Want Want China Times Media Group. Let’s check that image. The logo has the Want Want kid, " 中时集团" in red squares, and "CHINA TIMES GROUP". Is there a tree in this logo? Let’s fetch this image and analyze it. < /think > < tool_call > { "name": "tool-fetch-image-fetch_image", "arguments": "{"url": "https://upload.wikimedia.org/wikipedia/zh/thumb/0/05/…"}" } < /tool_call > < tool_response > Image downloaded from: https://upload.wikimedia.org/wikipedia/.. < /tool_response > Iteration 9 < think > Looking at this logo, I don’t see any trees. Wait, maybe the "newspaper’s Wikipedia poster" refers to the China Times (China Times Group) poster, not the media group’s poster. Let’s check the Wikipedia page for China Times Group ( 中国时报集团). Let’s search for " 中国时报集团 维基百科" and get its poster. Alternatively, maybe the "poster" is the Wikipedia infobox image for China Times Group. Let’s search for the China Times Group Wikipedia page. Let me call the google search for " 中国时报集团 维基百科". < /think > < tool_call > { "name": "tool-google-search-google_search", "arguments": "{"q": " 中国时报集团 维基百科"}" } < /tool_call > < tool_response > 1. "title": " 讨论:中国时报集团- 维基百科，自由的百科全书", "link": "https://zh.wikipedia.org/zh-my/…" 2. "title": " 中时新闻网", "link": "https://www.wikiwand.com/zh-hant/…" < /tool_response > Iteration 10 < think > Wait, maybe the Wikipedia page for China Times Group is under a different name? Let me try " 中时集团 维基百科". < /think > < tool_call > { "name": "tool-google-search-google_search", "arguments": "{"q": " 中时集团 维基百科"}" } < /tool_call > < tool_response > 1. "title": " 中国时报集团- 维基百科，自由的百科全书", "link": "https://zh.wikipedia.org/zh-cn/…" 2. "title": " 旺旺中时媒体集团 - 维基百科", "link": "https://zh.wikipedia.org/zh-cn/…" < /tool_response > Iteration 11 < think > Okay, so the newspaper company is China Times Group ( 中国时报集团), which was acquired by Want Want in 2008-2009. Now, the question is asking for the number of trees in the newspaper’s Wikipedia poster. Let’s find the Wikipedia poster (infobox image) for China Times Group. Let’s scrape the Wikipedia page for China Times Group and look for the infobox image. Let’s use the jina_scrape tool on that page, asking to find the infobox image and count the trees in it. < /think > < tool_call > { "name": "jina_scrape_llm_summary-scrape_and_extract_info", "arguments": "{"url": "https://zh.wikipedia.org/zh-cn/…", "info_to_extract": "Find the infobox image (poster) on the Wikipedia page for 中国时报集团and count the number of trees in it."}" } < /tool_call > < tool_response > {"success": false, "url": "https://zh.wikipedia.org/zh-cn/…", "extracted_info": "", "error": "Scraping failed (both Jina and Python): Python Scrape: Unexpected error occurred: Client error ’403 Forbidden’ for url ’https://zh.wikipedia.org/zh-cn/…"} < /tool_response > |
| Examples of our LMM-Searcher on VisBrowse-Bench (Part 4/4) Iteration 12 < think > 403 error again. Let’s try a different approach. Let’s use the image search for " 中国时报集团 维基百科Infobox image". Or maybe directly find the infobox image URL via Google search image. Let’s search for " 中国时报集团 维基百科poster" or " 中国时报集团 维基百科infobox". < /think > < tool_call > { "name": "tool-google-search-image_search", "arguments": "{"q": " 中国时报集团 维基百科infobox"}" } < /tool_call > < tool_response > 1. | Title: 中国时报集团- 维基百科，自由的百科全书| Image URL: https://upload.wikimedia.org/wikipedia/commons/e/ee/… | Source: https://zh.wikipedia.org/zh-tw/… 2. | Title: 中国时报- 维基百科，自由的百科全书| Image URL: https://upload.wikimedia.org/wikipedia/zh/9/9b/… | Source: https://zh.wikipedia.org/zh-cn/… < /tool_response > Iteration 13 < think > Wait, the first image is the China Times building with trees in front. Let’s count the trees in that image. Let’s fetch that image and analyze it. < /think > < tool_call > { "name": "tool-fetch-image-fetch_image", "arguments": "{"url": "https://upload.wikimedia.org/wikipedia/commons/e/ee/…"}" } < /tool_call > < tool_response > Image downloaded from: https://upload.wikimedia.org/…" < /tool_response > Iteration 14 < think > Now let’s count the trees in this image. Let’s see: in front of the building, there are several trees. Let’s list them: left side: one tree, then two more, …. Wait, but let me confirm once more. Let me list each tree with coordinates: 1. (x: 20, y: 800) - small tree 2. (x: 200, y: 850) - tree near pole 3. (x: 450, y: 880) - tree near first taxi 4. (x: 550, y: 890) - tree near second taxi 5. (x: 900, y: 850) - large tree on right Yes, that’s 5 trees. So the answer is 5. < /think > The food company is Want Want, which acquired the China Times Group ( 中时媒体集团) in 2009. The Wikipedia infobox poster for this newspaper shows 5 distinct trees in front of its headquarters building." Iteration 15 < User > Summarize the above conversation, and output the FINAL ANSWER to the original question. If a clear answer has already been provided earlier in the conversation, do not rethink or recalculate it — simply extract that answer and reformat it to match the required format below. If a definitive answer could not be determined, make a well-informed educated guess based on the conversation… < think > Okay, let’s summarize the conversation and extract the final answer. First, we identified the logo on the beverage as Want Want’s mascot. Then we found that Want Want acquired China Times Group in 2009. We looked at the Wikipedia poster (infobox image) for China Times Group, which shows the headquarters building with trees. Counting the distinct trees, we found 5.The original question asks for the number of trees in the newspaper’s Wikipedia poster. So the answer is boxed{5}. < /think > |