Multimodal agents use web and image tools to answer questions beyond their internal knowledge. Some search tasks use images mainly to identify an initial entity, then rely on text retrieval. These tasks offer limited insight into how agents connect information across images, objects, and search steps. We therefore focus on visual context management. This involves organizing and using visual information and linking it to other findings during search. Visually intensive tasks require agents to combine object appearance, local details, or spatial relations with external information. We introduce VisSearch-Bench, with 204 human-written tasks across 7 main domains and 25 subdomains. It covers Set-wise Visual Search, Sequential Visual Search, Hybrid Wide-Deep Search, Competitive Visual Disambiguation, and Visual-Structure-Guided Search. Each task provides a reference answer, traceable evidence, and scoring rules. Accuracy measures complete task success, while Process Score measures the correctness and completeness of individual results and subquestion answers. Across 16 multimodal models, the best system fully solves only 26.47% of tasks, even with search tools. These results highlight the challenge of linking visual evidence with search results to produce complete answers.
Figures & tables
Figure 1: Accuracy of models and agents on VisSearch-Bench , comparing direct answering with ReAct and DeerFlow. The dashed line marks the average accuracy across the systems shown.
Figure 2: Overview of VisSearch-Bench. The center presents five task families that require long-horizon visual search, while the surrounding panels show selected examples from the dataset across diverse content domains.
Benchmarks
Open- Web
Image Search
Multi- image
Multi- hop
Set Answers
Evidence Links
Partial Credit
BrowseComp
✓
—
—
✓
—
—
—
MMSearch
✓
✓
◐
◐
—
✓
◐
BrowseComp-VL
✓
✓
—
✓
—
—
—
MM-BrowseComp
✓
✓
✓
✓
—
✓
✓
MMSearch-Plus
✓
✓
✓
✓
—
◐
—
VisBrowse-Bench
✓
✓
✓
✓
—
—
—
Table 1: Comparison of representative open-web and multimodal search benchmarks. ✓ Full; ◐ Partial; — Not covered. InterLV-Search refers to its open-web Level 3.
Figure 3: Overview of the VisSearch-Bench construction pipeline. Human annotators curate public sources and organize them into traceable evidence graphs; structural constraints instantiate the five task families; each task is paired with a locked answer and gold trajectory; deterministic integrity gates, LLM-based post-filtering, and final human review verify the retained items.
Figure 4: Domain taxonomy and composition of VisSearch-Bench. The inner ring shows seven primary domains, and the outer ring shows 25 secondary domains. Arc area is proportional to the number of tasks in the 204-question benchmark. The right panel expands the full taxonomy and reports the task count within each category.
Statistic
Value
Total questions
204
Number of task families
5
Number of primary domains
7
Set-wise Visual Search
68 (33.3%)
Reference-solution steps (avg. / max.)
23.9 / 51
Sequential Visual Search
34 (16.7%)
Table 2: Dataset composition and structural complexity.
Setting
Full wrong → correct
Full correct → wrong
Correct answers
Acc. (%)
Qwen3.6-35B-A3B (Full: 15, 7.35%)
Caption
6
8
13
6.37
Text
6
9
12
5.88
Qwen3.5-397B-A17B (Full: 32, 15.69%)
Caption
8
10
30
14.71
Text
5
20
17
8.33
Table 6: ReAct answer changes from Full to each setting on 204 tasks. Group headers give the number of correct answers and accuracy under Full.
Figure 5: Tool use, efficiency, and performance on VisSearch-Bench.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Questions
Categories
Visual input
Metric used
Primary focus
MMSearch
300
14 subfields
171 queries
End-to-end score
Query reformulation, reranking, summarization, and answering
MMSearch-Plus
311
8 primary
All questions
Accuracy
Fine-grained visual cues and provenance-aware retrieval
MM-BrowseComp
224
22 subtasks
57% of questions
Strict Accuracy
Difficult multimodal browsing with verified reasoning checklists
VisBrowse-Bench
169
7 primary / 24 secondary
All questions
Accuracy
Visual-native search across webpages and image assets
Appendix
Table S1 : External benchmarks used in the cross-benchmark evaluation.
Category
Hyperparameter
Value
Scope / note
Evaluation Protocol
Sampling protocol
Pass@1
One fresh context per task
Question field
question_en
Identical across systems
Temperature
0.0
Omitted for GPT-5.5 and Claude Opus 4.8
Request timeout
180 s / 30 s
Model / tool requests
API attempt limit
1 or 3
Run-specific; includes first request
Appendix
Table S2 : Hyperparameters for generation, multimodal-agent inference, tool execution, and semantic evaluation.
Tool
Observation returned to the agent
Web search
Titles, snippets, and URLs for up to five results
Webpage reader
Cleaned page text, limited to 12,000 characters
Image search
Up to five images with source metadata
Reverse-image search
Visually related results and source pages
Image crop
A selected region from an available image
Appendix
Table S3 : Tool interface used by the ReAct agents.
Field
Type
Description
id
int64
Unique task ID from 1 to 204
category_name
string
One of the five task categories defined in the paper
domain
string
Primary content-domain assignment
question_en
string
English task question
answer
string
Reference answer
gold_trajectory
string
Reference solution with search steps and source links
Appendix
Table S4 : Fields in the English VisSearch-Bench release package.
Area
Method
Check
Action if an issue is found
Source evidence
Human review
Public source links, object–image matches, and supporting facts
Revise evidence
Visual evidence integration
Human review
Links between visual observations, object identities, and information needed for the answer
Revise task
Shortcut review
Human review
Text leakage and routes that may bypass the intended visual requirements
Revise task
Answer consistency
Human review
Agreement between the question, reference solution, answer, and scoring rules; set coverage where needed
Revise annotations
Record integrity
Program checks
Unique positive integer IDs, required text fields, category labels, and image-path lists
Correct records
Scoring prompts
Program checks
Both evaluation prompts and their question, reference-answer, and model-answer placeholders
Correct prompts
Appendix
Table S5 : Human review and program checks for benchmark construction and release.
Figure S1 : A successful Set-wise Visual Search trajectory from GLM-5.3-Flash (ReAct).
Figure S2 : A failed Set-wise Visual Search trajectory from DeepSeek-V4.1-Flash (ReAct).
Figure S3 : A successful Sequential Visual Search trajectory from GPT-5.5 (ReAct).
Figure S4 : A successful Sequential Visual Search trajectory from GPT-5.5 (DeerFlow).
Figure S5 : A failed Sequential Visual Search trajectory from GPT-5.5 (DeerFlow).
Figure S6 : A failed Sequential Visual Search trajectory from Qwen3.6-35B-A3B (DeerFlow).
Figure S7 : A successful Hybrid Wide–Deep Search trajectory from GPT-5.5 (DeerFlow).
Figure S8 : A failed Hybrid Wide–Deep Search trajectory from GPT-5.5 (ReAct).
Figure S9 : A failed Hybrid Wide–Deep Search trajectory from Qwen3.8-27B (ReAct).
Figure S10 : A successful Competitive Visual Disambiguation trajectory from Qwen3.5-397B-A17B (ReAct).
Figure S11 : A failed Competitive Visual Disambiguation trajectory from GPT-5.5 (DeerFlow).
Figure S12 : A failed Competitive Visual Disambiguation trajectory from Qwen3.5-397B-A17B (DeerFlow).
Figure S13 : A successful Visual-Structure-Guided Search trajectory from Kimi-K3 (ReAct).
Figure S14 : A successful Visual-Structure-Guided Search trajectory from Qwen3.6-35B-A3B (ReAct).
Figure S15 : A failed Visual-Structure-Guided Search trajectory from DeepSeek-V4.1-Flash (ReAct).
Figure S16 : A failed Visual-Structure-Guided Search trajectory from Qwen3.8-27B (ReAct).
Figure S17 : A failed Visual-Structure-Guided Search trajectory from GPT-5.5 (ReAct).