VisSearch-Bench: Evaluating Long Horizon Visual Search in Multimodal Agents
Organizations: Tsinghua University · Ant Group · Renmin University of China
Abstract
Multimodal agents use web and image tools to answer questions beyond their internal knowledge. Some search tasks use images mainly to identify an initial entity, then rely on text retrieval. These tasks offer limited insight into how agents connect information across images, objects, and search steps. We therefore focus on visual context management. This involves organizing and using visual information and linking it to other findings during search. Visually intensive tasks require agents to combine object appearance, local details, or spatial relations with external information. We introduce VisSearch-Bench, with 204 human-written tasks across 7 main domains and 25 subdomains. It covers Set-wise Visual Search, Sequential Visual Search, Hybrid Wide-Deep Search, Competitive Visual Disambiguation, and Visual-Structure-Guided Search. Each task provides a reference answer, traceable evidence, and scoring rules. Accuracy measures complete task success, while Process Score measures the correctness and completeness of individual results and subquestion answers. Across 16 multimodal models, the best system fully solves only 26.47% of tasks, even with search tools. These results highlight the challenge of linking visual evidence with search results to produce complete answers.
Figures & tables
| Benchmarks | Open- Web | Image Search | Multi- image | Multi- hop | Set Answers | Evidence Links | Partial Credit |
|---|---|---|---|---|---|---|---|
| BrowseComp | — | — | — | — | — | ||
| MMSearch | ◐ | ◐ | — | ◐ | |||
| BrowseComp-VL | — | — | — | — | |||
| MM-BrowseComp | — | ||||||
| MMSearch-Plus | — | ◐ | — | ||||
| VisBrowse-Bench | — | — | — |
| Statistic | Value |
|---|---|
| Total questions | 204 |
| Number of task families | 5 |
| Number of primary domains | 7 |
| Set-wise Visual Search | 68 (33.3%) |
| Reference-solution steps (avg. / max.) | 23.9 / 51 |
| Sequential Visual Search | 34 (16.7%) |
| Setting | Full wrong correct | Full correct wrong | Correct answers | Acc. (%) |
|---|---|---|---|---|
| Qwen3.6-35B-A3B (Full: 15, 7.35%) | ||||
| Caption | 6 | 8 | 13 | 6.37 |
| Text | 6 | 9 | 12 | 5.88 |
| Qwen3.5-397B-A17B (Full: 32, 15.69%) | ||||
| Caption | 8 | 10 | 30 | 14.71 |
| Text | 5 | 20 | 17 | 8.33 |
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | Questions | Categories | Visual input | Metric used | Primary focus |
|---|---|---|---|---|---|
| MMSearch | 300 | 14 subfields | 171 queries | End-to-end score | Query reformulation, reranking, summarization, and answering |
| MMSearch-Plus | 311 | 8 primary | All questions | Accuracy | Fine-grained visual cues and provenance-aware retrieval |
| MM-BrowseComp | 224 | 22 subtasks | 57% of questions | Strict Accuracy | Difficult multimodal browsing with verified reasoning checklists |
| VisBrowse-Bench | 169 | 7 primary / 24 secondary | All questions | Accuracy | Visual-native search across webpages and image assets |
| Category | Hyperparameter | Value | Scope / note |
| Evaluation Protocol | |||
| Sampling protocol | Pass@1 | One fresh context per task | |
| Question field | question_en | Identical across systems | |
| Temperature | 0.0 | Omitted for GPT-5.5 and Claude Opus 4.8 | |
| Request timeout | 180 s / 30 s | Model / tool requests | |
| API attempt limit | 1 or 3 | Run-specific; includes first request | |
| Tool | Observation returned to the agent |
|---|---|
| Web search | Titles, snippets, and URLs for up to five results |
| Webpage reader | Cleaned page text, limited to 12,000 characters |
| Image search | Up to five images with source metadata |
| Reverse-image search | Visually related results and source pages |
| Image crop | A selected region from an available image |
| Field | Type | Description |
|---|---|---|
| id | int64 | Unique task ID from 1 to 204 |
| category_name | string | One of the five task categories defined in the paper |
| domain | string | Primary content-domain assignment |
| question_en | string | English task question |
| answer | string | Reference answer |
| gold_trajectory | string | Reference solution with search steps and source links |
| Area | Method | Check | Action if an issue is found |
|---|---|---|---|
| Source evidence | Human review | Public source links, object–image matches, and supporting facts | Revise evidence |
| Visual evidence integration | Human review | Links between visual observations, object identities, and information needed for the answer | Revise task |
| Shortcut review | Human review | Text leakage and routes that may bypass the intended visual requirements | Revise task |
| Answer consistency | Human review | Agreement between the question, reference solution, answer, and scoring rules; set coverage where needed | Revise annotations |
| Record integrity | Program checks | Unique positive integer IDs, required text fields, category labels, and image-path lists | Correct records |
| Scoring prompts | Program checks | Both evaluation prompts and their question, reference-answer, and model-answer placeholders | Correct prompts |
Multimodal Search Agents
Multimodal search agents retrieve and reason across text, images, video, and other sources to answer complex questions.
See all 49 papers