AgentVista: Evaluating Multimodal Agents in Ultra-Challenging Realistic Visual Scenarios

Zhaochen Su; Jincheng Gao; Hangyu Guo; Zhenhua Liu; Lueyang Zhang; Xinyu Geng; Shijue Huang; Peng Xia

cs.CVFeb 26, 2026

AgentVista: Evaluating Multimodal Agents in Ultra-Challenging Realistic Visual Scenarios

Authors: Zhaochen Su, Jincheng Gao, Hangyu Guo, Zhenhua Liu, Lueyang Zhang, Xinyu Geng, Shijue Huang, Peng Xia, +5 more

Organizations: Hong Kong University of Science and Technology · University of North Carolina at Chapel Hill · Zhejiang University · National University of Singapore

Abstract

Real-world multimodal agents solve multi-step workflows grounded in visual evidence. For example, an agent can troubleshoot a device by linking a wiring photo to a schematic and validating the fix with online documentation, or plan a trip by interpreting a transit map and checking schedules under routing constraints. However, existing multimodal benchmarks mainly evaluate single-turn visual reasoning or specific tool skills, and they do not fully capture the realism, visual subtlety, and long-horizon tool use that practical agents require. We introduce AgentVista, a benchmark for generalist multimodal agents that spans 25 sub-domains across 7 categories, pairing realistic and detail-rich visual scenarios with natural hybrid tool use. Tasks require long-horizon tool interactions across modalities, including web search, image search, page navigation, and code-based operations for both image processing and general programming. Comprehensive evaluation of state-of-the-art models exposes significant gaps in their ability to carry out long-horizon multimodal tool use. Even the best model in our evaluation, Gemini-3-Pro with tools, achieves only 27.3% overall accuracy, and hard instances can require more than 25 tool-calling turns. We expect AgentVista to accelerate the development of more capable and reliable multimodal agents for realistic and ultra-challenging problem solving.

Figures & tables

Figure 1: A representative AgentVista task grounded in a real home-renovation scenario. The agent needs to match flooring styles across images, verify the target room, retrieve product specifications, and compute final cost via interleaved tool use.

Figure 2: Sampled AgentVista examples from each domain. Each query is grounded in complex, real-world visual scenes and is designed to elicit agentic tool use with multi-step reasoning toward a unique, verifiable answer.

Benchmark	VO.	VS.	TS.	CE.	Multi Image	# Turns
TIR-Bench	✓	✗	✗	✓	✓	2.92
Agent-X	✓	✗	✓	✓	✓	3.4
MMSearch-Plus	✗	✓	✓	✗	✓	4.6
BrowseComp-VL	✗	✓	✓	✓	✗	4.3
VisualToolBench	✓	✗	✓	✓	✗	4.46
AgentVista (Ours)	✓	✓	✓	✓	✓	12.67

Table 1: Comparison with representative multimodal agent benchmarks. Operation abbreviations: VO. (Visual Operations), VS. (Visual Search), TS. (Text Search), and CE. (Code Execution). Tool categories are based on the tools and signals used in these benchmarks. “# Turns” reports the average number of tool-calling turns by GPT-5 , used as a proxy for task complexity.

Figure 3: Overview of the AgentVista dataset construction pipeline, consisting of agent-centric filtering, expert finalization, execution filtering, and two-round verification to produce realistic and ultra-challenging multimodal agent tasks.

Figure 4: The categorization of AgentVista . The benchmark spans 7 major categories and 25 sub-domains, covering a broad range of realistic and challenging multimodal agent scenarios. Category abbreviations: Comm. (Commerce), Geog. (Geography), Ent. (Entertainment), Tech. (Technology), Soc. (Society), Acad. (Academics), and Cult. (Culture).

Statistic	Number
Total queries	209
Total images	308
Primary categories	7
Secondary categories	25
Average query length	401.4
Average answer length	40.8

Table 2: Summary statistics of the AgentVista benchmark.

Model	By Category							By Input Mode		Summary
Model	Comm.	Geog.	Ent.	Tech.	Soc.	Acad.	Cult.	Single.	Multi.	Overall	# Turns
Qwen3-VL-235B	7.14	7.69	7.69	26.47	16.00	20.00	13.33	11.84	15.79	12.92	2.34
GPT-4.1	16.67	15.38	10.26	29.41	20.00	20.00	13.33	15.13	24.56	17.70	1.74
o3	21.43	15.38	7.69	23.53	40.00	26.67	13.33	17.76	26.32	20.10	13.18
o4-mini	2.38	10.26	2.56	8.82	8.00	13.33	0.00	6.58	5.26	6.22	1.89
GPT-5	23.81	23.08	12.82	35.29	28.00	26.67	26.67	24.34	24.56	24.40	12.67

Table 3: Main results on our proposed AgentVista . Domain abbreviations: Comm. (Commerce), Geog. (Geography), Ent. (Entertainment), Tech. (Technology), Soc. (Society), Acad. (Academics), and Cult. (Culture). Input mode abbreviations: Single. (Single-image input) and Multi. (Multi-image input). The best-performing model in each category is in-bold , and the second best is underlined . Overall, Gemini-3-Pro achieves the highest accuracy among all evaluated models. All values are accuracies in %.

Figure 5: Tool-use distribution across models. GPT models rely more on the code interpreter, while Gemini and Claude models use web search most frequently.

Figure 6: Image manipulation operation distribution of code interpreter calls across four multimodal models. Tool usages are automatically categorized into image-editing and analysis-related types. Across models, crop is the most frequent operation, suggesting that many interactions rely on localized visual grounding before further reasoning.

Figure 7: Tool ablation on Gemini-3-Pro and Claude-Sonnet-4.5 . Both models perform best with the full tool suite, highlighting the importance of combining visual manipulation and retrieval.

Figure 8: Error category distribution on AgentVista across four multimodal models. Error types are automatically labeled by Gemini-3-Flash based on model trajectories. Across all models, visual misidentification is the dominant failure mode, indicating that many errors originate from incorrect grounding on fine-grained visual evidence.

Setting	$K{=}1$	$K{=}2$	$K{=}4$	$K{=}8$	$K{=}16$
Random1@ $K$	21.05	19.11	18.23	17.09	18.05
BoN@ $K$	21.05	24.88	26.32	28.23	30.62
Pass@ $K$	21.05	26.07	34.22	42.59	51.67

Table 4: Test-time scaling results under different sampling budgets

K

on Gemini-3-Flash . We report Random1@

K

as a lower bound, Best-of-

K

(BoN@

K

) selected by a reward model, and Pass@

K

as an upper bound. All values are accuracies in %.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	Comm.	Geog.	Ent.	Tech.	Soc.	Acad.	Cult.	Overall
Qwen3-VL-235B	7.14	7.69	7.69	26.47	16.00	20.00	13.33	12.92
DeepEyes-v2-7B	9.52	10.26	2.56	14.71	24.00	6.67	20.00	11.48
WebWatcher-32B	0.00	10.26	0.00	23.53	24.00	20.00	0.00	10.05

Appendix

Table 5: Results of representative open-source models on AgentVista by category. Domain abbreviations: Comm. (Commerce), Geog. (Geography), Ent. (Entertainment), Tech. (Technology), Soc. (Society), Acad. (Academics), and Cult. (Culture). The best-performing model in each category is in-bold , and the second best is underlined . All values are accuracies in %.

Explore similar work

WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agents
Aug 28, 2026Zongkai Liu, Hui Zhang, Liqiang Niu +7Multimodal IR Multimodal Search Agents
Weixin AI, Tencent · Sun Yat-sen University
VISTA: A Visual Harness for Reasoning in an Interactive World
Oct 1, 2026Qiushi Han, Keya Hu, Linlu Qiu +2Memory-Augmented VLMs Visual Reasoning
Massachusetts Institute of Technology
Paint What You See: Benchmarking Dexterous Visual Tool Use in Multimodal Agents
Aug 26, 2026Shudong Liu, Dongyang Chen, Enci Zhang +3AI Agent Benchmarks Tool Use
Peking University · SenseTime Research · Tsinghua University

Aug 28, 2026J/K move · Enter open · S save

WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agents

Zongkai Liu, Hui Zhang, Liqiang Niu, Zhen Cao+6

Weixin AI, Tencent · Sun Yat-sen University

Multimodal IR Multimodal Search Agents Multimodal Agents Multimodal Model Evaluation Multimodal Tool Use

Open paper AskPDF arXiv

Multimodal search agents extend parametric knowledge with newly emerging and long-tail evidence from the open web. Yet many existing agentic search environments often expose retrieved evidence only as text and omit tool-returned images from subsequent context, reducing visually grounded trajectories to text-only reasoning. Long-horizon interaction also compounds tool-call, response-length, timeout, and budget failures, which can discard salvageable trajectories, waste rollout computation, and disturb policy updates. To address these issues, we introduce WeAgent-Harness, a multimodal agentic harness that supports native text-vision interaction and runtime recovery. Retrieved images receive persistent disk references, allowing the model to inspect, process, and cite them throughout the trajectory. Based on this harness, we develop WeAgent-MMSearch, an integrated system spanning data construction, agentic post-training, and multimodal rollout. For data construction, a strong MLLM uses WeAgent-Harness to discover, synthesize, and verify MMSearch-style tasks and collect expert trajectories. During post-training, our Failure-Aware GSPO (FA-GSPO) recovers salvageable abnormal rollouts and filters invalid ones to improve bounded multimodal planning and search. We also introduce VisTarget-Bench, a 150-task human-verified benchmark that pairs each question with a held-out target image, distinguishing image-retrieval failures from visual-perception failures. Evaluation on VisTarget-Bench and seven public benchmarks shows that agentic post-training improves the average score by 19.22 points, enabling our model to outperform similarly sized open-source models and rival models with roughly ten times its parameter count.

Abstract

Explore similar work

WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agents

VISTA: A Visual Harness for Reasoning in an Interactive World

Paint What You See: Benchmarking Dexterous Visual Tool Use in Multimodal Agents