MKA: Evaluating Multimodal Knowledge Agents on end-to-end professional tasks
Authors: Anima Singh, Suhas Yogin, Weicheng Kuo, Ivor Rendulic, Ivan Jurin, Antoine Yang, Oran Kelly, Rohan Doshi, Hammad Ayyubi, Ioana Bica, Sergi Caelles, Matthew Conniff, Dan Graur, Anastasija Ilić, Ahmet Iscen, Chao Jia, Tomasz Kępa, Chunyuan Li, Noah Ó Donnaile, Filip Pavetic, Goran Petrovic, Khalid Salama, Zhicheng Wang, Polina Zablotskaia, Yiqian Zhang, Ming Zhao, Benjamin Caine, Mario Lučić, Melvin Johnson, Radu Soricut, Jean-Baptiste Alayrac
We introduce the Multimodal Knowledge Agent (MKA) benchmark, which measures how well autonomous agents perform end-to-end professional knowledge work that depends on visual understanding. Each task requires an agent to extract and combine information from multiple visually dense files, such as CAD blueprints, electronics schematics, radiographs, complex charts and geospatial maps, and to produce a standalone deliverable in .xlsx, .pptx, .pdf, .docx, .html or .png format. Existing benchmarks leave this capability largely untested: removing image-viewing tools leaves Gemini 4 Carbon's score on GDPval unchanged, and single-format evaluations such as GDP.pdf and GDP.xlsx do not require reasoning across files or modalities. MKA contains 200 tasks across 10 economic sectors. Deliverables are graded against expert-weighted rubrics by a sandboxed agentic autorater that agrees with human domain experts on 96.2% of criterion judgments in our expert-graded study (Cohen's κ=0.91), scored by weighted satisfaction rate (WSR) overall and on frontier criteria unsaturated by prior-generation models (Climb-WSR). Across the frontier models we evaluate (Gemini 4 Carbon, Claude Fable 5.2, Claude Opus 5.5, and GPT-6 Astra), Claude Fable 5.2 (56.6% Climb-WSR, 73.3% Overall WSR) and Gemini 4 Carbon (54.3% Climb-WSR, 71.9% Overall WSR) achieve the highest scores, while Carbon uses 5.7× fewer tokens. Even for the strongest models, Climb-WSR remains below 57%, leaving substantial headroom for multimodal agents.
Figures & tables
Figure 1: Overview of the 10 economic sectors in MKA (200 tasks total, 20 per sector), spanning blueprints, schematics, financial charts, clinical records, contracts, geospatial maps, scientific plots, and interactive dashboards.
Load-Bearing
Multi-Step
Heterogeneous
Open-Ended
Visual + Code
Benchmark
Vision
Tool Use
Multi-File
Deliverables
Grading
MMMU-Pro / CharXiv ( Yue et al., 2024b ; Wang et al., 2024c )
Yes
No
No
No
No
GDP.pdf ( Garre et al., 2026 )
Yes
No
No (PDFs Only)
No
No
GDP.xlsx ( Surge AI, 2026 )
No
Yes
No (Single File)
No
No
BrowseComp / MM-BrowseComp ( Wei et al., 2025 ; Li et al., 2025b )
Partial
Yes
No
No
No
OSWorld ( Xie et al., 2024 )
Yes (GUI)
Yes
Partial
Partial
No
Table 1: Comparison of MKA against representative frontier benchmarks across multimodal, browsing, GUI, tabular, and professional knowledge-work evaluations.
Figure 2: Distribution of the 1,459 input files across the 200 MKA tasks by format category.
Figure 3: Distribution of primary deliverable formats across the 200 MKA tasks.
Figure 4: Three example tasks from the open MKA-Showcase split. Each card shows the sector and expert role, a summary of the prompt, three of the task’s input files, a preview of the human golden deliverable, and the number of rubric criteria with one example criterion: a department staffing feeder audit (workforce planning analyst), a council rooftop solar screening (GIS analyst), and a nonlinear dynamics lab deck (mechanical engineering instructor).
Figure 5: Pairwise criterion agreement with human domain raters (error bars: 95% task-level bootstrap CIs). The Agentic Autorater reaches 96.2% agreement ( AHA ), matching 96.4% of the 99.8% inter-human ceiling ( AHH ).
Error Type
Definition (vs. Human Majority)
% of All Criteria
Conditional Rate (%)
False pass
Human fails, autorater passes
2.4 [1.3,3.8]
7.1 [3.6,11.4]
False fail
Human passes, autorater fails
1.4 [0.8,2.0]
2.1 [1.2,3.1]
Total error
Any disagreement with human majority
3.8 [2.4,5.4]
—
Table 2: Decomposition of the 3.8% per-criterion autorater error rate against the human majority verdict (95% task-level bootstrap CIs). Conditional rates normalize by the human-failed (34.2%) and human-passed (65.8%) base rates.
Model
Climb-WSR
Overall WSR
Regression-WSR
Claude Fable 5.2
56.56 ± 3.63
73.26 ± 2.45
95.71 ± 1.68
Gemini 4 Carbon
54.34 ± 3.24
71.89 ± 2.15
96.36 ± 0.91
Claude Opus 5.5
53.01 ± 3.54
70.98 ± 2.46
95.81 ± 1.32
GPT-6 Astra
50.24 ± 3.25
69.34 ± 2.10
95.17 ± 1.07
Table 3: Deliverable quality on MKA (task-level means ± 95% CI, %).
Figure 6: Deliverable quality across rubric splits ( a ; task-level means) and mean total tokens, output tokens and model turns per task ( b – d ; multipliers relative to Gemini 4 Carbon).
Figure 7: Deliverable quality (Climb-WSR) versus mean total tokens per task across MKA tasks (upper-left is better). GPT-6 Astra, Gemini 4 Carbon and Claude Fable 5.2 trace the token Pareto frontier, with Gemini 4 Carbon using fewer tokens than Claude Opus 5.5 at similar quality, and sitting at the knee before Fable 5.2’s 5.74× token jump.
Quality Δ (w/o − w/ view_image , pp)
Compute Ratio (w/o / w/ view_image )
Benchmark
Climb-WSR
Overall WSR
Reg.-WSR
Input Tok.
Output Tok.
Turns
MKA
−5.94
−4.58
−1.96
1.85×
1.40×
1.28×
GDPval
—
+0.08 (ns)
—
0.89×
1.05×
0.95×
Table 5: Tool ablation (w/ vs. w/o view_image ) for Gemini 4 Carbon on MKA and GDPval. Removing view_image leaves GDPval unchanged in both quality and cost, whereas MKA suffers a significant Climb-WSR drop alongside steep token and turn inflation.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Case Study & Arm
Climb-WSR
Overall WSR
Input Tok.
Output Tok.
Turns
1. CAD Review (w/ view_image )
76.3
88.7
1.36M
64.1k
21.7
1. CAD Review (w/o view_image )
63.3
82.4
6.21M
113.9k
51.0
Δ / Ratio
−13.0
−6.2
4.6×
1.8×
2.4×
2. Road Photo (w/ view_image )
86.7
96.0
4.87M
94.0k
43.0
2. Road Photo (w/o view_image )
14.3
64.7
18.06M
207.5k
76.5
Δ / Ratio
−72.4
−31.3
3.7×
2.2×
1.8×
Appendix
Table 6: Comparison of Gemini 4 Carbon w/ view_image and w/o view_image on two representative MKA tasks illustrating partial coding recovery versus perceptual collapse.
Figure 8: Case Study 1: the workaround recovers most of the score at 2–5 × the compute. (a) The CAD drawing as the agent w/ view_image sees it. (b) The Braille render that the agent w/o view_image printed from code_exec , cropped to the same region. Red rings mark the geometric features that the manufacturability judgment depends on. With view_image , the agent catches both in 3/3 runs, while w/o view_image catches them in 1/2.
Figure 9: Case Study 2: the workaround fails outright despite 2–4 × the compute. (a) A site photo from the inspection report. The red ring marks the region that holds the visual evidence the task asks for. (b) The ASCII luminance render that the agent w/o view_image printed for the same photo. The evidence is present in the render as a dark cluster, but the agent does not recognize it. The agent w/ view_image identifies it in every run. The agent w/o view_image misses it in all runs, and Climb-WSR drops from 86.7 to 14.3.
Real-world multimodal agents solve multi-step workflows grounded in visual evidence. For example, an agent can troubleshoot a device by linking a wiring photo to a schematic and validating the fix with online documentation, or plan a trip by interpreting a transit map and checking schedules under routing constraints. However, existing multimodal benchmarks mainly evaluate single-turn visual reasoning or specific tool skills, and they do not fully capture the realism, visual subtlety, and long-horizon tool use that practical agents require. We introduce AgentVista, a benchmark for generalist multimodal agents that spans 25 sub-domains across 7 categories, pairing realistic and detail-rich visual scenarios with natural hybrid tool use. Tasks require long-horizon tool interactions across modalities, including web search, image search, page navigation, and code-based operations for both image processing and general programming. Comprehensive evaluation of state-of-the-art models exposes significant gaps in their ability to carry out long-horizon multimodal tool use. Even the best model in our evaluation, Gemini-3-Pro with tools, achieves only 27.3% overall accuracy, and hard instances can require more than 25 tool-calling turns. We expect AgentVista to accelerate the development of more capable and reliable multimodal agents for realistic and ultra-challenging problem solving.
Zhaochen Su, Jincheng Gao, Hangyu Guo +10
Hong Kong University of Science and Technology · University of North Carolina at Chapel Hill · Zhejiang University +1
A large share of day-to-day work in professional domains happens inside PDF files: benefits packets, leases, datasheets, clinical guidelines, construction plans. Benchmarks for document AI have generally measured the required capabilities in isolation: OCR, layout analysis, chart reasoning, table QA, document VQA. A high score on any one of them does not necessarily reveal whether a model can answer a realistic question that someone in the field would actually ask about a specific PDF. GDP_pdf is a benchmark built to measure this directly. It consists of question-document pairs authored by working professionals in ten fields, and a candidate question was kept only when at least two frontier multimodal models failed it in a way that mattered: a wrong answer, missed decisive evidence, or a fabricated claim, rather than a superficial difference such as style. Each item comes with a rubric of atomic criteria, so we can report a graded rubric score as well as a strict task-level pass rate, and each item is tagged against a taxonomy of eleven capabilities in three tiers, spanning text extraction and grounding, table and chart comprehension, cross-referencing, spatial reasoning, and abstention on unsupported queries. We report results for seventeen frontier models on the 100-item benchmark: the best model passes only 30.7% of the items and the worst passes 2%. Most errors trace back to a small set of recurring loss patterns: misaligned tables, misread charts, skipped footnotes and exclusions, miscounted floor-plan symbols, scan noise, and amendments that supersede earlier text.
Language agents are increasingly deployed over accumulating multimodal information, yet existing benchmarks assume a human-human form with sparse visuals and straightforward content, evaluating neither reasoning over authentic multimodal file interaction nor the interpretation of concealed user information. We therefore introduce M3Exam, a query-centric multimodal conversational memory benchmark built on realistic user-agent interaction, with multi-dimensional evaluation spanning cross-modal grounding and implicit information inference. Benchmarking MLLMs and memory systems reveals persistent gaps in cross-modal grounding, cross session reasoning, and the efficiency cost of accumulating multimodal context. We further propose M3Proctor, a multimodal memory method that detects query modality bias and consumes raw visual sources only on demand, improving accuracy by 13% while cutting index-construction time and retrieved tokens by over 70%.
Zhengjun Huang, Wenxuan Liu, Zhoujin Tian +6
The Hong Kong University of Science and Technology · Peng Cheng Laboratory · Beijing University of Chemical Technology +4