MKA: Evaluating Multimodal Knowledge Agents on end-to-end professional tasks
Authors: Anima Singh, Suhas Yogin, Weicheng Kuo, Ivor Rendulic, Ivan Jurin, Antoine Yang, Oran Kelly, Rohan Doshi, Hammad Ayyubi, Ioana Bica, Sergi Caelles, Matthew Conniff, Dan Graur, Anastasija Ilić, Ahmet Iscen, Chao Jia, Tomasz Kępa, Chunyuan Li, Noah Ó Donnaile, Filip Pavetic, Goran Petrovic, Khalid Salama, Zhicheng Wang, Polina Zablotskaia, Yiqian Zhang, Ming Zhao, Benjamin Caine, Mario Lučić, Melvin Johnson, Radu Soricut, Jean-Baptiste Alayrac
We introduce the Multimodal Knowledge Agent (MKA) benchmark, which measures how well autonomous agents perform end-to-end professional knowledge work that depends on visual understanding. Each task requires an agent to extract and combine information from multiple visually dense files, such as CAD blueprints, electronics schematics, radiographs, complex charts and geospatial maps, and to produce a standalone deliverable in .xlsx, .pptx, .pdf, .docx, .html or .png format. Existing benchmarks leave this capability largely untested: removing image-viewing tools leaves Gemini 4 Carbon's score on GDPval unchanged, and single-format evaluations such as GDP.pdf and GDP.xlsx do not require reasoning across files or modalities. MKA contains 200 tasks across 10 economic sectors. Deliverables are graded against expert-weighted rubrics by a sandboxed agentic autorater that agrees with human domain experts on 96.2% of criterion judgments in our expert-graded study (Cohen's κ=0.91), scored by weighted satisfaction rate (WSR) overall and on frontier criteria unsaturated by prior-generation models (Climb-WSR). Across the frontier models we evaluate (Gemini 4 Carbon, Claude Fable 5.2, Claude Opus 5.5, and GPT-6 Astra), Claude Fable 5.2 (56.6% Climb-WSR, 73.3% Overall WSR) and Gemini 4 Carbon (54.3% Climb-WSR, 71.9% Overall WSR) achieve the highest scores, while Carbon uses 5.7× fewer tokens. Even for the strongest models, Climb-WSR remains below 57%, leaving substantial headroom for multimodal agents.
Figures & tables
Figure 1: Overview of the 10 economic sectors in MKA (200 tasks total, 20 per sector), spanning blueprints, schematics, financial charts, clinical records, contracts, geospatial maps, scientific plots, and interactive dashboards.
Load-Bearing
Multi-Step
Heterogeneous
Open-Ended
Visual + Code
Benchmark
Vision
Tool Use
Multi-File
Deliverables
Grading
MMMU-Pro / CharXiv ( Yue et al., 2024b ; Wang et al., 2024c )
Yes
No
No
No
No
GDP.pdf ( Garre et al., 2026 )
Yes
No
No (PDFs Only)
No
No
GDP.xlsx ( Surge AI, 2026 )
No
Yes
No (Single File)
No
No
BrowseComp / MM-BrowseComp ( Wei et al., 2025 ; Li et al., 2025b )
Partial
Yes
No
No
No
OSWorld ( Xie et al., 2024 )
Yes (GUI)
Yes
Partial
Partial
No
Table 1: Comparison of MKA against representative frontier benchmarks across multimodal, browsing, GUI, tabular, and professional knowledge-work evaluations.
Figure 2: Distribution of the 1,459 input files across the 200 MKA tasks by format category.
Figure 3: Distribution of primary deliverable formats across the 200 MKA tasks.
Figure 4: Three example tasks from the open MKA-Showcase split. Each card shows the sector and expert role, a summary of the prompt, three of the task’s input files, a preview of the human golden deliverable, and the number of rubric criteria with one example criterion: a department staffing feeder audit (workforce planning analyst), a council rooftop solar screening (GIS analyst), and a nonlinear dynamics lab deck (mechanical engineering instructor).
Figure 5: Pairwise criterion agreement with human domain raters (error bars: 95% task-level bootstrap CIs). The Agentic Autorater reaches 96.2% agreement ( AHA ), matching 96.4% of the 99.8% inter-human ceiling ( AHH ).
Error Type
Definition (vs. Human Majority)
% of All Criteria
Conditional Rate (%)
False pass
Human fails, autorater passes
2.4 [1.3,3.8]
7.1 [3.6,11.4]
False fail
Human passes, autorater fails
1.4 [0.8,2.0]
2.1 [1.2,3.1]
Total error
Any disagreement with human majority
3.8 [2.4,5.4]
—
Table 2: Decomposition of the 3.8% per-criterion autorater error rate against the human majority verdict (95% task-level bootstrap CIs). Conditional rates normalize by the human-failed (34.2%) and human-passed (65.8%) base rates.
Model
Climb-WSR
Overall WSR
Regression-WSR
Claude Fable 5.2
56.56 ± 3.63
73.26 ± 2.45
95.71 ± 1.68
Gemini 4 Carbon
54.34 ± 3.24
71.89 ± 2.15
96.36 ± 0.91
Claude Opus 5.5
53.01 ± 3.54
70.98 ± 2.46
95.81 ± 1.32
GPT-6 Astra
50.24 ± 3.25
69.34 ± 2.10
95.17 ± 1.07
Table 3: Deliverable quality on MKA (task-level means ± 95% CI, %).
Figure 6: Deliverable quality across rubric splits ( a ; task-level means) and mean total tokens, output tokens and model turns per task ( b – d ; multipliers relative to Gemini 4 Carbon).
Figure 7: Deliverable quality (Climb-WSR) versus mean total tokens per task across MKA tasks (upper-left is better). GPT-6 Astra, Gemini 4 Carbon and Claude Fable 5.2 trace the token Pareto frontier, with Gemini 4 Carbon using fewer tokens than Claude Opus 5.5 at similar quality, and sitting at the knee before Fable 5.2’s 5.74× token jump.
Quality Δ (w/o − w/ view_image , pp)
Compute Ratio (w/o / w/ view_image )
Benchmark
Climb-WSR
Overall WSR
Reg.-WSR
Input Tok.
Output Tok.
Turns
MKA
−5.94
−4.58
−1.96
1.85×
1.40×
1.28×
GDPval
—
+0.08 (ns)
—
0.89×
1.05×
0.95×
Table 5: Tool ablation (w/ vs. w/o view_image ) for Gemini 4 Carbon on MKA and GDPval. Removing view_image leaves GDPval unchanged in both quality and cost, whereas MKA suffers a significant Climb-WSR drop alongside steep token and turn inflation.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Case Study & Arm
Climb-WSR
Overall WSR
Input Tok.
Output Tok.
Turns
1. CAD Review (w/ view_image )
76.3
88.7
1.36M
64.1k
21.7
1. CAD Review (w/o view_image )
63.3
82.4
6.21M
113.9k
51.0
Δ / Ratio
−13.0
−6.2
4.6×
1.8×
2.4×
2. Road Photo (w/ view_image )
86.7
96.0
4.87M
94.0k
43.0
2. Road Photo (w/o view_image )
14.3
64.7
18.06M
207.5k
76.5
Δ / Ratio
−72.4
−31.3
3.7×
2.2×
1.8×
Appendix
Table 6: Comparison of Gemini 4 Carbon w/ view_image and w/o view_image on two representative MKA tasks illustrating partial coding recovery versus perceptual collapse.
Figure 8: Case Study 1: the workaround recovers most of the score at 2–5 × the compute. (a) The CAD drawing as the agent w/ view_image sees it. (b) The Braille render that the agent w/o view_image printed from code_exec , cropped to the same region. Red rings mark the geometric features that the manufacturability judgment depends on. With view_image , the agent catches both in 3/3 runs, while w/o view_image catches them in 1/2.
Figure 9: Case Study 2: the workaround fails outright despite 2–4 × the compute. (a) A site photo from the inspection report. The red ring marks the region that holds the visual evidence the task asks for. (b) The ASCII luminance render that the agent w/o view_image printed for the same photo. The evidence is present in the render as a dark cluster, but the agent does not recognize it. The agent w/ view_image identifies it in every run. The agent w/o view_image misses it in all runs, and Climb-WSR drops from 86.7 to 14.3.