Current multimodal benchmarks for scientific reasoning primarily evaluate local information extraction -- models recognize symbols and values and then perform textual inference. They do not assess whether models can reason over the global structural properties of formal diagrams, such as topology, conservation constraints, and the consistent mapping between visual patterns and algebraic expressions. We introduce FeynmanBench, a benchmark of over 2,000 tasks centered on Feynman diagrams spanning the electromagnetic, weak, and strong interactions of the Standard Model. Each instance couples a diagram image with minimal textual conventions and requires models to recover the full physical content -- vertex inventory, propagator types, topological connectivity, momentum routing, and the complete scattering amplitude. An automated generation and verification pipeline produces the diagrams, annotations, and reference answers under standardized rules. Evaluating 19 state-of-the-art multimodal LLMs, we find a consistent failure pattern: models achieve 70--95% on local recognition (vertex and propagator identification) but collapse to 13--17% on topological reconstruction (CP3), and near zero on full algebraic derivation (CP5). FeynmanBench offers a controlled testbed for multimodal reasoning over formal scientific diagrams and highlights fundamental limitations of current architectures in topology-sensitive scientific reasoning.
Figures & tables
Interaction sector
External-leg content
A1
A2
A3
A4
A5
QED
Photons, e± only
15
86
17
150
119
QED
Photons + generic charged fermions
11
112
12
106
72
Electroweak
Leptons and neutrinos
45
125
51
50
120
Electroweak
Includes W/Z
8
115
68
75
147
Electroweak
Includes Higgs
17
56
122
70
89
QCD
Quarks / gluons
7
38
74
51
28
Table 1. Diagram counts by interaction sector and topology class. Topology: tree (A1), one-loop 1PR (A2), one-loop 1PI (A3), two-loop 1PR (A4), two-loop 1PI (A5).
Figure 1. Workflow of FeynmanBench. Panels 1–4: diagram generation and annotation. Panel 5: amplitude derivation via Feynman rules. Panel 6: automated scoring against ground truth. The same pipeline that generates diagrams also produces their verifiable structural and symbolic ground truth, enabling scalable evaluation without manual answer authoring.
Figure 2. Model performance across checkpoints.
Figure 3. Prompt structure and task definitions with example expectations. The prompt provides minimal conventions (interaction type, external-leg identities, momentum orientation)—sufficient to resolve drawing ambiguities without supplying topological or algebraic answers.
Model
N
CP1
CP2
CP3
CP4
CP5
Score
GPT-5.5 †
272
84.2
77.6
45.6
39.3
15.8
52.5
Kimi K2.6 †
165
82.4
73.9
38.2
29.7
11.5
47.2
Gemini 3.1 Pro †
539
85.7
70.5
33.6
23.9
7.8
44.3
Gemini 3 Flash †
742
94.7
84.5
16.8
5.0
0.0
40.2
Qwen 3.6 Plus
2051
72.6
49.0
15.2
9.7
1.7
29.6
Doubao Seed 2.0 Pro
2046
79.1
44.3
13.3
5.4
0.9
28.6
Table 2. Overall checkpoint pass rates (%). Score=51∑jCPj . † : evaluated on a stratified subset. †† : not covered for this category.
Model
A1
A2
A3
A4
A5
Overall
Kimi K2.5 †
75.0
25.0
–
–
–
52.3
GPT-5.5 †
57.6
28.9
–
–
–
39.3
Kimi K2.6 †
53.3
20.8
–
–
–
29.7
Gemini 3.1 Pro †
39.8
18.0
32.2
8.2
27.6
23.9
Claude Sonnet 4.6
56.9
16.2
28.2
0.2
3.7
12.6
Qwen 3.6 Plus
48.5
11.0
21.4
0.6
3.3
9.7
Table 3. CP4 pass rates (%) by topology class (A1–A5), isolating the effect of graph complexity. Without this stratification, aggregate scores would obscure the fact that most models fail almost completely once reasoning requires multi-loop global consistency. † : subset. “–” denotes category not covered.
Figure 4. Representative examples of the four dominant failure modes. Each case corresponds to a pattern repeatedly observed across models and interaction sectors, rather than an isolated anomalous output.
Figure 5. Heatmaps by model and category for CP1–CP3.
Multimodal Large Language Models (MLLMs) excel at recognizing individual visual elements and reasoning over simple linear diagrams. However, when faced with complex topological structures involving branching paths, converging flows, and cyclic dependencies, their reasoning capabilities degrade sharply, even on tasks as basic as counting endpoints. Existing benchmarks fail to probe this gap, focusing on semantic comprehension rather than structural reasoning. We introduce ReactBench, a benchmark that reveals fundamental limitations in structural reasoning through chemical reaction diagrams. These real-world scientific diagrams offer an ideal testbed because they naturally span diverse structures from linear chains to cyclic graphs, while requiring both precise local recognition and coherent global reasoning. Our benchmark comprises 1,618 expert-annotated QA pairs across four hierarchical task dimensions. Extensive evaluation across 24 MLLMs reveals a significant performance gap exceeding 30% between anchor-based tasks and holistic structural reasoning tasks. Controlled ablations confirm this bottleneck lies in reasoning, not perception. These findings expose a fundamental deficit in structural understanding and establish directions for advancing visual reasoning.
Qiang Xu, Shengyuan Bai, Yu Wang +6
International Digital Economy Academy, Shenzhen, China · University of Wisconsin–Madison
Multimodal Large Language Models (MLLMs) have been growing the capability for scientific writing and collaboration. For example, OpenAI Prism is a free workspace for scientific writing and collaboration. One important feature in Prism is turning scientific diagrams directly into LaTeX TikZ code. In this paper, we build a benchmark, Diagram-MMU, a multi-modal benchmark designed to assess MLLMs' ability for scientific diagram parsing and understanding. Diagram-MMU features 3.7k curated diagrams and 18.3k human-validated questions across six domains. It evaluates MLLMs on three tasks common in vibe writing workspaces: diagram-to-code parsing, diagram-to-code editing, and diagram question answering, alongside agentic settings per task. The evaluation of 12 MLLMs reveals that diagram-to-code tasks are more challenging than diagram question answering: models can reason well over diagrams but struggle to parse and edit them, underscoring the need for methods to enhance MLLMs' capability in diagram-to-code generation. Under agentic settings, most models improve parsing and editing performance but degrade on question answering, while Claude-4.6 Opus consistently improves across all three tasks. Project Page: https://vi-ocean.github.io/projects/diagram-mmu.
Weihao Bo, Shan Zhang, Yanpeng Sun +7
Nanjing University of Science and Technology · Baidu Inc · AIML, Adelaide University +4
Evaluation of multimodal reasoning models is typically reduced to a single accuracy score, implicitly treating reasoning as a unitary capability. We introduce MathLens, a benchmark of textbook-style geometry problems that exposes this assumption by operationally decomposing performance into perception, reasoning, and multimodal-specific components. Each problem is derived from a symbolic specification and accompanied by visual diagrams, text-only variants, multimodal questions, and targeted perceptual probes, enabling controlled measurement of each component. Using this decomposition, we show that common training strategies are associated with systematically different capability profiles that are invisible under aggregate accuracy. Across the models we study, reinforcement learning primarily improves perceptual grounding and robustness to diagram variation, while our analysis suggests that textual SFT yields gains through reflective reasoning. In contrast, as perception and reasoning improve, a growing fraction of remaining errors fall outside these components and are categorized as multimodal-specific. These results suggest that apparent progress in multimodal reasoning reflects shifting balances among subskills rather than uniform advancement, motivating evaluation beyond scalar accuracy.
Jiwan Chung, Neel Joshi, Pratyusha Sharma +2
Microsoft Research AI Frontiers · Yonsei University · Seoul National University