FeynmanBench: Benchmarking Multimodal LLMs on Diagrammatic Physics Reasoning

Zeyu Wang; Jingye Xu; Xiaogang Li; Peiyao Xiao; Qinhao Kong; Ben Wang; Chengliang Xu; Zichao Chen

cs.AIApr 4, 2026

FeynmanBench: Benchmarking Multimodal LLMs on Diagrammatic Physics Reasoning

Authors: Zeyu Wang, Jingye Xu, Xiaogang Li, Peiyao Xiao, Qinhao Kong, Ben Wang, Chengliang Xu, Zichao Chen,

Organizations: Alibaba Group, Beijing, China · Skylenage, Beijing, China

Abstract

Current multimodal benchmarks for scientific reasoning primarily evaluate local information extraction -- models recognize symbols and values and then perform textual inference. They do not assess whether models can reason over the global structural properties of formal diagrams, such as topology, conservation constraints, and the consistent mapping between visual patterns and algebraic expressions. We introduce FeynmanBench, a benchmark of over 2,000 tasks centered on Feynman diagrams spanning the electromagnetic, weak, and strong interactions of the Standard Model. Each instance couples a diagram image with minimal textual conventions and requires models to recover the full physical content -- vertex inventory, propagator types, topological connectivity, momentum routing, and the complete scattering amplitude. An automated generation and verification pipeline produces the diagrams, annotations, and reference answers under standardized rules. Evaluating 19 state-of-the-art multimodal LLMs, we find a consistent failure pattern: models achieve 70--95% on local recognition (vertex and propagator identification) but collapse to 13--17% on topological reconstruction (CP3), and near zero on full algebraic derivation (CP5). FeynmanBench offers a controlled testbed for multimodal reasoning over formal scientific diagrams and highlights fundamental limitations of current architectures in topology-sensitive scientific reasoning.

Figures & tables

Interaction sector	External-leg content	A1	A2	A3	A4	A5
QED	Photons, $e^{\pm}$ only	15	86	17	150	119
QED	Photons + generic charged fermions	11	112	12	106	72
Electroweak	Leptons and neutrinos	45	125	51	50	120
Electroweak	Includes $W/Z$	8	115	68	75	147
Electroweak	Includes Higgs	17	56	122	70	89
QCD	Quarks / gluons	7	38	74	51	28

Table 1. Diagram counts by interaction sector and topology class. Topology: tree (A1), one-loop 1PR (A2), one-loop 1PI (A3), two-loop 1PR (A4), two-loop 1PI (A5).

Figure 1. Workflow of FeynmanBench. Panels 1–4: diagram generation and annotation. Panel 5: amplitude derivation via Feynman rules. Panel 6: automated scoring against ground truth. The same pipeline that generates diagrams also produces their verifiable structural and symbolic ground truth, enabling scalable evaluation without manual answer authoring.

Figure 2. Model performance across checkpoints.

Figure 3. Prompt structure and task definitions with example expectations. The prompt provides minimal conventions (interaction type, external-leg identities, momentum orientation)—sufficient to resolve drawing ambiguities without supplying topological or algebraic answers.

Model	$N$	CP1	CP2	CP3	CP4	CP5	Score
GPT-5.5 †	272	84.2	77.6	45.6	39.3	15.8	52.5
Kimi K2.6 †	165	82.4	73.9	38.2	29.7	11.5	47.2
Gemini 3.1 Pro †	539	85.7	70.5	33.6	23.9	7.8	44.3
Gemini 3 Flash †	742	94.7	84.5	16.8	5.0	0.0	40.2
Qwen 3.6 Plus	2051	72.6	49.0	15.2	9.7	1.7	29.6
Doubao Seed 2.0 Pro	2046	79.1	44.3	13.3	5.4	0.9	28.6

Table 2. Overall checkpoint pass rates (%).

\text{Score}=\frac{1}{5}\sum_{j}\text{CP}_{j}

\dagger

: evaluated on a stratified subset.

\dagger\dagger

: not covered for this category.

Model	A1	A2	A3	A4	A5	Overall
Kimi K2.5 †	75.0	25.0	–	–	–	52.3
GPT-5.5 †	57.6	28.9	–	–	–	39.3
Kimi K2.6 †	53.3	20.8	–	–	–	29.7
Gemini 3.1 Pro †	39.8	18.0	32.2	8.2	27.6	23.9
Claude Sonnet 4.6	56.9	16.2	28.2	0.2	3.7	12.6
Qwen 3.6 Plus	48.5	11.0	21.4	0.6	3.3	9.7

Table 3. CP4 pass rates (%) by topology class (A1–A5), isolating the effect of graph complexity. Without this stratification, aggregate scores would obscure the fact that most models fail almost completely once reasoning requires multi-loop global consistency.

\dagger

: subset. “–” denotes category not covered.

Figure 4. Representative examples of the four dominant failure modes. Each case corresponds to a pattern repeatedly observed across models and interaction sectors, rather than an isolated anomalous output.

Panel of Figure 5 — Figure 5. Heatmaps by model and category for CP1–CP3.

Scientific Reasoning in Language Models

Scientific reasoning in language models studies how AI systems use evidence, domain knowledge, and tools to solve problems and support discovery.

See all 144 papers

ReactBench: A Benchmark for Topological Reasoning in MLLMs on Chemical Reaction Diagrams
Apr 17, 2026Qiang Xu, Shengyuan Bai, Yu Wang +6Visual Reasoning Multimodal Model Evaluation
International Digital Economy Academy, Shenzhen, China · University of Wisconsin–Madison
Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams
Aug 12, 2026Weihao Bo, Shan Zhang, Yanpeng Sun +7Code Generation Multimodal QA
Nanjing University of Science and Technology · Baidu Inc · AIML, Adelaide University +4
What MLLMs Learn about When they Learn about Multimodal Reasoning
Oct 2, 2025Jiwan Chung, Neel Joshi, Pratyusha Sharma +2Multimodal Reasoning Multimodal Model Evaluation
Microsoft Research AI Frontiers · Yonsei University · Seoul National University

Apr 17, 2026J/K move · Enter open · S save

ReactBench: A Benchmark for Topological Reasoning in MLLMs on Chemical Reaction Diagrams

Qiang Xu, Shengyuan Bai, Yu Wang, He Cao+5

International Digital Economy Academy, Shenzhen, China · University of Wisconsin–Madison

Visual Reasoning Multimodal Model Evaluation

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) excel at recognizing individual visual elements and reasoning over simple linear diagrams. However, when faced with complex topological structures involving branching paths, converging flows, and cyclic dependencies, their reasoning capabilities degrade sharply, even on tasks as basic as counting endpoints. Existing benchmarks fail to probe this gap, focusing on semantic comprehension rather than structural reasoning. We introduce ReactBench, a benchmark that reveals fundamental limitations in structural reasoning through chemical reaction diagrams. These real-world scientific diagrams offer an ideal testbed because they naturally span diverse structures from linear chains to cyclic graphs, while requiring both precise local recognition and coherent global reasoning. Our benchmark comprises 1,618 expert-annotated QA pairs across four hierarchical task dimensions. Extensive evaluation across 24 MLLMs reveals a significant performance gap exceeding 30% between anchor-based tasks and holistic structural reasoning tasks. Controlled ablations confirm this bottleneck lies in reasoning, not perception. These findings expose a fundamental deficit in structural understanding and establish directions for advancing visual reasoning.

Abstract

Scientific Reasoning in Language Models

Explore similar work

ReactBench: A Benchmark for Topological Reasoning in MLLMs on Chemical Reaction Diagrams

Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams

What MLLMs Learn about When they Learn about Multimodal Reasoning