OmniPhys: A Unified Multimodal Benchmark for Physics Understanding and Generation from Chinese Educational Corpora

Hao Chen; Yumin Lin; Nadila Yushanjiang; Xin Lin; Min Zhang

cs.CLAug 26, 2026

OmniPhys: A Unified Multimodal Benchmark for Physics Understanding and Generation from Chinese Educational Corpora

Authors: Hao Chen, Yumin Lin, Nadila Yushanjiang, Xin Lin, Min Zhang

Organizations: East China Normal University

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated strong abilities in solving diverse visual and textual reasoning tasks. However, their development in the physics domain is significantly hindered by the lack of a comprehensive benchmark. To fill this gap, we introduce OmniPhys, a large-scale benchmark for multimodal physics understanding and reasoning, covering middle school through university-level problems from Chinese Educational Corpora. OmniPhys consists of 15,246 questions and 19,850 images, accompanied by detailed annotations that support fine-grained analysis of reasoning processes and knowledge usage. Beyond conventional evaluation, OmniPhys is a benchmark that systematically evaluates multimodal outputs in the physics domain, including models' ability to generate structured physics diagrams, which constitute a fundamental component of authentic physics problem solving. Extensive evaluations reveal critical gaps in the capabilities of current MLLMs, especially in complex reasoning and visual generation. To address this, we release OmniPhys to serve as a foundational resource for advancing multimodal intelligence in physics and scientific domains. Codes and data are available at https://github.com/ECNU-RAIL/OmniPhys-EMNLP2026.

Figures & tables

Figure 1: Overview of the OmniPhys Dataset. The benchmark encompasses five major physics disciplines, illustrated by representative samples: Mechanics , Electromagnetism , Optics , Thermodynamics , and Acoustics .

Text-Only Benchmarks
Dataset	Size (Phys.)	Lang.	Education Stage				Image Modality		Overlap
Dataset	Size (Phys.)	Lang.	PS	MS	HS	Uni	Input	Output	Overlap
PhysQA Ding et al. (2023)	1,008	EN	✗	✓	✗	✗	✗	✗	0.00%
C-Eval Huang et al. (2023)	601	ZH	✗	✓	✓	✓	✗	✗	0.00%
MMLU Hendrycks et al. (2020)	548	EN	✗	✗	✓	✓	✗	✗	0.00%
GAOKAO Zhang et al. (2023)	111	ZH	✗	✗	✓	✗	✗	✗	0.00%

Table 1: Comparison with Existing Physics Datasets. OmniPhys distinguishes itself by supporting multimodal outputs and covering the full spectrum of educational stages. (Lang.: Language, PS: Primary School, MS: Middle School, HS: High School, Uni: University) Overlap (rightmost column) indicates the percentage of samples in OmniPhys that overlap with existing benchmarks, computed via string similarity and perceptual hashing. Our dataset maintains near-zero overlap while uniquely supporting multimodal outputs across all stages.

Physics Domains
Category	Count
Mechanics	7,240
Electromagnetism	5,325
Optics	1,120
Thermodynamics	856
Acoustics	705

Table 2: Statistics of OmniPhys. The benchmark maintains a high-quality distribution across three educational stages. In response to the growing need for generative AI, we specifically curated a subset for Multimodal Generation , featuring expert-annotated diagrams.

Proprietary / Closed-source MLLMs
Model	Acoustics			Optics			Mechanics			Thermodynamics			Electricity			Overall
Model	$S_{1}$	$S_{2}$	$S_{3}$	$S_{1}$	$S_{2}$	$S_{3}$	$S_{1}$	$S_{2}$	$S_{3}$	$S_{1}$	$S_{2}$	$S_{3}$	$S_{1}$	$S_{2}$	$S_{3}$	$S_{1}$	$S_{2}$	$S_{3}$	$P_{obj}$	$P_{open}$
GPT-5.2	67.48	81.65	67.78	65.15	79.73	64.42	66.33	79.67	72.30	69.89	82.13	73.84	60.15	77.05	66.34	64.15	78.85	69.42	39.36	29.45
GPT-5.1	58.68	72.24	55.40	52.77	66.09	49.32	52.30	67.91	54.47	59.87	73.50	52.04	45.48	64.07	50.95	50.25	66.68	52.59	23.53	12.87
GPT-4o	46.21	64.07	47.36	41.15	55.99	35.03	39.41	54.01	40.33	49.02	60.46	46.76	36.63	52.24	38.28	39.04	53.93	39.50	12.28	4.61
o4-mini	62.39	68.86	58.46	59.19	67.79	49.08	64.85	72.01	60.01	70.76	77.39	63.86	59.55	67.78	52.63	62.75	70.36	56.48	39.34	21.50

Table 3: Main Results on OmniPhys. Metrics:

S_{1}

(Answer Accuracy),

S_{2}

S_{3}

(Process Quality for Objective/Open-ended tasks), and

P_{obj}

P_{open}

(Strict Mastery Rates). Formatting : In each section (Closed-source MLLMs/Open-source MLLMs), the best score is bolded and the second best is underlined. The global best across all models is highlighted in blue .

Figure 2: Performance Degradation Across Educational Stages. We report the pass rates (%) of 12 representative MLLMs across Junior High, Senior High, and University levels. Both closed-source (Left) and open-source (Right) models exhibit a consistent downward trend, validating the hierarchical difficulty design of OmniPhys.

Model	Evaluation Scores
Model	Human Eval.	Model Eval.
Nano Banana	0.23	0.67
Doubao-Seedream-4.0	0.18	0.59
Gemini-3-pro-Image	0.29	0.73
GPT-Image-1	0.27	0.75
Qwen-Image-Edit-Plus	0.10	0.51

Table 4: Performance comparison in multimodal output settings.

Figure 3: A case study of multimodal outputs in OmniPhys dataset.

Model	Full Set	Test-Mini	Drop
Doubao-seed-1.6	76.81	22.45	-70.8%
Gemini-3-pro	80.41	34.66	-56.9%
GLM-4.6v	71.12	19.90	-72.0%
GPT-5.2	65.60	21.82	-66.7%
Qwen3-vl-plus	68.24	18.76	-72.5%
Average	72.44	23.52	-67.8%

Table 5: Comparison of Average Scores (scaled to 0-100) on the full dataset versus the Test-Mini set. The significant score decline (avg. -67.8%) confirms the elevated difficulty of the subset, validating its distinct research value as a rigorous probe for robust physical reasoning.

Multimodal Large Language Models (MLLMs)							Large Language Models (LLMs)
Model	Text+Img		Text+Caption		Text-only		Model	Text+Caption		Text-only
Model	$P_{obj}$	$P_{open}$	$P_{obj}$	$P_{open}$	$P_{obj}$	$P_{open}$	Model	$P_{obj}$	$P_{open}$	$P_{obj}$	$P_{open}$
GPT-5.2	2.78	3.02	4.95	1.08	5.82	0.86	Deepseek-V3.2	5.58	1.08	5.70	1.73
GPT-5.1	2.55	1.08	4.00	0.86	2.55	1.08	GPT-3.5-Turbo	1.94	0.63	1.45	0.22
o4-mini	7.23	1.96	6.55	0.86	6.79	1.73	Qwen2.5-32B	3.64	0.02	3.24	0.22
Gemini-3-Pro	25.82	9.50	16.12	4.10	15.88	6.26	Qwen2.5-7B	1.82	0.05	0.97	0.05

Table 6: Ablation study results comparing MLLMs (left) and LLMs (right). The columns are arranged by decreasing modal information: from Text+Image to Text-only. We report Objective (

P_{obj}

) and Open-ended (

P_{open}

) Accuracy.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Screenshot of the Label Studio interface for human evaluation of multimodal outputs.

Metric	Full Set	Test-Mini
Sample Size	12,885	1,288
Avg. Reasoning Length (chars)	563	979
Visual Necessity Score	8.98	8.99
All-Model Failure Rate	1.6%	15.5%
Avg. Model Accuracy	72.44%	23.52%

Appendix

Table 7: Comparison of key statistics between the full dataset and the selected Test-Mini subset. The subset demonstrates significantly higher complexity and lower model solvability.

Filtering Task	Metric	Human-Human	Human-Machine
Visual Dependency	Agreement (%)	92.6%	89.3%
Visual Dependency	Cohen’s $\kappa$	0.86	0.82
Difficulty Screening	Agreement (%)	94.1%	91.5%
Difficulty Screening	Cohen’s $\kappa$	0.88	0.84

Appendix

Table 8: Alignment Statistics for Data Refinement. The "Human-Machine" column represents the consistency between our automated pipeline (using MLLMs and vLLM) and expert consensus. Scores above 0.80 signify strong reliability in our data quality control.

Metric	Pearson $r$	Spearman $\rho$	Agreement (%)	$\Delta$ Score
$S_{1}$ (Answer Accuracy)	0.89	0.87	92.4%	+1.2%
$S_{2}$ (Obj. Process)	0.86	0.84	89.7%	+2.5%
$S_{3}$ (Subj. Process)	0.82	0.81	85.5%	+4.2%
$P_{obj}$ (Mastery)	0.92	0.91	93.1%	+0.8%
$P_{open}$ (Mastery)	0.88	0.86	91.2%	+1.5%

Appendix

Table 9: Human-Machine Alignment Results. Comparison between expert consensus and the LLM-as-a-Judge framework across primary metrics.

\Delta

Score represents the mean difference (Machine

-

Human).

Figure 5: A middle school, Mechanics example of Original Physics Problems.

Figure 6: A high school, Electromagnetism example of Original Physics Problems.

Figure 7: A University, Optics example of Original Physics Problems.

Multimodal Reasoning

Multimodal reasoning studies how AI systems combine text with images, audio, video, and other signals to solve evidence-dependent tasks.

See all 501 papers

PhysElite: How Far Are LLMs from Solving Olympiad-Level Physics Problems?
Aug 25, 2026Ruoran Xu, Wending Gao, Liyunfeng Chen +5Physical Reasoning Scientific Reasoning in Language Models
Xi’an Jiaotong-Liverpool University
Physics-R1: An Audited Olympiad Corpus and Recipe for Visual Physics Reasoning
May 13, 2026Shan YangVisual Reasoning RL for Language Model Reasoning
Independent Researcher
PhysFieldBench: Can Multimodal Models Understand Physical Fields?
Sep 28, 2026Yuezhou Ma, Huikun Weng, Jialong Wu +5Physical Reasoning Multimodal Large Language Models
School of Software, BNRist, Tsinghua University, China

Aug 25, 2026J/K move · Enter open · S save

PhysElite: How Far Are LLMs from Solving Olympiad-Level Physics Problems?

Ruoran Xu, Wending Gao, Liyunfeng Chen, Aixin Shi+4

Xi’an Jiaotong-Liverpool University

Physical Reasoning Scientific Reasoning in Language Models Multimodal Model Evaluation

Open paper AskPDF arXiv

Understanding how (multimodal) large language models perform on physics problems requires benchmarks that reflect the difficulty and breadth of expert-level physical reasoning. Existing physics benchmarks remain limited in the following two important ways: (1) short of high-difficulty datasets, and (2) lack of comprehensive coverage of visual forms, knowledge points, and step-by-step solution processes. As a result, model performance on current datasets may not be fully representative of their ability to solve complex physics problems. To address these issues, we present PhysElite, a large-scale bilingual multimodal benchmark for Olympiad-level physics reasoning. PhysElite contains 11,586 Olympiad-tier problems. For each problem, we provide corresponding visual diagrams, step-by-step bilingual Chinese-English solution derivations, and the final answer. We benchmark 18 open-source and closed-source MLLMs, and find that even the strongest model reaches only 33.7% answer accuracy. We additionally conduct step-level process evaluation to diagnose where models fail in the reasoning chain. Our datasets are released at https://huggingface.co/datasets/physelite/PhysElite.

Abstract

Multimodal Reasoning

Explore similar work

PhysElite: How Far Are LLMs from Solving Olympiad-Level Physics Problems?

Physics-R1: An Audited Olympiad Corpus and Recipe for Visual Physics Reasoning

PhysFieldBench: Can Multimodal Models Understand Physical Fields?