Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving

Kaiyue Feng; Yilun Zhao; Yixin Liu; Tianyu Yang; Chen Zhao; John Sous; Arman Cohan

cs.AIMar 26, 2025

Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving

Authors: Kaiyue Feng, Yilun Zhao, Yixin Liu, Tianyu Yang, Chen Zhao, John Sous, Arman Cohan

Organizations: New York University · Yale University · Notre Dame University

Abstract

We introduce Physics, a benchmark for evaluating foundation models on advanced physics problem solving. It contains 1034 annotated, open-ended problems from PhD-qualifying exams across six core physics domains. Solving these problems requires applying domain-specific knowledge to physical setups and carrying out multi-step mathematical reasoning. Each problem includes a worked solution and independently checked reference answers. An LLM-based grading protocol evaluates individual answers and complete multi-part problems, supporting numeric, symbolic, and text responses. We evaluate 53 proprietary and open-weight models. On test problems without figures, high accuracy is concentrated in a small group of thinking models, with the strongest open-weight models approaching proprietary performance. Most open-weight models still solve fewer than half of these problems. Our analysis of Qwen3.5-9B reveals failures in applying physics knowledge and interpreting figures, with misread figures affecting several subquestions.

Figures & tables

Panel of Figure 1 — Figure 1: An example problem from Physics (Electromagnetism, three subquestions and four reference answers). The solution shows the derivation; answer labels and types support automated grading.

Figure 2: Overview of Physics . The pie chart uses the domain proportions of the final 1,034 problems. Annotated PhD-qualifying-exam problems pass through blind solving, auditing, and independent review (§ 3.3 ), then models produce structured responses (§ 4.1 ) for per-answer LLM grading (§ 4.2 ).

Figure 3: Dataset statistics of Physics . The panels show the distribution of answers per problem (left; percentage of problems) and answer types (right; percentage of answers). Text lengths count whitespace-delimited tokens in the raw question and solution strings, including LaTeX expressions.

Benchmark	Multi-modal	Size	Level	Question Type	Evaluation
JEEBench Arora et al. (2023)	✗	515	CEE	OE, MC	Rule-Based
MATH Hendrycks et al. (2021)	✗	12,500	K12–Comp	OE	Rule-Based
HARDMath Fan et al. (2025)	✗	1,466	Graduate	OE	Rule + Model
GSM8K Cobbe et al. (2021)	✗	8,500	K8	OE	Rule-Based
GPQA Rein et al. (2023)	✗	448	Graduate	MC	Rule-Based
SciQ Welbl et al. (2017)	✗	13,679	K4–K8	MC, OE	Rule-Based

Table 1: Comparison of Physics with related benchmarks. For Level , K1–K12: Elementary through High School, K4–K8: Grades 4–8, K8: up to Grade 8, HS: High School, CEE: University Entrance Examination, Comp: Competition, University: Undergraduate, PhD: PhD Qualifying Exams. For Question Type , OE: Open-ended Questions, MC: Multiple-choice Questions. Sizes use the sets and counting units indicated by the cited sources; multi-subject benchmarks are not restricted to their physics subsets.

Model	Think	AMO	E&M	CM	Opt.	QM	Stats.	Val	Test (w/ fig.)	Test
		(101)	(119)	(83)	(51)	(127)	(121)	(184)	(794)	(602)
Proprietary Models
GPT-6 Astra	✓	94.1	92.4	89.2	80.4	91.3	89.3	92.4	91.1	90.4
GPT-6 Sol	✓	93.1	92.4	89.2	86.3	88.2	89.3	90.2	90.6	90.0
GPT-5.6 Sol	✓	90.1	93.3	89.2	84.3	90.6	88.4	90.8	90.4	89.9
GPT-6 Luna	✓	83.2	85.7	85.5	78.4	83.5	78.5	83.2	82.6	82.7

Table 2: Strict accuracy (%). Domain columns and Test: 602 test problems without figures; Val: 184 validation problems without figures; Test (w/ fig.): all 794 test problems. Models are sorted by Test within groups; † marks text-only models (– for Test (w/ fig.)). Domains: AMO (Atomic Physics), E&M (Electromagnetism), CM (Classical Mechanics), Opt. (Optics), QM (Quantum Mechanics), Stats. (Thermodynamics and Statistical Mechanics).

Figure 4: Strict accuracy on test problems without figures by thinking status. Circles denote open-weight models, diamonds proprietary models. Squares show group means with model-resampling 95% intervals.

Figure 5: Strict accuracy on problems with versus without figures for each multimodal model. The dashed line denotes equal scores; the inset separates the overlapping proprietary-model points near the ceiling.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Year	Major	Assigned Subject(s)
1	2nd Year Undergraduate	Physics	Classical Mechanics, Electromagnetism
2	3rd Year Undergraduate	Physics	Quantum Mechanics, Optics
3	2nd Year Undergraduate	Theoretical Physics	Quantum Mechanics, Thermodynamics and Statistical Mechanics
4	3rd Year Undergraduate	Applied Physics	Thermodynamics and Statistical Mechanics, Atomic Physics
5	2nd Year Undergraduate	Engineering Physics	Electromagnetism, Classical Mechanics
6	3rd Year Undergraduate	Physics	Thermodynamics and Statistical Mechanics, Atomic Physics

Appendix

Table 3: Biographies of 7 annotators involved in the Physics benchmark construction.

Population	Problems	Strict (%)	CI low	CI high	Gap (pp)
Val (no fig.)	184	82.1	76.6	87.0	6.5
Val (all)	240	81.2	75.8	85.8	6.2
Test (no fig.)	602	80.6	77.4	83.7	5.0
Test (all)	794	80.5	77.8	83.4	4.8

Appendix

Table 4: GLM-5.3-Flash strict accuracy when judged by DeepSeek-V4.1-Flash. CI: 95% bootstrap interval over problems. Gap: GLM-5.3-Flash-judged minus DeepSeek-V4.1-Flash-judged strict accuracy on the same population, in percentage points .

Model	Think	Val (no fig.)			Test (no fig.)			Test (all)
		(184 problems)			(602 problems)			(794 problems)
		S	A	P	S	A	P	S	A	P
Proprietary Models
GPT-6 Astra	✓	92.4	96.2	96.1	90.4	94.9	94.6	91.1	95.2	95.0
GPT-6 Sol	✓	90.2	96.0	94.9	90.0	95.1	94.2	90.6	95.1	94.6
GPT-5.6 Sol	✓	90.8	95.8	95.0	89.9	94.9	93.5	90.4	95.0	93.8

Appendix

Table 5: Full results for all 53 models (%). S: strict accuracy, requiring every answer in a problem to be correct. A: answer-level accuracy, pooling all answers. P: partial credit, averaging the fraction of correct answers within each problem with equal problem weights. Val and Test (no fig.) exclude figure problems; Test (all) includes test problems with and without figures and applies only to multimodal models; † marks text-only models, with – for Test (all). Think uses ✓ /✗ for thinking/other models. Models are ordered within each group by descending strict Test (no fig.) accuracy .

Figure 6: Figure-free test strict accuracy versus total parameters (log scale) within five model families. The hollow marker for Qwen3.5-9B uses its nominal parameter count. Gemma E models use total parameters. Qwen3.5’s largest model is MoE; the Llama pair changes generation.

Older	Mapped successor	Older (%)	New (%)	Gain (pp) [95% CI]
Qwen2.5-Math-1.5B-Instruct	Qwen3.5-2B	8.3	20.9	+12.6 [9.5, 15.9]
gemma-2-2b-it	gemma-4-E2B-it	4.2	22.9	+18.8 [15.3, 22.3]
gemma-2-9b-it	gemma-4-E4B-it	9.6	35.5	+25.9 [22.3, 29.9]
InternLM3-8B-Instruct-AWQ	Intern-S1-mini	8.6	37.7	+29.1 [25.4, 33.2]
Phi-4	Phi-4-Reasoning-Vision-15B	28.6	15.8	$-$ 12.8 [ $-$ 16.1, $-$ 9.6]
DeepSeek-R1-Distill-Qwen-32B	DeepSeek-R1-0528-Qwen3-8B	42.2	45.8	+3.7 [ $-$ 0.3, 7.6]

Appendix

Table 6: Successive checkpoints from the same developer on figure-free test problems. Pairs share successors and often change size, modality or thinking. Intervals resample problems with checkpoints fixed.

Figure 7: Mean strict accuracy within short, middle, and long completed-response tertiles, averaged over 21 thinking models, with problem-resampling 95% intervals.

Panel of Figure 8 — Figure 8: Domain-level analyses on figure-free test problems.

Figure 9: Figure-free test strict accuracy versus share of responses that stopped at the output token limit. The symmetric-log axis has a linear threshold of 0.1%; color denotes thinking status, diamonds the four OpenAI models. Aria (gray square) has a separate marker for its post-answer run-on and is excluded from the non-thinking truncation pool .

Figure 10: The figure given with the question. The inner radius is labeled $b$ and the outer radius $a$ .

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,364 papers

PhysElite: How Far Are LLMs from Solving Olympiad-Level Physics Problems?
Aug 25, 2026Ruoran Xu, Wending Gao, Liyunfeng Chen +5Physical Reasoning Scientific Reasoning in Language Models
Xi’an Jiaotong-Liverpool University
Assessing AI in Introductory Physics Problem Solving
Jul 15, 2026Amir Bralin, N. Sanjay RebelloLLM Evaluation Multimodal Large Language Models
Texas Tech University, Lubbock, TX · Purdue University, West Lafayette, IN
How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks
Sep 11, 2026Ali Ansari, Haoran Sun, Andy Zeyi Liu +48LLM Evaluation Physical Reasoning

Aug 25, 2026J/K move · Enter open · S save

PhysElite: How Far Are LLMs from Solving Olympiad-Level Physics Problems?

Ruoran Xu, Wending Gao, Liyunfeng Chen, Aixin Shi+4

Xi’an Jiaotong-Liverpool University

Physical Reasoning Scientific Reasoning in Language Models Multimodal Model Evaluation

Open paper AskPDF arXiv

Understanding how (multimodal) large language models perform on physics problems requires benchmarks that reflect the difficulty and breadth of expert-level physical reasoning. Existing physics benchmarks remain limited in the following two important ways: (1) short of high-difficulty datasets, and (2) lack of comprehensive coverage of visual forms, knowledge points, and step-by-step solution processes. As a result, model performance on current datasets may not be fully representative of their ability to solve complex physics problems. To address these issues, we present PhysElite, a large-scale bilingual multimodal benchmark for Olympiad-level physics reasoning. PhysElite contains 11,586 Olympiad-tier problems. For each problem, we provide corresponding visual diagrams, step-by-step bilingual Chinese-English solution derivations, and the final answer. We benchmark 18 open-source and closed-source MLLMs, and find that even the strongest model reaches only 33.7% answer accuracy. We additionally conduct step-level process evaluation to diagnose where models fail in the reasoning chain. Our datasets are released at https://huggingface.co/datasets/physelite/PhysElite.

Abstract

LLM Evaluation

Explore similar work

PhysElite: How Far Are LLMs from Solving Olympiad-Level Physics Problems?

Assessing AI in Introductory Physics Problem Solving

How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks