We introduce Physics, a benchmark for evaluating foundation models on advanced physics problem solving. It contains 1034 annotated, open-ended problems from PhD-qualifying exams across six core physics domains. Solving these problems requires applying domain-specific knowledge to physical setups and carrying out multi-step mathematical reasoning. Each problem includes a worked solution and independently checked reference answers. An LLM-based grading protocol evaluates individual answers and complete multi-part problems, supporting numeric, symbolic, and text responses. We evaluate 53 proprietary and open-weight models. On test problems without figures, high accuracy is concentrated in a small group of thinking models, with the strongest open-weight models approaching proprietary performance. Most open-weight models still solve fewer than half of these problems. Our analysis of Qwen3.5-9B reveals failures in applying physics knowledge and interpreting figures, with misread figures affecting several subquestions.
Figures & tables
Figure 1: An example problem from Physics (Electromagnetism, three subquestions and four reference answers). The solution shows the derivation; answer labels and types support automated grading.
Figure 2: Overview of Physics . The pie chart uses the domain proportions of the final 1,034 problems. Annotated PhD-qualifying-exam problems pass through blind solving, auditing, and independent review (§ 3.3 ), then models produce structured responses (§ 4.1 ) for per-answer LLM grading (§ 4.2 ).
Figure 3: Dataset statistics of Physics . The panels show the distribution of answers per problem (left; percentage of problems) and answer types (right; percentage of answers). Text lengths count whitespace-delimited tokens in the raw question and solution strings, including LaTeX expressions.
Benchmark
Multi-modal
Size
Level
Question Type
Evaluation
JEEBench Arora et al. (2023)
✗
515
CEE
OE, MC
Rule-Based
MATH Hendrycks et al. (2021)
✗
12,500
K12–Comp
OE
Rule-Based
HARDMath Fan et al. (2025)
✗
1,466
Graduate
OE
Rule + Model
GSM8K Cobbe et al. (2021)
✗
8,500
K8
OE
Rule-Based
GPQA Rein et al. (2023)
✗
448
Graduate
MC
Rule-Based
SciQ Welbl et al. (2017)
✗
13,679
K4–K8
MC, OE
Rule-Based
Table 1: Comparison of Physics with related benchmarks. For Level , K1–K12: Elementary through High School, K4–K8: Grades 4–8, K8: up to Grade 8, HS: High School, CEE: University Entrance Examination, Comp: Competition, University: Undergraduate, PhD: PhD Qualifying Exams. For Question Type , OE: Open-ended Questions, MC: Multiple-choice Questions. Sizes use the sets and counting units indicated by the cited sources; multi-subject benchmarks are not restricted to their physics subsets.
Model
Think
AMO
E&M
CM
Opt.
QM
Stats.
Val
Test (w/ fig.)
Test
(101)
(119)
(83)
(51)
(127)
(121)
(184)
(794)
(602)
Proprietary Models
GPT-6 Astra
✓
94.1
92.4
89.2
80.4
91.3
89.3
92.4
91.1
90.4
GPT-6 Sol
✓
93.1
92.4
89.2
86.3
88.2
89.3
90.2
90.6
90.0
GPT-5.6 Sol
✓
90.1
93.3
89.2
84.3
90.6
88.4
90.8
90.4
89.9
GPT-6 Luna
✓
83.2
85.7
85.5
78.4
83.5
78.5
83.2
82.6
82.7
Table 2: Strict accuracy (%). Domain columns and Test: 602 test problems without figures; Val: 184 validation problems without figures; Test (w/ fig.): all 794 test problems. Models are sorted by Test within groups; † marks text-only models (– for Test (w/ fig.)). Domains: AMO (Atomic Physics), E&M (Electromagnetism), CM (Classical Mechanics), Opt. (Optics), QM (Quantum Mechanics), Stats. (Thermodynamics and Statistical Mechanics).
Figure 4: Strict accuracy on test problems without figures by thinking status. Circles denote open-weight models, diamonds proprietary models. Squares show group means with model-resampling 95% intervals.
Figure 5: Strict accuracy on problems with versus without figures for each multimodal model. The dashed line denotes equal scores; the inset separates the overlapping proprietary-model points near the ceiling.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
ID
Year
Major
Assigned Subject(s)
1
2nd Year Undergraduate
Physics
Classical Mechanics, Electromagnetism
2
3rd Year Undergraduate
Physics
Quantum Mechanics, Optics
3
2nd Year Undergraduate
Theoretical Physics
Quantum Mechanics, Thermodynamics and Statistical Mechanics
4
3rd Year Undergraduate
Applied Physics
Thermodynamics and Statistical Mechanics, Atomic Physics
5
2nd Year Undergraduate
Engineering Physics
Electromagnetism, Classical Mechanics
6
3rd Year Undergraduate
Physics
Thermodynamics and Statistical Mechanics, Atomic Physics
Appendix
Table 3: Biographies of 7 annotators involved in the Physics benchmark construction.
Population
Problems
Strict (%)
CI low
CI high
Gap (pp)
Val (no fig.)
184
82.1
76.6
87.0
6.5
Val (all)
240
81.2
75.8
85.8
6.2
Test (no fig.)
602
80.6
77.4
83.7
5.0
Test (all)
794
80.5
77.8
83.4
4.8
Appendix
Table 4: GLM-5.3-Flash strict accuracy when judged by DeepSeek-V4.1-Flash. CI: 95% bootstrap interval over problems. Gap: GLM-5.3-Flash-judged minus DeepSeek-V4.1-Flash-judged strict accuracy on the same population, in percentage points .
Model
Think
Val (no fig.)
Test (no fig.)
Test (all)
(184 problems)
(602 problems)
(794 problems)
S
A
P
S
A
P
S
A
P
Proprietary Models
GPT-6 Astra
✓
92.4
96.2
96.1
90.4
94.9
94.6
91.1
95.2
95.0
GPT-6 Sol
✓
90.2
96.0
94.9
90.0
95.1
94.2
90.6
95.1
94.6
GPT-5.6 Sol
✓
90.8
95.8
95.0
89.9
94.9
93.5
90.4
95.0
93.8
Appendix
Table 5: Full results for all 53 models (%). S: strict accuracy, requiring every answer in a problem to be correct. A: answer-level accuracy, pooling all answers. P: partial credit, averaging the fraction of correct answers within each problem with equal problem weights. Val and Test (no fig.) exclude figure problems; Test (all) includes test problems with and without figures and applies only to multimodal models; † marks text-only models, with – for Test (all). Think uses ✓ /✗ for thinking/other models. Models are ordered within each group by descending strict Test (no fig.) accuracy .
Figure 6: Figure-free test strict accuracy versus total parameters (log scale) within five model families. The hollow marker for Qwen3.5-9B uses its nominal parameter count. Gemma E models use total parameters. Qwen3.5’s largest model is MoE; the Llama pair changes generation.
Older
Mapped successor
Older (%)
New (%)
Gain (pp) [95% CI]
Qwen2.5-Math-1.5B-Instruct
Qwen3.5-2B
8.3
20.9
+12.6 [9.5, 15.9]
gemma-2-2b-it
gemma-4-E2B-it
4.2
22.9
+18.8 [15.3, 22.3]
gemma-2-9b-it
gemma-4-E4B-it
9.6
35.5
+25.9 [22.3, 29.9]
InternLM3-8B-Instruct-AWQ
Intern-S1-mini
8.6
37.7
+29.1 [25.4, 33.2]
Phi-4
Phi-4-Reasoning-Vision-15B
28.6
15.8
− 12.8 [ − 16.1, − 9.6]
DeepSeek-R1-Distill-Qwen-32B
DeepSeek-R1-0528-Qwen3-8B
42.2
45.8
+3.7 [ − 0.3, 7.6]
Appendix
Table 6: Successive checkpoints from the same developer on figure-free test problems. Pairs share successors and often change size, modality or thinking. Intervals resample problems with checkpoints fixed.
Figure 7: Mean strict accuracy within short, middle, and long completed-response tertiles, averaged over 21 thinking models, with problem-resampling 95% intervals.
Figure 8: Domain-level analyses on figure-free test problems.
Figure 9: Figure-free test strict accuracy versus share of responses that stopped at the output token limit. The symmetric-log axis has a linear threshold of 0.1%; color denotes thinking status, diamonds the four OpenAI models. Aria (gray square) has a separate marker for its post-answer run-on and is excluded from the non-thinking truncation pool .
Figure 10: The figure given with the question. The inner radius is labeled b and the outer radius a .
Understanding how (multimodal) large language models perform on physics problems requires benchmarks that reflect the difficulty and breadth of expert-level physical reasoning. Existing physics benchmarks remain limited in the following two important ways: (1) short of high-difficulty datasets, and (2) lack of comprehensive coverage of visual forms, knowledge points, and step-by-step solution processes. As a result, model performance on current datasets may not be fully representative of their ability to solve complex physics problems. To address these issues, we present PhysElite, a large-scale bilingual multimodal benchmark for Olympiad-level physics reasoning. PhysElite contains 11,586 Olympiad-tier problems. For each problem, we provide corresponding visual diagrams, step-by-step bilingual Chinese-English solution derivations, and the final answer. We benchmark 18 open-source and closed-source MLLMs, and find that even the strongest model reaches only 33.7% answer accuracy. We additionally conduct step-level process evaluation to diagnose where models fail in the reasoning chain. Our datasets are released at https://huggingface.co/datasets/physelite/PhysElite.
Reasoning or inference-scaling models are the new generation of Large Language Models (LLMs) capable of complex problem solving. To investigate their problem-solving capability in physics, we evaluated model o4-mini by OpenAI on solving traditional, end-of-chapter problems from Halliday and Resnick's "Fundamentals of Physics," spanning core topics in the undergraduate physics curriculum. Performance was analyzed across modality and problem difficulty. The model solved the problems with overall accuracy of about 90%, but performance depended strongly on representation: accuracy was much higher on text-only problems (96%) than on problems requiring coordinated interpretation of text and images (79%). Accuracy also declined significantly as the problem difficulty increased from low to medium to high. These results show that state-of-the-art LLMs can solve much of the standard introductory physics problems, but that their performance remains uneven and constrained by problem modality and problem difficulty.
Amir Bralin, N. Sanjay Rebello
Texas Tech University, Lubbock, TX · Purdue University, West Lafayette, IN
Low reported scores on leading physics benchmarks, including those featured in the Artificial Analysis Intelligence Index (2026), suggest that frontier language models still struggle with advanced physics, a demanding test of their scientific reasoning and quantitative problem-solving abilities. Yet this impression does not always align with domain experts' experiences using these models in their work. We revisit these reported findings by evaluating frontier models on six widely used physics benchmarks and auditing them with experts, focusing on text-only problems with verifiable final answers. For each subfield of physics, faculty and graduate researchers with relevant expertise carefully review problem statements, reference solutions, and model responses to distinguish genuine model errors from grader errors, incorrect reference solutions, and ambiguous or underspecified questions. Most audited cases initially evaluated as incorrect reflect these benchmarking issues rather than errors in the models' physics reasoning. We then ask experts to address these benchmarking issues by correcting erroneous reference solutions and repairing or excluding flawed questions. We find that GPT-5.6-Sol's measured mean@4 rises from 47.3% to 78.7% on HLE-Physics and from 61.0% to 87.2% on CMT-Benchmark, while its corrected pass@4 reaches 94.4% on the 54 retained CritPt challenges. Corrected scores are computed on the retained evaluation subsets following expert review. Scores on the audited subsets of UGPhysics, PRISM-Physics, and PHYBench also rise substantially after correction. These findings suggest that current benchmarks substantially understate frontier models' ability to solve well-posed physics problems. Near-saturation on these closed-ended tasks highlights the need for more demanding, expert-validated evaluations.