TheoPhysEval: Assessing AI agents at autonomous theoretical physics problem solving
Authors: A. Ramesh Chandra, Aaghaz Mahajan, Ainesh Sanyal, Mianqi Wang, Shubhadeep Chakraborty, Sudip Chakraborty, Shailesh Chandrasekharan, Rama Govindarajan, Akshit Goyal, Jonathan J. Heckman, Yasir Iqbal, Divya Jaganathan, Saumav Kapoor, Andreas Karch, Pushkar Khandare, Sandeep Krishna, Anupam Kundu, Alok Laddha, Tongyan Lin, Swarnendu Maity, Nilmani Mathur, Shani Nadir Meynet, Alessandro Mininno, Abhijeet Mishra, Vidur Niranjan, Abhishodh Prakash, Pratik Rath, Durbar Roy, Debargha Sarkar, Srikanth Sastry, Gary Shiu, Aninda Sinha, Thomas Van Riet, Praneeth Netrapalli, Suvrat Raju
Organizations: International Centre for Theoretical Sciences · Tata Institute of Fundamental Research · University of Texas at Austin · Kavli IPMU (WPI), The University of Tokyo · Harish-Chandra Research Institute · Homi Bhabha National Institute · Duke University · University of Pennsylvania · Indian Institute of Technology Madras · Harvard University · Jawaharlal Nehru Centre for Advanced Scientific Research · Chennai Mathematical Institute · University of California, San Diego · University of Wisconsin–Madison · Monash University · Indian Institute of Science · University of Calgary · KU Leuven · Google DeepMind
We present a suite of 50 unpublished research-level theoretical physics problems with verifiable answers. Without AI assistance, a typical problem in our collection could be solved by a world expert in about a day or less but would take considerably longer for experts in adjacent specialties. We test the ability of off-the-shelf AI agents to tackle these problems autonomously, without human hints. When given access to tools available in a realistic setting, including the ability to search the literature and write scripts for numerical and symbolic work, the best-performing agent, GPT-6-Astra harnessed in Codex, obtains the final answers with a near-perfect accuracy of 98% at an average cost under USD 3 per problem. On average, agents show only small changes in accuracy when denied web search and access to background research papers, suggesting significant memorization of the literature. These results suggest that state-of-the-art agents can accurately perform concrete calculations required across theoretical physics research. We comment on the broader implications of these findings and discuss possible explanations for the current lack of agent-led breakthroughs in theoretical physics despite their outstanding problem-solving abilities.
Figures & tables
Box 1: Three example problems and their expected solution trajectories.
Theme
Problems
Classical physics
5
Stochastic processes and biological physics
5
Nonequilibrium statistical mechanics
3
Lattice theories
3
Quantum phases of matter
5
Electronic structure theory
4
Table 1: Thematic distribution of problems
Box 2: An original question and an addition requesting a numerical answer.
Figure 3: Pass rates on the realistic track.
Figure 4: Pass rates on the isolated track.
Realistic
Isolated
Agent
Uncached input
Cached input
Output
Total
Uncached input
Cached input
Output
Total
Astra
2.93
33.55
0.92
37.41
1.87
21.66
1.00
24.53
Opus
11.78
192.25
7.15
211.18
18.13
203.09
7.21
228.43
Gemini
41.99
558.96
5.76
606.71
35.24
552.93
6.02
594.20
DeepSeek
12.84
989.73
23.02
1025.60
10.75
799.72
23.50
833.97
GLM
16.55
945.72
9.69
971.96
22.29
1079.79
13.52
1115.61
Table 2: Token usage for each agent and evaluation track, in millions of tokens. Rounded components may not sum exactly to the reported totals.
Agent
Realistic
Isolated
Astra
116.28
95.20
Opus
240.43
275.46
Gemini
95.01
90.48
DeepSeek
37.42
36.23
GLM
180 †
Table 3: Evaluation costs (USD) inferred from token usage. † GLM’s coding plan covered both tracks; API costs would have been USD 311.71 for the realistic track and USD 371.46 for the isolated track.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Numerical solution of the differential equation.
Problem
Astra
Opus
Gemini
DeepSeek
GLM
R
I
R
I
R
I
R
I
R
I
Boundary-layer perturbations ( § 1A.1.1 )
1
0
1
1
1
0
1
0
0
0
Autonomous equation ( § 1A.1.2 )
0
0
0
0
0
0
0
0
0
0
Inertial-particle attractors ( § 1A.1.3 )
1
1
0
1
0
0
0
0
0
0
Radiation damping ( § 1A.1.4 )
1
0
1
1
0
0
1
1
1
0
Two-body electrodynamics ( § 1A.1.5 )
1
1
1
1
1
1
0
0
0
0
Appendix
Table 4: Per-problem grades on the R and I tracks. A grade of 1 denotes a pass and 0 denotes a failure.
R
I
Problem
Uncached
Cached
Output
Total
Uncached
Cached
Output
Total
Boundary-layer perturbations ( § 1A.1.1 )
0.092
0.632
0.007
0.730
0.017
0.093
0.007
0.117
Autonomous equation ( § 1A.1.2 )
0.019
0.060
0.003
0.082
0.018
0.050
0.005
0.072
Inertial-particle attractors ( § 1A.1.3 )
0.128
5.588
0.062
5.778
0.115
3.241
0.074
3.430
Radiation damping ( § 1A.1.4 )
0.074
1.052
0.036
1.163
0.029
0.155
0.016
0.199
Two-body electrodynamics ( § 1A.1.5 )
0.040
0.214
0.012
0.266
0.027
0.173
0.014
0.215
Appendix
Table 5: Token usage by problem for Astra (millions).
R
I
Problem
Uncached
Cached
Output
Total
Uncached
Cached
Output
Total
Boundary-layer perturbations ( § 1A.1.1 )
0.288
0.902
0.084
1.274
0.423
2.098
0.110
2.631
Autonomous equation ( § 1A.1.2 )
1.175
10.628
0.436
12.240
1.301
9.117
0.383
10.801
Inertial-particle attractors ( § 1A.1.3 )
0.882
33.643
0.266
34.791
0.973
6.757
0.183
7.913
Radiation damping ( § 1A.1.4 )
0.216
5.232
0.185
5.633
0.331
4.575
0.183
5.089
Two-body electrodynamics ( § 1A.1.5 )
0.110
1.298
0.095
1.502
0.168
4.230
0.146
4.544
Appendix
Table 6: Token usage by problem for Opus (millions).
R
I
Problem
Uncached
Cached
Output
Total
Uncached
Cached
Output
Total
Boundary-layer perturbations ( § 1A.1.1 )
1.778
11.668
0.100
13.546
0.436
6.451
0.096
6.983
Autonomous equation ( § 1A.1.2 )
1.129
11.698
0.145
12.973
0.358
2.162
0.065
2.585
Inertial-particle attractors ( § 1A.1.3 )
1.459
20.230
0.119
21.807
1.981
20.144
0.303
22.428
Radiation damping ( § 1A.1.4 )
1.236
11.749
0.248
13.232
3.486
67.675
0.289
71.451
Two-body electrodynamics ( § 1A.1.5 )
0.476
5.943
0.127
6.546
0.677
4.590
0.087
5.355
Appendix
Table 7: Token usage by problem for Gemini (millions).
R
I
Problem
Uncached
Cached
Output
Total
Uncached
Cached
Output
Total
Boundary-layer perturbations ( § 1A.1.1 )
0.124
7.582
0.081
7.787
0.067
10.552
0.156
10.776
Autonomous equation ( § 1A.1.2 )
0.101
4.659
0.275
5.036
0.051
6.876
0.275
7.202
Inertial-particle attractors ( § 1A.1.3 )
0.589
53.369
0.707
54.665
1.278
94.423
1.928
97.629
Radiation damping ( § 1A.1.4 )
0.728
57.747
1.480
59.955
0.703
62.523
1.367
64.593
Two-body electrodynamics ( § 1A.1.5 )
0.055
16.744
0.183
16.982
0.016
9.047
0.195
9.258
Appendix
Table 8: Token usage by problem for DeepSeek (millions). * This task was cut off for exceeding the time limit and did not record token usage.
R
I
Problem
Uncached
Cached
Output
Total
Uncached
Cached
Output
Total
Boundary-layer perturbations ( § 1A.1.1 )
0.082
2.196
0.074
2.352
0.054
0.092
0.052
0.199
Autonomous equation ( § 1A.1.2 )
0.252
4.458
0.141
4.851
0.648
8.215
0.231
9.094
Inertial-particle attractors ( § 1A.1.3 )
3.837
61.382
0.332
65.551
7.213
87.645
0.350
95.207
Radiation damping ( § 1A.1.4 )
0.525
149.820
0.456
150.802
0.383
41.430
0.440
42.253
Two-body electrodynamics ( § 1A.1.5 )
0.122
2.330
0.113
2.565
0.159
3.929
0.152
4.240
Appendix
Table 9: Token usage by problem for GLM (millions).
Autonomous research agents use language models and tools to search literature, design and run experiments, analyze results, and support scientific discovery.
AI agents are playing an increasingly important role in physics research. However, existing physics benchmarks only evaluate fixed questions, isolated sub-tasks, or paper reproduction with given solutions, and rarely check research quality alongside correctness. We introduce FrontierPhysics, an end-to-end benchmark with 56 tasks made by 42 physicists based on their own research, covering nine domains across theoretical, computational, and experimental physics. Each task requires agents to review the literature, develop a research plan, conduct research and experiments, and produce structured outputs and scientific manuscripts. Combining deterministic verifiers with expert-authored rubrics, FrontierPhysics evaluates whether agents' research is correct and publishable. Across 22 models in 26 configurations, no configuration passes more than 46 of its 168 rollouts. Manual review of agent trajectories shows that frontier agents' main weaknesses are insufficiently rigorous reasoning and limited expert intuition, rather than gaps in physics knowledge or execution errors.
AI agents powered by large language models exhibit strong reasoning and problem-solving capabilities, enabling them to assist scientific research tasks such as formula derivation and code generation. However, whether these agents can reliably perform end-to-end reproduction from real scientific papers remains an open question. We introduce PRBench, a benchmark of 30 expert-curated tasks spanning 11 subfields of physics. Each task requires an agent to comprehend the methodology of a published paper, implement the corresponding algorithms from scratch, and produce quantitative results matching the original publication. Agents are provided only with the task instruction and paper content, and operate in a sandboxed execution environment. All tasks are contributed by domain experts from over 20 research groups at the School of Physics, Peking University, each grounded in a real published paper and validated through end-to-end reproduction with verified ground-truth results and detailed scoring rubrics. Using an agentified assessment pipeline, we evaluate a set of coding agents on PRBench and analyze their capabilities across key dimensions of scientific reasoning and execution. The best-performing agent, OpenAI Codex powered by GPT-5.3-Codex, achieves a mean overall score of 34%. All agents exhibit a zero end-to-end callback success rate, with particularly poor performance in data accuracy and code correctness. We further identify systematic failure modes, including errors in formula implementation, inability to debug numerical simulations, and fabrication of output data. Overall, PRBench provides a rigorous benchmark for evaluating progress toward autonomous scientific research.
The next frontier for artificial general intelligence is tackling unresolved scientific problems, calling for benchmarks that assess progress beyond established knowledge. We introduce OpenProblemBench, a benchmark of 82 unresolved problems drawn from the mathematics and theoretical physics literature. Each problem supplies the research context, assumptions, and prior progress needed to investigate the question. We select problems whose proposed solutions admit comparatively clear checks of their decisive mathematical or computational claims. Four evaluator models independently assess the correctness, completeness, and degree of progress of each submission without reference solutions. Across seven evaluated configurations, GPT-6-Astra achieves the highest mean judged solve rate of 14.0%, compared with 5.5-6.7% for the evaluated full-size open models and 2.4-3.7% for Flash models. Case comparisons connect stronger outcomes to changes in problem representation, general arguments that extend beyond finite evidence, and proofs of the steps needed to complete a solution. By grounding evaluation in questions arising from the research literature, OpenProblemBench provides a setting for investigating the capabilities and limitations of AI as a contributor to foundational theoretical science.
Zhiyi Li, Sihan Hu, Tianning Xiao +4
Department of Modern Physics, University of Science and Technology of China, Hefei, Anhui 230026, China · CAS Key Laboratory of Theoretical Physics, Institute of Theoretical Physics, Chinese Academy of Sciences, Beijing 100190, China · Hefei National Laboratory, University of Science and Technology of China, Hefei 230088, China +3