TheoPhysEval: Assessing AI agents at autonomous theoretical physics problem solving
Authors: A. Ramesh Chandra, Aaghaz Mahajan, Ainesh Sanyal, Mianqi Wang, Shubhadeep Chakraborty, Sudip Chakraborty, Shailesh Chandrasekharan, Rama Govindarajan, Akshit Goyal, Jonathan J. Heckman, Yasir Iqbal, Divya Jaganathan, Saumav Kapoor, Andreas Karch, Pushkar Khandare, Sandeep Krishna, Anupam Kundu, Alok Laddha, Tongyan Lin, Swarnendu Maity, Nilmani Mathur, Shani Nadir Meynet, Alessandro Mininno, Abhijeet Mishra, Vidur Niranjan, Abhishodh Prakash, Pratik Rath, Durbar Roy, Debargha Sarkar, Srikanth Sastry, Gary Shiu, Aninda Sinha, Thomas Van Riet, Praneeth Netrapalli, Suvrat Raju
Organizations: International Centre for Theoretical Sciences · Tata Institute of Fundamental Research · University of Texas at Austin · Kavli IPMU (WPI), The University of Tokyo · Harish-Chandra Research Institute · Homi Bhabha National Institute · Duke University · University of Pennsylvania · Indian Institute of Technology Madras · Harvard University · Jawaharlal Nehru Centre for Advanced Scientific Research · Chennai Mathematical Institute · University of California, San Diego · University of Wisconsin–Madison · Monash University · Indian Institute of Science · University of Calgary · KU Leuven · Google DeepMind
We present a suite of 50 unpublished research-level theoretical physics problems with verifiable answers. Without AI assistance, a typical problem in our collection could be solved by a world expert in about a day or less but would take considerably longer for experts in adjacent specialties. We test the ability of off-the-shelf AI agents to tackle these problems autonomously, without human hints. When given access to tools available in a realistic setting, including the ability to search the literature and write scripts for numerical and symbolic work, the best-performing agent, GPT-6-Astra harnessed in Codex, obtains the final answers with a near-perfect accuracy of 98% at an average cost under USD 3 per problem. On average, agents show only small changes in accuracy when denied web search and access to background research papers, suggesting significant memorization of the literature. These results suggest that state-of-the-art agents can accurately perform concrete calculations required across theoretical physics research. We comment on the broader implications of these findings and discuss possible explanations for the current lack of agent-led breakthroughs in theoretical physics despite their outstanding problem-solving abilities.
Figures & tables
Box 1: Three example problems and their expected solution trajectories.
Theme
Problems
Classical physics
5
Stochastic processes and biological physics
5
Nonequilibrium statistical mechanics
3
Lattice theories
3
Quantum phases of matter
5
Electronic structure theory
4
Table 1: Thematic distribution of problems
Box 2: An original question and an addition requesting a numerical answer.
Figure 3: Pass rates on the realistic track.
Figure 4: Pass rates on the isolated track.
Realistic
Isolated
Agent
Uncached input
Cached input
Output
Total
Uncached input
Cached input
Output
Total
Astra
2.93
33.55
0.92
37.41
1.87
21.66
1.00
24.53
Opus
11.78
192.25
7.15
211.18
18.13
203.09
7.21
228.43
Gemini
41.99
558.96
5.76
606.71
35.24
552.93
6.02
594.20
DeepSeek
12.84
989.73
23.02
1025.60
10.75
799.72
23.50
833.97
GLM
16.55
945.72
9.69
971.96
22.29
1079.79
13.52
1115.61
Table 2: Token usage for each agent and evaluation track, in millions of tokens. Rounded components may not sum exactly to the reported totals.
Agent
Realistic
Isolated
Astra
116.28
95.20
Opus
240.43
275.46
Gemini
95.01
90.48
DeepSeek
37.42
36.23
GLM
180 †
Table 3: Evaluation costs (USD) inferred from token usage. † GLM’s coding plan covered both tracks; API costs would have been USD 311.71 for the realistic track and USD 371.46 for the isolated track.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Numerical solution of the differential equation.
Problem
Astra
Opus
Gemini
DeepSeek
GLM
R
I
R
I
R
I
R
I
R
I
Boundary-layer perturbations ( § 1A.1.1 )
1
0
1
1
1
0
1
0
0
0
Autonomous equation ( § 1A.1.2 )
0
0
0
0
0
0
0
0
0
0
Inertial-particle attractors ( § 1A.1.3 )
1
1
0
1
0
0
0
0
0
0
Radiation damping ( § 1A.1.4 )
1
0
1
1
0
0
1
1
1
0
Two-body electrodynamics ( § 1A.1.5 )
1
1
1
1
1
1
0
0
0
0
Appendix
Table 4: Per-problem grades on the R and I tracks. A grade of 1 denotes a pass and 0 denotes a failure.
R
I
Problem
Uncached
Cached
Output
Total
Uncached
Cached
Output
Total
Boundary-layer perturbations ( § 1A.1.1 )
0.092
0.632
0.007
0.730
0.017
0.093
0.007
0.117
Autonomous equation ( § 1A.1.2 )
0.019
0.060
0.003
0.082
0.018
0.050
0.005
0.072
Inertial-particle attractors ( § 1A.1.3 )
0.128
5.588
0.062
5.778
0.115
3.241
0.074
3.430
Radiation damping ( § 1A.1.4 )
0.074
1.052
0.036
1.163
0.029
0.155
0.016
0.199
Two-body electrodynamics ( § 1A.1.5 )
0.040
0.214
0.012
0.266
0.027
0.173
0.014
0.215
Appendix
Table 5: Token usage by problem for Astra (millions).
R
I
Problem
Uncached
Cached
Output
Total
Uncached
Cached
Output
Total
Boundary-layer perturbations ( § 1A.1.1 )
0.288
0.902
0.084
1.274
0.423
2.098
0.110
2.631
Autonomous equation ( § 1A.1.2 )
1.175
10.628
0.436
12.240
1.301
9.117
0.383
10.801
Inertial-particle attractors ( § 1A.1.3 )
0.882
33.643
0.266
34.791
0.973
6.757
0.183
7.913
Radiation damping ( § 1A.1.4 )
0.216
5.232
0.185
5.633
0.331
4.575
0.183
5.089
Two-body electrodynamics ( § 1A.1.5 )
0.110
1.298
0.095
1.502
0.168
4.230
0.146
4.544
Appendix
Table 6: Token usage by problem for Opus (millions).
R
I
Problem
Uncached
Cached
Output
Total
Uncached
Cached
Output
Total
Boundary-layer perturbations ( § 1A.1.1 )
1.778
11.668
0.100
13.546
0.436
6.451
0.096
6.983
Autonomous equation ( § 1A.1.2 )
1.129
11.698
0.145
12.973
0.358
2.162
0.065
2.585
Inertial-particle attractors ( § 1A.1.3 )
1.459
20.230
0.119
21.807
1.981
20.144
0.303
22.428
Radiation damping ( § 1A.1.4 )
1.236
11.749
0.248
13.232
3.486
67.675
0.289
71.451
Two-body electrodynamics ( § 1A.1.5 )
0.476
5.943
0.127
6.546
0.677
4.590
0.087
5.355
Appendix
Table 7: Token usage by problem for Gemini (millions).
R
I
Problem
Uncached
Cached
Output
Total
Uncached
Cached
Output
Total
Boundary-layer perturbations ( § 1A.1.1 )
0.124
7.582
0.081
7.787
0.067
10.552
0.156
10.776
Autonomous equation ( § 1A.1.2 )
0.101
4.659
0.275
5.036
0.051
6.876
0.275
7.202
Inertial-particle attractors ( § 1A.1.3 )
0.589
53.369
0.707
54.665
1.278
94.423
1.928
97.629
Radiation damping ( § 1A.1.4 )
0.728
57.747
1.480
59.955
0.703
62.523
1.367
64.593
Two-body electrodynamics ( § 1A.1.5 )
0.055
16.744
0.183
16.982
0.016
9.047
0.195
9.258
Appendix
Table 8: Token usage by problem for DeepSeek (millions). * This task was cut off for exceeding the time limit and did not record token usage.
R
I
Problem
Uncached
Cached
Output
Total
Uncached
Cached
Output
Total
Boundary-layer perturbations ( § 1A.1.1 )
0.082
2.196
0.074
2.352
0.054
0.092
0.052
0.199
Autonomous equation ( § 1A.1.2 )
0.252
4.458
0.141
4.851
0.648
8.215
0.231
9.094
Inertial-particle attractors ( § 1A.1.3 )
3.837
61.382
0.332
65.551
7.213
87.645
0.350
95.207
Radiation damping ( § 1A.1.4 )
0.525
149.820
0.456
150.802
0.383
41.430
0.440
42.253
Two-body electrodynamics ( § 1A.1.5 )
0.122
2.330
0.113
2.565
0.159
3.929
0.152
4.240
Appendix
Table 9: Token usage by problem for GLM (millions).
Autonomous research agents use language models and tools to search literature, design and run experiments, analyze results, and support scientific discovery.
Department of Modern Physics, University of Science and Technology of China, Hefei, Anhui 230026, China · CAS Key Laboratory of Theoretical Physics, Institute of Theoretical Physics, Chinese Academy of Sciences, Beijing 100190, China · Hefei National Laboratory, University of Science and Technology of China, Hefei 230088, China +3