PHYRD: A Living Benchmark for Reproducing and Extending Theoretical Physics Research

Qituan Zhang; Guo Chen; Linfeng Zhang; Wenlong Zhang

cs.CLOct 23, 2026

PHYRD: A Living Benchmark for Reproducing and Extending Theoretical Physics Research

Authors: Qituan Zhang, Guo Chen, Linfeng Zhang, Wenlong Zhang

Organizations: EPIC Lab, Shanghai Jiao Tong University · Fudan University · University of Electronic Science and Technology of China · University of Science and Technology of China

Abstract

Large language models and agents have shown significant power in advanced physical research. In this paper, We introduce PHYRD, a living benchmark for theoretical-physics reasoning with 63 Reproduce and 43 Discover problems. Reproduce tests derivation of published results; Discover tests consequences of modified theoretical conditions. FormulaForest constructs the tasks with reference solutions and subquestion-level rubrics. Across five models, whole-problem accuracy ranges from 49.21% to 68.25% on Reproduce and from 18.60% to 74.42% on closed-book Discover. With tools, four models complete more Discover problems, but conceptual errors and missing arguments persist. Besides, our experimental results demonstrate the value of FormulaForest as synthetic data. On FrontierScience, a benchmark developed by OpenAI, it raises the accuracy of a Qwen-35B-A3B model from 61% to 72%.

Figures & tables

Benchmark	Physics focus	Tool use	Research tasks	Derivation under modified conditions	Continued expansion
MATH	✗	✗	✗	✗	✗
ScienceAgentBench	✗	✓	✓	—	—
PRL-Bench	✓	✓	✓	—	✓
HorizonMath	✗	—	✓	—	✓
MathArena	✗	✓	✓	—	✓
LiveBench	✗	✗	✗	✗	✓

Table 1: Coverage of representative benchmarks. ✓: included; ✗: outside scope; —: not specified. Modified-condition derivation examines consequences of changed premises; continued expansion includes release plans for PRL-Bench and PHYRD.

Figure 1: Examples from the two tracks. (a) Deriving a published soft graviton theorem. (b) Analyzing spectral collisions in a modified matrix family. Selected tasks and reference results are shown; complete statements and selected derivations appear in Appendix A.3 and A.4 .

Figure 2: Primary physics subfields in each track. Counts are shown where segments permit; CM: condensed matter, Qtm: quantum.

Figure 3: Reproduce results on 63 problems . Cells give complete solutions by primary subfield; the last two columns give global subquestion (SAcc) and whole-problem (WAcc) accuracy in percent.

Figure 4: Number of Reproduce responses by missed-subquestion count. “Miss 2+” includes responses with no correct subquestions.

Figure 5: Discover accuracy on the same 43 problems in the two evaluated settings. Paired bars give closed-book (blue) and tool-assisted (orange) percentages.

Figure 6: Discover outcomes on the same 43 problems per model. C: closed book; T: tool-assisted. Complete requires every subquestion; partial passes at least one but not all. The full 43-problem denominator is retained in both rows.

Figure 7: Problem-level Discover changes for four API models on 43 paired tasks each. Upper: subquestion fraction higher, equal, or lower with tools. Lower: complete solutions gained (blue), lost (orange), and net change (right).

Figure 8: Change in complete Discover solutions with tools, by primary subfield (tool-assisted minus closed-book). Column headings give the number of problems; Net sums the five fields. Color distinguishes gains, losses, and ties.

Figure 9: Problems solved completely by exactly

k

of five models. Segment widths give within-setting shares; segment labels give problem counts. The rightmost numbers are the setting totals.

Figure 10: Reproduce error composition by model and subfield. Colored segments give the four category shares among errors; numbers at right are counts. CM: condensed matter; Qtm: quantum; All pools subfields.

Figure 11: Discover error composition: closed-book (top) and tool-assisted (bottom). Segment widths and counts follow Figure 10 .

Figure 12: External distillation results on Olympiad-100 (a subset from FrontierScience benchmark, made by OpenAI) (accuracy, %) and PHYRD-40 (graded mean, 0–100). Teacher rows are references; BPM and ESCD are two on-policy distillation methods that use SciDeriv prompts, with the 35B student initialized by SFT Anonymous (2026) .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Track	3	4	5	6	7	8	9	Total
Reproduce	1	39	20	2	0	1	0	63
Discover	3	6	22	7	2	1	2	43

Appendix

Table 2: Number of problems with each subquestion count. Column headings 3–9 denote the number of public subquestions in a problem.

Matrix	Pole order	Coefficient ranks	Power growth
$I_{4}$	1	$4$	bounded
$J_{2}(1)\oplus J_{2}(1)$	2	$4,2$	$n$
$C(0,0)$	4	$4,3,2,1$	$n^{3}$

Appendix

Table 3: Resolvent and matrix-power behavior at the three centers. Coefficient ranks are ordered by increasing pole order.

ID	Field	Task / source	$k$
A001	Cond-Mat	Static Aristotelian algebra sector (R01)	5
A003	Cond-Mat	Post coupling variation in dipole Noether gauging (R01)	4
A004	Cond-Mat	Dilatation algebra sector (R01)	4
A009	Stat	Logarithmic boundary entropy in the deformed Ising model (R02)	4
A010	Cond-Mat	Spatial symmetries in a fracton scalar theory (R01)	5
A011	Cond-Mat	Dipole algebra sector (R01)	4

Appendix

Table 4: Reproduce task catalogue. The final column gives the number of public subquestions. Source keys refer to Table 4 .

Key	Source title and identifier
R01	Fractons, dipole symmetries and curved spacetime. https://doi.org/10.21468/scipostphys.12.6.205
R02	$T\bar{T}$ -deformed 2D quantum field theories. https://doi.org/10.1007/jhep10(2016)112
R03	Moving the CFT into the bulk with $T\bar{T}$ . https://doi.org/10.1007/jhep04(2018)010
R04	Lieb-Schultz-Mattis, Luttinger, and ’t Hooft - anomaly matching in lattice systems. https://doi.org/10.21468/scipostphys.15.2.051
R05	Scattering of two spinning black holes in post-Minkowskian gravity, to all orders in spin, and effective-one-body mappings. https://doi.org/10.1088/1361-6382/aaa3a8
R06	The IR-resummed Effective Field Theory of Large Scale Structures. https://doi.org/10.1088/1475-7516/2015/02/013

Appendix

Table 5: Source papers for the Reproduce task catalogue.

ID	Field	Source paper (arXiv identifier)	$k$
p01	Astro	Quantum de Sitter and Analytically-Continued Chern–Simons Theory (2608.07467)	4
p02	Quantum	Solvable Renyi Indices in Critical Stabilizer Entropy: From Cauchy Minors to Discrete Selberg Sums (2608.06995)	5
p03	HEP	Bootstrapping 3D Conformal Field Theories with Product Analytic Functionals (2608.06497)	3
p05	Quantum	Spectral Topology and Universal Krylov Dynamics (2608.07258)	3
p06	Astro	Formation of the Kerr black hole: an exact model (2608.06519)	4
p07	HEP	Topological defects in reflection positive topological field theories (2608.07217)	6

Appendix

Table 6: Discover source catalogue. Titles identify source papers, not the modified tasks. The final column gives the number of public subquestions.

Figure 13: Strict subquestion scores for the first 32 Reproduce problems. The remaining 31 appear in Figure 14 . Each cell shows correct over applicable public subquestions.

Figure 14: Strict subquestion scores for the remaining 31 Reproduce problems, on the same color scale as Figure 13 .

Figure 15: Strict subquestion scores for all 43 Discover problems in the closed-book setting. Grey NA cells mark problems without a final answer.

Figure 16: Strict subquestion scores for the same 43 Discover problems in the tool-assisted setting. Task order and color scale match Figure 15 .

SAcc	GPT	DeepSeek	GLM	Kimi	Qwen
Astro	91.00	86.83	93.33	94.17	89.33
CM	91.88	85.62	80.00	65.00	88.12
HEP	87.05	88.33	81.37	81.45	82.76
Qtm	75.00	83.33	70.00	60.00	86.67
Stat	100.00	75.00	75.00	66.67	75.00
All	88.33	86.88	82.25	79.66	84.30

Appendix

Table 7: Reproduce results by physics subfield (%). SAcc: mean subquestion accuracy; WAcc: whole-problem accuracy.

SAcc	GPT	DeepSeek	GLM	Kimi	Qwen
Astro	97.78	82.78	61.11	55.00	36.11
CM	93.33	85.00	60.00	74.17	49.17
HEP	86.33	77.12	59.52	59.93	54.20
Qtm	91.43	91.90	58.10	68.73	90.32
Stat	97.22	92.22	56.11	82.22	71.94
All	91.06	82.67	59.34	63.40	57.59

Appendix

Table 8: Closed-book Discover results by physics subfield (%). SAcc: mean subquestion accuracy; WAcc: whole-problem accuracy.

SAcc	GPT	DeepSeek	GLM	Kimi	Qwen
Astro	97.78	82.78	68.33	71.11	71.67
CM	95.83	78.33	56.67	80.00	60.83
HEP	93.67	93.00	63.80	72.02	65.61
Qtm	97.14	100.00	87.30	82.54	88.10
Stat	92.22	100.00	78.33	75.42	83.75
All	95.11	91.63	69.43	74.42	71.89

Appendix

Table 9: Tool-assisted Discover results by physics subfield (%). SAcc: mean subquestion accuracy; WAcc: whole-problem accuracy.

Reproduce
Model	C	D	A	I	Total
GPT-5.6 Sol	4	0	7	20	31
DeepSeek v4 Pro	12	2	11	10	35
GLM-5.2	11	3	29	4	47
Kimi K2.6	24	2	24	4	54
Qwen3.7-Max	19	3	13	9	44

Appendix

Table 10: Counts of primary error categories. C: formulaic/conceptual; D: derivation; A: calculation; I: incomplete or unsupported.

Figure 17: FormulaForest construction. Reproduce and Discover share derivation, blind review, and final checks. The expanded E.3 block shows the program-verified Discover route; dashed arrows mark revisions.

Scientific Reasoning in Language Models

Scientific reasoning in language models studies how AI systems use evidence, domain knowledge, and tools to solve problems and support discovery.

See all 144 papers

PRBench: End-to-end Paper Reproduction in Physics Research
Mar 29, 2026Shi Qiu, Junyi Deng, Yiwei Deng +48AI Agent Benchmarks AI Coding Agents
PRL-Bench: A Comprehensive Benchmark Evaluating LLMs' Capabilities in Frontier Physics Research
Apr 16, 2026Tingjia Miao, Wenkai Jin, Muhua Zhang +19LLM Evaluation AI Agent Benchmarks
School of Artificial Intelligence, Shanghai Jiao Tong University · Zhiyuan College, Shanghai Jiao Tong University · School of Physics and Astronomy, Shanghai Jiao Tong University +7
DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking
May 25, 2026Matt L. Wiemann, Lindsay M. Smith, Peter Melchior +4Physical Reasoning Scientific Discovery
Princeton University · Work done during internship at NYU/Polymathic AI. · Boston University +3

Mar 29, 2026J/K move · Enter open · S save

PRBench: End-to-end Paper Reproduction in Physics Research

Shi Qiu, Junyi Deng, Yiwei Deng, Haoran Dong+47

AI Agent Benchmarks AI Coding Agents Scientific Reproducibility Scientific Reasoning in Language Models

Open paper AskPDF arXiv

AI agents powered by large language models exhibit strong reasoning and problem-solving capabilities, enabling them to assist scientific research tasks such as formula derivation and code generation. However, whether these agents can reliably perform end-to-end reproduction from real scientific papers remains an open question. We introduce PRBench, a benchmark of 30 expert-curated tasks spanning 11 subfields of physics. Each task requires an agent to comprehend the methodology of a published paper, implement the corresponding algorithms from scratch, and produce quantitative results matching the original publication. Agents are provided only with the task instruction and paper content, and operate in a sandboxed execution environment. All tasks are contributed by domain experts from over 20 research groups at the School of Physics, Peking University, each grounded in a real published paper and validated through end-to-end reproduction with verified ground-truth results and detailed scoring rubrics. Using an agentified assessment pipeline, we evaluate a set of coding agents on PRBench and analyze their capabilities across key dimensions of scientific reasoning and execution. The best-performing agent, OpenAI Codex powered by GPT-5.3-Codex, achieves a mean overall score of 34%. All agents exhibit a zero end-to-end callback success rate, with particularly poor performance in data accuracy and code correctness. We further identify systematic failure modes, including errors in formula implementation, inability to debug numerical simulations, and fabrication of output data. Overall, PRBench provides a rigorous benchmark for evaluating progress toward autonomous scientific research.

Abstract

Scientific Reasoning in Language Models

Explore similar work

PRBench: End-to-end Paper Reproduction in Physics Research

PRL-Bench: A Comprehensive Benchmark Evaluating LLMs' Capabilities in Frontier Physics Research

DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking