FrontierPhysics: Evaluating Agents for End-to-End Physics Research

Bingran You; Wenjun Ke; Ria Rosenauer; Bo Liu; Wenbo Chen; Min Chen; Shuqi Xu; Yue Ma

cs.AINov 4, 2026

FrontierPhysics: Evaluating Agents for End-to-End Physics Research

Authors: Bingran You, Wenjun Ke, Ria Rosenauer, Bo Liu, Wenbo Chen, Min Chen, Shuqi Xu, Yue Ma, Jiechao Feng, Bo Chen, Xiangkai Sun, Jiarui Liu, Jinen Guo, Zeyu Zhou, Xu Huang, Zhuohan Li, Haoyu Li, Heguang Lin, Lijie Ding, Yadong Zeng, Kangqi Zhang, Xinge Liu, Hanwen Liu, Zhenghong Li, Alexander von Recum, Yuchen Zhu, Felix Schwarzfischer, Yizhou Zhao, Jicheng Wang, Binxu Li, Qian Yu, Chan Li, Cheng Dai, Di Wang, Haoyu Zhang, Jiankai Sun, Kaituo Feng, Mian Qin, Ruihong Yin, Yiqun Luo, Weiwei Wu, Shihong Pan, Tsung Yeh Hsieh, Xiao Huang, Yan Zhang, Sihan Li, Zirui Ren, Zengji Tu, Kyoung Whan Choe, Ke Sun, Vincenzo Galia, Vaisakhi Mishra, Jiashu He, Hongyang Wang, Zhiheng Yang, Shubham Gaur, Yifan Gu, Shuyi Wang, Zihao Zuo, Steven Dillmann, Yiyou Sun, Xinyang Han, Hanchung Lee, Molei Tao, Yue Zhao, Shilong Liu, Yuchen Wu, Xuandong Zhao, Xiangyi Li, Florian Brand, Gerbrand Ceder, Hartmut Haeffner

Abstract

AI agents are playing an increasingly important role in physics research. However, existing physics benchmarks only evaluate fixed questions, isolated sub-tasks, or paper reproduction with given solutions, and rarely check research quality alongside correctness. We introduce FrontierPhysics, an end-to-end benchmark with 56 tasks made by 42 physicists based on their own research, covering nine domains across theoretical, computational, and experimental physics. Each task requires agents to review the literature, develop a research plan, conduct research and experiments, and produce structured outputs and scientific manuscripts. Combining deterministic verifiers with expert-authored rubrics, FrontierPhysics evaluates whether agents' research is correct and publishable. Across 22 models in 26 configurations, no configuration passes more than 46 of its 168 rollouts. Manual review of agent trajectories shows that frontier agents' main weaknesses are insufficiently rigorous reasoning and limited expert intuition, rather than gaps in physics knowledge or execution errors.

Figures & tables

Figure 1: FrontierPhysics leaderboard. Overall pass rate (Pass@1) under the Table 2 configurations; a pass requires passing all verifier tests and blocker rubrics. Harnesses and inference level: GPT (Codex, max), Claude (Claude Code, max), Grok (Grok Build, xhigh), Gemini (Antigravity, high), Kimi (Claude Code, max), DeepSeek (Claude Code, max), MiMo (Claude Code, max), GLM (Claude Code, max), Mistral (Claude Code, max). Error bars show approximate 95% confidence intervals.

Figure 2: Overview of FrontierPhysics . (1) Physicists build tasks from their own research. (2) An admin team of physicists accepts those passing both reviews. (3) 4,364 rollouts cover 22 models and 4 harnesses, each is one trial of a task by a model in an agent harness. (4) A deterministic verifier and a reviewer agent with the task’s rubric grade each rollout (Eq. 1 ). (5) The admin team and contributors manually analyze failure modes of the 168 GPT-6 Astra rollouts (Fig. 6 , Table 4 ).

Reasoning and QA
Benchmark	Tasks	Phys.	Research Authenticity		Research Coverage		Evaluation
			Own Research	No Reference Given	Literature Graded	Write-up Graded	Expert Rubric	Deterministic Verifiers	Failure Analysis
CritPt [ 50 ]	71	71	$\checkmark$	$\checkmark$	$\times$	$\times$	$\times$	$\checkmark$	$\checkmark$
FrontierScience-Research [ 43 ]	60	20	$\checkmark$	$\checkmark$	$\times$	$\times$	$\checkmark$	$\times$	$\times$
Subtasks from scientific workflows
Terminal-Bench-Science [ 42 ]	70	5	$\checkmark$	$\checkmark$	$\times$	$\times$	$\times$	$\checkmark$	$\circ$

Table 1: Comparison of scientific research benchmarks. We compare scientific research benchmarks across the research authenticity, research components covered in the evaluation, and the evaluation methods. Meanings of labels: Tasks/Phys. : total/physics task counts in released task lists. Own Research : the tasks come from the contributors’ own research. No Reference Given : there is no reference (e.g., the original paper) provided to the agent. Literature Graded : the quality of agents’ literature review and synthesis is evaluated. Write-up Graded : agents write up scientific manuscripts and the manuscripts are graded. Expert Rubric : at least part of the evaluation protocol uses rubrics created by domain experts. Deterministic Verifiers : at least part of the evaluation protocol contains deterministic verifiers. Failure Analysis : domain experts manually check the trajectories and output papers of selected rollouts.

\checkmark

: documented;

\circ

: partial;

\times

: no.

Figure 3: (a), FrontierPhysics tasks QA pipeline. (b), FrontierPhysics tasks structure, shown for the example task rydberg-ising-critical-point . (c), How to run a task rollout.

Figure 4: Task composition and overall pass rate (%) by primary physics domain across all 56 tasks. Donut labels give task shares.

n

gives task counts.

Figure 5: Mean reported inference cost (top) and agent execution time (bottom) per rollout versus overall pass rate (%). Time excludes environment setup, verification and rubric review.

Table 2: Detailed Frontier-model performance across all 56 tasks. For the experiment we use the highest available inference effort by default, where configurable. GPT models also appear at xhigh. The Score column reports the score defined in Equation 1 , averaged over the three rollouts for each task and then across all tasks. Deterministic pass requires all verifier tests to pass and overall pass additionally requires all blocking rubric criteria to pass. Pass@1 gives the average pass rate over the three rollouts, while Pass@3 gives the percentage of tasks with at least one passing rollout. Uncertainties are approximate 95% binomial confidence intervals over all rollouts, as in Figure FrontierPhysics: Evaluating Agents for End-to-End Physics Research . Table 8 gives 95% confidence intervals from a bootstrap over tasks, which account for the correlation of the three trials of a task. Rows are ordered by overall Pass@1, then by deterministic Pass@1 and Score.

Figure 6: Failure mode analysis of GPT-6 Astra xhigh across 56 tasks (3 rollouts for each task). Counts

n

and percentages (

n/143

) use the final review labels in Table 5 . Each rollout may have multiple tags. Verifier/rubric proportions are retained from the original recorded assessment. Correcting the eight FALSE and one RRMIS outcomes changes the pass rate from 14.9% (25/168) to 20.2% (34/168).

Figure 7: Deterministic and overall pass rates (Pass@1, %) of the models in Fig. FrontierPhysics: Evaluating Agents for End-to-End Physics Research . Column height: deterministic pass rate (all verifier tests pass). Solid part: overall pass rate (blocker rubrics also pass). Error bars: approximate 95% confidence intervals (dot: deterministic; circle: overall).

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Example task. rydberg-ising-critical-point , the example of Table 4 . Left: its task.md . The cited proposal (Figure 9 ) defines

\sigma_{\mathrm{mid}}

as the order parameter on the central bond; the device simulates a Rydberg apparatus of at most three atoms. The tests that check each step in task.md (

\pm

: tolerance around withheld reference values) and the summarized rubric criteria that grade it.

Figure 9: The proposal of the example task. Top: proposal.pdf , the one-page brief that agent need to read first. It sets out the physics and the goal, and describe what are the questions should be answered after literature research. The last paragraph leaves four questions to the agent’s own research (numbered). Bottom: for each question, what the literature must supply.

#	Task	Domain	Also spans
1	arpes-sample-alignment	Condensed Matter Physics	Photonics & Imaging
2	baf2-ih001-strain-energy	Condensed Matter Physics	-
3	cpt-frustrated-chain-spectra	Condensed Matter Physics	-
4	eigenstate-optical-response-derivation	Condensed Matter Physics	Photonics & Imaging
5	lattice-band-geometry-discovery	Condensed Matter Physics	-
6	li-graphene-mlip-transferability	Condensed Matter Physics	Chemical Physics

Appendix

Table 3: Primary and secondary domain labels for all 56 FrontierPhysics tasks. A dash indicates no secondary domain label.

Figure 10: Input modalities and required research activities across the 56 accepted tasks. Each bar gives the percentage of tasks carrying that label. Modality and activity categories are non-exclusive: a single task can contribute to multiple categories, so the percentages within each panel need not sum to 100%.

Figure 11: Verifier pass rate per task and model. The bottom row is each model’s deterministic Pass@1 over all 56 tasks (Table 2 ). Unavailable outcomes contribute zero to the aggregate.

Figure 12: Rubric blocker pass rate per task and model. The bottom row is each model’s mean over all 56 tasks. Unavailable outcomes contribute zero to the aggregate.

Figure 13: Rubric score per task and model. The weighted non-blocker score (Equation 1 ) before the verifier and blocker gates, averaged over three rollouts. The bottom row is each model’s mean over all 56 tasks.

Figure 14: Score (Equation 1 ) per task and model. The bottom row reproduces Score in Table 2 . Rows follow Figure 11 . Missing rollouts count as zero.

Tag	Label	Description	Examples
Get the physics wrong	PHY	• Subtle mistakes in the formulas or theory applied; • Agents do not have such physics knowledge (no signal when checking agents’ trajectories).	mlg-quantum-hall-dot-hartree-fock material-agnostic-visual-inverse-dynamics
Scientific presenting	SP	Agents have a good understanding of the task, but the paper writing is problematic and cannot present the research professionally.	rixs-entanglement-witness
False grading	RRMIS	The reviewer agent’s rubric grading does not align with human researchers’ grading. For example, the rubric is correct and the worker agent does the right thing, but the reviewer agent makes a mistake (often by not following the instructions properly).	li-graphene-mlip-transferability
Literature mistake	LIT	• Agents cannot find the important papers (specified by the task author in rubrics as important papers that should be found) in the research stage; • In paper.pdf , agents cannot use the literature to support their claims correctly.	arpes-sample-alignment calibrated-cbct-local-reconstruction
Agents do not follow task.md properly	INS	• Agents miss certain instructions in task.md (do not make the judgement based only on agents’ artifacts; check their trajectories to see if agents ever recognize the instructions). • Agents don’t understand or misunderstand the instructions. In the example task, agents do not get that “we work in a nearest-neighbor-Rydberg-blockade regime” implies they should make approximations to the Hamiltonian or perform postselection.	glass-nonequilibrium-landscape-evolution low-dimensional-deep-network-learning rydberg-ising-critical-point (Trials 1–3; supporting; task-dominant: REAL)
Agents do not understand the reality constraints	REAL	In the example task, agents do not get the reality constraints of the simulated device.	rydberg-ising-critical-point

Appendix

Table 4: Failure mode taxonomy.

#	Task	Trial 1	Trial 2	Trial 3	Main finding
Condensed Matter Physics
1	arpes-sample-alignment	RIG LIT MIS	LIT RIG	LIT MIS	Agents recognized tilt–translation coupling but missed pose-recovery tolerances and omitted electron-optics or inverse-optimization references. Reviewers overlooked executed one-coordinate scans in Trials 1 and 3, although both still failed pose recovery.
2	baf2-ih001-strain-energy	INT RIG	INT RIG	RIG	Displayed Trials 1–2 insufficiently justified bulk-like ice selection and failed AgI transfer. Displayed Trial 3 passed AgI but did not justify its selected ice population despite crop-dependent results.
3	cpt-frustrated-chain-spectra	PASS	PASS	PASS	-
4	eigenstate-optical-response-derivation	INS MIS	INS MIS	INS	Agents did not start from the required current-density operator. PDF extraction and Lean environment setup also produced erroneous checks, while other verifier and blocker failures remained.
5	lattice-band-geometry-discovery	RIG LIT	FALSE	INS RIG	Trial 1 proved the scaling limit, but an early mistake spoiled its numerics; Trial 2’s correct numerics hit a run-time error because the task requested too large a grid, a false negative, and Trial 3 met the same problem.

Appendix

Table 5: Final review-sheet labels for 56 tasks. Cyan cells (eight FALSE, one RRMIS) identify the 9 trials whose final pass/fail outcomes should change; MIS denotes verifier or rubric errors that do not change the final outcome.

Table 6: Regrading with a second reviewer agent. GPT-5.6 Sol (xhigh), the default reviewer agent of all main results, and Claude Opus 5.5 grade the same first-round rollouts with the same rubrics. Rubric scores are as in Figure 13 . The pass/fail columns cover the rollouts that pass the verifier, the only ones whose outcome the reviewer agent decides;

\kappa

is Cohen’s kappa, and Opus only and Sol only count rollouts passed by just one of the two reviewer agents. Two Claude Opus 5.5 rollouts were not regraded.

Configuration	Configuration	Slots	Configuration
Claude Opus 5.5	GPT-5.6 Sol (max)	0	Grok 4.7
Claude Fable 5.1	GPT-5.6 Sol (xhigh)	0	Grok 4.6
Claude Opus 5	GPT-5.6 Luna (max)	0	Kimi K3.1 Pro
Claude Haiku 5.5	GPT-5.6 Luna (xhigh)	1	Kimi K3
GPT-6 Astra (max)	GPT-5.6 Terra (max)	0	MiMo V2.6 Pro
GPT-6 Astra (xhigh)	GPT-5.6 Terra (xhigh)	0	GLM-5.3

Appendix

Table 7: Slots without a valid result per configuration: no rollout, or a rollout without complete scoring. Each of them counts as a failure in the overall pass rate and as zero in the score. A rollout that ended at a time or memory limit, or that the model ended, is a scored failure and is not counted here. Every configuration has 168 slots.

Configuration	Est.	95% CI	Binomial	Est.	95% CI	3/3	$\geq$ 1/3
	Pass@1 (%)			Pass@3 (%)		Tasks passed
Claude Opus 5.5	27.4	[17.9, 36.9]	$\pm$ 6.7	44.6	[32.1, 57.1]	8	25
GPT-6 Astra (max) †	23.2	[13.7, 33.9]	$\pm$ 6.4	30.4	[17.9, 42.9]	9	17
GPT-6 Astra (xhigh) †	19.6	[11.3, 28.6]	$\pm$ 6.0	30.4	[17.9, 42.9]	4	17
GPT-6.1 Sol †	19.6	[11.3, 29.2]	$\pm$ 6.0	30.4	[19.6, 42.9]	6	17
GPT-6.1 Terra	14.3	[7.1, 22.6]	$\pm$ 5.3	21.4	[10.7, 32.1]	4	12

Appendix

Table 8: Overall pass rates with task-clustered uncertainty, over all 56 tasks and 3 trials per task; unavailable outcomes count as failures, as in Table 2 . 95% confidence intervals from a bootstrap over tasks, 10,000 resamples, which accounts for the correlation of the three trials of a task. The Binomial column repeats the previous approximate 95% binomial (Wald) half-width over 168 rollouts, which treats the rollouts as independent. Pass@3 is the share of tasks with at least one passing trial. 3/3 and

\geq

1/3 count the tasks (of 56) passed in all three trials and in at least one. † Not statistically separable from Claude Opus 5.5 (two-sided paired sign-flip permutation test over tasks, unadjusted

p\geq 0.05

Figure 15: Mean agent steps per rollout versus overall pass rate (%). Steps count recorded tool invocations, including failed and interrupted calls.

Figure 16: Mean reported output tokens per rollout, in thousands, versus overall pass rate (%). Token counts follow provider or harness usage accounting.

Figure 17: Agent execution time per rollout, in hours, per task and model: the mean over the recorded rollouts in each cell. The bottom row gives each model’s mean over tasks with records, as plotted in Figure 5 . A dash marks a task without a recorded time and a dot one with fewer than three.

Figure 18: Agent steps (tool calls) per rollout, per task and model: the mean over the recorded rollouts in each cell. The bottom row gives each model’s mean over tasks with records, as plotted in Figure 15 . Rows follow Figure 11 . A dash marks a task without a recorded step count and a dot one with fewer than three.

Figure 19: Output tokens per rollout, in thousands, per task and model: the mean over the recorded rollouts in each cell. The bottom row gives each model’s mean over tasks with records, as plotted in Figure 16 . Rows follow Figure 11 . A dash marks a task without an output-token record and a dot one with fewer than three.

Model	Agent time	Agent steps	Output tokens
Claude Opus 5.5	168 / 56	168 / 56	168 / 56
GPT-6 Astra (max)	168 / 56	168 / 56	168 / 56
GPT-6 Astra (xhigh)	168 / 56	168 / 56	168 / 56
GPT-6.1 Sol	168 / 56	168 / 56	168 / 56
GPT-6.1 Terra	168 / 56	168 / 56	168 / 56
GPT-6 Sol	168 / 56	168 / 56	168 / 56

Appendix

Table 9: Resource coverage: contributing rollouts / tasks for each model.

Model	Mean	Med.	P95	Mean	Med.	P95	Mean	Med.	P95	Mean	Med.	P95	Mean	Med.
	Tool calls			Steps			Time (h)			Cost ($)			Output tokens (M)
Claude Opus 5.5	219	204	371	191	168	339	2.44	2.06	5.38	26.95	24.86	51.89	0.38	0.39
GPT-6 Astra (max)	194	111	585	108	61	335	1.73	1.05	5.28	26.91	15.24	90.77	0.18	0.13
GPT-6 Astra (xhigh)	78	65	172	86	65	197	1.24	0.75	3.72	19.21	11.60	56.87	0.11	0.09
GPT-6.1 Sol	281	170	882	158	90	583	1.79	1.17	5.21	5.33	3.20	17.51	0.21	0.15
GPT-6.1 Terra	203	124	461	203	119	553	1.14	0.65	3.90	3.46	1.23	14.25	0.12	0.09

Appendix

Table 10: Resource use per rollout for each model configuration: mean, median (Med.) and 95th percentile (P95), pooled over the rollouts of Table 2 that record the value. These are the same selected rollouts as Figures 5 , 15 and 16 , and rows follow Table 2 . Tool calls count recorded tool invocations, including failed and interrupted calls. A step is one model call of the agent that returned a response, and one step can make several tool calls or none. Time is the execution-stage wall-clock duration in hours, excluding environment setup, verification and rubric review; because it is wall-clock time, slow provider routes and rate limiting inflate it. Cost is the reported inference cost per rollout in US dollars, as in Figure 5 (metered, priced from recorded tokens at list prices, or estimated), and output tokens are in millions. Unavailable values are omitted, not set to zero; † marks a statistic over fewer than all 168 designated rollouts of the configuration (Table 9 gives the coverage of time, tool calls and output tokens). P95 and the median use linear interpolation between order statistics (NumPy’s default). The figures instead average within each task and then across tasks; both means coincide wherever all 168 rollouts are recorded. In Claude Code runs, tool calls, steps and time include subagent work.

Task / reference	Weighted score	Blockers
QNTK-research	100%	2/2
consistent-two-phase-ls-vof	100%	1/1
cpt-frustrated-chain-spectra	N/A	0/1 - fail
cvqkd-calibrated-detector-noise	100%	2/3 - fail
diffuse-ib-rigid-body-fsi	100%	-
eigenstate-optical-response-derivation	100%	5/5

Appendix

Table 11: Evaluating rubric scores for all 33 tasks where the authors provided a manuscript or paper as a reference. Only the rubrics, that actually evaluated/ needed the present manuscript were evaluated. If no blocking rubric applied to the manuscript, it is shown as a dash. If no scoring rubric applied to the manuscript it is shown as NA.

Scientific Agent Benchmarks

See all 46 papers

PRBench: End-to-end Paper Reproduction in Physics Research
Mar 29, 2026Shi Qiu, Junyi Deng, Yiwei Deng +48AI Agent Benchmarks AI Coding Agents
PRL-Bench: A Comprehensive Benchmark Evaluating LLMs' Capabilities in Frontier Physics Research
Apr 16, 2026Tingjia Miao, Wenkai Jin, Muhua Zhang +19LLM Evaluation AI Agent Benchmarks
School of Artificial Intelligence, Shanghai Jiao Tong University · Zhiyuan College, Shanghai Jiao Tong University · School of Physics and Astronomy, Shanghai Jiao Tong University +7
DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking
May 25, 2026Matt L. Wiemann, Lindsay M. Smith, Peter Melchior +4Physical Reasoning Scientific Discovery
Princeton University · Work done during internship at NYU/Polymathic AI. · Boston University +3

Mar 29, 2026J/K move · Enter open · S save

PRBench: End-to-end Paper Reproduction in Physics Research

Shi Qiu, Junyi Deng, Yiwei Deng, Haoran Dong+47

AI Agent Benchmarks AI Coding Agents Scientific Reproducibility Scientific Reasoning in Language Models

Open paper AskPDF arXiv

AI agents powered by large language models exhibit strong reasoning and problem-solving capabilities, enabling them to assist scientific research tasks such as formula derivation and code generation. However, whether these agents can reliably perform end-to-end reproduction from real scientific papers remains an open question. We introduce PRBench, a benchmark of 30 expert-curated tasks spanning 11 subfields of physics. Each task requires an agent to comprehend the methodology of a published paper, implement the corresponding algorithms from scratch, and produce quantitative results matching the original publication. Agents are provided only with the task instruction and paper content, and operate in a sandboxed execution environment. All tasks are contributed by domain experts from over 20 research groups at the School of Physics, Peking University, each grounded in a real published paper and validated through end-to-end reproduction with verified ground-truth results and detailed scoring rubrics. Using an agentified assessment pipeline, we evaluate a set of coding agents on PRBench and analyze their capabilities across key dimensions of scientific reasoning and execution. The best-performing agent, OpenAI Codex powered by GPT-5.3-Codex, achieves a mean overall score of 34%. All agents exhibit a zero end-to-end callback success rate, with particularly poor performance in data accuracy and code correctness. We further identify systematic failure modes, including errors in formula implementation, inability to debug numerical simulations, and fabrication of output data. Overall, PRBench provides a rigorous benchmark for evaluating progress toward autonomous scientific research.

Abstract

Scientific Agent Benchmarks

Explore similar work

PRBench: End-to-end Paper Reproduction in Physics Research

PRL-Bench: A Comprehensive Benchmark Evaluating LLMs' Capabilities in Frontier Physics Research

DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking