From Research Goals to Scientific Closure: Evaluating Agents Through Evidence-Supported Physics Research
Authors: Yijia Mao, Song Dai, Tao Chen, Xiangyu Lu, Linqian Zeng, Chenxi Hu, Sitong Zhou, Dongyang He, Xiaodan Mao, Linxuan Zhang, Tong Zhu, Yafu Li, Fangchen Yu, Peng Ye, Feng He, Yu Cheng, Yun Luo
Organizations: Shanghai AI Laboratory · Shanghai Jiao Tong University · The HongKong University of Science and Technology(Guangzhou) · Nanyang Technological University
Can AI agents organize scientific investigations that lead to well-supported conclusions? To evaluate this capability, we introduce PhysClosureBench, a benchmark of 50 expert-verified, goal-specified research tasks spanning strong-field, attosecond, and neighboring physics areas. Each task specifies the research objective and essential scientific constraints, while leaving the agent to organize the investigation and determine how to generate and validate the evidence needed to answer the question. A 20-criterion Atomic Score measures local scientific accomplishment, while Research Closure (RC0--RC4) assesses how fully the investigation resolves the specified research question based on whether its indispensable evidence requirements and applicable scope conditions are satisfied. We evaluate seven frontier LLMs, where Claude-Opus-5 achieves the highest mean Atomic Score (6.29/10) but reaches RC4 in only 32.7% of attempts. Across the benchmark, 36.2% of attempts scoring at least 8/10 still fall short of RC4, while all seven models reach RC4 on substantially fewer tasks in all three runs than in at least one. Case-level analysis further illustrates how misinterpretation of generated results and unresolved physical applicability limits can lead to incomplete closure. Together, these results reveal a persistent gap between AI agents' local scientific accomplishments and their ability to achieve complete research closure.
Figures & tables
Figure 1: From a scientific goal to scientific closure. In PhysClosureBench, agents are required to achieve the provided scientific objective within the specified task setting and a resource-bounded computational environment.
Benchmark
Task formulation
Execution / artifacts
Research decisions
Evidence integration
Scientific closure
GPQA / HLE
Expert-level QA
×
×
×
×
CritPt
Physics reasoning
✓
×
×
×
SciCode
Scientific coding
✓
×
×
×
PaperBench
Research reproduction
✓
Partial
Partial
×
RE-Bench
Research-engineering tasks
✓
Partial
Partial
×
FrontierScience-R
Scientific research subproblems
×
✓
Partial
Partial
Table 1: Comparison of representative scientific AI benchmarks. A checkmark indicates direct assessment of a dimension, while “Partial” indicates limited or indirect assessment and a cross indicates no explicit assessment. Scientific closure concerns whether an investigation establishes a well-supported answer to its research question, including the required validation and scope conditions.
Figure 2: Two complementary views of scientific performance. Atomic Score (0–10) measures local accomplishment across F/O/I/M/V. Research Closure (RC0–RC4) assesses how fully the investigation resolves the research question. A high Atomic Score does not guarantee complete closure.
Figure 3: Scientific accomplishment and Research Closure across frontier models. (A) Mean Atomic Score and functional completion for Formulation (F), Output (O), Identification (I), and Mechanism (M). Atomic Scores are shown on the shared axis after multiplication by 10; functional completion is the mean attempt-level fraction of applicable criteria satisfied within each dimension. (B) Distribution of Research Closure levels across independent attempts. RC4 denotes complete closure, RC3 indicates that the main conclusion is supported, but material validation or scope gaps remain, and RC0–RC2 indicate different degrees of incomplete investigation.
Figure 4: Relationship between local scientific accomplishment and Research Closure. (A) Joint distribution of Atomic Score and Research Closure across all 1,050 independent attempts. Each cell reports the percentage of all attempts falling into the corresponding score bin and RC level. The highlighted region marks high-scoring attempts below RC4. (B) Unmet applicable criteria for Formulation (F), Output (O), Identification (I), and Mechanism (M) among the 229 attempts scoring at least 8/10, comparing RC4 ( n=146 ) with below RC4 ( n=83 ). Bars show 100% minus the mean attempt-level functional completion within each group. The inset reports the fraction of satisfied applicable Validation/Scope (V) criteria across the full benchmark, rather than within the high-scoring groups; N/A criteria are excluded.
Figure 5: Representative evidence and scientific closure. Case 30 compares representative runs from Kimi-K3, GPT-5.6-Sol, and Gemini-3.1-Pro, which reached RC4, RC3, and RC2, respectively. The shaded region illustrates a possible progression from numerical evidence to a supported conclusion; Gemini’s deviation represents the misinterpretation of the computed intensity trends. Paths are schematic, not execution times, Atomic Scores, or intermediate RC assignments. Inset plots schematically illustrate trends in the submitted results. Appendix E documents the research activities and evidence.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Domain distribution of the 50 benchmark tasks.
Case
CPU cores
Minutes
Case
CPU cores
Minutes
Case01
8
30
Case26
16
90
Case02
16
30
Case27
8
60
Case03
32
240
Case28
16
60
Case04
8
30
Case29
32
90
Case05
32
45
Case30
8
45
Case06
8
45
Case31
96
360
Appendix
Table 4: Case-level CPU and wall-time limits for the 50 benchmark tasks.
Dimension
Total
Mean
Median
Range
Mean score weight
F
164
3.28
3
2–4
16.4%
O
192
3.84
4
2–6
19.2%
I
163
3.26
3
3–4
16.3%
M
369
7.38
7
7–9
36.9%
V
112
2.24
2
1–3
11.2%
Total
1000
20.00
20
20–20
100.0%
Appendix
Table 5: Atom allocation across 50 cases. Total atom counts across all cases; Mean and Median are calculated from the 50 per-case atom counts for each dimension; Range gives their minimum and maximum. Mean score weight is the average percentage of each task’s 10-point maximum allocated to a dimension.
Model
F (%)
O (%)
I (%)
M (%)
V (%)
V satisfied/applicable
Claude Opus 5
85.6
60.1
54.6
52.8
84.0
263/313
GPT-5.6 Sol
83.8
60.5
47.9
46.0
74.2
241/325
Kimi K3
77.4
49.7
43.4
40.9
68.4
214/313
DeepSeek-V4.1-Flash
67.2
37.3
35.2
35.7
71.3
214/300
Qwen3.8-Max
62.3
28.7
24.1
25.9
54.7
156/285
GLM-5.3-Flash
56.7
23.7
20.3
23.3
65.6
177/270
Appendix
Table 6: Functional completion by model. F/O/I/M values are mean attempt-level completion rates, with terminal-zero attempts contributing zero. V values represent the percentage of applicable V criteria satisfied, excluding N/A criteria and terminal-zero attempts without atom records.
Model
Run
F
O
I
M
V
V satisfied/applicable
Claude Opus 5
R1
88.2
62.5
58.3
56.2
84.1
90/107
R2
83.7
63.6
56.8
55.5
82.5
85/103
R3
84.8
54.4
48.5
46.9
85.4
88/103
GPT-5.6 Sol
R1
86.8
58.7
49.3
48.3
76.8
86/112
R2
78.5
57.0
39.0
39.6
66.3
67/101
R3
86.2
65.9
55.3
50.0
78.6
88/112
Appendix
Table 7: Functional completion by models in three independent runs. Measures and applicability conventions follow Table 6 .
Model
N
RC0
RC1
RC2
RC3
RC4
Claude Opus 5
150
24 (16.0%)
49 (32.7%)
25 (16.7%)
3 (2.0%)
49 (32.7%)
GPT-5.6 Sol
150
28 (18.7%)
38 (25.3%)
43 (28.7%)
7 (4.7%)
34 (22.7%)
Kimi K3
150
48 (32.0%)
36 (24.0%)
32 (21.3%)
5 (3.3%)
29 (19.3%)
DeepSeek-V4.1-Flash
150
75 (50.0%)
25 (16.7%)
30 (20.0%)
4 (2.7%)
16 (10.7%)
Qwen3.8-Max
150
76 (50.7%)
39 (26.0%)
22 (14.7%)
2 (1.3%)
11 (7.3%)
GLM-5.3-Flash
150
87 (58.0%)
32 (21.3%)
19 (12.7%)
4 (2.7%)
8 (5.3%)
Appendix
Table 8: Research Closure distribution by model. Cells report count (percentage) over the 150 final scored attempts for each model.
Model
Run
N
RC0
RC1
RC2
RC3
RC4
Claude Opus 5
R1
50
8
14
7
1
20
R2
50
11
9
12
2
16
R3
50
5
26
6
0
13
GPT-5.6 Sol
R1
50
7
13
14
5
11
R2
50
15
9
13
2
11
R3
50
6
16
16
0
12
Appendix
Table 9: Research Closure counts by model and independent run. Each row reports 50 scored attempts.
V category
Description
Satisfied/applicable
Rate
Scope justification
Justify the conditions under which the conclusions are valid.
852/981
86.9%
Required validation
Perform validation checks required by the task, regardless of the chosen method.
93/156
59.6%
Risk-linked validation
Address specific risks arising from the method, results, or conclusions.
441/922
47.8%
Appendix
Table 10: Validation/Scope criteria across the benchmark.
Figure 8: Atomic-score distributions across investigations. Kernel density estimates show the distribution of Atomic Scores over all final scored attempts for each model, including terminal-zero outcomes. Dashed lines indicate mean Atomic Scores for each model. The distributions summarize score variation across the full set of benchmark investigations.
Investigation
Selected research steps
Key finding or unresolved issue
Kimi K3 10.0/RC4
Constructed scattering calculations; detected an angle-sign inconsistency; corrected and checked the results; interpreted the screening dependence.
Supported the interpretation within a physically justified scope.
GPT-5.6 Sol 9.5/RC3
Repaired the numerical solver; identified screening-dependent trends; explained their mechanism; recorded a high-field diagnostic.
Did not resolve the applicability concern revealed by its high-field diagnostic.
Gemini 3.1 Pro 4.5/RC2
Revised trajectory propagation and angle convention; generated Coulomb and screened results; summarized the intensity dependence.
Misidentified increasing or turning-over screened curves as monotonically decreasing.
Appendix
Table 11: Selected Case 30 research activities and key findings. The steps summarize recorded activities, not a complete or strictly ordered execution history. Scores and RC levels refer to final submissions.
Figure 9: Screened offsets contradict Gemini’s decreasing-trend claim. Replotted from the agent’s submitted numerical data. All 100 recorded points per curve are shown, with straight connectors and no fitting or smoothing. The angle θ is the submitted outgoing-trajectory offset in degrees, with I=2E02 ; λ is in atomic units.
Model
Mean score
Median
RC4
≥ 1 RC4
All 3 RC4
Claude Opus 5
6.29
6.75
32.7%
50.0%
18.0%
GPT-5.6 Sol
5.83
6.00
22.7%
40.0%
10.0%
Kimi K3
5.17
5.00
19.3%
40.0%
4.0%
DeepSeek-V4.1-Flash
4.50
4.00
10.7%
24.0%
0.0%
Qwen3.8-Max
3.46
2.00
7.3%
14.0%
2.0%
GLM-5.3-Flash
3.28
2.25
5.3%
12.0%
0.0%
Appendix
Table 12: Model-level performance and RC4 consistency. Mean and median Atomic Scores and attempt-level RC4 rates are computed over 150 final scored attempts per model. The two RC4 consistency measures are computed over the same 50-task denominator for every model.
Atom
Function
Representative criterion
F1
Formulation
Relate the outgoing half of the central-potential deflection to the attoclock offset, using consistent exit and field/vector-potential conventions, units, and approximation limits.
O2
Output
Calculate auditable offset-versus-intensity results for Coulomb and screened potentials matched to the same binding energy, including λ=1 and enough intermediate screening lengths to resolve changes with potential range.
I2
Identification
Identify the approach to Coulomb behavior at long range, changes in intensity dependence at intermediate screening lengths, and the suppression and relative offsets at strong screening supported by the data.
M2
Mechanism
Explain how the electron’s exit or impact distance, asymptotic kinetic energy, and potential range compete to change the offset response as screening length decreases.
V1
Validation/Scope
When numerical choices, binding-energy calibration, sampling, or curve structure raise a concrete risk to the conclusion, test that risk or restrict the conclusion accordingly. If no risk is triggered, record N/A without claiming positive validation evidence.
V3
Validation/Scope
Specify which conclusions the reduced scattering calculation supports, distinguishing them from unverified real-atom, quantum-dynamical, or universal tunnelling-time interpretations.
Appendix
Table 13: Representative Case 30 atomic criteria. Criteria are condensed from the full case-specific grading rubric for readability.
The paradigm of agentic science requires AI systems to conduct robust reasoning and engage in long-horizon, autonomous exploration. However, current scientific benchmarks remain confined to domain knowledge comprehension and complex reasoning, failing to evaluate the exploratory nature and procedural complexity of real-world research. In this work, we present research-oriented evaluations in theoretical and computational physics, a natural testbed with comprehensive domain knowledge, complex reasoning, and verifiable end-to-end workflows without reliance on experiments. Here we introduce PRL-Bench (Physics Research by LLMs), a benchmark designed to systematically map the capability boundaries of LLMs in executing end-to-end physics research. Constructed from 100 curated papers from the latest issues of Physical Review Letters since August 2025 and validated by domain experts, PRL-Bench covers five major theory- and computation-intensive subfields of modern physics: astrophysics, condensed matter physics, high-energy physics, quantum information, and statistical physics. Each task in the benchmark is designed to replicate the core properties of authentic scientific research, including exploration-oriented formulation, long-horizon workflows, and objective verifiability, thereby reconstructing the essential reasoning processes and research workflows of real physics research. Evaluation across frontier models shows that performance remains limited, with the best overall score below 50, revealing a pronounced gap between current LLM capabilities and the demands of real scientific research. PRL-Bench serves a reliable testbed for accessing next generation AI scientists advancing AI systems toward autonomous scientific discovery.
Tingjia Miao, Wenkai Jin, Muhua Zhang +19
School of Artificial Intelligence, Shanghai Jiao Tong University · Zhiyuan College, Shanghai Jiao Tong University · School of Physics and Astronomy, Shanghai Jiao Tong University +7
AI agents powered by large language models exhibit strong reasoning and problem-solving capabilities, enabling them to assist scientific research tasks such as formula derivation and code generation. However, whether these agents can reliably perform end-to-end reproduction from real scientific papers remains an open question. We introduce PRBench, a benchmark of 30 expert-curated tasks spanning 11 subfields of physics. Each task requires an agent to comprehend the methodology of a published paper, implement the corresponding algorithms from scratch, and produce quantitative results matching the original publication. Agents are provided only with the task instruction and paper content, and operate in a sandboxed execution environment. All tasks are contributed by domain experts from over 20 research groups at the School of Physics, Peking University, each grounded in a real published paper and validated through end-to-end reproduction with verified ground-truth results and detailed scoring rubrics. Using an agentified assessment pipeline, we evaluate a set of coding agents on PRBench and analyze their capabilities across key dimensions of scientific reasoning and execution. The best-performing agent, OpenAI Codex powered by GPT-5.3-Codex, achieves a mean overall score of 34%. All agents exhibit a zero end-to-end callback success rate, with particularly poor performance in data accuracy and code correctness. We further identify systematic failure modes, including errors in formula implementation, inability to debug numerical simulations, and fabrication of output data. Overall, PRBench provides a rigorous benchmark for evaluating progress toward autonomous scientific research.
AI coding agents are increasingly used for scientific work, but their end-to-end autonomous research capability remains difficult to verify. We present ResearchClawBench, a benchmark for evaluating autonomous scientific research across 40 tasks from 10 scientific domains. Each task is grounded in a real published paper, provides related literature and raw data, and hides the target paper during evaluation. Expert-curated multimodal rubrics decompose the target scientific artifacts into weighted criteria, enabling evaluation of target-paper-level re-discovery while leaving room for new discovery. We evaluate seven autonomous research (auto-research) agents under a unified protocol and seventeen native LLMs through the lightweight ResearchHarness. Current systems remain far from reliable re-discovery: the strongest autonomous agent, Claude Code, averages 21.5, and the strongest ResearchHarness LLM, Claude-Opus-4.7, averages 20.7, with an LLM frontier mean of only 26.5. Error analysis shows that failures concentrate in experimental protocol mismatch, evidence mismatch, and missing scientific core. ResearchClawBench provides a reproducible evaluation frontier for measuring progress toward autonomous scientific research.