cs.AIOct 15, 2026
SaveFrom Research Goals to Scientific Closure: Evaluating Agents Through Evidence-Supported Physics Research
Organizations: Shanghai AI Laboratory · Shanghai Jiao Tong University · The HongKong University of Science and Technology(Guangzhou) · Nanyang Technological University
Abstract
Can AI agents organize scientific investigations that lead to well-supported conclusions? To evaluate this capability, we introduce PhysClosureBench, a benchmark of 50 expert-verified, goal-specified research tasks spanning strong-field, attosecond, and neighboring physics areas. Each task specifies the research objective and essential scientific constraints, while leaving the agent to organize the investigation and determine how to generate and validate the evidence needed to answer the question. A 20-criterion Atomic Score measures local scientific accomplishment, while Research Closure (RC0--RC4) assesses how fully the investigation resolves the specified research question based on whether its indispensable evidence requirements and applicable scope conditions are satisfied. We evaluate seven frontier LLMs, where Claude-Opus-5 achieves the highest mean Atomic Score (6.29/10) but reaches RC4 in only 32.7% of attempts. Across the benchmark, 36.2% of attempts scoring at least 8/10 still fall short of RC4, while all seven models reach RC4 on substantially fewer tasks in all three runs than in at least one. Case-level analysis further illustrates how misinterpretation of generated results and unresolved physical applicability limits can lead to incomplete closure. Together, these results reveal a persistent gap between AI agents' local scientific accomplishments and their ability to achieve complete research closure.
Figures & tables
Figure 1: From a scientific goal to scientific closure. In PhysClosureBench, agents are required to achieve the provided scientific objective within the specified task setting and a resource-bounded computational environment.
| Benchmark | Task formulation | Execution / artifacts | Research decisions | Evidence integration | Scientific closure |
|---|---|---|---|---|---|
| GPQA / HLE | Expert-level QA | ||||
| CritPt | Physics reasoning | ✓ | |||
| SciCode | Scientific coding | ✓ | |||
| PaperBench | Research reproduction | ✓ | Partial | Partial | |
| RE-Bench | Research-engineering tasks | ✓ | Partial | Partial | |
| FrontierScience-R | Scientific research subproblems | ✓ | Partial | Partial |
Table 1: Comparison of representative scientific AI benchmarks. A checkmark indicates direct assessment of a dimension, while “Partial” indicates limited or indirect assessment and a cross indicates no explicit assessment. Scientific closure concerns whether an investigation establishes a well-supported answer to its research question, including the required validation and scope conditions.
Figure 2: Two complementary views of scientific performance. Atomic Score (0–10) measures local accomplishment across F/O/I/M/V. Research Closure (RC0–RC4) assesses how fully the investigation resolves the research question. A high Atomic Score does not guarantee complete closure.
Figure 3: Scientific accomplishment and Research Closure across frontier models. (A) Mean Atomic Score and functional completion for Formulation (F), Output (O), Identification (I), and Mechanism (M). Atomic Scores are shown on the shared axis after multiplication by 10; functional completion is the mean attempt-level fraction of applicable criteria satisfied within each dimension. (B) Distribution of Research Closure levels across independent attempts. RC4 denotes complete closure, RC3 indicates that the main conclusion is supported, but material validation or scope gaps remain, and RC0–RC2 indicate different degrees of incomplete investigation.
Figure 4: Relationship between local scientific accomplishment and Research Closure. (A) Joint distribution of Atomic Score and Research Closure across all 1,050 independent attempts. Each cell reports the percentage of all attempts falling into the corresponding score bin and RC level. The highlighted region marks high-scoring attempts below RC4. (B) Unmet applicable criteria for Formulation (F), Output (O), Identification (I), and Mechanism (M) among the 229 attempts scoring at least 8/10, comparing RC4 ( ) with below RC4 ( ). Bars show 100% minus the mean attempt-level functional completion within each group. The inset reports the fraction of satisfied applicable Validation/Scope (V) criteria across the full benchmark, rather than within the high-scoring groups; N/A criteria are excluded.
Figure 5: Representative evidence and scientific closure. Case 30 compares representative runs from Kimi-K3, GPT-5.6-Sol, and Gemini-3.1-Pro, which reached RC4, RC3, and RC2, respectively. The shaded region illustrates a possible progression from numerical evidence to a supported conclusion; Gemini’s deviation represents the misinterpretation of the computed intensity trends. Paths are schematic, not execution times, Atomic Scores, or intermediate RC assignments. Inset plots schematically illustrate trends in the submitted results. Appendix E documents the research activities and evidence.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Domain distribution of the 50 benchmark tasks.
| Case | CPU cores | Minutes | Case | CPU cores | Minutes |
|---|---|---|---|---|---|
| Case01 | 8 | 30 | Case26 | 16 | 90 |
| Case02 | 16 | 30 | Case27 | 8 | 60 |
| Case03 | 32 | 240 | Case28 | 16 | 60 |
| Case04 | 8 | 30 | Case29 | 32 | 90 |
| Case05 | 32 | 45 | Case30 | 8 | 45 |
| Case06 | 8 | 45 | Case31 | 96 | 360 |
Table 4: Case-level CPU and wall-time limits for the 50 benchmark tasks.
| Dimension | Total | Mean | Median | Range | Mean score weight |
|---|---|---|---|---|---|
| F | 164 | 3.28 | 3 | 2–4 | 16.4% |
| O | 192 | 3.84 | 4 | 2–6 | 19.2% |
| I | 163 | 3.26 | 3 | 3–4 | 16.3% |
| M | 369 | 7.38 | 7 | 7–9 | 36.9% |
| V | 112 | 2.24 | 2 | 1–3 | 11.2% |
| Total | 1000 | 20.00 | 20 | 20–20 | 100.0% |
Table 5: Atom allocation across 50 cases. Total atom counts across all cases; Mean and Median are calculated from the 50 per-case atom counts for each dimension; Range gives their minimum and maximum. Mean score weight is the average percentage of each task’s 10-point maximum allocated to a dimension.
| Model | F (%) | O (%) | I (%) | M (%) | V (%) | V satisfied/applicable |
|---|---|---|---|---|---|---|
| Claude Opus 5 | 85.6 | 60.1 | 54.6 | 52.8 | 84.0 | 263/313 |
| GPT-5.6 Sol | 83.8 | 60.5 | 47.9 | 46.0 | 74.2 | 241/325 |
| Kimi K3 | 77.4 | 49.7 | 43.4 | 40.9 | 68.4 | 214/313 |
| DeepSeek-V4.1-Flash | 67.2 | 37.3 | 35.2 | 35.7 | 71.3 | 214/300 |
| Qwen3.8-Max | 62.3 | 28.7 | 24.1 | 25.9 | 54.7 | 156/285 |
| GLM-5.3-Flash | 56.7 | 23.7 | 20.3 | 23.3 | 65.6 | 177/270 |
Table 6: Functional completion by model. F/O/I/M values are mean attempt-level completion rates, with terminal-zero attempts contributing zero. V values represent the percentage of applicable V criteria satisfied, excluding N/A criteria and terminal-zero attempts without atom records.
| Model | Run | F | O | I | M | V | V satisfied/applicable |
|---|---|---|---|---|---|---|---|
| Claude Opus 5 | R1 | 88.2 | 62.5 | 58.3 | 56.2 | 84.1 | 90/107 |
| R2 | 83.7 | 63.6 | 56.8 | 55.5 | 82.5 | 85/103 | |
| R3 | 84.8 | 54.4 | 48.5 | 46.9 | 85.4 | 88/103 | |
| GPT-5.6 Sol | R1 | 86.8 | 58.7 | 49.3 | 48.3 | 76.8 | 86/112 |
| R2 | 78.5 | 57.0 | 39.0 | 39.6 | 66.3 | 67/101 | |
| R3 | 86.2 | 65.9 | 55.3 | 50.0 | 78.6 | 88/112 |
Table 7: Functional completion by models in three independent runs. Measures and applicability conventions follow Table 6 .
| Model | N | RC0 | RC1 | RC2 | RC3 | RC4 |
|---|---|---|---|---|---|---|
| Claude Opus 5 | 150 | 24 (16.0%) | 49 (32.7%) | 25 (16.7%) | 3 (2.0%) | 49 (32.7%) |
| GPT-5.6 Sol | 150 | 28 (18.7%) | 38 (25.3%) | 43 (28.7%) | 7 (4.7%) | 34 (22.7%) |
| Kimi K3 | 150 | 48 (32.0%) | 36 (24.0%) | 32 (21.3%) | 5 (3.3%) | 29 (19.3%) |
| DeepSeek-V4.1-Flash | 150 | 75 (50.0%) | 25 (16.7%) | 30 (20.0%) | 4 (2.7%) | 16 (10.7%) |
| Qwen3.8-Max | 150 | 76 (50.7%) | 39 (26.0%) | 22 (14.7%) | 2 (1.3%) | 11 (7.3%) |
| GLM-5.3-Flash | 150 | 87 (58.0%) | 32 (21.3%) | 19 (12.7%) | 4 (2.7%) | 8 (5.3%) |
Table 8: Research Closure distribution by model. Cells report count (percentage) over the 150 final scored attempts for each model.
| Model | Run | N | RC0 | RC1 | RC2 | RC3 | RC4 |
|---|---|---|---|---|---|---|---|
| Claude Opus 5 | R1 | 50 | 8 | 14 | 7 | 1 | 20 |
| R2 | 50 | 11 | 9 | 12 | 2 | 16 | |
| R3 | 50 | 5 | 26 | 6 | 0 | 13 | |
| GPT-5.6 Sol | R1 | 50 | 7 | 13 | 14 | 5 | 11 |
| R2 | 50 | 15 | 9 | 13 | 2 | 11 | |
| R3 | 50 | 6 | 16 | 16 | 0 | 12 |
Table 9: Research Closure counts by model and independent run. Each row reports 50 scored attempts.
| V category | Description | Satisfied/applicable | Rate |
|---|---|---|---|
| Scope justification | Justify the conditions under which the conclusions are valid. | 852/981 | 86.9% |
| Required validation | Perform validation checks required by the task, regardless of the chosen method. | 93/156 | 59.6% |
| Risk-linked validation | Address specific risks arising from the method, results, or conclusions. | 441/922 | 47.8% |
Table 10: Validation/Scope criteria across the benchmark.
Figure 8: Atomic-score distributions across investigations. Kernel density estimates show the distribution of Atomic Scores over all final scored attempts for each model, including terminal-zero outcomes. Dashed lines indicate mean Atomic Scores for each model. The distributions summarize score variation across the full set of benchmark investigations.
| Investigation | Selected research steps | Key finding or unresolved issue |
|---|---|---|
| Kimi K3 10.0/RC4 | Constructed scattering calculations; detected an angle-sign inconsistency; corrected and checked the results; interpreted the screening dependence. | Supported the interpretation within a physically justified scope. |
| GPT-5.6 Sol 9.5/RC3 | Repaired the numerical solver; identified screening-dependent trends; explained their mechanism; recorded a high-field diagnostic. | Did not resolve the applicability concern revealed by its high-field diagnostic. |
| Gemini 3.1 Pro 4.5/RC2 | Revised trajectory propagation and angle convention; generated Coulomb and screened results; summarized the intensity dependence. | Misidentified increasing or turning-over screened curves as monotonically decreasing. |
Table 11: Selected Case 30 research activities and key findings. The steps summarize recorded activities, not a complete or strictly ordered execution history. Scores and RC levels refer to final submissions.
Figure 9: Screened offsets contradict Gemini’s decreasing-trend claim. Replotted from the agent’s submitted numerical data. All 100 recorded points per curve are shown, with straight connectors and no fitting or smoothing. The angle is the submitted outgoing-trajectory offset in degrees, with ; is in atomic units.
| Model | Mean score | Median | RC4 | 1 RC4 | All 3 RC4 |
|---|---|---|---|---|---|
| Claude Opus 5 | 6.29 | 6.75 | 32.7% | 50.0% | 18.0% |
| GPT-5.6 Sol | 5.83 | 6.00 | 22.7% | 40.0% | 10.0% |
| Kimi K3 | 5.17 | 5.00 | 19.3% | 40.0% | 4.0% |
| DeepSeek-V4.1-Flash | 4.50 | 4.00 | 10.7% | 24.0% | 0.0% |
| Qwen3.8-Max | 3.46 | 2.00 | 7.3% | 14.0% | 2.0% |
| GLM-5.3-Flash | 3.28 | 2.25 | 5.3% | 12.0% | 0.0% |
Table 12: Model-level performance and RC4 consistency. Mean and median Atomic Scores and attempt-level RC4 rates are computed over 150 final scored attempts per model. The two RC4 consistency measures are computed over the same 50-task denominator for every model.
| Atom | Function | Representative criterion |
|---|---|---|
| F1 | Formulation | Relate the outgoing half of the central-potential deflection to the attoclock offset, using consistent exit and field/vector-potential conventions, units, and approximation limits. |
| O2 | Output | Calculate auditable offset-versus-intensity results for Coulomb and screened potentials matched to the same binding energy, including and enough intermediate screening lengths to resolve changes with potential range. |
| I2 | Identification | Identify the approach to Coulomb behavior at long range, changes in intensity dependence at intermediate screening lengths, and the suppression and relative offsets at strong screening supported by the data. |
| M2 | Mechanism | Explain how the electron’s exit or impact distance, asymptotic kinetic energy, and potential range compete to change the offset response as screening length decreases. |
| V1 | Validation/Scope | When numerical choices, binding-energy calibration, sampling, or curve structure raise a concrete risk to the conclusion, test that risk or restrict the conclusion accordingly. If no risk is triggered, record N/A without claiming positive validation evidence. |
| V3 | Validation/Scope | Specify which conclusions the reduced scattering calculation supports, distinguishing them from unverified real-atom, quantum-dynamical, or universal tunnelling-time interpretations. |
Table 13: Representative Case 30 atomic criteria. Criteria are condensed from the full case-specific grading rubric for readability.