AI-Driven Reproducibility and Analytical Robustness in Scientific Discovery
Authors: Shicheng Liu, Krista Opsahl-Ong, Andrew Drozdov, Julian Killingback, Arnav Singhvi, Ronak Pradeep, Michael Bendersky, James C. Dickerson, Monica S. Lam
Organizations: Department of Computer Science, Stanford University · Databricks AI Research · University of Massachusetts Amherst · Division of Oncology, Stanford University School of Medicine
Reproducible and analytically robust research findings are essential to scientific progress, but evaluating these properties typically requires costly manual effort. We investigate whether LLM agents can help automate reproduction and robustness analysis, using a medical case study grounded in the Surveillance, Epidemiology, and End Results (SEER) database, a widely used U.S. cancer registry underlying tens of thousands of published studies. First, we introduce SEER-Repro, a benchmark of 20 published SEER-based papers together with the data and resources required to reproduce their findings. Second, we create an AI agent that successfully reproduces 77.4% of 2,190 atomic claims, while providing executable baselines for further analysis. Third, we develop an agentic analysis pipeline to test the robustness of the medical findings by identifying analytical choices underlying each finding, proposing defensible alternatives, and rerunning the analysis to determine whether those choices affect the conclusion. Across published papers in SEER-Repro, our pipeline identifies potentially consequential analytical choices in 9.5% of fully reproduced findings and flags them for expert review. These results show how LLM agents can support scientific review by reproducing findings, systematically testing their robustness, and directing expert attention toward potentially consequential analytical choices.
Figures & tables
Figure 1: AI-assisted robustness assessment pipeline to support expert review.
Figure 2: AI-driven analytical robustness with 3 complementary methods discussed in Sec 3.2
Paper (shorter name)
Journal (year)
Cit.
Findings
Atomic claims
Reproduced Close Not reproduced
Bone sarcoma survival by tumor site ( Hu et al., 2021 )
Cancers (2021)
37
56
188
187 (99.5%)
Sex disparities in cancer incidence ( Cook et al., 2009 )
Cancer Epidemiol. Biomarkers Prev. (2009)
466
36
151
140 (92.7%)
Inflammatory breast cancer incidence and survival ( Hance et al., 2005 )
J. Natl. Cancer Inst. (2005)
702
40
131
119 (90.8%)
Thyroid cancer trends ( Davies and Welch, 2014 )
JAMA Otolaryngol. Head Neck Surg. (2014)
1,628
33
91
80 (87.9%)
Age at oropharyngeal cancer diagnosis ( Cline et al., 2020 )
Cancers (2020)
29
75
154
129 (83.8%)
Adenoid cystic carcinoma of the breast ( Ghabach et al., 2010 )
Breast Cancer Res. (2010)
188
51
128
103 (80.5%)
Table 1: Atomic reproduction outcomes for the 20-paper SEER-Repro benchmark using GPT-5.6 Sol. Appendix Table 8 gives exact outcome counts and percentages; Appendix G lists full paper titles; Appendix Table 9 separates quantitative and qualitative claims.
Category
Meaning
Claims
%
Data issues
Unavailable data (212 cases) or a data-related explanation such as mismatches attributed to revised records and population denominators (39 cases).
251
59.8
Insufficiently specified analysis
The available paper and documentation do not uniquely specify the target analysis.
47
11.2
Unavailable-software approximations
Method substitutions involving approximations of unavailable software such as NCI Joinpoint.
28
6.7
Agent-side errors
Departures from the reported method or comparison (25), incorrect cohort (13), statistical inference (3), denominator or standardization (6), or an omitted analysis (1).
48
11.4
Suspected evaluation errors
Suspected errors in our claim extraction or grading.
20
4.8
Unresolved cause
The evidence does not identify a sufficiently specific cause of non-reproduction.
26
6.2
Table 2: Classification of all 420 non-reproduced atomic claims from 20 papers in SEER-Repro.
Paper
Eligible findings
Findings with consequential links
Perturbation Agent
Direct coding agent
Found by
Sol
Opus
Kimi
GLM
Codex
Claude Code
≥1 analyst
≥2 analysts
Bone sarcoma survival by tumor site
55
2
–
2
–
–
–
4
–
Sex disparities in cancer incidence
31
–
–
–
–
–
–
–
–
Inflammatory breast cancer incidence and survival
31
–
3
–
–
2
3
4
3
Thyroid cancer trends
24
–
–
–
–
2
–
2
–
Table 3: Model-assessed consequential links (scores 7–9): finding counts by paper and analyst. Counts include individual links and members of consequential joint links, deduplicated per analyst.
Figure 3: GPT-5.6 Sol consequence scores for analyses of published papers; shading marks scores 7–9.
Paper and original conclusion
Alternative analysis and result
Expert assessment
Bone sarcoma survival by tumor site. Sol, Perturbation Agent; score 9. Adjusted cancer-specific mortality did not significantly improve in the 2000s or 2010s relative to the 1990s.
Replace the SEER “Bones and Joints” recode with direct primary-site codes C40.0–C41.9. The implementation added 3,974 records; both period comparisons then showed significant improvements.
Likely less defensible alternative. The added records change the disease population, and closer examination of the database contents and manuals suggests that the alternative is less defensible than the original choice.
Adenoid cystic carcinoma of the breast. Codex, direct analysis; score 8. Nonsignificant standardized incidence ratios (SIRs) were interpreted as an absence of associated cancers.
Interpret the estimates as no statistically detected excess, while acknowledging that the confidence intervals still permit increased risk. Failure to reject the null does not establish absence of excess risk.
Limited qualification. The distinction is reasonable, but the paper already qualifies its conclusions with caveats. The reviewer judged this a clarification of interpretation rather than a conclusion-changing result.
Head and neck mucosal melanoma incidence. Claude Code, direct analysis; score 8. Increasing incidence was attributed primarily to nasal-cavity disease.
Directly compare nasal and non-nasal trends rather than infer a difference from their separate significance tests. The analyst reported positive trends in both groups and a nonsignificant interaction ( p=0.46 ).
Flagged for further review. The comparison may identify an overstatement, but noisy data and model-fit concerns complicate interpretation. Closer manuscript review and statistical consultation are needed; a nonsignificant interaction does not establish equal trends.
Table 4: Three expert-reviewed cases, one per assessment category.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Evolution of the SEER registry groupings relevant to our study. Historical SEER 13 and 18 expand the SEER 9 cohort; their successor groupings exclude Detroit. The separately retained SEER 9 supplement covers 1975–2018. Registry memberships and retained coverage follow NCI documentation ( National Cancer Institute, 2026b ; National Cancer Institute, 2026d ) .
Table 5: Documentation resources provided in SEER-Repro. Source URLs are those recorded for the local copies; formats were verified from the files.
Table
Cols.
Rows
Description
SEER 9 case records
250
5,075,266
Case-level cancer-registry export from the SEER 9 Program.
Population 1969–2024, 20 age groups
13
17,485,723
County denominators; White/Black/Other.
Population 1990–2024, 20 age groups
13
22,178,259
County denominators; expanded race/Hispanic origin.
Population 1969–2024, single-year ages
13
69,655,933
County denominators; White/Black/Other.
Population 1990–2024, single-year ages
13
78,667,287
County denominators; expanded race/Hispanic origin.
Recent trends annual SEER incidence rates
13
1,648,320
Annual incidence rates.
Appendix
Table 6: The 39 curated tables, with schema column counts and loaded row counts. SEER*Explorer tables use the November 2024 archive; RUCC denotes rural–urban continuum codes.
Paper (shorter name)
Neutral research prompt
Bone sarcoma survival by tumor site
Analyze tumor-site-specific survival trends for bone sarcomas in SEER 9 from 1980 to 2018, especially variation by histology, site, stage, age, and calendar period.
Sex disparities in cancer incidence
Analyze sex disparities in cancer incidence across SEER 9 from 1975 to 2004, especially male-to-female incidence rate ratios by cancer site, histology, period, and age.
Inflammatory breast cancer incidence and survival
Analyze inflammatory breast carcinoma incidence and survival trends in SEER 9 from 1988 to 2000, especially differences from locally advanced and non-T4 breast cancer by race and subtype.
Thyroid cancer trends
Analyze thyroid cancer incidence and mortality trends in the United States from 1975 to 2009, especially by tumor size, stage, histology, age, sex, and race.
Age at oropharyngeal cancer diagnosis
Analyze changes in age at diagnosis of oropharyngeal cancer in SEER 9 from 1975 to 2016, especially by sex, race, and human-papillomavirus-related tumor site.
Adenoid cystic carcinoma of the breast
Analyze breast adenoid cystic carcinoma in SEER 9 from 1977 to 2006, especially incidence, clinicopathologic characteristics, subsequent primary cancers, and relative survival.
Appendix
Table 7: Neutral research prompts supplied to both report-generation agents for the 20 benchmark cases, in the same paper order as Table 1 .
Paper (shorter name)
Findings
Atomic claims
Reproduced
Close
Not reproduced
Bone sarcoma survival by tumor site
56
188
187 (99.5%)
0 (0.0%)
1 (0.5%)
Sex disparities in cancer incidence
36
151
140 (92.7%)
1 (0.7%)
10 (6.6%)
Inflammatory breast cancer incidence and survival
40
131
119 (90.8%)
2 (1.5%)
10 (7.6%)
Thyroid cancer trends
33
91
80 (87.9%)
1 (1.1%)
10 (11.0%)
Age at oropharyngeal cancer diagnosis
75
154
129 (83.8%)
9 (5.8%)
16 (10.4%)
Adenoid cystic carcinoma of the breast
51
128
103 (80.5%)
2 (1.6%)
23 (18.0%)
Appendix
Table 8: Tabular breakdown of the reproduction outcomes shown in Table 1 , in the same paper order. Each outcome cell gives the number of atomic claims and the percentage of that paper's atomic claims. Findings counts refer to distinct parent findings represented by these claims.
Quantitative claims
Qualitative claims
Paper
Cit.
1% tolerance
5% tolerance
10% tolerance
All
Reprod.
Not reprod.
Bone sarcoma survival by tumor site
37
119 (88.1%)
135 (100.0%)
135 (100.0%)
135 (100.0%)
52 (98.1%)
1 (1.9%)
Sex disparities in cancer incidence
466
87 (75.0%)
114 (98.3%)
115 (99.1%)
116 (100.0%)
26 (74.3%)
9 (25.7%)
Inflammatory breast cancer incidence and survival
702
78 (74.3%)
98 (93.3%)
100 (95.2%)
105 (100.0%)
21 (80.8%)
5 (19.2%)
Thyroid cancer trends
1,628
52 (61.9%)
74 (88.1%)
75 (89.3%)
84 (100.0%)
6 (85.7%)
1 (14.3%)
Age at oropharyngeal cancer diagnosis
29
101 (73.2%)
117 (84.8%)
126 (91.3%)
138 (100.0%)
12 (75.0%)
4 (25.0%)
Appendix
Table 9: Atomic reproduction outcomes by claim type for the 20-paper SEER-Repro benchmark.
Shorter name
Full paper title
Bone sarcoma survival by tumor site
Trends in Tumor Site-Specific Survival of Bone Sarcomas from 1980 to 2018: A Surveillance, Epidemiology and End Results-Based Study
Sex disparities in cancer incidence
Sex Disparities in Cancer Incidence by Period and Age
Inflammatory breast cancer incidence and survival
Trends in Inflammatory Breast Carcinoma Incidence and Survival: The Surveillance, Epidemiology, and End Results Program at the National Cancer Institute
Thyroid cancer trends
Current Thyroid Cancer Trends in the United States
Age at oropharyngeal cancer diagnosis
Change in Age at Diagnosis of Oropharyngeal Cancer in the United States, 1975–2016
Adenoid cystic carcinoma of the breast
Adenoid cystic carcinoma of the breast in the United States (1977 to 2006): a population-based cohort study
Appendix
Table 10: Full titles corresponding to the shorter paper names in Table 1 .
Reproducing research results from papers and released code is central to scientific progress. Existing works have introduced benchmarks to evaluate whether LLM agents can assist with reproducibility, but they are difficult to scale due to their reliance on substantial manual effort for data curation and evaluation. We introduce ReproRepo, a scalable framework for reproducibility evaluation that leverages human-raised GitHub issues as naturally occurring supervision on realistic reproduction blockers. We instantiate ReproRepo on 1,149 recent machine learning papers from major conferences and evaluate four frontier model-agent configurations. Our results show that LLM agents, even without executing code, can identify many real-world reproducibility problems from paper-repository pairs: the best agent in our study, namely Codex with GPT-5.5, surfaces at least one semantically related human-reported blocker for ∼90% of papers in the study. Further analysis shows that agents are particularly effective for surfacing visible failures and identifying the right semantic region, but may still be insufficient in exact localization. ReproRepo can serve as a reusable, scalable framework for future evaluations of LLM agents on real-world reproducibility auditing. Our code is released at https://github.com/LithiumDA/ReproRepo.
Shanda Li, Qiuhong Anna Wei, Jingwu Tang +5
School of Computer Science, Carnegie Mellon University · Datadog
Recent work has used LLM agents to reproduce empirical social science results with access to both the data and code. We broaden this scope by asking: Can they reproduce results given only a paper's methods description and original data? We develop an agentic reproduction system that extracts structured methods descriptions from papers, runs reimplementations under strict information isolation -- agents never see the original code, results, or paper -- and enables deterministic, cell-level comparison of reproduced outputs to the original results. An error attribution step traces discrepancies through the system chain to identify root causes. Evaluating four agent scaffolds and four LLMs on 48 papers with human-verified reproducibility, we find that agents can largely recover published results, but performance varies substantially between models, scaffolds, and papers. Root cause analysis reveals that failures stem both from agent errors and from underspecification in the papers themselves.
Benjamin Kohler, David Zollikofer, Johanna Einsiedler +2
Scientific peer review increasingly struggles to assess reproducibility at the scale and complexity of modern research output. Evaluating reproducibility requires reconstructing experimental dependencies, methodological choices, data flows, and result-generating procedures, which often exceeds what human reviewers can provide. Agentic Reproducibility Assessment (ARA) formalizes reproducibility assessment as a structured reasoning task over scientific documents. Given a paper, ARA extracts a directed workflow graph linking sources, methods, experiments, and outputs, then evaluates its reconstructability using structural and content-based scores for reproducibility assessments. Experiments on 213 ReScience C articles - the largest cross-domain benchmark of human-validated computational reproducibility studies considered to date - demonstrate ARA's generalizability and consistent workflow reconstruction and assessment across LLMs, model temperatures, and scientific domains. ARA achieves ~61% accuracy on three benchmarks, and the highest accuracy reported on ReproBench (60.71% vs. 36.84%) and GoldStandardDB (61.68% vs. 43.56%), highlighting its potential to complement human review at scale and enabling next-generation peer review. Code and Data available: https://github.com/AndresLaverdeMarin/agentic_reproducibility_assessment.
Kevin Riehl, Andres L. Marin, Nikofors Zacharof +6
European Commission, Joint Research Centre · Ideas Forward