cs.AIOct 16, 2026
SaveAI-Driven Reproducibility and Analytical Robustness in Scientific Discovery
Organizations: Department of Computer Science, Stanford University · Databricks AI Research · University of Massachusetts Amherst · Division of Oncology, Stanford University School of Medicine
Abstract
Reproducible and analytically robust research findings are essential to scientific progress, but evaluating these properties typically requires costly manual effort. We investigate whether LLM agents can help automate reproduction and robustness analysis, using a medical case study grounded in the Surveillance, Epidemiology, and End Results (SEER) database, a widely used U.S. cancer registry underlying tens of thousands of published studies. First, we introduce SEER-Repro, a benchmark of 20 published SEER-based papers together with the data and resources required to reproduce their findings. Second, we create an AI agent that successfully reproduces 77.4% of 2,190 atomic claims, while providing executable baselines for further analysis. Third, we develop an agentic analysis pipeline to test the robustness of the medical findings by identifying analytical choices underlying each finding, proposing defensible alternatives, and rerunning the analysis to determine whether those choices affect the conclusion. Across published papers in SEER-Repro, our pipeline identifies potentially consequential analytical choices in 9.5% of fully reproduced findings and flags them for expert review. These results show how LLM agents can support scientific review by reproducing findings, systematically testing their robustness, and directing expert attention toward potentially consequential analytical choices.
Figures & tables
Figure 1: AI-assisted robustness assessment pipeline to support expert review.
Figure 2: AI-driven analytical robustness with 3 complementary methods discussed in Sec 3.2
| Paper (shorter name) | Journal (year) | Cit. | Findings | Atomic claims | Reproduced Close Not reproduced |
| Bone sarcoma survival by tumor site ( Hu et al., 2021 ) | Cancers (2021) | 37 | 56 | 188 | 187 (99.5%) |
| Sex disparities in cancer incidence ( Cook et al., 2009 ) | Cancer Epidemiol. Biomarkers Prev. (2009) | 466 | 36 | 151 | 140 (92.7%) |
| Inflammatory breast cancer incidence and survival ( Hance et al., 2005 ) | J. Natl. Cancer Inst. (2005) | 702 | 40 | 131 | 119 (90.8%) |
| Thyroid cancer trends ( Davies and Welch, 2014 ) | JAMA Otolaryngol. Head Neck Surg. (2014) | 1,628 | 33 | 91 | 80 (87.9%) |
| Age at oropharyngeal cancer diagnosis ( Cline et al., 2020 ) | Cancers (2020) | 29 | 75 | 154 | 129 (83.8%) |
| Adenoid cystic carcinoma of the breast ( Ghabach et al., 2010 ) | Breast Cancer Res. (2010) | 188 | 51 | 128 | 103 (80.5%) |
Table 1: Atomic reproduction outcomes for the 20-paper SEER-Repro benchmark using GPT-5.6 Sol. Appendix Table 8 gives exact outcome counts and percentages; Appendix G lists full paper titles; Appendix Table 9 separates quantitative and qualitative claims.
| Category | Meaning | Claims | % |
|---|---|---|---|
| Data issues | Unavailable data (212 cases) or a data-related explanation such as mismatches attributed to revised records and population denominators (39 cases). | 251 | 59.8 |
| Insufficiently specified analysis | The available paper and documentation do not uniquely specify the target analysis. | 47 | 11.2 |
| Unavailable-software approximations | Method substitutions involving approximations of unavailable software such as NCI Joinpoint. | 28 | 6.7 |
| Agent-side errors | Departures from the reported method or comparison (25), incorrect cohort (13), statistical inference (3), denominator or standardization (6), or an omitted analysis (1). | 48 | 11.4 |
| Suspected evaluation errors | Suspected errors in our claim extraction or grading. | 20 | 4.8 |
| Unresolved cause | The evidence does not identify a sufficiently specific cause of non-reproduction. | 26 | 6.2 |
Table 2: Classification of all 420 non-reproduced atomic claims from 20 papers in SEER-Repro.
| Paper | Eligible findings | Findings with consequential links | |||||||
| Perturbation Agent | Direct coding agent | Found by | |||||||
| Sol | Opus | Kimi | GLM | Codex | Claude Code | analyst | analysts | ||
| Bone sarcoma survival by tumor site | 55 | 2 | – | 2 | – | – | – | 4 | – |
| Sex disparities in cancer incidence | 31 | – | – | – | – | – | – | – | – |
| Inflammatory breast cancer incidence and survival | 31 | – | 3 | – | – | 2 | 3 | 4 | 3 |
| Thyroid cancer trends | 24 | – | – | – | – | 2 | – | 2 | – |
Table 3: Model-assessed consequential links (scores 7–9): finding counts by paper and analyst. Counts include individual links and members of consequential joint links, deduplicated per analyst.
Figure 3: GPT-5.6 Sol consequence scores for analyses of published papers; shading marks scores 7–9.
| Paper and original conclusion | Alternative analysis and result | Expert assessment |
|---|---|---|
| Bone sarcoma survival by tumor site. Sol, Perturbation Agent; score 9. Adjusted cancer-specific mortality did not significantly improve in the 2000s or 2010s relative to the 1990s. | Replace the SEER “Bones and Joints” recode with direct primary-site codes C40.0–C41.9. The implementation added 3,974 records; both period comparisons then showed significant improvements. | Likely less defensible alternative. The added records change the disease population, and closer examination of the database contents and manuals suggests that the alternative is less defensible than the original choice. |
| Adenoid cystic carcinoma of the breast. Codex, direct analysis; score 8. Nonsignificant standardized incidence ratios (SIRs) were interpreted as an absence of associated cancers. | Interpret the estimates as no statistically detected excess, while acknowledging that the confidence intervals still permit increased risk. Failure to reject the null does not establish absence of excess risk. | Limited qualification. The distinction is reasonable, but the paper already qualifies its conclusions with caveats. The reviewer judged this a clarification of interpretation rather than a conclusion-changing result. |
| Head and neck mucosal melanoma incidence. Claude Code, direct analysis; score 8. Increasing incidence was attributed primarily to nasal-cavity disease. | Directly compare nasal and non-nasal trends rather than infer a difference from their separate significance tests. The analyst reported positive trends in both groups and a nonsignificant interaction ( ). | Flagged for further review. The comparison may identify an overstatement, but noisy data and model-fit concerns complicate interpretation. Closer manuscript review and statistical consultation are needed; a nonsignificant interaction does not establish equal trends. |
Table 4: Three expert-reviewed cases, one per assessment category.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Evolution of the SEER registry groupings relevant to our study. Historical SEER 13 and 18 expand the SEER 9 cohort; their successor groupings exclude Detroit. The separately retained SEER 9 supplement covers 1975–2018. Registry memberships and retained coverage follow NCI documentation ( National Cancer Institute, 2026b ; National Cancer Institute, 2026d ) .
| Resource | Source URL | Format |
|---|---|---|
| SEER*Stat resources (6) | ||
| SEER*Stat November 2025 submission overview | https://seer.cancer.gov/data-software/documentation/seerstat/nov2025/ | HTML |
| Variable dictionary spreadsheet for SEER*Stat variables | https://seer.cancer.gov/data-software/documentation/seerstat/nov2025/seerstat-variable-dictionary-nov2025.xlsx | XLSX |
| Variable dictionary PDF for SEER*Stat variables | https://seer.cancer.gov/data-software/documentation/seerstat/nov2025/seerstat-variable-dictionary-nov2025.pdf | |
| Text data file description for SEER Research and Research Plus | https://seer.cancer.gov/data-software/documentation/seerstat/nov2025/TextData.FileDescription-nov2025.pdf | |
| SEER product comparison and availability notes | https://seer.cancer.gov/data/product-comparison.html | HTML |
Table 5: Documentation resources provided in SEER-Repro. Source URLs are those recorded for the local copies; formats were verified from the files.
| Table | Cols. | Rows | Description |
|---|---|---|---|
| SEER 9 case records | 250 | 5,075,266 | Case-level cancer-registry export from the SEER 9 Program. |
| Population 1969–2024, 20 age groups | 13 | 17,485,723 | County denominators; White/Black/Other. |
| Population 1990–2024, 20 age groups | 13 | 22,178,259 | County denominators; expanded race/Hispanic origin. |
| Population 1969–2024, single-year ages | 13 | 69,655,933 | County denominators; White/Black/Other. |
| Population 1990–2024, single-year ages | 13 | 78,667,287 | County denominators; expanded race/Hispanic origin. |
| Recent trends annual SEER incidence rates | 13 | 1,648,320 | Annual incidence rates. |
Table 6: The 39 curated tables, with schema column counts and loaded row counts. SEER*Explorer tables use the November 2024 archive; RUCC denotes rural–urban continuum codes.
| Paper (shorter name) | Neutral research prompt |
|---|---|
| Bone sarcoma survival by tumor site | Analyze tumor-site-specific survival trends for bone sarcomas in SEER 9 from 1980 to 2018, especially variation by histology, site, stage, age, and calendar period. |
| Sex disparities in cancer incidence | Analyze sex disparities in cancer incidence across SEER 9 from 1975 to 2004, especially male-to-female incidence rate ratios by cancer site, histology, period, and age. |
| Inflammatory breast cancer incidence and survival | Analyze inflammatory breast carcinoma incidence and survival trends in SEER 9 from 1988 to 2000, especially differences from locally advanced and non-T4 breast cancer by race and subtype. |
| Thyroid cancer trends | Analyze thyroid cancer incidence and mortality trends in the United States from 1975 to 2009, especially by tumor size, stage, histology, age, sex, and race. |
| Age at oropharyngeal cancer diagnosis | Analyze changes in age at diagnosis of oropharyngeal cancer in SEER 9 from 1975 to 2016, especially by sex, race, and human-papillomavirus-related tumor site. |
| Adenoid cystic carcinoma of the breast | Analyze breast adenoid cystic carcinoma in SEER 9 from 1977 to 2006, especially incidence, clinicopathologic characteristics, subsequent primary cancers, and relative survival. |
Table 7: Neutral research prompts supplied to both report-generation agents for the 20 benchmark cases, in the same paper order as Table 1 .
| Paper (shorter name) | Findings | Atomic claims | Reproduced | Close | Not reproduced |
|---|---|---|---|---|---|
| Bone sarcoma survival by tumor site | 56 | 188 | 187 (99.5%) | 0 (0.0%) | 1 (0.5%) |
| Sex disparities in cancer incidence | 36 | 151 | 140 (92.7%) | 1 (0.7%) | 10 (6.6%) |
| Inflammatory breast cancer incidence and survival | 40 | 131 | 119 (90.8%) | 2 (1.5%) | 10 (7.6%) |
| Thyroid cancer trends | 33 | 91 | 80 (87.9%) | 1 (1.1%) | 10 (11.0%) |
| Age at oropharyngeal cancer diagnosis | 75 | 154 | 129 (83.8%) | 9 (5.8%) | 16 (10.4%) |
| Adenoid cystic carcinoma of the breast | 51 | 128 | 103 (80.5%) | 2 (1.6%) | 23 (18.0%) |
Table 8: Tabular breakdown of the reproduction outcomes shown in Table 1 , in the same paper order. Each outcome cell gives the number of atomic claims and the percentage of that paper's atomic claims. Findings counts refer to distinct parent findings represented by these claims.
| Quantitative claims | Qualitative claims | ||||||
|---|---|---|---|---|---|---|---|
| Paper | Cit. | 1% tolerance | 5% tolerance | 10% tolerance | All | Reprod. | Not reprod. |
| Bone sarcoma survival by tumor site | 37 | 119 (88.1%) | 135 (100.0%) | 135 (100.0%) | 135 (100.0%) | 52 (98.1%) | 1 (1.9%) |
| Sex disparities in cancer incidence | 466 | 87 (75.0%) | 114 (98.3%) | 115 (99.1%) | 116 (100.0%) | 26 (74.3%) | 9 (25.7%) |
| Inflammatory breast cancer incidence and survival | 702 | 78 (74.3%) | 98 (93.3%) | 100 (95.2%) | 105 (100.0%) | 21 (80.8%) | 5 (19.2%) |
| Thyroid cancer trends | 1,628 | 52 (61.9%) | 74 (88.1%) | 75 (89.3%) | 84 (100.0%) | 6 (85.7%) | 1 (14.3%) |
| Age at oropharyngeal cancer diagnosis | 29 | 101 (73.2%) | 117 (84.8%) | 126 (91.3%) | 138 (100.0%) | 12 (75.0%) | 4 (25.0%) |
Table 9: Atomic reproduction outcomes by claim type for the 20-paper SEER-Repro benchmark.
| Shorter name | Full paper title |
|---|---|
| Bone sarcoma survival by tumor site | Trends in Tumor Site-Specific Survival of Bone Sarcomas from 1980 to 2018: A Surveillance, Epidemiology and End Results-Based Study |
| Sex disparities in cancer incidence | Sex Disparities in Cancer Incidence by Period and Age |
| Inflammatory breast cancer incidence and survival | Trends in Inflammatory Breast Carcinoma Incidence and Survival: The Surveillance, Epidemiology, and End Results Program at the National Cancer Institute |
| Thyroid cancer trends | Current Thyroid Cancer Trends in the United States |
| Age at oropharyngeal cancer diagnosis | Change in Age at Diagnosis of Oropharyngeal Cancer in the United States, 1975–2016 |
| Adenoid cystic carcinoma of the breast | Adenoid cystic carcinoma of the breast in the United States (1977 to 2006): a population-based cohort study |
Table 10: Full titles corresponding to the shorter paper names in Table 1 .
More in this topic
AI-Assisted Scientific Research
AI-assisted scientific research uses machine learning and AI agents to support discovery, analysis, experimentation, and scientific communication.
See all 250 papers