Scientific Reproducibility

Ask about this topic

Overview

The field asks whether findings can be recovered from reported methods and artifacts, and how measurement, preprocessing, statistical analysis, software, and evaluation choices affect that recovery. Common approaches include sharing versioned data, code, and environments; preregistration and provenance records; standardized workflows and checklists; independent reruns; and benchmarks that test coding or language-model agents end to end. Audits across fields reveal missing artifacts, underspecified procedures, hidden analytic flexibility, and sensitivity to datasets, configurations, or software; agents can reduce reproduction effort but still struggle with tacit assumptions and validating scientific meaning rather than merely producing runnable code. The central challenge is turning transparency into faithful, durable verification: open artifacts and automation help, but neither guarantees comparable results nor establishes that a claim is sound.

Latest papers

CardsList

108 papers13 papers in the last four weeks, level with the four weeks before. 0.1% of all new papers.

  1. AI-Driven Reproducibility and Analytical Robustness in Scientific Discovery

    Oct 16, 2026Shicheng Liu, Krista Opsahl-Ong, Andrew Drozdov +6AI-Assisted Scientific ResearchScientific Reproducibility

    Department of Computer Science, Stanford University · Databricks AI Research · University of Massachusetts Amherst +1

  2. Author-in-the-loop Agentic Reproducibility Auditing for a Machine Learning Conference

    Oct 15, 2026Yukun Dong, Linjun Zhang, Tian ZhengML ReproducibilityScientific Reproducibility

    Rutgers University · Columbia University

  3. AIPOCH Open-Science: An Open-Source AI Research Workbench with Reproducible .science Packages

    Oct 12, 2026Wen Ouyang, Bocheng Huang, Qianyu Yao +76Scientific ReproducibilityAI-Assisted Scientific Research

  4. Re-executing Coding-Agent Evaluations: Unchanged Decisions and Unresolved Policy Risk

    Oct 10, 2026Tianyu Ding, Yaocheng Tong, Muzhi Yang +3Benchmark ReproducibilityAutomated Agent Evaluation

    Independent Researchers

  5. Kernel-based External Validity of Experimental Studies

    Oct 9, 2026Federico Matteucci, Gerhard NeumannScientific Reproducibility

    Karlsruhe Institute of Technology

  6. BioStudyBench: Evaluating Agents on Post-Cutoff Biomedical Studies

    Oct 6, 2026David Li, Shaamil Karim, Christian GensbiglerScientific Agent BenchmarksAI Agent Benchmarks

    University of Waterloo

  7. Kurate: Scalable Scientific Quality Analysis

    Oct 5, 2026Matthew J. Vowels, Jamie CumminsScientific Reproducibility

    Kivira Health, U.S.A. · University of Bern, Bern, Switzerland

  8. An Automated and Reproducible Workflow for Crack Identification and Damage Assessment of Fusion Materials

    Oct 2, 2026Rinkle Juneja, Viktor Reshniak, Richard K. Archibald +4Microscopy Image AnalysisMaterials Science

    Oak Ridge National Laboratory, Oak Ridge, TN 37831

  9. Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated Software

    Sep 30, 2026Bhanu Prakash Vangala, Tanu MalikCoding AgentsCode Generation

    Department of Electrical Engineering and Computer Science, University of Missouri–Columbia, Missouri, USA

  10. Can an AI Agent Rediscover a Blaschke-Curve Invariant?

    Sep 29, 2026Yunus E. ZeytuncuAI Agents for Scientific DiscoveryAI Agent Evaluation

    University of Michigan-Dearborn

  11. Research-Native by Construction: Minimal Nodes, Re-verifiable Workflows, and Compounding Memory for Long-Horizon Scientific Agents

    Sep 28, 2026Di Wang, Yu Liu, Bing Cui +5Runtime Enforcement for AI AgentsScientific Reproducibility

    IdeaHorizon Team

  12. Papers Without Code: Availability of GitHub Repositories Linked in *CL Publications

    Sep 28, 2026Selina Meyer, Michael RothScientific Reproducibility

    University of Technology Nuremberg

  13. Reconstructing Implicit Scientific Knowledge: Evaluating LLM Agents through End-to-End Reproduction of Astronomy

    Sep 28, 2026Yuehui Wang, Xinyu Qi, Guirong Xue +4Scientific ReproducibilityScientific Reasoning in Language Models

    Research Center for Scientific Data Hub, Zhejiang Lab, Hangzhou, China

  14. Position: Let's Strengthen Verifiability If We Can't Enforce Reproducibility

    Sep 25, 2026Samet Hicsonmez, Nermin Samet, Renaud MarletML ReproducibilityScientific Reproducibility

    Independent Researcher Paris, France · Valeo.ai Paris, France

  15. Amplify: A Lightweight Library for Reproducible Nonlinear Programming Problems in Robotics

    Sep 23, 2026Nelson RosaRobotic OptimizationTrajectory Optimization

  16. Reproducible AI Requires Reproducible Randomness

    Sep 22, 2026Anthony Bertrand, Tom Schmitt, Engelbert Mephu Nguifo +1ML ReproducibilityScientific Reproducibility

    UCA, LIMOS · UCA · LIMOS, UCA +1

  17. Gricea: An Open Science Platform for Conversational AI Research

    Sep 18, 2026Nikhil Sharma, Yunlin Gong, Xinyang Cheng +1Scientific ReproducibilityConversational Agents

    Johns Hopkins University Baltimore, MD, USA

  18. PaperDoctor: Evidence-Grounded and Actionable Feedback for Scientific Papers in Progress

    Sep 15, 2026Kevin Qinghong Lin, Siyuan Hu, Pan Lu +14AI-Assisted Scientific ResearchScientific Reproducibility

    University of Oxford · 3National University of Singapore · 2Stanford University +3

  19. A Programmable Optics Cloud Laboratory

    Sep 14, 2026Sachin Vaidya, Caio Silva, Seou Choi +2Laboratory AutomationScientific Reproducibility

    Research Laboratory of Electronics, Massachusetts Institute of Technology, Cambridge, MA 02139, USA · NSF Institute for Artificial Intelligence and Fundamental Interactions and the Department of Physics, Massachusetts Institute of Technology, Cambridge, MA 02139, USA

  20. OpenAI4S: Code as Action, Science as Sessions

    Sep 14, 2026Gongbo Zhang, Hao Li, Yu Wang +15Data ProvenanceRuntime Enforcement for AI Agents

    Peking University Shenzhen Graduate School, Shenzhen, 518055, China · Tsinghua-Peking Joint Center for Life Sciences, School of Life Sciences, Tsinghua University, Beijing, China · Independent Researcher +1

  21. LabAgent: Customize Any Research Hubs for Scientific Discoveries Using AI Agents

    Sep 11, 2026Lei Liu, Yikun Zhang, Jialin Chen +7LLM Agent HarnessesScientific Reproducibility

    Department of Biostatistics, Yale University · Department of Computer Science, Northeastern University · EWSC Center, Broad Institute of MIT and Harvard +5

  22. Overview of the NLPCC 2026 Shared Task 11: Agent-Based Experiment Reproduction from Scientific Papers

    Sep 10, 2026Hanhua Hong, Yizhi Li, Luu Gia Huy +3Process-Based Agent EvaluationAI Agent Evaluation

    The University of Manchester · IQuest Research · Vietnam National University +2

  23. IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications

    Sep 9, 2026Yiling Ma, Yilun Zhao, Sihong Wu +2Benchmark DesignClarification Question Generation

    Yale University · TCS Research

  24. SCIRIGOR:Evaluating Open-Ended Scientific Analysis Beyond Final Scores

    Sep 5, 2026Bowen Liu, Shuo Nie, Bodong Du +1Scientific Agent BenchmarksLLM Agent Evaluation

    HKUST · Alibaba Group

  25. Model Retirement Creates Reproducibility Risk in Biomedical AI Publications

    Sep 4, 2026Nathan Wolfrath, Meghan Conroy, Thomas Kosten +7ML ReproducibilityScientific Reproducibility

    Bud and Sue Selig Hub for Surgical Data Science, Department of Surgery, Medical College of Wisconsin · Division of Biostatistics, Data Science Institute, Medical College of Wisconsin · Department of Computer and Electrical Engineering, Marquette University +1

  26. Can Coding Agents Reproduce Official Statistics? Metadata, Retry Budget and the Limits of Execution Feedback in a Controlled Eurostat Benchmark

    Sep 2, 2026Sabina-Cristiana NeculaData Analysis AgentsAI Agent Benchmarks

    Alexandru Ioan Cuza University of Iași, Faculty of Economics and Business Administration, Iași, Romania

  27. Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific Research

    Aug 27, 2026Lezhi Yu, Xiaogang Xu, Yuhua Zhou +2AI Agent AuditingAI-Assisted Scientific Research

    College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Zhejiang Lab, Hangzhou, China

  28. 6.5% of the Neuro-Symbolic Literature Can Be Reproduced from Its Published Artifacts, a Six-Stage Audit Framework and First Instantiation

    Aug 26, 2026Brandon Colelough, Vladimir Martirosyan, Ishan Tamrakar +6Scientific ReproducibilityML Reproducibility

    Department of Computer Science, University of Maryland College Park, Maryland, USA

  29. Provenance Before Prose: Claim-Locked Reporting for Statistical Text Generation

    Aug 26, 2026Xiao Fan, Jingyuan Li, Hongbin Guo +2Evidence-Grounded GenerationScientific Communication

    Xidian University

  30. SA-Bench: Evaluating Semantic Alignment in LLM-Based Paper Reproduction

    Aug 25, 2026Xue Hu, Zewei Pan, Zeli Su +2ML ReproducibilityScientific Code Generation

    Beihang University · Shanghai Jiao Tong University · Minzu University of China +3