Scientific Reproducibility

Ask about this topic

Overview

The field asks whether findings can be recovered from reported methods and artifacts, and how measurement, preprocessing, statistical analysis, software, and evaluation choices affect that recovery. Common approaches include sharing versioned data, code, and environments; preregistration and provenance records; standardized workflows and checklists; independent reruns; and benchmarks that test coding or language-model agents end to end. Audits across fields reveal missing artifacts, underspecified procedures, hidden analytic flexibility, and sensitivity to datasets, configurations, or software; agents can reduce reproduction effort but still struggle with tacit assumptions and validating scientific meaning rather than merely producing runnable code. The central challenge is turning transparency into faithful, durable verification: open artifacts and automation help, but neither guarantees comparable results nor establishes that a claim is sound.

Latest papers

CardsList

108 papers13 papers in the last four weeks, level with the four weeks before. 0.1% of all new papers.

  1. IoT-Zoo: A Container-Based Framework for Heterogeneous IoT Device Profiles and Reproducible Traffic Capture

    Apr 7, 2026Vagner E. Quincozes, Diego Kreutz, Silvio E. QuincozesInternet of ThingsScientific Reproducibility

    Federal Fluminense University (UFF) · AI Horizon Labs and PPGES – Federal University of Pampa (UNIPAMPA)

  2. RESCORE: LLM-Driven Simulation Recovery in Control Systems Research Papers

    Apr 6, 2026Vineet Bhat, Shiqing Wei, Ali Umut Kaypak +3Scientific Code GenerationScientific Reproducibility

    ECE Dept, Tandon School of Engineering, New York University, Brooklyn, NY 11201, USA

  3. PRBench: End-to-end Paper Reproduction in Physics Research

    Mar 29, 2026Shi Qiu, Junyi Deng, Yiwei Deng +48AI Agent BenchmarksAI Coding Agents

  4. How Open Must Language Models be to Enable Reliable Scientific Inference?

    Mar 27, 2026James A. Michaelov, Catherine Arnett, Tyler A. Chang +7AI-Assisted Scientific ResearchLLM Reliability

    Massachusetts Institute of Technology · EleutherAI · University of California San Diego +2

  5. Do Papers Tell the Whole Story? A Benchmark and Framework for Uncovering Hidden Implementation Gaps in Bioinformatics

    Mar 23, 2026Tianxiang Xu, Xiaoyan Zhu, Xin Lai +4Software Engineering BenchmarksScientific Reproducibility

    Xi’an Jiaotong University

  6. Code Sharing In Prediction Model Research: A Scoping Review

    Mar 16, 2026Thomas Sounack, Raffaele Giancotti, Catherine A. Gao +8Scientific ReproducibilityML Reproducibility

    Dana-Farber Cancer Institute · University of Calabria · Northwestern Feinberg School of Medicine +9

  7. Supporting Workflow Reproducibility by Linking Bioinformatics Tools across Papers and Executable Code

    Mar 9, 2026Clémence Sebe, Olivier Ferret, Aurélie Névéol +3Entity ResolutionScientific Reproducibility

    Universit´e Paris-Saclay, CNRS, Laboratoire Interdisciplinaire des Sciences du Num´erique, 91400, Orsay, France · Universit´e Paris-Saclay, CEA, List, F-91120, Palaiseau, France · Department of Computer Science, Humboldt-Universität zu Berlin, 10099, Berlin, Germany +1

  8. Supporting Artifact Evaluation with LLMs: A Study with Published Security Research Papers

    Mar 6, 2026David Heye, Karl Kindermann, Robin Decker +5Scientific ReproducibilityAutomated Evaluation

    Communication and Distributed Systems, RWTH Aachen University, Germany · Data Stream Management and Analysis, RWTH Aachen University, Germany

  9. Benchmark of Benchmarks: Unpacking Influence and Code Repository Quality in LLM Safety Benchmarks

    Mar 3, 2026Junjie Chu, Xinyue Shen, Ye Leng +3Benchmark ReproducibilityLLM Safety Benchmarks

    CISPA Helmholtz Center for Information Security · University of Waterloo · Flexera

  10. Bridging the Reproducibility Divide: Open Source Software's Role in Standardizing Healthcare AI

    Mar 2, 2026John Wu, Zhenbang Wu, Jimeng SunOpen-Source Software DevelopmentML Reproducibility

    University of Illinois, Urbana-Champaign, USA

  11. What Papers Don't Tell You: Recovering Tacit Knowledge for Automated Paper Reproduction

    Mar 2, 2026Lehui Li, Ruining Wang, Haochen Song +8Scientific Code GenerationScientific Reproducibility

    School of Software, Shandong University China · Beijing Institute of Technology China · Zhejiang University China +4

  12. Agentic Scientific Simulation: Execution-Grounded Model Construction and Reconstruction

    Feb 27, 2026Knut-Andreas Lie, Olav Møyner, Elling Svee +1Physics-Based SimulationScientific Code Generation

    SINTEF Digital, Oslo, Norway · Department of Mathematical Sciences, NTNU, Trondheim, Norway

  13. Criteria-first, semantics-later: reproducible structure discovery in image-based sciences

    Feb 17, 2026Jan BumbergerScientific ReproducibilityAI for Science

    Helmholtz Centre for Environmental Research – UFZ, Research Data Management - RDM, Permoserstraße 15, Leipzig, 04318, Germany · Helmholtz Centre for Environmental Research – UFZ, Department Monitoring and Exploration Technologies, Permoserstraße 15, Leipzig, 04318, Germany · German Centre for Integrative Biodiversity Research (iDiv) Halle-Jena-Leipzig, Puschstraße 4, Leipzig, 04103, Germany

  14. ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences

    Feb 11, 2026Bang Nguyen, Dominik Soós, Qian Ma +8LLM Agent EvaluationAI Agent Benchmarks

    University of Notre Dame · Computer Science and Engineering · Notre Dame, IN, United States +8

  15. RegCheck: A tool for structured comparisons between study registrations and papers

    Jan 19, 2026Jamie Cummins, Beth Clarke, Ian Hussey +1Scientific Reproducibility

    Institute of Psychology, University of Bern · Bennett Institute of Applied Data Science, University of Oxford

  16. Standardization of Post-Publication Code Verification is Possible with the Support of the Community

    Jan 12, 2026Susana Lopez-Moreno, Sangil Kim, Eric Dolores-CuencaScientific ReproducibilityML Reproducibility

    Department of Mathematics, Pusan National University, Busan, South Korea · Industrial Mathematics Center, Pusan National University, Busan, South Korea · Humanoid Olfactory Display Center, Pusan National University, Yangsan, Gyeongsangnam-do, South Korea +2

  17. Can synthetic data reproduce real-world findings in epidemiology? A replication study using adversarial random forests

    Aug 19, 2025Jan Kapar, Kathrin Günther, Lori Ann Vallis +27Synthetic Tabular Data GenerationSynthetic Data Evaluation

    Leibniz Institute for Prevention Research and Epidemiology—BIPS, Bremen, Germany · Faculty of Mathematics and Computer Science, University of Bremen, Bremen, Germany · Department of Human Health and Nutritional Sciences, University of Guelph, Guelph, Ontario, Canada +23

  18. CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark

    Sep 17, 2024Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir +2AI Agent EvaluationAI Agent Benchmarks

    Princeton University