Scientific Reproducibility

Overview

The field asks whether findings can be recovered from reported methods and artifacts, and how measurement, preprocessing, statistical analysis, software, and evaluation choices affect that recovery. Common approaches include sharing versioned data, code, and environments; preregistration and provenance records; standardized workflows and checklists; independent reruns; and benchmarks that test coding or language-model agents end to end. Audits across fields reveal missing artifacts, underspecified procedures, hidden analytic flexibility, and sensitivity to datasets, configurations, or software; agents can reduce reproduction effort but still struggle with tacit assumptions and validating scientific meaning rather than merely producing runnable code. The central challenge is turning transparency into faithful, durable verification: open artifacts and automation help, but neither guarantees comparable results nor establishes that a claim is sound.

Latest papers

CardsList

117 papers8 papers in the last four weeks, down 50% on the four weeks before. 0.1% of all new papers.

  1. Benchmarking Real-World Robotic Manipulation through Stochastic Challenges

    Nov 9, 2026Elias Gläßer, Gülhan Şikaroğlu, Pavlo Kupyn +3Robot Manipulation BenchmarksScientific Reproducibility

  2. Replicability, Reproducibility, and Robustness in Political and Social Science Research with Large Language Models

    Nov 4, 2026Joan C. Timoneda, Sebastián Vallejo VeraScientific ReproducibilityLLM Reliability

  3. Patching the Protocol: A Taxonomy of Pre-Registration Repairs in LLM-Proposer Experiments

    Nov 3, 2026Peter Chen, Ni Zhang, Siyuan ZhangML ReproducibilityScientific Reproducibility

  4. Reimagining computational quantum research with AI scientists

    Nov 2, 2026Yuxuan Du, Hanqing Zhao, Xinbiao Wang +3AI Agents for Scientific DiscoverySelf-Improving Agents

  5. A Scientific Claim Stability Framework for Evaluating MRI Morphometry Pipelines

    Nov 2, 2026Duy-Cat Can, Dang-Duat Tran, Anh-Khang Nguyen +6Scientific ReproducibilityMagnetic Resonance Imaging

  6. RoboSeed: Multidimensional Evaluation of Vision-Language-Action and World-Action Models: Capability, Resource Affordability, and Research Evidence

    Oct 29, 2026Xin Ma, Furong Chen, Xinlong Man +9Robot Foundation ModelsWorld Action Models

  7. Pinpointing Pivot Points: The DECIDE Framework for Data Analysis Decision Importances

    Oct 28, 2026Zachary T. Rewolinski, Jan Simson, Frauke Kreuter +2Sensitivity AnalysisScientific Reproducibility

  8. AI-Driven Reproducibility and Analytical Robustness in Scientific Discovery

    Oct 16, 2026Shicheng Liu, Krista Opsahl-Ong, Andrew Drozdov +6AI-Assisted Scientific ResearchScientific Reproducibility

  9. Author-in-the-loop Agentic Reproducibility Auditing for a Machine Learning Conference

    Oct 15, 2026Yukun Dong, Linjun Zhang, Tian ZhengML ReproducibilityScientific Reproducibility

  10. Kernel-based External Validity of Experimental Studies

    Oct 9, 2026Federico Matteucci, Gerhard NeumannScientific Reproducibility

  11. BioStudyBench: Evaluating Agents on Post-Cutoff Biomedical Studies

    Oct 6, 2026David Li, Shaamil Karim, Christian GensbiglerScientific Agent BenchmarksAI Agent Benchmarks

  12. Kurate: Scalable Scientific Quality Analysis

    Oct 5, 2026Matthew J. Vowels, Jamie CumminsScientific Reproducibility

  13. An Automated and Reproducible Workflow for Crack Identification and Damage Assessment of Fusion Materials

    Oct 2, 2026Rinkle Juneja, Viktor Reshniak, Richard K. Archibald +4Microscopy Image AnalysisMaterials Science

  14. Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated Software

    Sep 30, 2026Bhanu Prakash Vangala, Tanu MalikCoding AgentsCode Generation

  15. Papers Without Code: Availability of GitHub Repositories Linked in *CL Publications

    Sep 28, 2026Selina Meyer, Michael RothScientific Reproducibility

  16. Reconstructing Implicit Scientific Knowledge: Evaluating LLM Agents through End-to-End Reproduction of Astronomy

    Sep 28, 2026Yuehui Wang, Xinyu Qi, Guirong Xue +4Scientific ReproducibilityScientific Reasoning in Language Models

  17. Position: Let's Strengthen Verifiability If We Can't Enforce Reproducibility

    Sep 25, 2026Samet Hicsonmez, Nermin Samet, Renaud MarletML ReproducibilityScientific Reproducibility

  18. Reproducible AI Requires Reproducible Randomness

    Sep 22, 2026Anthony Bertrand, Tom Schmitt, Engelbert Mephu Nguifo +1ML ReproducibilityScientific Reproducibility

  19. Gricea: An Open Science Platform for Conversational AI Research

    Sep 18, 2026Nikhil Sharma, Yunlin Gong, Xinyang Cheng +1Scientific ReproducibilityConversational Agents

  20. PaperDoctor: Evidence-Grounded and Actionable Feedback for Scientific Papers in Progress

    Sep 15, 2026Kevin Qinghong Lin, Siyuan Hu, Pan Lu +14AI-Assisted Scientific ResearchScientific Reproducibility

  21. A Programmable Optics Cloud Laboratory

    Sep 14, 2026Sachin Vaidya, Caio Silva, Seou Choi +2Laboratory AutomationScientific Reproducibility

  22. OpenAI4S: Code as Action, Science as Sessions

    Sep 14, 2026Gongbo Zhang, Hao Li, Yu Wang +15Data ProvenanceRuntime Enforcement for AI Agents

  23. LabAgent: Customize Any Research Hubs for Scientific Discoveries Using AI Agents

    Sep 11, 2026Lei Liu, Yikun Zhang, Jialin Chen +7LLM Agent HarnessesScientific Reproducibility

  24. Overview of the NLPCC 2026 Shared Task 11: Agent-Based Experiment Reproduction from Scientific Papers

    Sep 10, 2026Hanhua Hong, Yizhi Li, Luu Gia Huy +3Process-Based Agent EvaluationAI Agent Evaluation