Scientific Reproducibility

Follow this topic Ask about this topic

Momentum

11 papers in the last four weeks, up 175% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 76

All topics

VERITAS: Towards a General-Purpose Replication Tool for Scientific Research
Jul 3, 2026Haokun Liu, Filbert Aurelian Tjiaranata, Chenhao TanAI Coding Agents Scientific Reproducibility
University of Indonesia
Coding-agents can replicate scientific machine learning papers
Jul 2, 2026Atharva Hans, Ilias BilionisCoding Agents LLM Agent Evaluation
School of Mechanical Engineering, Purdue University, West Lafayette, IN, 47907, USA
The Agentic Garden of Forking Paths
Jul 1, 2026Jiacheng Miao, Jonathan K Pritchard, James ZouAI Agents for Scientific Discovery Scientific Reproducibility
Department of Biomedical Data Science, Stanford University · Department of Genetics, Stanford University · Department of Biology, Stanford University +2
Mitigating LLM-based p-Hacking by Preregistering for the Next LLM
Jun 26, 2026Maria Thomas, Kristina Gligoric, Nihar B. ShahScientific Reproducibility
Johns Hopkins University · Carnegie Mellon University
Computational Analysis of Heart Rate Variability in Healthy Adults
Jun 25, 2026María J. Lado, Arturo J. Méndez, Leandro Rodriguez-Liñares +4Electrocardiography Scientific Reproducibility
Escola Superior de Enxeñaría Informática, IFCAE - Universidade de Vigo, Campus As Lagoas, Ourense, 32004, Spain. · Universidad Politécnica de Madrid, Escuela Tecn Super Ingn Telecomunicac, Informat Proc & Telecommun Ctr, Speech Technol & Machine Learning Grp, Madrid, 28040, Spain
AI-Assisted Computational Reproducibility on the FABRIC Testbed
Jun 24, 2026Komal Thareja, Paul Ruth, Berent Aldikacti +1AI Coding Agents AI-Assisted Scientific Research
RENCI, University of North Carolina at Chapel Hill, NC, USA · University of Massachusetts Amherst, MA, USA
Life After Benchmark Saturation: A Case Study of CORE-Bench
Jun 23, 2026Nitya Nadgir, Sayash Kapoor, Kangheng Liu +11OOD Generalization AI Agent Evaluation
1Independent · 2Princeton University · 3UC Berkeley +1
PixJail: Self-Evolving Paper-to-Pipeline Reproduction for Text-to-Image Jailbreak Evaluation
Jun 23, 2026Leyi Sheng, Han Sun, Zhen Sun +4AI Safety Evaluation Multimodal Jailbreak Attacks
1The Hong Kong University of Technology and Science (Guangzhou) · 2East China Normal University · 3Shanghai Qi Zhi Institute +4
Making AI Scientists Auditable from Evidence to Claim
Jun 17, 2026Zijian Wang, Hanqi Li, Ziyue Yang +17Data Provenance Scientific Reproducibility
X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Suzhou Laboratory, Suzhou, China · Shanghai Innovation Institution, Shanghai, China +1
ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues
Jun 16, 2026Shanda Li, Qiuhong Anna Wei, Jingwu Tang +5LLM Agent Evaluation AI Agent Benchmarks
School of Computer Science, Carnegie Mellon University · Datadog
The Shift Toward Open and Reproducible AI Research
Jun 15, 2026Kevin L Coakley, Thijs Snelleman, Holger Hoos +1ML Reproducibility Scientific Reproducibility
Department of Computer Science, Norwegian University of Science and Technology, Sem Sælands Vei 7-9, Trondheim, 7031, Trønderlag, Norway. · San Diego Supercomputer Center, University of California San Diego, 9500 Gilman Dr, La Jolla, 92093, California, USA. · Chair of AI Methodology, RWTH Aachen University, Theaterstrasse 35-39, Aachen, 52062, North Rhine-Westphalia, Germany. +1
Agentic Publication Protocol: An Attempt to Modernize Scientific Publication
Jun 15, 2026Sirui Lu, Xiao-Liang QiAI-Assisted Scientific Research Scientific Communication
1Max-Planck-Institut für Quantenoptik, Garching, Germany · 2Munich Center for Quantum Science and Technology (MCQST), Schellingstrasse 4, D-80799 München, Germany · 3Leinweber Institute for Theoretical Physics, Stanford University, Stanford, CA, USA
Automated reproducibility assessments in the social and behavioral sciences using large language models
Jun 11, 2026Tobias Holtdirk, Pietro Marcolongo, Anna Steinberg Schulten +7AI-Assisted Scientific Research Scientific Reproducibility
1LMU Munich, Munich, Germany · 2Munich Center for Machine Learning, Munich, Germany · University of Maryland, College Park, USA +1
AI Coding Agents Can Reproduce Social Science Findings
Jun 9, 2026Meysam Alizadeh, Mohsen Mosleh, Fabrizio Gilardi +2AI Coding Agents AI Agent Benchmarks
University of Oxford · University of Zurich · Carnegie Mellon University +1
Deterministic Integrity Gates for LLM-Assisted Clinical Manuscript Preparation: An Auditable Biomedical Informatics Architecture
Jun 8, 2026Yoojin Nam, Jinhoon Jeong, Namkug KimLLM Answer Verification LLM Auditing
Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Republic of Korea · Department of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Republic of Korea · Aperivue, Incheon, Republic of Korea +2
Traxia: A Framework for Verifiable, Agent-Native Scientific Publishing
Jun 6, 2026Wisdom DogahData Provenance Scholarly Peer Review
Faculty of Computing and Mathematical Sciences, University of Mines and Technology (UMaT), Tarkwa, Ghana. · BlackMatrix AI Research, Accra, Ghana.
Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generation
Jun 6, 2026Katelyn Xiaoying Mei, Yi-Li Hsu, Minjoon Choi +5Human-in-the-Loop Evaluation Scientific Reproducibility
University of Washington · National Tsing Hua University · Seoul National University +2
Cluster Analysis with Resampling for Validation and Exploration (CARVE)
May 29, 2026Kai R. Wycik, Tiffany M. Tang, Tarek M. Zikry +1Clustering Scientific Reproducibility
Department of Statistics, Columbia University, New York, NY, USA · Center for Theoretical Neuroscience, Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY, USA · Department of Applied and Computational Mathematics and Statistics, University of Notre Dame, Notre Dame, IN, USA +2
ERICA: Quantifying Replicability of Cluster Analysis
May 29, 2026Siamak K. Sorooshyari, Manuel A. Rivas, Robert TibshiraniClustering ML Reproducibility
Department of Statistics Stanford University Stanford, CA 94305, USA · Department of Biomedical Data Science Stanford University Stanford, CA 94305, USA · Department of Statistics and Biomedical Data Science Stanford University Stanford, CA 94305, USA
Replicable Simulation-Based Robot Validation through Provenance
May 28, 2026Argentina Ortega, Samuel Wiest, Frederik Pasch +1Data Provenance Robotics Simulation
University of Bremen, Germany. · Karlsruhe University of Applied Sciences, Germany.
ResearchLoop: An Evidence-Gated Control Plane for AI-Assisted Research
May 27, 2026Yihan Xia, Taotao WangAI-Assisted Scientific Research Scientific Reproducibility
Shenzhen University, China
UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities
May 27, 2026Pengyu Zhu, Lijun Li, Yaxing Lyu +8LLM Agent Evaluation AI Agent Benchmarks
1Beijing University of Posts and Telecommunications · 2Shanghai Artificial Intelligence Laboratory · 4North China Electric Power University +1
ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidence
May 25, 2026Rui Meng, Bhavana Dalvi Mishra, Jiefeng Chen +10Data Provenance Citation Verification
Google Cloud AI Research
SafetyRepro: Configuration-Conditional Rank Instability on Alignment Benchmarks
May 25, 2026Yanhang Li, Zhichao Fan, Zexin ZhuangPairwise Comparison Pairwise Preference Evaluation
Northeastern University, Boston, MA, USA · University of Illinois Urbana-Champaign, Urbana, IL, USA · Southern Methodist University, Dallas, TX, USA
Position: AI for Science Should Treat Measurement-to-Dataset Pipelines as Inference Components
May 23, 2026Ling Zhan, Xiaoyao Yu, Tao JiaAI for Science Uncertainty Quantification
College of Computer and Information Science, Southwest University, Chongqing, China · Chongqing Key Laboratory of Brain-Inspired Cognitive Computing and Educational Rehabilitation for Children with Special Needs, Chongqing Normal University, Chongqing, China · College of Computer and Information Science, Chongqing Normal University, Chongqing, China
What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema
May 20, 2026Mahdi Naser Moghadasi, Faezeh GhaderiBenchmark Auditing LLM Auditing
BrightMind AI, Texas Tech University · Research Division, BrightMind AI, Seattle, WA · Texas Tech University, Lubbock, TX +2
Collider-Bench: Benchmarking AI Agents with Particle Physics Analysis Reproduction
May 13, 2026Darius A. Faroughy, Sofia Palacios Schweitzer, Ian Pang +2AI Agents for Scientific Discovery AI Agent Evaluation
New High Energy Theory Center Department of Physics & Astronomy Rutgers University · Faculty of Computing & Data Sciences Boston University
It's not the Language Model, it's the Tool: Deterministic Mediation for Scientific Workflows
May 13, 2026Marios Adamidis, Danae Katrisioti, Yannis Tzitzikas +1LLM Tool Use Scientific Reproducibility
Department of Materials Science and Technology, University of Crete, Heraklion, Greece · Institute of Electronic Structure and Laser, FORTH, Heraklion, Greece · Computer Science Department, University of Crete, Heraklion, Greece +2
Decomposing the Generalization Gap in PROTAC Activity Prediction: Variance Attribution and the Inter-Laboratory Ceiling
May 12, 2026Thor Klamt, Wolfgang Nejdl, Ming TangMolecular Property Prediction Scientific Reproducibility
L3S Research Center, Leibniz Universität Hannover, Appelstraße 9a, 30167 Hannover, Germany · Institute of Data Science (Knowledge-Based Systems), Faculty of Electrical Engineering and Computer Science,2026 Leibniz Universität Hannover, Appelstraße 9a, 30167 Hannover, Germany
Quantifying the Reconstructability of Astrophysical Methods with Large Language Models and Information Theory: A Case Study in Spectral Reconstruction
May 11, 2026Hsing Wen Lin, Zong-Fu SieLLM Auditing Scientific Reproducibility
Department of Physics, University of Michigan, Ann Arbor, MI 48109, USA · Michigan Institute for Data and AI in Society, University of Michigan, Ann Arbor, MI 48109, USA · Independent Researcher, Taiwan

Jul 3, 2026, cs.AIJ/K move · Enter open · S save

VERITAS: Towards a General-Purpose Replication Tool for Scientific Research

Haokun Liu, Filbert Aurelian Tjiaranata, Chenhao Tan

University of Indonesia

AI Coding Agents Scientific Reproducibility Scientific Claim Verification

Open paper AskPDF arXiv

AI tools are accelerating scientific publication while the systems that review it struggle to keep up, and independent verification of published research has become both harder and more important. As manual replication is slow and expensive, a growing line of work uses coding agents to automate parts of the process. Existing efforts are largely packaged as benchmarks with companion agents that only run inside the benchmark's own pipeline, and no general-purpose replication tool exists. We present VERITAS, a domain-agnostic replication framework built around CLI coding agents. Given a paper, a code repository, or both, VERITAS extracts the paper's claims, runs the methodology while resolving issues as they arise, and judges each claim against the evidence from experiment runs. The pipeline returns an importance-weighted Replication Score, a severity-rated log of every fix applied, and the patched codebase. We evaluate VERITAS on CORE-Bench and ReplicationBench, 65 papers spanning computer science, social science, medicine, and astrophysics. Against two strong Claude Code baselines on the same model and host environment, VERITAS achieves state-of-the-art performance and leads on every metric on both benchmarks.

Momentum

Related topics

Latest papers 76