Benchmark Reproducibility

Follow this topic Ask about this topic

Latest papers

27 papers5 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

ReproBridge: Building Reproducibility into ML Research Artifacts
Nov 5, 2026Yubo Wang, Rui Meng, Jaehyun Nam +2ML Reproducibility Benchmark Reproducibility
University of Waterloo · Google
TestBED: An Open-Source Toolkit for Bayesian Experimental Design
Nov 4, 2026Gavin Kerrigan, Marcel Hedman, Sahel Iqbal +1Bayesian Experimental Design Benchmark Reproducibility
Department of Statistics, University of Oxford
InformedXRD: Reproducible Benchmarks and Physics-Informed Evaluation for Powder Diffraction Symmetry Classification
Nov 1, 2026Edward G Friedman, Elizabeth J. Baggett, Abhishek Shetty +5Benchmark Reproducibility Automated Evaluation
School of Computer Science, Carnegie Mellon University · NIST Center for Neutron Research · Dept. of Physics, Boston College +3
ReuScope: Auditing Data Reuse and Evaluation Sensitivity in Capsule Vision 2024
Oct 30, 2026Eunseob Choi, Kyeonghun Kim, Hyuk-Jae Lee +1ML Reproducibility Benchmark Reproducibility
Gwangju Institute of Science and Technology · OUTTA · Seoul National University
Re-executing Coding-Agent Evaluations: Unchanged Decisions and Unresolved Policy Risk
Oct 10, 2026Tianyu Ding, Yaocheng Tong, Muzhi Yang +3Benchmark Reproducibility Automated Agent Evaluation
Independent Researchers
Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing
Oct 9, 2026Oktie Hassanzadeh, Dharmashankar SubramanianBenchmark Reproducibility LLM Evaluation
IBM Yorktown Heights, NY, USA
Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing
Oct 7, 2026Arnold Olympio, Juan Manuel Servera Bondroit, Wael Abdelmalek +2Benchmark Reproducibility LLM Inference
Lucerne University of Applied Sciences and Arts, Switzerland · Microsoft, Switzerland · Uthereal AG, Switzerland +1
Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluation
Sep 29, 2026Mario Sanz-Guerrero, Minh Duc Bui, Manuel Mager +1LLM Evaluation Language Model Generation Evaluation
Johannes Gutenberg University Mainz, Germany · Universidad Iberoamericana, Mexico · University of Colorado Boulder, USA
AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines
Sep 16, 2026Li ChenGPU Kernel Optimization Benchmark Design
Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints
Sep 3, 2026Haoyaun Zhu, Jie ZhangLLM Evaluation LLM-as-a-Judge
School of Electronic and Electrical Engineering, University of Sheffield, Sheffield S10 2TN, UK · R&D Department, Ranplan Wireless Network Design Ltd., Cambridge CB23 3UY, UK · R&D Department, Cambridge AI+ Ltd., Cambridge CB23 3UY, UK
SA-Bench: Evaluating Semantic Alignment in LLM-Based Paper Reproduction
Aug 25, 2026Xue Hu, Zewei Pan, Zeli Su +2ML Reproducibility Scientific Code Generation
Beihang University · Shanghai Jiao Tong University · Minzu University of China +3
Noise Floor Audit for Agent Benchmarks
Aug 23, 2026Yihang Chen, Pin Qian, Su Wang +4Benchmark Reproducibility LLM Agent Reliability
Reproducible Multimodal Affordance Prediction
Aug 18, 2026Tommaso Apicella, Alessio Xompero, Andrea CavallaroBenchmark Reproducibility Multimodal Model Evaluation
Istituto Italiano di Tecnologia, Italy · Independent Researcher · EPFL, Switzerland
Forensic Reproducibility Audit of a Radiology Vision-Language Model Benchmark: From Intended Protocol to Released Artifact
Jul 28, 2026Mateusz KozłowskiVLM Evaluation Radiology VLMs
Independent Researcher, Kraków, Poland
Metrics or Mirage? An Audit of Evaluation Inconsistencies in Colonoscopy Polyp Segmentation Benchmarks
Jul 9, 2026Aisha Urooj, Zain Ul Abdien, Neelu MadanPolyp Segmentation Benchmark Reproducibility
Lunit, South Korea · RPTU Kaiserslautern, Germany · Aalborg University & Pioneer Center, Denmark
Necessary but Not Sufficient: Temperature Control and Reproducibility in LLM-as-Judge Safety Evaluations
Jun 24, 2026Hiroki TambaLLM Evaluation LLM-as-a-Judge
Tamba Research Academy
ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues
Jun 16, 2026Shanda Li, Qiuhong Anna Wei, Jingwu Tang +5LLM Agent Evaluation AI Agent Benchmarks
School of Computer Science, Carnegie Mellon University · Datadog
UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities
May 27, 2026Pengyu Zhu, Lijun Li, Yaxing Lyu +8LLM Agent Evaluation AI Agent Benchmarks
1Beijing University of Posts and Telecommunications · 2Shanghai Artificial Intelligence Laboratory · 4North China Electric Power University +1
SafetyRepro: Configuration-Conditional Rank Instability on Alignment Benchmarks
May 25, 2026Yanhang Li, Zhichao Fan, Zexin ZhuangPairwise Comparison Pairwise Preference Evaluation
Northeastern University, Boston, MA, USA · University of Illinois Urbana-Champaign, Urbana, IL, USA · Southern Methodist University, Dallas, TX, USA
What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema
May 20, 2026Mahdi Naser Moghadasi, Faezeh GhaderiBenchmark Auditing LLM Auditing
BrightMind AI, Texas Tech University · Research Division, BrightMind AI, Seattle, WA · Texas Tech University, Lubbock, TX +2
The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility
May 19, 2026David Pape, Jonathan Evertz, Lea SchönherrLLM Evaluation LLM Inference
CISPA Helmholtz Center for Information Security, Saarbrücken, Germany
ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Review
May 4, 2026Kevin Riehl, Andres L. Marin, Nikofors Zacharof +6AI-Assisted Scientific Research Scientific Reproducibility
European Commission, Joint Research Centre · Ideas Forward
Benchmark of Benchmarks: Unpacking Influence and Code Repository Quality in LLM Safety Benchmarks
Mar 3, 2026Junjie Chu, Xinyue Shen, Ye Leng +3Benchmark Reproducibility LLM Safety Benchmarks
CISPA Helmholtz Center for Information Security · University of Waterloo · Flexera
Medical Imaging AI Competitions Lack Fairness
Dec 19, 2025Annika Reinke, Evangelia Christodoulou, Sthuthi Sadananda +34Medical Image Benchmarks Dataset Bias
German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Germany · DKFZ Heidelberg, Helmholtz Imaging, Germany · Engineering Faculty, Heidelberg University, Germany +32
Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility
Jan 18, 2025Jialun Cao, Yuk-Kit Chan, Zixuan Ling +12Benchmark Construction Benchmark Design
The Hong Kong University of Science and Technology · Guangzhou HKUST Fok Ying Tung Research Institute · The Chinese University of Hong Kong +6
Real Faults in Deep Learning Fault Benchmarks: How Real Are They?
Dec 20, 2024Gunel Jahangirova, Nargiz Humbatova, Jinhan Kim +2Software Engineering Benchmarks Benchmark Reproducibility
Department of Informatics, King’s College London, London, UK · Software Institute, Università della Svizzera italiana (USI), Lugano, Switzerland · School of Computing, KAIST, Daejeon, Republic of Korea
CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark
Sep 17, 2024Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir +2AI Agent Evaluation AI Agent Benchmarks
Princeton University

Nov 5, 2026J/K move · Enter open · S save

ReproBridge: Building Reproducibility into ML Research Artifacts

Yubo Wang, Rui Meng, Jaehyun Nam, Tomas Pfister+1

University of Waterloo · Google

ML Reproducibility Benchmark Reproducibility

Open paper AskPDF arXiv

Reproducing machine learning experiments requires reconstructing procedures from information scattered across papers, released code, and supporting artifacts. For coding agents, reproduction success can therefore depend on their ability to fill in missing details, rather than on the released artifacts alone. We ask whether preserving recovered procedures and decisions improves subsequent independent reproduction. We define a standardized artifact design comprising a refined codebase, environment setup, experiment launchers, and a reproduction guide, and introduce ReproBridge, an agent workflow for constructing these refined reproducible artifacts. ReproBridge-Bench provides 1,029 tasks from 387 experiments across 36 papers, anchored by five human-verified reference scores per task. Agents attempt tasks in fresh sessions with raw code or refined artifacts; a separate Auditor checks execution evidence. Across four evaluated models, refined artifacts raise mean reproduction success from 76.4% to 92.6% and reduce downstream API spending by 19.7% on average. The cross-model success-rate spread narrows from 21.6 to 3.9 percentage points, with all four models achieving mean success rates above 90%. Removing any component reduces success in the ablation study, with the largest losses from code and environment removal. These results suggest that preserving execution knowledge helps even models with lower raw-code success reproduce experiments more reliably and at lower API cost.