ML Reproducibility

ML: Machine Learning

Machine learning reproducibility asks whether reported results can be recovered from the data, code, configurations, and execution environments—and whether they hold beyond one run. Common approaches include releasing versioned artifacts and provenance, standardizing benchmark splits and metrics, controlling seeds and numerical computation, and using independent replications with multi-seed uncertainty estimates. These methods are applied across settings such as LLM inference, biomedical prediction, recommender systems, and robotics, where results can shift with hardware, preprocessing, evaluation protocols, or data choice. A central tension is that deterministic reruns and executable code do not guarantee faithful reproduction or robust conclusions, while thorough audits and repeated experiments can be costly.

Latest papers

138 papers23 papers in the last four weeks, up 15% on the four weeks before. 0.1% of all new papers.

The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility
May 19, 2026David Pape, Jonathan Evertz, Lea SchönherrLLM Evaluation LLM Inference
CISPA Helmholtz Center for Information Security, Saarbrücken, Germany
ExECG: An Explainable AI Framework for ECG models
May 19, 2026Jong-Hwan Jang, Yong-yeon JoExplainable Artificial Intelligence Healthcare
Medical AI Co. Ltd., Seoul, Republic of Korea
MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibility
May 15, 2026Sasi Kiran Gaddipati, Diyana Muhammed, Farhana Keya +2AI Agent Benchmarks ML Reproducibility
1TIB—Leibniz Information Centre for Science and Technology, Hannover, Germany · 2L3S Research Center, Leibniz University Hannover, Hannover, Germany
parallelcbf: A composable safety-filter and auditability framework for tensor-parallel reinforcement learning
May 15, 2026Yijun Lu, Zilei Yang, Yuyin MaReinforcement Learning Control Barrier Functions
Croissant Baker: Metadata Generation for Discoverable, Governable, and Reusable ML Datasets
May 14, 2026Rafi Al Attrach, Rajna Fani, Sebastian Lobentanzer +17Dataset Discovery ML Reproducibility
Technical University of Munich · Massachusetts Institute of Technology · Helmholtz Munich +15
Improving Reproducibility in Evaluation through Multi-Level Annotator Modeling
May 13, 2026Deepak Pandita, Flip Korn, Chris Welty +1Inter-Rater Reliability LLM Evaluation
Rochester Institute of Technology · Google Research
No One Knows the State of the Art in Geospatial Foundation Models
May 12, 2026Isaac Corley, Nils Lehmann, Caleb Robinson +6Geospatial Foundation Models ML Reproducibility
Taylor Geospatial · Technical University of Munich · Microsoft AI for Good Research Lab +6
Rollout Cards: A Reproducibility Standard for Agent Research
May 12, 2026Charlie Masters, Ziyuan Liu, Stefano V. AlbrechtAI Agent Evaluation Agent Evaluation
Deepflow · Nanyang Technological University
From Accuracy to Auditability: A Survey of Determinism in Financial AI Systems
May 11, 2026Ruizhe Zhou, Xiaoyang Liu, Gaoyuan Du +4Algorithmic Auditing ML Reproducibility
Amazon.com USA
BioMedArena: An Open-source Toolkit for Building and Evaluating Biomedical Deep Research Agents
May 7, 2026Jinge Wu, Hongjian Zhou, Mingde Zeng +8AI Agents for Scientific Discovery Deep Research Agents
University of Oxford · University College London · Technical University of Munich +1
Ensuring Reliability in Programming Knowledge Tracing: A Re-evaluation of Attention-augmented Models and Experimental Protocols
May 6, 2026Jaewook Kim, Hyeoncheol KimProgramming Education Temporal Data Leakage
Korea University, Seoul, Republic of Korea
RamanBench: A Large-Scale Benchmark for Machine Learning on Raman Spectroscopy
May 3, 2026Mario Koddenbrock, Christoph Lange, Robin Legner +6Benchmark Design Scientific ML
HTW Berlin · TU Berlin · KWS SAAT, Einbeck +4
ECG-biometrics-bench: A Unified Framework for Reproducible Benchmarking of ECG Biometrics
May 2, 2026Milad ParvanBiometric Identification Benchmark Design
Independent Researcher · Milan, Italy
SCARV: Structure-Constrained Aggregation for Stable Sample Ranking in Redundant NLP Datasets
May 1, 2026Xu Zheng, Feiyu Wu, Linhong Wu +2Training Data Selection ML Reproducibility
School of Cyber Engineering, Xidian University
A Reproducibility Analysis of PO4ISR: Diagnosing and Mitigating Semantic Drift in LLM-Based Session Recommendation
Apr 29, 2026Aditya Tiwari, Konduri Naga Lakshmi Rekha, Rajesh Kumar MundotiyaSequential Recommendation LLM Prompting
MATRA Lab Department of Computer Science and Engineering Indian Institute of Technology Bhilai Chhattisgarh, India 491002
Spreadsheet Modeling Experiments Using GPTs on Small Problem Statements and the Wall Task
Apr 28, 2026Thomas A. Grossman, Yuan Chen, Sopiko DatuashviliML Reproducibility Spreadsheet Automation
University of San Francisco 2130 Fulton St. San Francisco, CA 94117 USA
NeuroClaw Technical Report
Apr 27, 2026Cheng Wang, Zhibin He, Zhihao Peng +7Multi-Agent LLM Systems AI Agent Benchmarks
Electronic Engineering Department, The Chinese University of Hong Kong, China · School of Electronic Information, Northwest University, China · Department of Computer Science, Emory University, Atlanta, GA, USA +3
A Tale of Two Variances: When Single-Seed Benchmarks Fail in Bayesian Deep Learning
Apr 25, 2026Qishi Zhan, Minxuan Hu, Liang He +2Bayesian Neural Networks Deep Ensembles
Department of Mathematical and Statistical Sciences Marquette University, USA · Cornell Ann S. Bowers College of Computing and Information Science Cornell University, USA · Physics Science and Engineering Tongji University, China +2
Introducing Background Temperature to Characterise Hidden Randomness in Large Language Models
Apr 24, 2026Alberto Messina, Stefano ScottaLLM Inference ML Reproducibility
1RAI - Radiotelevisione Italiana, Centre for Research, Technological Innovation and Experimentation (CRITS)
Replicable Bandits with UCB based Exploration
Apr 21, 2026Rohan Deb, Udaya Ghai, Karan Singh +1Multi-Armed Bandits Contextual Bandits
University of Illinois Urbana-Champaign · Amazon · Carnegie Mellon University
AblateCell: A Reproduce-then-Ablate Agent for Virtual Cell Repositories
Apr 21, 2026Xue Xia, Chengkai Yao, Mingyu Tsoi +10AI Coding Agents ML Reproducibility
Shanghai Artificial Intelligence Laboratory · The Hong Kong University of Science and Technology (Guangzhou) · University of California San Diego +4
Improving reproducibility by controlling random seed stability in machine learning based estimation via bagging
Apr 20, 2026Nicholas Williams, Alejandro SchulerCross-Fitting Debiased ML
Division of Biostatistics, University of California, Berkeley, California, U.S.A.
bioLeak: Leakage-Aware Modeling and Diagnostics for Machine Learning in R
Apr 13, 2026Selçuk KorkmazData Leakage ML Reproducibility
Department of Biostatistics and Medical Informatics, Faculty of Medicine, Trakya University, 22030 Edirne, Türkiye
Replicable Composition
Apr 12, 2026Kiarash Banihashem, MohammadHossein Bateni, Hossein Esfandiari +2Sample Complexity ML Reproducibility
University of Maryland, College Park, MD, USA · Google Research, New York City, NY, USA · Google Research, London, UK
OpenPRC: A Unified Open-Source Framework for Physics-to-Task Evaluation in Physical Reservoir Computing
Apr 8, 2026Yogesh Phalak, Wen Sin Lor, Apoorva Khairnar +3Reservoir Computing Physics-Based Simulation
Department of Mechanical Engineering, Virginia Tech, Blacksburg, VA, USA · Department of Philosophy, Virginia Tech, Blacksburg, VA, USA · Department of Computer Science, Virginia Tech, Blacksburg, VA, USA
Reproducibility study on how to find Spurious Correlations, Shortcut Learning, Clever Hans or Group-Distributional non-robustness and how to fix them
Apr 6, 2026Ole Delzer, Sidney BenderGroup Distributionally Robust Optimization Spurious Correlation Robustness
Machine Learning Group Technische Universität Berlin
BlazeFL: Fast and Deterministic Federated Learning Simulation
Apr 4, 2026Kitsuya Azuma, Takayuki NishioFederated Learning ML Reproducibility
Institute of Science Tokyo, Tokyo, Japan
An Artifact-based Agent Framework for Adaptive and Reproducible Medical Image Processing
Mar 31, 2026Lianrui Zuo, Yihao Liu, Gaurav Rudravaram +15Agentic Workflows Scientific Workflow Automation
Dept. of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, USA · Dept. of Computer Science, Vanderbilt University, Nashville, TN, USA · Dept. of Surgery, Vanderbilt University Medical Center, Nashville, TN, USA +10
Schrödinger's Seed: Purr-fect Initialization for an Impurr-fect Universe
Mar 31, 2026Mi chen, Renhao YeNeural Network Initialization ML Reproducibility
Kapteyn Astronomical Institute, University of Groningen, Postbus 800, 9700 AV Groningen, The Netherlands · School of Astronomy and Space Science, University of Chinese Academy of Sciences, 1 East Yanqi Lake Rd., Beijing 100049, P.R. China · Shanghai Astronomical Observatory, Chinese Academy of Sciences, 80 Nandan Rd., Shanghai 200030, China
Bit-Identical Medical Deep Learning via Structured Orthogonal Initialization
Mar 30, 2026Yakov Pyotr ShkolnikovML Reproducibility Neural Network Initialization
Independent Researcher, Mountain View, CA, United States

May 19, 2026J/K move · Enter open · S save

The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility

David Pape, Jonathan Evertz, Lea Schönherr

CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

LLM Evaluation LLM Inference ML Reproducibility Benchmark Reproducibility

Open paper AskPDF arXiv

Progress in LLMs is increasingly measured through standardized benchmarks, where state-of-the-art improvements are often separated by fractions of a percentage point. At the same time, the computational cost of evaluating modern LLMs has driven widespread adoption of specialized inference backends, software systems that execute trained models efficiently at inference time. While critical for scalability, system-level optimizations, such as custom CUDA kernels and reduced-precision arithmetic, can alter token probabilities and introduce non-determinism, possibly cascading into divergent generation. In this work, we first survey the inference landscape, identifying 200 distinct engines, and analyze 35,000 ML publications, finding that the specific inference stack is rarely reported despite this widespread diversity. We then present a systematic empirical study of how inference backends affect LLM benchmark results. Holding model weights, decoding parameters, and hardware constant, we evaluate five widely used inference engines, including vLLM, SGLang, and llama

.

cpp, across multiple open-weight models and established benchmarks. We show that the choice of backend alone can shift benchmark scores by up to 16.6 percentage points and induce high rates of output disagreement. By isolating backend optimizations and tracing the execution pipeline, we find this divergence is driven by system-level optimizations like prefix caching and CUDA graphs, custom kernels, and engine-specific defaults in logit processing. Our findings identify the inference backend as a previously unreported but consequential hyperparameter in the evaluation of LLM and advocate standardized reporting of inference stacks to improve the reproducibility and interpretability of benchmark comparisons.