ML Reproducibility

ML: Machine Learning

Follow this topic Ask about this topic

Momentum

14 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 88

All topics

The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility
May 19, 2026David Pape, Jonathan Evertz, Lea SchönherrLLM Evaluation LLM Inference
CISPA Helmholtz Center for Information Security, Saarbrücken, Germany
ExECG: An Explainable AI Framework for ECG models
May 19, 2026Jong-Hwan Jang, Yong-yeon JoExplainable Artificial Intelligence Healthcare
Medical AI Co. Ltd., Seoul, Republic of Korea
MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibility
May 15, 2026Sasi Kiran Gaddipati, Diyana Muhammed, Farhana Keya +2AI Agent Benchmarks ML Reproducibility
1TIB—Leibniz Information Centre for Science and Technology, Hannover, Germany · 2L3S Research Center, Leibniz University Hannover, Hannover, Germany
parallelcbf: A composable safety-filter and auditability framework for tensor-parallel reinforcement learning
May 15, 2026Yijun Lu, Zilei Yang, Yuyin MaReinforcement Learning Control Barrier Functions
Croissant Baker: Metadata Generation for Discoverable, Governable, and Reusable ML Datasets
May 14, 2026Rafi Al Attrach, Rajna Fani, Sebastian Lobentanzer +17ML Reproducibility
Technical University of Munich · Massachusetts Institute of Technology · Helmholtz Munich +15
Improving Reproducibility in Evaluation through Multi-Level Annotator Modeling
May 13, 2026Deepak Pandita, Flip Korn, Chris Welty +1Inter-Rater Reliability LLM Evaluation
Rochester Institute of Technology · Google Research
Rollout Cards: A Reproducibility Standard for Agent Research
May 12, 2026Charlie Masters, Ziyuan Liu, Stefano V. AlbrechtAI Agent Evaluation Agent Evaluation
Deepflow · Nanyang Technological University
From Accuracy to Auditability: A Survey of Determinism in Financial AI Systems
May 11, 2026Ruizhe Zhou, Xiaoyang Liu, Gaoyuan Du +4Algorithmic Auditing ML Reproducibility
Amazon.com USA
BioMedArena: An Open-source Toolkit for Building and Evaluating Biomedical Deep Research Agents
May 7, 2026Jinge Wu, Hongjian Zhou, Mingde Zeng +8AI Agents for Scientific Discovery Deep Research Agents
University of Oxford · University College London · Technical University of Munich +1
Ensuring Reliability in Programming Knowledge Tracing: A Re-evaluation of Attention-augmented Models and Experimental Protocols
May 6, 2026Jaewook Kim, Hyeoncheol KimProgramming Education Temporal Data Leakage
Korea University, Seoul, Republic of Korea
RamanBench: A Large-Scale Benchmark for Machine Learning on Raman Spectroscopy
May 3, 2026Mario Koddenbrock, Christoph Lange, Robin Legner +6Benchmark Design Scientific ML
HTW Berlin · TU Berlin · KWS SAAT, Einbeck +5
ECG-biometrics-bench: A Unified Framework for Reproducible Benchmarking of ECG Biometrics
May 2, 2026Milad ParvanBiometric Identification Benchmark Design
Independent Researcher · Milan, Italy
SCARV: Structure-Constrained Aggregation for Stable Sample Ranking in Redundant NLP Datasets
May 1, 2026Xu Zheng, Feiyu Wu, Linhong Wu +2Training Data Selection ML Reproducibility
School of Cyber Engineering, Xidian University
A Reproducibility Analysis of PO4ISR: Diagnosing and Mitigating Semantic Drift in LLM-Based Session Recommendation
Apr 29, 2026Aditya Tiwari, Konduri Naga Lakshmi Rekha, Rajesh Kumar MundotiyaSequential Recommendation LLM Prompting
MATRA Lab Department of Computer Science and Engineering Indian Institute of Technology Bhilai Chhattisgarh, India 491002
Spreadsheet Modeling Experiments Using GPTs on Small Problem Statements and the Wall Task
Apr 28, 2026Thomas A. Grossman, Yuan Chen, Sopiko DatuashviliML Reproducibility Spreadsheet Automation
University of San Francisco 2130 Fulton St. San Francisco, CA 94117 USA
NeuroClaw Technical Report
Apr 27, 2026Cheng Wang, Zhibin He, Zhihao Peng +7Multi-Agent LLM Systems AI Agent Benchmarks
Electronic Engineering Department, The Chinese University of Hong Kong, China · School of Electronic Information, Northwest University, China · Department of Computer Science, Emory University, Atlanta, GA, USA +3
A Tale of Two Variances: When Single-Seed Benchmarks Fail in Bayesian Deep Learning
Apr 25, 2026Qishi Zhan, Minxuan Hu, Liang He +2Bayesian Neural Networks Deep Ensembles
Department of Mathematical and Statistical Sciences Marquette University, USA · Cornell Ann S. Bowers College of Computing and Information Science Cornell University, USA · Physics Science and Engineering Tongji University, China +2
Introducing Background Temperature to Characterise Hidden Randomness in Large Language Models
Apr 24, 2026Alberto Messina, Stefano ScottaLLM Inference ML Reproducibility
1RAI - Radiotelevisione Italiana, Centre for Research, Technological Innovation and Experimentation (CRITS)
Replicable Bandits with UCB based Exploration
Apr 21, 2026Rohan Deb, Udaya Ghai, Karan Singh +1Multi-Armed Bandits Contextual Bandits
University of Illinois Urbana-Champaign · Amazon · Carnegie Mellon University
AblateCell: A Reproduce-then-Ablate Agent for Virtual Cell Repositories
Apr 21, 2026Xue Xia, Chengkai Yao, Mingyu Tsoi +10AI Coding Agents ML Reproducibility
Shanghai Artificial Intelligence Laboratory · The Hong Kong University of Science and Technology (Guangzhou) · University of California San Diego +4
Improving reproducibility by controlling random seed stability in machine learning based estimation via bagging
Apr 20, 2026Nicholas Williams, Alejandro SchulerDebiased ML Ensemble Learning
Division of Biostatistics, University of California, Berkeley, California, U.S.A.
A Random Matrix Theory Perspective on the Consistency of Diffusion Models
Feb 2, 2026Binxu Wang, Jacob Zavatone-Veth, Cengiz PehlevanDiffusion Model Sampling ML Reproducibility
Kempner Institute, Harvard University, Boston, MA, USA · Center for Brain Science, Harvard University, Cambridge, MA, USA · Society of Fellows, Harvard University, Cambridge, MA, USA +1
Replayable Financial Agents: A Determinism-Faithfulness Assurance Harness for Tool-Using LLM Agents
Jan 17, 2026Raffi KhatchadourianAI Agent Reliability LLM Agent Evaluation
IBM Financial Services Market, New York, NY, USA
EvoXplain: When Machine Learning Models Agree on Predictions but Disagree on Why -- Measuring Mechanistic Multiplicity Across Training Runs
Dec 23, 2025Chama BensmailCancer Genomics Model Selection
University of Hertfordshire
Glucose-ML: A collection of longitudinal diabetes datasets for development of robust AI solutions
Jul 18, 2025Temiloluwa Prioleau, Baiying Lu, Yanjun CuiForecasting Benchmarks Blood Glucose Forecasting
Department of Computer Science Emory University Atlanta, GA 30322 · Department of Computer Science Dartmouth College Hanover, NH 03755
Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility
Jan 18, 2025Jialun Cao, Yuk-Kit Chan, Zixuan Ling +12Benchmark Construction Benchmark Design
The Hong Kong University of Science and Technology · Guangzhou HKUST Fok Ying Tung Research Institute · The Chinese University of Hong Kong +6
Replicability is Asymptotically Free in Multi-armed Bandits
Feb 12, 2024Junpei Komiyama, Shinji Ito, Yuichi Yoshida +1Multi-Armed Bandits ML Reproducibility
New York University · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · RIKEN AIP +2
No One Knows the State of the Art in Geospatial Foundation Models
Date pendingIsaac Corley, Nils Lehmann, Caleb Robinson +6Geospatial Foundation Models ML Reproducibility
Taylor Geospatial · Technical University of Munich · Microsoft AI for Good Research Lab +6

May 19, 2026, cs.LGJ/K move · Enter open · S save

The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility

David Pape, Jonathan Evertz, Lea Schönherr

CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

LLM Evaluation LLM Inference ML Reproducibility

Open paper AskPDF arXiv

Progress in LLMs is increasingly measured through standardized benchmarks, where state-of-the-art improvements are often separated by fractions of a percentage point. At the same time, the computational cost of evaluating modern LLMs has driven widespread adoption of specialized inference backends, software systems that execute trained models efficiently at inference time. While critical for scalability, system-level optimizations, such as custom CUDA kernels and reduced-precision arithmetic, can alter token probabilities and introduce non-determinism, possibly cascading into divergent generation. In this work, we first survey the inference landscape, identifying 200 distinct engines, and analyze 35,000 ML publications, finding that the specific inference stack is rarely reported despite this widespread diversity. We then present a systematic empirical study of how inference backends affect LLM benchmark results. Holding model weights, decoding parameters, and hardware constant, we evaluate five widely used inference engines, including vLLM, SGLang, and llama

.

cpp, across multiple open-weight models and established benchmarks. We show that the choice of backend alone can shift benchmark scores by up to 16.6 percentage points and induce high rates of output disagreement. By isolating backend optimizations and tracing the execution pipeline, we find this divergence is driven by system-level optimizations like prefix caching and CUDA graphs, custom kernels, and engine-specific defaults in logit processing. Our findings identify the inference backend as a previously unreported but consequential hyperparameter in the evaluation of LLM and advocate standardized reporting of inference stacks to improve the reproducibility and interpretability of benchmark comparisons.

Momentum

Related topics

Latest papers 88