ML Reproducibility

ML: Machine Learning

Follow this topic Ask about this topic

Momentum

14 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 85

All topics

Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing
Oct 7, 2026Arnold Olympio, Juan Manuel Servera Bondroit, Wael Abdelmalek +2LLM Inference ML Reproducibility
Lucerne University of Applied Sciences and Arts, Switzerland · Microsoft, Switzerland · Uthereal AG, Switzerland +1
Same Text, Different Prediction: Serving-Context Nondeterminism in Text Classifiers
Oct 6, 2026Santhosh Kumar Kasa, Siva Rajesh Kasa, Sumit NegiText Classification ML Reproducibility
Amazon
Beyond the Model: The Critical Role of Data Filtering in Clinical Machine Learning
Oct 5, 2026Noah Subedar, Colin Campbell, Wenjing Zhang +3Training Data Curation ML Reproducibility
School of Computer Science, University of Guelph, Ontario, Canada · St. Joseph’s Healthcare Hamilton, Hamilton, Ontario, Canada · The Research Institute of St. Joe’s Hamilton, Hamilton, Ontario, Canada +1
Technical Report on the Turba Fertilizer Machine Learning Stack in Morocco
Oct 5, 2026Abdelghani Belgaid, Zakaria Mahmoud, Fahd Chibani +3Surrogate Modeling ML Reproducibility
Turba Open Lab
Measuring and Reducing Cross-Vendor Mismatch in Language Models
Oct 4, 2026Erland Hilman Fuadi, Chong Tian, Xiaosong Ma +1Efficient Neural Network Inference ML Reproducibility
Mohamed bin Zayed University of Artificial Intelligence
Model validation in machine learning: A scenario-based guide from hold-out splits to nested group cross-validation in biomedical and applied research
Oct 1, 2026Mehmet Baygin, Sengul Dogan, Turker TuncerData Leakage ML Reproducibility
Department of Computer Engineering, Faculty of Engineering and Architecture, Erzurum Technical University, Erzurum, Türkiye · Department of Digital Forensics Engineering, Technology Faculty, Firat University, Elazig, Türkiye
Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluation
Sep 29, 2026Mario Sanz-Guerrero, Minh Duc Bui, Manuel Mager +1LLM Evaluation Language Model Generation Evaluation
Johannes Gutenberg University Mainz, Germany · Universidad Iberoamericana, Mexico · University of Colorado Boulder, USA
Making Cross-Continental Federated Learning Repeatable with FLIP: a Multi-Application Study
Sep 28, 2026Rafael Garcia-Dias, Alexandre Triay Bagur, Chayanin Tangwiriyasakul +20Healthcare ML Reproducibility
School of Biomedical Engineering & Imaging Sciences, King’s College London, London, UK · Bangkok Dusit Medical Services, Bangkok, Thailand
Training Witnesses: Trusting the Training without Trusting the Trainer
Sep 27, 2026Houjun Liu, Pratyusha SharmaLLM Auditing ML Reproducibility
Microsoft Research
Identical Runs, Different Results: Benchmarking AI Coding Agents on Open-Weight Models
Sep 27, 2026Eduardo Ariño de la Rubia, Szilard PafkaAI Coding Agents LLM Agent Evaluation
Central European University · Epoch
RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?
Sep 23, 2026Mithil Salunkhe, Haochen Ding, Samridhi Verma +1LLM Agent Evaluation AI Agent Benchmarks
University of Illinois Urbana-Champaign · National Center for Supercomputing Applications
Reproducible AI Requires Reproducible Randomness
Sep 22, 2026Anthony Bertrand, Tom Schmitt, Engelbert Mephu Nguifo +1ML Reproducibility Scientific Reproducibility
UCA, LIMOS · Université Clermont Auvergne, Clermont Auvergne INP, CNRS, LIMOS, F-63000 Clermont–Ferrand, France · UCA +2
Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures
Sep 22, 2026Liam Cooper, Shinnung Jeong, Hyeran Jeon +2GPU Kernel Optimization LLM Inference Acceleration
Georgia Institute of Technology · University of California, Merced
Exposing Blind Spots in Deep Imbalanced Regression Evaluation
Sep 21, 2026Noah C. Puetz, Jens U. Brandt, Marc Hilbert +3Class-Imbalanced Learning ML Reproducibility
TH Köln · Leiden University · Toyota Racing
Rethinking How We Evaluate Methodological Progress in Health AI
Sep 16, 2026Florent Pollet, Matthew McDermottML Reproducibility Longitudinal EHR Modeling
Department of Biomedical Informatics, Columbia University
TuiML: Machine Learning for AI Agents
Sep 16, 2026Nilesh Verma, Nick Lim, Albert Bifet +1Tool-Using Agents ML Reproducibility
AI Institute, University of Waikato Hamilton 3216, New Zealand · LTCI, T´el´ecom Paris, Institut Polytechnique de Paris 19 place Marguerite Perey, 91120 Palaiseau, France
OPEN-1B: A Fully Auditable Training Run
Sep 15, 2026John Donaghy, Brian Wilcox, Oğuzhan Ersoy +6Neural Network Verification LLM Auditing
Gensyn
Model Retirement Creates Reproducibility Risk in Biomedical AI Publications
Sep 7, 2026Nathan Wolfrath, Meghan Conroy, Thomas Kosten +7ML Reproducibility Scientific Reproducibility
Bud and Sue Selig Hub for Surgical Data Science, Department of Surgery, Medical College of Wisconsin · Division of Biostatistics, Data Science Institute, Medical College of Wisconsin · Department of Computer and Electrical Engineering, Marquette University +1
Training seeds and model-selection stability in recommender-system evaluation
Sep 2, 2026Juan Manuel Rodriguez, Oleg Lesota, Antonela TommaselRecommender Systems ML Reproducibility
Aalborg University Aalborg, Denmark · Johannes Kepler University Linz Linz, Austria · ISISTAN, CONICET-UNCPBA Tandil, Argentina
A Composable Evaluation System for Reproducible Omni-Modal Foundation Model Evaluation
Sep 1, 2026Hodong Lee, Sanghee Park, Dohoon Ryu +4ML Reproducibility Multimodal Foundation Models
NAVER Cloud AI · Korea University · KAIST AI +1
Thesis Proposal: Toward a Human-Centered and Perspective-Aware Framework for Reproducible ML Evaluation and AI Alignment
Aug 31, 2026Deepak Pandita, Christopher M. HomanAI Alignment Automated Evaluation
Rochester Institute of Technology
MURANO: Design, Run, and Reproduce Mechanistic Interpretability Experiments as Composable Pipelines
Aug 31, 2026Alireza Bayat Makou, Emirhan Böge, Phu Gia Hoang +5LLM Interpretability Interpretable ML
Ubiquitous Knowledge Processing (UKP) Lab, Technical University of Darmstadt, Germany · Cluster of Excellence “Reasonable Artificial Intelligence” (RAI), hessian.AI, Germany · National Research Center for Applied Cybersecurity ATHENE, Germany +1
Workflow Cards: Structured Summaries of Workflow Executions Using Provenance Data
Aug 11, 2026Nicola Giuseppe Marchioro, Gabriele Padovani, Amal Gueroudji +5Data Provenance ML Reproducibility
University of Trento, Trento, Italy · Argonne National Laboratory, Lemont, IL, USA · Oak Ridge National Laboratory, Oak Ridge, TN, USA
Seeds Before Objectives: Rethinking Evaluation for Low-Resource Garhwali ASR
Aug 11, 2026Karamvir Singh Batra, Prathamjyot Singh, Ashima Sood +2ASR Evaluation ML Reproducibility
Thapar Institute of Engineering and Technology, Patiala, Punjab, India · Ulster University, Londonderry, United Kingdom · Ulster University, Belfast, United Kingdom
The Evaluation Protocol Determines the Result: An Independent Reproduction of LeWorldModel on TwoRoom
Aug 10, 2026Joyjeet SinghWorld Model-Based Planning Long-Horizon Agent Evaluation
Independent researcher
Why Ranking Anomaly Detection Algorithms Isn't as Reliable as You May Think
Aug 5, 2026Simon Klüttermann, Jérôme Rutinowski, Frederik Polachowski +1Benchmark Design ML Reproducibility
Carnegie Mellon University, Pittsburgh, USA · TU Dortmund University, Dortmund, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Germany +2
Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility
Aug 4, 2026Mohsen Hariri, Weicong Chen, Nahal Shahini +11LLM Evaluation Test-Time Scaling
Case Western Reserve University
Stage-Replay Divergence Follows the KV Cache: Fixed-Prefix Precision Controls and Bidirectional Cache Transplantation
Jul 30, 2026Alexander Boesgaard LorupLLM Inference KV Caching
Openhagen
From Expert Reduction to Behavioral Divergence: Tracing Numerical State through Sparse MoE Inference
Jul 30, 2026Tianyang ZhuSparse Mixture-of-Experts Mixture-of-Experts Inference
Independent Researcher
One Run Is Not an Idea: The Implementation Lottery in Automated Research
Jul 29, 2026Jingjie Ning, Shanshan Zhong, Xiaochuan Li +2ML Reproducibility Autonomous Research Agents
School of Computer Science, Carnegie Mellon University
Scaling Laws for Classical Machine Learning on Tabular Data: A Benchmark Study
Jul 23, 2026Kaihua DingTabular ML Scaling Laws
University of Pennsylvania
GlucoTune: A Unified Framework for Blood Glucose Preprocessing, Forecasting, and Benchmarking in Diabetes
Jul 23, 2026Davide Marelli, Giorgia Rigamonti, Mirko Paolo Barbato +1Type 1 Diabetes Time Series Forecasting
Department of Informatics, Systems and Communication, University of Milano-Bicocca, Milano 20126, Italy
Teach it to stop, not just to click
Jul 19, 2026Barada Sahu, Shivesh PandeyAI Agent Reliability Agent Reliability
Cabal AI · Para AI
Is Model Instability just Noise to be Tolerated or a Property that can be Managed?
Jul 11, 2026Amirali Rayegan, Lunxiao Li, Tim MenziesStochastic Optimization Automated Software Engineering
North Carolina State University Raleigh, North Carolina, USA
GPUSimBench: Towards Scalable and Reliable GPU-Accelerated Simulators in Embodied AI
Jul 6, 2026Huzhenyu Zhang, Shenghai Yuan, Wenrui Yan +4GPU Acceleration Robotics Simulation
Shanghai AI Laboratory, China · MIRAI, Russia · Nanyang Technological University, Singapore +2
Grokking Is Conditional and Fragile: A Fully-Tractable, Multi-Seed Study at 12K Parameters
Jul 6, 2026Yoshiyuki OotaniPhase Transitions ML Reproducibility
Independent Researcher
Coding-agents can replicate scientific machine learning papers
Jul 2, 2026Atharva Hans, Ilias BilionisCoding Agents LLM Agent Evaluation
School of Mechanical Engineering, Purdue University, West Lafayette, IN, 47907, USA
The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation
Jun 18, 2026Nicolas Dufour, Alexei A. Efros, Patrick PérezML Reproducibility Image Generation Evaluation
Kyutai · UC Berkeley
ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues
Jun 16, 2026Shanda Li, Qiuhong Anna Wei, Jingwu Tang +5LLM Agent Evaluation AI Agent Benchmarks
School of Computer Science, Carnegie Mellon University · Datadog
The Shift Toward Open and Reproducible AI Research
Jun 15, 2026Kevin L Coakley, Thijs Snelleman, Holger Hoos +1ML Reproducibility Scientific Reproducibility
Department of Computer Science, Norwegian University of Science and Technology, Sem Sælands Vei 7-9, Trondheim, 7031, Trønderlag, Norway. · San Diego Supercomputer Center, University of California San Diego, 9500 Gilman Dr, La Jolla, 92093, California, USA. · Chair of AI Methodology, RWTH Aachen University, Theaterstrasse 35-39, Aachen, 52062, North Rhine-Westphalia, Germany. +1
Incentives and Evidence in Learned Service Orchestration
Jun 15, 2026Syed Izhan Khilji, Alireza Furutanpey, Schahram DustdarRL Control Reinforcement Learning
†Distributed Systems Group (DSG), TU Wien, Vienna, Austria · ‡Coovally.ai, Barcelona, Spain
RecourseBench: A Modular Framework for Reproducible Algorithmic Recourse Evaluation
Jun 15, 2026Zahra Khotanlou, Hashir Ahmed, Chenghao Tan +2Benchmark Design Automated Evaluation
University of Waterloo
Mojo: A Promising Tool for Scalable Financial AI Efficiency
Jun 14, 2026Henry HanQuantitative Finance ML Reproducibility
Data Science and Artificial Intelligence Innovation Laboratory, School of Engineering and Computer Science, Baylor University, USA
Snyk VulnBench JS 1.0: Can LLMs Find the Same Bugs Twice?
Jun 14, 2026Liran Tal, Johannes Kloos, Arsenii Rudich +2Software Vulnerability Detection Static Code Analysis
Snyk
BrainSurgery: Reproducible and Reliable Declarative Weight Manipulations for Model Editing and Upcycling
Jun 8, 2026Gianluca Barmina, Annemette Broch Pirchert, Andrea Blasi Núñez +2ML Reproducibility
University of Southern Denmark
Execution Realism and Reproducibility in LLM-Based Trading Systems: A Systematic Scoping Review and Evidence Audit
Jun 6, 2026Junyi Yao, Zihao Zheng, Baichuan Li +1ML Reproducibility Finance
Washington University in St. Louis, St. Louis, MO 63130 USA. · Department of Operations Research and Engineering Management, Southern Methodist University, Dallas, TX 75275 USA.
Accelerating Reproducible Research in Synthetic EHR Generation
Jun 5, 2026Jalen Jiang, Chufan Gao, Ethan Rasmussen +2Synthetic Data Generation ML Reproducibility
University of Illinois Urbana-Champaign
Jacobi-Anger Method for Deterministic Initialization in Implicit Neural Representation
Jun 4, 2026Mohammed Alsakabi, Kejia Hu, John M. Dolan +1Implicit Neural Representations Neural Network Initialization
Department of Electrical and Computer Engineering, College of Engineering · The Robotics Institute, School of Computer Science Carnegie Mellon University, Pittsburgh, PA, USA
Tight list replicability bounds via a novel sphere covering theorem
Jun 4, 2026Ari Blondal, Hamed Hatami, Pooya Hatami +2ML Reproducibility Statistical Learning Theory
McGill University · Ohio State University
Revisiting Vul-RAG: Reproducibility and Replicability of RAG-based Vulnerability Detection with Open-Weight Models
Jun 3, 2026Sabrina Kaniewski, Fabian Schmidt, Tobias HeerRetrieval-Augmented Generation LLM Evaluation
Institute for Secure Networked Systems, Esslingen University, Esslingen, Germany · Institute for Intelligent Systems, Esslingen University, Esslingen, Germany
Reproducibility is the New Copyleft: Defining AGI-oriented Reproducible Builds
Jun 2, 2026Masayuki HattaML Reproducibility AI Governance
Surugadai University, Hanno Saitama 3570046, Japan
ERICA: Quantifying Replicability of Cluster Analysis
May 29, 2026Siamak K. Sorooshyari, Manuel A. Rivas, Robert TibshiraniClustering ML Reproducibility
Department of Statistics Stanford University Stanford, CA 94305, USA · Department of Biomedical Data Science Stanford University Stanford, CA 94305, USA · Department of Statistics and Biomedical Data Science Stanford University Stanford, CA 94305, USA
Modeling Robotics Dataset Construction as an Artifact-Based Build Process
May 29, 2026Leon Pohl, Lukas Beer, George Sebastian +1Robotics Automated Software Engineering
Institute for Autonomous Driving, University of the Bundeswehr Munich, 85579 Neubiberg, Germany
A Unified and Reproducible Experimentation Framework for Speech Understanding
May 29, 2026Jing Peng, Junhao Du, Chenghao Wang +21Audio-Language Model Evaluation ML Reproducibility
1X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University · 2AISpeech Ltd, Suzhou, China · 3ETH Zürich, Switzerland +3
Croissant Tasks: A Metadata Format for Reproducible Machine Learning Evaluations
May 28, 2026Omar Benjelloun, Leonardo Martins Bianco, Isabelle Guyon +8ML Reproducibility
Google DeepMind, Paris, France · ChaLearn, CA, USA · Université Paris-Saclay, Gif-sur-Yvette, France +8
From paper to benchmark: agentic, framework-based reproduction of under-specified methods in machine health intelligence
May 27, 2026Raffael Theiler, Ludovico Comito, David Leko +3Agentic Code Generation ML Reproducibility
EPFL · University of Roma “La Sapienza”
Picid: A Modular Evaluation Infrastructure for Reproducible PHM Across Tasks and Domains
May 27, 2026Lev Telyatnikov, Raffael Theiler, Leandro Von Krannichfeldt +1Benchmark Design Predictive Maintenance
EPFL
stable-worldmodel: A Platform for Reproducible World Modeling Research and Evaluation
May 20, 2026Lucas Maes, Quentin Le Lidec, Luiz Facury +9World Models ML Reproducibility
Mila & Université de Montréal · New York University · Universidade Federal de Minas Gerais +4
HyperBench: Standardizing and Scaling Synthetic Evaluation for Hyperspectral Super-Resolution
May 20, 2026Ritik Shah, Marco F. DuarteBenchmark Design Remote Sensing Image Super-Resolution
University of Massachusetts Amherst, MA 01003
A Reproducible Log-Driven AutoML Framework for Interpretable Pipeline Optimization in Healthcare Risk Prediction
May 19, 2026Rui Huang, Lican HuangClass-Imbalanced Learning Interpretable ML
School of Basic Medicine, Hangzhou Normal University, No.2318, Yuhangtang Rd, Yuhang District, Hangzhou, 311121, Zhejiang, China. · Research Department, Hangzhou Domain Zones Technology Co.Ltd.,2026 200 Zhenhua Rd, West District, Hangzhou, 310030, Zhejiang, China.

Oct 7, 2026, cs.PFJ/K move · Enter open · S save

Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing

Arnold Olympio, Juan Manuel Servera Bondroit, Wael Abdelmalek, Guang Lu+1

Lucerne University of Applied Sciences and Arts, Switzerland · Microsoft, Switzerland · Uthereal AG, Switzerland+1

LLM Inference ML Reproducibility LLM Evaluation LLM Reliability

Open paper AskPDF arXiv

Reproducible benchmarking of Large Language Model (LLM) inference is challenging because repeated measurements can vary with execution and system state. We present the Sequential Isolation Methodology, a controlled benchmarking and regression-testing protocol designed to reduce between-run measurement variance while deliberately varying workload concurrency. We evaluate three representative open-source LLMs on an NVIDIA A100 80GB GPU using vLLM 0.9.1 across six context sizes and eight concurrency levels, with five repetitions per configuration. The final protocol reduces average coefficient of variation (CV) from 15.2% in the least controlled methodology stage to 2.2% under the final protocol; using CV computed across the five repetition-level median (P50) TTFT values per configuration, 113 of 144 configurations (78.5%) achieve CV below 3%. The measurements also show a marked latency transition between 200 and 500 concurrent users on the tested stack and descriptive differences in P99 latency across the three models. We additionally provide an explicit cost break-even model with sensitivity to API pricing. The protocol is intended to provide a stable reference for reproducible comparison and regression testing rather than to predict absolute behavior under uncontrolled production traffic. Infrastructure-as-Code and benchmark scripts support replication of the experimental environment.

Momentum

Related topics

Latest papers 85