ML Reproducibility

ML: Machine Learning

Follow this topic Ask about this topic

Momentum

14 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 88

All topics

Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing
Oct 7, 2026Arnold Olympio, Juan Manuel Servera Bondroit, Wael Abdelmalek +2LLM Inference ML Reproducibility
Lucerne University of Applied Sciences and Arts, Switzerland · Microsoft, Switzerland · Uthereal AG, Switzerland +1
Same Text, Different Prediction: Serving-Context Nondeterminism in Text Classifiers
Oct 6, 2026Santhosh Kumar Kasa, Siva Rajesh Kasa, Sumit NegiText Classification ML Reproducibility
Amazon
Beyond the Model: The Critical Role of Data Filtering in Clinical Machine Learning
Oct 5, 2026Noah Subedar, Colin Campbell, Wenjing Zhang +3Training Data Curation ML Reproducibility
School of Computer Science, University of Guelph, Ontario, Canada · St. Joseph’s Healthcare Hamilton, Hamilton, Ontario, Canada · The Research Institute of St. Joe’s Hamilton, Hamilton, Ontario, Canada +1
Technical Report on the Turba Fertilizer Machine Learning Stack in Morocco
Oct 5, 2026Abdelghani Belgaid, Zakaria Mahmoud, Fahd Chibani +3Surrogate Modeling ML Reproducibility
Turba Open Lab
Measuring and Reducing Cross-Vendor Mismatch in Language Models
Oct 4, 2026Erland Hilman Fuadi, Chong Tian, Xiaosong Ma +1Efficient Neural Network Inference ML Reproducibility
Mohamed bin Zayed University of Artificial Intelligence
Model validation in machine learning: A scenario-based guide from hold-out splits to nested group cross-validation in biomedical and applied research
Oct 1, 2026Mehmet Baygin, Sengul Dogan, Turker TuncerData Leakage ML Reproducibility
Department of Computer Engineering, Faculty of Engineering and Architecture, Erzurum Technical University, Erzurum, Türkiye · Department of Digital Forensics Engineering, Technology Faculty, Firat University, Elazig, Türkiye
Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluation
Sep 29, 2026Mario Sanz-Guerrero, Minh Duc Bui, Manuel Mager +1LLM Evaluation Language Model Generation Evaluation
Johannes Gutenberg University Mainz, Germany · Universidad Iberoamericana, Mexico · University of Colorado Boulder, USA
Making Cross-Continental Federated Learning Repeatable with FLIP: a Multi-Application Study
Sep 28, 2026Rafael Garcia-Dias, Alexandre Triay Bagur, Chayanin Tangwiriyasakul +20Healthcare ML Reproducibility
School of Biomedical Engineering & Imaging Sciences, King’s College London, London, UK · Bangkok Dusit Medical Services, Bangkok, Thailand
Training Witnesses: Trusting the Training without Trusting the Trainer
Sep 27, 2026Houjun Liu, Pratyusha SharmaLLM Auditing ML Reproducibility
Microsoft Research
Identical Runs, Different Results: Benchmarking AI Coding Agents on Open-Weight Models
Sep 27, 2026Eduardo Ariño de la Rubia, Szilard PafkaAI Coding Agents LLM Agent Evaluation
Central European University · Epoch
RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?
Sep 23, 2026Mithil Salunkhe, Haochen Ding, Samridhi Verma +1LLM Agent Evaluation AI Agent Benchmarks
University of Illinois Urbana-Champaign · National Center for Supercomputing Applications
Reproducible AI Requires Reproducible Randomness
Sep 22, 2026Anthony Bertrand, Tom Schmitt, Engelbert Mephu Nguifo +1ML Reproducibility Scientific Reproducibility
UCA, LIMOS · Université Clermont Auvergne, Clermont Auvergne INP, CNRS, LIMOS, F-63000 Clermont–Ferrand, France · UCA +2
Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures
Sep 22, 2026Liam Cooper, Shinnung Jeong, Hyeran Jeon +2GPU Kernel Optimization LLM Inference Acceleration
Georgia Institute of Technology · University of California, Merced
Exposing Blind Spots in Deep Imbalanced Regression Evaluation
Sep 21, 2026Noah C. Puetz, Jens U. Brandt, Marc Hilbert +3Class-Imbalanced Learning ML Reproducibility
TH Köln · Leiden University · Toyota Racing
Rethinking How We Evaluate Methodological Progress in Health AI
Sep 16, 2026Florent Pollet, Matthew McDermottML Reproducibility Longitudinal EHR Modeling
Department of Biomedical Informatics, Columbia University
TuiML: Machine Learning for AI Agents
Sep 16, 2026Nilesh Verma, Nick Lim, Albert Bifet +1Tool-Using Agents ML Reproducibility
AI Institute, University of Waikato Hamilton 3216, New Zealand · LTCI, T´el´ecom Paris, Institut Polytechnique de Paris 19 place Marguerite Perey, 91120 Palaiseau, France
OPEN-1B: A Fully Auditable Training Run
Sep 15, 2026John Donaghy, Brian Wilcox, Oğuzhan Ersoy +6Neural Network Verification LLM Auditing
Gensyn
Model Retirement Creates Reproducibility Risk in Biomedical AI Publications
Sep 7, 2026Nathan Wolfrath, Meghan Conroy, Thomas Kosten +7ML Reproducibility Scientific Reproducibility
Bud and Sue Selig Hub for Surgical Data Science, Department of Surgery, Medical College of Wisconsin · Division of Biostatistics, Data Science Institute, Medical College of Wisconsin · Department of Computer and Electrical Engineering, Marquette University +1
Training seeds and model-selection stability in recommender-system evaluation
Sep 2, 2026Juan Manuel Rodriguez, Oleg Lesota, Antonela TommaselRecommender Systems ML Reproducibility
Aalborg University Aalborg, Denmark · Johannes Kepler University Linz Linz, Austria · ISISTAN, CONICET-UNCPBA Tandil, Argentina
A Composable Evaluation System for Reproducible Omni-Modal Foundation Model Evaluation
Sep 1, 2026Hodong Lee, Sanghee Park, Dohoon Ryu +4ML Reproducibility Multimodal Foundation Models
NAVER Cloud AI · Korea University · KAIST AI +1
Thesis Proposal: Toward a Human-Centered and Perspective-Aware Framework for Reproducible ML Evaluation and AI Alignment
Aug 31, 2026Deepak Pandita, Christopher M. HomanAI Alignment Automated Evaluation
Rochester Institute of Technology
MURANO: Design, Run, and Reproduce Mechanistic Interpretability Experiments as Composable Pipelines
Aug 31, 2026Alireza Bayat Makou, Emirhan Böge, Phu Gia Hoang +5LLM Interpretability Interpretable ML
Ubiquitous Knowledge Processing (UKP) Lab, Technical University of Darmstadt, Germany · Cluster of Excellence “Reasonable Artificial Intelligence” (RAI), hessian.AI, Germany · National Research Center for Applied Cybersecurity ATHENE, Germany +1
Workflow Cards: Structured Summaries of Workflow Executions Using Provenance Data
Aug 11, 2026Nicola Giuseppe Marchioro, Gabriele Padovani, Amal Gueroudji +5Data Provenance ML Reproducibility
University of Trento, Trento, Italy · Argonne National Laboratory, Lemont, IL, USA · Oak Ridge National Laboratory, Oak Ridge, TN, USA
Seeds Before Objectives: Rethinking Evaluation for Low-Resource Garhwali ASR
Aug 11, 2026Karamvir Singh Batra, Prathamjyot Singh, Ashima Sood +2ASR Evaluation ML Reproducibility
Thapar Institute of Engineering and Technology, Patiala, Punjab, India · Ulster University, Londonderry, United Kingdom · Ulster University, Belfast, United Kingdom
The Evaluation Protocol Determines the Result: An Independent Reproduction of LeWorldModel on TwoRoom
Aug 10, 2026Joyjeet SinghWorld Model-Based Planning Long-Horizon Agent Evaluation
Independent researcher
Why Ranking Anomaly Detection Algorithms Isn't as Reliable as You May Think
Aug 5, 2026Simon Klüttermann, Jérôme Rutinowski, Frederik Polachowski +1Benchmark Design ML Reproducibility
Carnegie Mellon University, Pittsburgh, USA · TU Dortmund University, Dortmund, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Germany +2
Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility
Aug 4, 2026Mohsen Hariri, Weicong Chen, Nahal Shahini +11LLM Evaluation Test-Time Scaling
Case Western Reserve University
Stage-Replay Divergence Follows the KV Cache: Fixed-Prefix Precision Controls and Bidirectional Cache Transplantation
Jul 30, 2026Alexander Boesgaard LorupLLM Inference KV Caching
Openhagen
From Expert Reduction to Behavioral Divergence: Tracing Numerical State through Sparse MoE Inference
Jul 30, 2026Tianyang ZhuSparse Mixture-of-Experts Mixture-of-Experts Inference
Independent Researcher
One Run Is Not an Idea: The Implementation Lottery in Automated Research
Jul 29, 2026Jingjie Ning, Shanshan Zhong, Xiaochuan Li +2ML Reproducibility Autonomous Research Agents
School of Computer Science, Carnegie Mellon University

Oct 7, 2026, cs.PFJ/K move · Enter open · S save

Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing

Arnold Olympio, Juan Manuel Servera Bondroit, Wael Abdelmalek, Guang Lu+1

Lucerne University of Applied Sciences and Arts, Switzerland · Microsoft, Switzerland · Uthereal AG, Switzerland+1

LLM Inference ML Reproducibility LLM Evaluation LLM Reliability

Open paper AskPDF arXiv

Reproducible benchmarking of Large Language Model (LLM) inference is challenging because repeated measurements can vary with execution and system state. We present the Sequential Isolation Methodology, a controlled benchmarking and regression-testing protocol designed to reduce between-run measurement variance while deliberately varying workload concurrency. We evaluate three representative open-source LLMs on an NVIDIA A100 80GB GPU using vLLM 0.9.1 across six context sizes and eight concurrency levels, with five repetitions per configuration. The final protocol reduces average coefficient of variation (CV) from 15.2% in the least controlled methodology stage to 2.2% under the final protocol; using CV computed across the five repetition-level median (P50) TTFT values per configuration, 113 of 144 configurations (78.5%) achieve CV below 3%. The measurements also show a marked latency transition between 200 and 500 concurrent users on the tested stack and descriptive differences in P99 latency across the three models. We additionally provide an explicit cost break-even model with sensitivity to API pricing. The protocol is intended to provide a stable reference for reproducible comparison and regression testing rather than to predict absolute behavior under uncontrolled production traffic. Infrastructure-as-Code and benchmark scripts support replication of the experimental environment.

Momentum

Related topics

Latest papers 88