cs.LGSep 25, 2026

Position: Let's Strengthen Verifiability If We Can't Enforce Reproducibility

Authors: Samet Hicsonmez, Nermin Samet, Renaud Marlet

Organizations: Independent Researcher Paris, France · Valeo.ai Paris, France

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

In the field of Machine Learning, many papers contain empirical results supporting claimed statements or illustrating the performance of a proposed method. However, most practitioners know that (1) results are generally hard to reproduce, and increasingly so, (2) code is not often available to do so, and (3) it hinders the development of research. In this position paper, we analyze and quantify these issues, and make concrete proposals to improve result checkability, if not reproducibility. Code and supporting materials are available at https://github.com/giddyyupp/position-enforce-verifiability.

Figures & tables

Figure 1: Proportion of accepted papers at CVPR, ICCV, ICLR, ICML and NeurIPS 2021–2025 that publicly (a) share train or test code [reducing], (b) provide pretrained model weights [reducing], or (c) acknowledge previous repositories [growing].
Figure 2: Proportion of papers per topic in accepted papers to CVPR, ICCV, ICLR, ICML and NeurIPS 2021–2025 (left), and code availability in each topic (right). Except for a few topics, the general trend shows a proportional decrease in open-sourcing official codes.
Figure 3: Mean & median citation counts for publications at CVPR, ICCV, ICLR, ICML and NeurIPS (2021–2025), grouped by code availability.
IndicatorStatus or rating
Experiment logs✓consistent with paper
?inconclusive wrt paper
✗inconsistent with paper
∅\boldsymbol{\varnothing}unavailable (default)
Checked metrics✓similar enough
?questionably different
Table 1: Provided indicators of reproducibility.
Figure 4: Overview of the proposed verification workflow. At submission time, training and inference logs are attached to the submission package and verified by one or several reviewers (Level 1). The metric verification step (Level 2), indicated with red background, requires the submission of JSON file(s) that contain model predictions on target dataset(s), along with an evaluation script. The assigned metric validator runs the evaluation script and compares with the numbers in the main paper. Finally, the AC updates the corresponding statuses ( Table 1 ) in the submission site.
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

ParameterYears
20212022202320242025
Accepted Papers (Main Track)16602064235927192872
PDFs Downloaded16602071235327132871
URLs Extracted16572069234927112868
LLM Detected Code URLs9601400155818491962
No URL in PDF595556693737757
Appendix
Table 2: Detailed statistics for CVPR.
ParameterYears
20212022202320242025
Accepted Papers (Main Track)1617-2160-2701
PDF Downloaded1612-2156-2701
URL Extracted1610-2153-2698
LLM Detected Code URL1008-1508-1851
No URL in PDF520-556-752
Appendix
Table 3: Detailed statistics for ICCV.
ParameterYears
20212022202320242025
Accepted Papers (Main Track)8601095158422963827
PDF Downloaded8601095157422633708
URL Extracted8601090156822613703
LLM Detected Code URL695918131719453272
No URL in PDF117128181228261
Appendix
Table 4: Detailed statistics for ICLR.
ParameterYears
20212022202320242025
Accepted Papers (Main Track)11831233186526343339
PDF Downloaded11831233182826103341
URL Extracted11791232182726063336
LLM Detected Code URL763883140220582734
No URL in PDF292238301367399
Appendix
Table 5: Detailed statistics for ICML.
ParameterYears
20212022202320242025
Accepted Papers (Main Track)23342673321840375290
PDF Downloaded23342671321840355287
URL Extracted23272424320240295280
LLM Detected Code URL15441674233534795066
No URL in PDF637576684234
Appendix
Table 6: Detailed statistics for NeurIPS.
Figure 5: Example log plots from MLFlow.
Figure 6: Example log plots from TensorBoard.
Figure 7: Example log plots from W&B.
More in this topic

ML Reproducibility

Machine learning reproducibility studies whether model results and scientific claims can be independently checked and reliably repeated.

See all 136 papers

Explore similar work

CardsList

  1. ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues

    Jun 16, 2026Shanda Li, Qiuhong Anna Wei, Jingwu Tang +5LLM Agent EvaluationAI Agent Benchmarks

    School of Computer Science, Carnegie Mellon University · Datadog

  2. Standardization of Post-Publication Code Verification is Possible with the Support of the Community

    Jan 12, 2026Susana Lopez-Moreno, Sangil Kim, Eric Dolores-CuencaScientific ReproducibilityML Reproducibility

    Department of Mathematics, Pusan National University, Busan, South Korea · Industrial Mathematics Center, Pusan National University, Busan, South Korea · Humanoid Olfactory Display Center, Pusan National University, Yangsan, Gyeongsangnam-do, South Korea +2

  3. Coding-agents can replicate scientific machine learning papers

    Jul 2, 2026Atharva Hans, Ilias BilionisCoding AgentsLLM Agent Evaluation

    School of Mechanical Engineering, Purdue University, West Lafayette, IN, 47907, USA