cs.SEMar 16, 2026

Code Sharing In Prediction Model Research: A Scoping Review

Authors: Thomas Sounack, Raffaele Giancotti, Catherine A. Gao, Lasai Barreñada, Hyeonhoon Lee, Hyung-Chul Lee, Leo Anthony Celi, Karel G. M. Moons, +3 more

Organizations: Dana-Farber Cancer Institute · University of Calabria · Northwestern Feinberg School of Medicine · KU Leuven · Seoul National University Hospital · Seoul National University College of Medicine · Massachusetts Institute of Technology · Harvard T.H. Chan School of Public Health · UMC Utrecht · University of Birmingham · University Hospitals Birmingham NHS Foundation Trust · Harvard Medical School

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Analytical code is essential for reproducing diagnostic and prognostic prediction model research, yet code availability in the published literature remains limited. While the TRIPOD statements set standards for reporting prediction model methods, they do not define explicit standards for repository structure and documentation. This review quantifies current code-sharing practices to inform the development of TRIPOD-Code, a TRIPOD extension reporting guideline focused on code sharing. We conducted a scoping review of PubMed-indexed articles citing TRIPOD or TRIPOD+AI as of Aug 11, 2025, restricted to studies retrievable via the PubMed Central Open Access API. Eligible studies developed, updated, or validated multivariable prediction models. A large language model-assisted pipeline was developed to screen articles and extract code availability statements and repository links. Repositories were assessed with the same LLM against 14 predefined reproducibility-related features. Our code is made publicly available. Among 3,967 eligible articles, 12.2% included code sharing statements. Code sharing increased over time, reaching 15.8% in 2025, and was higher among TRIPOD+AI-citing studies than TRIPOD-citing studies. Sharing prevalence varied widely by journal and country. Repository assessment showed substantial heterogeneity in reproducibility features: most repositories contained a README file (80.5%), but fewer specified dependencies (37.6%; version-constrained 21.6%) or were modular (42.4%). In prediction model research, code sharing remains relatively uncommon, and when shared, often falls short of being reusable. These findings provide an empirical baseline for the TRIPOD-Code extension and underscore the need for clearer expectations beyond code availability, including documentation, dependency specification, licensing, and executable structure.

Figures & tables

Figure 1: Modified PRISMA Flow diagram. We determined that an article was sharing code (n=482) if: it shared code in the Appendix (n=75), its repository provider was not supported (n=27) or its repository was included in the review (n=380).
Figure 2: Proportion of articles sharing their code by year, for the article cohort (n=3,967). Articles that only cited the TRIPOD statement are displayed in blue, and articles that only cited the TRIPOD+AI statement are displayed in orange. 17 articles cited both statements, of which 3 reported their code, and are excluded from this figure.
Figure 3: (Top) Proportion of articles sharing their code by country and (Bottom) number of articles by country, for the article cohort (n=3,967). Countries with more than 10 articles in the articles cohort are included.
Figure 4: Proportion of articles reporting their code by journal, for the article cohort (n=3,967). Journals with more than 10 articles in the cohort are included.
Figure 5: Repository characteristics – quality criteria against percentage of repositories, for the repository cohort (n=380). README files that provide an overview of the purpose and expected outputs of their code repository are shown as Complete, while those that do not are shown as Incomplete.
Figure 6: Repository characteristics by journal, for the repository cohort (n=380), for journals with more than 5 repositories. The legend displays journals sorted by descending average across criteria.
Figure 7: Annual distribution of programming languages across repositories, for the repository cohort (n=380) (languages appearing in >5 repositories overall). Years with less than 5 repositories were excluded.
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8
Figure 9
 PrecisionRecallF1 ScoreSupport
Out of scope0.930.940.93105
In scope0.980.980.98395
Weighted average0.970.970.97 
Appendix
Table 10
 PrecisionRecallF1 ScoreSupport
Contains README1.001.001.0029
README purpose and outputs0.701.000.8224
Contains Requirements0.621.000.7629
Requirements dependency versions1.000.860.928
Contains license1.001.001.0029
Sufficient Code Documentation0.710.710.7129
Appendix
Table 11
Code Sharing SentenceNumber Of Occurrences
The complete code used for model development and evaluation in this project is publicly available on GitHub:8
The source code for preprocessing and analyzing the data is available on GitHub3
The code to build and train the model is openly available on github.2
The code used to conduct the analysis is available on GitHub2
The implementation of the deep learning framework is available at:2
Code is available at2
Appendix
Table 12
Figure 13

Explore similar work

CardsList
Aug 25, 2026

ReproAgent: Contract-Guided Paper-to-Code Reproduction

Paper-to-code reproduction asks scientific AI agents to turn research papers into executable repositories that preserve the paper's method, protocol and artifacts. This is difficult because the specification is split: explicit paper content such as algorithms, metrics and artifacts is often lost across long agent trajectories, while implicit details such as framework defaults and conventions inherited from related work are absent from the paper. We introduce ReproAgent, a four-stage Prepare--Plan--Generate--Repair pipeline built around a persistent implementation contract with two channels: an implementation-requirement channel that turns paper snippets into code obligations, and a reference-evidence channel that retrieves content and structure evidence from related repositories. Both are bound to work packages, projected into file-level contracts, and consumed across generation and repair. On PaperBench Code-Dev, ReproAgent reaches the highest mean score among same-backbone scaffolds under both Claude-Sonnet-4.5 and Gemini-3-Flash. End-to-end channel ablations and per-paper cases support the contribution of both channels. Code and experimental artifacts are publicly available.

Xue Hu, Zewei Pan, Zhongyuan Wang +3

Beihang University · Shanghai Jiao Tong University · Peking University +3

Scientific Code GenerationAgentic Code GenerationCode Generation
AskPDF
Jun 15, 2026

The Shift Toward Open and Reproducible AI Research

The reproducibility crisis has directed the AI research community toward improving documentation practices. Several studies have identified methodological issues, and in response, the most impactful venues in the field have introduced reproducibility checklists. We seek to understand whether documentation practices have changed over time by assessing all published papers at five leading AI conferences over the past decade. Seven reproducibility variables were identified, quality-assured and used to analyse 56 800 publications. Our analysis reveals that in the period 2014 to 2024, documentation practices have improved; papers sharing both code and data increased nearly sixfold, from 11% to 64% Building on empirical reproducibility rates from a prior study, we estimate - inferred from documentation practices, not direct testing - that reproducibility increased from 28% in 2014 to 64% in 2024. Improvements in documentation practices predate the introduction of reproducibility checklists, suggesting these changes reflect a broader movement toward open science rather than a direct response to formal requirements.

Kevin L Coakley, Thijs Snelleman, Holger Hoos +1

Department of Computer Science, Norwegian University of Science and Technology, Sem Sælands Vei 7-9, Trondheim, 7031, Trønderlag, Norway. · San Diego Supercomputer Center, University of California San Diego, 9500 Gilman Dr, La Jolla, 92093, California, USA. · Chair of AI Methodology, RWTH Aachen University, Theaterstrasse 35-39, Aachen, 52062, North Rhine-Westphalia, Germany. +1

ML ReproducibilityScientific Reproducibility
AskPDF
Aug 17, 2026

Foundation Agents Meet Agentic Deep Research: Evidence-Grounded Clinical Code Forecasting

Next-encounter ICD forecasting predicts which standardized diagnosis codes will be documented at a future visit from the longitudinal record available beforehand. The task is prospective and multi-label: the target note does not yet exist, and several codes may be correct. Structured EHR foundation models capture recurrence and temporal progression, whereas language foundation models generate flexible diagnostic hypotheses. We introduce ICD-Deepresearch, a DeepResearch workflow that composes these predictive foundation models with medical search and ICD dictionaries. Because no source reveals the future code set, research evaluates candidate transitions by linking patient evidence, external clinical relations, and exact code semantics under a fixed top-K budget. Candidate Generation uses SparseEHR to produce an EHR Prior that initializes two bounded Research Expansion rounds; an independent GPT-5 Direct Forecast supplies complementary candidates. Final Selection validates, deduplicates, and jointly ranks both paths, after which a separate module writes rationales without changing predictions. Finally ICD-Deepresearch achieves patient-averaged precision/recall of 24.60/35.09% on MIMIC-III and 25.14/48.32% on MIMIC-IV. Physicians rate 51% and 68% of its retrieved documents useful, compared with 22% and 39% for standalone GPT-5 web search and 32% and 41% for Medical Deep Research. ICD-Deepresearch therefore improves over the registered local comparators while retrieving evidence with higher physician-rated usefulness than the standalone research systems

Junda Wang, Meysam Ghaffari, Akshat Choube +3

University of Massachusetts Amherst, MA, USA · Optum, MN, USA · University of Massachusetts Lowell, MA, USA

Clinical PredictionDeep Research AgentsLongitudinal EHR Modeling
AskPDF