ReproBridge: Building Reproducibility into ML Research Artifacts

Yubo Wang; Rui Meng; Jaehyun Nam; Tomas Pfister; Jinsung Yoon

cs.AINov 5, 2026

ReproBridge: Building Reproducibility into ML Research Artifacts

Authors: Yubo Wang, Rui Meng, Jaehyun Nam, Tomas Pfister, Jinsung Yoon

Organizations: University of Waterloo · Google

Abstract

Reproducing machine learning experiments requires reconstructing procedures from information scattered across papers, released code, and supporting artifacts. For coding agents, reproduction success can therefore depend on their ability to fill in missing details, rather than on the released artifacts alone. We ask whether preserving recovered procedures and decisions improves subsequent independent reproduction. We define a standardized artifact design comprising a refined codebase, environment setup, experiment launchers, and a reproduction guide, and introduce ReproBridge, an agent workflow for constructing these refined reproducible artifacts. ReproBridge-Bench provides 1,029 tasks from 387 experiments across 36 papers, anchored by five human-verified reference scores per task. Agents attempt tasks in fresh sessions with raw code or refined artifacts; a separate Auditor checks execution evidence. Across four evaluated models, refined artifacts raise mean reproduction success from 76.4% to 92.6% and reduce downstream API spending by 19.7% on average. The cross-model success-rate spread narrows from 21.6 to 3.9 percentage points, with all four models achieving mean success rates above 90%. Removing any component reduces success in the ablation study, with the largest losses from code and environment removal. These results suggest that preserving execution knowledge helps even models with lower raw-code success reproduce experiments more reliably and at lower API cost.

Figures & tables

Figure 1: Research coverage and benchmark composition. (a) Eight ICML-adapted topic families and 22 subdomains cover 36 papers; sector angles encode paper counts. Outer marks show 144 paper–model pairs from the first evaluation with Codex auditing. Each paper’s four clockwise marks follow Sonnet 4.6, Sonnet 5, Opus 5, and Gemini 3.8 Flash. Green bars encode RRA-minus-Raw gains from the blue zero ring; dashed rings mark 25, 50, and 100 pp. Red/grey ticks indicate decreases/ties without magnitude. Legend counts are pairs; positive bins are

(a,b]

. (b) Paper selection. (c) Procedure and hardware are separate partitions of the same 1,029 verified tasks.

Cat.	Verification outcome	Interpretation	Tasks	Share (%)
1	Reproduced: stable	Compatible reference, low variability	701	47.9
2	Reproduced: variable	Compatible reference, higher variability	328	22.4
3	Numerical mismatch	Executed, but numerically incompatible	183	12.5
4	Execution blocked	Missing artifacts, dependencies, or code defects	186	12.7
5	Resource/access limit	Compute capacity or restricted access	65	4.4
Downstream evaluation (Categories 1 + 2)			1,029	70.3

Table 1: Reference verification outcomes. A task is one selected own-method measurement from a paper’s main tables. Counts and shares cover all 1,463 curated tasks from 36 papers. Categories 1 and 2 have verified reference measurements compatible with the published score and differ at the 10% relative-variation threshold; together they supply the 1,029 downstream tasks. Categories 3–5 distinguish numerical disagreement, execution blockers, and resource/access limits.

Figure 2: ReproBridge artifact construction. Six roles form three stages: Specify (Intake, Planner), Execute (Builder, construction Reproducer), and Preserve (Distiller, Packager). Footers list stage outputs. Solid arrows show construction; dashed links show independent review and revision. The final handoff contains refined code, environment setup, launchers, and a guide. Downstream reproduction and auditing are separate evaluation steps.

Role	Input and responsibility	Output and check
Intake	Paper, pinned raw code, selected tasks; read and map the inputs	Input manifest, experiment groups, artifact availability, initial ambiguities
Planner	Task mapping and ambiguities; specify the measurement before execution	Task specifications with source references, recorded choices, execution plans, draft launchers
Builder	Dependency and artifact requirements; construct a working environment	Locked dependencies, setup and data scripts, container definition, smoke-test evidence
Reproducer	Plans and verified setup; execute and record justified changes	File-backed measurements, code changes, repetition records, explicit failure reasons
Distiller	Execution evidence, repairs, and decisions; synthesize the handoff	Reproduction guide, change record, aggregation and variability guidance
Packager	Code, setup, launchers, and documents; assemble a portable deliverable	Refined artifact and clean-shell verification evidence

Table 2: Construction skills: inputs, responsibility, and persistent outputs. A checkpoint reviewer is reused at different stages in separate contexts. The construction Reproducer is not the fresh downstream session used to evaluate reuse.

Figure 3: A refined artifact for SCIQL. Left: selected guide entries specify the six-criterion, five-seed protocol, unweighted aggregation, and checks for silently empty datasets. Right: an alias applied before unpickling preserves episode classes; CPU PyTorch avoids a cuDNN conflict with GPU JAX; launchers connect configurations to execution, output collection, and aggregation. The four components preserve executable steps and the reasons for them.

Model	Raw	RRA	Gain	Raw	RRA	Red.	Raw	RRA	Red.
	Success (%)			Tokens (B)			API (USD)
Sonnet 4.6	$72.3_{\pm 1.1}$	$\mathbf{91.6}_{\pm 3.4}$	+19.3	$8.48_{\pm 0.38}$	$6.80_{\pm 0.76}$	19.8	$3{,}165_{\pm 142}$	$2{,}512_{\pm 286}$	20.6
Sonnet 5	$76.5_{\pm 2.8}$	$\mathbf{93.3}_{\pm 1.4}$	+16.8	$4.98_{\pm 0.55}$	$4.52_{\pm 0.24}$	9.2	$1{,}726_{\pm 175}$	$1{,}317_{\pm 102}$	23.7
Opus 5	$89.2_{\pm 1.2}$	$\mathbf{94.7}_{\pm 2.2}$	+5.5	$3.01_{\pm 0.16}$	$2.82_{\pm 0.29}$	6.3	$3{,}480_{\pm 165}$	$2{,}865_{\pm 312}$	17.7
Gemini 3.8	$67.6_{\pm 1.9}$	$\mathbf{90.8}_{\pm 1.1}$	+23.2	$2.24_{\pm 0.26}$	$1.83_{\pm 0.09}$	18.3	$392_{\pm 41}$	$326_{\pm 16}$	16.8

Table 3: Main results on 36 test papers and 1,029 tasks. Raw uses the released codebase; RRA uses the refined reproducible artifact. Entries report mean

\pm

standard deviation over three evaluations (SD in subscripts). Success requires numerical compatibility and no confirmed audit violation. Gain is RRA minus Raw in percentage points; Red. is

100(\mathrm{Raw}-\mathrm{RRA})/\mathrm{Raw}

, computed from the means. Tokens (B: billions) and API USD sum Reproducer sessions, including failed attempts; construction, auditing, and GPU costs are excluded.

Figure 4: Paper-wise reproduction success in the first evaluation. Each panel contains one point per retained paper, comparing the percentage of the same tasks that succeed with Raw (horizontal axis) and the refined artifact, RRA (vertical axis), under primary Codex scoring. Marker area is proportional to task count. Green points above the equality line improve, blue points on the line tie, and red points below it regress. Overlapping observations retain their exact positions.

	(a) Paper-macro, 95% interval				(b) Task-level, 80% interval
Model	Raw	RRA	Gain	I/T/W	Raw	RRA	Gain
Sonnet 4.6	69.9	89.7	+19.8	25/8/3	61.3	88.2	+26.9
Sonnet 5	75.1	90.6	+15.5	23/9/4	63.9	90.5	+26.5
Opus 5	87.5	94.2	+6.7	16/15/5	82.0	92.3	+10.3
Gemini 3.8 Flash	67.2	87.9	+20.7	25/5/6	47.7	86.8	+39.1

Table 5: Gains persist with equal paper weights and stricter scoring. First-evaluation results under primary Codex scoring. (a) Paper-macro success gives equal weight to all 36 papers with the default 95% interval; I/T/W counts improved, tied, and worse papers. (b) Task-level success uses the 80% interval on the same 1,029 tasks, with category labels fixed. Raw and RRA are success percentages; Gain is RRA minus Raw in percentage points, calculated before rounding.

Figure 5: Recovering failures and shortening preparation. First-evaluation results across four models and 1,029 tasks per model. Left: positive bars count Raw failures becoming RRA successes, grouped by the Raw outcome: numerical mismatch, execution/resource barrier (Not run), or audit-invalid numerical match. Negative bars count Raw successes lost with RRA; labels give recovery and reversal totals. Right: median time to first scored execution across paper sessions (minutes); hollow and filled markers denote Raw and RRA, respectively.

Full RRA without one component:
Handoff	Success (%)	Loss (pp)
Raw code	73.9	—
Ordinary outputs	81.2	—
Full RRA	93.0	—
Refined codebase	87.8	5.2
Environment setup	88.5	4.5

Table 6: Handoff and component ablations. Sonnet 5, 287 tasks, 12 papers, one evaluation. Loss is the drop from full RRA (pp); removals follow the adjacent protocol.

Reproducer	Input	Codex		Claude
		Invalid / matched	Success (%)	Invalid / matched	Success (%)
Sonnet 4.6	Raw	63/805	72.1	41/805	74.2
Sonnet 4.6	RRA	23/968	91.8	6/968	93.5
Sonnet 5	Raw	51/838	76.5	4/838	81.0
Sonnet 5	RRA	0/962	93.5	0/962	93.5
Opus 5	Raw	43/961	89.2	0/961	93.4

Table 7: Sensitivity to the Auditor in a single evaluation. Codex and Claude inspect the same numerically matching outcomes for each model and handoff. Invalid / matched reports the Auditor’s invalidations and the number of numerical matches; Success is

100(\mathrm{matched}-\mathrm{invalid})/1{,}029

. Raw denotes the released codebase and RRA the refined artifact.

Figure 6: Audit judgments against human verification. Two humans cross-verify 409 tasks from eight papers under both Raw and refined-artifact (RRA) conditions (818 outcomes). Each 409-outcome matrix has human invalid/valid rows and Auditor flag/pass columns. TP and FN are detected and missed invalid outcomes; FP and TN are flagged and accepted valid outcomes. Both Auditors are scored against the same human labels.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Primary research area	Paper IDs	Papers
Deep learning	35, 71, 84, 95, 96, 116, 129, 145	8
Trustworthy ML	7, 20, 24, 90, 124, 135, 169, 187	8
General ML & theory	1, 40, 66, 149, 223, 230	6
ML systems & optimization	81, 113, 121, 136, 141, 238	6
Reinforcement learning & robotics	2, 58, 134	3
Probabilistic methods	50, 112	2

Appendix

Table 8: Paper-level assignments to the eight ICML-adapted topic families in Figure 1 . IDs refer to the paper inventory in Table 9 . Each paper contributes one count; no task-level weighting is applied.

ID	Paper	Tasks	S4.6	S5	O5	G3.8
1	( Li et al., 2026a )	12	9 / 9	9 / 9	11 / 9	12 / 9
2	( Petitbois et al., 2026 )	15	2 / 15	15 / 15	15 / 15	10 / 15
7	( Ferrere et al., 2026 )	82	75 / 76	75 / 81	80 / 77	62 / 75
20	( Zhou et al., 2026 )	12	10 / 12	2 / 12	9 / 12	11 / 9
24	( Angell et al., 2026 )	3	3 / 3	3 / 2	3 / 3	3 / 2
35	( Li et al., 2026c )	40	9 / 40	28 / 40	40 / 40	16 / 40

Appendix

Table 9: Retained papers and successful task counts from a single evaluation, raw / RRA, under the primary Codex scoring rule. The four columns are Sonnet 4.6 (S4.6), Sonnet 5 (S5), Opus 5 (O5), and Gemini 3.8 Flash (G3.8). IDs are dataset identifiers, not ranks.

ID	Paper	Criteria
133	( Ding et al., 2026 )	C1
156	( Słupiński and Lipiński, 2026 )	C1
44	( Xu et al., 2026 )	C1
27	( Sun et al., 2026 )	C1 / C3
218	( Zhao et al., 2026a )	C1 / C3
219	( Zhang et al., 2026b )	C1

Appendix

Table 10: Excluded papers in the supplied 50-paper manifest. C1: code/artifact availability; C2: node capacity; C3: per-experiment compute; C4: closed-model API dependence. The first listed code is the primary reason. Selection criteria are described in Appendix A .

Figure 7: Inside a reproduction guide. Condensed SCIQL entries retain all twelve section headings in their original order. The document records hardware and software prerequisites, data acquisition, launch commands, repetitions, output fields and aggregation rules, path conventions, ambiguities, known pitfalls, and unsupported scope. The example links a renamed-package issue to a compatibility launcher and a minimum-episode check, preserving both an executable route and the reason for it. Commands and output keys specify how to produce and extract new measurements.

Role / label	Model identifier	Harness	Effort
Construction	claude-opus-5[1m]	Claude Code	max
Sonnet 4.6	claude-sonnet-4-6[1m]	Claude Code	max
Sonnet 5	claude-sonnet-5[1m]	Claude Code	max
Opus 5	claude-opus-5[1m]	Claude Code	max
Gemini 3.8 Flash	gemini-3.8-flash	Gemini CLI	high
Primary Auditor	gpt-5.6-sol	Codex CLI	high

Appendix

Table 11: Model identifiers and harness settings for main-experiment construction and downstream evaluation. “Max” and “high” are tool-specific effort settings, not a shared compute scale.

Scope	Auditor	Flagged	TP	FP	FN	Precision (%)	Recall (%)
All	Codex	35	31	4	8	88.6	79.5
All	Claude	28	24	4	15	85.7	61.5
Raw	Codex	29	26	3	7	89.7	78.8
Raw	Claude	24	20	4	13	83.3	60.6
RRA	Codex	6	5	1	1	83.3	83.3
RRA	Claude	4	4	0	2	100.0	66.7

Appendix

Table 12: Audit predictions checked against human verification on 409 outcomes per condition (818 overall). Flagged = TP + FP; TP and FP are correct and incorrect invalid flags, and FN counts missed invalid outcomes. Human-confirmed invalid counts are 33 for Raw and six for RRA. Precision = TP/(TP + FP); recall = TP/(TP + FN).

Figure 8: Gains across reference-variability levels. Blue circles show stable references (Cat. 1, 701 tasks); amber squares show higher-variability references (Cat. 2, 328 tasks), separated by the 10% relative-variation threshold. Markers and right-hand labels give refined-artifact (RRA) minus Raw success in percentage points in the first evaluation.

Model	Cat.	Raw (%)	RRA (%)	Gain (pp)
Sonnet 4.6	1	71.3	93.6	+22.3
Sonnet 4.6	2	73.8	88.1	+14.3
Sonnet 5	1	75.6	95.4	+19.8
Sonnet 5	2	78.4	89.3	+11.0
Opus 5	1	88.9	96.9	+8.0
Opus 5	2	89.9	92.1	+2.1

Appendix

Table 13: Success by reference variability in the first evaluation. Stable (Cat. 1,

n=701

) and higher-variability (Cat. 2,

n=328

) tasks use the same fixed compatibility thresholds in Raw and refined-artifact (RRA) conditions. Gain is RRA minus Raw in percentage points, calculated before rounding the displayed rates.

Model	Raw	RRA	Recoveries	Reversals	Net
Sonnet 4.6	742	945	234	31	203
Sonnet 5	787	962	208	33	175
Opus 5	918	981	100	37	63
Gemini 3.8 Flash	696	932	271	35	236
Total	3,143	3,820	813	136	677

Appendix

Table 14: First-evaluation success counts and transitions, with 1,029 tasks per model. Recovery: Raw failure to RRA success; reversal: Raw success to RRA failure. Net is recoveries minus reversals.

Model	Raw	RRA	Raw	RRA
	Papers with patches		First scored execution (h)
Sonnet 4.6	26	11	0.50	0.27
Sonnet 5	30	14	0.90	0.53
Opus 5	24	16	0.54	0.32
Gemini 3.8 Flash	33	13	0.73	0.26

Appendix

Table 15: Downstream repair and time-to-execution diagnostics. Patch counts are papers with a Reproducer code diff out of 36, not independent defects. Time is the median to the first scored execution, not end-to-end runtime.

Explanation	Est. outcomes	Share (%)
Protocol	210	25.8
Launcher	76	9.4
Env	87	10.7
Code	99	12.2
Artifact	79	9.7
Integrity	217	26.7

Appendix

Table 16: Explanations for recovered outcomes. Estimated counts allocate case-review shares across 813 recoveries, with rounding that preserves the total.

RRA outcome	Reversals	Share (%)
Numerical mismatch	84	61.8
Execution blocked	10	7.4
Resource/access limit	31	22.8
Audit-invalid match	11	8.1

Appendix

Table 17: Failure states after a reversal. Counts and shares of all 136 Raw-success-to-RRA-failure outcomes across four models.

ID	Paper	Experiments	Tasks	CPU-only tasks
1	TESS	4	12	0
20	EnergyDPS	3	12	6
50	ControlConsistencyBridges	13	18	0
112	AuxiliaryMCMC	2	19	19
129	KPEKTS	5	20	0
141	POET-X	57	57	0

Appendix

Table 18: Randomly selected papers for component ablations. IDs match Table 9 . Tasks are scored measurements associated with the listed experiments; CPU-only tasks are a subset of Tasks. The same 12 papers and 287 tasks are used in every condition.

Category	Findings	Papers	Distinct tasks
Hyperparameters	41	26	287
Objective	41	22	400
Data	40	20	298
Architecture	33	18	209
Evaluation procedure	30	20	261
Training procedure	22	16	176

Appendix

Table 19: Static paper–code contradiction findings by recorded category. Scope is determined from evidence, not category names. Distinct tasks and papers are deduplicated within a row but can overlap between rows; these columns must not be summed. Counts describe agent-reviewed records.

Reproducer	Raw	RRA	Gain (pp)	Raw	RRA	Reduction (%)
	Success rate (%)			API USD
Sonnet 5	75.6	92.2	+16.6	1,883	1,396	25.9
Gemini 3.8 Flash	59.7	91.2	+31.5	478	389	18.6

Appendix

Table 20: Development-set results from a single evaluation. Success and downstream Reproducer API spending on the ten papers used for workflow development, reported separately from test results. RRA denotes refined reproducible artifacts; gain is RRA minus Raw in percentage points, and API reduction is

100(\mathrm{Raw}-\mathrm{RRA})/\mathrm{Raw}

. Construction, auditing, and GPU costs are excluded.

Model	Tokens (B)	Reduction (%)	API USD	Reduction (%)
Sonnet 4.6	8.48 / 6.80	19.8	3,165 / 2,512	20.6
Sonnet 5	4.98 / 4.52	9.2	1,726 / 1,317	23.7
Opus 5	3.01 / 2.82	6.3	3,480 / 2,865	17.7
Gemini 3.8 Flash	2.24 / 1.83	18.3	392 / 326	16.8

Appendix

Table 21: Three-evaluation mean paired Reproducer costs from Table 3 , raw / RRA. Each model uses the same papers in both arms. Tokens include recorded input, output, cache writes/reads, and separately recorded thinking; USD uses frozen accounting rates. Construction, audit, and GPU costs are excluded. Reductions use the means.

ML Reproducibility

Machine learning reproducibility studies whether model results and scientific claims can be independently checked and reliably repeated.

See all 144 papers

ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues
Jun 16, 2026Shanda Li, Qiuhong Anna Wei, Jingwu Tang +5LLM Agent Evaluation AI Agent Benchmarks
School of Computer Science, Carnegie Mellon University · Datadog
RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?
Sep 23, 2026Mithil Salunkhe, Haochen Ding, Samridhi Verma +1LLM Agent Evaluation AI Agent Benchmarks
University of Illinois Urbana-Champaign · National Center for Supercomputing Applications
Coding-agents can replicate scientific machine learning papers
Jul 2, 2026Atharva Hans, Ilias BilionisCoding Agents LLM Agent Evaluation
School of Mechanical Engineering, Purdue University, West Lafayette, IN, 47907, USA

Jun 16, 2026J/K move · Enter open · S save

ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues

Shanda Li, Qiuhong Anna Wei, Jingwu Tang, Valerie Chen+4

School of Computer Science, Carnegie Mellon University · Datadog

LLM Agent Evaluation AI Agent Benchmarks ML Reproducibility Scientific Reproducibility Benchmark Reproducibility

Open paper AskPDF arXiv

Reproducing research results from papers and released code is central to scientific progress. Existing works have introduced benchmarks to evaluate whether LLM agents can assist with reproducibility, but they are difficult to scale due to their reliance on substantial manual effort for data curation and evaluation. We introduce ReproRepo, a scalable framework for reproducibility evaluation that leverages human-raised GitHub issues as naturally occurring supervision on realistic reproduction blockers. We instantiate ReproRepo on 1,149 recent machine learning papers from major conferences and evaluate four frontier model-agent configurations. Our results show that LLM agents, even without executing code, can identify many real-world reproducibility problems from paper-repository pairs: the best agent in our study, namely Codex with GPT-5.5, surfaces at least one semantically related human-reported blocker for

\sim

90% of papers in the study. Further analysis shows that agents are particularly effective for surfacing visible failures and identifying the right semantic region, but may still be insufficient in exact localization. ReproRepo can serve as a reusable, scalable framework for future evaluations of LLM agents on real-world reproducibility auditing. Our code is released at https://github.com/LithiumDA/ReproRepo.

Abstract

ML Reproducibility

Explore similar work

ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues

RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?

Coding-agents can replicate scientific machine learning papers