cs.AISep 23, 2026

RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?

Authors: Mithil Salunkhe, Haochen Ding, Samridhi Verma, Volodymyr Kindratenko

Organizations: University of Illinois Urbana-Champaign · National Center for Supercomputing Applications

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Reproducing a machine learning paper involves most research steps, from installing software and debugging to running experiments, work that AI agents increasingly do. We introduce RECLAIM, a benchmark of 100 NeurIPS 2025 papers that can be rebuilt yearly from new conferences. For each paper we fix in advance the result to reproduce, what counts as a successful reproduction, and a GPU-hour budget. An agent must reproduce that result using the paper and whatever its authors released. What the authors released decides the difficulty tier. Run-tier releases include code, data, and weights; Retrain-tier releases lack weights, so the agent trains the model; Reimplement-tier releases lack code, so the agent writes it. A separate language model grades runs from logs and outputs rather than agents' reports. We run four agents once per paper; the best agent in each tier reproduces only 41% of Run-tier papers, 27% at Retrain, and 15% at Reimplement, where every agent does worst. Failed attempts use on average 29% of their budget, so most stop with budget left. The most common agent error is writing the method without checking any part against the paper's numbers, in 63 of 400 runs.

Figures & tables

Figure 1: Overview of RECLAIM and its evaluation pipeline. The figure shows an example run with Qwen3.6-27B on the TTPL paper ( Chen et al., 2025d ) .
Figure 2: Failure counts for the two weakest and two strongest agents. The top four failures become rarer with stronger agents.
Figure 3: Three stages narrow a 1,000-paper sample to the released 100-paper benchmark, with every availability call and compute estimate manually audited.
Figure 4: Reproduction rate and mean audit score by agent and tier, and the share of the granted H100-hours each agent spent.
Figure 5: Primary failure mode by tier over the 400 cells, in the order of Appendix G .
Figure 6: Mean audit score and share of granted GPU-hours spent, by the paper’s compute band.
Appendix figures & tables50 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Round 97 of MiniMax-M2.7 on FairImagen ( Fu et al., 2025 ) as the event transcript records it, from the reasoning to the metered run_gpu result.
ToolFunction
workspace_bashBash with the episode workspace as the working directory, on the node that runs the harness, with no GPU passed through. Cloning, installs, edits, inspection. Timeout default 900 s, maximum 3,600 s. Returns stdout, stderr, exit code, and duration.
write_fileWrite a UTF-8 file under the workspace or evidence roots, up to 200,000 characters, overwriting an existing file at that path.
edit_fileReplace one exact string in an existing file. The match must be unique unless replace_all is set. A bare repo-relative path is located by suffix search when it resolves to exactly one file.
update_planReplace the agent’s ordered checklist of steps with statuses pending , in_progress , or completed . At most one step may be in_progress .
fetch_urlFetch a public http(s) URL and reduce HTML to text, default 24,000 characters.
list_partitionsRun sinfo and return each Slurm partition’s node counts by state, walltime limit, and GPU resources, plus the profile default. Read-only.
Appendix
Table 1: The seven tools available to the model, with the code defaults and caps each enforces.
FieldContent
paper_idThe paper’s arXiv ID.
claimThe claim the agent worked toward, in the agent’s own words.
what_ranWhat was executed, from the environment build to the run.
scoring_commandOne command that reproduces the measured number from a clean state. Empty when nothing ran.
measurementsArray of measured metrics. Each entry records the metric name with units, the observed value, the paper’s reference value or null , the scope the value was measured over, and an array of citations into evidence/ supporting the observed value.
agent_assessmentOne of reproduced , partial , not_reproduced , could_not_run .
Appendix
Table 2: The nine fields of report.json , all required by the schema.
SettingDeepSeek-V4-FlashQwen3.6-27BMiniMax-M2.7Muse Spark 1.2
Served checkpointdeepseek-ai/DeepSeek-V4-Flash-0731Qwen/Qwen3.6-27B-FP8cyankiwi/MiniMax-M2.7-AWQ-4bitmuse-spark-1.2-contributor
Advertised namesame as checkpointsame as checkpointMiniMaxAI/MiniMax-M2.7same as checkpoint
Precisionnative INT8/FP8FP84-bit AWQ (W4A16)vendor-served
KV cache dtypefp8fp8not setvendor-served
GPUs per replica212vendor-served
Tensor parallel212vendor-served
Appendix
Table 3: Served configuration of the four agent models, from the tier launch scripts and the serve profiles.
Figure 8: Prompt, cached prompt, and completion tokens per labeled run by agent on a shared log axis, one tick per run, with a bar over the middle half of runs and the median in black.
Figure 9: Cost per labeled run by agent at the rates of Table 4 , with the median ticked.
Figure 10: Token cost against GPU cost per labeled run by agent. The dashed line marks equal cost.
 Rate ($/M)Tokens (M)Cost ($) 
AgentPromptCachedCompletionPromptCompletionTokensGPUTotalPer run
DeepSeek-V4-Flash0.140.00280.281,077.911.1115.041,337.641,352.6813.66
Qwen3.6-27B0.45–2.70489.16.24236.93797.631,034.5610.45
MiniMax-M2.70.300.061.20573.86.6282.21528.61610.826.17
Muse Spark 1.21.250.154.251,494.633.53724.911,112.341,837.2418.56
All agents   3,635.457.501,059.083,776.224,835.3012.21
Appendix
Table 4: Token usage and cost of the 396 runs that started.
  By tierBy band
AgentTier or measureReproducedMean score8 h32 h96 h
DeepSeek-V4-FlashRun14/34 (41%)5.29   
 Retrain9/33 (27%)5.45   
 Reimplement4/33 (12%)4.64   
 All tiers27/100 (27%)5.13   
 Mean score  6.004.103.33
Appendix
Table 5: Reproduced cells, mean audit score, and grant spent over all 400 agent-paper cells, by agent, tier, and compute band.
ToolFunction and limits
list_run_filesLists files and directories in the run bundle, optionally recursive. At most 200 entries, flagged when truncated. Prunes the .git , pycache , .venv , and node_modules subtrees.
read_run_fileReads one text file by relative path. Default cap 40,000 characters, maximum 200,000, flagged when truncated.
bashRuns one shell command with the run bundle as the working directory. Timeout 60 s, after which the whole process group is killed. Returns stdout, stderr, and the exit code.
write_run_fileWrites a new text file of up to 200,000 characters into the bundle, such as a re-scoring script to run with bash and cite. Refuses to overwrite an existing file, so agent artifacts stay intact.
Appendix
Table 6: The four run-directory tools available to the auditor, with the function and limits of each.
Figure 11: Auditor tool rounds per labeled run, by verdict. The dashed line marks the median.
ScoreBandConditionsVerdict
10Faithful, exactRight experiment executed, C1 criterion met, no flags, correct protocol, split, and scale; a stochastic metric seed-fixed or averaged.reproduced
9Faithful, single drawAs 10, with one unseeded draw of a stochastic metric that still lands inside the bar.reproduced
8Minor caveatsC1 criterion met with sound provenance; low-severity flags only, or one medium flag whose evidence shows the deviation does not change the measured quantity. The bar is never loosened at grade time.reproduced
7Near-reproductionThe authors’ own pipeline, clean provenance, honest, and the result lands just outside the bar, or inside it with an unresolved protocol deviation, or all but one arm of a multi-part claim is met.partial
6Clear partialThe right quantity executed with sound provenance, and the result clearly misses the bar, or the pinned bar is mis-specified and the agent reproduced the reproducible sibling quantity.partial
5Honest off-targetNo authors’ evaluation pipeline existed, so the agent reconstructed the protocol in good faith, ran the whole evaluation, disclosed the result, and it diverged from the target.not_reproduced
Appendix
Table 7: The frozen audit rubric, with the conditions for each band and the verdict derived from each score.
Figure 12: Audit scores, colored by the verdict bands of Table 7 .
Figure 13: Anti-cheat flags the auditor raised on the labeled runs, by kind and severity, with the runs each kind disqualified through a high-severity flag.
Figure 14: The verdict derivation of the code in Figure . The integrity checks run first, and the score then maps to a band, colored as in Figure 12 .
TierSelected0–88–3232–96>> 96FlaggedRevisedΣ\Sigma H100-h
Run ( Easy )6713351451461,961
Retrain ( Medium )6713391142231,754
Reimplement ( Hard )6624231181442,190
Total2005097361750135,905
Appendix
Table 8: The 200-paper audit pool by tier and pre-audit H100-hour band, with rows flagged for compute review, rows the automatic arithmetic check revised, and summed audited compute.
Figure 15: The audited compute of the 100 evaluation papers, one dot per paper on a log axis, with a tick at each tier’s median and papers below the axis floor in the leftmost slot.
Figure 16: The 100 evaluation papers by the month of their arXiv identifier, stacked by tier.
 Evaluation splitDevelopment split
TierPapers0–88–3232–96Audited H100-hPapers
Run342770142.715
Retrain3316134380.914
Reimplement331698630.185
All1005929121,153.814
Appendix
Table 9: Composition of the two splits by tier and compute band, with papers binned by the audited H100-hour estimate.
FieldTypeDescription
Top-level fields
custom_idstringarXiv ID, the primary key.
central_claimstringThe paper’s central empirical claim in one sentence.
claim_evidencestringQuoted paper text grounding the claim.
mre_configstringProse recipe for the cheapest configuration that tests the claim, withheld from the agent.
agent_taskstringClassifier-written reproduction instruction, released with the row and shown to neither the agent nor the auditor.
Appendix
Table 10: The fields of a dataset row.
arXivTierBandH100-hMetricBarTarget value
2502.05795Run0–81.00Validation perplexity…point estimate25.76
2502.06684Run0–8<< 0.01Mean accuracy (%) on 10…point estimate77.0 +/- 1.8%
2503.18430Run0–80.64Average Precision (AP)point estimate46.3%
2503.23035Run0–80.25PSNR (dB)point estimate27.69
2504.20571Run0–80.16MATH500 accuracy (%)point estimate73.6%
2505.10978Run0–82.00Success rate (%)point estimate86.7%
Appendix
Table 11: The 100 evaluation papers, one row each, sorted by tier, then compute band, then arXiv ID. Metric names and target values longer than 28 characters are cut and closed with an ellipsis.
arXivTierBandH100-hMetricBarTarget value
2505.18513Run0–80.01LDS (Linear Datamodeling…point estimate21.11
2506.09045Run0–8<< 0.01end-to-end latency speeduppoint estimate2.68x
2507.02546Run0–80.01Metric point map relative…threshold4.44%
2510.21323Run0–80.01Unified concept extraction…point estimateUnified concept set shared…
2511.07099Run0–80.01Speaker similarity score…point estimate0.113
2502.06067Retrain0–82.0095% CI coverage ratethreshold1.0 (100%)
Appendix
Table 12: The 14 development papers, in the same columns, sort order, and truncation as Table 11 . This split is disjoint from the evaluation set and is used for harness development only.
ModeDeepSeek-V4-FlashQwen3.6-27BMiniMax-M2.7Muse Spark 1.2All
Reproduced2713102272
Ran, outside tolerance261672776
Reimplemented but did not check the result131627763
Build and dependency failures1811424
Wrong artifact measured64131538
Wrong experiment911161046
Appendix
Table 13: Primary failure mode by agent.
ModeDefinitionEvidence signatureRubric anchor
reproduced-clean ReproducedThe agent runs the pinned experiment, using the released artifacts the tier provides, and the graded number lands inside the pinned bar.A number the run computed, traced to a command in the transcript that ran the experiment on the released artifacts the tier provides.8–10
near-miss-partial Ran, outside toleranceThe agent runs the right pipeline and measures a number that misses the bar.A completed run of the pinned quantity with clean provenance, and a measured value on the wrong side of the bar.6–7
reimplement-without-validating Reimplemented but did not check the resultThe agent writes its own version of the method, or part of it, and never checks any stage against a reference it did not produce.A component written from scratch in the run’s own code, and a reference number the paper supplies, most often its baseline in the same table as the target, that the transcript never compares against.4
environment-fights Build and dependency failuresMost of the run goes to build and dependency failures in the released stack, and the experiment never produces a valid number.Repeated install, compile, and binary-compatibility failures against the authors’ pinned versions, with the same packages failing across runs on the same host.2
artifact-provenance-mismatch Wrong artifact measuredThe agent measures a different checkpoint, dataset, split, or scale from the one the dataset row names.A downloaded artifact whose model card, README, or example disagrees with the dataset row, and a measurement taken on it. The wrong_split_scale_dataset flag marks the same break.0 with a high flag
scope-substitution Wrong experimentThe agent runs a different experiment from the one the pinned claim is about.A round where the agent names the pinned arm and picks a different one, and a graded number from the substitute.0 or 2
Appendix
Table 14: The nine failure modes, one primary mode per run. The rubric anchor is the band whose conditions match the mode’s evidence signature, and scores spread around it.
Figure 17: The 36 runs the two relabel passes re-read, from the label each carried to its mode of record, with ribbons in the color of the destination mode and widths in runs.
Figure 18: Primary failure mode by agent and tier over the 400 cells, each bar normalized to its sweep’s cells, in the order and colors of Figure 5 .
Figure 19: Rounds used and share of the H100 grant spent at exit, one tick per labeled run, with each mode’s median as a tick. Never launched an experiment and Echoed a shipped number exit earliest.
Figure 20: Run card for DeepSeek-V4-Flash on TabDPT (arXiv 2410.18164) at the Run tier, with the decisive round 53 in gold.
Figure 21: Activity of the ten annotated runs by round, colored by the kind of tool call, with the annotated episode, its decisive round, and the first graded-pipeline launch marked.
Figure 22: Decisive round 53 of DeepSeek-V4-Flash on TabDPT (arXiv 2410.18164), where the agent picks the fix, and round 77, where the patched script runs on the GPU.
Figure 23: Run card for DeepSeek-V4-Flash on LayerNorm Scaling (arXiv 2502.05795) at the Run tier, with the decisive round 22 in gold.
Figure 24: Decisive round 22 of DeepSeek-V4-Flash on LayerNorm Scaling (arXiv 2502.05795), with the agent’s reasoning, the tool call, and its result.
Figure 25: Run card for Muse Spark 1.2 on TAMI (arXiv 2510.23577) at the Retrain tier, with the decisive round 31 in gold.
Figure 26: Decisive round 31 of Muse Spark 1.2 on TAMI (arXiv 2510.23577), with the one-line opener the transcript carries in place of reasoning, the tool call, and its result.
Figure 27: Run card for DeepSeek-V4-Flash on Default MoE (arXiv 2504.12463) at the Retrain tier, with the decisive round 91 in gold.
Figure 28: Decisive round 91 of DeepSeek-V4-Flash on Default MoE (arXiv 2504.12463), with the agent’s reasoning, the tool call, and its result.
Figure 29: Run card for Muse Spark 1.2 on Toto (arXiv 2505.14766) at the Run tier, with the decisive round 25 in gold.
Figure 30: Round 24 of Muse Spark 1.2 on Toto (arXiv 2505.14766), the editable install of the authors’ repository, and the decisive round 25 where the probe reports a CPU-only torch.
Figure 31: Run card for DeepSeek-V4-Flash on CQ-DINO (arXiv 2503.18430) at the Run tier, with the decisive round 91 in gold.
Figure 32: Decisive round 91 of DeepSeek-V4-Flash on CQ-DINO (arXiv 2503.18430), with the agent’s reasoning, the tool call, and its result.
Figure 33: Run card for MiniMax-M2.7 on SemCoT (arXiv 2510.24940) at the Retrain tier, with the decisive round 12 in gold.
Figure 34: Decisive round 12 of MiniMax-M2.7 on SemCoT (arXiv 2510.24940), with the agent’s reasoning, the tool call, and its result.
Figure 35: Run card for Qwen3.6-27B on TTPL (arXiv 2506.02392) at the Run tier, with the decisive round 36 in gold.
Figure 36: Decisive round 36 of Qwen3.6-27B on TTPL (arXiv 2506.02392), with the agent’s reasoning, the tool call, and its result.
Figure 37: Run card for MiniMax-M2.7 on Neural Block Linearization (arXiv 2505.21077) at the Retrain tier, with the decisive round 99 in gold.
Figure 38: Round 96 of MiniMax-M2.7 on Neural Block Linearization (arXiv 2505.21077), the layer-skip benchmark, and the decisive round 99 with the file the agent wrote.
Figure 39: Run card for Muse Spark 1.2 on DLRT-HeavyBall (arXiv 2506.17475) at the Reimplement tier, with the decisive round 34 in gold.
Figure 40: Decisive round 34 of Muse Spark 1.2 on DLRT-HeavyBall (arXiv 2506.17475), with the update rule the agent wrote from the paper, and the round-37 run with torch SGD.
Figure 41: Compute charged and wall-clock elapsed by round for the ten annotated runs, each as a share of the run’s total, with the decisive round and the first graded-pipeline launch marked.
Figure 42: The auditor’s session on each of the ten annotated runs, one cell per tool call in order and colored by what it opened or ran, with the pinned verdict and the flags it raised.

Explore similar work

CardsList

  1. Coding-agents can replicate scientific machine learning papers

    Jul 2, 2026Atharva Hans, Ilias BilionisReproducibilityCoding Agents

    School of Mechanical Engineering, Purdue University, West Lafayette, IN, 47907, USA

  2. ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues

    Jun 16, 2026Shanda Li, Qiuhong Anna Wei, Jingwu Tang +5ReproducibilityResearch Automation

    School of Computer Science, Carnegie Mellon University · Datadog

  3. Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results

    Apr 23, 2026Benjamin Kohler, David Zollikofer, Johanna Einsiedler +2ReproducibilityComputational Social Science

    ETH Zurich · University of Basel