Organizations: School of Software, Shandong University China · Beijing Institute of Technology China · Zhejiang University China · Dept. of Math & Statistics, Boston University USA · College of AI, Tsinghua University China · Alibaba Group China · The Hong Kong Polytechnic University Hong Kong SAR, China
Automated paper reproduction -- generating executable code from academic papers -- is bottlenecked not by information retrieval but by the tacit knowledge that papers inevitably leave implicit. We formalize this challenge as the progressive recovery of three types of tacit knowledge -- relational, somatic, and collective -- and propose \method, a graph-based agent framework with a dedicated mechanism for each: node-level relation-aware aggregation recovers relational knowledge by analyzing implementation-unit-level reuse and adaptation relationships between the target paper and its citation neighbors; execution-feedback refinement recovers somatic knowledge through iterative debugging driven by runtime signals; and graph-level knowledge induction distills collective knowledge from clusters of papers sharing similar implementations. On an extended ReproduceBench spanning 3 domains, 10 tasks, and 40 recent papers, \method{} achieves an average performance gap of 10.04% against official implementations, improving over the strongest baseline by 24.68%. The code will be publicly released upon acceptance; the repository link will be provided in the final version.
Figures & tables
Figure 1. Illustrative examples of the three types of tacit knowledge in automated paper reproduction. Left : Relational—implementation units are reused, adapted, or newly created relative to neighbor papers. Middle : Somatic—execution feedback from a sandbox environment guides iterative refinement. Right : Collective—common practices are induced from clusters of related implementations.
Figure 2. Overview of PaperRepro . The framework comprises three stages: SSGP prunes the citation graph to retain implementation-relevant neighbors; node-level relation-aware aggregation analyzes reuse/adapt/new relationships and assembles an initial implementation; execution-feedback refinement and graph-level knowledge induction iteratively improve it via runtime debugging and community-induced knowledge.
Figure 3. Node-Level and Graph-Level Graph Reasoning. Left (Node-Level Relation-Aware Aggregation) : For each neighbor vi∈Nprune(vt) , relation analysis produces a structured annotation Γt,i that categorizes implementation units into directly reusable ( Ureuse ), adaptable ( Uadapt ), and new ( Unew ); neighborhood aggregation selects among competing candidates by priority p=w(vt,vi)+β⋅1[reuse] to construct Cvt(init) . Right (Graph-Level Knowledge Induction) : The paper graph is partitioned into subgraphs {Gj(t)} via Louvain clustering on SSGP edge weights; within each subgraph, the agent reproduces member papers, performs cross-paper induction over execution feedback, and writes recurring patterns into the subgraph knowledge base Kj(t) .
Domain
Task
ReAct
OpenHands
Paper2Code
DeepCode
AutoReproduce
Ours
MMRec
81.66 ± 12.3
67.44 ± 14.7
70.13 ± 15.2
64.58 ± 13.4
36.42 ± 8.7
20.17 ± 4.6
GeneralRec
77.85 ± 14.1
58.77 ± 11.6
32.11 ± 12.4
28.66 ± 10.8
28.34 ± 7.2
8.77 ± 2.4
RecSys
SeqRec
80.33 ± 13.5
71.09 ± 16.2
65.88 ± 14.8
61.24 ± 12.1
34.76 ± 9.1
3.38 ± 1.9
Classification
78.57 ± 15.4
62.18 ± 12.9
68.44 ± 16.1
63.11 ± 13.7
38.23 ± 8.4
5.03 ± 2.6
LongTerm
82.11 ± 11.8
75.33 ± 14.3
37.66 ± 10.5
34.22 ± 9.2
31.87 ± 7.6
3.27 ± 1.7
ShortTerm
76.88 ± 16.7
55.44 ± 13.2
64.79 ± 14.6
60.13 ± 12.4
35.67 ± 8.9
6.01 ± 3.5
Table 1. Performance Gap ( ↓ ) across three domains and ten task types. Lower values indicate better reproduction quality. Results are averaged over 5 independent runs; we report mean ± std.
Method
Param.
Exp.
Avg.
κ
Paper2Code
4.67
4.23
5.12
4.67
0.71
DeepCode
5.12
4.89
5.56
5.19
0.68
AutoReproduce
6.23
5.78
6.67
6.23
0.72
Ours
8.45
8.12
8.89
8.49
0.76
Table 2. Human evaluation results ( ↑ ). Three domain experts independently assess each reproduction across three dimensions (1–10 scale). We report mean and Fleiss’ κ .
SSGP
Node
Exec.
Graph
RecSys
TimeSeries
∘
✔
✔
✔
44.33
47.66
✔
∘
✔
✔
21.77
25.44
✔
✔
∘
✔
49.55
52.33
✔
✔
✔
∘
36.88
38.66
✔
✔
✔
✔
10.77
9.94
Table 3. Ablation study on key components. Performance Gap ( ↓ ).
Figure 4. Cross-model transferability of Kcol . The induced knowledge base yields consistent gains when transferred across different backbone LLMs.
Figure 5. Subgraph partitioning comparison. Our edge-weight-based partitioning vs. random partitioning across training epochs.
Figure 6. Knowledge base evolution across training epochs. Induced knowledge progresses from engineering rules to training tricks to methodological insights.
Figure 7. Code-source breakdown. Proportions of reused, adapted, and newly generated code across tasks.
Figure 8. Neighborhood aggregation case study. The priority score selects FREEDOM’s reuse candidate over LATTICE’s adapted variant for PGL’s kNN graph construction.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9. SSGP analysis. (a) Input ablation: Method+Experiments achieves optimal performance across all tasks. (b) Sensitivity to Kkeep : difficult tasks require smaller K for precise pruning.
Paper-to-code reproduction asks scientific AI agents to turn research papers into executable repositories that preserve the paper's method, protocol and artifacts. This is difficult because the specification is split: explicit paper content such as algorithms, metrics and artifacts is often lost across long agent trajectories, while implicit details such as framework defaults and conventions inherited from related work are absent from the paper. We introduce ReproAgent, a four-stage Prepare--Plan--Generate--Repair pipeline built around a persistent implementation contract with two channels: an implementation-requirement channel that turns paper snippets into code obligations, and a reference-evidence channel that retrieves content and structure evidence from related repositories. Both are bound to work packages, projected into file-level contracts, and consumed across generation and repair. On PaperBench Code-Dev, ReproAgent reaches the highest mean score among same-backbone scaffolds under both Claude-Sonnet-4.5 and Gemini-3-Flash. End-to-end channel ablations and per-paper cases support the contribution of both channels. Code and experimental artifacts are publicly available.
Xue Hu, Zewei Pan, Zhongyuan Wang +3
Beihang University · Shanghai Jiao Tong University · Peking University +3
Scientific machine learning papers typically make computational claims, e.g., that the relative mean square error is less than 5% or that the 95% predictive credible interval covers the test data. A coding agent can be prompted to replicate those claims from paper materials alone, but the prompt does not by itself reliably preserve progress or check whether generated evidence supports the paper's claims. We introduce Paper-replication, a workflow that makes each selected paper claim a target with recorded evidence, and implement it as a coding-agent skill. The workflow makes the agent record those targets, reconstruct the paper's method, run computational experiments, link generated outputs to provenance and comparisons with the paper's claims, record where matched evidence appears in the replication report, and pass validation checks before completion. We evaluate Paper-replication on twelve independent runs across four scientific machine learning papers. All twelve workspaces pass the completion gate, and all 158 recorded targets are matched with report coverage. Even in this completed workspace state, repeated runs differ in how papers are divided into targets, in numerical fidelity to the source papers, in elapsed replication time, in the number of intermediate executions replaced before final evidence is accepted, and in the rules used to accept evidence. Paper-replication makes completion depend on workspace evidence and validation checks rather than on the agent's final message.
Atharva Hans, Ilias Bilionis
School of Mechanical Engineering, Purdue University, West Lafayette, IN, 47907, USA
Reproducing a machine learning paper involves most research steps, from installing software and debugging to running experiments, work that AI agents increasingly do. We introduce RECLAIM, a benchmark of 100 NeurIPS 2025 papers that can be rebuilt yearly from new conferences. For each paper we fix in advance the result to reproduce, what counts as a successful reproduction, and a GPU-hour budget. An agent must reproduce that result using the paper and whatever its authors released. What the authors released decides the difficulty tier. Run-tier releases include code, data, and weights; Retrain-tier releases lack weights, so the agent trains the model; Reimplement-tier releases lack code, so the agent writes it. A separate language model grades runs from logs and outputs rather than agents' reports. We run four agents once per paper; the best agent in each tier reproduces only 41% of Run-tier papers, 27% at Retrain, and 15% at Reimplement, where every agent does worst. Failed attempts use on average 29% of their budget, so most stop with budget left. The most common agent error is writing the method without checking any part against the paper's numbers, in 63 of 400 runs.
Mithil Salunkhe, Haochen Ding, Samridhi Verma +1
University of Illinois Urbana-Champaign · National Center for Supercomputing Applications