What Papers Don't Tell You: Recovering Tacit Knowledge for Automated Paper Reproduction

Lehui Li; Ruining Wang; Haochen Song; Yaoxin Mao; Tong Zhang; Yuyao Wang; Jiayi Fan; Yitong Zhang

cs.AIMar 2, 2026

What Papers Don't Tell You: Recovering Tacit Knowledge for Automated Paper Reproduction

Authors: Lehui Li, Ruining Wang, Haochen Song, Yaoxin Mao, Tong Zhang, Yuyao Wang, Jiayi Fan, Yitong Zhang,

Organizations: School of Software, Shandong University China · Beijing Institute of Technology China · Zhejiang University China · Dept. of Math & Statistics, Boston University USA ·

Abstract

Automated paper reproduction -- generating executable code from academic papers -- is bottlenecked not by information retrieval but by the tacit knowledge that papers inevitably leave implicit. We formalize this challenge as the progressive recovery of three types of tacit knowledge -- relational, somatic, and collective -- and propose \method, a graph-based agent framework with a dedicated mechanism for each: node-level relation-aware aggregation recovers relational knowledge by analyzing implementation-unit-level reuse and adaptation relationships between the target paper and its citation neighbors; execution-feedback refinement recovers somatic knowledge through iterative debugging driven by runtime signals; and graph-level knowledge induction distills collective knowledge from clusters of papers sharing similar implementations. On an extended ReproduceBench spanning 3 domains, 10 tasks, and 40 recent papers, \method{} achieves an average performance gap of 10.04% against official implementations, improving over the strongest baseline by 24.68%. The code will be publicly released upon acceptance; the repository link will be provided in the final version.

Figures & tables

Figure 1. Illustrative examples of the three types of tacit knowledge in automated paper reproduction. Left : Relational—implementation units are reused, adapted, or newly created relative to neighbor papers. Middle : Somatic—execution feedback from a sandbox environment guides iterative refinement. Right : Collective—common practices are induced from clusters of related implementations.

Figure 2. Overview of PaperRepro . The framework comprises three stages: SSGP prunes the citation graph to retain implementation-relevant neighbors; node-level relation-aware aggregation analyzes reuse/adapt/new relationships and assembles an initial implementation; execution-feedback refinement and graph-level knowledge induction iteratively improve it via runtime debugging and community-induced knowledge.

Figure 3. Node-Level and Graph-Level Graph Reasoning. Left (Node-Level Relation-Aware Aggregation) : For each neighbor $v_{i}\in\mathcal{N}_{\text{prune}}(v_{t})$ , relation analysis produces a structured annotation $\Gamma_{t,i}$ that categorizes implementation units into directly reusable ( $\mathcal{U}^{\mathrm{reuse}}$ ), adaptable ( $\mathcal{U}^{\mathrm{adapt}}$ ), and new ( $\mathcal{U}^{\mathrm{new}}$ ); neighborhood aggregation selects among competing candidates by priority $p=w(v_{t},v_{i})+\beta\cdot\mathbb{1}[\text{reuse}]$ to construct $\mathcal{C}_{v_{t}}^{(\mathrm{init})}$ . Right (Graph-Level Knowledge Induction) : The paper graph is partitioned into subgraphs $\{\mathcal{G}^{(t)}_{j}\}$ via Louvain clustering on SSGP edge weights; within each subgraph, the agent reproduces member papers, performs cross-paper induction over execution feedback, and writes recurring patterns into the subgraph knowledge base $\mathcal{K}^{(t)}_{j}$ .

Domain	Task	ReAct	OpenHands	Paper2Code	DeepCode	AutoReproduce	Ours
	MMRec	81.66 $\pm$ 12.3	67.44 $\pm$ 14.7	70.13 $\pm$ 15.2	64.58 $\pm$ 13.4	36.42 $\pm$ 8.7	20.17 $\pm$ 4.6
	GeneralRec	77.85 $\pm$ 14.1	58.77 $\pm$ 11.6	32.11 $\pm$ 12.4	28.66 $\pm$ 10.8	28.34 $\pm$ 7.2	8.77 $\pm$ 2.4
RecSys	SeqRec	80.33 $\pm$ 13.5	71.09 $\pm$ 16.2	65.88 $\pm$ 14.8	61.24 $\pm$ 12.1	34.76 $\pm$ 9.1	3.38 $\pm$ 1.9
	Classification	78.57 $\pm$ 15.4	62.18 $\pm$ 12.9	68.44 $\pm$ 16.1	63.11 $\pm$ 13.7	38.23 $\pm$ 8.4	5.03 $\pm$ 2.6
	LongTerm	82.11 $\pm$ 11.8	75.33 $\pm$ 14.3	37.66 $\pm$ 10.5	34.22 $\pm$ 9.2	31.87 $\pm$ 7.6	3.27 $\pm$ 1.7
	ShortTerm	76.88 $\pm$ 16.7	55.44 $\pm$ 13.2	64.79 $\pm$ 14.6	60.13 $\pm$ 12.4	35.67 $\pm$ 8.9	6.01 $\pm$ 3.5

Table 1. Performance Gap (

\downarrow

) across three domains and ten task types. Lower values indicate better reproduction quality. Results are averaged over 5 independent runs; we report mean

\pm

std.

	Method	Param.	Exp.	Avg.	$\kappa$
Paper2Code	4.67	4.23	5.12	4.67	0.71
DeepCode	5.12	4.89	5.56	5.19	0.68
AutoReproduce	6.23	5.78	6.67	6.23	0.72
Ours	8.45	8.12	8.89	8.49	0.76

Table 2. Human evaluation results (

\uparrow

). Three domain experts independently assess each reproduction across three dimensions (1–10 scale). We report mean and Fleiss’

\kappa

SSGP	Node	Exec.	Graph	RecSys	TimeSeries
$\circ$	✔	✔	✔	44.33	47.66
✔	$\circ$	✔	✔	21.77	25.44
✔	✔	$\circ$	✔	49.55	52.33
✔	✔	✔	$\circ$	36.88	38.66
✔	✔	✔	✔	10.77	9.94

Table 3. Ablation study on key components. Performance Gap (

\downarrow

Figure 4. Cross-model transferability of $\mathcal{K}_{\mathrm{col}}$ . The induced knowledge base yields consistent gains when transferred across different backbone LLMs.

Figure 5. Subgraph partitioning comparison. Our edge-weight-based partitioning vs. random partitioning across training epochs.

Figure 6. Knowledge base evolution across training epochs. Induced knowledge progresses from engineering rules to training tricks to methodological insights.

Figure 7. Code-source breakdown. Proportions of reused, adapted, and newly generated code across tasks.

Panel of Figure 8 — Figure 8. Neighborhood aggregation case study. The priority score selects FREEDOM’s reuse candidate over LATTICE’s adapted variant for PGL’s kNN graph construction.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9. SSGP analysis. (a) Input ablation: Method+Experiments achieves optimal performance across all tasks. (b) Sensitivity to $K_{\text{keep}}$ : difficult tasks require smaller $K$ for precise pruning.

Figure 10. Expert annotation validation. Comparison between expert consensus (left) and our semantic pruning (right). Red: target paper; Purple: critical neighbors; Orange: relevant neighbors. Our method retains all expert-identified critical nodes.

Scientific Code Generation

Scientific code generation develops systems that turn scientific questions, papers, and workflows into executable, scientifically valid programs.

See all 52 papers

ReproAgent: Contract-Guided Paper-to-Code Reproduction
Aug 25, 2026Xue Hu, Zewei Pan, Zhongyuan Wang +3Scientific Code Generation Agentic Code Generation
Beihang University · Shanghai Jiao Tong University · Peking University +3
Coding-agents can replicate scientific machine learning papers
Jul 2, 2026Atharva Hans, Ilias BilionisCoding Agents LLM Agent Evaluation
School of Mechanical Engineering, Purdue University, West Lafayette, IN, 47907, USA
RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?
Sep 23, 2026Mithil Salunkhe, Haochen Ding, Samridhi Verma +1LLM Agent Evaluation AI Agent Benchmarks
University of Illinois Urbana-Champaign · National Center for Supercomputing Applications

Aug 25, 2026J/K move · Enter open · S save

ReproAgent: Contract-Guided Paper-to-Code Reproduction

Xue Hu, Zewei Pan, Zhongyuan Wang, Zhou Liu+2

Beihang University · Shanghai Jiao Tong University · Peking University+3

Scientific Code Generation Agentic Code Generation Code Generation

Open paper AskPDF arXiv

Paper-to-code reproduction asks scientific AI agents to turn research papers into executable repositories that preserve the paper's method, protocol and artifacts. This is difficult because the specification is split: explicit paper content such as algorithms, metrics and artifacts is often lost across long agent trajectories, while implicit details such as framework defaults and conventions inherited from related work are absent from the paper. We introduce ReproAgent, a four-stage Prepare--Plan--Generate--Repair pipeline built around a persistent implementation contract with two channels: an implementation-requirement channel that turns paper snippets into code obligations, and a reference-evidence channel that retrieves content and structure evidence from related repositories. Both are bound to work packages, projected into file-level contracts, and consumed across generation and repair. On PaperBench Code-Dev, ReproAgent reaches the highest mean score among same-backbone scaffolds under both Claude-Sonnet-4.5 and Gemini-3-Flash. End-to-end channel ablations and per-paper cases support the contribution of both channels. Code and experimental artifacts are publicly available.

Abstract

Scientific Code Generation

Explore similar work

ReproAgent: Contract-Guided Paper-to-Code Reproduction

Coding-agents can replicate scientific machine learning papers

RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?