ReproAgent: Contract-Guided Paper-to-Code Reproduction

Xue Hu; Zewei Pan; Zhongyuan Wang; Zhou Liu; Zeli Su; Wentao Zhang

cs.AIAug 25, 2026

ReproAgent: Contract-Guided Paper-to-Code Reproduction

Authors: Xue Hu, Zewei Pan, Zhongyuan Wang, Zhou Liu, Zeli Su, Wentao Zhang

Organizations: Beihang University · Shanghai Jiao Tong University · Peking University · Minzu University of China · Zhongguancun Academy · Beijing Key Laboratory of Data Intelligence and Security (Peking University)

Abstract

Paper-to-code reproduction asks scientific AI agents to turn research papers into executable repositories that preserve the paper's method, protocol and artifacts. This is difficult because the specification is split: explicit paper content such as algorithms, metrics and artifacts is often lost across long agent trajectories, while implicit details such as framework defaults and conventions inherited from related work are absent from the paper. We introduce ReproAgent, a four-stage Prepare--Plan--Generate--Repair pipeline built around a persistent implementation contract with two channels: an implementation-requirement channel that turns paper snippets into code obligations, and a reference-evidence channel that retrieves content and structure evidence from related repositories. Both are bound to work packages, projected into file-level contracts, and consumed across generation and repair. On PaperBench Code-Dev, ReproAgent reaches the highest mean score among same-backbone scaffolds under both Claude-Sonnet-4.5 and Gemini-3-Flash. End-to-end channel ablations and per-paper cases support the contribution of both channels. Code and experimental artifacts are publicly available.

Figures & tables

Figure 1: Overview of ReproAgent . A four-stage pipeline, Prepare, Plan, Generate, and Repair, carries an implementation-requirement channel and a reference-evidence channel into a shared file-level implementation contract. The implementation-requirement channel preserves paper-derived code obligations across generation and repair, while the reference-evidence channel supplies package-local content and structure evidence from related repositories.

Figure 2: ReproAgent overview. The implementation-requirement channel extracts code-facing obligations from the target paper, while the reference-evidence channel retrieves content and structure evidence from related repositories. The shared contract binds both channels into work packages, global and file-level contracts, generation, and validation, yielding a repository that is checked for both requirement fidelity and runtime correctness. Representative unit and evidence records appear in Appendix A .

Figure 3: Full-suite PaperBench Code-Dev comparison across reported paper-to-code systems.

Paper	BasicAgent	IterAgent	DeepCode	ReproAgent (ours)
Adaptive Pruning	16.6	22.4 $\uparrow$ 5.8	54.4 $\uparrow$ 37.8	67.8 $\uparrow$ 51.2
All in One	24.8	11.5 $\downarrow$ 13.3	75.9 $\uparrow$ 51.1	76.9 $\uparrow$ 52.1
BAM	25.2	18.7 $\downarrow$ 6.5	74.8 $\uparrow$ 49.6	61.8 $\uparrow$ 36.6
BBOX	10.7	15.2 $\uparrow$ 4.5	64.4 $\uparrow$ 53.7	86.4 $\uparrow$ 75.7
Bridging Data Gaps	15.8	8.1 $\downarrow$ 7.7	58.1 $\uparrow$ 42.3	66.0 $\uparrow$ 50.2
FRE	22.7	17.5 $\downarrow$ 5.2	81.4 $\uparrow$ 58.7	76.8 $\uparrow$ 54.1

Table 1: Per-paper PaperBench Code-Dev comparison with Claude backbones. Claude columns for BasicAgent and IterAgent follow the PaperBench report ( Starace et al., 2025 ) ; DeepCode follows its concurrent report ( Li et al., 2025b ) . BasicAgent and IterAgent use Claude-3.5-Sonnet, while DeepCode and ReproAgent (ours) use Claude-4.5-Sonnet. Subscripts show absolute-point differences from BasicAgent on the same paper; red upward and green downward arrows indicate the sign of the difference. best and second-best mark the row-best and row-second-best scores among the shown Claude columns.

Paper	BasicAgent	IterAgent	AiScientist	ReproAgent (ours)
Adaptive Pruning	24.5	3.0 $\downarrow$ 21.5	27.2 $\uparrow$ 2.7	41.9 $\uparrow$ 17.4
All in One	20.9	45.1 $\uparrow$ 24.2	46.3 $\uparrow$ 25.4	51.6 $\uparrow$ 30.7
BAM	48.5	45.0 $\downarrow$ 3.5	56.6 $\uparrow$ 8.1	48.5
BBOX	15.4	8.3 $\downarrow$ 7.1	33.8 $\uparrow$ 18.4	15.6 $\uparrow$ 0.2
Bridging Data Gaps	12.6	12.4 $\downarrow$ 0.2	23.1 $\uparrow$ 10.5	50.6 $\uparrow$ 38.0
FRE	21.7	23.9 $\uparrow$ 2.2	35.2 $\uparrow$ 13.5	26.9 $\uparrow$ 5.2

Table 2: Per-paper PaperBench Code-Dev comparison with Gemini. Gemini columns for BasicAgent, IterAgent, and AiScientist follow the AiScientist engineering report ( Chen et al., 2026 ) ; all Gemini columns use Gemini-3-Flash. Subscripts show absolute-point differences from BasicAgent on the same paper; red upward and green downward arrows indicate the sign of the difference. best and second-best mark the row-best and row-second-best scores among the shown Gemini columns.

Figure 4: Gemini channel ablations on all 20 papers. Removing reference evidence (top) or implementation requirements (bottom) lowers the mean; judge provenance and token diagnostics are in the appendix.

Setting	Mean	Median	$\Delta$
Full contract	39.7	41.8	—
$-$ Reference evidence	21.6	21.4	$-$ 18.1
$-$ Implementation requirements	25.6	24.6	$-$ 14.1

Table 3: Gemini-3-Flash ablation results over 20 papers;

\Delta

is the mean drop from the full contract.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Backbone / setting	Reported score
BasicAgent ( Starace et al., 2025 ) a	Gemini-2.0-Flash	5.0
	o3-mini	5.1
	GPT-4o	7.7
	o1	19.5
	Claude-3.5-Sonnet	35.4
IterativeAgent ( Starace et al., 2025 ) a	o3-mini	16.4

Appendix

Table 4: Supplemental aggregate PaperBench baselines. Reported scores are percentages. Rows differ in scaffold, backbone, runtime budget, and reporting protocol, and provide context for the full-suite comparison rather than a primary controlled comparison.

Paper	Full	w/o impl.	w/o ref.	Case role
Sample-specific Masks	53.8	21.3	28.0	Clear two-channel benefit: the full run preserves SMM-specific optimization, mask generation, interpolation, baselines, and ablations.
BAM	48.5	30.8	20.7	Large positive case: both removed-channel settings score substantially below the full contract, with the largest gap for reference evidence.
Bridging Data Gaps	50.6	38.4	13.9	Reference-heavy case: paper obligations remain useful, but removing reference evidence sharply reduces concrete protocol and repository-structure recovery.
PINN	61.3	53.3	52.2	Standardized-prior case: all variants remain strong because PINN objectives and PDE-loss structure are common, but the full contract is still best.
What Will My Model Forget	16.8	10.7	8.4	Scale and coverage stress case: the task has many long-tail details, and the full contract still leads both ablations.
LCA-on-the-Line	41.8	16.1	33.8	Boundary case: no-reference keeps a visible checklist for ObjectNet/LCA details, while the full run remains the strongest setting.

Appendix

Table 5: Representative Gemini ablation cases. The case role column summarizes inspection observations alongside the score columns. “w/o impl.” removes the implementation-requirement channel; “w/o ref.” removes the reference-evidence channel.

Figure 5: Code-level evidence for two representative ablation cases. The full generated repositories expose different concrete surfaces: Sample-specific Masks relies on implementation-requirement preservation for method-specific masks, interpolation, baselines, and ablations, while Bridging Data Gaps relies more heavily on reference evidence for protocol and training-artifact wiring.

Paper	Claude		Gemini		w/o ref.		w/o req.
	Score	Files	Score	Files	Score	Files	Score	Files
Adaptive Pruning	67.8	46	41.9	92	6.2	23	14.3	24
All in One	76.9	77	51.6	48	23.7	16	28.0	36
BAM	61.8	37	48.5	47	20.7	18	30.8	38
BBOX	86.4	280	15.6	70	14.6	15	14.2	75
Bridging Data Gaps	66.0	151	50.6	23	13.9	17	38.4	19

Appendix

Table 6: Per-paper scores and generated repository sizes for our full and ablated runs. File counts come from the corresponding run metadata and are reported as process metadata rather than scoring inputs.

Setting	Papers	Score	Judge tok.	Pipe tok.	Hours	Calls
ReproAgent / Claude	20	73.7	12,868,729	17,833,795	9.3	35.4
ReproAgent / Gemini full	20	39.7	6,243,588	17,459,007	5.6	11.7
Gemini w/o reference evidence	20	21.6	6,582,787	4,034,007	1.5	13.8
Gemini w/o implementation requirements	20	25.6	4,668,761	4,429,321	1.3	12.6

Appendix

Table 7: Usage metadata for our runs. Judge-token cells use root-level or summed leaf metadata depending on how the PaperBench summary stores usage; pipeline-token cells use exported run metadata or the estimate described above.

Paper	Claude judge	Gemini judge	Gemini pipe	No-ref judge	No-ref pipe	No-req judge	No-req pipe
Adaptive pruning	5,423,412	2,868,933	23,047,479	1,980,736	2,236,269	3,341,264	5,919,631
All-in-one	5,538,759	3,780,967	9,264,024	3,429,304	3,936,893	2,250,662	2,630,553
BAM	30,766,863	7,768,083	16,097,296	8,827,504	4,969,452	1,892,690	15,027,467
BBOX	8,567,577	774,669	58,268,936	4,903,571	4,131,573	4,373,037	4,509,759
Bridging data gaps	4,746,750	2,300,086	8,820,069	1,864,915	4,264,829	2,412,905	4,406,997
FRE	37,387,691	13,780,748	8,425,041	10,006,741	2,975,272	12,737,673	3,925,572

Appendix

Table 8: Per-paper token metadata. “Judge” is PaperBench grading-token usage, with leaf metadata summed for summaries that store usage at leaf level; “pipe” is generation-token usage from exported run metadata when available.

Phase	Stage	Engineering role
Prepare	Input normalization	Normalize the target, task type, entities, and explicit constraints.
Prepare	Paper chunking	Segment paper text into section-level chunks with overflow handling.
Prepare	Requirement extraction	Extract implementation requirements that need concrete code, artifacts, interfaces, or validation targets.
Prepare	Reference acquisition	Collect reference repositories and survey file trees, README files, and likely reusable modules.
Plan	Work-package planning	Group prepared requirements into implementation-oriented packages such as data, method, training, evaluation, and artifact writing.
Plan	Package reference evidence	Attach package-local content and structure evidence; this supports package decisions but does not decide the final file tree.

Appendix

Table 9: Complete stage sequence of the ReproAgent pipeline. Stages are executed in order; within-phase stages may share intermediate artifacts.

Stage	Audit statistic	Mean	Median	Range
Prepare	Extracted implementation units per paper	33.8	34.0	28–38
Prepare	Candidate reference repositories requested per paper	24.6	25.0	10–30
Prepare	Prepared reference repositories per paper	1.1	0.0	0–9
Prepare	Implementation units grounded by references per paper	6.2	0.0	0–29
Plan	Work packages per paper	19.6	21.5	3–38
Plan	Contract assertions per paper	6.9	7.0	5–10

Appendix

Table 10: Coarse audit statistics from the 20 Gemini full runs. These logged counts summarize the scale of preparation, planning, generation, and runtime interaction recorded during execution; they are not inputs to PaperBench judging.

Explore similar work

Coding-agents can replicate scientific machine learning papers
Jul 2, 2026Atharva Hans, Ilias BilionisCoding Agents LLM Agent Evaluation
School of Mechanical Engineering, Purdue University, West Lafayette, IN, 47907, USA
ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues
Jun 16, 2026Shanda Li, Qiuhong Anna Wei, Jingwu Tang +5LLM Agent Evaluation AI Agent Benchmarks
School of Computer Science, Carnegie Mellon University · Datadog
HiRAS: A Hierarchical Multi-Agent Framework for Paper-to-Code Generation and Execution
Apr 20, 2026Hanhua Hong, Yizhi LI, Jiaoyan Chen +4Multi-Agent LLM Systems Code Generation
The University of Manchester · Institute for Infocomm Research (I²R), A*STAR · ELLIS Manchester +1

Jul 2, 2026J/K move · Enter open · S save

Coding-agents can replicate scientific machine learning papers

Atharva Hans, Ilias Bilionis

School of Mechanical Engineering, Purdue University, West Lafayette, IN, 47907, USA

Coding Agents LLM Agent Evaluation ML Reproducibility Scientific Reproducibility Scientific Workflow Automation Scientific Claim Verification

Open paper AskPDF arXiv

Scientific machine learning papers typically make computational claims, e.g., that the relative mean square error is less than 5% or that the 95% predictive credible interval covers the test data. A coding agent can be prompted to replicate those claims from paper materials alone, but the prompt does not by itself reliably preserve progress or check whether generated evidence supports the paper's claims. We introduce Paper-replication, a workflow that makes each selected paper claim a target with recorded evidence, and implement it as a coding-agent skill. The workflow makes the agent record those targets, reconstruct the paper's method, run computational experiments, link generated outputs to provenance and comparisons with the paper's claims, record where matched evidence appears in the replication report, and pass validation checks before completion. We evaluate Paper-replication on twelve independent runs across four scientific machine learning papers. All twelve workspaces pass the completion gate, and all 158 recorded targets are matched with report coverage. Even in this completed workspace state, repeated runs differ in how papers are divided into targets, in numerical fidelity to the source papers, in elapsed replication time, in the number of intermediate executions replaced before final evidence is accepted, and in the rules used to accept evidence. Paper-replication makes completion depend on workspace evidence and validation checks rather than on the agent's final message.

Abstract

Explore similar work

Coding-agents can replicate scientific machine learning papers

ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues

HiRAS: A Hierarchical Multi-Agent Framework for Paper-to-Code Generation and Execution