Organizations: Beihang University · Shanghai Jiao Tong University · Minzu University of China · Peking University · Zhongguancun Academy · Beijing Key Laboratory of Data Intelligence and Security (Peking University)
LLM agents can generate paper reproduction code, yet often produce scientifically unfaithful implementations. We define this failure mode as semantic drift, where generated code silently diverges from the paper's specifications. We introduce SemanticAlign-Bench(SA-Bench), a diagnostic benchmark covering 30 papers from ICLR, ICML and NeurIPS 2025. For each paper, we decompose its specifications into atomic and verifiable implementation claims, which we call Semantic Alignment Units (SAUs) and evaluate repositories along four diagnostic dimensions spanning numerical, methodological, protocol and ordering drift. In total, we construct 1,491 SAUs across five ML domains and evaluate 12 generator configurations (4 models × 3 scaffolds). Even the strongest configuration (Claude+PaperCoder) achieves a mean SAU score of only 0.301 out of 1.0, with an overall mean of 0.221 across 360 evaluations. A failure taxonomy reveals that agents attempt most requirements but implement them incorrectly, with implementation mismatch and stubs accounting for the majority of zero-scored claims. Our analysis further indicates that scaffolds optimized for executability provide limited leverage for scientific reproduction; narrowing the gap requires scaffolds that prioritize semantic specification verification. The benchmark, annotations and evaluation pipeline are publicly available.
Figures & tables
Figure 1: Overview of SA-Bench. The benchmark consists of a curated paper collection and per-paper SAU files. Given a paper, a coding agent generates a reproduction repository, which is then scored against the paper’s SAU claims to produce a diagnostic report.
Dimension
PaperBench
LMR-BENCH
SciReplicate
SciCoQA
SA-Bench (Ours)
Evaluation unit
Rubric node
Function
Function
QA pair
Claim (SAU)
Static evaluation
Partial *
—
—
✓
✓
Failure taxonomy
—
—
—
Partial †
✓
Graded scoring
Binary
Binary
Binary
Binary
5-level rubric
Table 1: Comparison between SA-Bench and existing benchmarks. ✓ denotes supported; — denotes not supported.
Figure 2: Four diagnostic dimensions of semantic drift.
Figure 3: SA-Bench construction pipeline. A code-first filter separates implementable content from theoretical discussion; three specialized agents (D1 Numerical, D2 Method, D3 Protocol) extract SAU claims in parallel with per-section spawning to avoid attention collapse. D4 claims are derived from D2/D3 ordering annotations during a merge phase. All claims undergo mandatory human review before inclusion in the benchmark.
Figure 4: Example SAU scoring output for a single claim (D1-006, emergent-planning-rl). The judge outputs structured JSON with claim-level scores, cited evidence with file:line references and per-dimension aggregates. Full judge prompt templates are in Appendix F .
Score
Label
Criterion
1.0
Faithful
All semantic content correctly implemented.
0.75
Minor deviation
Core correct; deviation is an optimization variant or implementation detail.
0.5
Key omission
Main logic present but a material step or component missing.
0.25
Core substitution
Core mechanism replaced by a different approach.
0
Absent
No implementation or logic directly contradicts paper.
Table 2: Five-level scoring rubric.
Figure 5: Overall SAS distribution by model and scaffold. Each dot is one paper; crossbars show mean ± 95% CI.
Model
BasicAgent
PaperCoder
OpenHands
Claude-Sonnet-4.6
0.272
0.301 (+0.029)
0.277 (+0.005)
DeepSeek-V4-Pro
0.268
0.241 ( − 0.027)
0.282 (+0.014)
Gemini-2.5-Flash
0.150
0.256 (+0.106)
0.243 (+0.093)
GPT-4o
0.081
0.197 (+0.116)
0.082 (+0.001)
Table 3: Mean SAS per model and scaffold. Deltas relative to BasicAgent: red = gain, green = regression.
Factor
Overall
D1
D2
D3
D4
By Model
Claude-Sonnet-4.6
0.283
0.413
0.293
0.201
0.225
DeepSeek-V4-Pro
0.263
0.358
0.294
0.186
0.216
Gemini-2.5-Flash
0.217
0.268
0.248
0.161
0.189
GPT-4o
0.120
0.120
0.140
0.092
0.130
By Scaffold
Table 4: Marginal means by model and scaffold across all four drift dimensions. Model range: 0.120–0.283; scaffold range: 0.193–0.249.
Figure 6: Per-dimension mean SAU score heatmaps
Scaffold
Model
Overall
D1
D2
D3
D4
Claude-Sonnet-4.6
0.272
0.391
0.307
0.169
0.219
BasicAgent
DeepSeek-V4-Pro
0.268
0.379
0.308
0.180
0.204
Gemini-2.5-Flash
0.150
0.134
0.210
0.099
0.158
GPT-4o
0.081
0.047
0.121
0.058
0.100
Claude-Sonnet-4.6
0.301
0.470
0.299
0.209
0.226
PaperCoder
DeepSeek-V4-Pro
0.241
0.339
0.278
0.148
0.200
Table 5: Mean SAS for all 12 generator configurations, grouped by scaffold. Bold marks the best score per column; underline marks the second best.
Figure 7: Domain × Dimension heatmap of mean SAU scores across all 12 generator configurations.
Figure 8: Score-level distribution of all 17,892 claim judgments
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Paper ID
Title
Venue
Total
D1
D2
D3
D4
adjoint-matching
Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control
ICLR 2025
47
17
20
7
3
avg-reward-pg
Global Convergence of Policy Gradient in Average Reward MDPs
ICLR 2025
22
9
7
3
3
ca2-vdm
Ca2-VDM: Efficient Autoregressive Video Diffusion Model with Causal Generation and Cache Sharing
ICML 2025
48
22
10
8
8
cara
Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers
ICML 2025
41
19
11
10
1
conformal-bayesian-quadrature
Conformal Prediction as Bayesian Quadrature
ICML 2025
38
25
6
3
4
diffusion-convergence-rate
Instance-dependent Convergence Theory for Diffusion Models
ICLR 2025
42
22
12
4
4
Appendix
Table 6: Full benchmark paper list with SAU counts by drift type.
Statistic
D1
D2
D3
D4
Total
Total claims
523
503
300
165
1,491
% of total
35.1%
33.7%
20.1%
11.1%
100%
Mean per paper
17.4
16.8
10.0
5.5
49.7
Appendix
Table 7: SAU claims per dimension for the 30-paper benchmark.
Paper
Score
Pass
Total
Steps
Time (s)
Cost
fre
0.331
113
306
24
909.1
$2.11
sample-specific-masks
0.603
66
87
81
652.4
$6.10
mechanistic-understanding
0.238
29
36
37
1252.5
$5.81
pinn
0.482
112
126
35
909.4
$4.10
all-in-one
0.117
15
92
81
478.1
$3.41
Total/Avg.
0.581
335
647
52
840.3
$21.53
Appendix
Table 8: Per-paper pilot results with resource usage. Score is the gpt-4o judge’s leaf-aggregated mean. Steps and time are read from the agent’s step timings; cost is the recorded API spend.
Paper
Judge verdict (abridged, verbatim)
all-in-one
“The submission lacks any explicit implementation of the [Euler-Maruyama] discretization process, and no reference to the number of steps (500) is observable in the provided files. The absence of such code means the submission does not meet the resolution criteria.”
fre
“The submission does not meet the resolution criteria. Although it partially implements reward functions and describes the evaluation tasks for the walker domain, there is no actual code to set up or interact with the ExORL walker environment. The absence of integration with the custom_dmc_tasks repository… indicates a failure to satisfy the criterion.”
sample-specific-masks
“Instead of 32 output channels, the third layer implements 64 output channels. The required BatchNorm operation immediately following the third convolution is missing.”
mechanistic-understanding
“The code processes only a subset of 295 prompts instead of the required 1199 prompts from the RealToxicityPrompts challenge set. There is no clear integration showing measurement across all 20 token generations for every prompt.”
pinn
“While it has implemented the ability to extract the s , y , and rho vectors with the function extract_lbfgs_state … it does not integrate this functionality into the training pipeline to actually save these vectors at the end of training. Without saving the L-BFGS state, the criterion is not fully resolved.”
Appendix
Table 9: Verbatim excerpts from GPT-4o judge verdicts on score-0 leaf nodes from the pilot runs. Quotations are abridged for length; ellipses mark omitted text.
Dimension
PaperBench
Ours
Taxonomy
Free-form rubric tree
D1–D4 diagnostic
Extraction
Author-paired manual
Multi-agent section-level
Filtering
Implicit (author judgment)
Code-first hard rules
Granularity control
Human rubric tree
Rule-constrained + human review
Effort per paper
Author-time intensive
Multi-agent parallel + 1–2 h human
Appendix
Table 10: Comparison of SAU extraction methodology with PaperBench’s manually authored rubric trees.
Paper
Before
After
Del
Mod
Add
Merge
Dominant Issue
diffusion-convergence-rate
12
6
6
1
2
2
Baseline contamination
conformal-bayesian-quadrature
18
15
0
0
1
5
Over-splitting
navil
35
31
2
10
1
1
Type misclassification
luno
19
25
0
9
5
0
Omissions, source fixes
moe-pot
32
35
7
13
10
0
Mixed: result values + retypes
Appendix
Table 11: Representative per-paper review statistics, drawn from review_log.json . Before = agent-extracted candidates submitted for review (modifiable subset); After = candidates after review. Del/Mod/Add/Merge are explicit reviewer actions on the modifiable subset. The five papers illustrate different dominant failure modes.
SAU Claim
Repo Implementation
Type
Score
Rationale
“Adam optimizer, lr=0.001”
AdamW(lr=0.001)
D1
0.75
Optimization variant: Adam → AdamW is a functional improvement; does not alter experimental conclusions.
“Adam optimizer, lr=0.001”
SGD(lr=0.1)
D1
0.25
Core violation: different optimizer family + order-of-magnitude lr error; training dynamics fundamentally altered.
“Adam optimizer, lr=0.001”
Adam(lr=0.003)
D1
0.75
Minor deviation: order-of-magnitude consistent; specific value within 3× .
“Score matching loss (Eq. 5)”
Code implements KL divergence
D2
0.25
Core violation: different algorithm family; paper’s core claim no longer holds.
“Score matching loss (Eq. 5)”
Correct score matching but missing ε regularization term
D2
0.5
Key omission: core algorithm present but incomplete.
“Compare against PPO, SAC, TD3”
Only PPO and SAC implemented
D3
0.5
Key omission: 1 of 3 baselines missing.
Appendix
Table 13: Annotated scoring examples for judge calibration.
Table 21
Figure 9: Score distribution across all 360 evaluations. The heavy right skew confirms semantic drift is pervasive.
All phases merged; flat LR schedule. Training recipe as prose
Ordering collapse
Appendix
Table 18: Representative failure cases from Claude + PaperCoder (best configuration), annotated with failure taxonomy types.
Paper
Overall
Std
D1
D2
D3
D4
conformal-bayesian-quadrature
0.304
0.113
0.524
0.306
0.194
0.193
mrq
0.271
0.073
0.331
0.306
0.167
0.281
olmoe
0.260
0.096
0.277
0.296
0.201
0.264
sc-fno
0.257
0.094
0.319
0.255
0.196
0.257
neural-operator-flow-matching-pde
0.255
0.100
0.285
0.236
0.222
0.276
ca2-vdm
0.248
0.109
0.419
0.227
0.148
0.198
Appendix
Table 19: Per-paper mean SAS across all 12 generators, sorted by overall mean. These are paper-level means (equally weighted across papers); SAU-level means in the main text weight each claim equally and may differ slightly.
Paper
Scaff
Model
Overall
D1
D2
D3
D4
Time(s)
.py
adjoint
BasicAgent
Claude-Sonnet-4.6
0.208
0.382
0.225
0.143
0.083
938
12
DeepSeek-V4-Pro
0.208
0.279
0.138
0.250
0.167
861
9
Gemini-2.5-Flash
0.136
0.029
0.287
0.143
0.083
678
3
GPT-4o
0.110
0.015
0.138
0.036
0.250
189
4
PaperCoder
Claude-Sonnet-4.6
0.304
0.632
0.250
0.250
0.083
2408
12
DeepSeek-V4-Pro
0.303
0.471
0.113
0.214
0.417
2342
6
Appendix
Table 20: Complete per-configuration per-paper SAU scores and code generation metrics. Time(s) : end-to-end agent wall-clock seconds. .py : number of Python source files generated by the agent in the resulting repository.
Reproducing research results from papers and released code is central to scientific progress. Existing works have introduced benchmarks to evaluate whether LLM agents can assist with reproducibility, but they are difficult to scale due to their reliance on substantial manual effort for data curation and evaluation. We introduce ReproRepo, a scalable framework for reproducibility evaluation that leverages human-raised GitHub issues as naturally occurring supervision on realistic reproduction blockers. We instantiate ReproRepo on 1,149 recent machine learning papers from major conferences and evaluate four frontier model-agent configurations. Our results show that LLM agents, even without executing code, can identify many real-world reproducibility problems from paper-repository pairs: the best agent in our study, namely Codex with GPT-5.5, surfaces at least one semantically related human-reported blocker for ∼90% of papers in the study. Further analysis shows that agents are particularly effective for surfacing visible failures and identifying the right semantic region, but may still be insufficient in exact localization. ReproRepo can serve as a reusable, scalable framework for future evaluations of LLM agents on real-world reproducibility auditing. Our code is released at https://github.com/LithiumDA/ReproRepo.
Shanda Li, Qiuhong Anna Wei, Jingwu Tang +5
School of Computer Science, Carnegie Mellon University · Datadog
Recent work has used LLM agents to reproduce empirical social science results with access to both the data and code. We broaden this scope by asking: Can they reproduce results given only a paper's methods description and original data? We develop an agentic reproduction system that extracts structured methods descriptions from papers, runs reimplementations under strict information isolation -- agents never see the original code, results, or paper -- and enables deterministic, cell-level comparison of reproduced outputs to the original results. An error attribution step traces discrepancies through the system chain to identify root causes. Evaluating four agent scaffolds and four LLMs on 48 papers with human-verified reproducibility, we find that agents can largely recover published results, but performance varies substantially between models, scaffolds, and papers. Root cause analysis reveals that failures stem both from agent errors and from underspecification in the papers themselves.
Benjamin Kohler, David Zollikofer, Johanna Einsiedler +2
Paper-to-code reproduction asks scientific AI agents to turn research papers into executable repositories that preserve the paper's method, protocol and artifacts. This is difficult because the specification is split: explicit paper content such as algorithms, metrics and artifacts is often lost across long agent trajectories, while implicit details such as framework defaults and conventions inherited from related work are absent from the paper. We introduce ReproAgent, a four-stage Prepare--Plan--Generate--Repair pipeline built around a persistent implementation contract with two channels: an implementation-requirement channel that turns paper snippets into code obligations, and a reference-evidence channel that retrieves content and structure evidence from related repositories. Both are bound to work packages, projected into file-level contracts, and consumed across generation and repair. On PaperBench Code-Dev, ReproAgent reaches the highest mean score among same-backbone scaffolds under both Claude-Sonnet-4.5 and Gemini-3-Flash. End-to-end channel ablations and per-paper cases support the contribution of both channels. Code and experimental artifacts are publicly available.
Xue Hu, Zewei Pan, Zhongyuan Wang +3
Beihang University · Shanghai Jiao Tong University · Peking University +3