SA-Bench: Evaluating Semantic Alignment in LLM-Based Paper Reproduction

Xue Hu; Zewei Pan; Zeli Su; Zhou Liu; Wentao Zhang

cs.AIAug 25, 2026

SA-Bench: Evaluating Semantic Alignment in LLM-Based Paper Reproduction

Authors: Xue Hu, Zewei Pan, Zeli Su, Zhou Liu, Wentao Zhang

Organizations: Beihang University · Shanghai Jiao Tong University · Minzu University of China · Peking University ·

Abstract

LLM agents can generate paper reproduction code, yet often produce scientifically unfaithful implementations. We define this failure mode as semantic drift, where generated code silently diverges from the paper's specifications. We introduce SemanticAlign-Bench(SA-Bench), a diagnostic benchmark covering 30 papers from ICLR, ICML and NeurIPS 2025. For each paper, we decompose its specifications into atomic and verifiable implementation claims, which we call Semantic Alignment Units (SAUs) and evaluate repositories along four diagnostic dimensions spanning numerical, methodological, protocol and ordering drift. In total, we construct 1,491 SAUs across five ML domains and evaluate 12 generator configurations (4 models $\times$ 3 scaffolds). Even the strongest configuration (Claude+PaperCoder) achieves a mean SAU score of only 0.301 out of 1.0, with an overall mean of 0.221 across 360 evaluations. A failure taxonomy reveals that agents attempt most requirements but implement them incorrectly, with implementation mismatch and stubs accounting for the majority of zero-scored claims. Our analysis further indicates that scaffolds optimized for executability provide limited leverage for scientific reproduction; narrowing the gap requires scaffolds that prioritize semantic specification verification. The benchmark, annotations and evaluation pipeline are publicly available.

Figures & tables

Figure 1: Overview of SA-Bench. The benchmark consists of a curated paper collection and per-paper SAU files. Given a paper, a coding agent generates a reproduction repository, which is then scored against the paper’s SAU claims to produce a diagnostic report.

Dimension	PaperBench	LMR-BENCH	SciReplicate	SciCoQA	SA-Bench (Ours)
Evaluation unit	Rubric node	Function	Function	QA pair	Claim (SAU)
Static evaluation	Partial *	—	—	✓	✓
Failure taxonomy	—	—	—	Partial †	✓
Graded scoring	Binary	Binary	Binary	Binary	5-level rubric

Table 1: Comparison between SA-Bench and existing benchmarks.

\checkmark

denotes supported; — denotes not supported.

Figure 2: Four diagnostic dimensions of semantic drift.

Figure 3: SA-Bench construction pipeline. A code-first filter separates implementable content from theoretical discussion; three specialized agents (D1 Numerical, D2 Method, D3 Protocol) extract SAU claims in parallel with per-section spawning to avoid attention collapse. D4 claims are derived from D2/D3 ordering annotations during a merge phase. All claims undergo mandatory human review before inclusion in the benchmark.

Figure 4: Example SAU scoring output for a single claim (D1-006, emergent-planning-rl). The judge outputs structured JSON with claim-level scores, cited evidence with file:line references and per-dimension aggregates. Full judge prompt templates are in Appendix F .

Score	Label	Criterion
1.0	Faithful	All semantic content correctly implemented.
0.75	Minor deviation	Core correct; deviation is an optimization variant or implementation detail.
0.5	Key omission	Main logic present but a material step or component missing.
0.25	Core substitution	Core mechanism replaced by a different approach.
0	Absent	No implementation or logic directly contradicts paper.

Table 2: Five-level scoring rubric.

Figure 5: Overall SAS distribution by model and scaffold. Each dot is one paper; crossbars show mean

\pm

95% CI.

Model	BasicAgent	PaperCoder	OpenHands
Claude-Sonnet-4.6	0.272	0.301 (+0.029)	0.277 (+0.005)
DeepSeek-V4-Pro	0.268	0.241 ( $-$ 0.027)	0.282 (+0.014)
Gemini-2.5-Flash	0.150	0.256 (+0.106)	0.243 (+0.093)
GPT-4o	0.081	0.197 (+0.116)	0.082 (+0.001)

Table 3: Mean SAS per model and scaffold. Deltas relative to BasicAgent: red

=

gain, green

=

regression.

By Model
Factor	Overall	D1	D2	D3	D4
Claude-Sonnet-4.6	0.283	0.413	0.293	0.201	0.225
DeepSeek-V4-Pro	0.263	0.358	0.294	0.186	0.216
Gemini-2.5-Flash	0.217	0.268	0.248	0.161	0.189
GPT-4o	0.120	0.120	0.140	0.092	0.130
By Scaffold

Table 4: Marginal means by model and scaffold across all four drift dimensions. Model range: 0.120–0.283; scaffold range: 0.193–0.249.

Figure 6: Per-dimension mean SAU score heatmaps

Scaffold	Model	Overall	D1	D2	D3	D4
	Claude-Sonnet-4.6	0.272	0.391	0.307	0.169	0.219
BasicAgent	DeepSeek-V4-Pro	0.268	0.379	0.308	0.180	0.204
	Gemini-2.5-Flash	0.150	0.134	0.210	0.099	0.158
	GPT-4o	0.081	0.047	0.121	0.058	0.100
	Claude-Sonnet-4.6	0.301	0.470	0.299	0.209	0.226
PaperCoder	DeepSeek-V4-Pro	0.241	0.339	0.278	0.148	0.200

Table 5: Mean SAS for all 12 generator configurations, grouped by scaffold. Bold marks the best score per column; underline marks the second best.

Figure 7: Domain $\times$ Dimension heatmap of mean SAU scores across all 12 generator configurations.

Figure 8: Score-level distribution of all 17,892 claim judgments

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Paper ID	Title	Venue	Total	D1	D2	D3	D4
adjoint-matching	Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control	ICLR 2025	47	17	20	7	3
avg-reward-pg	Global Convergence of Policy Gradient in Average Reward MDPs	ICLR 2025	22	9	7	3	3
ca2-vdm	Ca2-VDM: Efficient Autoregressive Video Diffusion Model with Causal Generation and Cache Sharing	ICML 2025	48	22	10	8	8
cara	Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers	ICML 2025	41	19	11	10	1
conformal-bayesian-quadrature	Conformal Prediction as Bayesian Quadrature	ICML 2025	38	25	6	3	4
diffusion-convergence-rate	Instance-dependent Convergence Theory for Diffusion Models	ICLR 2025	42	22	12	4	4

Appendix

Table 6: Full benchmark paper list with SAU counts by drift type.

Statistic	D1	D2	D3	D4	Total
Total claims	523	503	300	165	1,491
% of total	35.1%	33.7%	20.1%	11.1%	100%
Mean per paper	17.4	16.8	10.0	5.5	49.7

Appendix

Table 7: SAU claims per dimension for the 30-paper benchmark.

Paper	Score	Pass	Total	Steps	Time (s)	Cost
fre	0.331	113	306	24	909.1	$2.11
sample-specific-masks	0.603	66	87	81	652.4	$6.10
mechanistic-understanding	0.238	29	36	37	1252.5	$5.81
pinn	0.482	112	126	35	909.4	$4.10
all-in-one	0.117	15	92	81	478.1	$3.41
Total/Avg.	0.581	335	647	52	840.3	$21.53

Appendix

Table 8: Per-paper pilot results with resource usage. Score is the gpt-4o judge’s leaf-aggregated mean. Steps and time are read from the agent’s step timings; cost is the recorded API spend.

Paper	Judge verdict (abridged, verbatim)
all-in-one	“The submission lacks any explicit implementation of the [Euler-Maruyama] discretization process, and no reference to the number of steps (500) is observable in the provided files. The absence of such code means the submission does not meet the resolution criteria.”
fre	“The submission does not meet the resolution criteria. Although it partially implements reward functions and describes the evaluation tasks for the walker domain, there is no actual code to set up or interact with the ExORL walker environment. The absence of integration with the custom_dmc_tasks repository… indicates a failure to satisfy the criterion.”
sample-specific-masks	“Instead of 32 output channels, the third layer implements 64 output channels. The required BatchNorm operation immediately following the third convolution is missing.”
mechanistic-understanding	“The code processes only a subset of 295 prompts instead of the required 1199 prompts from the RealToxicityPrompts challenge set. There is no clear integration showing measurement across all 20 token generations for every prompt.”
pinn	“While it has implemented the ability to extract the s , y , and rho vectors with the function extract_lbfgs_state … it does not integrate this functionality into the training pipeline to actually save these vectors at the end of training. Without saving the L-BFGS state, the criterion is not fully resolved.”

Appendix

Table 9: Verbatim excerpts from GPT-4o judge verdicts on score-0 leaf nodes from the pilot runs. Quotations are abridged for length; ellipses mark omitted text.

Dimension	PaperBench	Ours
Taxonomy	Free-form rubric tree	D1–D4 diagnostic
Extraction	Author-paired manual	Multi-agent section-level
Filtering	Implicit (author judgment)	Code-first hard rules
Granularity control	Human rubric tree	Rule-constrained $+$ human review
Effort per paper	Author-time intensive	Multi-agent parallel $+$ 1–2 h human

Appendix

Table 10: Comparison of SAU extraction methodology with PaperBench’s manually authored rubric trees.

Paper	Before	After	Del	Mod	Add	Merge	Dominant Issue
diffusion-convergence-rate	12	6	6	1	2	2	Baseline contamination
conformal-bayesian-quadrature	18	15	0	0	1	5	Over-splitting
navil	35	31	2	10	1	1	Type misclassification
luno	19	25	0	9	5	0	Omissions, source fixes
moe-pot	32	35	7	13	10	0	Mixed: result values $+$ retypes

Appendix

Table 11: Representative per-paper review statistics, drawn from review_log.json . Before = agent-extracted candidates submitted for review (modifiable subset); After = candidates after review. Del/Mod/Add/Merge are explicit reviewer actions on the modifiable subset. The five papers illustrate different dominant failure modes.

SAU Claim	Repo Implementation	Type	Score	Rationale
“Adam optimizer, lr=0.001”	AdamW(lr=0.001)	D1	0.75	Optimization variant: Adam $\rightarrow$ AdamW is a functional improvement; does not alter experimental conclusions.
“Adam optimizer, lr=0.001”	SGD(lr=0.1)	D1	0.25	Core violation: different optimizer family + order-of-magnitude lr error; training dynamics fundamentally altered.
“Adam optimizer, lr=0.001”	Adam(lr=0.003)	D1	0.75	Minor deviation: order-of-magnitude consistent; specific value within $3\times$ .
“Score matching loss (Eq. 5)”	Code implements KL divergence	D2	0.25	Core violation: different algorithm family; paper’s core claim no longer holds.
“Score matching loss (Eq. 5)”	Correct score matching but missing $\varepsilon$ regularization term	D2	0.5	Key omission: core algorithm present but incomplete.
“Compare against PPO, SAC, TD3”	Only PPO and SAC implemented	D3	0.5	Key omission: 1 of 3 baselines missing.

Appendix

Table 13: Annotated scoring examples for judge calibration.

Table 21

Figure 9: Score distribution across all 360 evaluations. The heavy right skew confirms semantic drift is pervasive.

Dim.	Paper Specification	Agent Output / Root Cause	Failure Type
D1	Training: 900K levels, 250M transitions; UNet: ch_mult=[1,2,4,8]	Uses 10K levels, ch_mult=[1,1,2,4]. Specs in appendix, defaults not overridden	Impl. mismatch
D2	Adjoint matching loss with correction term; SCoRe: two-turn RL with asymmetric reward	Standard flow matching without correction; single-turn RL. Mathematical distinctions missed	Wrong impl.
D3	17 datasets (DAVIS, YT-VOS, MOSE, etc.); 5 baselines (PPO, SAC, TD3, Dreamer-v3, MPC)	Data loading for 3 datasets; only PPO implemented	External knowledge
D4	4-phase pipeline; warmup $\rightarrow$ cosine decay $\rightarrow$ eval	All phases merged; flat LR schedule. Training recipe as prose	Ordering collapse

Appendix

Table 18: Representative failure cases from Claude + PaperCoder (best configuration), annotated with failure taxonomy types.

Paper	Overall	Std	D1	D2	D3	D4
conformal-bayesian-quadrature	0.304	0.113	0.524	0.306	0.194	0.193
mrq	0.271	0.073	0.331	0.306	0.167	0.281
olmoe	0.260	0.096	0.277	0.296	0.201	0.264
sc-fno	0.257	0.094	0.319	0.255	0.196	0.257
neural-operator-flow-matching-pde	0.255	0.100	0.285	0.236	0.222	0.276
ca2-vdm	0.248	0.109	0.419	0.227	0.148	0.198

Appendix

Table 19: Per-paper mean SAS across all 12 generators, sorted by overall mean. These are paper-level means (equally weighted across papers); SAU-level means in the main text weight each claim equally and may differ slightly.

Paper	Scaff	Model	Overall	D1	D2	D3	D4	Time(s)	.py
adjoint	BasicAgent	Claude-Sonnet-4.6	0.208	0.382	0.225	0.143	0.083	938	12
		DeepSeek-V4-Pro	0.208	0.279	0.138	0.250	0.167	861	9
		Gemini-2.5-Flash	0.136	0.029	0.287	0.143	0.083	678	3
		GPT-4o	0.110	0.015	0.138	0.036	0.250	189	4
	PaperCoder	Claude-Sonnet-4.6	0.304	0.632	0.250	0.250	0.083	2408	12
		DeepSeek-V4-Pro	0.303	0.471	0.113	0.214	0.417	2342	6

Appendix

Table 20: Complete per-configuration per-paper SAU scores and code generation metrics. Time(s) : end-to-end agent wall-clock seconds. .py : number of Python source files generated by the agent in the resulting repository.

ML Reproducibility

Machine learning reproducibility studies whether model results and scientific claims can be independently checked and reliably repeated.

See all 130 papers

ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues
Jun 16, 2026Shanda Li, Qiuhong Anna Wei, Jingwu Tang +5LLM Agent Evaluation AI Agent Benchmarks
School of Computer Science, Carnegie Mellon University · Datadog
Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results
Apr 23, 2026Benjamin Kohler, David Zollikofer, Johanna Einsiedler +2AI Coding Agents LLM Agent Evaluation
ETH Zurich · University of Basel
ReproAgent: Contract-Guided Paper-to-Code Reproduction
Aug 25, 2026Xue Hu, Zewei Pan, Zhongyuan Wang +3Scientific Code Generation Agentic Code Generation
Beihang University · Shanghai Jiao Tong University · Peking University +3

Jun 16, 2026J/K move · Enter open · S save

ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues

Shanda Li, Qiuhong Anna Wei, Jingwu Tang, Valerie Chen+4

School of Computer Science, Carnegie Mellon University · Datadog

LLM Agent Evaluation AI Agent Benchmarks ML Reproducibility Scientific Reproducibility Benchmark Reproducibility

Open paper AskPDF arXiv

Reproducing research results from papers and released code is central to scientific progress. Existing works have introduced benchmarks to evaluate whether LLM agents can assist with reproducibility, but they are difficult to scale due to their reliance on substantial manual effort for data curation and evaluation. We introduce ReproRepo, a scalable framework for reproducibility evaluation that leverages human-raised GitHub issues as naturally occurring supervision on realistic reproduction blockers. We instantiate ReproRepo on 1,149 recent machine learning papers from major conferences and evaluate four frontier model-agent configurations. Our results show that LLM agents, even without executing code, can identify many real-world reproducibility problems from paper-repository pairs: the best agent in our study, namely Codex with GPT-5.5, surfaces at least one semantically related human-reported blocker for

\sim

90% of papers in the study. Further analysis shows that agents are particularly effective for surfacing visible failures and identifying the right semantic region, but may still be insufficient in exact localization. ReproRepo can serve as a reusable, scalable framework for future evaluations of LLM agents on real-world reproducibility auditing. Our code is released at https://github.com/LithiumDA/ReproRepo.

Abstract

ML Reproducibility

Explore similar work

ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues

Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results

ReproAgent: Contract-Guided Paper-to-Code Reproduction