Reproducibility checks during peer review require a shared understanding of what authors' code is expected to reproduce. We present an author-in-the-loop collaborative audit that makes author declarations explicit inputs to an AI agent's investigation and assessment. A structured checklist records reproduction instructions and limitations; a two-stage workflow on GitHub connects agent checks to author clarification, revision, and agent reassessment. The agent separately assesses whether results have been reproduced and whether authors need to provide clarification or make repairs, while human reviewers approve requests and decide whether concerns are resolved. We deployed the system across 36 conference papers and 133 experimental tables and figures, raising 20 issues after human review. Further checks supported closing all the issues, although full reproduction remained unverified in some cases. We further compared repeated audits with complete and blank checklists to examine how author declarations shaped the agent's investigation and assessment. Our work contributes a collaborative workflow and agent design that use author knowledge to guide reproducibility auditing. The conference deployment provides empirical evidence that this approach can help identify and address reproducibility problems through collaboration among authors, agents, and reviewers.
Figures & tables
Figure 1: The two-stage collaborative audit workflow. In the first stage, the author’s checklist guides the initial agent code check. Papers with human-approved issues enter the second stage for response and reassessment, with human reviewers deciding issue resolution. The workflow aims to help authors and reviewers clarify what can be reproduced and what requires clarification or repair.
Figure 2: GitHub organization and audit records supporting the two stages of collaborative reproducibility checking.
Figure 3: Synthetic demonstration of the two-stage collaborative audit. (a) Author declarations guide the initial check, and a reviewer approves the issue. (b) The author clarifies the metric and revises the code. (c) Fresh output supports a recommendation to close the issue, which a reviewer accepts.
Figure 5: Using author declarations to guide checks of paper results. The guidance column illustrates checklist information about the scope and feasibility of the execution paths investigated.
Figure 6: Assessing reproduction outcomes separately from the need for author action. (a) The agent evaluates audit evidence and separately decides whether author action is needed. (b) Full reproduction exceeds the audit’s time budget, but the authors still need to supply missing core code. (c) The demo covers only one city, as the authors declared, so no further action is requested.
Figure 7: Records supporting run-2 and auditability, continuing the synthetic Table D1 example in Figure 3 . (a) A fixed record links the original GitHub issue and findings to the author reply and revised code. (b) Each issue has a workspace with numbered execution records and raw output, alongside separate checker and verifier assessments. The expanded attempt shows a local execution of the corrected synthetic code.
Audit records
Coverage
Recorded model
Findings
Average token usage
Official initial audit (run-1)
36 papers; 133 result items
GPT-5.5
The run-1 audit yielded 20 open GitHub issues after human approval.
Not available
Official reassessment (run-2)
10 papers with 20 original GitHub issues
GPT-5.6-Sol
The run-2 audit recommended closing 15 issues. All 20 were ultimately closed following author responses, further checks, and human review.
13.09 million
Checklist comparison experiment
The same 10 papers; three repeats with or without checklists
GPT-5.6-Sol
Audit records showed how declarations informed investigation planning and judgments about author action. Complete checklists yielded fewer requests overall.
25.13 million
Table 1: Audit configurations and main findings.
Repeat
Complete checklist
Blank checklist
Difference
1
22
28
+6
2
25
29
+4
3
21
27
+6
Table 2: Number of result items judged by the agent to require author action in each condition. Each repeat checks the same 39 result items from ten papers with a complete or blank checklist. Difference is the count with a blank checklist minus the count with a complete checklist.
Figure 8: Findings and published issues in the initial audit. (a) Finding tags across 133 result items from 36 papers, including findings that did not warrant author action. (b) Labels on the 20 published issues. (c) Their distribution across ten anonymized papers, with unique issue counts shown separately. Each issue concerns one result item. Tags and labels can overlap: one published issue carries both missing_code and missing_data , giving 21 issue labels in total.
Reported action
Papers
Updated or completed code
10
Added input datasets or example data
4
Supplied precomputed experimental outputs
1
Revised the manuscript
2
Corrected the reproducibility checklist
1
Clarified reproduction scope or interpretation of outputs
5
Table 3: Changes and clarifications reported by authors of the ten papers with open GitHub issues. Each paper is counted once per category, and categories may overlap.
Path to closure
Concerns
Agent recommended closure after successful execution
9
Agent recommended closure after confirming that missing code had been supplied
6
Human review supported closure without further author replies
2
Further author clarification and reviewer checks supported closure
3
Total
20
Table 4: Closure pathways for the 20 original concerns across ten papers.
Reproducing research results from papers and released code is central to scientific progress. Existing works have introduced benchmarks to evaluate whether LLM agents can assist with reproducibility, but they are difficult to scale due to their reliance on substantial manual effort for data curation and evaluation. We introduce ReproRepo, a scalable framework for reproducibility evaluation that leverages human-raised GitHub issues as naturally occurring supervision on realistic reproduction blockers. We instantiate ReproRepo on 1,149 recent machine learning papers from major conferences and evaluate four frontier model-agent configurations. Our results show that LLM agents, even without executing code, can identify many real-world reproducibility problems from paper-repository pairs: the best agent in our study, namely Codex with GPT-5.5, surfaces at least one semantically related human-reported blocker for ∼90% of papers in the study. Further analysis shows that agents are particularly effective for surfacing visible failures and identifying the right semantic region, but may still be insufficient in exact localization. ReproRepo can serve as a reusable, scalable framework for future evaluations of LLM agents on real-world reproducibility auditing. Our code is released at https://github.com/LithiumDA/ReproRepo.
Shanda Li, Qiuhong Anna Wei, Jingwu Tang +5
School of Computer Science, Carnegie Mellon University · Datadog
Scientific peer review increasingly struggles to assess reproducibility at the scale and complexity of modern research output. Evaluating reproducibility requires reconstructing experimental dependencies, methodological choices, data flows, and result-generating procedures, which often exceeds what human reviewers can provide. Agentic Reproducibility Assessment (ARA) formalizes reproducibility assessment as a structured reasoning task over scientific documents. Given a paper, ARA extracts a directed workflow graph linking sources, methods, experiments, and outputs, then evaluates its reconstructability using structural and content-based scores for reproducibility assessments. Experiments on 213 ReScience C articles - the largest cross-domain benchmark of human-validated computational reproducibility studies considered to date - demonstrate ARA's generalizability and consistent workflow reconstruction and assessment across LLMs, model temperatures, and scientific domains. ARA achieves ~61% accuracy on three benchmarks, and the highest accuracy reported on ReproBench (60.71% vs. 36.84%) and GoldStandardDB (61.68% vs. 43.56%), highlighting its potential to complement human review at scale and enabling next-generation peer review. Code and Data available: https://github.com/AndresLaverdeMarin/agentic_reproducibility_assessment.
Kevin Riehl, Andres L. Marin, Nikofors Zacharof +6
European Commission, Joint Research Centre · Ideas Forward
Large language model agents are increasingly capable of conducting research autonomously, producing research documents alongside the code and experiments that ostensibly support them. Yet whether the reported findings are consistently supported by corresponding implementations and execution evidence remains largely unexplored: existing review practices primarily assess textual quality and cannot reliably identify inconsistencies such as hard-coded metrics, unimplemented methods, or unsupported experimental results. We present ReAgent, an automated auditing framework for assessing the consistency between agent-generated research documents and their associated repositories. ReAgent constructs structured representations of scientific claims from research documents and uses them to guide repository analysis and evidence collection. Static auditing examines whether claimed methodologies, implementations, and experimental configurations are consistently reflected in the repository, while dynamic auditing executes relevant experiments and collects execution evidence to assess empirical findings. By combining static analysis with dynamic evidence, ReAgent identifies inconsistencies that may remain hidden under either perspective alone, such as experiments that reproduce reported numbers while deviating from the claimed methodology. The collected evidence and audit decisions are organized into a structured repository-level audit report, enabling transparent evidence traceability. We evaluate ReAgent on a manually curated benchmark of agent-generated research document--repository pairs and compare it against representative static and reproduction-based baselines. Experimental results demonstrate that ReAgent effectively identifies inconsistencies between reported research findings and their supporting repository evidence.