Reproducibility checks during peer review require a shared understanding of what authors' code is expected to reproduce. We present an author-in-the-loop collaborative audit that makes author declarations explicit inputs to an AI agent's investigation and assessment. A structured checklist records reproduction instructions and limitations; a two-stage workflow on GitHub connects agent checks to author clarification, revision, and agent reassessment. The agent separately assesses whether results have been reproduced and whether authors need to provide clarification or make repairs, while human reviewers approve requests and decide whether concerns are resolved. We deployed the system across 36 conference papers and 133 experimental tables and figures, raising 20 issues after human review. Further checks supported closing all the issues, although full reproduction remained unverified in some cases. We further compared repeated audits with complete and blank checklists to examine how author declarations shaped the agent's investigation and assessment. Our work contributes a collaborative workflow and agent design that use author knowledge to guide reproducibility auditing. The conference deployment provides empirical evidence that this approach can help identify and address reproducibility problems through collaboration among authors, agents, and reviewers.
Figures & tables
Figure 1: The two-stage collaborative audit workflow. In the first stage, the author’s checklist guides the initial agent code check. Papers with human-approved issues enter the second stage for response and reassessment, with human reviewers deciding issue resolution. The workflow aims to help authors and reviewers clarify what can be reproduced and what requires clarification or repair.
Figure 2: GitHub organization and audit records supporting the two stages of collaborative reproducibility checking.
Figure 3: Synthetic demonstration of the two-stage collaborative audit. (a) Author declarations guide the initial check, and a reviewer approves the issue. (b) The author clarifies the metric and revises the code. (c) Fresh output supports a recommendation to close the issue, which a reviewer accepts.
Figure 5: Using author declarations to guide checks of paper results. The guidance column illustrates checklist information about the scope and feasibility of the execution paths investigated.
Figure 6: Assessing reproduction outcomes separately from the need for author action. (a) The agent evaluates audit evidence and separately decides whether author action is needed. (b) Full reproduction exceeds the audit’s time budget, but the authors still need to supply missing core code. (c) The demo covers only one city, as the authors declared, so no further action is requested.
Figure 7: Records supporting run-2 and auditability, continuing the synthetic Table D1 example in Figure 3 . (a) A fixed record links the original GitHub issue and findings to the author reply and revised code. (b) Each issue has a workspace with numbered execution records and raw output, alongside separate checker and verifier assessments. The expanded attempt shows a local execution of the corrected synthetic code.
Audit records
Coverage
Recorded model
Findings
Average token usage
Official initial audit (run-1)
36 papers; 133 result items
GPT-5.5
The run-1 audit yielded 20 open GitHub issues after human approval.
Not available
Official reassessment (run-2)
10 papers with 20 original GitHub issues
GPT-5.6-Sol
The run-2 audit recommended closing 15 issues. All 20 were ultimately closed following author responses, further checks, and human review.
13.09 million
Checklist comparison experiment
The same 10 papers; three repeats with or without checklists
GPT-5.6-Sol
Audit records showed how declarations informed investigation planning and judgments about author action. Complete checklists yielded fewer requests overall.
25.13 million
Table 1: Audit configurations and main findings.
Repeat
Complete checklist
Blank checklist
Difference
1
22
28
+6
2
25
29
+4
3
21
27
+6
Table 2: Number of result items judged by the agent to require author action in each condition. Each repeat checks the same 39 result items from ten papers with a complete or blank checklist. Difference is the count with a blank checklist minus the count with a complete checklist.
Figure 8: Findings and published issues in the initial audit. (a) Finding tags across 133 result items from 36 papers, including findings that did not warrant author action. (b) Labels on the 20 published issues. (c) Their distribution across ten anonymized papers, with unique issue counts shown separately. Each issue concerns one result item. Tags and labels can overlap: one published issue carries both missing_code and missing_data , giving 21 issue labels in total.
Reported action
Papers
Updated or completed code
10
Added input datasets or example data
4
Supplied precomputed experimental outputs
1
Revised the manuscript
2
Corrected the reproducibility checklist
1
Clarified reproduction scope or interpretation of outputs
5
Table 3: Changes and clarifications reported by authors of the ten papers with open GitHub issues. Each paper is counted once per category, and categories may overlap.
Path to closure
Concerns
Agent recommended closure after successful execution
9
Agent recommended closure after confirming that missing code had been supplied
6
Human review supported closure without further author replies
2
Further author clarification and reviewer checks supported closure
3
Total
20
Table 4: Closure pathways for the 20 original concerns across ten papers.