cs.AIApr 15, 2026
SaveAI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot
Organizations: Department of Computer Science, The University of Texas at Austin · Department of Computing Science, University of Alberta · Department of Electrical Engineering and Computer Science, University of Michigan · School of Information, The University of Texas at Austin · Department of Linguistics, The University of Texas at Austin · Sony AI · The Valley Library, Oregon State University · Alberta Machine Intelligence Institute
Abstract
Scientific peer review faces mounting strain as submission volumes surge, making it increasingly difficult to sustain review quality, consistency, and timeliness. Recent advances in AI have led the community to consider its use in peer review, yet a key unresolved question is whether AI can generate technically sound reviews at real-world conference scale. Here we report the first large-scale field deployment of AI-assisted peer review: every main-track submission at AAAI-26 received one clearly identified AI review from a state-of-the-art system. The system combined frontier models, tool use, and safeguards in a multi-stage process to generate reviews for all 22,977 full-review papers in less than a day. A large-scale survey of AAAI-26 authors and program committee members showed that participants not only found AI reviews useful, but actually preferred them to human reviews on key dimensions such as technical accuracy and research suggestions. We also introduce a novel benchmark and find that our system substantially outperforms a simple LLM-generated review baseline at detecting a variety of scientific weaknesses. Together, these results show that state-of-the-art AI methods can already make meaningful contributions to scientific peer review at conference scale, opening a path toward the next generation of synergistic human-AI teaming for evaluating research.
Figures & tables
Figure 1: The AAAI-26 AI review system (a) and review generation timeline (b). For every submission to the AAAI-26 main track that entered the full review phase, the AI review system took as input the submitted paper in PDF form and generated an AI review that was then included in the first phase of the two-phase AAAI-26 review process. The AI review system first pre-processed the paper to resample all images to a consistent resolution of 250 DPI, and then used olmOCR [ Poznanski et al. 2025 ] to convert the PDF version of the paper to markdown. Both PDF and markdown versions of the paper were provided to the the subsequent stages. Five core review stages assessed the story, presentation, evaluations, correctness, and significance of the paper. An initial review was generated, compiling findings from all stages, into a consistent format. A self-critique stage checked the review for unsubstantiated claims, missing details, and inconsistencies with the paper. The final review stage revised the review to address the self-critique and compiled a final review. Logs, checkpoints, and review reports generated at all stages were saved for auditing and human oversight. The final review was assessed by a quality-checking critic, and a human inspected the critical judgments to identify potential concerns such as ethical concerns, revealing author identities, or missing structural elements in the review. The review generation timeline (b) shows the progress of all staged generations for all 22,977 papers. An initial batch of of the papers were run through the core review stages and the outputs inspected manually, before proceeding to complete all remaining stages and process all remaining papers. After the final manual inspection stage, a few papers were flagged for re-processing after manual PDF conversions to handle exceptional graphics formats.
| Review Type | Authors | PC | SPC | AC |
|---|---|---|---|---|
| AI Reviews | 575 | 1305 | 117 | 12 |
| Human Reviews | 2500 | 879 | 433 | 13 |
| Total | 3075 | 2184 | 550 | 25 |
Table 1: Number of responses by the respondent roles and review types, including the reviewer program committee (PC), senior program committee (SPC), and area chairs (AC). Response counts are further broken down by review type being surveyed (AI or human reviews). There were 5,834 responses in total.
Figure 2: Survey responses: AI vs. human review comparisons (a) and AI review questions (b). The left figure shows the differences in the mean response score between AI and human reviews for each of the nine review-quality criteria. In six out of nine criteria, AI reviews were rated higher than human reviews. The preference towards AI reviews was stronger for authors than for PC, SPC, and ACs. All p-values show strong statistical significance at the level. The right figure shows the responses to the questions posed for just the AI reviews. Overall, there is strong support among the respondents that the AI reviews were useful in the AAAI-26 pilot and that they would be useful in future peer review processes. Respondents also expressed that the AI reviews demonstrated capabilities that were beyond what they expected from AI, despite indicating that they went into the process expecting that the AI reviews would be useful. Responses to questions posed specifically to the reviewers (labelled ‘(R)’) show that the AI reviews both raised points that a human reviewer would have difficulty catching, while simultaneously overlooking points that a human reviewer would likely have caught — indicating the complementary strengths of AI vs. human reviews. The majority of reviewers indicated that the AI reviews did not change their evaluation or interpretation of the paper, though a minority did indicate that they did. Authors (labelled ‘(A)’) were mixed in their views about the AI reviews raising points that the human reviews omitted.
Figure 3: Top five most frequent positive and negative themes specific to the AAAI-26 AI Review Pilot found in written feedback from authors and program committee members. The percentages refer to the percentage of mentions belonging to that particular theme over all classified mentions specific to this pilot.
Figure 4: The SPECS Review Benchmark curation and analysis workflow (a) and stage-by-criterion detection rates (b) from evaluating each stage in the AAAI-26 AI Review System for the SPECS criteria. Starting from the accepted papers listed in a venue proceedings (AAAI-25 in our case), the SPECS curation system samples across proceedings categories, matches each paper to an arXiv source release, and retains papers whose L a T e X sources compile successfully. For each curated paper, an LLM generates controlled source-level perturbations targeting the five SPECS criteria: S tory, P resentation, E valuations, C orrectness, and S ignificance. Each perturbation is accepted only if the cited source span matches the original file and the edited paper recompiles without errors, yielding executable synthetic errors for evaluation. The full AAAI-26 AI Review System and the criterion-targeted intermediate stages are then run on these perturbed papers, and a judge model determines whether each review explicitly identifies the injected error with supporting evidence from the review text. The stage-by-criterion detection-rate matrix in (b) summarizes criterion-wise detection behavior: each row corresponds to a perturbation type, while each column corresponds to a core stage in the AAAI-26 AI Review System. Each cell is computed independently as the fraction of perturbations of the row criterion detected by the column stage; rows do not sum to 1 because a perturbation can be detected by multiple stages (or none). The diagonal entries indicate criterion-aligned detection of the target core stage and the corresponding criterion, while off-diagonal entries reveal cross-criterion detection patterns. The targetedness margin is the difference between a row’s diagonal value and its largest off-diagonal value, indicating how much more strongly each stage detects its intended criterion than the other stages.
| Criterion | Baseline | Targeted | Final | T–B | -value | F–B | -value | |
|---|---|---|---|---|---|---|---|---|
| Story | 153 | 0.3529 | 0.6667 | 0.6732 | +0.3137 | +0.3203 | ||
| Presentation | 173 | 0.4162 | 0.5202 | 0.5665 | +0.1040 | 0.0051 | +0.1503 | |
| Evaluations | 159 | 0.5157 | 0.6730 | 0.7547 | +0.1572 | 0.0006 | +0.2390 | |
| Correctness | 144 | 0.6111 | 0.6944 | 0.7639 | +0.0833 | 0.0576 | +0.1528 | |
| Significance | 154 | 0.2597 | 0.5325 | 0.4481 | +0.2727 | +0.1883 | 0.0003 | |
| All SPECS criteria | 783 | 0.4291 | 0.6143 | 0.6386 | +0.1852 | +0.2095 |
Table 2: SPECS benchmark results by criterion, comparing the AAAI-26 AI Review System (Final) to the baseline (Baseline) and criterion-targeted stages (Targeted). Improvements over baseline are listed in the columns, and corresponding p-values are reported for each comparison.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Prompt component | Where it is used | Synopsis of Prompt |
|---|---|---|
| Base instruction | Prepended to all stages | Explains the task to the model: to generate a factual, objective review without scores or recommendations through a sequence of subsequent steps. Also explains how to use the PDF and OCR-derived markdown as complementary sources. |
| Story prompt | Story stage | Asks the model to analyze the paper’s problem formulation, claimed gap in prior work, core contribution, and whether the evidence supports the main story. |
| Presentation prompt | Presentation stage | Asks the model to assess clarity, organization, readability, and whether the technical narrative is easy for a researcher to follow. |
| Evaluations prompt | Evaluations stage | Asks the model to inspect baselines, datasets, metrics, statistical evidence, empirical support for claims, and reproducibility-related weaknesses, using the available code interpreter tool if needed. |
| Correctness prompt | Correctness stage | Asks the model to verify equations, proofs, algorithms, figures, and tables, using the available code interpreter tool if needed. |
| Significance prompt | Significance stage | Asks the model to identify closely related published work at top-tier venues and use it to contextualize novelty, competitiveness, and missing comparisons, with specific instructions to restrict results to published work only, and to disregard preprints or non-peer-reviewed work. |
Table 3: Descriptions for prompts used in the AAAI-26 AI review system. The table lists the prompt components, where each is used, and a high-level synopsis of what each prompt asks the model to do.
| Questionnaire item | A/AI | A/H | R/AI | R/H |
| Overall impressions | ||||
| This review was a thorough review for this conference | ||||
| This review demonstrated capabilities beyond what I expected from AI | ||||
| This review changed my evaluation or interpretation of the paper | ||||
| This review changed my evaluation of the paper | ||||
| Overall, AAAI-26 AI reviews were useful | ||||
Table 4: Questionnaire items by respondent group and review type. ‘A’ denotes author questionnaires and ‘R’ denotes the shared reviewer-form structure used for PC, SPC, and AC respondents. ‘AI’ and ‘H’ denote questionnaires attached to AI reviews and human reviews, respectively.
| Criterion | R1 | R2 | Consensus | |
|---|---|---|---|---|
| Story | 6 | 5 | 5 | 5 |
| Presentation | 7 | 3 | 4 | 3 |
| Evaluations | 8 | 5 | 6 | 5 |
| Correctness | 7 | 7 | 6 | 6 |
| Significance | 7 | 3 | 4 | 3 |
| All SPECS criteria | 35 | 23 | 25 | 22 |
Table 5: SPECS benchmark human oversight. Reviewers R1 and R2 independently reviewed each perturbation. The responses here are the verdicts of the perturbation that they declared valid.