Overview

Automated peer review studies how computational systems can evaluate manuscripts, identify weaknesses, check claims, and assist tasks such as reviewer matching, rebuttal, and meta-review. Common approaches include prompting and fine-tuning language models, retrieval-augmented and multi-agent review, rubric-based scoring, and tool-assisted checks of references, code, or experiments. Evaluation uses conference-paper datasets, expert annotations, outcome prediction, and injected-error or claim-verification benchmarks to test dimensions such as correctness, coverage, calibration, and actionability. Systems can provide useful and sometimes complementary feedback, but unreliable judgments, hallucinations, bias, and vulnerability to strategic rewriting or prompt injection leave human oversight and robust evaluation central concerns.

Latest papers

122 papers11 papers in the last four weeks, up 10% on the four weeks before. 0.1% of all new papers.

Nov 3, 2025

"Give a Positive Review Only": An Early Investigation Into In-Paper Prompt Injection Attacks and Defenses for AI Reviewers

With the rapid advancement of AI models, their deployment across diverse tasks has become increasingly widespread. A notable emerging application is leveraging AI models to assist in reviewing scientific papers. However, recent reports have revealed that some papers contain hidden, injected prompts designed to manipulate AI reviewers into providing overly favorable evaluations. In this work, we present an early systematic investigation into this emerging threat. We propose two classes of attacks: (1) static attack, which employs a fixed injection prompt, and (2) iterative attack, which optimizes the injection prompt against a simulated reviewer model to maximize its effectiveness. Both attacks achieve striking performance, frequently inducing full evaluation scores when targeting frontier AI reviewers. Furthermore, we show that these attacks are robust across various settings. To counter this threat, we explore a simple detection-based defense. While it substantially reduces the attack success rate, we demonstrate that an adaptive attacker can partially circumvent this defense. Our findings underscore the need for greater attention and rigorous safeguards against prompt-injection threats in AI-assisted peer review.
Sep 18, 2025

Calibrated Generative AI as Meta-Reviewer: A Systemic Functional Linguistics Discourse Analysis of Reviews of Peer Reviews

This study investigates the use of generative AI to support formative assessment through machine-generated reviews of peer reviews in graduate online courses at a public university in the United States. The analysis draws on Systemic Functional Linguistics and Appraisal Theory to examine how 120 metareviews construct meaning across ideational and interpersonal dimensions as well as the textual dimension. The findings suggest that generative AI can approximate key rhetorical and relational features of effective human feedback by providing clear direction while maintaining a supportive stance. The reviews demonstrated a balance between praise and constructive critique alongside alignment with rubric expectations and structured staging that foregrounded student agency. By modeling these qualities AI metafeedback has the potential to scaffold feedback literacy and enhance learner engagement with peer review. We conclude that carefully calibrated AI can contribute to hybrid human AI feedback ecosystems when embedded in thoughtfully designed workflows. However its effective integration into educational practice requires critical attention to its limitations and adaptation to specific educational contexts.