cs.CLNov 13, 2026

AI Reviews Are Biased Towards the Appendix

Authors: Saptarshi Ghosh, Mohammad Saim, Tianyu Jiang

Organizations: University of Cincinnati

Abstract

Large language models (LLMs) are increasingly used by reviewers to highlight weaknesses during the reviewing process, and their outputs often shape the reviews that humans submit. Prior work studies how LLM reviews align with human ones, yet we know little about which parts of a paper these models actually criticize. In this work, we investigate 2,000 papers from ACL, EMNLP, COLM, and ICLR, and show that the latest GPT and Claude models draw about 50% or more of their review weaknesses from the appendix. This behavior holds across the API, coding-agent, and web access methods for using LLMs to review a paper. In contrast, we find that humans draw only 8.4% of their weaknesses from the appendix, and these weaknesses primarily revolve around a paper's presentation and content related to its core idea. Meanwhile, LLMs draw weaknesses from deep in the appendix that concern fine-grained technical details not central to the paper's contributions. We anticipate that our analysis will contribute to the scholarly discourse on AI reviews. Our code is publicly available at: https://github.com/cincynlp/review_appendix.

Figures & tables

Explore similar work

CardsList
  1. Writing literature reviews with AI: principles, hurdles and some lessons learned

    Mar 8, 2026Saadi Lahlou, Annabelle Gouttebroze, Atrina Oraee +1LLM SycophancyLLM Evaluation

  2. Review Arcade: On the Human Alignment and Gameability of LLM Reviews

    May 27, 2026Hans Ole Hatzel, Sebastian Steindl, Jan StrichLLM-as-a-JudgeHuman-AI Agreement

  3. LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers

    May 25, 2026Lingyao Li, Junjie Xiong, Changjia Zhu +5LLM EvaluationLLM-as-a-Judge