Scientific discoveries must be communicated clearly to realize their full potential. Without effective communication, even the most groundbreaking findings risk being overlooked or misunderstood. The primary way scientists communicate their work and receive feedback from the community is through peer review. However, the current system often provides inconsistent feedback between reviewers, ultimately hindering the improvement of a manuscript and limiting its potential impact. In this paper, we introduce a novel method APRES powered by Large Language Models (LLMs) to update a scientific papers text based on an evaluation rubric. Our automated method discovers a rubric that is highly predictive of future citation counts, and integrate it with APRES in an automated system that revises papers to enhance their quality and impact. Crucially, this objective should be met without altering the core scientific content. We demonstrate the success of APRES, which improves future citation prediction by 19.6% in mean averaged error over the next best baseline, and show that our paper revision process yields papers that are preferred over the originals by human expert evaluators 79% of the time. Our findings provide strong empirical support for using LLMs as a tool to help authors stress-test their manuscripts before submission. Ultimately, our work seeks to augment, not replace, the essential role of human expert reviewers, for it should be humans who discern which discoveries truly matter, guiding science toward advancing knowledge and enriching lives.
Figures & tables
Figure 1: APRES is a two-stage framework that utilizes the same agentic search scaffold (left) for two distinct tasks. The right details the inner-loop logic, executed in two stages, at each node of the search tree. Stage 1 - Rubric Search (top right): an agentic search process discovers a rubric that is highly predictive of a paper’s future impact as measured by citation counts. Stage 2 - Paper Improvement (bottom right): the discovered rubric from stage 1 is used as a guide for iteratively revising and enhancing a paper through another agentic search process.
Table 1: Comparison of our work to related approaches. APRES is the first method to integrate the discovery of predictive evaluation criteria with a closed-loop system for automated paper revision.
Category
# Paper
Percentage
Oral
427
1.6%
Spotlight
1,451
5.4%
Poster
11,344
42.4%
Reject
8,916
33.4%
Withdrawn
4,569
17.1%
Total
26,707
100%
Table 2: Paper types in our dataset.
Figure 2: Performance of agentic search (MultiAIDE) in predicting citation counts, measured in Mean Absolute Error (MAE). Our MultiAIDE search approach converges to a lower MAE compared to several strong baselines, including a model using human reviewer scores (Human scores baseline), an MLP on SPECTER paper embeddings, a negative binomial model on the principal components of SPECTER embeddings (Paper embedding + PCA), and an alternative search method (Prompt breeder). The x-axis represents the number of iterations for search methods.
Figure 3: The word cloud generated from participants’ reasons for preferring a revised paper.
Figure 4: Improvement in predicted paper quality from our iterative revision process using OpenAI-o3 and Gemini 2.5 Pro LLMs. Our agentic ‘Rewriter’ improves the scores for both borderline and clear reject papers more. The higher final scores obtained for borderline papers suggests our rubric discovery approach is more effective at fixing presentational flaws as opposed to fundamental scientific ones. The y-axis is averaging of the scores from all rubric items.
Category
# Papers
Percentage
3/3 participants preferred
174
47.8%
2/3 participants preferred
113
31.0%
1/3 participants preferred
65
17.9%
0/3 participants preferred
12
3.3%
Total
364
100%
Table 3: Percentage of human participant preferences for the revised papers.
Paper Type
‘Accept’
‘Borderline’
‘Reject’
APRES w/o R∗
1.02
1.24
1.21
APRES w/o MultiAIDE
1.13
1.46
1.34
APRES
1.67
3.33
2.98
Table 4: Ablation of APRES components.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1 : LLM-derived Glicko2 ratings strongly correlate with final conference decisions. As the Glicko2 rating increases, the proportion of rejected papers (red) decreases while the proportion of accepted papers—posters (blue), spotlights (yellow), and orals (green) steadily rises.
Figure A2 : LLM-based review committees are more consistent than human committees. The heatmap shows the pairwise disagreement rate between various committee configurations. Each axis corresponds to a different base LLM judge (i.e., OpenAI-o1 , OpenAI-o3 , Gemini 2.5 Pro , and Gemini 2.5 Flash ).
Figure A3 : Distribution of citations in our dataset.
Figure A4 : Improvement in predicted paper quality during the iterative revision process using the o1 model. The agentic rewriter improves the scores for both ‘borderline’ (solid lines) and ’rejected’ (dashed lines) papers. The higher final scores achieved for borderline papers suggests our approach is more effective at fixing presentational flaws than fundamental scientific ones.
Figure A5 : Improvement in predicted paper quality during the iterative revision process using the Gemini 2.5 Flash model. The agentic rewriter improves the scores for both ‘borderline’ (solid lines) and ’rejected’ (dashed lines) papers. The higher final scores achieved for borderline papers suggests our approach is more effective at fixing presentational flaws as opposed to fundamental scientific ones.
Criterion
Key Questions
Clarity
Is the text easy to read and understand? Is the language precise and unambiguous? Does the text flow logically? Is the core message communicated effectively?
Correctness
Does one version contain apparent technical inaccuracies or oversimplifications (we guarantee that the contribution of both papers is the same).
Contribution & Impact
Does one version do a better job of framing the paper’s contribution? Is the significance of the work made more apparent and compelling in one over the other?
Presentation & Style
Is the grammar, spelling, and punctuation flawless? Is the tone appropriately academic and professional? Is the text concise, or is it unnecessarily verbose?
Peer review is central to scientific quality control, yet it can undervalue papers that later achieve substantial citation impact. While frontier large language models have shown promise in automating aspects of peer review, they primarily mimic human reviewer preferences rather than predict long-term scientific value. We introduce ReviewGuard, a two-stage framework that aligns LLM-generated reviews with citation-based estimates of long-term scientific impact rather than contemporaneous reviewer judgments. On 20,861 AI/ML papers from OpenReview augmented with Semantic Scholar citation data, ReviewGuard achieves a Spearman correlation of \r{ho} = 0.776 with future citations on rejected-then-published papers, outperforming human reviewers (\r{ho} = 0.492) and a supervised Expert model (\r{ho} = 0.681). Under the same decision threshold, ReviewGuard flags 10.2% of high-impact rejected papers, compared with 1.8% for human reviewers, corresponding to a 5.6x improvement. Our results demonstrate that impact-aligned reinforcement learning can provide editors with a complementary signal for identifying high-potential work, without replacing human judgment.
Abdur Rasool, Xiaohui Huang, Yanqing Hu +1
Department of Statistics and Data Science Southern University of Science and Technology Shenzhen 518055, China
Recent advances in large language models (LLMs) have enabled AI systems to assist scientific research and peer review. However, an essential capability for reliable AI-assisted scientific workflows remains underexplored: verifying whether reviewer feedback leads to meaningful and evidence-supported manuscript improvements. We introduce AutoSupervision, which evaluates whether scientific manuscript revisions genuinely address reviewer concerns through grounded evidence. AutoSupervision leverages transparent peer-review records as a natural source of supervision, where reviewer comments specify scientific concerns, author responses describe claimed resolutions, and revised manuscripts provide evidence of changes. Given reviewer comments, author responses, and revised manuscripts, models must characterize reviewer concerns, determine whether concerns have been addressed, and identify supporting manuscript evidence. We construct AutoSupervision from 56,000 Nature Communications articles and corresponding review records. Then we conducted experiments on LLMs, the ablation study, and the case study. Our results show that while LLMs perform well in characterizing reviewer concerns, with GPT-5.5 achieving a score of 0.754, evidence-based verification remains the primary bottleneck, with the best-performing model reaching only 0.501.
Haobo Li, Eunseo Jung, Wenxiao Zhao +8
1Shanghai AI Laboratory · 3The Hong Kong University of Science and Technology · University of California, Los Angeles
Automated reviewing systems are increasingly evaluated based on the quality of the reviews they produce. Yet a review is only useful if acting on it leads to a measurable improvement in the paper. We present AppliedScientist, a closed-loop system that couples an autonomous AI scientist with an AI reviewer, and evaluate it by iteratively revising rejected papers from a range of research subfields. To mirror how human authors build on earlier drafts, the AI scientist has access to its previous versions during revision. To avoid bias from prior judgments, however, each review is generated independently, with the reviewer having no memory of earlier feedback or scores. We compare three revision settings: one initialized with the original venue reviews, one initialized with AI-generated reviews, and autonomous self-revision using the same fixed prompt in every round. Because the reviewer both guides and evaluates the revision, we also assess the human-initialized revisions using Stanford Reviewer as an independent evaluator. Reviewer-guided revision consistently improves more than fixed-prompt self-revision, and Stanford Reviewer also assigns higher scores to later revisions. AppliedScientist resolves 128 of 150 execution-related weaknesses (85.3%), but only 2 of 18 idea-related weaknesses (11.1%), suggesting that iterative revision is effective at improving experiments and implementation, but rarely changes concerns about novelty or significance.
Vidushee Vats, Karun Sharma, Shengzhi Li +1
Department of Computer Science University of Massachusetts Boston