Can AI Reviewers See Your Efforts to Improve Your Paper? A Large-Scale Study of Paper Revision Trajectories
Organizations: MBZUAI · Virginia Tech · University of Maryland · University of Notre Dame
Abstract
AI reviewers are increasingly used to assess scientific manuscripts, yet it remains unclear whether their judgments reflect the improvements authors make through revision. We investigate this question using PaperTrajectory-2026, the 2026 release of PaperTrajectory, a dataset of 6,959 paper trajectories across five AI conferences, with automatic annotations of changes along 14 dimensions and separate judgments of overall quality. The annotations show that major improvements emphasize empirical evidence and framing, while later revisions tend toward incremental improvement. Using independent assessments of manuscript versions from eight AI reviewers, we examine whether annotated major improvements receive higher scores, which revision dimensions are associated with score changes, and whether user opinions can outweigh observed revision gains. Every reviewer leaves scores unchanged on most major improvements, and agreement on the direction of score changes is weak. Revision dimensions show some reviewer-specific associations with score increases, but provide limited predictive discrimination and no consistent guide across reviewers. Even when a revision earns a higher neutral score, a brief negative user opinion can erase or reverse that advantage, with substantial variation across reviewers. These findings show that substantive revisions are not consistently reflected in review scores and that observed scoring gains can be vulnerable to user framing, motivating assessments that connect manuscript evidence to overall judgments and remain stable when no scientific evidence changes.
Figures & tables
| Dimension | GPT-5.5 | GPT-5.6 Sol | GPT-6 Astra | Gemini 3.5 | Gemini 3.8 | Sonnet | Haiku | GLM |
|---|---|---|---|---|---|---|---|---|
| Methods | 1.57 ∗ | 0.91 | 1.11 | 1.78 ∗ | 1.14 | 1.20 | 1.28 | 1.31 |
| Coverage | 1.20 | 0.99 | 0.84 | 1.32 | 1.37 | 1.63 ∗ | 0.89 | 1.05 |
| Baselines | 1.05 | 0.96 | 1.34 | 2.06 ∗ | 1.18 | 1.40 | 1.06 | 1.76 ∗ |
| Statistics | 1.34 ∗ | 0.92 | 0.66 ∗ | 1.14 | 1.28 | 1.38 ∗ | 1.26 | 1.12 |
| Theory | 1.30 | 1.17 | 0.80 | 1.07 | 1.39 ∗ | 1.40 ∗ | 1.10 | 1.15 |
| Claims | 1.12 | 1.23 | 1.31 | 2.00 ∗ | 1.67 ∗ | 1.21 | 1.19 | 1.49 ∗ |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Venue | Papers | Versions | Transitions |
|---|---|---|---|
| ICLR | 2,372 | 5,637 | 3,265 |
| ICML | 2,239 | 5,135 | 2,896 |
| ACL | 772 | 1,782 | 1,010 |
| CVPR | 943 | 2,088 | 1,145 |
| ECCV | 633 | 1,409 | 776 |
| Total | 6,959 | 16,051 | 9,092 |
| Keyword | Dimension | Scope |
|---|---|---|
| Methods | Method and model | Algorithm, architecture, objective, or inference changes; additional description alone does not establish a new method. |
| Coverage | Dataset and task coverage | Changes in represented datasets, tasks, populations, or test environments, beyond renamed benchmarks. |
| Baselines | Baselines and comparison methods | Changes to the set of methods actually evaluated in the corresponding comparison, not citations alone. |
| Protocol | Evaluation protocol and settings | Changes in splits, metrics, controls, budgets, or procedures; disclosure of an unchanged setting is assessed as technical disclosure. |
| Ablations | Ablations, robustness, and analyses | Changes to ablations, sensitivity, robustness, error analyses, or case studies; appendix location is not a criterion. |
| Statistics | Statistics and uncertainty | Changes in runs, seeds, intervals, tests, or uncertainty reporting; changed numerical outcomes alone are assessed as concrete results. |
| Dimensions | Major | Minor | Lift |
| Results + Visuals | 92.5 | 38.6 | 1.02 |
| Ablations + Results | 89.4 | 39.4 | 1.03 |
| Protocol + Results | 88.2 | 30.9 | 1.04 |
| Ablations + Visuals | 86.9 | 33.1 | 1.04 |
| Protocol + Visuals | 85.2 | 26.1 | 1.03 |
| Protocol + Ablations | 82.9 | 26.1 | 1.05 |
| Major improvement (%) | Major-change dimensions | |||
| Stratum | First | Last | First | Last |
| 2 transitions | 22.5 | 12.1 | 5.6 | 3.0 |
| 3 or more transitions | 22.7 | 7.5 | 5.8 | 2.4 |
| ICLR | 23.6 | 9.3 | 5.8 | 2.6 |
| ICML | 23.7 | 17.3 | 5.6 | 3.5 |
| ACL | 16.6 | 6.0 | 5.6 | 2.5 |
| Reviewer | Higher (%) | Same (%) | Lower (%) |
|---|---|---|---|
| GPT-5.5 | 40.50 [38.07, 43.04] | 54.92 [52.41, 57.39] | 4.58 [3.51, 5.66] |
| GPT-5.6 Sol | 35.38 [32.95, 37.85] | 58.56 [56.02, 61.05] | 6.06 [4.85, 7.30] |
| GPT-6 Astra | 21.02 [18.98, 23.23] | 77.63 [75.38, 79.76] | 1.35 [0.75, 1.96] |
| Gemini 3.5 Flash | 20.61 [18.62, 22.64] | 76.76 [74.66, 78.88] | 2.64 [1.83, 3.51] |
| Gemini 3.8 Flash | 40.95 [38.46, 43.48] | 55.74 [53.23, 58.25] | 3.31 [2.43, 4.22] |
| Claude Sonnet 5 | 38.00 [35.55, 40.31] | 58.46 [56.07, 61.01] | 3.54 [2.59, 4.52] |
| Reviewer | Higher | Same | Lower |
|---|---|---|---|
| ICLR | |||
| GPT-5.5 | 37.48 [33.57, 41.41] | 57.79 [53.67, 61.82] | 4.73 [2.99, 6.53] |
| GPT-5.6 Sol | 29.25 [25.43, 32.99] | 64.27 [60.24, 68.25] | 6.48 [4.62, 8.54] |
| GPT-6 Astra | 17.86 [14.64, 21.16] | 80.91 [77.64, 84.20] | 1.23 [0.35, 2.27] |
| Gemini 3.5 Flash | 20.88 [17.57, 24.16] | 76.49 [73.03, 79.93] | 2.63 [1.41, 4.02] |
| Gemini 3.8 Flash | 43.68 [39.68, 47.89] | 52.46 [48.17, 56.49] | 3.86 [2.31, 5.42] |
| Conference | Reviewer | Modal OA | Mode (%) | Ties at mode (%) |
|---|---|---|---|---|
| ICLR | GPT-5.5 | 5 | 51.23 | 59.63 |
| Sol | 3 | 66.79 | 66.48 | |
| Astra | 3 | 85.01 | 86.83 | |
| Gemini 3.5 | 8 | 75.52 | 97.24 | |
| Gemini 3.8 | 6 | 50.85 | 57.84 | |
| Sonnet | 6 | 54.27 | 73.43 |
| Reviewers raising scores | Observed count | Conference | Conference + initial OA |
|---|---|---|---|
| 0 | 97 | 60.34 [48.00, 73.00] | 146.24 [129.00, 163.00] |
| 1 | 314 | 234.21 [211.98, 257.00] | 281.50 [255.00, 307.00] |
| 2 | 334 | 395.47 [363.00, 426.00] | 311.90 [284.00, 341.00] |
| 3 | 291 | 380.51 [349.00, 412.00] | 264.46 [238.98, 291.00] |
| 4 | 190 | 229.48 [206.00, 253.00] | 192.47 [169.00, 216.00] |
| 5 | 96 | 88.99 [73.98, 104.00] | 123.30 [104.00, 142.00] |
| Sample | Pairs | OR [95% CI] | |
|---|---|---|---|
| All complete cases | 1,420 | 1.571 [1.247, 1.978] | 0.0017 |
| Earliest transition | 1,379 | 1.574 [1.245, 1.991] | 0.0018 |
| Shared revision pairs | 1,352 | 1.538 [1.214, 1.949] | 0.0054 |
| Reviewer | Pairs | AUC [95% CI] | Log loss [95% CI] | Brier [95% CI] |
|---|---|---|---|---|
| GPT-5.5 | 1,420 | 0.6071 [0.5760, 0.6375] | 0.6698 [0.6516, 0.6921] | 0.2347 [0.2278, 0.2420] |
| GPT-5.6 Sol | 1,420 | 0.5098 [0.4798, 0.5407] | 0.6633 [0.6454, 0.6817] | 0.2328 [0.2252, 0.2408] |
| GPT-6 Astra | 1,420 | 0.5683 [0.5313, 0.6068] | 0.5167 [0.4863, 0.5494] | 0.1641 [0.1518, 0.1772] |
| Gemini 3.5 Flash | 1,416 | 0.6408 [0.6051, 0.6760] | 0.4980 [0.4668, 0.5303] | 0.1581 [0.1465, 0.1703] |
| Gemini 3.8 Flash | 1,416 | 0.6005 [0.5705, 0.6288] | 0.6651 [0.6520, 0.6804] | 0.2360 [0.2303, 0.2427] |
| Claude Sonnet 5 | 1,408 | 0.6022 [0.5730, 0.6340] | 0.6548 [0.6373, 0.6718] | 0.2306 [0.2232, 0.2378] |
| Brier | Log loss | |||
|---|---|---|---|---|
| Reviewer | Baseline | Reduction [95% CI] | Baseline | Reduction [95% CI] |
| GPT-5.5 | 0.2413 | 0.6757 | ||
| GPT-5.6 Sol | 0.2291 | 0.6508 | ||
| GPT-6 Astra | 0.1651 | 0.5123 | ||
| Gemini 3.5 Flash | 0.1634 | 0.5085 | ||
| Gemini 3.8 Flash | 0.2423 | 0.6778 | ||
| Reviewer | Opposing opinions Strict reversal | Criticism only Strict reversal | Criticism only Erase/reverse | ||
|---|---|---|---|---|---|
| GPT-5.5 | 1,483 | 600 | 97.7 [96.4, 98.7] | 54.3 [50.4, 58.6] | 97.8 [96.5, 99.0] |
| GPT-5.6 Sol | 1,483 | 524 | 74.6 [70.9, 78.2] | 11.3 [8.5, 14.0] | 80.9 [77.6, 84.2] |
| GPT-6 Astra | 1,483 | 312 | 7.1 [4.3, 9.9] | 0.0 [0.0, 0.0] | 24.7 [20.1, 29.4] |
| Gemini 3.5 Flash | 1,480 | 305 | 50.2 [44.5, 55.7] | 37.0 [31.5, 42.4] | 69.8 [64.6, 74.7] |
| Gemini 3.8 Flash | 1,479 | 606 | 28.2 [24.7, 31.6] | 12.4 [9.9, 15.1] | 60.1 [56.2, 64.0] |
| Claude Sonnet 5 | 1,418 | 540 | 89.1 [86.4, 91.7] | 51.3 [47.2, 55.6] | 95.7 [94.0, 97.3] |
| Reviewer | Opposing opinions | Criticism only | |
|---|---|---|---|
| Strict reversal | Strict reversal | Erase/reverse | |
| Eight-model common sample | |||
| GPT-5.5 | 97.6 | 54.2 | 98.3 |
| GPT-5.6 Sol | 74.1 | 10.9 | 80.7 |
| GPT-6 Astra | 7.2 | 0.0 | 25.3 |
| Gemini 3.5 Flash | 48.9 | 35.9 | 67.8 |
| One-category advantage | At least two categories | |||
|---|---|---|---|---|
| Reviewer | Reversed / total | Rate [95% CI] | Reversed / total | Rate [95% CI] |
| GPT-5.5 | 539 / 549 | 98.2 [97.1, 99.3] | 47 / 51 | 92.2 [83.9, 98.2] |
| GPT-5.6 Sol | 358 / 475 | 75.4 [71.4, 79.1] | 33 / 49 | 67.3 [54.2, 80.6] |
| GPT-6 Astra | 21 / 267 | 7.9 [4.9, 11.1] | 1 / 45 | 2.2 [0.0, 7.7] |
| Gemini 3.5 Flash | 119 / 216 | 55.1 [48.3, 61.8] | 34 / 89 | 38.2 [28.4, 48.8] |
| Gemini 3.8 Flash | 146 / 431 | 33.9 [29.5, 38.3] | 25 / 175 | 14.3 [9.3, 19.5] |
| Reviewer | Neutral | Both positive | Both negative | Negative neutral |
|---|---|---|---|---|
| GPT-5.5 | 7.64 | 4.30 | 2.74 | -4.90 [-5.68, -4.18] |
| GPT-5.6 Sol | 6.49 | 8.71 | 3.32 | -3.17 [-3.91, -2.40] |
| GPT-6 Astra | 4.59 | 5.88 | 4.55 | -0.04 [-0.46, 0.39] |
| Gemini 3.5 Flash | 5.08 | 4.24 | 7.59 | 2.51 [1.14, 3.88] |
| Gemini 3.8 Flash | 10.03 | 9.80 | 11.64 | 1.61 [0.62, 2.58] |
| Claude Sonnet 5 | 7.18 | 6.01 | 5.34 | -1.83 [-2.68, -1.06] |
| Conference | Reviewer | Opposing opinions | Criticism only | ||
|---|---|---|---|---|---|
| Strict: | Erase/reverse: | ||||
| ICLR | GPT-5.5 | 214 | 98.6 | 52.3 | 99.1 |
| Sol | 167 | 79.6 | 10.8 | 82.6 | |
| Astra | 102 | 3.9 | 0.0 | 22.5 | |
| Gemini 3.5 | 119 | 58.8 | 49.6 | 70.6 | |
| Gemini 3.8 | 249 | 31.3 | 13.3 | 67.9 | |
Scholarly Peer Review
Scholarly peer review studies how experts evaluate research, make publication decisions, and improve the fairness, reliability, and transparency of those processes.
See all 86 papers