Peer-review feedback often arrives too late for authors to make meaningful revisions. We study an author-facing LLM system that moves part of this stress test before submission: it generates a broad pool of atomic concerns and compresses them into a short report. We evaluate agreement with historical reviews and, separately, the possible validity of concerns they omit. From 10,000 ICLR 2026 submissions, we use 3,398 manuscripts with accessible versions that predate review. On a ten-paper diagnostic, independent sampling covers 44.9% of historical issues; deduplication and refill reaches 78.7% strict and 84.9% seriousness-weighted coverage, at 3.6× more requests and 5.2× more tokens. A hidden Top-32 Oracle preserves the full 79.3% weighted coverage of a 256-candidate pool, but paper-only selectors retain only 40--44%. LLM review therefore provides broad coverage with a large candidate pool but compresses poorly; ablations identify representative selection and matcher sensitivity as the main sources of this gap.
Figures & tables
Figure 1: Data construction pipeline.
Cohort
Oral
Poster
Reject
Withdrawn
Total
OpenReview snapshot
224
5,127
4,473
176
10,000
Strict historical cohort
61
1,331
1,940
66
3,398
Train
44
958
1,397
47
2,446
Validation
5
106
155
6
272
Held-out test
12
267
388
13
680
Table 1: Corpus construction and frozen paper-family split. Decisions are normalized from the explicit decision note and current venue state.
Strategy
Covered
Strict
Weighted
Unique
Requests
Tokens
Active req. h
Independent batches
80/178
0.449
0.495
656/5,120
525
18.66M
2.23
Deduplicate + refill
140/178
0.787
0.849
5,120 retained
1,905
96.61M
95.82
Table 2: DeepSeek generation strategies on ten papers at 512 retained candidates per paper.
Figure 2: Candidate-generation coverage and cost.
Selector
Strict
Weighted
Major
Precision
Qwen pointwise, one/cluster
.350
.402
.643
.191
Kimi tournament, one/cluster
.390
.442
.727
.206
Cluster Oracle (hidden best)
.699
.755
.873
.384
Unrestricted Top-32 Oracle
.739
.793
.940
.406
Full 256 pool
.739
.793
.940
.051
Table 3: Top-32 compression on ten training papers. All rows use the same frozen candidate–issue matrix. Oracle selectors access that matrix and are upper bounds, not deployable systems.
Figure 3: Candidate clusters and their historical-review matches.
Individual recall is 0.387/0.423; union recall is 0.535, showing complementary misses.
Single vs. mixed 512
DeepSeek-only vs. Sol+DeepSeek
Mixed recall is 0.764 vs. 0.449; just 12.8% of the single-model pool is unique.
Persona submodes
Fixed persona–submode quotas
Better early prefixes, but 74.5% duplicate rejection and zero completed 512 pools.
Novelty stopping
Stop after two low-yield waves
87.6% fewer calls, but macro recall drops from 0.798 to 0.610.
Structured exploration
Section–category–persona slots
More clean unique concerns, but 56.7% no_issue outputs.
Appendix
Table 4: Candidate-generation experiments. All recall values use frozen atomic human issues and a same-issue threshold of 0.70 unless noted.
Strategy
Retained
Requests
Tokens
Wall
Macro recall
Weighted
Full dedup + refill
5,120
1,905
96.61M
n/r
0.798
0.849
Adaptive early stop
1,125
236
7.45M
15.0 min
0.610
0.660
Relative change
−78.0%
−87.6%
−92.3%
—
−0.188
−0.189
Appendix
Table 5: Cost–coverage consequence of novelty-based early stopping. Counts and tokens are totals over the same ten-paper development cohort. Wall time was directly observed only for the adaptive run; n/r means not recorded.
Paper only
Paper + reviews
Judge
A/R rec.
Acc.
AUROC
A/R rec.
Acc.
AUROC
DeepSeek V4 Flash
.30/.80
.467
.530
.10/1.00
.400
.500
Kimi-K3
.30/.40
.333
.460
.70/.80
.733
.820
Inkling NVFP4
.70/.60
.667
.640
.30/.80
.467
.700
GPT-5.6 Sol High
.50/.40
.467
.520
.70/1.00
.800
.920
Appendix
Table 6: Binary Accept/Reject prediction on 15 training papers. “A/R rec.” gives Accept and Reject recall. Reviews contain prose only.
Scholarly peer review studies how experts evaluate research, make publication decisions, and improve the fairness, reliability, and transparency of those processes.