A Missing Piece for Trustworthy AI Reviewers: From Benchmarking Rhetorical Robustness to SciCore Review

Chenguang Wang; Ming Li; Chengrui Fan; Jianpeng Chen; Han Chen; Tianyi Zhou; Dawei Zhou

cs.CLSep 30, 2026

A Missing Piece for Trustworthy AI Reviewers: From Benchmarking Rhetorical Robustness to SciCore Review

Authors: Chenguang Wang, Ming Li, Chengrui Fan, Jianpeng Chen, Han Chen, Tianyi Zhou, Dawei Zhou

Organizations: Virginia Tech · University of Maryland · MBZUAI

Abstract

AI reviewers can assign different judgments to manuscripts that report the same science in different wording, potentially rewarding rhetorical optimization over scientific improvement. We formulate Rhetorical Robustness as the joint requirement of stability across content-preserving rewrites and discrimination across papers. We introduce RobustReview, a controlled full-manuscript benchmark with 1,260 manuscript versions, and evaluate 30 reviewer configurations. The benchmark reveals false robustness, where low rewrite sensitivity coincides with score collapse across papers, and shows that human alignment and rhetorical robustness rank reviewers differently. Moreover, the evaluated content-focused prompting protocol does not consistently improve robustness across backbones. Motivated by these findings, we introduce SciCore, a dual-branch reviewer that averages a full-manuscript judgment with a judgment based on an extracted, structured science core. This design combines manuscript-level assessment with a content-normalized view intended to reduce rhetorical sensitivity. In our primary GPT-5.5 comparison, SciCore achieves a leading joint stability-discrimination profile among the benchmarked reviewers while maintaining competitive human alignment. These results identify rhetorical robustness as a distinct evaluation target and demonstrate the potential of science-core review to improve it.

Figures & tables

Figure 1: Overview of the benchmark and method. RobustReview evaluates rhetorical robustness across controlled rhetorical variants, while SciCore averages a full-manuscript judgment with a content-normalized judgment obtained by extracting and reviewing the science core.

System	Protocol	Human alignment		Within-paper stability		Joint stability-discrimination
System	Protocol	H-MAE $\downarrow$	Spearman $\uparrow$	MAD $\downarrow$	Drift SD $\downarrow$	ICC $\uparrow$	SPR $\uparrow$	Discrim. $\uparrow$
General-purpose LLM reviewers
GPT-5.5 ( OpenAI, 2026 )	Standard	1.294	0.448	0.598	0.956	0.615	0.555	0.649
	Strict	1.078	0.529	0.766	1.202	0.632	0.507	0.672
	Persistent	1.261	0.423	0.476	0.875	0.697	0.586	0.682
GPT-5-mini ( OpenAI, 2025 )	Standard	1.711	0.417	0.543	0.945	0.401	0.390	0.586

Table 1: Main comparison of rhetorical robustness and human alignment. MAD and Drift SD directly measure rewrite-induced within-paper instability. ICC, SPR, and discriminability are joint stability-discrimination metrics: each evaluates within-paper consistency relative to cross-paper variation or separation and should not be interpreted as a between-paper-only measure. Arrows indicate the preferred direction. The best point estimate in each column is shown in bold, the second-best is underlined, and the third-best is italicized.

Configuration	Human alignment		Within-paper stability		Joint stability-discrimination
Configuration	H-MAE $\downarrow$	Spearman $\uparrow$	MAD $\downarrow$	Drift SD $\downarrow$	ICC $\uparrow$	SPR $\uparrow$	Discrim. $\uparrow$
Manuscript-Based Review
Manuscript-Standard	1.294	0.448	0.598	0.956	0.615	0.555	0.649
Manuscript-Strict	1.078	0.529	0.766	1.202	0.632	0.507	0.672
ReconstructReview	1.490	0.133	0.651	1.101	0.623	0.566	0.664
Science-Core Branch

Table 2: Within-backbone analysis of SciCore and its components. Every configuration uses GPT-5.5. MAD and Drift SD are direct within-paper metrics, whereas ICC, SPR, and discriminability jointly relate within-paper consistency to cross-paper variation or separation. The final SciCore row averages the Manuscript-Strict and Core-Adapted scores. The best point estimate in each column is shown in bold and the second-best is underlined.

Extractor Model	Comparison	GPT-5.5 producer			Opus 4.8 producer
Extractor Model	Comparison	Mean	Median	P05	Mean	Median	P05
GPT-5-mini	All matched	0.975	0.977	0.964	0.973	0.976	0.962
GPT-5-mini	Different-paper control	0.668	0.668	0.582	0.670	0.671	0.585
GPT-5.5	All matched	0.978	0.979	0.966	0.978	0.979	0.965
GPT-5.5	Different-paper control	0.616	0.617	0.516	0.619	0.620	0.518
Gemini-3.5-Flash-Lite	All matched	0.963	0.965	0.934	0.961	0.964	0.934

Table 3: Science-core cosine similarity across extractor models and rewrite producers. All matched comparisons pair the science core extracted from each rewritten manuscript with the core extracted from the original manuscript of the same paper; different-paper controls pair it with the 59 nonmatching original cores. Each matched cell contains 600 pairs and each control cell contains 35,400 comparisons per producer. P05 denotes the fifth percentile of pair-level similarities.

Manuscript-Based Review
Configuration	Soundness			Contribution			Presentation †			Confidence
Configuration	MAD $\downarrow$	SPR $\uparrow$	$\rho_{\mathrm{H}}$ $\uparrow$	MAD $\downarrow$	SPR $\uparrow$	$\rho_{\mathrm{H}}$ $\uparrow$	MAD $\downarrow$	SPR $\uparrow$	$\rho_{\mathrm{H}}$ $\uparrow$	MAD $\downarrow$	SPR $\uparrow$	$\rho_{\mathrm{H}}$ $\uparrow$
Manuscript-Standard	0.196	0.547	0.099	0.218	0.587	0.350	0.084	0.371	-0.109	0.232	0.375	-0.013
ReconstructReview	0.112	0.625	0.182	0.214	0.507	0.301	0.240	0.375	-0.094	0.289	0.402	0.052
Science-Core Branch
Core-Standard	0.135	0.645	0.046	0.139	0.593	0.226	0.197	0.524	-0.078	0.000	0.000	–

Table 4: Secondary-score behavior across GPT-5.5 branch configurations. For each secondary score, we report direct within-paper stability (MAD), joint stability-discrimination (SPR), and Spearman correlation with the corresponding mean human score. Presentation is marked as diagnostic because its meaning differs between manuscript-based and direct science-core review. A dash denotes an undefined Spearman correlation due to constant scores; SPR is defined as zero for fully constant outputs.

Backbone	Configuration	Human alignment		Within-paper stability		Joint stability-discrimination
Backbone	Configuration	H-MAE $\downarrow$	Spearman $\uparrow$	MAD $\downarrow$	Drift SD $\downarrow$	ICC $\uparrow$	SPR $\uparrow$	Discrim. $\uparrow$
GPT-5.5	Manuscript-Standard	1.294	0.448	0.598	0.956	0.615	0.555	0.649
GPT-5.5	Core-Adapted	1.461	0.285	0.292	0.650	0.751	0.710	0.695
GPT-5-mini	Manuscript-Standard	1.711	0.417	0.543	0.945	0.401	0.390	0.586
GPT-5-mini	Core-Adapted	1.794	0.290	0.367	0.866	0.408	0.371	0.577
GLM-5.2	Manuscript-Standard	2.044	$-0.034$	1.500	2.341	0.226	0.322	0.558

Table 5: Cross-backbone evaluation of the science-core branch. For each backbone, Manuscript-Standard applies the Standard protocol to the full paper, whereas Core-Adapted uses the same backbone for science-core extraction and review. Bold marks the better result within each backbone pair.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

System	Protocol	Human alignment		Within-paper stability		Joint stability-discrimination
System	Protocol	H-MAE $\downarrow$	Spearman $\uparrow$	MAD $\downarrow$	Drift SD $\downarrow$	ICC $\uparrow$	SPR $\uparrow$	Discrim. $\uparrow$
General-purpose LLM reviewers
GPT-5.5	Standard	0.529	0.099	0.196	0.432	0.520	0.547	0.614
	Strict	0.583	0.273	0.272	0.494	0.494	0.441	0.627
	Persistent	0.549	0.070	0.227	0.481	0.531	0.532	0.619
GPT-5-mini	Standard	0.401	0.214	0.071	0.262	0.061	0.401	0.501

Appendix

Table 6: Full Soundness results for secondary scores. Metrics and column order match Table 1 . The best result in each column is shown in bold, the second-best is underlined, and the third-best is italicized.

System	Protocol	Human alignment		Within-paper stability		Joint stability-discrimination
System	Protocol	H-MAE $\downarrow$	Spearman $\uparrow$	MAD $\downarrow$	Drift SD $\downarrow$	ICC $\uparrow$	SPR $\uparrow$	Discrim. $\uparrow$
General-purpose LLM reviewers
GPT-5.5	Standard	0.441	$-0.109$	0.084	0.287	0.232	0.371	0.510
	Strict	0.393	0.220	0.044	0.208	0.306	0.271	0.515
	Persistent	0.402	0.285	0.085	0.284	0.402	0.440	0.529
GPT-5-mini	Standard	0.410	0.181	0.114	0.324	0.263	0.220	0.526

Appendix

Table 7: Full Presentation results for secondary scores. Metrics and column order match Table 1 . The best result in each column is shown in bold, the second-best is underlined, and the third-best is italicized. A dash denotes an undefined Spearman correlation.

System	Protocol	Human alignment		Within-paper stability		Joint stability-discrimination
System	Protocol	H-MAE $\downarrow$	Spearman $\uparrow$	MAD $\downarrow$	Drift SD $\downarrow$	ICC $\uparrow$	SPR $\uparrow$	Discrim. $\uparrow$
General-purpose LLM reviewers
GPT-5.5	Standard	0.513	0.350	0.218	0.442	0.600	0.587	0.620
	Strict	0.407	0.409	0.181	0.416	0.583	0.520	0.626
	Persistent	0.416	0.493	0.217	0.455	0.629	0.583	0.651
GPT-5-mini	Standard	0.658	0.515	0.160	0.374	0.442	0.435	0.572

Appendix

Table 8: Full Contribution results for secondary scores. Metrics and column order match Table 1 . The best result in each column is shown in bold, the second-best is underlined, and the third-best is italicized.

System	Protocol	Human alignment		Within-paper stability		Joint stability-discrimination
System	Protocol	H-MAE $\downarrow$	Spearman $\uparrow$	MAD $\downarrow$	Drift SD $\downarrow$	ICC $\uparrow$	SPR $\uparrow$	Discrim. $\uparrow$
General-purpose LLM reviewers
GPT-5.5	Standard	0.500	$-0.013$	0.232	0.466	0.272	0.375	0.550
	Strict	0.516	$-0.142$	0.142	0.367	0.193	0.304	0.520
	Persistent	0.466	0.139	0.093	0.304	0.263	0.450	0.517
GPT-5-mini	Standard	0.489	–	0.000	0.000	–	0.000	0.500

Appendix

Table 9: Full Confidence results for secondary scores. Metrics and column order match Table 1 . The best result in each column is shown in bold, the second-best is underlined, and the third-best is italicized. A dash denotes an undefined Spearman correlation or ICC.

Configuration	Human alignment		Within-paper stability		Joint stability-discrimination
Configuration	H-MAE $\downarrow$	Spearman $\uparrow$	MAD $\downarrow$	Drift SD $\downarrow$	ICC $\uparrow$	SPR $\uparrow$	Discrim. $\uparrow$
Single full-manuscript reviews
Manuscript-Standard	1.294	0.448	0.598	0.956	0.615	0.555	0.649
Manuscript-Strict	1.078	0.529	0.766	1.202	0.632	0.507	0.672
Manuscript-Persistent	1.261	0.423	0.476	0.875	0.697	0.586	0.682
Two-score averages

Appendix

Table 11: Full-manuscript averaging controls on RobustReview . All configurations use GPT-5.5 and the same 60 papers with 21 versions each. Protocol names denote full-manuscript reviews. Scores are averaged separately for each original and rewrite before computing the metrics, with no rounding. SciCore averages Manuscript-Strict and Core-Adapted. The three-protocol control averages three review scores. All entries are point estimates.

Metric	Manuscript- Strict	Core-Adapted	Strict + Standard	Strict + Persistent
H-MAE	$-0.006$ $[-0.011,\,-0.001]$	$-0.389$ $[-0.564,\,-0.214]$	$-0.058$ $[-0.087,\,-0.029]$	$+0.006$ $[+0.001,\,+0.011]$
Spearman	$-0.041$ $[-0.068,\,-0.014]$	$+0.203$ $[+0.053,\,+0.366]$	$-0.007$ $[-0.012,\,-0.002]$	$-0.006$ $[-0.011,\,-0.001]$
MAD	$-0.290$ $[-0.348,\,-0.232]$	$+0.184$ $[+0.084,\,+0.285]$	$-0.084$ $[-0.112,\,-0.056]$	$-0.074$ $[-0.101,\,-0.047]$
Drift SD	$-0.515$ $[-0.575,\,-0.455]$	$+0.037$ $[+0.006,\,+0.071]$	$-0.133$ $[-0.167,\,-0.099]$	$-0.143$ $[-0.180,\,-0.106]$
ICC	$+0.143$ $[+0.101,\,+0.185]$	$+0.024$ $[+0.004,\,+0.049]$	$+0.025$ $[+0.010,\,+0.040]$	$+0.017$ $[+0.005,\,+0.029]$
SPR	$+0.145$ $[+0.101,\,+0.189]$	$-0.058$ $[-0.101,\,-0.012]$	$+0.032$ $[+0.015,\,+0.049]$	$+0.017$ $[+0.005,\,+0.029]$

Appendix

Table 12: Paired bootstrap comparisons for SciCore . All configurations use GPT-5.5 and the same 60 complete paper families. Each cell shows the difference ( SciCore minus the column configuration), followed by its marginal 95% percentile interval from 5,000 paired paper-family resamples. Negative differences favor SciCore for H-MAE, MAD, and Drift SD; positive differences favor SciCore for the other metrics.

System	Protocol	Human alignment		Within-paper stability		Joint stability-discrimination
System	Protocol	H-MAE $\downarrow$	Spearman $\uparrow$	MAD $\downarrow$	Drift SD $\downarrow$	ICC $\uparrow$	SPR $\uparrow$	Discrim. $\uparrow$
GPT-5.5	Standard	$[1.067,\,1.531]$	$[0.213,\,0.640]$	$[0.476,\,0.717]$	$[0.809,\,1.075]$	$[0.492,\,0.698]$	$[0.441,\,0.630]$	$[0.614,\,0.679]$
	Strict	$[0.884,\,1.287]$	$[0.305,\,0.700]$	$[0.603,\,0.927]$	$[1.038,\,1.338]$	$[0.520,\,0.709]$	$[0.416,\,0.593]$	$[0.632,\,0.702]$
	Persistent	$[1.037,\,1.491]$	$[0.161,\,0.638]$	$[0.364,\,0.593]$	$[0.715,\,1.017]$	$[0.573,\,0.770]$	$[0.430,\,0.696]$	$[0.640,\,0.716]$
GPT-5-mini	Standard	$[1.387,\,2.048]$	$[0.148,\,0.634]$	$[0.431,\,0.664]$	$[0.821,\,1.057]$	$[0.276,\,0.504]$	$[0.263,\,0.487]$	$[0.553,\,0.616]$
	Strict	$[1.014,\,1.471]$	$[0.142,\,0.616]$	$[0.749,\,1.046]$	$[1.111,\,1.306]$	$[0.283,\,0.481]$	$[0.352,\,0.487]$	$[0.567,\,0.625]$

Appendix

Table 13: Paper-family bootstrap intervals for the primary comparison. Entries are marginal 95% percentile intervals from 5,000 paired resamples of the 60 complete paper families; the corresponding point estimates appear in Table 1 . All original and rewritten versions stay together within each resample.

Rhetorical condition	GPT-5.5 producer			Opus 4.8 producer
Rhetorical condition	Mean	Median	P05	Mean	Median	P05
Novelty stance	0.975	0.977	0.963	0.965	0.976	0.960
Scope framing	0.976	0.976	0.969	0.976	0.977	0.965
Evidence framing	0.972	0.974	0.957	0.973	0.975	0.958
Contribution salience	0.974	0.976	0.964	0.975	0.976	0.963
Technical register	0.976	0.976	0.968	0.974	0.975	0.964

Appendix

Table 14: Condition-level science-core similarity for GPT-5-mini. P05 denotes the fifth percentile across comparisons. The cross-paper baseline compares variant cores with nonmatching original cores.

Rhetorical condition	GPT-5.5 producer			Opus 4.8 producer
Rhetorical condition	Mean	Median	P05	Mean	Median	P05
Novelty stance	0.979	0.979	0.968	0.979	0.981	0.967
Scope framing	0.978	0.978	0.965	0.980	0.980	0.969
Evidence framing	0.976	0.978	0.964	0.977	0.979	0.961
Contribution salience	0.976	0.978	0.963	0.977	0.979	0.965
Technical register	0.978	0.978	0.968	0.976	0.977	0.966

Appendix

Table 15: Condition-level science-core similarity for GPT-5.5. P05 denotes the fifth percentile across comparisons. The cross-paper baseline compares variant cores with nonmatching original cores.

Rhetorical condition	GPT-5.5 producer			Opus 4.8 producer
Rhetorical condition	Mean	Median	P05	Mean	Median	P05
Novelty stance	0.963	0.967	0.939	0.965	0.965	0.940
Scope framing	0.961	0.965	0.927	0.964	0.967	0.944
Evidence framing	0.958	0.961	0.930	0.961	0.963	0.935
Contribution salience	0.959	0.964	0.934	0.961	0.963	0.930
Technical register	0.962	0.965	0.940	0.961	0.964	0.935

Appendix

Table 16: Condition-level science-core similarity for Gemini-3.5-Flash-Lite. P05 denotes the fifth percentile across comparisons. The cross-paper baseline compares variant cores with nonmatching original cores.

$\bm{\alpha}$	Within-paper stability		Joint stability-discrimination			Human alignment
$\bm{\alpha}$	MAD $\downarrow$	Drift SD $\downarrow$	ICC $\uparrow$	SPR $\uparrow$	Discrim. $\uparrow$	H-MAE $\downarrow$	Spearman $\uparrow$
0.00	0.598	0.956	0.615	0.555	0.649	1.294	0.448
0.05	0.577	0.911	0.633	0.566	0.699	1.294	0.441
0.10	0.556	0.867	0.652	0.577	0.699	1.293	0.441
0.15	0.534	0.824	0.671	0.590	0.699	1.292	0.441
0.20	0.513	0.784	0.690	0.603	0.701	1.291	0.441

Appendix

Table 17: Complete science-core weight sensitivity with Manuscript-Standard. All entries are unnormalized numerical results. Arrows indicate the preferred direction.

$\bm{\alpha}$	Within-paper stability		Joint stability-discrimination			Human alignment
$\bm{\alpha}$	MAD $\downarrow$	Drift SD $\downarrow$	ICC $\uparrow$	SPR $\uparrow$	Discrim. $\uparrow$	H-MAE $\downarrow$	Spearman $\uparrow$
0.00	0.766	1.202	0.632	0.507	0.672	1.078	0.529
0.05	0.736	1.143	0.645	0.516	0.712	1.061	0.513
0.10	0.707	1.085	0.659	0.526	0.712	1.044	0.513
0.15	0.677	1.028	0.674	0.538	0.712	1.032	0.513
0.20	0.648	0.972	0.689	0.550	0.713	1.022	0.513

Appendix

Table 18: Complete science-core weight sensitivity with Manuscript-Strict. All entries are unnormalized numerical results. Arrows indicate the preferred direction.

$\bm{\alpha}$	Within-paper stability		Joint stability-discrimination			Human alignment
$\bm{\alpha}$	MAD $\downarrow$	Drift SD $\downarrow$	ICC $\uparrow$	SPR $\uparrow$	Discrim. $\uparrow$	H-MAE $\downarrow$	Spearman $\uparrow$
0.00	0.476	0.875	0.697	0.586	0.682	1.261	0.423
0.05	0.461	0.830	0.712	0.601	0.727	1.269	0.408
0.10	0.447	0.787	0.726	0.617	0.727	1.276	0.408
0.15	0.432	0.746	0.741	0.634	0.727	1.284	0.408
0.20	0.418	0.707	0.755	0.652	0.729	1.291	0.408

Appendix

Table 19: Complete science-core weight sensitivity with Manuscript-Persistent. All entries are unnormalized numerical results. Arrows indicate the preferred direction.

Task	Cost (USD)
RobustReview construction	$6,688.24
Review-only (OpenAI API)	$3,767.59
Review-only (OpenRouter API)	$870.60
SciCore evaluation	$3,043.48
ReconstructReview	$8,193.82
AI Scientist	$391.55

Appendix

Table 20: API expenditure by experimental task. Review-only costs are aggregated by billing route rather than by model. Costs are the recorded expenditures from the provider API consoles.

Automated Peer Review

Automated peer review uses computational systems, especially language models, to assess research papers, generate feedback, and support editorial decisions.

See all 109 papers

How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review
Aug 10, 2026Ming Li, Chenguang Wang, Xirui Li +5Reward Hacking LLM Evaluation
University of Maryland · Virginia Tech · University of Waterloo +1
When AI Reviews Train AI Reviewers: Scientific-Judgment Collapse and Mitigation
Sep 17, 2026Sy-Tuyen Ho, Minghui Liu, Hieu Tran +1Automated Peer Review LLM Reliability
University of Maryland, College Park
Gaming AI-Assisted Peer Reviews Poses New Risks to the Scientific Community
Jun 8, 2026Lin Li, Qi Zhang, Xander Davies +2Adversarial Attacks Adversarial Attacks on LLMs
OATML, University of Oxford · MBZUAI

Aug 10, 2026J/K move · Enter open · S save

How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review

Ming Li, Chenguang Wang, Xirui Li, Xinyue Zeng+4

University of Maryland · Virginia Tech · University of Waterloo+1

Reward Hacking LLM Evaluation Scholarly Peer Review Automated Peer Review

Open paper AskPDF arXiv

As large language models increasingly participate in scientific evaluation, we investigate a potential form of reward hacking: how rhetorical choices shape AI-review judgments when reported scientific content is preserved and how these effects vary across evaluation conditions. We construct a controlled corpus of 4,200 full-paper manuscripts derived from 120 anonymized ICLR 2026 submissions. Two LLM rewriters transform six rhetorical dimensions in opposing directions, and five LLM reviewers evaluate the resulting manuscripts under standard and strict protocols. We also test joint, recursive, and reviewer-guided rewriting. Our results show that rhetorical sensitivity is structured rather than uniform. Evidence framing and novelty stance produce the largest positive-negative contrasts in overall assessment, with scope framing forming a weaker second tier; the remaining dimensions have smaller or less stable effects. This hierarchy persists across human-assessed quality levels, but score movement depends strongly on the AI reviewer's original score: lower scores tend to rise, higher scores tend to fall, and directional contrasts are clearest in the middle ranges. More elaborate workflows do not reliably yield larger gains. Joint rewriting is strongly rewriter-dependent, reviewer guidance does not consistently outperform an unguided second pass, and repeated rewriting yields diminishing, configuration-dependent returns. Across conditions, the rewriter primarily determines the separation between opposing variants, whereas the reviewer determines the magnitude and sign of their score effects. Strict review lowers mean OA by 1.36 points without consistently changing rhetorical sensitivity. These findings identify when rhetorical presentation influences AI scientific review and motivate evaluation systems robust to content-preserving variation in scientific writing.

Abstract

Automated Peer Review

Explore similar work

How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review

When AI Reviews Train AI Reviewers: Scientific-Judgment Collapse and Mitigation

Gaming AI-Assisted Peer Reviews Poses New Risks to the Scientific Community