Breaking the Illusion of Review Reliability under Static Evaluation: SCOPE Fuzzing for LLM-based Scientific Reviewers

Zhuo Chen; Hao Zeng; Jiawei Liu; Guoxiu He; Le Cai; Liu Haotan; Li Wenbo; Yong Huang

cs.AISep 29, 2026

Breaking the Illusion of Review Reliability under Static Evaluation: SCOPE Fuzzing for LLM-based Scientific Reviewers

Authors: Zhuo Chen, Hao Zeng, Jiawei Liu, Guoxiu He, Le Cai, Liu Haotan, Li Wenbo, Yong Huang, +1 more

Organizations: Wuhan University · East China Normal University

Abstract

The rapid growth of submissions and reviewing workload has accelerated the use of large language models (LLMs) in peer review. Prior studies suggest that LLM-based reviewers can penalize content perturbations, such as overclaiming, indicating a certain degree of reliability. Yet these conclusions are largely based on a narrow set of perturbation strategies instantiated with static templates, providing limited evidence of actual reliability. In this paper, we construct a three-level evaluation framework covering perturbations to surface presentation, argumentative logic, and value judgment. Experiments on representative LLM-based reviewers reveal two limitations of static evaluation: stratified vulnerability, where perturbation effects depend on whether the paper's original review score is high or low, and perturbation undercoverage, where a single template misses vulnerabilities exposed by diverse realizations. To address these limitations, we propose SCOPE-Fuzzer, a strategy-aware fuzzer that combines feedback-driven strategy selection with adaptive mutation of paper content. By iteratively probing reviewers with dynamic perturbations, SCOPE-Fuzzer consistently uncovers vulnerabilities overlooked by static evaluation and other baselines.

Figures & tables

Figure 1: For reliability against content perturbations, aggregate results from static evaluation suggest that LLM-based reviewers can penalize perturbations, while our analysis reveals their stratified vulnerability: papers with lower initial scores are more susceptible to perturbations. Also, static evaluation is inefficient to discover effective instantiations in the perturbation space.

Table 2

Figure 2: Multi-probe evaluation with TREAP perturbations.

Figure 4: Fuzzing effectiveness of SCOPE and baselines.

Figure 5: Ablation of the selector and the mutator.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Framework of TriScope Review Evaluation Against Perturbation (Treap)

Figure 7: Case study for lexical syntactic complexification, verbosity increase and overclaiming.

Figure 8: Case study for logic adjustment, value alignment and empathy elicitation.

Figure 9: Human validation results across the six TREAP strategies. Top: the proportion of retained responses agreeing that the paired texts preserve the same core research content; error bars show 95% Wilson confidence intervals. Bottom: mean academic-plausibility ratings for the original and perturbed texts on a five-point scale.

Panel of Figure 10 — Figure 10: Prompt template used for LLM-based paper review.

Review System	Mean Benign	Median	Mode
QWEN3	7.65	8	8
GPT4o-mini	7.84	8	8
SEA	5.84	6	6
Openreviewer	4.47	5	3
DeepReview	5.56	6	6
MARG	4.96	5	5

Appendix

Table 4: Statistic results of benign review scores across review systems.

Reviewer	Metric	Overclaiming		Logic Adjustment
		High	Low	High	Low
QWEN3	$PSR_{net}$	-14.20	+26.09	-15.88	+29.41
QWEN3	$PG$	-0.15	0.26	-0.16	0.31
GPT-4o-mini	$PSR_{net}$	-4.72	+34.78	-3.94	+30.43
GPT-4o-mini	$PG$	-0.04	0.34	-0.03	0.30
SEA	$PSR_{net}$	-13.75	+42.37	-13.75	+42.37

Appendix

Table 5: SV under reasoning-level perturbations.

Reviewer	Metric	Value Alignment		Empathy Elicitation
		High	Low	High	Low
QWEN3	$PSR_{net}$	-19.07	+25.72	-18.45	+33.33
QWEN3	$PG$	-0.20	0.27	-0.20	0.35
GPT-4o-mini	$PSR_{net}$	-3.94	+26.09	-3.15	+26.09
GPT-4o-mini	$PG$	-0.03	0.26	-0.03	0.26
SEA	$PSR_{net}$	-20.42	+36.50	-13.79	+34.92

Appendix

Table 6: SV under value-level perturbations.

$\mathrm{PSR}_{\mathrm{net}}$ (%) $\uparrow$
Reviewer	Method	Budget
		1	2	3	4	5	6	7	8	9	10
SEA	PAA	-2.15	+20.43	+25.80	+32.26	+36.56	+36.56	+37.64	+38.71	+40.86	+41.94
	MPS	-5.38	+13.98	+29.03	+31.18	+36.55	+38.71	–	–	–	–
	SPD	-11.83	+11.83	+22.58	+29.03	+36.56	+38.71	+39.79	+41.94	+43.01	+43.01
	SCOPE	+15.05 *	+27.96 ***	+38.71	+43.01	+48.39	+51.61	+52.69	+53.76	+54.84	+58.06

Appendix

Table 7: Main fuzzing results under different budgets.

\mathrm{PSR}_{\mathrm{net}}=\mathrm{PSR}-\mathrm{PFR}

is reported in percentage points. Higher

\mathrm{PSR}_{\mathrm{net}}

and PG indicate stronger perturbation effects.

Reviewer	Baseline	$\Delta$ PG	$p_{\mathrm{adj}}$
SEA	PAA	$\mathbf{+0.172}$	$\mathbf{<0.01}$
SEA	MPS	$\mathbf{+0.215}$	$\mathbf{<0.01}$
SEA	SPD	$\mathbf{+0.225}$	$\mathbf{<0.01}$
OpenReviewer	PAA	$+0.150$	$0.213$
OpenReviewer	MPS	$\mathbf{+0.333}$	$\mathbf{<0.01}$
OpenReviewer	SPD	$+0.129$	$0.255$

Appendix

Table 8: Paired comparison of SCOPE against the baselines on perturbation gain (PG).

\Delta\mathrm{PG}=\mathrm{PG}_{\textsc{scope}}-\mathrm{PG}_{\text{baseline}}

, where a positive value favors SCOPE. The reported effect is the mean paper-level paired PG difference.

p

denotes the two-sided paired permutation-test

p

-value across the baseline comparisons. Bold indicates a statistically significant improvement.

Figure 11: Ablation study on the heuristic selection methods.

Explore similar work

LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers
May 25, 2026Lingyao Li, Junjie Xiong, Changjia Zhu +5Peer Review Divergence
University of South Florida · Missouri University of Science and Technology · University of Alabama +3
LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challenges
Jun 23, 2026Thi Huyen Nguyen, Zahra AhmadiPeer Review Large Language Model Evaluation
L3S Research Center, Leibniz University Hannover Hannover, Germany · Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed) Hannover, Germany
SafeReview: Defending LLM-based Review Systems Against Adversarial Hidden Prompts
Apr 29, 2026Yuan Xin, Yixuan Weng, Minjun Zhu +5Peer Review Adversarial Prompts
1CISPA · 2Westlake University · 3Southern University of Science and Technology +1

May 25, 2026, cs.CLJ/K move · Enter open · S save

LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers

Lingyao Li, Junjie Xiong, Changjia Zhu, Runlong Yu+4

University of South Florida · Missouri University of Science and Technology · University of Alabama+3

Peer Review Divergence

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used in academic peer review, yet their reliability, alignment with human judgment, and robustness to adversarial attacks remain poorly understood. We present a systematic benchmark of LLM-as-a-Reviewer on 898 papers stratified from NeurIPS and ICLR, evaluating 12 LLMs along three axes: rating calibration, divergence from human reviewers, and resistance to prompt injection embedded via an invisible font-mapping attack. We find that LLMs systematically overrate weaker submissions and diverge from humans in topical emphasis, under-flagging Clarity and over-flagging Reproducibility, while producing reviews two to three times longer with lower lexical diversity and a more standardized vocabulary. Prompt injection remains highly effective. Simple hidden instructions can promote low-scoring papers to acceptance-level ratings in a substantial fraction of cases, with effectiveness varying sharply across model families. While LLMs offer utility in structuring evaluations, their integration into peer review requires safeguards against both intrinsic biases and adversarial risks.

Abstract

Explore similar work

LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers

LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challenges

SafeReview: Defending LLM-based Review Systems Against Adversarial Hidden Prompts