Evaluating LLM-Based Grant Proposal Review via Structured Perturbations

William Thorne; Joseph James; Yang Wang; Chenghua Lin; Diana Maynard

cs.CLMar 9, 2026

Evaluating LLM-Based Grant Proposal Review via Structured Perturbations

Authors: William Thorne, Joseph James, Yang Wang, Chenghua Lin, Diana Maynard

Organizations: University of Sheffield, UK · University of Manchester, UK

Abstract

As AI-assisted grant proposals outpace manual review capacity in a kind of ``Malthusian trap'' for the research ecosystem, this paper investigates the capabilities and limitations of LLM-based grant reviewing for high-stakes evaluation. Using six EPSRC proposals, we develop a perturbation-based framework probing LLM sensitivity across six quality axes: funding, timeline, competency, alignment, clarity, and impact. We compare three review architectures: single-pass review, section-by-section analysis, and a 'Council of Personas' ensemble emulating expert panels. The section-level approach significantly outperforms alternatives in both detection rate and scoring reliability, while the computationally expensive council method performs no better than baseline. Detection varies substantially by perturbation type, with alignment issues readily identified but clarity flaws largely missed by all systems. Human evaluation shows LLM feedback is largely valid but skewed toward compliance checking over holistic assessment. We conclude that current LLMs may provide supplementary value within EPSRC review but exhibit high variability and misaligned review priorities. We release our code and any non-protected data.

Figures & tables

Axis	Perturbation Strategy	Targeted Criterion
Funding	Inflating/lowering budgets; removing cost justifications; misaligning resource allocation with project priorities.	Value for money; compliance with UKRI financial policy.
Timeline	Extending periods beyond call limits; unrealistic task compression; misaligning milestones with logical work progression.	Project feasibility; operational justification.
Competency	Removing/replacing key personnel; weakening evidence of technical skills in the team capability section.	Demonstration of requisite skills and leadership.
Alignment	Modifying opportunity aims; switching “What we’re looking for” sections; introducing cross-disciplinary mandates.	Strategic fit to call; adherence to funding body values.
Clarity	Removing acronym expansions; introducing vagueness in methods; removing novelty markers; deleting factual background.	Accessibility; technical comprehensibility; academic rigour; specificity.
Impact	Replacing key stakeholders with irrelevant parties; modifying outcome scope; removing long-term/short-term outcomes.	Contribution to field; stakeholder engagement.

Table 1: Summary of perturbation axes, strategies, and their mapping to EPSRC assessment criteria. Examples and further details can be found in subsection A.4 and subsection A.5 .

	Baseline			Section-Level			Council
Perturbation Axis	C	P	I	C	P	I	C	P	I
Funding (n=1688)	159	20	440	220	40	358	115	51	285
Competency (n=889)	28	12	353	32	10	240	15	7	192
Alignment (n=953)	122	2	222	183	1	163	84	0	176
Clarity (n=1859)	6	0	668	99	5	571	12	4	494
Impact (n=639)	10	5	221	71	5	157	7	0	163

Table 2: Perturbation identification rates across review systems. C = Correct (perturbation identified with negative sentiment), P = Partial (ambiguous or indirect identification), I = Incorrect (missed or positive sentiment). Verdicts are determined by majority vote across three independent judge models.

Figure 1: Detection scores across review systems and perturbation categories. Darker cells indicate higher detection rates. The section-level system shows strongest performance on alignment and impact perturbations, while all systems fail on clarity-based changes.

	$\sigma^{2}_{\text{total}}$	$\sigma^{2}_{\text{between}}$	$\sigma^{2}_{\text{within}}$	ICC ( $\uparrow$ )
Baseline	0.87	0.13	0.80	0.14
Council	0.91	0.11	0.88	0.11
Section-Level	0.88	0.49	0.49	0.50

Table 3: Variance decomposition across review systems. ICC (intra-class correlation) measures the proportion of variance attributable to true differences between proposals versus noise from repeated evaluation.

	Human		Baseline		Council		Section-Level
Category	Ret	Ctr	Ret	Ctr	Ret	Ctr	Ret	Ctr
Alignment	79.6%	2.8%	68.2%	0.0%	71.8%	1.9%	80.0%	0.0%
Competency	67.7%	2.1%	76.8%	1.8%	66.7%	1.6%	52.2%	0.0%
Ethics	97.7%	0.0%	95.0%	0.0%	94.0%	1.2%	90.0%	0.0%
Funding	81.3%	8.0%	84.0%	0.0%	89.5%	4.8%	100.0%	0.0%
Impact	77.8%	1.2%	85.3%	0.0%	73.8%	1.0%	66.7%	0.0%

Table 4: Retained (Ret) and Contradiction (Ctr) Rates of claims. Retained rate is computed from the exclusive set , where consensus claims are removed, isolating claims unique to each reviewer. Higher retained percentages indicate greater reviewer-specific contribution (unique claims), whereas lower values suggest substantial overlap

Figure 2: Table 11 shows the scale for severity. The dashed line on agreement indicates the neutral agreement.

	Review	Spearman $\rho$	Significance
Agreement	Human	0.258	0.014
	Baseline	0.431	$<0.001$
	Council	0.732	$<0.001$
	Section-Level	0.613	0.006
Severity	Human	0.250	0.018
	Baseline	0.306	0.008

Table 5: Spearman correlations between valence and outcome measures by method.

	Negative	Neutral	Positive
Human	17.4%	33.1%	49.5%
Baseline	41.7%	20.8%	37.5%
Council	22.2%	25.9%	52.0%
Section-Level	48.1%	12.3%	39.6%

Table 6: Valence distribution (percentage of total claims per source).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Human	LLM	Total
ALIGNMENT	13	11	24
COMPETENCY	30	26	56
ETHICS	12	21	33
FUNDING	38	40	78
IMPACT	52	36	88
CLARITY	144	54	198

Appendix

Table 7: Claims used for human evaluation.

Bracket and Example Removal
Category / Variant	Example Content
Original	The framework supports multiple modalities (such as text, image, and audio) to ensure versatility in downstream tasks.
	The system provides several security features (including end-to-end encryption and two-factor authentication) for user protection.
Brackets Removed	The framework supports multiple modalities to ensure versatility in downstream tasks.
	The system provides several security features for user protection.
Bracket + Example Removed	The framework supports multiple modalities.

Appendix

Table 8: Examples of perturbations.

	Summary Totals			Directly Allocated			Staffing	Directly Incurred
Variant	Full Funding	Org Cont.	Applied	Staff	Estates	Other	%FTE	Staff	Equip.	Travel	Other
Original	£25,000	£5,000	£20,000	£8,000	£2,000	£1,000	40%	£5,000	£2,000	£1,000	£1,000
High Org Cont.	£25,000	£21,000	£4,000	£1,500	£500	£250	40%	£1,000	£250	£250	£250
Low Eq / High Other	£25,000	£5,000	£20,000	£8,000	£2,000	£1,000	40%	£5,000	£100	£1,000	£2,900
Low Staff Cost	£25,000	£5,000	£20,000	£480	£4,520	£2,000	40%	£8,000	£2,000	£1,500	£1,500
Low Staff FTE	£25,000	£5,000	£20,000	£8,000	£2,000	£1,000	1%	£5,000	£2,000	£1,000	£1,000

Appendix

Table 9: Funding perturbations with constant Full Funding across all variants.

Axis	Component	Sub-component	Aspects
1. Competency	Team Capability	Experience & Track Record	expertise_domain, track_record_outputs, track_record_leadership, career_stage_appropriateness
		Skills & Expertise	skill_coverage, skill_gaps, complementarity
		Leadership & Management	communication_ability, team_development, project_management, cross_sector_influence
2. Funding	Resources & Justification	Resource Specification	staff_justification, travel_justification, compute_resources, resource_completeness
		Appropriateness	staff_time_realistic, resource_alternatives
		Value for Money	outcome_proportionality, impact_optimization

Appendix

Table 10: Taxonomy of grant proposals

Rating	Definition	Examples
None	Purely factual or administrative. No bearing on scientific merit or deliverability.	“The first letter of the sentence was not capitalised.”
Little	Minor issues that are easily correctable or do not affect core assessment criteria.	“Figure 3 is difficult to read.”
Some	Valid observations affecting secondary criteria. Would influence score by $\pm 0.5$ points.	“The budget justification for travel could be more detailed.”
Substantial	Significant strengths or weaknesses directly affecting Quality or Importance. Would shift score by $\pm 1$ – $2$ points.	“The proposed methodology represents a genuine advance over current techniques.”
Pivotal	Fundamental issues affecting viability or exceptional strengths. Changes fundable/non-fundable status.	“The underlying theoretical framework contradicts established principles.”

Appendix

Table 11: Impact rating definitions for different levels of claim severity.

Review	Effort	Wall Clock	Input	Output	Total
Baseline	Low	00:20:37	21,088,675	437,826	21,526,501
	Medium	00:34:04	21,088,675	1,157,919	22,246,594
	High	01:37:06	21,088,675	3,628,537	24,717,212
Section-Level	Low	02:04:33	27,231,048	8,532,425	35,763,473
	Medium	02:40:23	27,828,422	11,221,317	39,049,739
	High	04:22:37	28,094,907	18,489,158	46,584,065

Appendix

Table 12: Wall-clock time of GPT-OSS-20B and token breakdown from producing reviews for all original proposals and perturbations at each reasoning level. Each was generated 5 times to assess variability (total: 910). Council on high reasoning was stopped at 6 hours (25%) completion due to excessive runtime cost.

Figure 3: Table 11 shows the scale for severity. The dashed line on agreement indicates the neutral stance (2 on the scale).

Automated Peer Review

Automated peer review uses computational systems, especially language models, to assess research papers, generate feedback, and support editorial decisions.

See all 109 papers

LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers
May 25, 2026Lingyao Li, Junjie Xiong, Changjia Zhu +5LLM Evaluation LLM-as-a-Judge
University of South Florida · Missouri University of Science and Technology · University of Alabama +3
How Closely Do LLM Reviews Align with Human Peer Review?
Aug 4, 2026Abraham Camelo-Guerrero, Jairo Diaz-RodriguezLLM Evaluation Human-AI Agreement
School of Information Technology York University Toronto, Ontario M3J 1P3 · Department of Mathematics and Statistics York University Toronto, Ontario M3J 1P3
ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agents
Apr 15, 2026Zhuofeng Li, Yi Lu, Dongfu Jiang +7Automated Peer Review Scholarly Peer Review
Texas A&M University · University of Waterloo · UC San Diego +2

May 25, 2026J/K move · Enter open · S save

LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers

Lingyao Li, Junjie Xiong, Changjia Zhu, Runlong Yu+4

University of South Florida · Missouri University of Science and Technology · University of Alabama+3

LLM Evaluation LLM-as-a-Judge Adversarial Attacks on LLMs LLM Reliability Language Model Bias Evaluation Automated Peer Review

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used in academic peer review, yet their reliability, alignment with human judgment, and robustness to adversarial attacks remain poorly understood. We present a systematic benchmark of LLM-as-a-Reviewer on 898 papers stratified from NeurIPS and ICLR, evaluating 12 LLMs along three axes: rating calibration, divergence from human reviewers, and resistance to prompt injection embedded via an invisible font-mapping attack. We find that LLMs systematically overrate weaker submissions and diverge from humans in topical emphasis, under-flagging Clarity and over-flagging Reproducibility, while producing reviews two to three times longer with lower lexical diversity and a more standardized vocabulary. Prompt injection remains highly effective. Simple hidden instructions can promote low-scoring papers to acceptance-level ratings in a substantial fraction of cases, with effectiveness varying sharply across model families. While LLMs offer utility in structuring evaluations, their integration into peer review requires safeguards against both intrinsic biases and adversarial risks.

Abstract

Automated Peer Review

Explore similar work

LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers

How Closely Do LLM Reviews Align with Human Peer Review?

ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agents