Style over Substance? Linguistic Bias in LLM-based Peer Review Copy title

Sai Suresh Macharla Vasu; Sumedh Santosh Joshi; Philipp Christmann; Ivaxi Sheth; Ruta Binkyte; Koel Dutta Chowdhury; Raouf Kerkouche; Hui-Po Wang

cs.CLOct 26, 2026

Style over Substance? Linguistic Bias in LLM-based Peer Review Copy title

Authors: Sai Suresh Macharla Vasu, Sumedh Santosh Joshi, Philipp Christmann, Ivaxi Sheth, Ruta Binkyte, Koel Dutta Chowdhury, Raouf Kerkouche, Hui-Po Wang,

Organizations: CISPA Helmholtz Center for Information Security · German Research Center for Artificial Intelligence (DFKI) · University of Technology Nuremberg · INRIA

Abstract

The number of paper submissions to major AI conferences has increased dramatically in recent years, substantially increasing the peer-review workload of researchers. At the same time, large language models (LLMs) are increasingly used to support peer review, both by individual reviewers and by conference organizations. While LLM capabilities have improved considerably, their evaluations may be influenced by linguistic characteristics that are unrelated to scientific content. In this work, we introduce a controlled experimental framework for studying such effects by systematically perturbing the linguistic style of research papers. Specifically, we rewrite papers to exhibit (i) linguistic features associated with native or non-native English writing, and (ii) high- or low-confidence linguistic features. To construct these interventions, we derive fine-grained rewriting instructions grounded in the linguistic literature and apply them using LLMs. Crucially, our interventions are designed to preserve the underlying scientific content. We then evaluate the resulting papers using different LLM reviewers. Our experiments expose a critical vulnerability in LLM-based peer review: reviewers systematically reward confident rhetoric independent of scientific content, raising serious concerns about the robustness of their assessments.

Figures & tables

Figure 1: Visualization of our controlled framework for linguistic rewriting of research papers.

Rewrite	Example
Original	Systematically studying such effects is challenging, as naturally occurring papers with different linguistic styles also differ in their research content and quality. We therefore propose a controlled experimental framework for perturbing a research paper along a specific linguistic dimension, while preserving its underlying research content.
Native	Systematically investigating such effects is no straightforward matter , since naturally occurring papers written in divergent linguistic styles also tend to differ in their research content and quality. We therefore put forward a controlled experimental framework for perturbing a research paper along one specific linguistic dimension while leaving its underlying research content intact .
Non-native	To study such effects in a systematic way is difficult , because naturally occurring papers that differ in linguistic style also differ in their research content and quality. We therefore propose a controlled experimental framework which perturbs a research paper along one specific linguistic dimension, while the underlying research content is preserved.
High-confidence	Systematically studying such effects poses a clear methodological challenge , since naturally occurring papers with different linguistic styles invariably differ in their research content and quality as well. We therefore introduce a controlled experimental framework that perturbs a research paper along a specific linguistic dimension while preserving its underlying research content.
Low-confidence	A systematic study of such effects appears challenging, in part because naturally occurring papers differing in linguistic style tend also to differ in their research content and quality. We therefore propose a controlled experimental framework intended to perturb a research paper along a specific linguistic dimension, while seeking to preserve its underlying research content.

Table 1: Example rewrites of the same scientific passage, taken from our own introduction, along the two linguistic dimensions. Colored spans highlight selected linguistic changes induced by the language-proficiency and expressed-confidence interventions. Further examples from our methodology and results sections are provided in Appendix B .

Rewrite	Language Proficiency			Expressed Confidence			Content Integrity
	MTLD	AG	GF		Boosters/1K words	Hedges/1K words	Semantic Similarity	TF-IDF
Original	60.8	7.7	17.0		1.9	1.5	–	–
Native	63.9	8.4	17.4		1.7	2.3	0.997	0.992
Non-native	58.0	7.2	16.8		1.6	1.5	0.998	0.995
High-confidence	60.8	7.8	17.2		3.2	0.8	0.999	0.998
Low-confidence	62.9	7.7	17.2		0.7	13.6	0.998	0.993

Table 2: Validation of the language-proficiency and confidence rewrites. For language proficiency, we report lexical diversity ( MTLD ), lexical sophistication ( AG ), and textual complexity ( GF ). For confidence, we report the average frequency of boosters and hedges per 1,000 words. For both rewrite dimensions, we additionally report our measures for content integrity based on Semantic Similarity and lexical similarity ( TF-IDF ) to the original paper. Highlighted values mark the higher value within each pair of rewrites, for the measures targeted by the respective rewrite dimension.

Gpt-5.5	Claude Sonnet 4.6	Gemini 3.5 Flash	Gemma3	Qwen3 VL	Glm4.7
0.48	0.49	1.24	0.52	0.49	0.85

Table 3: Pooled within-paper standard deviation of reviewer scores across repeated generations.

					Decision Flips (%)
Comparison	Reviewer	Preference (%)	Ratio	$p$	First $\uparrow$	Second $\uparrow$	Average Scores
Native vs. Non-native	Gpt-5.5	17 / 14 (69)	1.3 $\times$	0.15	6.2	3.0	3.6 / 3.6
	Claude Sonnet 4.6	19 / 21 (60)	0.9 $\times$	0.67	7.5	8.5	4.5 / 4.5
	Gemini 3.5 Flash	33 / 32 (36)	1.0 $\times$	0.78	11.1	10.9	4.7 / 4.7
	Gemma3	24 / 19 (57)	1.2 $\times$	0.13	0.2	0.0	7.2 / 7.2
	Qwen3 VL	19 / 17 (63)	1.1 $\times$	0.51	0.0	0.0	8.0 / 8.0

Table 4: Pairwise reviewer preferences for native vs. non-native and high-confidence vs. low-confidence rewrites (504 papers). Preference : % of papers where the first / second version scores higher (ties in parentheses); Ratio : first-to-second version preference ratio;

p

: two-sided sign test, significant values (

p<0.05

) in bold. Decision Flips : % of papers where only the first / second version is accepted. Average Scores : mean overall rating of the first / second version. Highlighted values indicate the preferred direction.

Comparison	Condition	Model	Preference (%)	Ratio	$p$
Native vs. Non-native	Given High-confidence	Gemma3	27 / 19 (54)	1.4 $\times$	0.24
		Qwen3 VL	20 / 17 (63)	1.2 $\times$	0.66
		Glm4.7	39 / 29 (33)	1.4 $\times$	0.19
	Given Low-confidence	Gemma3	32 / 19 (49)	1.7 $\times$	0.06
		Qwen3 VL	17 / 25 (58)	0.7 $\times$	0.27
		Glm4.7	37 / 30 (33)	1.2 $\times$	0.45

Table 5: Pairwise reviewer preferences across the combined linguistic rewrites (126 papers). Preference : % of papers where the first / second version scores higher (ties in parentheses); Ratio : first-to-second version preference ratio;

p

: two-sided sign test, significant values (

p<0.05

) in bold. Highlighted values indicate the preferred direction.

Figure 2: Mean aspect-level review polarity. For each paper and aspect, we score review texts as #positive mention – #negative mentions, compute within-paper difference between rewrites and average over all papers. Left: native

-

non-native . Right: high-confidence

-

low-confidence . Bars show the top-10 aspects with largest

|\Delta|

. Error bars are 95% bootstrap CIs.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Rewrite	Example
Original	The validity of our experimental setup depends on two central properties of the generated rewrites. First, the rewrite should preserve the scientific content of the original paper. Second, the rewriting process should systematically induce the intended linguistic characteristics. We therefore evaluate both content integrity and linguistic change for each rewriting dimension.
Native	The validity of our experimental setup rests on two central properties of the generated rewrites. First, a rewrite ought to preserve the scientific content of the original paper. Second, the rewriting procedure should reliably induce the intended linguistic characteristics. We therefore assess both content integrity and linguistic change for each rewriting dimension.
Non-native	The validity of our experimental setup is based on two main properties of the generated rewrites. First, the rewrite should keep the scientific content of the original paper. Second, the rewriting process should induce the intended linguistic characteristics in a systematic way . For this reason , we evaluate both content integrity and linguistic change for each rewriting dimension.
High-confidence	The validity of our experimental setup rests on two central properties of the generated rewrites. First, the rewrite preserves the scientific content of the original paper. Second, the rewriting process systematically induces the intended linguistic characteristics. We therefore evaluate both content integrity and linguistic change for each rewriting dimension.
Low-confidence	The validity of our experimental setup appears to rest on two properties of the generated rewrites. First, the rewrite should, as far as possible , preserve the scientific content of the original paper. Second, the rewriting process should plausibly induce the intended linguistic characteristics in a reasonably systematic manner . We therefore attempt to assess both content integrity and linguistic change for each rewriting dimension.

Appendix

Table 6: Example rewrites of a passage from our methodology section along the two linguistic dimensions. Colored spans highlight selected linguistic changes induced by the language-proficiency and expressed-confidence interventions.

Rewrite	Example
Original	For language proficiency, decision flips are largely balanced, which is consistent with the absence of a systematic score preference. For confidence, flips predominantly favor the high-confidence version. We observe the most drastic effect for Claude Sonnet 4.6 : in 21.0% of reviews, Claude Sonnet 4.6 accepts the high-confidence version and reject the low-confidence version, with only 2.8% of papers accepted only for the low-confidence rewrite.
Native	With respect to language proficiency, decision flips are for the most part balanced, a pattern consistent with the absence of any systematic score preference. For confidence, by contrast , flips predominantly favor the high-confidence version. The most pronounced effect emerges for Claude Sonnet 4.6 : in 21.0% of reviews, Claude Sonnet 4.6 accepts the high-confidence version while rejecting the low-confidence one, whereas a mere 2.8% of papers are accepted solely in their low-confidence rewrite.
Non-native	For language proficiency, decision flips are mostly balanced, which is in line with the absence of a systematic score preference. For confidence, flips mostly favor the high-confidence version. The most drastic effect is observed for Claude Sonnet 4.6 : in 21.0% of reviews, Claude Sonnet 4.6 accepts the high-confidence version and rejects the low-confidence version, and only 2.8% of papers are accepted only for the low-confidence rewrite.
High-confidence	For language proficiency, decision flips are largely balanced, which is fully consistent with the absence of a systematic score preference. For confidence, flips clearly favor the high-confidence version. The most drastic effect occurs for Claude Sonnet 4.6 : in 21.0% of reviews, Claude Sonnet 4.6 accepts the high-confidence version and rejects the low-confidence version, with only 2.8% of papers accepted only for the low-confidence rewrite.
Low-confidence	For language proficiency, decision flips appear to be largely balanced, which would seem consistent with the absence of a systematic score preference. For confidence, flips tend to favor the high-confidence version. The most drastic effect appears to occur for Claude Sonnet 4.6 : in 21.0% of reviews, Claude Sonnet 4.6 accepts the high-confidence version and rejects the low-confidence version, with only 2.8% of papers accepted only for the low-confidence rewrite.

Appendix

Table 7: Example rewrites of a passage from our results section along the two linguistic dimensions. Colored spans highlight selected linguistic changes induced by the language-proficiency and expressed-confidence interventions.

					Decision Flips (%)
Comparison	Prompt	Preference (%)	Ratio	$p$	First $\uparrow$	Second $\uparrow$	Avg. Scores
Native vs. Non-native	Default	17 / 21 (62)	0.8 $\times$	0.47	7.1	7.9	4.3 / 4.3
	Brief	19 / 14 (67)	1.3 $\times$	0.44	7.9	4.8	4.5 / 4.4
	Minimal	15 / 17 (67)	0.9 $\times$	0.76	7.1	4.8	4.6 / 4.6
	Bare	14 / 14 (71)	1.0 $\times$	1.00	3.2	4.8	4.7 / 4.7
	Biased	44 / 10 (45)	4.3 $\times$	$<$ 0.001	19.0	4.0	4.8 / 4.4

Appendix

Table 8: Prompt sensitivity of Claude Sonnet 4.6 on a subset of 126 papers. Columns as in Table 4 ; significant

p

-values (

p<0.05

) in bold. Rows below the rule use the Biased prompt, which explicitly instructs the reviewer to weight linguistic style.

				Decision Flips (%)
Prompt	Reviewer	Preference (%)	Ratio	First $\uparrow$	Second $\uparrow$	Average Scores
Default	Gpt-5.5	16 / 13 (71)	1.2 $\times$	3.2	4.0	3.5 / 3.5
	Claude Sonnet 4.6	17 / 21 (62)	0.8 $\times$	7.1	7.9	4.3 / 4.3
	Gemini 3.5 Flash	34 / 30 (36)	1.1 $\times$	9.5	11.9	4.8 / 4.7
	Gemma3	31 / 19 (50)	1.6 $\times$	0.8	0.0	7.2 / 7.1
	Qwen3 VL	19 / 13 (67)	1.4 $\times$	0.0	0.0	7.9 / 7.8

Appendix

Table 9: native vs. non-native preferences under the Default and the Biased reviewer prompt (Figure 7 ), on the same 126 papers. Columns as in Table 4 ; † marks ratios that are statistically significant (two-sided sign test,

p<0.05

Panel of Figure 3 — Figure 3: Preference ratios per review criterion for native vs. non-native (left) and high-confidence vs. low-confidence (right) rewrites. Only ratios that are statistically significant (two-sided sign test, $p<0.05$ ) are shown; grey cells indicate no significant preference.

Panel of Figure 4 — Figure 4: Paired prevalence shifts (pp) for positive and negative aspect mentions for native $-$ non-native .

Panel of Figure 5 — Figure 5: Paired prevalence shifts (pp) for positive and negative aspect mentions for high-confidence $-$ low-confidence .

Panel of Figure 6 — Figure 6: The prompt used for the LLM reviewer. The prompt specifies the review criteria, defines the likert scale, and emphasizes standard responsibilities for reviewers at major machine learning conferences. The prompt also specifies the expected output format to ensure seamless integration into our framework.

Panel of Figure 7 — Figure 7: The review prompt used for our analysis simulating a biased LLM reviewer.

Panel of Figure 8 — Figure 8: Rewriting prompt for native -associated language.

Panel of Figure 9 — Figure 9: Rewriting prompt for non-native -associated language.

Panel of Figure 10 — Figure 10: Rewriting prompt for high-confidence language.

Panel of Figure 11 — Figure 11: Rewriting prompt for low-confidence language.

Panel of Figure 12 — Figure 12: Rewriting prompt for the native & high-confidence interaction condition.

Panel of Figure 13 — Figure 13: Rewriting prompt for the native & low-confidence interaction condition.

Panel of Figure 14 — Figure 14: Rewriting prompt for the non-native & high-confidence interaction condition.

Panel of Figure 15 — Figure 15: Rewriting prompt for the non-native & low-confidence interaction condition.

Panel of Figure 16 — Figure 16: Prompt for annotating aspects within the review prose.

Linguistic Bias in Language Models

Linguistic bias in language models studies how language varieties, identity cues, and cultural context shape model outputs and judgments.

See all 90 papers

LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers
May 25, 2026Lingyao Li, Junjie Xiong, Changjia Zhu +5LLM Evaluation LLM-as-a-Judge
University of South Florida · Missouri University of Science and Technology · University of Alabama +3
Judging a Review by its Cover: A Reliability Analysis of LLM-based Peer Review Evaluation Metrics
Sep 20, 2026Shakiba Amirshahi, Sajad Ebrahimi, Hai Son Le +2Automated Peer Review LLM-as-a-Judge
Reviewerly Toronto, Ontario, Canada · University of California, Berkeley Berkeley, California, United States · University of Toronto Toronto, Ontario, Canada
LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challenges
Jun 23, 2026Thi Huyen Nguyen, Zahra AhmadiScholarly Peer Review LLM Security
L3S Research Center, Leibniz University Hannover Hannover, Germany · Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed) Hannover, Germany

May 25, 2026J/K move · Enter open · S save

LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers

Lingyao Li, Junjie Xiong, Changjia Zhu, Runlong Yu+4

University of South Florida · Missouri University of Science and Technology · University of Alabama+3

LLM Evaluation LLM-as-a-Judge Adversarial Attacks on LLMs LLM Reliability Language Model Bias Evaluation Automated Peer Review

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used in academic peer review, yet their reliability, alignment with human judgment, and robustness to adversarial attacks remain poorly understood. We present a systematic benchmark of LLM-as-a-Reviewer on 898 papers stratified from NeurIPS and ICLR, evaluating 12 LLMs along three axes: rating calibration, divergence from human reviewers, and resistance to prompt injection embedded via an invisible font-mapping attack. We find that LLMs systematically overrate weaker submissions and diverge from humans in topical emphasis, under-flagging Clarity and over-flagging Reproducibility, while producing reviews two to three times longer with lower lexical diversity and a more standardized vocabulary. Prompt injection remains highly effective. Simple hidden instructions can promote low-scoring papers to acceptance-level ratings in a substantial fraction of cases, with effectiveness varying sharply across model families. While LLMs offer utility in structuring evaluations, their integration into peer review requires safeguards against both intrinsic biases and adversarial risks.

Abstract

Linguistic Bias in Language Models

Explore similar work

LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers

Judging a Review by its Cover: A Reliability Analysis of LLM-based Peer Review Evaluation Metrics

LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challenges