How Robust Is Multimodal Claim Verification to LLM Rewriting?

Yun-Ang Wu; Xanh Ho; Andre Greiner-Petter; Sunisth Kumar; Tian Cheng Xia; Florian Boudin; Akiko Aizawa

cs.CLOct 2, 2026

How Robust Is Multimodal Claim Verification to LLM Rewriting?

Authors: Yun-Ang Wu, Xanh Ho, Andre Greiner-Petter, Sunisth Kumar, Tian Cheng Xia, Florian Boudin, Akiko Aizawa

Organizations: NII LLMC, Japan

Abstract

LLMs are known to introduce stylistic changes into generated text, yet how these stylistic shifts affect model decisions on scientific tasks remains underexplored. In this paper, we focus on multimodal claim verification, where the goal is to determine whether a textual claim is grounded in a given piece of evidence. We apply two rewriting strategies: natural rewriting, which simulates how researchers routinely use LLMs to polish academic text, and controlled injection, which inserts a single LLM-associated word to isolate the effect of vocabulary choice. We evaluate 11 open-weight models spanning five VLM families and ranging from 2B to 38B parameters. We find that models are robust to these modifications: most show no significant drop in accuracy, and compared to prior work on review-score manipulation, verification appears far more stable. However, consistent probability shifts do occur. Hedging-oriented conditions produce significant shifts across nearly all models, while boosting conditions show a weaker effect and general polishing conditions (e.g., grammar correction, fluency improvement) have little effect.

Figures & tables

Figure 1: A claim verification task requires deciding whether a claim is supported or refuted by evidence. This figure shows how a rational decision-maker should adjust

p(\text{support})

when the claim’s intensity modifier changes. A stronger modifier (e.g., “considerably” to “dramatically”) makes a supported claim harder to verify and a refuted claim easier to refute, both lowering

p(\text{support})

. However, this reasoning only applies when the modification is semantically meaningful. Our central question: when the modification is stylistic and does not alter the claim’s semantic content, does the model still exhibit the same behavioral shift?

Condition	Claim
Original	Additionally, 46.6% of participants were primary school graduates, 9.4% reported alcohol use, and 13% were smokers, with an average duration of smoking of 22.83 ± 6.94 years.
Natural Rewriting
Language Polishing (LP)	Additionally, 46.6 % of participants were primary - school graduates, 9.4 % reported alcohol use, and 13 % were smokers, with an average smoking duration of 22.83 ± 6.94 years.
Fluency Improvement (FI)	Moreover, 46.6 % of the participants had completed primary school; 9.4 % reported consuming alcohol; and 13 % were smokers, with an average smoking history of 22.83 ± 6.94 years.
Academic Style Alignment (AS)	Furthermore, 46.6 % of participants had completed primary education, 9.4 % reported alcohol consumption, and 13 % were smokers, with a mean smoking duration of 22.83 ± 6.94 years.
Outcome-Aware (OA)	Notably, 46.6 % of the cohort were primary - school graduates, 9.4 % reported alcohol consumption, and 13 % were smokers, with a mean smoking history of 22.83 ± 6.94 years.

Table 1: Rewrite examples for claim val_tab_0484 from SciClaimEval ( Ho et al., 2026 ) . Natural rewriting paraphrases claims using LLM prompts that simulate common academic editing scenarios, while controlled injection inserts a single LLM-associated word into the original claim.

			Natural Rewriting					Controlled Rewriting
Model		Base	LP	FI	AS	OA	CT	D1	C1
gemma-4-26B-A4B-it	P-Acc	0.548	0.546	0.537	0.529	0.532	0.524	0.560	0.538
gemma-4-26B-A4B-it	F1	0.789	0.785	0.782	0.773	0.770	0.790	0.792	0.790
gemma-4-31B-it	P-Acc	0.610	0.585	0.579 *	0.581	0.565 ***	0.557 ***	0.606	0.586
gemma-4-31B-it	F1	0.824	0.814	0.809 *	0.810	0.798 ***	0.806 *	0.822	0.816
GLM-4.6V-Flash	P-Acc	0.571	0.566	0.563	0.565	0.563	0.539	0.565	0.568

Table 2: Claim verification performance under different rewriting conditions. Base : original unmodified claims. Significance levels: *

q<0.05

, **

q<0.01

, ***

q<0.001

(BH-corrected). 3 3 3 Due to computational constraints, dev set results for InternVL3-38B were not available.

				Natural Rewriting					Controlled Rewriting
Model		Base		LP	FI	AS	OA	CT	D1	C1
gemma-4-26B-A4B-it	$p_{\text{sup}}$	0.851	$\Delta p_{\text{sup}}$	-0.011	-0.024 ***	-0.024 ***	-0.045 ***	+0.037 ***	-0.011 *	+0.016 *
gemma-4-26B-A4B-it	$p_{\text{ref}}$	0.321	$\Delta p_{\text{ref}}$	+0.007	-0.004	+0.002	-0.019 *	+0.070 ***	-0.013 *	+0.021 ***
gemma-4-31B-it	$p_{\text{sup}}$	0.919	$\Delta p_{\text{sup}}$	-0.002	-0.016 ***	-0.020 **	-0.040 ***	+0.019 **	-0.007	+0.005
gemma-4-31B-it	$p_{\text{ref}}$	0.315	$\Delta p_{\text{ref}}$	+0.024 ***	+0.022 ***	+0.015 *	+0.006	+0.078 ***	+0.010	+0.036 ***
GLM-4.6V-Flash	$p_{\text{sup}}$	0.823	$\Delta p_{\text{sup}}$	+0.009	-0.004	-0.005	-0.017	+0.056 ***	+0.007	+0.022 ***

Table 3: Changes in

p(\text{support})

on the supported and refuted subsets after claim rewriting. Values under each rewriting condition represent

\Delta p=p^{\prime}-p

, where

p

is the base prediction and

p^{\prime}

is the prediction after rewriting. The values of

p(\text{support})

are computed separately for the supported subset (denoted as

p_{\text{sup}}

) and the refuted subset (denoted as

p_{\text{ref}}

). We observe statistically consistent effects for CT and C1 across models. Significance levels: *

q<0.05

, **

q<0.01

, ***

q<0.001

(BH-corrected).

Figure 2: Shift decomposition compass. Each point represents one rewriting condition, averaged over all models. The

x

-axis shows the mean change in

p(\text{support})

on supporting evidence pairs (

\Delta p_{\text{sup}}

) and the

y

-axis on refuting evidence pairs (

\Delta p_{\text{ref}}

). Circles denote different rewriting conditions. Points near the origin indicate robustness to rewriting.

Dataset	#Dev	#Test
SciClaimEval
Full	747	917
Unpaired removed	704	872

Table 4: Dataset statistics.

Figure 3: Average

\Delta p

for the Qwen3-VL model family. The x-axis shows the mean change in

p(\text{support})

, i.e., the average predicted probability of the support label across the whole dataset. The y-axis represents model size (2B, 4B, 8B, and 32B).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Condition	Rewriter	$\Delta p_{\text{sup}}$	$\Delta p_{\text{ref}}$
CT	gpt-oss	$+0.054$	$+0.030$
		$[0.030,\,0.078]$	$[0.012,\,0.050]$
	Gemma	$+0.056$	$+0.037$
		$[0.032,\,0.080]$	$[0.020,\,0.058]$
OA	gpt-oss	$-0.035$	$-0.021$
		$[-0.058,\,-0.012]$	$[-0.039,\,-0.002]$

Appendix

Table 5:

\Delta p_{\text{sup}}

and

\Delta p_{\text{ref}}

for rewrites generated by gpt-oss and Gemma-4-31B-it on the dev set, verified by Qwen3-VL-8B. Brackets show 95% bootstrap CIs (1,000 resamples). For Gemma, C1/D1 rewrites that do not insert exactly one pool word are excluded.

	Full	Unflagged	Flagged
CT $\Delta p_{\text{sup}}$	$+0.062$	$+0.055$	$+0.080$
	$[0.044,\,0.078]$	$[0.037,\,0.074]$	$[0.051,\,0.109]$
CT $\Delta p_{\text{ref}}$	$+0.045$	$+0.044$	$+0.046$
	$[0.030,\,0.059]$	$[0.027,\,0.061]$	$[0.020,\,0.075]$
C1 $\Delta p_{\text{sup}}$	$+0.017$	$+0.016$	$+0.020$
	$[0.006,\,0.028]$	$[0.003,\,0.029]$	$[-0.002,\,0.040]$

Appendix

Table 6:

\Delta p_{\text{sup}}

and

\Delta p_{\text{ref}}

on the full, unflagged, and flagged subsets (Qwen3-VL-8B). Brackets show 95% bootstrap CIs (1,000 resamples).

Panel of Figure 4 — Figure 4: Prompt template used for natural rewriting. The [Rewriting Prompt] placeholder is instantiated with one of five rewriting categories: Language Polishing, Fluency Improvement, Academic Style Alignment, Outcome-Aware, or Cautious Tone. See Table 9 for details.

Panel of Figure 5 — Figure 5: Prompt template used for controlled rewriting. The {word_list} placeholder is instantiated with a randomly sampled subset of candidate words from either the decorative or conservative word pool, and {claim} is instantiated with the original claim.

Cond.	Judg.	Pres. (%)	$n_{2}$	$P_{o}$	$\kappa$	AC1
CT	96	78.1	27	0.667	0.088	0.475
C1	97	84.5	26	0.846	0.246	0.807
OA	98	74.5	27	0.741	0.291	0.591
D1	100	90.0	29	0.897	0.343	0.877
All	391	81.8	109	0.789	0.251	0.706

Appendix

Table 7: Human annotation results. Judg.: number of judgments. Pres.: percentage of judgments finding the meaning preserved.

n_{2}

: claims with two judgments;

P_{o}

(observed agreement), Fleiss’

\kappa

, and Gwet’s AC1 are computed on these only. CT and C1 are sampled from the unflagged subset.

Case 1: high shift val_fig_0285 Condition: CT Gold: Supported $p$ : 0.0 $\rightarrow$ 0.9 ( $\Delta p=+0.9$ ) Flipped: yes
Original	The results indicated that there were more bacterial categories in clinical samples than cultures by media.
Rewritten	The results suggested that a greater number of bacterial categories were observed in clinical samples than were identified through culture media.
Evidence
Case 2: high shift val_fig_0052 Condition: OA Gold: Supported $p$ : 0.9 $\rightarrow$ 0.0 ( $\Delta p=-0.9$ ) Flipped: yes
Original	It is also intriguing to observe from Figure 5 that Vanilla LLaMA-2-70B excelled in Writing, but full-parameter fine-tuning led to a decline in these areas, a phenomenon known as the “Alignment Tax” (Ouyang et al., 2022 ) .
Rewritten	Figure 5 compellingly demonstrates that the vanilla LLaMA-2-70B model attains peak performance on writing tasks, yet once the model is subjected to full-parameter fine-tuning its scores markedly decline—a clear manifestation of the well-documented “Alignment Tax” (Ouyang et al., 2022).

Appendix

Table 8: Example cases for Qwen3-VL-8B-Instruct.

p

is the probability of Supported estimated from 10 sampled answers, before and after rewriting. Words changed by the rewrite are in bold. Evidence images are taken from SciClaimEval ( Ho et al., 2026 ) .

Panel of Figure 6 — Figure 6: Annotation guidelines provided to annotators for evaluating semantic consistency between original and rewritten claims.

Category	Variant	Prompt
Language Polishing (LP)	LP-1	Fix any grammar mistakes in the following text.
	LP-2	Proofread this text and correct all grammatical errors.
	LP-3	Can you fix the grammar of this text?
	LP-4	Please correct any grammatical issues in the text below.
	LP-5	Review this text for grammar problems and rewrite it with corrections.
Fluency Improvement (FI)	FI-1	Rewrite this text to make it more fluent and readable.

Appendix

Table 9: Prompt strategies for natural rewriting experiments. Each category contains five prompt variants to control for variance introduced by prompt phrasing.

		Natural Rewriting					Controlled Rewriting
Model		LP	FI	AS	OA	CT	D1	C1
gemma-4-26B-A4B-it	Supported	4.9%	5.2%	7.9%	8.9%	7.9%	4.5%	5.1%
gemma-4-26B-A4B-it	Refuted	6.7%	7.0%	8.5%	8.9%	11.5%	5.7%	5.5%
gemma-4-31B-it	Supported	2.9%	3.2%	3.9%	6.1%	4.2%	3.0%	2.4%
gemma-4-31B-it	Refuted	5.7%	6.5%	7.9%	9.6%	11.8%	5.1%	7.2%
GLM-4.6V-Flash	Supported	6.7%	7.5%	8.5%	11.0%	10.8%	4.9%	6.5%

Appendix

Table 10: Flip rate: percentage of claims whose predicted label (threshold

0.5

p(\text{support})

) changes after rewriting, in either direction. Rows group claims by gold label: Supported (claim paired with supporting evidence) and Refuted (claim paired with refuting evidence). Colour intensity scales with flip rate.

Category	Words
Conservative (C)	acknowledges, allegedly, apparently, approximately, arguably, basically, broadly, comparatively, conceivably, essentially, generally, hinting, inadequately, largely, mainly, marginally, moderately, mostly, often, ostensibly, partially, perhaps, plausibly, possibly, potential, presumably, primarily, purportedly, relatively, roughly, seemingly, slightly, somewhat, supposedly, tentatively, typically, underexplored, unexplored, usually
Decorative (D)	accentuates, adept, advocates, affirming, aptly, attains, augmenting, avenue, boast, bolster, burgeoning, capitalizing, catalyze, combating, commendable, compelling, compellingly, comprehensive, consolidates, crafted, critical, crucial, culminating, deeper, dependability, dependable, detrimentally, disrupts, distinctive, effectively, effortlessly, elevate, emphasises, emphasising, emphasize, empowers, enduring, enhance, essential, excel, excellently, exceptional, exceptionally, exhaustive, expansive, formidable, fortify, foster, foundational, fresh, fundamental, groundbreaking, illuminate, imperative, impressive, impressively, ingenious, innovative, intriguing, invaluable, leverage, lucidly, meticulous, meticulously, multifaceted, notable, notably, noteworthy, optimizing, orchestrating, outperform, paving, pinpoint, pioneering, pivotal, poised, potent, pressing, promise, pronounced, propelling, remarkable, renowned, revolutionize, seamless, seamlessly, showcase, significant, solidify, strategically, streamline, substantiated, surged, surmount, surpass, swift, swiftly, thorough, transcend, transform, transformative, uncharted, uncovering, underscore, unlocking, unparalleled, unraveling, unveil, uphold, valuable, versatile, versatility, warranting, well-rounded

Appendix

Table 11: Word pools used in inject rewriting. These words are frequently included by LLMs during writing and serve as recognizable markers of LLM-generated text. Decorative words are sourced from Kobak et al. (2025) and Matsui (2025) ; conservative words are drawn from the same sources and supplemented with LLM-assisted hedging terms. Category C (Conservative) contains 39 words; Category D (Decorative) contains 114 words.

Language Model Robustness

Language model robustness studies how reliably language models preserve correct, safe behavior under input changes, distribution shifts, and adversarial pressure.

See all 493 papers

Style over Substance? Linguistic Bias in LLM-based Peer Review Copy title
Oct 26, 2026Sai Suresh Macharla Vasu, Sumedh Santosh Joshi, Philipp Christmann +6Linguistic Bias in Language Models Bias in LLM-as-a-Judge
CISPA Helmholtz Center for Information Security · German Research Center for Artificial Intelligence (DFKI) · University of Technology Nuremberg +1
From 'May' to 'Is': Certainty Distortion in Language Model Rewriting
Jun 6, 2026Catarina G Belem, Shang Wu, Hongyu Yao +3Scientific Communication Language Model Calibration
Department of Computer Science, University of California Irvine · Massachusetts Institute of Technology · Department of Cognitive Science, University of California Irvine
How LLMs Distort Our Written Language
Mar 18, 2026Marwa Abdulhai, Isadora White, Yanming Wan +4Text Generation
UC Berkeley · UC San Diego · University of Washington +2

Oct 26, 2026J/K move · Enter open · S save

Style over Substance? Linguistic Bias in LLM-based Peer Review Copy title

Sai Suresh Macharla Vasu, Sumedh Santosh Joshi, Philipp Christmann, Ivaxi Sheth+5

CISPA Helmholtz Center for Information Security · German Research Center for Artificial Intelligence (DFKI) · University of Technology Nuremberg+1

Linguistic Bias in Language Models Bias in LLM-as-a-Judge Automated Peer Review

Open paper AskPDF arXiv

The number of paper submissions to major AI conferences has increased dramatically in recent years, substantially increasing the peer-review workload of researchers. At the same time, large language models (LLMs) are increasingly used to support peer review, both by individual reviewers and by conference organizations. While LLM capabilities have improved considerably, their evaluations may be influenced by linguistic characteristics that are unrelated to scientific content. In this work, we introduce a controlled experimental framework for studying such effects by systematically perturbing the linguistic style of research papers. Specifically, we rewrite papers to exhibit (i) linguistic features associated with native or non-native English writing, and (ii) high- or low-confidence linguistic features. To construct these interventions, we derive fine-grained rewriting instructions grounded in the linguistic literature and apply them using LLMs. Crucially, our interventions are designed to preserve the underlying scientific content. We then evaluate the resulting papers using different LLM reviewers. Our experiments expose a critical vulnerability in LLM-based peer review: reviewers systematically reward confident rhetoric independent of scientific content, raising serious concerns about the robustness of their assessments.

Abstract

Language Model Robustness

Explore similar work

Style over Substance? Linguistic Bias in LLM-based Peer Review Copy title

From 'May' to 'Is': Certainty Distortion in Language Model Rewriting

How LLMs Distort Our Written Language