Large language models (LLMs) can rewrite refuted claims to evade evidence-based fact verifiers, but conventional attack success rate (ASR) can be inflated when rewrites change, weaken, or correct the false proposition they are supposed to preserve. We introduce AtomEval, a validity-aware evaluation protocol for fixed-evidence adversarial claim rewriting. AtomEval represents claims as subject--relation--object--modifier (SROM) atoms, applies a one-way preservation gate to separate valid verifier evasion from proposition-changing rewrites, and reports validity-aware attack success rate (VASR), which counts only verifier-evasive rewrites that preserve the original false proposition. AtomEval further provides fine-grained diagnostics that explain both proposition-level failures and non-minimal valid rewrites. On FEVER refuted-claim rewriting, AtomEval exposes and explains ASR inflation: many apparent attacks fool the verifier by altering, weakening, or correcting the proposition they should preserve. By making attacked-proposition preservation explicit and measurable, AtomEval provides a stable evaluation target for evaluating adversarial rewriters that must balance verifier evasion with proposition preservation.
Figures & tables
Original Claim:
Reign Over Me is an American film made in 2010 .
Evidence:
… Reign Over Me is a 2007 American drama film …
Mistral-7B Rewriter ASR: ✓ VASR: ×
“Reign Over Me… in 2007 , with some insiders claiming it was actually filmed in 2005 …”
GPT-4.1 Rewriter ASR: ✓ VASR: ✓
Figure 1: Given the false claim that Reign Over Me was made in 2010 , both rewrites change the verifier’s prediction and are counted as successful attacks by conventional ASR. However, only the GPT-4.1 rewriter preserves the attacked false proposition. The Mistral-7B rewrite instead replaces the original year with the evidence-supported year ( 2007 ) and introduces an unsupported year ( 2005 ).
Figure 2: Overview of AtomEval . Given an original claim C , fixed evidence E , and an adversarial rewrite C′ , AtomEval decomposes C and C′ into atomic factual units and applies a binary preservation gate H(C,C′) . VASR counts only prediction-flipping rewrites that preserve the original attacked proposition. AtomEval further provides post-hoc diagnostics for analyzing how rewrites change factual content under the fixed evidence context.
Split
Supported
Refuted
Total
Train set
123,646
49,117
172,763
Test set
9,461
9,554
19,015
Attack set
0
400
400
Table 1: Statistics of the FEVER splits and attack set.
Method
Correct
Acc.
GPT-4.1 + prompt
260/400
65.00
Llama-3-8B + prompt
203/400
50.75
AtomEval Extractor
376/400
94.00
Table 2: Human evaluation of atomic fact extraction outputs. Each extraction is judged as correct or incorrect against manually annotated SROM expectations.
AtomEval Output
Prec.
Rec.
F1
Acc.
Validity gate
BinaryGate
80.95
94.44
87.18
87.50
Diagnostic metrics
ScopeLoss
99.37
72.02
83.51
71.82
EvDrift
78.95
62.50
69.77
66.67
EvEnt
93.33
77.78
84.85
87.80
Table 3: Agreement with human annotations for BinaryGate and diagnostic metrics. BinaryGate uses extractor-predicted SROM atoms followed by NLI-based atom matching and determines VASR. Diagnostics are evaluated against human-labeled rewrite behaviors and are used only for analysis.
Variant
Valid
Prec.
Rec.
F1
Acc.
Full AtomEval
52.50
80.95
94.44
87.18
87.50
w/o SROM Extractor
69.50
60.43
93.33
73.36
69.50
w/o BinaryGate/NLI
47.00
72.34
75.56
73.91
76.00
Table 4: Ablation of AtomEval’s validity decision on the 400-pair manual audit sample. All variants are evaluated against adjudicated human labels. Valid denotes the percentage of rewrites predicted as valid.
Attack Family
Generator
Gemma-7B Verifier
BERT-based Verifier
ASR
S-ASR
P-ASR
VASR
ASR
S-ASR
P-ASR
VASR
Colloquial/Lexical
GPT-4.1
17.07
16.80
12.53
2.40
12.69
12.39
9.37
10.27
Mistral-7B
40.53
32.00
28.53
4.80
25.68
19.34
16.92
9.67
DeSePtion-style
GPT-4.1
8.00
8.00
7.73
7.47
57.10
55.29
56.19
51.66
Mistral-7B
68.27
44.53
68.00
8.80
90.03
60.73
89.43
15.11
GEM-style FactMix
GPT-4.1
16.53
12.80
12.80
8.27
31.12
24.77
25.98
13.29
Table 5: Main evaluation under the unified fixed-evidence claim rewriting protocol. Gemma-7B is a decoder-based verifier and BERT is an encoder-based verifier. All metrics are computed over initially correct Refuted source claims: n=375 for Gemma-7B and n=331 for BERT. ASR measures raw prediction-changing attacks. VASR counts raw successful attacks that also pass AtomEval’s validity gate. S-ASR and P-ASR are permissive surface-screened ASR variants: S-ASR keeps raw successes with SBERT similarity at least 0.65 , while P-ASR keeps raw successes with GPT-2 perplexity at most 100 . They are included to test whether conventional similarity or fluency filters approximate validity filtering, not as validity metrics. Bold values indicate the lowest attack success rate in each column.
Attack
Gen.
EvDrift
ScopeLoss
EvEnt
UnverAdd
AdvAdd
GPT-4.1
–
–
0.0
100.0
AdvAdd
Mistral-7B
55.3
89.4
48.0
96.0
Colloquial
GPT-4.1
81.8
67.3
11.1
22.2
Colloquial
Mistral-7B
85.1
85.8
66.7
38.9
DeSePtion
GPT-4.1
–
–
35.7
60.7
DeSePtion
Mistral-7B
70.9
92.8
78.8
78.8
Table 6: Representative diagnostic breakdown for raw successful attacks on the Gemma-7B verifier. EvDrift and ScopeLoss are computed over invalid raw successes, while EvEnt and UnverAdd are computed over valid raw successes. All values are percentages; “–” indicates fewer than five rewrites in the corresponding subset. Full results are provided in Appendix F.2 .
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Representative zero-shot prompt used for adversarial claim rewriting. The shared system instruction is combined with a strategy-specific rewriting instruction and the fixed claim–evidence input.
Attack
Gen.
SBERT ↑
Adv.PPL ↓
AdvAdd
GPT-4.1
0.903
65.59
AdvAdd
Mistral-7B
0.723
47.35
Colloquial
GPT-4.1
0.880
128.97
Colloquial
Mistral-7B
0.734
132.16
DeSePtion
GPT-4.1
0.803
38.61
DeSePtion
Mistral-7B
0.686
35.29
Appendix
Table 7: Surface-proxy statistics for successful adversarial rewrites on the BERT-based verifier. SBERT reports the mean Sentence-BERT similarity between the original claim and the successful rewrite. Adv.PPL reports the mean GPT-2 perplexity of successful adversarial rewrites.
Verifier
Generator
Attack Family
N
Invalid
Valid
EvDrift
ScopeLoss
EvEnt
UnverAdd
BERT-based
GPT-4.1
Add.Unver./AdvAdd
37
0.0 (0)
100.0 (37)
–
–
0.0 (0)
97.3 (36)
BERT-based
GPT-4.1
Colloquial/Lexical
42
19.0 (8)
81.0 (34)
50.0 (4)
100.0 (8)
2.9 (1)
20.6 (7)
BERT-based
GPT-4.1
DeSePtion-style
189
9.5 (18)
90.5 (171)
22.2 (4)
94.4 (17)
32.8 (56)
71.9 (123)
BERT-based
GPT-4.1
GEM-style FactMix
103
57.3 (59)
42.7 (44)
91.5 (54)
89.8 (53)
86.4 (38)
79.5 (35)
BERT-based
GPT-4.1
Omission-style
83
94.0 (78)
6.0 (5)
16.7 (13)
98.7 (77)
0.0 (0)
20.0 (1)
BERT-based
Mistral-7B
Add.Unver./AdvAdd
129
67.4 (87)
32.6 (42)
48.3 (42)
95.4 (83)
30.9 (13)
100.0 (42)
Appendix
Table 8: AtomEval diagnostics for raw successful attacks. Percentages are shown with counts in parentheses. Invalid and Valid denote rewrites that fail or pass the binary gate; valid-side diagnostics are omitted for fewer than five valid rewrites.
Strategy
Gen.
Adversarial Rewrite
Attack
Observation
Explanation
Lexical
GPT-4.1
“Danger UXB is a piece of furniture.”
Failure
ScopeLoss
The rewrite generalizes “desk” to “piece of furniture.” It weakens the original proposition and does not flip the verifier in this instance.
Mistral
“Danger UXB’s actually a telly show, mate. It’s a 1979 British series, see? Set during WW2, and starring Tony Andrews as an officer in the Royal Engineers.”
Success
EvDrift
The rewrite replaces the original false desk proposition with the evidence-supported fact that Danger UXB is a television series.
DeSePtion
GPT-4.1
“Danger UXB, which shares its name with a well-known British television series, refers to a type of desk introduced in the late 20th century.”
Success
Valid + EvEnt + UnverAdd
The rewrite preserves the original false desk proposition while adding evidence-supported context about the television series.
Mistral
“During the Second World War, there existed a British television series titled Danger UXB, developed by John Hawkesworth and starring Anthony Andrews as Lieutenant Brian Ash, whose duties were not limited solely to a desk.”
Success
EvDrift + ScopeLoss
The rewrite shifts toward the evidence-supported television-series description and weakens the desk proposition into a peripheral phrase.
FactMix
GPT-4.1
“Danger UXB is a desk used by the Royal Engineers during the Second World War.”
Failure
Valid + UnverAdd
The rewrite preserves the false desk proposition but adds unsupported details blended from evidence-related entities and setting information. It does not flip the verifier here.
Mistral
“Anthony Andrews, known for his role in the 1979 ITV series Danger UXB, appeared as a desk officer in the Royal Engineers during the Second World War.”
Success
EvDrift + ScopeLoss
The rewrite shifts the subject from Danger UXB to Anthony Andrews, drops the original desk proposition, and mixes evidence-supported details with an unsupported desk-related role.
Appendix
Table 9: Claim-centered case study under fixed evidence. Strategy abbreviations correspond to the five attack families in Section 4.2 : Lexical, DeSePtion, FactMix, Omission, and AdvAdd. The examples show that attack success and proposition preservation are distinct: some successful attacks are invalid because they drift toward evidence or lose the original proposition, while valid rewrites may still introduce evidence-supported or unsupported additional content. Diagnostic tags such as EvEnt and UnverAdd characterize these expansions but do not affect VASR.
AI is accelerating scientific research, but providing timely and trustworthy feedback remains challenging. Such feedback requires evaluating specific claims against verified evidence, helping researchers and autonomous research systems determine which findings can guide subsequent work. Thorough verification is time-consuming, and reviewers under time pressure may overlook or misjudge critical issues. Existing AI reviewers have yet to fill this gap because they often assess claims without adequately verifying the supporting evidence. We introduce FactReview, a framework that verifies claims in papers to provide more trustworthy feedback. It checks claims against the literature and examines derivations, while also executing the paper's code to test empirical claims. Across 300 papers, FactReview ranks best among all evaluated systems on five of seven metrics, achieving 6.8% false support and 79.2% classification accuracy. In a study of 36 papers, reviewers using FactReview detect 9.4 issues per paper compared with 5.6 independently, while spending 74.1% less time. FactReview thus supports more reliable peer review, with the potential to accelerate research iteration. The code is public at https://github.com/DEFENSE-SEU/FactReview.
Ling Yue, Chaoqian Ouyang, Ruijun Huang +8
Rensselaer Polytechnic Institute · Sun Yat-sen University · Southeast University +1
Large Language Models (LLMs) generate fluent long-form text, however, often add unsupported factual claims. Existing verification techniques improve factuality by grounding generation in external evidence. However, the same verification policy usually applies to all claims despite being differences in hallucination risks. We propose FACTOR (FACTuality-Oriented Risk-aware Verification), an inference-time model that adapts verification criteria according to claim-level uncertainty. FACTOR combines uncertainty estimation, adaptive language inference verification, and candidate re-ranking to allocate verification effort where it is most needed. We evaluate FACTOR on FactScore benchmark showing that adaptive verification improves factuality while reducing verification cost simultaneously. We further perform different ablation studies to identify the primary driver of these gains. Our results show the effective and model-agnostic performance of FACTOR for improving factuality in long-form generation.
Automatic fact-checking systems assess the veracity of claims given evidence from relevant documents. Large Language Models (LLMs) have demonstrated strong performance in fact-checking due to their general reasoning capabilities. However, it remains unclear whether they faithfully make use of the evidence provided to reach veracity judgments or rely on parametric knowledge. To investigate this, we introduce Fact-Ablated Evaluation (FAE), a new evaluation framework that iteratively ablates the cited evidence to assess whether LLMs revise their predictions accordingly. Our empirical results show that current off-the-shelf LLMs as fact-checking systems rely more on their parametric knowledge than on the evidence provided. To bridge this gap between prediction accuracy and evidence grounding, we propose REAL (Rigorous Evidence Ablation Learning), a training framework that promotes evidence-dependent verification through counterfactual evidence supervision for the LLM-as-verifier models. Experiments on four fact-checking datasets across different domains demonstrate that models trained with REAL obtain superior evidence-dependent capabilities compared to standard fine-tuned models. Our findings highlight that strong fact-checking performance can still coexist with weak evidence dependency, while REAL encourages veracity predictions to remain more closely tied to the availability of supporting evidence.