cs.CLApr 9, 2026
SaveAtomEval: Validity-Aware Atomic Evaluation of Adversarial Claim Rewriting in Fact Verification
Organizations: Zhejiang University · The Hong Kong University of Science and Technology (Guangzhou)
Abstract
Large language models (LLMs) can rewrite refuted claims to evade evidence-based fact verifiers, but conventional attack success rate (ASR) can be inflated when rewrites change, weaken, or correct the false proposition they are supposed to preserve. We introduce AtomEval, a validity-aware evaluation protocol for fixed-evidence adversarial claim rewriting. AtomEval represents claims as subject--relation--object--modifier (SROM) atoms, applies a one-way preservation gate to separate valid verifier evasion from proposition-changing rewrites, and reports validity-aware attack success rate (VASR), which counts only verifier-evasive rewrites that preserve the original false proposition. AtomEval further provides fine-grained diagnostics that explain both proposition-level failures and non-minimal valid rewrites. On FEVER refuted-claim rewriting, AtomEval exposes and explains ASR inflation: many apparent attacks fool the verifier by altering, weakening, or correcting the proposition they should preserve. By making attacked-proposition preservation explicit and measurable, AtomEval provides a stable evaluation target for evaluating adversarial rewriters that must balance verifier evasion with proposition preservation.
Figures & tables
| Original Claim: |
| Reign Over Me is an American film made in 2010 . |
| Evidence: |
| … Reign Over Me is a 2007 American drama film … |
| Mistral-7B Rewriter ASR: ✓ VASR: |
| “Reign Over Me… in 2007 , with some insiders claiming it was actually filmed in 2005 …” |
| GPT-4.1 Rewriter ASR: ✓ VASR: ✓ |
Figure 1: Given the false claim that Reign Over Me was made in 2010 , both rewrites change the verifier’s prediction and are counted as successful attacks by conventional ASR. However, only the GPT-4.1 rewriter preserves the attacked false proposition. The Mistral-7B rewrite instead replaces the original year with the evidence-supported year ( 2007 ) and introduces an unsupported year ( 2005 ).
Figure 2: Overview of AtomEval . Given an original claim , fixed evidence , and an adversarial rewrite , AtomEval decomposes and into atomic factual units and applies a binary preservation gate . VASR counts only prediction-flipping rewrites that preserve the original attacked proposition. AtomEval further provides post-hoc diagnostics for analyzing how rewrites change factual content under the fixed evidence context.
| Split | Supported | Refuted | Total |
|---|---|---|---|
| Train set | 123,646 | 49,117 | 172,763 |
| Test set | 9,461 | 9,554 | 19,015 |
| Attack set | 0 | 400 | 400 |
Table 1: Statistics of the FEVER splits and attack set.
| Method | Correct | Acc. |
|---|---|---|
| GPT-4.1 + prompt | 260/400 | 65.00 |
| Llama-3-8B + prompt | 203/400 | 50.75 |
| AtomEval Extractor | 376/400 | 94.00 |
Table 2: Human evaluation of atomic fact extraction outputs. Each extraction is judged as correct or incorrect against manually annotated SROM expectations.
| AtomEval Output | Prec. | Rec. | F1 | Acc. |
|---|---|---|---|---|
| Validity gate | ||||
| BinaryGate | 80.95 | 94.44 | 87.18 | 87.50 |
| Diagnostic metrics | ||||
| ScopeLoss | 99.37 | 72.02 | 83.51 | 71.82 |
| EvDrift | 78.95 | 62.50 | 69.77 | 66.67 |
| EvEnt | 93.33 | 77.78 | 84.85 | 87.80 |
Table 3: Agreement with human annotations for BinaryGate and diagnostic metrics. BinaryGate uses extractor-predicted SROM atoms followed by NLI-based atom matching and determines VASR. Diagnostics are evaluated against human-labeled rewrite behaviors and are used only for analysis.
| Variant | Valid | Prec. | Rec. | F1 | Acc. |
|---|---|---|---|---|---|
| Full AtomEval | 52.50 | 80.95 | 94.44 | 87.18 | 87.50 |
| w/o SROM Extractor | 69.50 | 60.43 | 93.33 | 73.36 | 69.50 |
| w/o BinaryGate/NLI | 47.00 | 72.34 | 75.56 | 73.91 | 76.00 |
Table 4: Ablation of AtomEval’s validity decision on the 400-pair manual audit sample. All variants are evaluated against adjudicated human labels. Valid denotes the percentage of rewrites predicted as valid.
| Attack Family | Generator | Gemma-7B Verifier | BERT-based Verifier | ||||||
|---|---|---|---|---|---|---|---|---|---|
| ASR | S-ASR | P-ASR | VASR | ASR | S-ASR | P-ASR | VASR | ||
| Colloquial/Lexical | GPT-4.1 | 17.07 | 16.80 | 12.53 | 2.40 | 12.69 | 12.39 | 9.37 | 10.27 |
| Mistral-7B | 40.53 | 32.00 | 28.53 | 4.80 | 25.68 | 19.34 | 16.92 | 9.67 | |
| DeSePtion-style | GPT-4.1 | 8.00 | 8.00 | 7.73 | 7.47 | 57.10 | 55.29 | 56.19 | 51.66 |
| Mistral-7B | 68.27 | 44.53 | 68.00 | 8.80 | 90.03 | 60.73 | 89.43 | 15.11 | |
| GEM-style FactMix | GPT-4.1 | 16.53 | 12.80 | 12.80 | 8.27 | 31.12 | 24.77 | 25.98 | 13.29 |
Table 5: Main evaluation under the unified fixed-evidence claim rewriting protocol. Gemma-7B is a decoder-based verifier and BERT is an encoder-based verifier. All metrics are computed over initially correct Refuted source claims: for Gemma-7B and for BERT. ASR measures raw prediction-changing attacks. VASR counts raw successful attacks that also pass AtomEval’s validity gate. S-ASR and P-ASR are permissive surface-screened ASR variants: S-ASR keeps raw successes with SBERT similarity at least , while P-ASR keeps raw successes with GPT-2 perplexity at most . They are included to test whether conventional similarity or fluency filters approximate validity filtering, not as validity metrics. Bold values indicate the lowest attack success rate in each column.
| Attack | Gen. | EvDrift | ScopeLoss | EvEnt | UnverAdd |
|---|---|---|---|---|---|
| AdvAdd | GPT-4.1 | – | – | 0.0 | 100.0 |
| AdvAdd | Mistral-7B | 55.3 | 89.4 | 48.0 | 96.0 |
| Colloquial | GPT-4.1 | 81.8 | 67.3 | 11.1 | 22.2 |
| Colloquial | Mistral-7B | 85.1 | 85.8 | 66.7 | 38.9 |
| DeSePtion | GPT-4.1 | – | – | 35.7 | 60.7 |
| DeSePtion | Mistral-7B | 70.9 | 92.8 | 78.8 | 78.8 |
Table 6: Representative diagnostic breakdown for raw successful attacks on the Gemma-7B verifier. EvDrift and ScopeLoss are computed over invalid raw successes, while EvEnt and UnverAdd are computed over valid raw successes. All values are percentages; “–” indicates fewer than five rewrites in the corresponding subset. Full results are provided in Appendix F.2 .
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Representative zero-shot prompt used for adversarial claim rewriting. The shared system instruction is combined with a strategy-specific rewriting instruction and the fixed claim–evidence input.
| Attack | Gen. | SBERT ↑ | Adv.PPL ↓ |
|---|---|---|---|
| AdvAdd | GPT-4.1 | 0.903 | 65.59 |
| AdvAdd | Mistral-7B | 0.723 | 47.35 |
| Colloquial | GPT-4.1 | 0.880 | 128.97 |
| Colloquial | Mistral-7B | 0.734 | 132.16 |
| DeSePtion | GPT-4.1 | 0.803 | 38.61 |
| DeSePtion | Mistral-7B | 0.686 | 35.29 |
Table 7: Surface-proxy statistics for successful adversarial rewrites on the BERT-based verifier. SBERT reports the mean Sentence-BERT similarity between the original claim and the successful rewrite. Adv.PPL reports the mean GPT-2 perplexity of successful adversarial rewrites.
| Verifier | Generator | Attack Family | N | Invalid | Valid | EvDrift | ScopeLoss | EvEnt | UnverAdd |
|---|---|---|---|---|---|---|---|---|---|
| BERT-based | GPT-4.1 | Add.Unver./AdvAdd | 37 | 0.0 (0) | 100.0 (37) | – | – | 0.0 (0) | 97.3 (36) |
| BERT-based | GPT-4.1 | Colloquial/Lexical | 42 | 19.0 (8) | 81.0 (34) | 50.0 (4) | 100.0 (8) | 2.9 (1) | 20.6 (7) |
| BERT-based | GPT-4.1 | DeSePtion-style | 189 | 9.5 (18) | 90.5 (171) | 22.2 (4) | 94.4 (17) | 32.8 (56) | 71.9 (123) |
| BERT-based | GPT-4.1 | GEM-style FactMix | 103 | 57.3 (59) | 42.7 (44) | 91.5 (54) | 89.8 (53) | 86.4 (38) | 79.5 (35) |
| BERT-based | GPT-4.1 | Omission-style | 83 | 94.0 (78) | 6.0 (5) | 16.7 (13) | 98.7 (77) | 0.0 (0) | 20.0 (1) |
| BERT-based | Mistral-7B | Add.Unver./AdvAdd | 129 | 67.4 (87) | 32.6 (42) | 48.3 (42) | 95.4 (83) | 30.9 (13) | 100.0 (42) |
Table 8: AtomEval diagnostics for raw successful attacks. Percentages are shown with counts in parentheses. Invalid and Valid denote rewrites that fail or pass the binary gate; valid-side diagnostics are omitted for fewer than five valid rewrites.
| Strategy | Gen. | Adversarial Rewrite | Attack | Observation | Explanation |
|---|---|---|---|---|---|
| Lexical | GPT-4.1 | “Danger UXB is a piece of furniture.” | Failure | ScopeLoss | The rewrite generalizes “desk” to “piece of furniture.” It weakens the original proposition and does not flip the verifier in this instance. |
| Mistral | “Danger UXB’s actually a telly show, mate. It’s a 1979 British series, see? Set during WW2, and starring Tony Andrews as an officer in the Royal Engineers.” | Success | EvDrift | The rewrite replaces the original false desk proposition with the evidence-supported fact that Danger UXB is a television series. | |
| DeSePtion | GPT-4.1 | “Danger UXB, which shares its name with a well-known British television series, refers to a type of desk introduced in the late 20th century.” | Success | Valid + EvEnt + UnverAdd | The rewrite preserves the original false desk proposition while adding evidence-supported context about the television series. |
| Mistral | “During the Second World War, there existed a British television series titled Danger UXB, developed by John Hawkesworth and starring Anthony Andrews as Lieutenant Brian Ash, whose duties were not limited solely to a desk.” | Success | EvDrift + ScopeLoss | The rewrite shifts toward the evidence-supported television-series description and weakens the desk proposition into a peripheral phrase. | |
| FactMix | GPT-4.1 | “Danger UXB is a desk used by the Royal Engineers during the Second World War.” | Failure | Valid + UnverAdd | The rewrite preserves the false desk proposition but adds unsupported details blended from evidence-related entities and setting information. It does not flip the verifier here. |
| Mistral | “Anthony Andrews, known for his role in the 1979 ITV series Danger UXB, appeared as a desk officer in the Royal Engineers during the Second World War.” | Success | EvDrift + ScopeLoss | The rewrite shifts the subject from Danger UXB to Anthony Andrews, drops the original desk proposition, and mixes evidence-supported details with an unsupported desk-related role. |
Table 9: Claim-centered case study under fixed evidence. Strategy abbreviations correspond to the five attack families in Section 4.2 : Lexical, DeSePtion, FactMix, Omission, and AdvAdd. The examples show that attack success and proposition preservation are distinct: some successful attacks are invalid because they drift toward evidence or lose the original proposition, while valid rewrites may still introduce evidence-supported or unsupported additional content. Diagnostic tags such as EvEnt and UnverAdd characterize these expansions but do not affect VASR.
More in this topic
Adversarial Text Attacks
Adversarial text attacks test how small or meaning-preserving text changes can fool language models, classifiers, and safety systems.
See all 78 papers