BLUFF: Benchmarking the Detection of False and Synthetic Content across 58 Low-Resource Languages

Jason Lucas; Matt Murtagh-White; Adaku Uchendu; Ali Al-Lawati; Michiharu Yamashita; Dominik Macko; Ivan Srba; Robert Moro

cs.CLFeb 28, 2026

BLUFF: Benchmarking the Detection of False and Synthetic Content across 58 Low-Resource Languages

Authors: Jason Lucas, Matt Murtagh-White, Adaku Uchendu, Ali Al-Lawati, Michiharu Yamashita, Dominik Macko, Ivan Srba, Robert Moro,

Organizations: Penn State University, USA · Trinity College Dublin, Ireland · MIT Lincoln Lab, USA · Visa Research, USA ·

Abstract

Multilingual falsehoods threaten information integrity worldwide, yet detection benchmarks remain confined to English or a few high-resource languages, leaving low-resource linguistic communities without robust defense tools. We introduce BLUFF, a comprehensive benchmark for detecting false and synthetic content, spanning 79 languages with over 202K samples, combining human-written fact-checked content (122K+ samples across 57 languages) and LLM-generated content (79K+ samples across 71 languages). BLUFF uniquely covers both high-resource "big-head" (20) and low-resource "long-tail" (59) languages, addressing critical gaps in multilingual research on detecting false and synthetic content. Our dataset features four content types (human-written, LLM-generated, LLM-translated, and hybrid human-LLM text), bidirectional translation (English $\leftrightarrow$ X), 39 textual modification techniques (36 manipulation tactics for fake news, 3 AI-editing strategies for real news), and varying edit intensities generated using 19 diverse LLMs. We present AXL-CoI (Adversarial Cross-Lingual Agentic Chainof-Interactions), a novel multi-agentic framework for controlled fake/real news generation, paired with mPURIFY, a quality filtering pipeline ensuring dataset integrity. Experiments reveal state-of-theart detectors suffer up to 25.3% F1 degradation on low-resource versus high-resource languages. BLUFF provides the research community with a multilingual benchmark, extensive linguistic-oriented benchmark evaluation, comprehensive documentation, and opensource tools to advance equitable falsehood detection. Dataset and code are available at: https://jsl5710.github.io/BLUFF/

Figures & tables

	Languages				Text Transformation		Degree of Edits			Authorship				Scale
Dataset	Lang.	Big-Head	Long-Tail	Regions	D.Tactics	AI.Edits	Light	Moderate	Complete	HWT	MGT	MTT	HAT	Samples	Orgs.
NewsPolyML ( Mohtaj et al., 2024 )	5	$\checkmark$	—	5	—	—	—	—	—	$\checkmark$	$\times$	$\times$	$\times$	32k	5
TALLIP ( De et al., 2021 )	5	$\checkmark$	$\checkmark$	5	—	—	—	—	—	$\checkmark$	$\times$	$\checkmark$	$\times$	—	—
MM-COVID ( Li et al., 2020 )	6	$\checkmark$	—	6	—	—	—	—	—	$\checkmark$	$\times$	$\times$	$\times$	11k	—
PHEME ( Zubiaga et al., 2016 )	15	$\checkmark$	$\checkmark$	—	—	—	—	—	—	$\checkmark$	$\times$	$\times$	$\times$	6k	—
X-FACT ( Gupta and Srikumar, 2021 )	25	$\checkmark$	$\checkmark$	—	—	—	—	—	—	$\checkmark$	$\times$	$\times$	$\times$	31k	85

Table 1. Multilingual disinformation datasets with

\geq 5

languages, organized under four higher-level categories.

Figure 1. BLUFF Adversarial Cross-Lingual Chain-of-Interactions Agentic (AXL-CoI) Framework Diagram

Consistency Dimension
Metric	Method(s)	Agg.	Pass (R/F)
Logical	MENLI, FrugalScore	vote/avg	99.1%/97.5%
Factual	AlignScore (XLM-R)	score	98.2%/96.9%
Semantic	BERTScore (XLM-R)	score	98.7%/97.1%
Sentiment	Original vs Generated	vote	99.4%/95.8%
Combined	–	–	97.9%/93.8%

Table 2. mPURIFY standard AEM methods across evaluation dimensions. Label-based metrics use majority voting across tools; score-based metrics use averaging. Pass rates shown as (Real/Fake).

Metric	Comparison	Real	Fake	Pass (R/F)
Consistency Dimension
Factual	NA/SM (Src) vs Orig	$\geq$ 4.0	$\leq$ 3.0	98.5%/97.2%
Logical	NA/SM (Src) vs Orig	$\geq$ 4.0	$\leq$ 4.0	99.2%/97.7%
Semantic	NA/SM (Src) vs Orig	$\geq$ 4.0	$\leq$ 3.0	98.5%/96.8%
Contextual	NA/SM (Src) vs Orig	$\geq$ 4.0	$\leq$ 3.0	98.7%/94.7%
Combined	–	ALL pass	ALL pass	98.3%/94.1%

Table 3. mPURIFY threshold configuration and pass rates across all evaluation dimensions. Real news applies stricter thresholds (

\geq

4.0) to ensure authenticity, while fake news accepts moderate quality (

\geq

3.0) to preserve manipulation diversity. Pass rates shown as (Real/Fake).

Panel of Figure 2 — Figure 2. Generation $\rightarrow$ Defect Removal $\rightarrow$ LLM-mPURIFY pipeline. Overlapping bars show retention at each stage: generated samples (blue, 100%), post-defect filtering (hatched orange), and post-mPURIFY (solid green). GPT-4.1 variants retain highest quality (86–87%), while Llama4-Scout shows highest rejection rate (99.8% filtered).

Encoder-based (FT)
Model	Cross-lingual †			Multilingual ‡
Model	Big-Head	Long-Tail	$\Delta$	Big-Head	Long-Tail	$\Delta$
mBERT	86.5	75.8	+10.7	95.6	94.2	+1.4
mDeBERTa	96.1	88.0	+8.1	98.3	90.4	+7.9
XLM-RoBERTa	93.5	84.5	+9.0	94.3	94.2	+0.1
XLM-100 a	47.2	38.6	+8.6	70.0	65.2	+4.8

Table 4. Binary veracity classification (Real/Fake). Macro-F1 (%) reported. Best in bold .

\Delta

= Big-Head

-

Long-Tail. For decoders:

\dagger

= cross-lingual prompt,

\ddagger

= native prompt.

Encoder-based (FT)
Model	Cross-lingual			Multilingual
Model	Big-Head	Long-Tail	$\Delta$	Big-Head	Long-Tail	$\Delta$
mBERT	61.0	45.6	+15.4	62.7	63.4	–0.7
mDeBERTa	63.5	53.3	+10.2	59.2	64.0	–4.8
XLM-RoBERTa	55.7	46.2	+9.5	60.4	63.6	–3.2
XLM-RoBERTa-large	66.4	55.7	+10.7	64.7	67.8	–3.1

Table 5. Multiclass Veracity Classification (8 classes). Macro-F1 (%) reported. Best in bold . Random baseline = 12.5%.

Encoder-based
Model	Cross-lingual			Multilingual
Model	Big-Head	Long-Tail	$\Delta$	Big-Head	Long-Tail	$\Delta$
mBERT	87.3	80.2	+7.1	85.5	91.2	–5.7
mDeBERTa	87.2	83.0	+4.2	84.1	89.5	–5.4
XLM-RoBERTa	78.9	79.7	–0.8	82.4	87.6	–5.2
XLM-RoBERTa-large	49.3	44.4	+4.9	77.0	81.2	–4.2

Table 6. Binary Synthetic Text Detection (Human/Machine). Macro-F1 (%) reported. Best in bold . Random baseline = 50%.

Encoder-based
Model	Cross-lingual			Multilingual
Model	Big-Head	Long-Tail	$\Delta$	Big-Head	Long-Tail	$\Delta$
mBERT	80.6	57.2	+23.4	77.3	79.0	–1.7
mDeBERTa	76.4	59.9	+16.5	75.3	77.5	–2.2
XLM-RoBERTa	73.0	57.6	+15.4	74.3	77.3	–3.0
XLM-RoBERTa-large	77.1	62.1	+15.0	71.9	76.9	–5.0

Table 7. Multiclass Synthetic Text Detection (4 classes). Macro-F1 (%) reported. Best in bold . Random baseline = 25%.

Panel of Figure 3 — Figure 3. Cross-lingual transfer heatmaps (macro-F1, averaged across 10 encoder models) for binary veracity classification. (a) Language family: within-family avg. 66.6%, cross-family 51.2%. (b) Syntax: VSO targets prove challenging (28–63%). (c) Script: same-script avg. 68.8%, cross-script 52.4%; Arabic targets consistently poor (30–47%). Abbreviations—Family: Indo=Indo-European, Sino=Sino-Tibetan, Drav=Dravidian, AuA=Austroasiatic, AuN=Austronesian, Afro=Afro-Asiatic, Turk=Turkic, Creo=Creole. Script: Lat=Latin, Cyr=Cyrillic, Ara=Arabic, Ind=Indic, Gre=Greek.

	mDeBERTa	mBERT	XLM-E	XLM-T	S-BERT	XLM-R-L	XLM-R	XLM-17	XLM-100
Big-head	65.1	64.3	53.1	48.8	46.2	46.3	46.2	30.7	30.7
Long-tail	59.5	54.3	49.2	52.2	50.2	52.6	53.7	25.8	25.8
Overall	67.3	64.3	52.8	49.1	47.5	46.8	46.8	30.3	28.2

Table 8. External evaluation (macro-F1 %). Best in bold .

Appendix figures & tables86 assets

Supplementary material from the paper’s appendix.

Appendix

Variant	Source	Target	Lang.	Samples
Fake News (eng_x)	English	70 languages	70	105,000
Fake News (x_eng)	50 languages	English	50	75,000
Real News (eng_x)	English	70 languages	70	105,000
Real News (x_eng)	50 languages	English	50	75,000
Total	360,000 samples across 71 unique languages

Appendix

Table 9. BLUFF bidirectional translation architecture (assuming 1,500 samples per language per variant)

Language Category	eng_x Coverage	x_eng Coverage
Head Languages (19)	19 languages	15 languages
Tail Languages (52)	51 languages	35 languages
Total Unique Languages	70 languages	50 languages
Combined Coverage	71 unique languages

Appendix

Table 10. Language distribution across eng_x and x_eng variants

Characteristic	Original CoI	BLUFF CoI
	(Dialogue Summarization)	(News Generation)
Chain Count	8 chains	8–10 chains
Domain	Customer service dialogues	News articles
Languages	Monolingual (English)	Multilingual (71 languages)
Translation Direction	Not applicable	Bidirectional (eng_x, x_eng)
Primary Task	Abstractive summarization	Content manipulation/editing

Appendix

Table 11. Comparison of original CoI framework for dialogue summarization versus adapted BLUFF CoI framework for multilingual news generation

Chain	Fake News Pipeline	Real News Pipeline
1	Analyst/Examiner	Analyst/Examiner (identical)
2	Creator/Manipulator (inject 2 of 36 tactics at 3 severity levels)	Dynamic Editor (1 of 3 editing techniques)
3	Auditor/Change Tracker (document all modifications)	Validator/Quality Checker (flag inaccuracies)
4	Editor/Refiner (polish while preserving manipulation)	Adjuster/Fixer (apply accuracy corrections)
5	Validator/Quality Checker (verify manipulation completeness)	Translator (convert per eng_x/x_eng)
6	Adjuster/Fixer (fix missing manipulations)	Localization QA/Reviewer (refine translation)

Appendix

Table 12. Chain-by-chain comparison of fake and real news generation pipelines

Panel of Figure 4 — Figure 4. Block diagram of the Dynamic Persona Cycling mechanism (Alogrithm 1 ). The generator $G$ initializes a persona pool $\mathcal{P}$ with success and failure counters. For each input $x$ , the model is prompted with persona $p_{idx}$ . Success updates the counter and proceeds to the next input. Refusal updates the failure counter, cycles $idx$ (creating a new persona if needed), and retries the same input $x$ .

Large Language Models (LLMs)
Type	Model	Standard	Imperson. (F3)	w/o Mutation	Full ADIS
OpenAI	GPT-4.1	12.4	34.2	78.6	100.0
OpenAI	GPT-4.1-mini	18.6	41.8	82.4	100.0
Google	Gemini 2.5 Pro	8.2	28.4	71.2	100.0
	Gemini 2.0 Flash	14.8	38.6	79.8	100.0
	Gemini 1.5 Pro	16.2	42.4	81.6	100.0

Appendix

Table 13. ADIS ablation study: Bypass success rates (%) across 19 frontier models under four configurations. Standard Prompt serves as baseline. Each cell reports the percentage of 500 samples that successfully bypassed safety guardrails. Full ADIS achieves 100% bypass across all models.

Metric Category	Feature Count	Key Purpose
Consistency Metrics	17	Alignment verification
Change Validity Metrics	8	Edit accuracy assessment
Translation Quality Metrics	7	Cross-lingual quality
Manipulation Detection	2	Fake/real classification
Total Core Features	34	Comprehensive evaluation

Appendix

Table 14. Distribution of evaluation features across metric categories in the mPURIFY framework

Figure 11. mPURIFY pipeline filters data ( x ) through defect detection ( DEFCT ), dual evaluation scoring—LLM-AEM ( LLM-based ) and S-AEM (standard metrics)—and threshold-based selection.

Consistency Dimension				Validation Dimension
Metric	Real	Fake	Pass (R/F)	Metric	Real	Fake	Pass (R/F)
Factual	$\geq$ 4.0	$\leq$ 3.0	98.5%/97.2%	Change	$\geq$ 4.0	$\geq$ 3.0	99.9%/96.2%
Logical	$\geq$ 4.0	$\leq$ 4.0	99.2%/97.7%	Technique	$\geq$ 4.0	$\geq$ 3.0	99.9%/94.1%
Semantic	$\geq$ 4.0	$\leq$ 3.0	98.5%/96.8%	Edit	Expected vs Eval		Qualitative
Contextual	$\geq$ 4.0	$\leq$ 3.0	98.7%/94.7%	Degree	Expected vs Eval		Qualitative
Combined	ALL pass	ALL pass	98.3%/94.1%	Combined	Change & Tech pass		99.0%/93.9%

Appendix

Table 15. mPURIFY threshold configuration and pass rates across all evaluation dimensions. Real news applies stricter thresholds (

\geq

4.0) to ensure authenticity, while fake news accepts moderate quality (

\geq

3.0) to preserve manipulation diversity.

Stage	Real	Real	Fake	Fake
	Kept	Removed	Kept	Removed
Start	43,703	0	43,508	0
Consistency	42,975	728	40,934	2,574
Validation	42,945	30	39,281	1,653
Translation	42,281	664	36,674	2,607
Manipulation	41,779	502	36,664	10

Appendix

Table 16. Sequential filtering results showing cumulative sample retention at each mPURIFY stage.

	Real News			Fake News
Dimension	Kept	Rem.	%	Kept	Rem.	%
Consistency	42,975	728	98.3	40,934	2,574	94.1
Validation	43,262	441	99.0	40,854	2,654	93.9
Translation	42,757	946	97.8	39,203	4,305	90.1
Manipulation	42,418	1,285	97.1	42,924	584	98.7
All	41,779	1,924	95.6	36,664	6,844	84.3

Appendix

Table 17. Independent dimension pass rates (before sequential filtering).

Label	Kept	Rem.	%Kept	Status
Real News (Threshold: $\geq$ 4 for all metrics)
consistent	42,731	27	99.9	$\checkmark$
partial	244	519	32.0	$\checkmark$
inconsistent	0	176	0.0	$\checkmark$
Fake News (Threshold: $\leq$ 3 for most metrics)
consistent	0	897	0.0	$\times$

Appendix

Table 18. Consistency dimension label correctness analysis.

Label	Kept	Rem.	%Kept	Status
Real News (Threshold: $\geq$ 4)
fully	42,701	19	100.0	$\checkmark$
partial	0	160	0.0	$\triangle$
inaccurate	2	216	0.9	$\triangle$
Fake News (Threshold: $\geq$ 3)
fully	40,210	1,038	97.5	$\checkmark$

Appendix

Table 19. Validation dimension label correctness analysis.

	Real News		Fake News
Status	Match	%	Match	%
Kept	41,779	100.0	25,234	68.8
Removed	1,830	95.1	5,393	78.8
Total	43,609	99.8	30,627	70.4

Appendix

Table 20. Topic match analysis between original and LLM-evaluated content.

	Real News		Fake News
Sentiment	Count	%	Count	%
Positive	9,075	21.7	5,134	14.0
Negative	7,845	18.8	25,570	69.7
Neutral	21,831	52.3	2,258	6.2
Other	3,028	7.2	3,702	10.1
Match Rate	99.6%		23.6%

Appendix

Table 21. Sentiment match analysis and distribution in kept samples.

Real News
Description	Exp.	Count	Exp	Eval	Match
10–20% edit	Light	19,175	2.0	2.31	68.7%
30–50% edit	Mod.	19,426	3.0	2.37	29.4%
100% edit	Complete	5,072	5.0	4.11	16.8%
Fake News
Inconspicuous	Minor	14,744	1.0	4.36	0.4%

Appendix

Table 22. Degree of modification: expected vs LLM-evaluated scores.

Real News
Category	Count	Exp	Eval	High%
High (5)	43,692	5	4.96	99.0%
Fake News by Degree
Minor	14,744	5	4.56	90.5%
Medium	14,629	3	4.57	90.8%
Critical	14,118	1	4.58	90.5%

Appendix

Table 23. Edit quality: expected vs LLM-evaluated scores.

Panel of Figure 12 — Figure 12. Sequential filtering process showing kept vs removed samples at each mPURIFY stage (top) and per-stage removal counts with percentages (bottom). Real News (green) and Fake News (white with red lines) are shown side by side, with removed portions stacked on top. Real news maintains 95.6% retention while fake news retains 84.3% after all filters.

Panel of Figure 13 — Figure 13. Consistency dimension score distributions across four metrics: Factual, Logical, Semantic, and Contextual consistency. Real news (green) concentrates at score 5, while fake news (red) clusters at scores 1–2, reflecting successful manipulation. Dashed lines indicate quality thresholds: Real News $\geq$ 4.0 (green) and Fake News $\leq$ 3.0 (red). Pass rates shown in subplot titles indicate the percentage of samples meeting respective thresholds.

Panel of Figure 14 — Figure 14. Validation dimension score distributions for Change Detection, Edit Quality, Technique Application, and Degree Assessment. Change Detection and Technique Application include quality thresholds: Real News $\geq$ 4.0 (blue) and Fake News $\geq$ 3.0 (orange). Edit Quality and Degree Assessment show expected vs evaluated comparisons without thresholds. Degree Assessment reveals divergent patterns: real news clusters at score 2–3 (light editing) while fake news peaks at score 5 (critical manipulation).

Panel of Figure 15 — Figure 15. Translation dimension score distributions across six quality metrics: Accuracy, Fluency, Terminology, Localization, Coherence, and Semantic Preservation. Both real and fake news exhibit high translation quality (scores 4–5), with pass rates exceeding 90% for all metrics. Dashed lines indicate quality thresholds: Real News $\geq$ 4.0 (light magenta) and Fake News $\geq$ 3.0 (dark magenta).

Panel of Figure 16 — Figure 16. Manipulation dimension analysis showing score distribution (top left), pass/fail counts (top right), pass rate comparison (bottom left), and summary statistics (bottom right). Real News is expected to have no manipulation (score $\leq$ 1.0, mean: 1.04) while Fake News should show manipulation (score $\geq$ 2.0, mean: 4.80). The distribution confirms clear separation between classes with 97.9% combined accuracy.

Panel of Figure 17 — Figure 17. Mean scores across all mPURIFY dimensions with threshold lines. Consistency (CON): Fact.=Factual, Log.=Logical, Sem.=Semantic, Cont.=Contextual. Validation (VAL): Change=Change Detection, Edit=Edit Quality, Tech.=Technique Application, Degree=Degree Assessment. Translation (TRA): Acc.=Accuracy, Flu.=Fluency, Term.=Terminology, Loc.=Localization, Coh.=Coherence, Sem.=Semantic Preservation. Manipulation (MAN): Manip.=Manipulation Score. Consistency metrics show maximal class separation (real $\approx$ 4.9, fake $\approx$ 1.3–2.2). Translation metrics exhibit uniformly high scores ( $>$ 4.6). Manipulation detection achieves near-perfect separation (real: 1.04, fake: 4.80).

Panel of Figure 18 — Figure 18. Categorical label analysis showing keep/remove decisions by label type across Consistency (top), Manipulation (middle), and Validation (bottom) dimensions. Percentage boxes indicate retention rates per category. The framework correctly filters based on label semantics: “consistent” fake news (failed manipulation) is removed while “inconsistent” fake news (successful manipulation) is retained.

Key Findings
Metric	Value
Original Samples	87,211
Final Retained	78,443 (89.9%)
Real News Retained	41,779 (95.6%)
Fake News Retained	36,664 (84.3%)
Topic Match (Real/Fake)	99.8% / 70.4%

Appendix

Table 24. mPURIFY filtering summary and key findings.

Category	Examples
Indigenous	Quechua, Navajo, Inuktitut, Māori
Regional	Wolof, Sinhala, Assamese
Minority	Romani, Kurdish, Uyghur
Creole/Pidgin	Haitian Creole, Nigerian Pidgin
Limited Digital	Amharic, Somali, Nepali, Khmer

Appendix

Table 25. Long-tail language categories with representative examples.

Dataset	Total	Sampled	Lang.	Sources	Trans.	Ref.
Global News	90K	82K	1	31+ orgs	Eng $\rightarrow$ X	( Codings, 2022 )
CNN/Daily Mail	300K+	82K	1	2 orgs	Eng $\rightarrow$ X	( Nallapati et al., 2016 )
MassiveSumm	28.8M	51K	78	150+ orgs	Bidirectional	( Varab and Schluter, 2021 )
Visual News	1M+	82K	1	4 orgs	Eng $\rightarrow$ X	( Liu et al., 2021 )

Appendix

Table 26. Source news corpora for BLUFF generation. Sampling: 20,480 articles

\times

4 subsets per dataset, except MassiveSumm (up to 1K per language).

Panel of Figure 19 — Figure 19. Distribution of web content by language, illustrating the “big-head” vs “long-tail” disparity. The top 21 languages account for 88.3% of digital content, while 50+ long-tail languages share just 11.7%. English alone dominates with 52.1%. Data source: ( Statista Research Department, 2024 ) .

Region	HWT	HWT %	MGT	MGT %
Other/Unclassified	83,864	68.4%	0	0.0%
North America	5,421	4.4%	57,187	72.9%
International/Wire Services	17,103	13.9%	2,529	3.2%
Europe	5,477	4.5%	6,580	8.4%
South Asia	1,555	1.3%	7,188	9.2%
Latin America	6,566	5.4%	26	0.0%

Appendix

Table 27. Regional distribution of BLUFF samples across human-written (HWT) and LLM-generated (MGT) content. The dataset covers 12 geographic regions with 331 source organizations (130 human + 201 LLM).

Rank	HWT Organization	Samples	MGT Organization	Samples
1	Propaganda Diary	78,787	CNN	38,197
2	Agence France-Presse	16,865	The Times of India	5,378
3	human_MG_MT	4,469	VOA News	4,264
4	PolitiFact	4,048	ETF Daily News	3,511
5	Maldita	1,706	BBC News	3,395
6	Chequeado	1,146	Forbes	1,986

Appendix

Table 28. Top 20 source organizations for human-written (HWT) and LLM-generated (MGT) samples in BLUFF . HWT sources emphasize fact-checking organizations, while MGT sources draw from mainstream news outlets.

Name	Checkpoint ID	Creator	Family	Accessibility	Type	# lang	Source	Architecture	Context Window	Param.	Cost
Generation Models – Large Language Models (LLM)
GPT-4.1	gpt-4.1	OpenAI	GPT	Closed API	Instruct	50	OpenAI API	Decoder	128K	–	$200
GPT-4.1 (Apr)	gpt-4.1-2025-04-14	OpenAI	GPT	Closed API	Instruct	50	OpenAI API	Decoder	128K	–	$200
Qwen 3-Next 80B	Qwen/Qwen3-Next-80B	Alibaba	Qwen	Open Source	Instruct	12	HuggingFace	Decoder	32K	80B	Free
Llama 4 Maverick	meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8	Meta	Llama	Open Source	Instruct	28	HuggingFace	Decoder	128K	402B	Free
Llama 4 Scout	meta-llama/Llama-4-Scout-17B-16E-Instruct	Meta	Llama	Open Source	Instruct	28	HuggingFace	Decoder	16K	109B	Free

Appendix

Table 29. Multilingual LLMs for Generation, Detection, and Evaluation in the BLUFF Framework. Generation models are divided into Large Language Models (LLM) for instruction-following and Large Reasoning Models (LRM) for complex reasoning tasks. Detection models include both encoder-based classifiers and decoder-based models using prompting strategies. All evaluation models employ decoder architectures.

Dimension	Real News	Fake News
Edit Degree	Light (10–20%), Moderate (30–50%), Complete (100%)	Inconspicuous, Moderate, Alarming
Transformation	Rewrite, Polish, Edit	36 manipulation tactics (2 per sample)
Translation	Eng $\rightarrow$ X (70 langs) \| X $\rightarrow$ Eng (50 langs)
Format	News articles \| Social media posts
Authorship	HWT, MGT, MTT, HAT
Combinations	$3\times 3\times 2\times 2\times 4=\mathbf{144}$	$3\times\binom{36}{2}\times 2\times 2\times 4=\mathbf{30,240}$

Appendix

Table 30. BLUFF variation space. Real news: 144 combinations; Fake news: 30,240 combinations (using

\binom{36}{2}=630

tactic pairs). Combined with 78 languages, this yields comprehensive coverage of multilingual disinformation patterns.

Dataset	#Lang	Category	Domain	Type	Size	Author	Ref.
English Monolingual
Med-MMHL	1	big-head	Medical	NW	40.6k	HWT, MGT	( Sun et al., 2023 )
F3	1	big-head	News, Social	NW, SM	13k	HWT, MGT	( Lucas et al., 2023 )
Synthetic Lies	1	big-head	Health	SM	1k	HWT, MGT	( Zhou et al., 2023 )
LIAR	1	big-head	Politics	NW	12.8k	HWT	( Wang, 2017 )
FEVER	1	big-head	Various	Wiki	185k	HWT	( Thorne et al., 2018 )

Appendix

Table 31. Monolingual disinformation datasets. Category : big-head = high-resource, long-tail = low-resource.

Dataset	#Lang	Category	Domain	Type	Size	Author	Ref.
Limited Multilingual (2–10 languages)
Deceiver	2	big-head	Politics, COVID-19	NW, SM	600	HWT	( Vargas et al., 2021 )
CrossFake	2	big-head	COVID-19	NW	28k	HWT	( Du et al., 2021 )
FakeNewsSpreader	2	big-head	Politics, COVID-19	NW, SM	15k	HWT	( Santosh et al., 2021 )
CT-FAN	2	big-head	General	NW	3.4k	HWT	( Köhler et al., 2022 )
Italian Disinfo	2	big-head	Election	NW, SM	16.9k	HWT	( Pierri et al., 2020 )

Appendix

Table 32. Multilingual disinformation datasets grouped by language coverage: limited (2–10), moderate (11–30), extensive (31–60), and comprehensive (61+). Category : big-head = high-resource, long-tail = low-resource. Most datasets exhibit severely imbalanced long-tail distributions.

#	Tactic	Definition & Explanation	Example
1	Sensational Appeal	Crafting content with exaggerated or shocking elements to capture attention and encourage sharing.	"Elon Musk announces AI-powered 5G towers on Mars."
2	Emotionally Charged	Utilizing content that evokes strong emotions (fear, anger, joy) to influence opinions without critical analysis.	"5G towers cause severe health issues."
3	Psychologically Manipulative	Leveraging tactics that exploit cognitive biases or emotional vulnerabilities to influence beliefs or behaviors.	"AI will lead to mass unemployment."
4	Misleading Statistics	Presenting data in a deceptive manner (e.g., cherry-picking numbers) to support a false narrative.	"Isolated 5G malfunction portrayed as systemic."
5	Fabricated Evidence	Creating fake documents, images, or videos to support a claim that is untrue.	"Deepfake of Trump endorsing AI."
6	Source Masking & Fake Credibility	Disguising the origin of information by creating fake sources or impersonating credible entities to lend false legitimacy.	"Fake news website mimics reputable outlet."

Appendix

Table 33. Disinformation tactics used in BLUFF ’s adversarial generation framework (AXL-CoI). Each tactic is randomly infused pairwise (2 of 36) at three intensity levels (minor, medium, critical), yielding 1,890 unique combinations. Taxonomy adapted from CISA ( Cybersecurity and Infrastructure Security Agency, 2022 ) , Culloty and Suiter ( Culloty and Suiter, 2021 ) , Cunningham ( Cunningham, 2020 ) , and Pherson et al. ( Pherson et al., 2021 ) .

#	Strategy	Definition & Explanation	Example
Original Text: "The new AI system is very good. It can do many things fast. Users like it alot because it helps them work better and saves time."
1	Rewrite	Completely restructuring the original text while preserving the core message and factual content. This involves changing sentence structure, word choice, and overall flow to produce a substantially different version that maintains semantic equivalence.	"Leveraging advanced machine learning capabilities, this innovative artificial intelligence platform delivers exceptional performance across diverse applications, earning widespread user adoption through its ability to streamline workflows and dramatically reduce task completion times."
2	Polish	Enhancing the clarity, readability, and professional quality of the text without fundamentally altering its structure or meaning. This includes improving grammar, fixing awkward phrasing, enhancing transitions, and ensuring consistent tone throughout.	"The new AI system performs exceptionally well . It can execute many things quickly . Users appreciate it because it helps them work more efficiently and saves considerable time."
3	Refine	Making targeted, surgical improvements to specific elements like minor adjustments, grammar corrections, and small error fixes to the text while preserving the original content and structure. This involves enhancing quality with minimal content alteration.	"The new AI system is very good. It can do many things fast. Users like it a lot because it helps them work better and saves time."

Appendix

Table 34. AI-editing strategies used in BLUFF ’s human-AI collaborative text generation. Each strategy is applied at three intensity levels (light, moderate, complete), yielding 9 unique combinations. These strategies simulate real-world scenarios where humans refine AI outputs or AI assists human writing. Taxonomy adapted from ( Artemova et al., 2025 ) . Blue text indicates changes from the original.

Dimension	#	Metric	Type	Real Thresh.	Fake Thresh.
Consistency	C1	Factual Consistency: Accuracy of facts and details from original	Score (1–5) + Label	$\geq$ 4.0	$\leq$ 3.0
	C2	Logical Consistency: Absence of contradictions, logical structure	Score (1–5) + Label	$\geq$ 4.0	$\leq$ 4.0
	C3	Semantic Consistency: Preservation of key meaning and intent	Score (1–5) + Label	$\geq$ 4.0	$\leq$ 3.0
	C4	Contextual Consistency: Alignment with broader context and tone	Score (1–5) + Label	$\geq$ 4.0	$\leq$ 3.0
	C5	Topic Match: Agreement of main topic (1–2 words)	Label (matched/mismatched)	Match	Match
	C6	Sentiment Match: Emotional alignment (positive/neutral/negative)	Label (matched/mismatched)	Match	–

Appendix

Table 35. mPURIFY evaluation metrics checklist. Each generated sample is evaluated across 34 features spanning 4 quality dimensions: Consistency (17), Validation (8), Translation (7), and Manipulation (2). Score-based metrics use asymmetric thresholds: real news requires

\geq

4.0 (high fidelity), fake news accepts

\geq

3.0 (allowing deliberate deviations). Full metric specifications appear in subsection C.1 .

Figure 20. BLUFF Crawler framework for human-written data collection leveraging IFCN and Credibility Coalition approved fact-checking sources. The main crawler component extracts six data types.

Figure 21. Step 1: Claims data and metadata extraction from fact-checking sources.

Figure 22. Step 2: News content extraction from fact-check articles.

Figure 23. Step 3: Original source content retrieval.

Figure 24. Step 4: Social engagement data collection.

Abbrev.	Content Type	Chain (R/F)	Language
Orig	Original seed article	–	Source
NA (Src)	News Article	C4/C6	Source
NA (Tgt)	News Article (translated)	C5/C7	Target
SM (Src)	Social Media post	C8/C10	Source
SM (Tgt)	Social Media post (translated)	C8/C10	Target

Appendix

Table 36. Content notation mapping for S-AEM evaluation.

Pair	Reference	Candidate	Format	Direction
P1	Orig	NA (Src)	Article	Src $\rightarrow$ Src
P2	Orig	NA (Tgt)	Article	Src $\rightarrow$ Tgt
P3	Orig	SM (Src)	Post	Src $\rightarrow$ Src
P4	Orig	SM (Tgt)	Post	Src $\rightarrow$ Tgt
T1	NA (Src)	NA (Tgt)	Article	Src $\rightarrow$ Tgt
T2	SM (Src)	SM (Tgt)	Post	Src $\rightarrow$ Tgt

Appendix

Table 37. S-AEM comparison pairs for quality evaluation.

Output Column	Pair	Ground Truth	Claim (Check)
halluc_na_src	P1	Orig	NA (Src)
halluc_na_tgt	P2	Orig	NA (Tgt)
halluc_sm_src	P3	Orig	SM (Src)
halluc_sm_tgt	P4	Orig	SM (Tgt)

Appendix

Table 38. Hallucination detection configuration using SelfCheckGPT.

Output Column	Pair	Reference	Hypothesis	Method
menli_na_src	P1	Orig	NA (Src)	MENLI
menli_na_tgt	P2	Orig	NA (Tgt)	MENLI
menli_sm_src	P3	Orig	SM (Src)	MENLI
menli_sm_tgt	P4	Orig	SM (Tgt)	MENLI
frugal_na_src	P1	Orig	NA (Src)	FrugalScore
frugal_na_tgt	P2	Orig	NA (Tgt)	FrugalScore

Appendix

Table 39. Logical consistency metrics configuration.

Output Column	Pair	Context	Claim
align_na_src	P1	Orig	NA (Src)
align_na_tgt	P2	Orig	NA (Tgt)
align_sm_src	P3	Orig	SM (Src)
align_sm_tgt	P4	Orig	SM (Tgt)

Appendix

Table 40. Factual consistency using AlignScore (XLM-R Large).

Output Column	Pair	Reference	Candidate
bert_na_src	P1	Orig	NA (Src)
bert_na_tgt	P2	Orig	NA (Tgt)
bert_sm_src	P3	Orig	SM (Src)
bert_sm_tgt	P4	Orig	SM (Tgt)

Appendix

Table 41. Semantic consistency using BERTScore (XLM-R).

Output Column	Pair	Text 1	Text 2	Model
sent_na_src	P1	Orig	NA (Src)	RoBERTa-ML
sent_na_tgt	P2	Orig	NA (Tgt)	RoBERTa-ML
sent_sm_src	P3	Orig	SM (Src)	Twitter-XLM-R
sent_sm_tgt	P4	Orig	SM (Tgt)	Twitter-XLM-R

Appendix

Table 42. Sentiment consistency using multilingual sentiment classifiers.

Output Column	Pair	Source	Translation	Method
comet_na	T1	NA (Src)	NA (Tgt)	COMET-QE
comet_sm	T2	SM (Src)	SM (Tgt)	COMET-QE
labse_na	T1	NA (Src)	NA (Tgt)	LaBSE
labse_sm	T2	SM (Src)	SM (Tgt)	LaBSE
yisi_na	T1	NA (Src)	NA (Tgt)	YiSi-2
yisi_sm	T2	SM (Src)	SM (Tgt)	YiSi-2

Appendix

Table 43. Translation semantic quality metrics (averaged).

Output Column	Input	Expected	Detectors
langid_na	NA (Tgt)	Target lang	fasttext, pycld3, Polyglot
langid_sm	SM (Tgt)	Target lang	fasttext, pycld3, Polyglot

Appendix

Table 44. Language identification using majority voting across three detectors.

Output Column	Pair	Sentence 1	Sentence 2	Expected
transdir_na	T1	NA (Src)	NA (Tgt)	Src $\rightarrow$ Tgt
transdir_sm	T2	SM (Src)	SM (Tgt)	Src $\rightarrow$ Tgt

Appendix

Table 45. Translation direction verification.

Output Column	Input	Expected Label	Condition
auth_orig	Orig	HWT	Always
auth_na_src	NA (Src)	HAT or MGT	minor–moderate $\rightarrow$ HAT; complete/critical $\rightarrow$ MGT
auth_na_tgt	NA (Tgt)	MTT	Always (translated)
auth_sm_src	SM (Src)	HAT or MGT	minor–moderate $\rightarrow$ HAT; complete/critical $\rightarrow$ MGT
auth_sm_tgt	SM (Tgt)	MTT	Always (translated)

Appendix

Table 46. Authorship classification using LLM-DetectAIve.

Output Column	Pair	Text 1	Text 2
edit_na_src	P1	Orig	NA (Src)
edit_na_tgt	P2	Orig	NA (Tgt)
edit_sm_src	P3	Orig	SM (Src)
edit_sm_tgt	P4	Orig	SM (Tgt)

Appendix

Table 47. Edit distance metrics (averaged).

Dimension	Metric	Columns	Aggregation
Hallucination	SelfCheckGPT	4	vote
Consistency	MENLI (Logical)	4	vote/avg
	FrugalScore (Logical)	4	score
	AlignScore (Factual)	4	score
	BERTScore (Semantic)	4	score
	Sentiment	4	vote

Appendix

Table 48. S-AEM output columns summary (43 total).

Method	Multilingual	Coverage
SelfCheckGPT (NLI)	✓	XLM-R based
MENLI	✓	cross_lingual=True
FrugalScore	✗	English only
AlignScore	✓	XLM-R Large
BERTScore	✓	130 languages
Sentiment (articles)	✓	16+ languages

Appendix

Table 49. S-AEM multilingual support summary.

Panel of Figure 25 — Figure 25. Language coverage across BLUFF subsets. Of 79 unique languages, 49 appear in both subsets, 22 are exclusive to the AI-generated data, and 8 are exclusive to the human-written data. Big-head ( $\bullet$ ) and long-tail ( $\circ$ ) classifications based on web content distribution ( W3Techs, 2024 ) .

Code	Lang	Count	R/F	Code	Lang	Count	R/F	Code	Lang	Count	R/F
agm	$\circ$ Angaatiha	1	0/0	hun	$\circ$ Hungarian	13,306	0/13,294	ron	$\circ$ Romanian	222	0/197
ara	$\bullet$ Arabic	355	0/351	ind	$\bullet$ Indonesian	537	0/233	rus	$\bullet$ Russian	13,623	0/13,617
asm	$\circ$ Assamese	29	0/29	ita	$\bullet$ Italian	13,381	0/13,206	sin	$\circ$ Sinhala	67	0/65
ben	$\circ$ Bengali	263	0/260	jpn	$\bullet$ Japanese	5	0/5	slk	$\circ$ Slovak	13,410	0/13,194
bos	$\circ$ Bosnian	99	0/91	kat	$\circ$ Georgian	440	0/416	slv	$\circ$ Slovenian	1	0/1
bul	$\circ$ Bulgarian	189	0/142	kaz	$\circ$ Kazakh	2	0/2	spa	$\bullet$ Spanish	11,480	130/9,183

Appendix

Table 50. Language distribution in the human-written (HWT) subset of BLUFF . Data sourced from IFCN-certified fact-checking organizations and CredCatalog-indexed sources across 57 languages. Each of the 122,836 unique samples contains a human-written component (article or post) paired with its machine-generated counterpart, translations, and summary, yielding 573,133 total text instances after passing through our mPURIFY quality pipeline. Languages are classified as big-head (

\bullet

, high-resource) or long-tail (

\circ

, low-resource) based on web content distribution ( W3Techs, 2024 ) .

Code	Lang	Count	R/F	H/M/T	Code	Lang	Count	R/F	H/M/T	Code	Lang	Count	R/F	H/M/T
afr	$\circ$ Afrikaans	1,300	675/625	1k/300/1.3k	hin	$\circ$ Hindi	1,211	644/567	924/287/1.2k	per	$\bullet$ Persian ∗	1,144	656/488	937/207/1.1k
amh	$\circ$ Amharic †	635	373/262	539/96/635	hrv	$\circ$ Croatian	1,136	644/492	896/240/1.1k	pol	$\bullet$ Polish	1,280	666/614	995/285/1.3k
ara	$\bullet$ Arabic	1,263	658/605	970/293/1.3k	hun	$\circ$ Hungarian	1,248	640/608	950/298/1.2k	por	$\bullet$ Portuguese	1,286	692/594	983/303/1.3k
aze	$\circ$ Azerbaijani	1,254	660/594	977/277/1.3k	ibo	$\circ$ Igbo †	418	314/104	376/42/418	ron	$\circ$ Romanian	1,272	651/621	979/293/1.3k
ban	$\circ$ Balinese	1,053	630/423	829/224/1.1k	ind	$\bullet$ Indonesian	1,272	646/626	987/285/1.3k	rus	$\bullet$ Russian	1,296	667/629	997/299/1.3k
ben	$\circ$ Bengali	1,206	642/564	951/255/1.2k	ita	$\bullet$ Italian	1,295	667/628	1k/273/1.3k	sin	$\circ$ Sinhala	961	550/411	750/211/961

Appendix

Table 51. Language distribution in the AI-generated subset of BLUFF . 79,943 unique samples across 71 languages after mPURIFY quality filtering. Each sample contains 4 associated content fields with binary labels: HAT (Human-AI Text), MGT (Machine-Generated Text), MTT (Machine-Translated Text), and HWT (Human-Written Text). R/F shows Real/Fake news distribution. Languages with fewer than 100 samples in either veracity class were augmented with 300 additional samples ( † ). Languages are classified as big-head (

\bullet

, high-resource) or long-tail (

\circ

, low-resource) based on web content distribution ( W3Techs, 2024 ) .

Classification	Big-Head Languages	Long-tail Languages
Genetic Relationship
Major Language Families
Indo-European	eng , deu , nld , spa , por , fra , ita , pol , rus , ces , ukr , fas , ell	afr , dan , nor , swe , ron , cat , hrv , bos , bul , srp , slk , slv † , mkd , hin , ben , pan , mar , guj , nep , urd , sin , kur , sqi , lit , lav , asm † , ori † , glg †
Sino-Tibetan	zho	mya
Tai-Kadai		tha
[1pt/1pt]

Appendix

Table 52. Hierarchical classification of BLUFF languages (79 total) by genetic, script, and syntactic features ( International Organization for Standardization, 2007 ; Hammarström et al., 2024 ; International Organization for Standardization, 2004 ; Dryer and Haspelmath, 2013 ) . Languages are categorized as big-head (high-resource, 20 languages) or long-tail (low-resource, 59 languages) based on web content distribution ( W3Techs, 2024 ) . Genetic Relationship: Major families (Indo-European, Sino-Tibetan) and minor families (Afro-Asiatic, Dravidian, Creole). Script Relationship: Major (Latin) and minor scripts (Cyrillic, Arabic, CJK, Indic). Syntactic Relationship: Word orders (SVO, SOV, VSO, Free). Big-head languages marked with green ; long-tail with gray . † = human-written only.

Region	Big-Head Languages	Long-tail Languages
Europe
Western Europe	eng , deu , fra , nld	afr
Southern Europe	spa , por , ita , ell	cat , sqi , glg †
Eastern Europe	rus , ukr , pol , ces	hun , ron , bul , slk , kaz †
Balkans		hrv , bos , srp , mkd , slv †
Northern Europe		dan , nor , swe , fin , est , lav , lit

Appendix

Table 53. Regional distribution of BLUFF languages (79 total) across 6 continents and 15 sub-regions ( United Nations Statistics Division, 2024 ) . Languages span Europe (28), Asia (24), Africa (11), Middle East (2), Americas/Caribbean (4), and other categories (3). Big-head languages (top 20 by web content ( W3Techs, 2024 ) ) marked with green ; long-tail with gray . † = human-written only. Note: Some languages appear in multiple regions (e.g., afr in Western Europe and Southern Africa).

Task	Classes	#	Baseline
Veracity Binary	Real, Fake	2	50.0%
Veracity Multiclass	HWT-Real, MGT-Real, MTT-Real, HAT-Real,	8	12.5%
	HWT-Fake, MGT-Fake, MTT-Fake, HAT-Fake
MGT Binary	Human, Machine	2	50.0%
MGT Multiclass	HWT, MGT, MTT, HAT	4	25.0%

Appendix

Table 54. Task definitions and class distributions.

Setting	Train	Val	Test	Big-head	Long-tail
Multilingual (60/15/25)	38,533	9,633	16,055	4,993	11,062
Cross-lingual	15,968	1,996	46,257	1,997	44,260
External	47,142	5,238	370,379	322	1,057

Appendix

Table 55. Main training settings. Splits shared across all 4 tasks.

Family	Train	Val	Test	Big-h.	Long-t.	Langs
Indo-European	29,305	3,663	27,589	6,987	20,602	42
Indic	8,174	1,022	54,003	19,961	34,042	12
Afro-Asiatic	4,972	621	58,006	18,963	39,043	7
Uralic	2,396	300	61,225	19,961	41,264	3
Dravidian	2,395	299	61,227	19,961	41,266	3
Austronesian	2,037	255	61,674	18,963	42,711	4

Appendix

Table 56. Cross-lingual splits by language family (15 families).

Script	Train	Val	Test	Big-h.	Long-t.	Langs
Latin	28,829	3,604	28,184	7,984	20,200	48
Indic	8,174	1,022	54,003	19,961	34,042	12
Cyrillic	3,996	499	59,226	17,964	41,262	6
Arabic	3,192	399	60,230	17,966	42,264	4
CJK	2,395	299	61,227	16,967	44,260	3
Greek	798	100	63,223	18,963	44,260	1

Appendix

Table 57. Cross-lingual splits by script type (11 scripts).

Word Order	Train	Val	Test	Big-h.	Long-t.	Langs
SVO	29,450	3,681	27,408	5,987	21,421	43
SOV	17,759	2,220	42,022	15,970	26,052	25
Free	3,368	421	60,010	18,963	41,047	5
VSO	798	100	63,223	18,963	44,260	1

Appendix

Table 58. Cross-lingual splits by syntactic word order (4 types).

Parameter	Value
Optimizer	AdamW
Learning rate	2e-5
Batch size	8
Epochs	3
Weight decay	0.01
Warmup ratio	0.1 (10% of training steps)

Appendix

Table 59. Encoder fine-tuning hyperparameters.

Model	Identifier	Params	Languages
mBERT	bert-base-multilingual-cased	177M	104
mDeBERTa	microsoft/mdeberta-v3-base	278M	100+
XLM-RoBERTa	xlm-roberta-base	278M	100
XLM-RoBERTa-L	xlm-roberta-large	559M	100
XLM-100 a	xlm-mlm-100-1280	570M	100
XLM-17 b	xlm-mlm-17-1280	570M	17

Appendix

Table 60. Encoder model specifications.

Parameter	Value
Max new tokens	10
Temperature	0.1 (near-deterministic)
Top-p	1.0
Quantization	4-bit (models $\geq$ 7B)
Prompt mode	Cross-lingual (English prompts)
	Native (target language prompts)

Appendix

Table 61. Decoder inference hyperparameters.

Model	Identifier	Params	Quant.
Gemma-3-270M	google/gemma-3-270m-it	270M	None
Qwen3-0.6B	Qwen/Qwen3-0.6B	0.6B	None
Gemma-3-1B	google/gemma-3-1b-it	1B	None
Llama-3.2-1B	meta-llama/Llama-3.2-1B-Instruct	1B	None
Mistral-7B	mistralai/Mistral-7B-Instruct-v0.3	7B	4-bit
Qwen3-8B	Qwen/Qwen3-8B	8B	4-bit

Appendix

Table 62. Decoder model specifications.

Resource	Specification
GPUs	8 $\times$ NVIDIA H100 80GB
Training time (encoder)	2–4 hours per model/task
Inference time (decoder)	4–8 hours per model/task
Total GPU hours	$\sim$ 1,200 hours
Framework	PyTorch 2.1 + HuggingFace Transformers 4.43

Appendix

Table 63. Computational requirements.

Model	Overall	Big-head	Long-tail	$\Delta$	Time (min)
mDeBERTa	67.3	65.1	59.5	+5.6	6.6
mBERT	64.3	64.3	54.3	+10.0	3.3
XLM-E (InfoXLM)	52.8	53.1	49.2	+3.9	25.8
XLM-T (Twitter)	49.1	48.8	52.2	$-$ 3.4	9.9
S-BERT (LaBSE)	47.5	46.2	50.2	$-$ 4.0	3.4
XLM-R-Large	46.8	46.3	52.6	$-$ 6.3	26.4

Appendix

Table 64. External evaluation results. Macro-F1 (%) and inference time.

\Delta

= Big-head

-

Long-tail. Best in bold .

Source	mBERT	S-BERT	XLM-100	XLM-17	XLM-E	XLM-R	XLM-R-L	XLM-T	mDeB.	Samples	Langs
constraint_hi	.578	.590	.327	.327	.675	.672	.667	.645	.327	2,000	1
LIF-FakeBrCorpus	.615	.473	.311	.311	.602	.462	.489	.577	.311	2,000	1
MIDe	.490	.420	.365	.365	.413	.450	.443	.428	.365	2,000	2
CoAID	.581	.422	.296	.296	.514	.416	.407	.403	.296	2,000	1
MM-COVID	.526	.367	.308	.308	.368	.480	.479	.474	.308	2,000	27
Multiclaim	.487	.570	.207	.207	.624	.618	.619	.532	.207	1,751	39

Appendix

Table 65. Source-level performance matrix (macro-F1). Best model per source in bold . Sources ordered by sample count.

Source	Language(s)	Eval	Samples
Evaluated Sources (14)
constraint_hi	Hindi	✓	2,000
FakeCoVID	Multilingual (29)	✓	1,097
LIF-FakeBrCorpus	Portuguese	✓	2,000
Multiclaim	Multilingual (39)	✓	1,751
CHECKED	Chinese	✓	1,344

Appendix

Table 66. External source inventory. ✓= evaluated, – = no language overlap.

BIG-HEAD LANGUAGES
Group	Family	Lang	mBERT	mDeBERTa	XLM-R	XLM-100	XLM-17	XLM-B	XLM-T	XLM-E	XLM-V	S-BERT
	Afro-Asiatic	ara	83.3	98.9	49.9	54.2	72.2	82.8	49.9	45.4	75.5	49.9
	Austroasiatic	vie	94.3	100.0	92.7	79.6	61.3	97.0	86.6	62.0	84.5	98.0
	Austronesian	ind	97.7	94.3	94.1	64.0	73.5	94.0	91.8	55.7	87.4	98.0
	Indo-European	ces	49.5	94.9	49.5	82.7	55.5	98.4	49.5	51.8	82.2	49.5
		deu	78.0	100.0	89.6	75.5	89.7	96.3	94.3	39.0	88.9	87.1

Appendix

Table 67. Binary veracity classification per language (Multilingual Setting) – Macro-F1 %. Languages grouped by resource level and family. Best per row in bold .

BIG-HEAD LANGUAGES
Group	Family	Lang	mBERT	mDeBERTa	XLM-R	XLM-100	XLM-17	XLM-B	XLM-T	XLM-E	XLM-V	S-BERT
	Afro-Asiatic	ara	49.7	49.9	49.9	63.6	29.1	83.5	49.9	96.9	68.0	99.6
	Austroasiatic	vie	67.7	94.8	88.6	25.0	60.7	91.1	74.5	95.0	98.0	94.6
	Austronesian	ind	81.6	96.6	92.6	25.0	53.3	79.7	88.0	95.3	81.2	97.2
	Indo-European	ces	49.3	74.7	74.7	25.2	28.4	78.7	69.5	96.1	98.0	92.5
		deu	49.1	100.0	82.6	41.7	47.0	97.3	82.8	84.2	67.8	99.3

Appendix

Table 68. Binary veracity classification per language (Cross-lingual Setting) – Macro-F1 %. Languages grouped by resource level and family. Best per row in bold .


		Gemma-270M			Qwen3-0.6B			Gemma-1B			Llama-3.2-1B			Mistral-7B			Llama-3.1-8B			Qwen3-8B
Family	Lg	T1	T2	T3	T1	T2	T3	T1	T2	T3	T1	T2	T3	T1	T2	T3	T1	T2	T3	T1	T2	T3
Afro-Asiatic	ara	37.9	36.8	23.4	30.2	46.9	44.4	25.2	24.9	31.4	67.5	58.7	32.5	39.6	41.4	48.4	67.6	38.8	41.1	61.6	67.8	57.5
Austroasiatic	vie	34.2	34.0	38.9	28.8	48.4	21.6	27.1	25.2	24.3	47.5	31.4	33.3	51.2	62.1	39.2	76.8	37.7	35.0	86.9	73.5	54.1
Austronesian	ind	34.7	37.0	22.6	43.6	40.0	54.5	46.8	58.4	54.6	30.8	30.8	36.1	53.8	72.3	37.0	49.1	73.1	55.2	66.4	56.6	44.9
Indo-European	por	19.8	21.3	33.5	58.6	47.3	28.5	41.8	26.8	48.0	30.9	46.3	44.5	76.1	53.6	82.9	47.1	64.9	37.3	83.7	67.9	61.4

Appendix

Table 69. Binary veracity classification per language (Decoder Models) – Macro-F1 %. T1 = Native Prompt, T2 = Cross-lingual Prompt, T3 = English Translated. Best per row in bold .


Family	Lg	mDeBERTa	XLM-R	XLM-R-L	mBERT	XLM-100	XLM-17	XLM-T	XLM-E	S-BERT
Afro-Asiatic	ara	66.4	42.0	41.9	45.8	62.5	15.1	40.0	60.5	77.3
Austroasiatic	vie	94.4	69.5	62.5	52.3	25.0	23.5	37.1	61.2	65.4
Austronesian	ind	74.2	65.7	66.3	71.6	25.4	16.1	74.3	67.1	61.1
Indo-European	por	87.0	85.5	88.6	81.1	68.9	22.4	52.5	73.8	85.6
	pol	32.5	82.7	94.0	62.1	59.3	42.8	72.4	45.7	75.5

Appendix

Table 70. Multiclass veracity classification per language (Encoder Models, Cross-lingual) – Macro-F1 %. Best per row in bold .


Family	Lg	mDeBERTa	XLM-R	XLM-R-L	mBERT	XLM-100	XLM-17	XLM-T	XLM-E	S-BERT
Afro-Asiatic	ara	50.3	89.4	83.1	36.1	74.6	35.0	33.3	56.1	81.0
Austroasiatic	vie	86.7	35.1	73.9	35.0	64.2	52.8	62.9	40.1	57.7
Austronesian	ind	40.6	56.4	84.5	74.7	25.8	50.4	37.2	48.1	38.2
Indo-European	por	61.2	89.8	68.5	53.5	53.9	28.3	73.9	72.7	68.2
	pol	63.4	51.9	81.5	62.0	37.7	72.0	51.7	60.3	87.2

Appendix

Table 71. Multiclass veracity classification per language (Encoder Models, Multilingual) – Macro-F1 %. Best per row in bold .


		Gemma-270M			Gemma-1B			Llama-3.2-1B			Qwen3-0.6B			Mistral-7B			Qwen3-8B			Llama-3.1-8B
Family	Lg	T1	T2	T3	T1	T2	T3	T1	T2	T3	T1	T2	T3	T1	T2	T3	T1	T2	T3	T1	T2	T3
Afro-Asiatic	ara	1.7	11.6	1.6	5.3	12.3	3.1	1.5	10.1	3.6	2.1	7.0	4.0	0.5	7.2	2.2	4.8	4.0	4.9	2.4	7.1	0.5
Austroasiatic	vie	2.8	10.2	3.7	5.4	7.6	6.8	4.3	10.8	3.0	3.0	3.8	3.1	3.2	7.2	4.9	0.7	3.2	2.6	0.8	6.9	1.3
Austronesian	ind	0.5	13.7	0.5	5.1	9.3	8.2	3.8	10.5	2.3	3.8	3.7	2.3	2.6	2.4	2.0	2.4	3.5	2.2	3.2	2.0	1.9
Indo-European	por	3.4	11.6	2.5	3.8	12.0	3.1	4.9	5.1	1.7	5.9	8.1	2.4	1.7	5.2	1.1	4.1	6.6	2.7	2.9	7.1	2.7

Appendix

Table 72. Multiclass veracity classification per language (Decoder Models, 0-shot) – Macro-F1 %. T1 = Native, T2 = Cross-lingual, T3 = Translate. Best per row in bold . Random baseline = 12.5%.


Family	Lg	mDeBERTa	XLM-R	XLM-R-L	mBERT	XLM-100	XLM-17	XLM-T	XLM-E	S-BERT
Afro-Asiatic	ara	72.4	63.6	93.0	76.1	67.3	89.3	64.4	100.0	87.4
Austroasiatic	vie	87.5	85.4	68.7	71.7	89.6	96.3	84.4	100.0	100.0
Austronesian	ind	74.3	100.0	79.5	91.1	71.6	79.4	74.1	100.0	85.0
Indo-European	ces	100.0	98.0	95.4	100.0	76.6	91.3	91.2	90.7	100.0
	deu	71.2	78.2	65.1	78.5	57.4	98.7	88.1	70.7	78.1

Appendix

Table 73. Binary Synthetic Text Detection per language (Encoder Models, Multilingual) – Macro-F1 %. Best per row in bold .


Family	Lg	mDeBERTa	XLM-R	XLM-R-L	mBERT	XLM-100	XLM-17	XLM-T	XLM-E	S-BERT
Afro-Asiatic	ara	94.1	77.7	65.1	88.3	88.5	84.6	86.0	59.2	80.2
Austroasiatic	vie	68.5	66.7	30.4	89.7	70.8	50.0	58.6	53.3	85.1
Austronesian	ind	100.0	61.6	61.5	77.0	90.8	50.0	82.4	59.0	100.0
Indo-European	ces	91.3	67.6	38.3	93.7	82.8	53.9	94.4	86.0	78.9
	deu	100.0	77.9	25.0	100.0	86.1	72.8	63.4	85.7	68.6

Appendix

Table 74. Binary Synthetic Text Detection per language (Encoder Models, Cross-lingual) – Macro-F1 %. Best per row in bold .


		Gemma-270M			Gemma-1B			Llama-3.2-1B			Qwen3-0.6B			Mistral-7B			Qwen3-8B			Llama-3.1-8B
Family	Lg	T1	T2	T3	T1	T2	T3	T1	T2	T3	T1	T2	T3	T1	T2	T3	T1	T2	T3	T1	T2	T3
Afro-Asiatic	ara	59.8	48.2	59.2	44.6	67.9	40.5	75.0	51.8	35.4	47.1	49.5	64.9	72.3	72.8	60.8	43.6	59.9	52.8	37.1	54.4	46.7
Austroasiatic	vie	59.6	45.6	48.5	55.5	56.2	49.7	73.2	55.4	47.5	56.6	58.5	62.8	47.7	57.5	56.0	57.1	46.2	52.5	54.4	53.2	60.8
Austronesian	ind	57.3	72.0	55.3	58.0	51.2	45.8	63.4	38.6	36.3	64.3	44.3	44.6	66.2	69.8	53.4	62.3	39.6	42.5	56.2	35.2	55.0
Indo-European	por	52.8	71.4	55.0	71.4	64.9	51.4	56.9	43.5	35.4	57.7	55.8	58.2	44.6	60.5	55.9	64.0	42.5	38.4	36.4	35.0	50.8

Appendix

Table 75. Binary Synthetic Text Detection per language (Decoder Models, 0-shot) – Macro-F1 %. T1 = Native, T2 = Cross-lingual, T3 = Translate. Best per row in bold . Random baseline = 50%.


Family	Lg	mDeBERTa	XLM-R	XLM-R-L	mBERT	XLM-100	XLM-17	XLM-T	XLM-E	S-BERT
Afro-Asiatic	ara	76.1	52.6	77.7	89.0	60.9	95.8	91.4	92.2	71.4
Austroasiatic	vie	69.0	96.7	92.1	100.0	33.1	47.5	59.4	49.0	57.7
Austronesian	ind	83.5	63.4	70.4	97.2	43.0	42.6	94.1	94.4	90.2
Indo-European	ces	60.2	94.7	60.7	53.4	49.9	84.8	86.2	99.8	66.9
	deu	68.7	81.0	79.8	71.8	71.7	58.1	76.5	43.4	66.4

Appendix

Table 76. Multiclass Synthetic Text Detection per language (Encoder Models, Multilingual) – Macro-F1 %. Best per row in bold .


Family	Lg	mDeBERTa	XLM-R	XLM-R-L	mBERT	XLM-100	XLM-17	XLM-T	XLM-E	S-BERT
Afro-Asiatic	ara	51.3	49.5	51.1	77.1	37.1	98.9	65.0	76.3	53.7
Austroasiatic	vie	69.6	89.0	82.0	100.0	32.0	40.7	76.4	88.1	98.9
Austronesian	ind	67.2	100.0	79.1	85.2	59.5	82.7	96.8	57.4	59.4
Indo-European	ces	55.9	49.2	94.0	100.0	36.3	67.4	59.8	87.6	59.5
	deu	63.3	47.3	74.3	100.0	70.8	55.9	94.4	48.6	58.1

Appendix

Table 77. Multiclass Synthetic Text Detection per language (Encoder Models, Cross-lingual) – Macro-F1 %. Best per row in bold .


		Gemma-270M			Gemma-1B			Llama-3.2-1B			Qwen3-0.6B			Mistral-7B			Qwen3-8B			Llama-3.1-8B
Family	Lg	T1	T2	T3	T1	T2	T3	T1	T2	T3	T1	T2	T3	T1	T2	T3	T1	T2	T3	T1	T2	T3
Afro-Asiatic	ara	15.8	13.8	2.4	25.7	8.6	24.4	17.7	10.3	14.9	12.8	2.7	9.8	21.6	20.5	12.5	6.5	12.7	3.2	4.2	5.4	8.8
Austroasiatic	vie	17.8	7.8	14.3	5.9	22.1	10.1	25.0	7.8	7.0	2.7	2.6	10.6	7.6	28.4	8.3	24.4	20.8	16.9	0.5	8.9	8.1
Austronesian	ind	21.2	32.8	15.9	23.2	19.9	20.4	31.1	1.2	14.3	28.6	5.6	17.8	21.1	29.8	6.7	7.3	19.6	0.6	9.4	6.7	10.8
Indo-European	por	10.8	19.0	2.5	5.4	5.9	25.4	17.8	10.6	12.3	29.0	25.9	24.0	20.2	22.9	9.0	16.6	5.3	12.1	4.1	18.1	10.5

Appendix

Table 78. Multiclass Synthetic Text Detection per language (Decoder Models, 0-shot) – Macro-F1 %. T1 = Native, T2 = Cross-lingual, T3 = Translate. Best per row in bold . Random baseline = 25%.

Fake News Detection

Fake news detection identifies false or misleading claims by analyzing content, context, evidence, and how information spreads.

See all 93 papers

Many Ways to Be Fake: Benchmarking Fake News Detection Under Strategy-Driven AI Generation
Apr 10, 2026Xinyu Wang, Sai Koneru, Wenbo Zhang +3Fake News Detection Synthetic Benchmark Generation
Pennsylvania State University University Park, PA 16802, USA
DetectRL-X: Towards Reliable Multilingual and Real-World LLM-Generated Text Detection
May 15, 2026Junchao Wu, Yefeng Liu, Chenyu Zhu +8Multilingual Language Model Evaluation AI-Generated Text Detection
NLP2CT Lab, Department of Computer and Information Science, University of Macau · Alibaba Group · Xiamen University
When Spoof Detectors Travel: Evaluation Across 66 Languages in the Low-Resource Language Spoofing Corpus
Mar 2, 2026Kirill Borodin, Vasiliy Kudryavtsev, Maxim Maslov +2Audio Deepfake Detection Speech Processing
lab260, Moscow Technical University of Communications and Informatics, Russia · BitmanagerAI, UAE

Apr 10, 2026J/K move · Enter open · S save

Many Ways to Be Fake: Benchmarking Fake News Detection Under Strategy-Driven AI Generation

Xinyu Wang, Sai Koneru, Wenbo Zhang, Wenliang Zheng+2

Pennsylvania State University University Park, PA 16802, USA

Fake News Detection Synthetic Benchmark Generation

Open paper AskPDF arXiv

Recent advances in large language models (LLMs) have enabled the large-scale generation of highly fluent and deceptive news-like content. While prior work has often treated fake news detection as a binary classification problem, modern fake news increasingly arises through human-AI collaboration, where strategic inaccuracies are embedded within otherwise accurate and credible narratives. These mixed-truth cases represent a realistic and consequential threat, yet they remain underrepresented in existing benchmarks. To address this gap, we introduce MANYFAKE, a synthetic benchmark containing 6,798 fake news articles generated through multiple strategy-driven prompting pipelines that capture many ways fake news can be constructed and refined. Using this benchmark, we evaluate a range of state-of-the-art fake news detectors. Our results show that even advanced reasoning-enabled models approach saturation on fully fabricated stories, but remain brittle when falsehoods are subtle, optimized, and interwoven with accurate information.

Abstract

Fake News Detection

Explore similar work

Many Ways to Be Fake: Benchmarking Fake News Detection Under Strategy-Driven AI Generation

DetectRL-X: Towards Reliable Multilingual and Real-World LLM-Generated Text Detection

When Spoof Detectors Travel: Evaluation Across 66 Languages in the Low-Resource Language Spoofing Corpus