Retrieving Climate Change Disinformation by Narrative
Authors: Max Upravitelev, Veronika Solopova, Charlott Jakob, Premtim Sahitaj, Sebastian Möller, Vera Schmitt
Organizations: Technische Universität Berlin · German Research Center for Artificial Intelligence (DFKI) · BIFOLD – Berlin Institute for the Foundations of Learning and Data · Centre for European Research in Trusted AI (CERTAIN)
Detecting climate disinformation narratives typically relies on fixed taxonomies, which do not accommodate emerging narratives. Thus, we re-frame narrative detection as a retrieval task: given a narrative's core message as a query, rank texts from a corpus by alignment with that narrative. This formulation requires no predefined label set and can accommodate emerging narratives. We repurpose three climate disinformation datasets (CARDS, Climate Obstruction, climate change subset of PolyNarrative) for retrieval evaluation and propose SpecFi, a framework that generates hypothetical documents to bridge the gap between abstract narrative descriptions and their concrete textual instantiations. SpecFi uses community summaries from graph-based community detection as few-shot examples for generation, achieving a MAP of 0.505 on CARDS without access to narrative labels. We further introduce narrative variance, an embedding-based difficulty metric, and show via partial correlation analysis that standard retrieval degrades on high-variance narratives (BM25 loses 63.4% of MAP), while SpecFi-CS remains robust (32.7% loss). Our analysis also reveals that unsupervised community summaries converge on descriptions close to expert-crafted taxonomies, suggesting that graph-based methods can surface narrative structure from unlabeled text.
Figures & tables
Figure 1: Overview: The SpecFi Retrieval Pipeline
narratives
mean texts
std texts
mean words
std words
mean words
std words
total
disinfo %
n
per n
per n
per n
per n
per text
per text
texts
of all texts
CARDS
17
67.65
57.49
7.61
3.96
65.35
57.60
2904
39.6
CO
7
38.29
27.20
20.50
3.56
28.27
11.95
255
73.3
PN
51
1.98
1.39
9.06
2.82
601.83
293.73
41
73.2
PN-UKR
27
2
1.47
9.32
3.30
740.62
382.17
13
100
PN-CC
23
2
1.32
8.77
2.12
495.71
120.74
17
100
Table 1: Quantitative statistics of the used datasets.
NDCG
NDCG
Average
Setup
Models
MAP
@10
@100
R-Precision
zero
4o, OI-E
0.321
0.509
0.487
0.370
shot
G,Q4B
0.313
0.469
0.456
0.371
G-a,Q4B
0.295
0.428
0.436
0.308
static*
4o, OI-E
0.488
0.713
0.649
0.487
G, Q4B
0.435
0.635
0.616
0.435
Table 2: Results on the CARDS dataset. static* is included for reference only due to its reliance on labels. All metrics are averaged over 10 runs. We report a standard deviation of <0.01 for all performance metrics. The model abbreviations are: 4o=gpt-4o, OI-E= text-embedding-3-large, G=gemma-3-27b-it, G-a=gemma-3-27b-it abliterated, Q4B=Qwen3-Embedding-4b.
NDCG
Avg.
Setup/Model
MAP
@10
R-Prec.
BM25
0.326
0.472
0.298
Qwen3-E-4B
0.499
0.607
0.491
SpecFi-DR
0.519
0.644
0.496
SpecFi-DR-a
0.482
0.604
0.494
SpecFi-CS
0.491
0.618
0.49
Table 3: Evaluation on CO. Qwen3-E-4B=Qwen3-Embedding-4B
NDCG
Avg.
Setup/Model
MAP
@10
R-Prec.
BM25
0.311
0.378
0.219
Qwen3-E-4B
0.502
0.598
0.374
SpecFi-DR
0.443
0.621
0.370
SpecFi-DR-a
0.386
0.536
0.275
SpecFi-CS
0.458
0.626
0.372
Table 4: Evaluation on PN
NDCG
Avg.
s /
Setup/Model
MAP
@10
R-Prec.
narrative
NodeRAG only
0.259
0.506
0.323
1.931
BM25
0.080
0.125
0.119
0.011
thenlper/gte-large
0.215
0.394
0.272
2.092
OpenAI-E
0.262
0.507
0.323
0.452
Qwen3-E-4B
0.299
0.523
0.352
6.645
Table 5: Retrieval performance of individual pipeline components on CARDS, serving as baselines for the statistical analysis in §5. Models: OpenAI-E=text-embedding-3-large, Qwen3-E-4B=Qwen3-Embedding-4b. Runtimes were measured on a system with a H100 GPU.
Narrative
Narrative
Setup
Dataset
Distinct.
Variance
BM25
CARDS
-0.240
-0.525 *
CO
-0.357
-0.071
PN
0.369 *
0.319 *
QWEN-E-4B
CARDS
-0.066
-0.556 *
CO
-0.679
0.000
Table 6: Spearman’s ρ between MAP and narrative metrics. FDR-corrected significance: * p<0.05, ** p<0.01.
Figure 2: Partial correlation between MAP and narrative variance on CARDS, controlling for mi . Each point represents one narrative. BM25 and QWEN-E-4B show steep negative slopes; SpecFi-CS-a shows no significant trend. All p -values are FDR-corrected (Benjamini–Hochberg across all tests in Table 7 ).
Figure 3: Mean MAP for narratives above and below median Vi on CARDS. BM25 loses 63.4% of its MAP on high-variance narratives; SpecFi-CS-a loses 32.7%.
Original correlations
Setup
Di
Vi
BM25
-0.240 (.530)
-0.525 (.123)
QWEN-E-4B
-0.066 (.874)
-0.556 (.122)
SpecFi-DR-a
0.147 (.704)
-0.578 (.122)
SpecFi-CS-a
0.282 (.468)
-0.324 (.468)
Partial correlations (controlling for mi )
Table 7: Spearman’s ρ between MAP and narrative metrics on CARDS. FDR-corrected p -values; bold p<0.05 .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: AP results per narrative on the CARDS dataset
QWEN
SpecFi
SpecFi
#
Metric
Setup
BM25
-E-4B
-DR-a
-CS-a
1
RHO
BM25
1.000
0.824
0.544
0.365
QWEN-E-4B
0.824
1.000
0.679
0.520
SpecFi-DR-a
0.544
0.679
1.000
0.892
SpecFi-CS-a
0.365
0.520
0.892
1.000
FDR-p
BM25
0.000
0.000
0.060
0.249
Appendix
Table 8: Spearman’s rho and FDR-corrected p-values for correlation analysis of performances on the MAP metric. #1 documents the results on CARDS, #2 on CO and #3 on PN.
ID
Narrative Label
Pattern
Community Summary
1: Global warming is not happening
1_1
Ice/permafrost/snow cover isn’t melting
partial
The text explores the concept of anthropogenic global warming as a myth, questioning its validity and the narratives surrounding it.
1_2
Heading into ice age/global cooling
converg.
There are emerging voices cautioning against the narrative of catastrophic global warming, suggesting that the Earth may be entering a cooling cycle.
1_3
Weather is cold/snowing
drift
Severe weather events, such as unexpected snowfall, significantly affect city operations and highlight the need for preparedness in urban planning.
1_4
Climate hasn’t warmed over the last decade(s)
partial
Maps generated by climate models, used by the IPCC, are criticized as ‘fantasy maps’ that do not accurately reflect Earth’s climate history or current state.
1_6
Sea level rise is exaggerated/not accelerating
converg.
Recent studies indicate that the rate of sea level rise has remained consistent, contradicting some climate model predictions, which raises questions about their reliability.
Appendix
Table 9: CARDS narrative taxonomy with community summaries generated within the NodeRAG framework. Pattern categories: convergence (summary aligns with taxonomy label), partial (aligns at super-claim level), collapse (identical summary shared with sibling sub-narrative), drift (correct topic, wrong stance or focus), incoherent (summary unrelated to narrative).
Technische Universität Berlin · German Research Center for Artificial Intelligence (DFKI) · BIFOLD – Berlin Institute for the Foundations of Learning and Data +2