Automatically verifying climate-related claims against scientific literature is a challenging task, complicated by the specialised nature of scholarly evidence and the diversity of rhetorical strategies underlying climate disinformation. ClimateCheck 2026 is the second iteration of a shared task addressing this challenge, expanding on the 2025 edition with tripled training data and a new disinformation narrative classification task. Running from January to February 2026 on the CodaBench platform, the competition attracted 20 registered participants and 8 leaderboard submissions, with systems combining dense retrieval pipelines, cross-encoder ensembles, and large language models with structured hierarchical reasoning. In addition to standard evaluation metrics (Recall@K and Binary Preference), we adapt an automated framework to assess retrieval quality under incomplete annotations, exposing systematic biases in how conventional metrics rank systems. A cross-task analysis further reveals that not all climate disinformation is equally verifiable, potentially implicating how future fact-checking systems should be designed.
Figures & tables
Figure 1: Instance from ClimateCheck 2026. Task 1: Given a claim, systems must retrieve relevant abstracts and use them for verification. Task 2: Given a claim, systems must predict all disinformation narratives associated with it.
Label
Train
Test
Supports
1399
749
Refutes
451
266
Not Enough Information (NEI)
1173
889
Total
3023
1904
Table 1: Label distribution for Task 1 across training and test splits.
Task 1.1
Task 1.2
Team
R@2
R@5
Bpref
Score 1.1
P
R
F1
Score 1.2
Baseline
0.213
0.403
0.459
0.431
0.683
0.682
0.679
1.082
Ant Bridge Wang et al. (2025) †
0.218
0.451
0.495
0.473
0.729
0.726
0.725
1.176
ClimateSense Ehrhart et al. (2026)
0.221
0.443
0.489
0.466
0.742
0.744
0.740
1.183
berkbubus
0.206
0.429
0.453
0.441
0.391
0.414
0.328
0.757
DFKI-IML Liang et al. (2026)
0.193
0.352
0.401
0.377
0.670
0.641
0.620
0.972
Table 2: Results for Task 1. Task 1.1 is evaluated using Recall@2 (R@2), Recall@5 (R@5), Binary Preference (Bpref), and Score 1.1 . Task 1.2 is evaluated using weighted Precision (P), Recall (R), F1, and Score 1.2 . Best results among 2026 submissions are shown in bold . Shaded cells indicate scores that outperform the baseline.
Team
P (macro)
R (macro)
F1 (macro)
F1 (micro)
F1 (weighted)
Baseline
0.530
0.574
0.514
0.798
0.784
ahilbert Hilbert et al. (2026)
0.707
0.631
0.625
0.844
0.821
XplaiNLP Foroutan et al. (2026)
0.625
0.640
0.597
0.801
0.806
ClimateSense Ehrhart et al. (2026)
0.670
0.568
0.583
0.876
0.844
alextsiakalou
0.574
0.586
0.548
0.844
0.837
Table 3: Results for Task 2. Systems are evaluated using macro-averaged Precision (P), Recall (R), and F1, as well as micro- and weighted F1. Macro-F1 is the official ranking metric, reflecting performance across all narrative categories regardless of class frequency. Best results are shown in bold . Shaded cells indicate scores that outperform the baseline.
Team
Ev2R (r)
Sver
# Evaluated
Avg. Unannotated
Claims
Abstracts per Claim
Baseline
0.606
0.600
112
1.92
ClimateSense Ehrhart et al. (2026)
0.559
0.559
111
1.77
berkbubus
0.554
0.468
106
1.67
DFKI-IML Liang et al. (2026)
0.568
0.527
135
1.97
gardlz
0.585
0.528
134
2.16
Table 4: Ev2R evaluation results for unannotated abstracts retrieved in Task 1. Ev2R (r) is the mean of the reference-based and proxy-reference components evaluating task 1.1. Sver is the added verification score evaluating task 1.2. Only the final leaderboard submission per team is evaluated.
Figure 2: Per-claim average Recall@5 across all submitted systems, sorted in ascending order. Colours indicate retrieval difficulty.
Figure 3: Confusion matrices for the baseline, ClimateSense, and DFKI-IML predictions on task 1.2, normalised by claim. SUP = Supports, REF = Refutes, NEI = Not Enough Information.
Figure 4: Per-label F1 scores across participating systems and IAA (Krippendorff’s α , leftmost column). Labels sorted by mean system F1 (ascending); only labels with more than 3 instances in the test set are shown.
Figure 5: Refutation accuracy by narrative group based on the CARDS taxonomy for the baseline and ClimateSense systems.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Label
Narrative
N
Prev.
α
κ
PA
0 — No disinformation narrative
0_0
No disinformation narrative detected
776
.715
.722
.722
.921
1 — Global warming is not happening
1_0
Global warming is not happening (general)
12
.004
.175
.132
.133
1_1
Ice/permafrost/snow cover isn’t melting
28
.017
.672
.676
.681
1_2
We’re heading into an ice age/global cooling
23
.011
.556
.570
.575
Appendix
Table 5: Per-label IAA sorted by top-level category. N = items where ≥ 1 annotator assigned the label; Prev. = prevalence; α = Krippendorff’s alpha; κ = avg. pairwise Cohen’s kappa; PA = positive agreement (Dice).
Figure 6: Prompt used for the Ev 2 R reference-based scorer adapted for the ClimateCheck shared task based on the prompt provided by Akhtar et al. (2024) . Both the reference evidence and the retrieved evidence are decomposed into atomic facts representing generalised climate science statements before being assessed against each other.
Figure 8: Confusion matrices for the berkbubus, gardlz, and ytsoneva predictions on task 1.2, normalised by claim. SUP = Supports, REF = Refutes, NEI = Not Enough Information.
Narrative Group
R@5
N
Baseline
0
0.992
124
1
0.933
15
2
0.938
16
3
1.000
6
4
1.000
6
Appendix
Table 6: R@5 by narrative group for both systems. N indicates the number of claims per group. Results show minimal variance across narrative groups, indicating that retrieval difficulty is narrative-agnostic.
Technische Universität Berlin · German Research Center for Artificial Intelligence (DFKI) · BIFOLD – Berlin Institute for the Foundations of Learning and Data +1