Checking whether a generated sentence is supported by its source document is a standard safeguard for AI-written text, and the strongest published checkers are single models. We describe GauntletScore, an evidence engine whose few rules and thresholds were set on the benchmark's development split. It puts typed questions to independent readers from different providers and records each answer as a measurement in an evidence ledger. It combines the measurements with a fixed Bayesian rule that counts a contradiction only when two readers agree. On the full LLM-AggreFact test split (29,320 claims across 11 datasets), following the leaderboard's approach, GauntletScore achieves 79.51 balanced accuracy, above every average published on the leaderboard as of 15 November 2026. That places it level with the best of them, nimble-v4-max (79.45 from its published dataset scores; the engine's 95% interval, 78.47 to 80.53, contains it). On the same claims it beats two of its three readers used alone (Readers A and B) by 1.66 and 2.17 points (95% intervals 0.80 to 2.53 and 1.29 to 3.06). Without AggreFact-CNN, described below, the leads are 0.90 and 1.32 and both intervals stay above zero. The engine also records the evidence behind every verdict, which a single model does not. Its largest margin is on full-length news articles checked against multi-sentence summaries (AggreFact-CNN), a format that resembles much regulated writing. There it scores 76.45, the best on the leaderboard, against 70.1 for the next best and 63.9 to 67.8 for the six systems with the highest averages. The test split was run once, on an engine frozen before the first test claim, under a stopping plan committed in advance. We release per-claim predictions and the scoring script.
Figures & tables
Figure 1: The engine’s stages, from a claim and its source document to a verdict with its reason.
Dataset
Claims
RAGTruth
16,371
ExpertQA
3,702
LFQA
1,911
REVEAL
1,710
FactCheck-GPT
1,566
ClaimVerify
1,088
Table 1: Claims per dataset in the LLM-AggreFact test split.
Model
Size
Avg
CNN
XSum
MediaS
MeetB
WiCE
REVEAL
ClaimVerify
FactCheck
ExpertQA
LFQA
RAGTruth
GauntletScore (this work)
–
79.51
76.45
74.95
74.74
80.38
81.94
90.63
77.52
80.89
59.98
89.33
87.84
nimble-v4-max
27B
79.5
67.6
78.6
73.3
82.4
87.1
89.0
75.9
80.2
60.4
89.0
90.5
Clef
27B
79.0
67.3
75.7
75.3
81.1
85.8
89.6
78.6
80.8
59.0
89.0
87.3
Decider v1.2
27B
78.7
65.5
77.9
72.4
80.3
87.1
90.3
75.8
78.7
60.6
88.4
89.0
Jev 1.13.0
–
78.7
67.2
75.5
74.5
83.7
83.9
90.9
76.9
77.0
59.9
89.2
87.0
d1-ultra
–
78.5
63.9
77.6
74.4
79.9
82.7
91.6
76.8
79.0
60.2
87.9
89.2
Table 2: LLM-AggreFact test split, balanced accuracy (%). Published rows read from the leaderboard ( https://llm-aggrefact.github.io/ ) on 9 November and rechecked on 15 November 2026; the leaderboard shows one decimal. The eleven published scores for nimble-v4-max average 79.45. The published rows are the six systems with the highest averages.
Dataset
Engine
Reader A alone
Reader B alone
Engine minus A
Engine minus B
Average of 11
79.51
77.85
77.34
+ 1.66
+ 2.17
AggreFact-CNN
76.45
67.12
65.77
+ 9.33
+ 10.68
LFQA
89.33
84.87
87.14
+ 4.46
+ 2.19
ClaimVerify
77.52
73.54
74.41
+ 3.98
+ 3.11
REVEAL
90.63
88.04
90.78
+ 2.59
− 0.15
TofuEval-MediaS
74.74
72.37
69.98
+ 2.37
+ 4.76
Table 3: The engine against Readers A and B alone, on the same claims, balanced accuracy (%). Full test split, 29,320 claims. Intervals are under Statistical tests below.
System
Supported claims kept (true-positive rate)
Unsupported claims caught (true-negative rate)
Balanced accuracy
Engine
82.5
77.5
80.00
Reader A alone
89.5
67.8
78.67
Reader B alone
89.0
66.6
77.81
Reader C alone
79.7
79.5
79.63
Table 4: Supported claims kept and unsupported claims caught, 2,992-claim sample (%). Means over the 11 datasets.
Document length (words)
Claims
Engine
Reader A alone
Reader B alone
Engine minus A
Engine minus B
Under 200
8,524
80.99
80.20
80.49
+ 0.8
+ 0.5
200 to 399
11,185
80.17
78.45
77.62
+ 1.7
+ 2.5
400 to 799
6,056
75.75
74.17
73.91
+ 1.6
+ 1.8
800 and over
3,555
77.44
74.71
74.30
+ 2.7
+ 3.1
Table 5: Balanced accuracy by source-document length, full test split (%): Each cell is the mean over the datasets that have both labels in that band (8 or 9 of the 11).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Supported claims kept
Unsupported claims caught
Dataset
Supported
Unsupported
Engine
A alone
B alone
Engine
A alone
B alone
AggreFact-CNN
501
57
95.0
97.4
98.2
57.9
36.8
33.3
AggreFact-XSum
285
273
64.9
76.5
84.9
85.0
75.1
62.3
ClaimVerify
789
299
85.8
91.9
90.6
69.2
55.2
58.2
ExpertQA
2,971
731
60.5
70.1
63.2
59.5
51.3
57.3
FactCheck-GPT
376
1,190
72.9
78.7
71.3
88.9
83.9
89.2
Appendix
Table 6: Supported claims kept (true-positive rate) and unsupported claims caught (true-negative rate), full test split (%). The last row gives total counts and mean rates.
Dataset
Engine
Reader C alone
Engine minus C
Average of 11
80.00
79.63
+ 0.37
AggreFact-CNN
77.09
74.99
+ 2.10
AggreFact-XSum
74.26
76.47
− 2.21
ClaimVerify
77.21
76.10
+ 1.11
ExpertQA
63.24
60.29
+ 2.95
FactCheck-GPT
79.78
78.31
+ 1.47
Appendix
Table 7: The engine and Reader C alone on the 2,992-claim sample, balanced accuracy (%). 272 claims per dataset, so single-dataset gaps of 2 to 3 points are within noise. Readers A and B on the full split are in Table 3 .
Large language models (LLMs) can achieve strong fact-checking accuracy, yet forced binary decisions conceal a critical reliability problem: systems may issue confident verdicts even when supporting evidence is weak, sparse, or internally inconsistent. We address this issue through Evidence Chain Evaluation (ECE), a selective fact-checking framework that permits abstention via an uncertain verdict instead of requiring a true/false decision for every claim. The evaluated system is a tool-using verification agent that gathers evidence through web search, scholarly search, and executable checks, and then returns a structured verdict with confidence and source-level metadata. On ECE-Bench, ECE achieves 91.6% standard accuracy, 93.7% coverage, and 97.8% selective accuracy on answered claims. Although ECE does not outperform the strongest retrieval baseline on aggregate calibration metrics such as Expected Calibration Error, Brier score, or AURC, it delivers a clear selective-prediction trade-off: the system maintains very high accuracy on answered claims while deferring 6 of 95 cases. These deferred cases are concentrated in lower-reliability evidence settings (5/6 at source level L4), supporting the view that abstention functions as a safety-oriented mechanism for handling epistemically weak evidence. Code is available at https://github.com/ cheshireyang/ECE.git
Automated fact-checking (AFC) systems powered by large language model (LLM) agents still face challenges in evidence retrieval and claim verification. Existing approaches improve their capabilities through computationally costly post-training or human-in-the-loop feedback that is difficult to scale. To transfer professional fact-checking expertise to LLM agents efficiently, we distill publicly available guidelines published by Agence France-Presse (AFP) Fact Check into a reusable, modular Professional Fact-Checking Skill (PFC-Skill). We then present SkillCheck, the first AFC system that gives LLM agents dynamic access to PFC-Skill to guide evidence retrieval and claim verification at inference time. Extensive experiments demonstrate that SkillCheck improves upon skill-free DEFAME baselines across multiple LLM backbones, with gains of 2.2--4.8 pp. in Accuracy and 1.5--6.3 pp. in Macro-F1 on the AVeriTeC benchmark. Notably, SkillCheck with GPT-4o-Mini exceeds DEFAME with GPT-4o by 3.4 pp. in Accuracy while cutting inference cost by roughly 61%. The performance gains also extend to open-web retrieval. Ablations show that incorporating the skill during evidence retrieval provides most of the gains, while incorporation into both stages yields the best overall performance. Case studies show that PFC-Skill enables more targeted search and closer scrutiny of evidence. To conclude, our work shows that LLM agents can "stand on the shoulders of giants" by drawing on the best practices of professional fact-checkers, substantially enhancing AFC performance without extra training or continuous human supervision.
Xinwen Chen, Haorui He, Yupeng Li
Department of Interactive Media, Hong Kong Baptist University · School of Computing and Data Science, The University of Hong Kong · AI and Social Good Lab, AI Media Centre, Hong Kong Baptist University
Large language models (LLMs) often hallucinate, yet most existing fact-checking methods treat factuality evaluation as a binary classification problem, offering limited interpretability and failing to capture fine-grained error types. In this paper, we introduce InFi-Check, a framework for interpretable and fine-grained fact-checking of LLM outputs. Specifically, we first propose a controlled data synthesis pipeline that generates high-quality data featuring explicit evidence, fine-grained error type labels, justifications, and corrections. Based on this, we further construct large-scale training data and a manually verified benchmark InFi-Check-FG for fine-grained fact-checking of LLM outputs. Building on these high-quality training data, we further propose InFi-Checker, which can jointly provide supporting evidence, classify fine-grained error types, and produce justifications along with corrections. Experiments show that InFi-Checker achieves state-of-the-art performance on InFi-Check-FG and strong generalization across various downstream tasks, significantly improving the utility and trustworthiness of factuality evaluation.
Yuzhuo Bai, Shuzheng Si, Kangyang Luo +5
Tsinghua University · DeepLang AI · Fudan University