cs.AIOct 19, 2026

Standing on the Shoulders of Giants: Enhancing LLM Agents for Automated Fact-Checking with Human Expert Skills

Authors: Xinwen Chen, Haorui He, Yupeng Li

Organizations: Department of Interactive Media, Hong Kong Baptist University · School of Computing and Data Science, The University of Hong Kong · AI and Social Good Lab, AI Media Centre, Hong Kong Baptist University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Automated fact-checking (AFC) systems powered by large language model (LLM) agents still face challenges in evidence retrieval and claim verification. Existing approaches improve their capabilities through computationally costly post-training or human-in-the-loop feedback that is difficult to scale. To transfer professional fact-checking expertise to LLM agents efficiently, we distill publicly available guidelines published by Agence France-Presse (AFP) Fact Check into a reusable, modular Professional Fact-Checking Skill (PFC-Skill). We then present SkillCheck, the first AFC system that gives LLM agents dynamic access to PFC-Skill to guide evidence retrieval and claim verification at inference time. Extensive experiments demonstrate that SkillCheck improves upon skill-free DEFAME baselines across multiple LLM backbones, with gains of 2.2--4.8 pp. in Accuracy and 1.5--6.3 pp. in Macro-F1 on the AVeriTeC benchmark. Notably, SkillCheck with GPT-4o-Mini exceeds DEFAME with GPT-4o by 3.4 pp. in Accuracy while cutting inference cost by roughly 61%. The performance gains also extend to open-web retrieval. Ablations show that incorporating the skill during evidence retrieval provides most of the gains, while incorporation into both stages yields the best overall performance. Case studies show that PFC-Skill enables more targeted search and closer scrutiny of evidence. To conclude, our work shows that LLM agents can "stand on the shoulders of giants" by drawing on the best practices of professional fact-checkers, substantially enhancing AFC performance without extra training or continuous human supervision.

Figures & tables

Figure 1. Overview of LLM agents enhanced with human expert skills for automated fact-checking (AFC).
Figure 2. PFC-Skill is incorporated into the evidence retrieval and claim verification stages of an AFC system.
Figure 3. Structure of PFC-Skill .
MethodLLMAccuracyMacro-F1
Score (%)95% CIScore (%)95% CI
Post-training methods
DebateCVLlama-3.1-8B72.8---
HerOLlama-3.1-8B70.2---
Training-free methods
Zero-shotGPT-4o62.1---
Table 1. AFC results on AVeriTeC. Green superscripts indicate absolute improvements, in pp., over the corresponding skill-free DEFAME setting. For DEFAME and SkillCheck , uncertainty is reported as 95% bootstrap confidence intervals on matched claims. ∗\ast indicates statistically significant improvements ( p<0.05p<0.05 , paired bootstrap test).
BackboneSystemInputOutputTotal CostLatency
Tokens (K)Cost ($)Tokens (K)Cost ($)($)(s)
GPT-4oDEFAME2.970.00740.370.00370.011143.11
SkillCheck26.410.06600.430.00430.070351.33
GPT-4o-MiniDEFAME2.890.00040.390.00020.000743.34
SkillCheck26.750.00400.550.00030.004353.41
Table 2. Average token usage, LLM inference cost, and latency per claim. Token counts are reported in thousands (K). Input, output, and total cost are rounded independently.
Figure 4. Comparison of Accuracy and inference cost per claim between SkillCheck and w/o Skill (i.e., DEFAME).
Figure 5. AFC results on ClaimReview2024+ date{}_{\text{date}} .
Figure 6. Ablation results. (a) compares skill-free DEFAME ( w/o Skill ), raw AFP guidelines ( w/o Distillation ), stage-specific skills ( w/ Stage-Specific Skill ), and the unified PFC-Skill ( SkillCheck ). (b) compares skill-free DEFAME ( w/o ER & CV ), skill incorporation during ER alone ( w/o CV ), and incorporation into both ER and CV ( SkillCheck ).
 DEFAMESkillCheck
 Fact-Checking RecordFact-Checking RecordSelected Skill
Claim 1, Refuted : US President Trump made vaccines voluntary, not mandatory, in the US.
Query1. “US President Trump vaccines voluntary mandatory September 2020”1. “US President Trump vaccines voluntary not mandatory September 2020”; 2. “Trump statement vaccines voluntary mandatory September 2020”; 3. “Trump executive order vaccines voluntary mandatory September 2020”; 4. “ US federal vaccine policy September 2020”. 
EvidenceTrump’s order encouraged vaccination without specifying voluntary or mandatory status . Another source described a COVID-19 vaccine mandate for service members .Trump’s September 20, 2020 order encouraged vaccination. A Department of Defense document described a military mandate and noted that public health responsibilities primarily lie with state and local governments . 
JudgmentMandates applied to specific groups , while Trump focused on vaccine development and distribution . These are conflicting: ‘ Conflicting/Cherry-picking ’ . ✗Vaccination mandates primarily fall under state and local authority . No federal mandate or declaration of voluntary vaccination for the general public was identified: ‘ Refuted ’ . ✓ 
Claim 2, Refuted : Amy Coney Barrett, associate justice of the Supreme Court of the United States, said during her Supreme Court confirmation hearing that she was not using notes because she has seven kids and is used to stupid questions.
Table 3. Case studies illustrating how the skill alters fact-checking behaviors. Selected search queries are shown; evidence and judgments are abridged. Effective behaviors, useful skills, and correct judgments are highlighted in green , while problematic behaviors and incorrect judgments are marked in purple . Positive cases are highlighted in blue , while failures are in yellow .
Reference FileReference Content
Claim_decomposition.mdBreaking complex claims into atomic, verifiable sub-questions; Claim type identification; Investigation prioritization.
Search_query_strategies.mdConcrete search query construction techniques by claim type; Iterative refinement strategies; Query anti-patterns.
Source_credibility_evaluation.md4-tier source reliability hierarchy; Source evaluation checklist; Rules for handling denials, expert citations, and prior fact-checks.
Cross_verification_methods.mdMulti-source verification strategies, including source independence criteria, consistency checks, and handling conflicting evidence.
Evidence_sufficiency_evaluation.mdCriteria for evidence sufficiency; Evidence weight taxonomy; Handling absence of evidence; When to continue or stop retrieval.
Contextual_analysis.mdDetecting out-of-context usage, cherry-picking, misleading framing, and omitted context; Distinguishing misleading claims from incomplete context.
Table 4. Detailed descriptions of the reference files in PFC-Skill used in SkillCheck .
More in this topic

Automated Fact-Checking

Automated fact-checking detects checkable claims, retrieves evidence, and assesses whether that evidence supports or refutes them.

See all 136 papers

Explore similar work

CardsList
Nov 5, 2025

Hybrid Fact-Checking that Integrates Knowledge Graphs, Large Language Models, and Search-Based Retrieval Agents Improves Interpretable Claim Verification

Large language models (LLMs) excel in generating fluent utterances but can lack reliable grounding in verified information. At the same time, knowledge-graph-based fact-checkers deliver precise and interpretable evidence, yet suffer from limited coverage or latency. By integrating LLMs with knowledge graphs and real-time search agents, we introduce a hybrid fact-checking approach that leverages the individual strengths of each component. Our system comprises three autonomous steps: 1) a Knowledge Graph (KG) Retrieval for rapid one-hop lookups in DBpedia, 2) an LM-based classification guided by a task-specific labeling prompt, producing outputs with internal rule-based logic, and 3) a Web Search Agent invoked only when KG coverage is insufficient. Our pipeline achieves an F1 score of 0.93 on the FEVER benchmark on the Supported/Refuted split without task-specific fine-tuning. To address Not enough information cases, we conduct a targeted reannotation study showing that our approach frequently uncovers valid evidence for claims originally labeled as Not Enough Information (NEI), as confirmed by both expert annotators and LLM reviewers. With this paper, we present a modular, opensource fact-checking pipeline with fallback strategies and generalization across datasets.

Shaghayegh Kolli, Richard Rosenbaum, Timo Cavelius +3

Technical University Munich

LLM GroundingAutomated Fact-Checking
AskPDF
Mar 5, 2026

Leveraging LLM Parametric Knowledge for Fact Checking without Retrieval

Trustworthiness is a core research challenge for agentic AI systems built on Large Language Models (LLMs). To enhance trust, natural language claims from diverse sources, including human-written text, web content, and model outputs, are commonly checked for factuality by retrieving external knowledge and using an LLM to verify the faithfulness of claims to the retrieved evidence. As a result, such methods are constrained by retrieval errors and external data availability, while leaving the models intrinsic fact-verification capabilities largely unused. We propose the task of fact-checking without retrieval, focusing on the verification of arbitrary natural language claims, independent of their source. To study this setting, we introduce a comprehensive evaluation framework focused on generalization, testing robustness to (i) long-tail knowledge, (ii) variation in claim sources, (iii) multilinguality, and (iv) long-form generation. Across 9 datasets, 18 methods and 3 models, our experiments indicate that logit-based approaches often underperform compared to those that leverage internal model representations. Building on this finding, we introduce INTRA, a method that exploits interactions between internal representations and achieves state-of-the-art performance with strong generalization. More broadly, our work establishes fact-checking without retrieval as a promising research direction that can complement retrieval-based frameworks, improve scalability, and enable the use of such systems as reward signals during training or as components integrated into the generation process.

Artem Vazhentsev, Maria Marina, Daniil Moskovskiy +8

MBZUAI · S-NLP Group · AXXX +3

Automated Fact-CheckingFactual Knowledge in Language Models
AskPDF
Sep 8, 2026

Evaluating and Improving Evidence-Grounded Fact-Checking in LLMs via Multi-Round Evidence Ablation

Automatic fact-checking systems assess the veracity of claims given evidence from relevant documents. Large Language Models (LLMs) have demonstrated strong performance in fact-checking due to their general reasoning capabilities. However, it remains unclear whether they faithfully make use of the evidence provided to reach veracity judgments or rely on parametric knowledge. To investigate this, we introduce Fact-Ablated Evaluation (FAE), a new evaluation framework that iteratively ablates the cited evidence to assess whether LLMs revise their predictions accordingly. Our empirical results show that current off-the-shelf LLMs as fact-checking systems rely more on their parametric knowledge than on the evidence provided. To bridge this gap between prediction accuracy and evidence grounding, we propose REAL (Rigorous Evidence Ablation Learning), a training framework that promotes evidence-dependent verification through counterfactual evidence supervision for the LLM-as-verifier models. Experiments on four fact-checking datasets across different domains demonstrate that models trained with REAL obtain superior evidence-dependent capabilities compared to standard fine-tuned models. Our findings highlight that strong fact-checking performance can still coexist with weak evidence dependency, while REAL encourages veracity predictions to remain more closely tied to the availability of supporting evidence.

Xingyu Deng, Mingzi Cao, Nikolaos Aletras +2

University of Sheffield · Sheffield, UK

LLM EvaluationLLM GroundingAutomated Fact-Checking
AskPDF