Standing on the Shoulders of Giants: Enhancing LLM Agents for Automated Fact-Checking with Human Expert Skills

Xinwen Chen; Haorui He; Yupeng Li

cs.AIOct 19, 2026

Standing on the Shoulders of Giants: Enhancing LLM Agents for Automated Fact-Checking with Human Expert Skills

Authors: Xinwen Chen, Haorui He, Yupeng Li

Organizations: Department of Interactive Media, Hong Kong Baptist University · School of Computing and Data Science, The University of Hong Kong · AI and Social Good Lab, AI Media Centre, Hong Kong Baptist University

Abstract

Automated fact-checking (AFC) systems powered by large language model (LLM) agents still face challenges in evidence retrieval and claim verification. Existing approaches improve their capabilities through computationally costly post-training or human-in-the-loop feedback that is difficult to scale. To transfer professional fact-checking expertise to LLM agents efficiently, we distill publicly available guidelines published by Agence France-Presse (AFP) Fact Check into a reusable, modular Professional Fact-Checking Skill (PFC-Skill). We then present SkillCheck, the first AFC system that gives LLM agents dynamic access to PFC-Skill to guide evidence retrieval and claim verification at inference time. Extensive experiments demonstrate that SkillCheck improves upon skill-free DEFAME baselines across multiple LLM backbones, with gains of 2.2--4.8 pp. in Accuracy and 1.5--6.3 pp. in Macro-F1 on the AVeriTeC benchmark. Notably, SkillCheck with GPT-4o-Mini exceeds DEFAME with GPT-4o by 3.4 pp. in Accuracy while cutting inference cost by roughly 61%. The performance gains also extend to open-web retrieval. Ablations show that incorporating the skill during evidence retrieval provides most of the gains, while incorporation into both stages yields the best overall performance. Case studies show that PFC-Skill enables more targeted search and closer scrutiny of evidence. To conclude, our work shows that LLM agents can "stand on the shoulders of giants" by drawing on the best practices of professional fact-checkers, substantially enhancing AFC performance without extra training or continuous human supervision.

Figures & tables

Figure 1. Overview of LLM agents enhanced with human expert skills for automated fact-checking (AFC).

Figure 2. PFC-Skill is incorporated into the evidence retrieval and claim verification stages of an AFC system.

Method	LLM	Accuracy		Macro-F1
Method	LLM	Score (%)	95% CI	Score (%)	95% CI
Post-training methods
DebateCV	Llama-3.1-8B	72.8	-	-	-
HerO	Llama-3.1-8B	70.2	-	-	-
Training-free methods
Zero-shot	GPT-4o	62.1	-	-	-

Table 1. AFC results on AVeriTeC. Green superscripts indicate absolute improvements, in pp., over the corresponding skill-free DEFAME setting. For DEFAME and SkillCheck , uncertainty is reported as 95% bootstrap confidence intervals on matched claims.

\ast

indicates statistically significant improvements (

p<0.05

, paired bootstrap test).

Backbone	System	Input		Output		Total Cost	Latency
Backbone	System	Tokens (K)	Cost ($)	Tokens (K)	Cost ($)	($)	(s)
GPT-4o	DEFAME	2.97	0.0074	0.37	0.0037	0.0111	43.11
GPT-4o	SkillCheck	26.41	0.0660	0.43	0.0043	0.0703	51.33
GPT-4o-Mini	DEFAME	2.89	0.0004	0.39	0.0002	0.0007	43.34
GPT-4o-Mini	SkillCheck	26.75	0.0040	0.55	0.0003	0.0043	53.41

Table 2. Average token usage, LLM inference cost, and latency per claim. Token counts are reported in thousands (K). Input, output, and total cost are rounded independently.

Figure 4. Comparison of Accuracy and inference cost per claim between SkillCheck and w/o Skill (i.e., DEFAME).

Panel of Figure 5 — Figure 5. AFC results on ClaimReview2024+ ${}_{\text{date}}$ .

Panel of Figure 6 — Figure 6. Ablation results. (a) compares skill-free DEFAME ( w/o Skill ), raw AFP guidelines ( w/o Distillation ), stage-specific skills ( w/ Stage-Specific Skill ), and the unified PFC-Skill ( SkillCheck ). (b) compares skill-free DEFAME ( w/o ER & CV ), skill incorporation during ER alone ( w/o CV ), and incorporation into both ER and CV ( SkillCheck ).

	DEFAME	SkillCheck
	Fact-Checking Record	Fact-Checking Record	Selected Skill
Claim 1, Refuted : US President Trump made vaccines voluntary, not mandatory, in the US.
Query	1. “US President Trump vaccines voluntary mandatory September 2020”	1. “US President Trump vaccines voluntary not mandatory September 2020”; 2. “Trump statement vaccines voluntary mandatory September 2020”; 3. “Trump executive order vaccines voluntary mandatory September 2020”; 4. “ US federal vaccine policy September 2020”.
Evidence	Trump’s order encouraged vaccination without specifying voluntary or mandatory status . Another source described a COVID-19 vaccine mandate for service members .	Trump’s September 20, 2020 order encouraged vaccination. A Department of Defense document described a military mandate and noted that public health responsibilities primarily lie with state and local governments .
Judgment	Mandates applied to specific groups , while Trump focused on vaccine development and distribution . These are conflicting: ‘ Conflicting/Cherry-picking ’ . ✗	Vaccination mandates primarily fall under state and local authority . No federal mandate or declaration of voluntary vaccination for the general public was identified: ‘ Refuted ’ . ✓
Claim 2, Refuted : Amy Coney Barrett, associate justice of the Supreme Court of the United States, said during her Supreme Court confirmation hearing that she was not using notes because she has seven kids and is used to stupid questions.

Table 3. Case studies illustrating how the skill alters fact-checking behaviors. Selected search queries are shown; evidence and judgments are abridged. Effective behaviors, useful skills, and correct judgments are highlighted in green , while problematic behaviors and incorrect judgments are marked in purple . Positive cases are highlighted in blue , while failures are in yellow .

Reference File	Reference Content
Claim_decomposition.md	Breaking complex claims into atomic, verifiable sub-questions; Claim type identification; Investigation prioritization.
Search_query_strategies.md	Concrete search query construction techniques by claim type; Iterative refinement strategies; Query anti-patterns.
Source_credibility_evaluation.md	4-tier source reliability hierarchy; Source evaluation checklist; Rules for handling denials, expert citations, and prior fact-checks.
Cross_verification_methods.md	Multi-source verification strategies, including source independence criteria, consistency checks, and handling conflicting evidence.
Evidence_sufficiency_evaluation.md	Criteria for evidence sufficiency; Evidence weight taxonomy; Handling absence of evidence; When to continue or stop retrieval.
Contextual_analysis.md	Detecting out-of-context usage, cherry-picking, misleading framing, and omitted context; Distinguishing misleading claims from incomplete context.

Table 4. Detailed descriptions of the reference files in PFC-Skill used in SkillCheck .

Automated Fact-Checking

Automated fact-checking detects checkable claims, retrieves evidence, and assesses whether that evidence supports or refutes them.

See all 136 papers

Hybrid Fact-Checking that Integrates Knowledge Graphs, Large Language Models, and Search-Based Retrieval Agents Improves Interpretable Claim Verification
Nov 5, 2025Shaghayegh Kolli, Richard Rosenbaum, Timo Cavelius +3LLM Grounding Automated Fact-Checking
Technical University Munich
Leveraging LLM Parametric Knowledge for Fact Checking without Retrieval
Mar 5, 2026Artem Vazhentsev, Maria Marina, Daniil Moskovskiy +8Automated Fact-Checking Factual Knowledge in Language Models
MBZUAI · S-NLP Group · AXXX +3
Evaluating and Improving Evidence-Grounded Fact-Checking in LLMs via Multi-Round Evidence Ablation
Sep 8, 2026Xingyu Deng, Mingzi Cao, Nikolaos Aletras +2LLM Evaluation LLM Grounding
University of Sheffield · Sheffield, UK

Nov 5, 2025J/K move · Enter open · S save

Hybrid Fact-Checking that Integrates Knowledge Graphs, Large Language Models, and Search-Based Retrieval Agents Improves Interpretable Claim Verification

Shaghayegh Kolli, Richard Rosenbaum, Timo Cavelius, Lasse Strothe+2

Technical University Munich

LLM Grounding Automated Fact-Checking

Open paper AskPDF arXiv

Large language models (LLMs) excel in generating fluent utterances but can lack reliable grounding in verified information. At the same time, knowledge-graph-based fact-checkers deliver precise and interpretable evidence, yet suffer from limited coverage or latency. By integrating LLMs with knowledge graphs and real-time search agents, we introduce a hybrid fact-checking approach that leverages the individual strengths of each component. Our system comprises three autonomous steps: 1) a Knowledge Graph (KG) Retrieval for rapid one-hop lookups in DBpedia, 2) an LM-based classification guided by a task-specific labeling prompt, producing outputs with internal rule-based logic, and 3) a Web Search Agent invoked only when KG coverage is insufficient. Our pipeline achieves an F1 score of 0.93 on the FEVER benchmark on the Supported/Refuted split without task-specific fine-tuning. To address Not enough information cases, we conduct a targeted reannotation study showing that our approach frequently uncovers valid evidence for claims originally labeled as Not Enough Information (NEI), as confirmed by both expert annotators and LLM reviewers. With this paper, we present a modular, opensource fact-checking pipeline with fallback strategies and generalization across datasets.

Abstract

Automated Fact-Checking

Explore similar work

Hybrid Fact-Checking that Integrates Knowledge Graphs, Large Language Models, and Search-Based Retrieval Agents Improves Interpretable Claim Verification

Leveraging LLM Parametric Knowledge for Fact Checking without Retrieval

Evaluating and Improving Evidence-Grounded Fact-Checking in LLMs via Multi-Round Evidence Ablation