When Imperfect Automated Fact-Checking Backfires: Social Learning in the Presence of Misinformation

Zixuan Liu; Ming Cao

cs.AINov 5, 2026

When Imperfect Automated Fact-Checking Backfires: Social Learning in the Presence of Misinformation

Authors: Zixuan Liu, Ming Cao

Organizations: Faculty of Science and Engineering, University of Groningen, 9747 AG Groningen, The Netherlands

Abstract

Automated fact-checking tools, cheap and fast but imperfect, are increasingly available on social platforms. We study when such tools can backfire in a sequential social learning model of misinformation. Agents observe an article's reliability and their predecessors' actions, which indicate whether they liked or disliked the article. They may also pay to fact-check it. We examine how welfare, defined as the probability of a correct cascade, changes when a free, error-prone automated tool supplements costly but perfectly accurate manual fact-checking. Two opposing effects arise. Broader use of fact-checking helps agents escape wrong cascades. At the same time, errors from the automated tool can mislead agents who would otherwise have verified the article perfectly, reducing the likelihood of correct cascades. We prove that adding the tool strictly lowers welfare for articles of intermediate reliability when truthful articles are not too informative and manual checking is sufficiently costly. Simulations show that the tool can help on either side of the intermediate range, and that it improves on manual-only checking on average only above a threshold accuracy.

Figures & tables

Fig. 1 : Expected payoff of an agent with posterior

\pi

without fact-checking,

{V}

, and with it,

\widehat{V}-K

. Agents with

\pi\in(\underline{\pi},\overline{\pi})

would follow the verdict, and those with

\pi\in(\pi_{-},\pi_{+})

, where

\widehat{V}-K>V

, also find it worth the cost and fact-check. Here

e_{1}=0.4

e_{2}=0.1

K=0.15

Panel of Figure 2 — Fig. 2 : Expected payoff of an agent with posterior $\pi$ if she does not fact-check, $V$ , and if she fact-checks using method ${\cal M}$ , $\widehat{V}_{\cal M}-K_{\cal M}$ , where $\widehat{V}_{\cal M}$ is given in ( 3 ). The highest curve (thick) is her expected payoff and the strip below shows her choice. $\pi^{\star}$ is where the curves of ${{\cal M}_{\mathrm{H}}}$ and ${{\cal M}_{\mathrm{A}}}$ cross. At the ends, where $\widehat{V}_{\mathrm{A}}=V$ (as ${{\cal M}_{\mathrm{A}}}$ is free), she is indifferent and does not check (see Section II-B ). This tie-break is immaterial, since no verdict of ${{\cal M}_{\mathrm{A}}}$ would change her action. $\pi^{\mathrm{H}}_{-}:=\pi_{-}({{\cal M}_{\mathrm{H}}})$ , $\pi^{\mathrm{A}}_{\pm}:=\pi_{\pm}({{\cal M}_{\mathrm{A}}})$ , ${K_{\mathrm{H}}}=0.3$ .

Panel of Figure 3 — Fig. 3 : Welfare change in regime (iii) : ${K_{\mathrm{H}}}=0.3$ , ${e_{1\mathrm{A}}}=0.3$ , ${e_{2\mathrm{A}}}=0.1$ . Lines: the thresholds ( 11 )–( 14 ) bounding the regions where Proposition 2 certifies the sign ( $+$ , $-$ ), and $0$ where a cascade starts at $t=1$ under both method sets. Bold tick marks $\bar{p}$ .

Panel of Figure 4 — Fig. 4 : Welfare change in regime (ii) : ${K_{\mathrm{H}}}=0.3$ , ${e_{1\mathrm{A}}}=0.45$ , ${e_{2\mathrm{A}}}=0.15$ . Lines: the thresholds ( 11 )–( 14 ) and $r^{\star}$ bounding the regions where Proposition 3 certifies the sign ( $+$ , $-$ strict, $\leq 0$ weak, $0$ ). Bold ticks mark $p_{\kappa}$ and $\bar{p}$ . The horizontal dotted line in (b) marks $p=p_{\kappa}$ .

Fig. 5 : Welfare

\overline{W}

averaged over

r\sim\mathrm{Unif}(0,1)

against the balanced accuracy

\alpha=1-\frac{1}{2}({e_{1\mathrm{A}}}+{e_{2\mathrm{A}}})

{{\cal M}_{\mathrm{A}}}

along

{e_{1\mathrm{A}}}=3{e_{2\mathrm{A}}}

{K_{\mathrm{H}}}=0.3

p=0.55<\bar{p}

are fixed. Estimates are based on

20000

articles, with

\pm 1

standard deviation shown in bands. Vertical lines:

\tau_{\mathrm{use}}

, below which no agent uses

{{\cal M}_{\mathrm{A}}}

;

\tau_{\mathrm{all}}

, above which every agent who checks does;

\tau_{0}

, above which

\{{{\cal M}_{\mathrm{H}}},{{\cal M}_{\mathrm{A}}}\}

has higher welfare than

\{{{\cal M}_{\mathrm{H}}}\}

. Top strip: regimes (i) – (iii) . Diamond, square: the

{{\cal M}_{\mathrm{A}}}

of Sections IV-A , IV-B .

Automated Fact-Checking

Automated fact-checking detects checkable claims, retrieves evidence, and assesses whether that evidence supports or refutes them.

See all 142 papers

No Hindsight for LLM Fact-Checkers: Measuring Leakage Channels in Misinformation Detection
Oct 4, 2026Kuan-Hua Wu Lu, Yohanes Andre SetiawanAutomated Fact-Checking Data Contamination in Language Models
Georgia Institute of Technology Atlanta, Georgia, USA
Gaming Consensus: Coordinated Manipulation in Crowdsourced Fact-Checking
Jul 2, 2026Nikil Roashan Selvam, Jay Baxter, Sophie Hilgard +4Social Media Analysis Algorithmic Auditing
Stanford University, Stanford, CA, USA · X Community Notes, xAI, Palo Alto, USA
Taxonomy of Risks on Automated Fact-Checking Systems Considering its Propagation
Jun 24, 2026Jun Yajima, Tatsuya Oka, Takao OkuboAI Risk Management Automated Fact-Checking
1Fujitsu Limited · Institute of Information Security

Oct 4, 2026J/K move · Enter open · S save

No Hindsight for LLM Fact-Checkers: Measuring Leakage Channels in Misinformation Detection

Kuan-Hua Wu Lu, Yohanes Andre Setiawan

Georgia Institute of Technology Atlanta, Georgia, USA

Automated Fact-Checking Data Contamination in Language Models LLM Auditing

Open paper AskPDF arXiv

As automated fact-checking scales on social media, large language model (LLM) verdict scores can look stronger than warranted. One reason is that evaluations mix in information that was not knowable at claim time. Two channels are easy to conflate: outcomes memorized in pre-training and retrieved evidence published after the claim. Yet standard benchmarks rarely separate the two. In this study we measure both channels on AVeriTeC and QuanTemp++ by reconstructing point-in-time evidence conditions and probing for outcome information encoded in model representations. We find substantial evidence of parametric leakage, that can be hidden by the aggregate accuracy, while a simple representation bottleneck reduces this future leakage more efficiently than a mutual-information-based training penalty. We also find that allowing post-claim evidence inflates zero-shot accuracy by 6.3 points in AVeriTeC while the effect is negligible in QuanTemp++, where retrieval provides little post-claim evidence. These results show that misinformation benchmarks can overstate fact-checking performance when they do not account for what information was actually available at claim time.

Abstract

Automated Fact-Checking

Explore similar work

No Hindsight for LLM Fact-Checkers: Measuring Leakage Channels in Misinformation Detection

Gaming Consensus: Coordinated Manipulation in Crowdsourced Fact-Checking

Taxonomy of Risks on Automated Fact-Checking Systems Considering its Propagation