Automated fact-checking tools, cheap and fast but imperfect, are increasingly available on social platforms. We study when such tools can backfire in a sequential social learning model of misinformation. Agents observe an article's reliability and their predecessors' actions, which indicate whether they liked or disliked the article. They may also pay to fact-check it. We examine how welfare, defined as the probability of a correct cascade, changes when a free, error-prone automated tool supplements costly but perfectly accurate manual fact-checking. Two opposing effects arise. Broader use of fact-checking helps agents escape wrong cascades. At the same time, errors from the automated tool can mislead agents who would otherwise have verified the article perfectly, reducing the likelihood of correct cascades. We prove that adding the tool strictly lowers welfare for articles of intermediate reliability when truthful articles are not too informative and manual checking is sufficiently costly. Simulations show that the tool can help on either side of the intermediate range, and that it improves on manual-only checking on average only above a threshold accuracy.
Figures & tables
Fig. 1 : Expected payoff of an agent with posterior π without fact-checking, V , and with it, V−K . Agents with π∈(π,π) would follow the verdict, and those with π∈(π−,π+) , where V−K>V , also find it worth the cost and fact-check. Here e1=0.4 , e2=0.1 , K=0.15 .
Fig. 2 : Expected payoff of an agent with posterior π if she does not fact-check, V , and if she fact-checks using method M , VM−KM , where VM is given in ( 3 ). The highest curve (thick) is her expected payoff and the strip below shows her choice. π⋆ is where the curves of MH and MA cross. At the ends, where VA=V (as MA is free), she is indifferent and does not check (see Section II-B ). This tie-break is immaterial, since no verdict of MA would change her action. π−H:=π−(MH) , π±A:=π±(MA) , KH=0.3 .
Fig. 3 : Welfare change in regime (iii) : KH=0.3 , e1A=0.3 , e2A=0.1 . Lines: the thresholds ( 11 )–( 14 ) bounding the regions where Proposition 2 certifies the sign ( + , − ), and 0 where a cascade starts at t=1 under both method sets. Bold tick marks pˉ .
Fig. 4 : Welfare change in regime (ii) : KH=0.3 , e1A=0.45 , e2A=0.15 . Lines: the thresholds ( 11 )–( 14 ) and r⋆ bounding the regions where Proposition 3 certifies the sign ( + , − strict, ≤0 weak, 0 ). Bold ticks mark pκ and pˉ . The horizontal dotted line in (b) marks p=pκ .
Fig. 5 : Welfare W averaged over r∼Unif(0,1) against the balanced accuracy α=1−21(e1A+e2A) of MA along e1A=3e2A . KH=0.3 , p=0.55<pˉ are fixed. Estimates are based on 20000 articles, with ±1 standard deviation shown in bands. Vertical lines: τuse , below which no agent uses MA ; τall , above which every agent who checks does; τ0 , above which {MH,MA} has higher welfare than {MH} . Top strip: regimes (i) – (iii) . Diamond, square: the MA of Sections IV-A , IV-B .
As automated fact-checking scales on social media, large language model (LLM) verdict scores can look stronger than warranted. One reason is that evaluations mix in information that was not knowable at claim time. Two channels are easy to conflate: outcomes memorized in pre-training and retrieved evidence published after the claim. Yet standard benchmarks rarely separate the two. In this study we measure both channels on AVeriTeC and QuanTemp++ by reconstructing point-in-time evidence conditions and probing for outcome information encoded in model representations. We find substantial evidence of parametric leakage, that can be hidden by the aggregate accuracy, while a simple representation bottleneck reduces this future leakage more efficiently than a mutual-information-based training penalty. We also find that allowing post-claim evidence inflates zero-shot accuracy by 6.3 points in AVeriTeC while the effect is negligible in QuanTemp++, where retrieval provides little post-claim evidence. These results show that misinformation benchmarks can overstate fact-checking performance when they do not account for what information was actually available at claim time.
Kuan-Hua Wu Lu, Yohanes Andre Setiawan
Georgia Institute of Technology Atlanta, Georgia, USA
Crowdsourced fact-checking systems have been adopted by major social media companies such as X, Meta, TikTok and Google with the aim of combating misleading information at scale without relying on centralized editorial control. These systems have been developed around a common underlying concept: a bridging mechanism that identifies notes flagging misleading information when they receive support from people with different perspectives rather than simple majority support. To our knowledge the only publicly disclosed bridging algorithms deployed for fact-checking are based on matrix factorization, as deployed by both X and Meta, augmented with additional components addressing abuse, targeted manipulation, and contributor brigades. This work examines the core matrix factorization portion of these systems, presenting theoretical and empirical evaluations of the degree to which coordinated users could vote strategically by leveraging the latent representations to fabricate the appearance of synthetic consensus within the bridging mechanism. Using historic production data, we find that up to 10.7% of lower quality notes could be manipulated above consensus thresholds using less than 10 ratings. We complement these findings with a theoretical analysis, revealing counterintuitively that rating a note as "Not Helpful" can increase its helpfulness score, as well as a cost model quantifying manipulation effort. We have developed and deployed mitigations within X's Community Notes algorithm to address synthetic consensus.
Nikil Roashan Selvam, Jay Baxter, Sophie Hilgard +4
Stanford University, Stanford, CA, USA · X Community Notes, xAI, Palo Alto, USA
In recent years, the posting of fake news including disinformation and misinformation on social networking services (SNS) has become a social problem. To combat this fake news, fact-checking that is the process of assessing the veracity of posts on SNS has become increasingly important. While fact-checking is currently performed by fact-checking organizations, it is difficult to fact-check all posts on SNS. Therefore, the use of automated fact-checking systems is effective. Recent automated fact-checking systems utilize artificial intelligence and large language models, so there are risks of incorrect judgments and posting incorrect results on social media which can lead to the spread of misinformation or to engage in defamation. In this paper, as a first step toward enabling the safe use of automated fact-checking systems, we categorize the specific risks on automated fact-checking systems. In this categorizing, we consider a three-stage risk propagation: risk factors, hazardous situations, and harm. Our analysis revealed that 32 specific risks exist in automated fact-checking systems. In this paper, we utilize the categorized risks as analytical cues (guide words) to present the risk assessment of the automated fact-checking system DEFAME. This assessment result indicates that risks that cannot be derived using STRIDE, a conventional IT security risk assessment method can be derived using our guide words.
Jun Yajima, Tatsuya Oka, Takao Okubo
1Fujitsu Limited · Institute of Information Security