cs.CLFeb 21, 2026

Contradiction to Consensus: Dual Perspective, Multi Source Retrieval Based Claim Verification with Source Level Disagreement using LLM

Authors: Md Badsha Biswas, Ozlem Uzuner

Organizations: George Mason University, Fairfax, VA, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The spread of misinformation across digital platforms can pose significant societal risks. Claim verification, a.k.a. fact-checking, systems can help identify potential misinformation. However, their efficacy is limited by the knowledge sources that they rely on. Most automated claim verification systems depend on a single knowledge source and utilize the supporting evidence from that source; they ignore the disagreement of their source with others. This limits their knowledge coverage and transparency. To address these limitations, we present a novel system for open-domain claim verification (ODCV) that leverages large language models (LLMs), multi-perspective evidence retrieval, and cross-source disagreement analysis. Our approach introduces a novel retrieval strategy that collects evidence for both the original and the negated forms of a claim, enabling the system to capture supporting and contradicting information from diverse sources: Wikipedia, PubMed, and Google. These evidence sets are filtered, deduplicated, and aggregated across sources to form a unified and enriched knowledge base that better reflects the complexity of real-world information. This aggregated evidence is then used for claim verification using LLMs. We further enhance interpretability by analyzing model confidence scores to quantify and visualize inter-source disagreement. Through extensive evaluation on four benchmark datasets with five LLMs, we show that knowledge aggregation not only improves claim verification but also reveals differences in source-specific reasoning. Our findings underscore the importance of embracing diversity, contradiction, and aggregation in evidence for building reliable and transparent claim verification systems

Figures & tables

Figure 1: Workflow of the proposed system. A claim and its negation are used to retrieve sentence-level evidence from Wikipedia, PubMed, and Google; candidate sentences are ranked and deduplicated, keeping the top- pp per source. The resulting candidate sentences from each source are aggregated into a cumulative evidence set EiE_{i} , which a zero-shot LLM uses to verify claims; log-probabilities for each source are reported and visualized to convey agreement and uncertainty.
DatasetModelKnowledge SourceOriginal Claim Original + Negated Claim
APRF1 APRF1
SCIFACTLlama 70BWikipedia0.4300.4680.4070.415 0.2300.3760.3350.203
Pubmed0.5970.5880.5970.584 0.6170.6090.6260.605
Google0.5500.5430.5580.530 0.6070.6150.6200.573
Llama 405BWikipedia0.4470.4890.4200.425 0.4430.4750.4210.427
Pubmed0.5930.5830.5920.576 0.6170.6060.6220.599
Table 1: Original-only vs. original+negated evidence in zero-shot evaluation. We report Accuracy (A), Precision (P), Recall (R), and macro- F1F_{1} across datasets, models, and knowledge sources (Wikipedia, PubMed, Google).
DatasetModelKnowledge SourceOriginal Claim Original + Negated Claim
APRF1 APRF1
LiarMistralWikipedia0.2100.3340.1840.096 0.2110.3230.1850.097
Pubmed0.2140.1340.1840.105 0.2060.1160.1760.095
Google0.3750.6080.3610.356 0.3810.6310.3650.362
PubHealthLlama 70BWikipedia0.2450.3330.3130.197 0.2410.3250.3180.195
Pubmed0.1990.3130.3150.163 0.2070.3410.3310.169
Table 2: Extension of Table 1 : Comparison of Original Claims vs. Original + Negated Claims
Dataset Model Knowledge Source
  Wikipedia Pubmed Google Merged(W+P+G)
SciFact Llama 70B 0.430 0.597 0.550 0.610
 Llama 405B 0.447 0.593 0.580 0.597
 Phi-4 0.410 0.583 0.590 0.583
 Qwen 2.5 0.437 0.583 0.587 0.607
 Mistral 0.393 0.573 0.583 0.617
Table 3: Comparative Accuracy of Individual vs. Aggregated Knowledge Sources (Wikipedia+PubMed+Google)
Figure 2: Confidence distribution (KDE) across knowledge sources for the Averitec dataset, illustrating variation in model certainty and inter-source disagreement for claim verification
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

DatasetSourceDomainLabelClaims
SCIFACTScienceScientificSupported1400
Refuted
Not Enough Info
AveritecFactcheckMixedSupported4568
Refuted
Conflicting evidence/cherrypicking
Appendix
Table 4: Overview of Benchmark Datasets
Figure 3: Confidence distribution (KDE) across different knowledge sources for the LIAR dataset, illustrating variation in model certainty and inter-source disagreement during claim verification
Figure 4: Confidence distribution (KDE) across different knowledge sources for the Pubhealth dataset, illustrating variation in model certainty and inter-source disagreement during claim verification
Figure 5: Confidence distribution (KDE) across different knowledge sources for the SCIFact dataset, illustrating variation in model certainty and inter-source disagreement during claim verification
Figure 6: Prompt template for veracity prediction
Figure 7: Violin Plot for Pubhealth
More in this topic

Claim Verification

Claim verification assesses whether statements are supported, contradicted, or unresolved by relevant evidence.

See all 59 papers

Explore similar work

CardsList
May 28, 2026

CommunityFact: A Dynamic, Multilingual, Multi-domain Benchmark for Misinformation Detection in the Wild

Misinformation verification increasingly occurs in public, fast-moving, and multilingual online settings, where static benchmarks provide an incomplete measure of model reliability. We introduce CommunityFact, a refreshable benchmark for misinformation detection in the wild, with three major goals: coverage, granularity, and redistributability. This release contains 15,992 standalone claims across five languages and two domains. We evaluate ten LLMs under varying inference-time capabilities, including thinking and web-search. Our results show that closed-input verification remains challenging, web access yields the largest gains, and web-enabled LLMs' source-selection policies are systematically misaligned with the sources human Community Notes raters converge on -- a gap that closes through model-specific mechanisms of retrieval expansion or pruning. We further find substantial variation across language-domain slices and across the evidence ecosystems used by web-enabled systems. Beyond evaluation, CommunityFact positions Community Notes as a training signal for claim-conditioned source suggesters that could improve factual verification on novel claims.

Sahajpreet Singh, Insyirah Mujtahid, Min-Yen Kan +1

National University of Singapore

Multilingual Language Model EvaluationLLM EvaluationFake News DetectionAutomated Fact-CheckingLLM Reliability
AskPDF
Nov 5, 2025

Hybrid Fact-Checking that Integrates Knowledge Graphs, Large Language Models, and Search-Based Retrieval Agents Improves Interpretable Claim Verification

Large language models (LLMs) excel in generating fluent utterances but can lack reliable grounding in verified information. At the same time, knowledge-graph-based fact-checkers deliver precise and interpretable evidence, yet suffer from limited coverage or latency. By integrating LLMs with knowledge graphs and real-time search agents, we introduce a hybrid fact-checking approach that leverages the individual strengths of each component. Our system comprises three autonomous steps: 1) a Knowledge Graph (KG) Retrieval for rapid one-hop lookups in DBpedia, 2) an LM-based classification guided by a task-specific labeling prompt, producing outputs with internal rule-based logic, and 3) a Web Search Agent invoked only when KG coverage is insufficient. Our pipeline achieves an F1 score of 0.93 on the FEVER benchmark on the Supported/Refuted split without task-specific fine-tuning. To address Not enough information cases, we conduct a targeted reannotation study showing that our approach frequently uncovers valid evidence for claims originally labeled as Not Enough Information (NEI), as confirmed by both expert annotators and LLM reviewers. With this paper, we present a modular, opensource fact-checking pipeline with fallback strategies and generalization across datasets.

Shaghayegh Kolli, Richard Rosenbaum, Timo Cavelius +3

Technical University Munich

LLM GroundingAutomated Fact-Checking
AskPDF
Mar 30, 2026

Courtroom-Style Multi-Agent Debate with Progressive RAG and Role-Switching for Controversial Claim Verification

Large language models (LLMs) remain unreliable for high-stakes claim verification due to hallucinations and shallow reasoning. While retrieval-augmented generation (RAG) and multi-agent debate (MAD) address this, they are limited by one-pass retrieval and unstructured debate dynamics. We propose a courtroom-style multi-agent framework, PROClaim, that reformulates verification as a structured, adversarial deliberation. Our approach integrates specialized roles (e.g., Plaintiff, Defense, Judge) with Progressive RAG (P-RAG) to dynamically expand and refine the evidence pool during the debate. Furthermore, we employ evidence negotiation, self-reflection, and heterogeneous multi-judge aggregation to enforce calibration, robustness, and diversity. In zero-shot evaluations on the Check-COVID benchmark, PROClaim achieves 81.7% accuracy, outperforming standard multi-agent debate by 10.0 percentage points, with P-RAG driving the primary performance gains (+7.5 pp). We ultimately demonstrate that structural deliberation and model heterogeneity effectively mitigate systematic biases, providing a robust foundation for reliable claim verification. Our code and data are publicly available at https://github.com/mnc13/PROClaim.

Masnun Nuha Chowdhury, Nusrat Jahan Beg, Umme Hunny Khan +3

Systems and Software Lab (SSL), Department of Computer Science and Engineering, Islamic University of Technology, Dhaka, Bangladesh

Claim VerificationMulti-Agent DebateMulti-Agent LLMsRetrieval-Augmented Generation
AskPDF