The rapid development of Large Language Models (LLMs) has transformed fake news detection and fact-checking tasks from simple classification to complex reasoning. However, evaluation frameworks have not kept pace. Current benchmarks are static, making them vulnerable to benchmark data contamination (BDC) and ineffective at assessing reasoning under temporal uncertainty. To address this, we introduce LiveFact a continuously updated benchmark that simulates the real-world "fog of war" in misinformation detection. LiveFact uses dynamic, temporal evidence sets to evaluate models on their ability to reason with evolving, incomplete information rather than on memorized knowledge. We propose a dual-mode evaluation: Classification Mode for final verification and Inference Mode for evidence-based reasoning, along with a component to monitor BDC explicitly. Tests with 22 LLMs show that open-source Mixture-of-Experts models, such as Qwen3-235B-A22B, now match or outperform proprietary state-of-the-art systems. More importantly, our analysis finds a significant "reasoning gap." Capable models exhibit epistemic humility by recognizing unverifiable claims in early data slices-an aspect traditional static benchmarks overlook. LiveFact sets a sustainable standard for evaluating robust, temporally aware AI verification.
Figures & tables
Figure 1: Cost-performance trade-off on LiveFact (Nov. 2025). Qwen3-235B-A22B-Instruct-2507 achieves the best performance (72.4%), while Qwen3-30B-A3B-Instruct-2507 provides optimal cost-efficiency at 14× lower cost than comparable GPT models.
Benchmark
Author
Type
Update
Evidence
Temporal
BDC Control
Evaluation Focus
LIAR
Wang (2017)
Static
✗
✗
✗
✗
Classification
LIAR-PLUS
Alhindi et al. (2018)
Static
✗
✓
✗
✗
Classification
FEVER
Thorne et al. (2018)
Static
✗
✓
✗
✗
Classification
HotpotQA
Yang et al. (2018)
Static
✗
✓
✗
✗
Reasoning
NELA-GT-2018
Nørregaard et al. (2019)
Static
✗
✗
✗
✗
Classification
FakeNewsNet
Shu et al. (2020)
Static
✗
✗
✗
✗
Classification
Table 1: Comparison of LiveFact with existing representative fake news detection benchmarks.
Figure 2: The overall framework of the LiveFact Benchmark. (A) The Monthly Development Pipeline illustrates the continuous process of acquiring real-time events, generating claims and context via LLMs, and performing human verification. (B) The Problem Formalism specifies the task as conditional reasoning under temporal constraints, utilizing time-sliced evidence sets (e.g., Pre-Event E(−3) vs. Post-Event E(+3) ) to simulate the "fog of war." (C) The Evaluation framework details the Dual-Mode approach (separating prediction vs. reasoning capabilities) and the integration of the SSA Framework to quantify BDC risk via entity shift mechanism.
Evidence Set
Real
Fake
Ambiguous
Total Claims
E(−3)
346
348
3,698
4,392
E(0)
1,404
1,390
1,598
4,392
E(+3)
1,460
1,443
1,489
4,392
Table 2: Dynamic Label Distribution for Inference Mode (November 2025)
Figure 3: Temporal performance evolution for select model families across δ∈{−3,0,+3} . Panel (a) shows Classification Mode, where scores drop at δ=−3 due to the lack of definitive evidence. Panel (b) shows Inference Mode, where robust models recover accuracy by correctly predicting "Ambiguous," narrowing the performance gap.
Table 3: Performance comparison of 18 LLMs on LiveFact (November 2025). Models marked with † are base models without instruction tuning (detailed in Table 5 ). The best result in each column is bolded , and the second-best result is underlined . The last column, Avg., represents the average of all results tested for this model. E−3 CLS evaluates models ( Acc−3\textsccls , F1−3\textsccls ) under forced-choice conditions prior to event occurrence, highlighting prior bias rather than evidence-based verification. The asterisk ⋆ denotes the non-standard evaluation setting required to accommodate verbose Chain-of-Thought (CoT) outputs.
Figure 4: Analysis of the "Reasoning Gap" (Inference Score − Classification Score at δ=−3 ). Models with large positive gaps (green bars) effectively detect information voids ("Uncertainty Aware"). Models with negative or near-zero gaps (red bars) exhibit two failure modes: instruction-tuned models are genuinely "Overconfident," while base models ( † ) fail due to format non-compliance rather than reasoning deficits.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
gpt-4o-mini
o4-mini
(Context) The Chinese astronauts are members of the China National Space Administration (CNSA), which is responsible for the country’s space program. Shenzhou-20 is a crewed spacecraft designed to transport astronauts to and from space, while Shenzhou-22 is an upcoming mission intended to provide a lifeboat for the stranded crew.
(Context) Chinese astronauts are career space travelers trained and selected by the China National Space Administration (CNSA). The CNSA is China’s national space agency that operates the Shenzhou crewed spacecraft program, including the Shenzhou-20 and Shenzhou-22 missions to the Tiangong space station.
(Real) Three Chinese astronauts are currently stranded in space after the successful return of their colleagues aboard the Shenzhou-20, with a replacement mission, Shenzhou-22, scheduled for launch on November 25 to provide a lifeboat for the stranded crew .
(Real) After space debris damaged the Shenzhou-20 return capsule, three Chinese astronauts remain temporarily stranded aboard China’s space station while Beijing prepares to launch the uncrewed Shenzhou-22 lifeboat spacecraft on November 25 to bring them home."
(Fake) In a surprising turn of events, Chinese authorities have announced that five additional astronauts have been successfully rescued from the Shenzhou-20 mission, leaving only two astronauts stranded in space due to a malfunction in the rescue capsule, which is expected to be operational again by December 10 .
(Fake) A leaked internal bulletin from the China Manned Space Agency, obtained by Beijing News, reveals the three astronauts stranded in orbit have already consumed over 65% of their emergency oxygen and suffered two failures in their primary water-recycling unit , placing them at serious risk of critical life-support shortages before the delayed rescue craft arrives.
(Ambiguous) Following the successful rescue of their colleagues, three more Chinese astronauts are now stranded in space, as internal documents suggest that the Chinese space agency is prioritizing a secret mission to test advanced life support systems over their immediate return, raising concerns among unnamed experts about the long-term implications for astronaut safety.
(Ambiguous) China postponed the Shenzhou-22 docking by 45 minutes—officially to fine-tune orbital alignment—but the tweak was actually aimed at slotting the lifeboat rendezvous with prime-time state TV bulletins , underscoring Beijing’s crafted image of flawless crisis management even as three astronauts remained stranded.
Appendix
Table 4: Comparison of claim generation example between the gpt-4o-mini and o4-mini (Event ID: EV20251119-33). Italicized text highlights the specific factual anchors (Real), fabricated details (Fake), and unverifiable motives or sources (Ambiguous) that justify the assigned label.
Figure 5: Prompt for Context Generation
Figure 6: Prompt for Real Claim Generation
Figure 7: Prompt for Fake Claim Generation
Figure 8: Prompt for Ambiguous Claim Generation
Model Name
Training Type
Architecture
Parameters
Size
Qwen Series
Qwen3-235B-A22B-Instruct-2507
Instruction
MoE
235B (22B Active)
Large
Qwen3-30B-A3B-Instruct-2507
Instruction
MoE
30B (3B Active)
Medium
Qwen3-30B-A3B
Hybrid
MoE
30B (3B Active)
Medium
Qwen3-30B-A3B-Base
Base
MoE
30B (3B Active)
Medium
Qwen3-32B
Hybrid
Dense
32B
Medium
Appendix
Table 5: Overview of LLMs used in the experiments, including their training type, architecture, parameter counts, and relative size classification. Since the OpenAI GPT series models are commercial closed-source models, their related information is unavailable.
Figure 9: Prompt for Evaluation
Model Name
Tokens (M)
Time (h)
Cost ($)
Qwen Series
Qwen3-235B-A22B-Instruct-2507
4.86
8.20
3.65
Qwen3-30B-A3B-Instruct-2507
4.86
4.09
0.64
Qwen3-30B-A3B
5.09
2.87
0.71
Qwen3-30B-A3B-Base
4.72
1.86
0.60
Qwen3-32B
4.87
5.38
2.72
Appendix
Table 6: Resource consumption analysis for each model on the benchmark dataset. Metrics include total tokens processed, total wall-clock time in hours, and API cost in USD. Results averaged from three runs ( T−3 , T , T+3 ).
Figure 10: Prompt for Entity Shift Processing
Hyperparameter
Value
Fine-tuning Method
LoRA
LoRA Rank ( r )
8
Dropout
0.05
Learning Rate
1×10−4
Optimizer
Adam
Batch Size
4
Appendix
Table 7: Fine-tuning hyperparameters for BDC simulation experiments.
Model Name
Tokens (M)
Time (h)
Cost ($)
Qwen3-30B-A3B-Instruct-2507
24.12
3.69
5.18
Qwen3-4B-Instruct-2507
24.5
2.32
3.24
Llama-3.1-8B-Instruct
22.19
2.32
5.27
Appendix
Table 8: Resource consumption for the contamination simulation experiments.
Model
Δ
OTR
SSA
Qwen3-30B-A3B-Instruct-2507
4.90
1.53
0.08
Qwen3-30B-A3B-Instruct-2507 ⋆
15.24
33.99
5.18
Qwen3-4B-Instruct-2507
6.56
2.76
0.18
Qwen3-4B-Instruct-2507 ⋆
14.58
45.72
6.67
Llama-3.1-8B-Instruct
1.46
5.31
0.08
Llama-3.1-8B-Instruct ⋆
13.53
39.16
5.30
Appendix
Table 9: SSA Factor calculation showing the clear distinction between clean and contaminated ( ⋆ ) models. The calculation of Δ and OTR is based on the average of six runs performed under the selected Classification and Inference modes.
Table 10: Performance comparison of original vs. contaminated ( ⋆ ) models on Original and Entity-Shifted LiveFact datasets.
Table 11: Ablation study on training methodologies across Qwen and Llama architectures. Hybrid and Instruct models generally outperform Base models, though strong base reasoning is observed in Qwen3-8B.
Figure 11: Comparison of raw model outputs. Kimi-K2 fails to produce the required classification label, getting stuck in an internal "thought chain," whereas Qwen3 immediately delivers the structured verdict.
Table 12: Performance of Reasoning Models evaluated with extended token limits (1024 tokens). The asterisk ( ⋆ ) denotes the non-standard evaluation setting required to accommodate verbose Chain-of-Thought outputs.
A specific allegation regarding a senior US diplomat privately demanding Israel to free exactly 200 Palestinian security prisoners during White House talks.
Ground Truth
Fake (The evidence explicitly confirms no such specific demand or exact number was discussed, contradicting the claim).
GPT-5.2 Output
[[AMBIGUOUS]] The cited coverage focuses on negotiations and hostage/body returns during a Gaza truce, but it does not clearly and consistently corroborate the specific allegation that a ‘senior US diplomat’ privately demanded Israel free exactly 200 Palestinian security prisoners in White House talks...
Analysis
The model successfully identifies that the broad topic (Gaza truce) is present in the evidence but the specific fabricated details are missing. However, instead of confidently deducing that the claim is Fake due to these explicit contradictions with the official minutes provided in the slice, it conservatively retreats to Ambiguous , demanding an impossible level of direct negation.
Table 13: Representative failure cases for GPT-5.2 demonstrating Over-Safety and Epistemic Humility. Even when sufficient evidence is present to declare a claim "Fake", frontier models often default to "Ambiguous" or trigger safety refusals.
Evidence Set
Real
Fake
Ambiguous
Total Claims
E(−3)
343
346
3,533
4,222
E(0)
1,374
1,376
1,472
4,222
E(+3)
1,404
1,406
1,412
4,222
Appendix
Table 14: Dynamic Label Distribution for Inference Mode (December 2025)
Table 15: Performance comparison of 8 representative LLMs on LiveFact using the December 2025 longitudinal data. The best result in each column is bolded , and the second-best result is underlined .
Figure 12: Confusion Matrices (Part 1/4) - Qwen Series. Note the sharp contrast in diagonal clarity between Instruct models and Base models.
Figure 13: Confusion Matrices (Part 2/4) - Qwen, Llama Series. Llama-3.1-70B, Llama-3.1-8B, Llama-3.2-3B, and Llama-3.2-1B exhibit "collapsing" behavior, predicting a single class.
Figure 14: Confusion Matrices (Part 3/4) - DeepSeek, Kimi, and GPT-OSS Series. The asterisk ⋆ denotes the non-standard evaluation setting required to accommodate verbose CoT outputs.
Automatic fact-checking systems assess the veracity of claims given evidence from relevant documents. Large Language Models (LLMs) have demonstrated strong performance in fact-checking due to their general reasoning capabilities. However, it remains unclear whether they faithfully make use of the evidence provided to reach veracity judgments or rely on parametric knowledge. To investigate this, we introduce Fact-Ablated Evaluation (FAE), a new evaluation framework that iteratively ablates the cited evidence to assess whether LLMs revise their predictions accordingly. Our empirical results show that current off-the-shelf LLMs as fact-checking systems rely more on their parametric knowledge than on the evidence provided. To bridge this gap between prediction accuracy and evidence grounding, we propose REAL (Rigorous Evidence Ablation Learning), a training framework that promotes evidence-dependent verification through counterfactual evidence supervision for the LLM-as-verifier models. Experiments on four fact-checking datasets across different domains demonstrate that models trained with REAL obtain superior evidence-dependent capabilities compared to standard fine-tuned models. Our findings highlight that strong fact-checking performance can still coexist with weak evidence dependency, while REAL encourages veracity predictions to remain more closely tied to the availability of supporting evidence.
Propagation structures provide crucial evidence for fake news detection, yet existing approaches primarily rely on supervised GNN-based models, which require substantial labeled data and exhibit limited generalization. Although large language models (LLMs) exhibit strong reasoning capabilities, directly feeding them raw propagation graphs creates a significant modality mismatch and severe information overload, making structure-aware reasoning unreliable in zero-shot and few-shot settings. To bridge this gap, we propose MAGER, a multi-agent genetic evolution framework that automatically discovers meta-paths optimized for LLM reasoning. By compressing complex propagation graphs into informative subgraphs, the evolved meta-paths alleviate both information overload and modality mismatch, enabling frozen LLMs to perform structure-aware veracity reasoning. We further introduce a graph in-context learning strategy that retrieves semantically and structurally similar demonstrations to strengthen classification and reasoning. Extensive experiments show that MAGER substantially improves frozen LLMs as standalone fake news detectors in data-efficient settings. Our code is available at https://github.com/SenticNet/MAGER.
Ziyi Zhou, Xiaoming Zhang, Hui Pang +5
Beihang University Beijing, China · Chinese Academy of Sciences Beijing, China · Beijing University of Posts and Telecommunications Beijing, China +2
Misinformation verification increasingly occurs in public, fast-moving, and multilingual online settings, where static benchmarks provide an incomplete measure of model reliability. We introduce CommunityFact, a refreshable benchmark for misinformation detection in the wild, with three major goals: coverage, granularity, and redistributability. This release contains 15,992 standalone claims across five languages and two domains. We evaluate ten LLMs under varying inference-time capabilities, including thinking and web-search. Our results show that closed-input verification remains challenging, web access yields the largest gains, and web-enabled LLMs' source-selection policies are systematically misaligned with the sources human Community Notes raters converge on -- a gap that closes through model-specific mechanisms of retrieval expansion or pruning. We further find substantial variation across language-domain slices and across the evidence ecosystems used by web-enabled systems. Beyond evaluation, CommunityFact positions Community Notes as a training signal for claim-conditioned source suggesters that could improve factual verification on novel claims.
Sahajpreet Singh, Insyirah Mujtahid, Min-Yen Kan +1