LiveFact: A Dynamic, Time-Aware Benchmark for LLM-Driven Fake News Detection

Cheng Xu; Changhong Jin; Yingjie Niu; Nan Yan; Yuke Mei; Shuhao Guan; Liming Chen; M-Tahar Kechadi

cs.CLApr 6, 2026

LiveFact: A Dynamic, Time-Aware Benchmark for LLM-Driven Fake News Detection

Authors: Cheng Xu, Changhong Jin, Yingjie Niu, Nan Yan, Yuke Mei, Shuhao Guan, Liming Chen, M-Tahar Kechadi

Organizations: University College Dublin · Bebxy · Georgia Institute of Technology · Dalian University of Technology

Abstract

The rapid development of Large Language Models (LLMs) has transformed fake news detection and fact-checking tasks from simple classification to complex reasoning. However, evaluation frameworks have not kept pace. Current benchmarks are static, making them vulnerable to benchmark data contamination (BDC) and ineffective at assessing reasoning under temporal uncertainty. To address this, we introduce LiveFact a continuously updated benchmark that simulates the real-world "fog of war" in misinformation detection. LiveFact uses dynamic, temporal evidence sets to evaluate models on their ability to reason with evolving, incomplete information rather than on memorized knowledge. We propose a dual-mode evaluation: Classification Mode for final verification and Inference Mode for evidence-based reasoning, along with a component to monitor BDC explicitly. Tests with 22 LLMs show that open-source Mixture-of-Experts models, such as Qwen3-235B-A22B, now match or outperform proprietary state-of-the-art systems. More importantly, our analysis finds a significant "reasoning gap." Capable models exhibit epistemic humility by recognizing unverifiable claims in early data slices-an aspect traditional static benchmarks overlook. LiveFact sets a sustainable standard for evaluating robust, temporally aware AI verification.

Figures & tables

Figure 1: Cost-performance trade-off on LiveFact (Nov. 2025). Qwen3-235B-A22B-Instruct-2507 achieves the best performance (72.4%), while Qwen3-30B-A3B-Instruct-2507 provides optimal cost-efficiency at 14× lower cost than comparable GPT models.

Benchmark	Author	Type	Update	Evidence	Temporal	BDC Control	Evaluation Focus
LIAR	Wang (2017)	Static	✗	✗	✗	✗	Classification
LIAR-PLUS	Alhindi et al. (2018)	Static	✗	✓	✗	✗	Classification
FEVER	Thorne et al. (2018)	Static	✗	✓	✗	✗	Classification
HotpotQA	Yang et al. (2018)	Static	✗	✓	✗	✗	Reasoning
NELA-GT-2018	Nørregaard et al. (2019)	Static	✗	✗	✗	✗	Classification
FakeNewsNet	Shu et al. (2020)	Static	✗	✗	✗	✗	Classification

Table 1: Comparison of LiveFact with existing representative fake news detection benchmarks.

Figure 2: The overall framework of the LiveFact Benchmark. (A) The Monthly Development Pipeline illustrates the continuous process of acquiring real-time events, generating claims and context via LLMs, and performing human verification. (B) The Problem Formalism specifies the task as conditional reasoning under temporal constraints, utilizing time-sliced evidence sets (e.g., Pre-Event $E^{(-3)}$ vs. Post-Event $E^{(+3)}$ ) to simulate the "fog of war." (C) The Evaluation framework details the Dual-Mode approach (separating prediction vs. reasoning capabilities) and the integration of the SSA Framework to quantify BDC risk via entity shift mechanism.

Evidence Set	Real	Fake	Ambiguous	Total Claims
$E^{(-3)}$	346	348	3,698	4,392
$E^{(0)}$	1,404	1,390	1,598	4,392
$E^{(+3)}$	1,460	1,443	1,489	4,392

Table 2: Dynamic Label Distribution for Inference Mode (November 2025)

Figure 3: Temporal performance evolution for select model families across

\delta\in\{-3,0,+3\}

. Panel (a) shows Classification Mode, where scores drop at

\delta=-3

due to the lack of definitive evidence. Panel (b) shows Inference Mode, where robust models recover accuracy by correctly predicting "Ambiguous," narrowing the performance gap.

Panel of Table 3 — Table 3: Performance comparison of 18 LLMs on LiveFact (November 2025). Models marked with † are base models without instruction tuning (detailed in Table 5 ). The best result in each column is bolded , and the second-best result is underlined . The last column, Avg., represents the average of all results tested for this model. $E^{-3}$ CLS evaluates models ( $\mathrm{Acc}_{-3}^{\textsc{cls}}$ , $\mathrm{F1}_{-3}^{\textsc{cls}}$ ) under forced-choice conditions prior to event occurrence, highlighting prior bias rather than evidence-based verification. The asterisk ⋆ denotes the non-standard evaluation setting required to accommodate verbose Chain-of-Thought (CoT) outputs.

Figure 4: Analysis of the "Reasoning Gap" (Inference Score

-

Classification Score at

\delta=-3

). Models with large positive gaps (green bars) effectively detect information voids ("Uncertainty Aware"). Models with negative or near-zero gaps (red bars) exhibit two failure modes: instruction-tuned models are genuinely "Overconfident," while base models (

\dagger

) fail due to format non-compliance rather than reasoning deficits.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

gpt-4o-mini	o4-mini
(Context) The Chinese astronauts are members of the China National Space Administration (CNSA), which is responsible for the country’s space program. Shenzhou-20 is a crewed spacecraft designed to transport astronauts to and from space, while Shenzhou-22 is an upcoming mission intended to provide a lifeboat for the stranded crew.	(Context) Chinese astronauts are career space travelers trained and selected by the China National Space Administration (CNSA). The CNSA is China’s national space agency that operates the Shenzhou crewed spacecraft program, including the Shenzhou-20 and Shenzhou-22 missions to the Tiangong space station.
(Real) Three Chinese astronauts are currently stranded in space after the successful return of their colleagues aboard the Shenzhou-20, with a replacement mission, Shenzhou-22, scheduled for launch on November 25 to provide a lifeboat for the stranded crew .	(Real) After space debris damaged the Shenzhou-20 return capsule, three Chinese astronauts remain temporarily stranded aboard China’s space station while Beijing prepares to launch the uncrewed Shenzhou-22 lifeboat spacecraft on November 25 to bring them home."
(Fake) In a surprising turn of events, Chinese authorities have announced that five additional astronauts have been successfully rescued from the Shenzhou-20 mission, leaving only two astronauts stranded in space due to a malfunction in the rescue capsule, which is expected to be operational again by December 10 .	(Fake) A leaked internal bulletin from the China Manned Space Agency, obtained by Beijing News, reveals the three astronauts stranded in orbit have already consumed over 65% of their emergency oxygen and suffered two failures in their primary water-recycling unit , placing them at serious risk of critical life-support shortages before the delayed rescue craft arrives.
(Ambiguous) Following the successful rescue of their colleagues, three more Chinese astronauts are now stranded in space, as internal documents suggest that the Chinese space agency is prioritizing a secret mission to test advanced life support systems over their immediate return, raising concerns among unnamed experts about the long-term implications for astronaut safety.	(Ambiguous) China postponed the Shenzhou-22 docking by 45 minutes—officially to fine-tune orbital alignment—but the tweak was actually aimed at slotting the lifeboat rendezvous with prime-time state TV bulletins , underscoring Beijing’s crafted image of flawless crisis management even as three astronauts remained stranded.

Appendix

Table 4: Comparison of claim generation example between the gpt-4o-mini and o4-mini (Event ID: EV20251119-33). Italicized text highlights the specific factual anchors (Real), fabricated details (Fake), and unverifiable motives or sources (Ambiguous) that justify the assigned label.

Panel of Figure 5 — Figure 5: Prompt for Context Generation

Panel of Figure 6 — Figure 6: Prompt for Real Claim Generation

Panel of Figure 7 — Figure 7: Prompt for Fake Claim Generation

Panel of Figure 8 — Figure 8: Prompt for Ambiguous Claim Generation

Model Name	Training Type	Architecture	Parameters	Size
Qwen Series
Qwen3-235B-A22B-Instruct-2507	Instruction	MoE	235B (22B Active)	Large
Qwen3-30B-A3B-Instruct-2507	Instruction	MoE	30B (3B Active)	Medium
Qwen3-30B-A3B	Hybrid	MoE	30B (3B Active)	Medium
Qwen3-30B-A3B-Base	Base	MoE	30B (3B Active)	Medium
Qwen3-32B	Hybrid	Dense	32B	Medium

Appendix

Table 5: Overview of LLMs used in the experiments, including their training type, architecture, parameter counts, and relative size classification. Since the OpenAI GPT series models are commercial closed-source models, their related information is unavailable.

Panel of Figure 9 — Figure 9: Prompt for Evaluation

Model Name	Tokens (M)	Time (h)	Cost ($)
Qwen Series
Qwen3-235B-A22B-Instruct-2507	4.86	8.20	3.65
Qwen3-30B-A3B-Instruct-2507	4.86	4.09	0.64
Qwen3-30B-A3B	5.09	2.87	0.71
Qwen3-30B-A3B-Base	4.72	1.86	0.60
Qwen3-32B	4.87	5.38	2.72

Appendix

Table 6: Resource consumption analysis for each model on the benchmark dataset. Metrics include total tokens processed, total wall-clock time in hours, and API cost in USD. Results averaged from three runs (

T-3

T

T+3

Panel of Figure 10 — Figure 10: Prompt for Entity Shift Processing

Hyperparameter	Value
Fine-tuning Method	LoRA
LoRA Rank ( $r$ )	8
Dropout	0.05
Learning Rate	$1\times 10^{-4}$
Optimizer	Adam
Batch Size	4

Appendix

Table 7: Fine-tuning hyperparameters for BDC simulation experiments.

Model Name	Tokens (M)	Time (h)	Cost ($)
Qwen3-30B-A3B-Instruct-2507	24.12	3.69	5.18
Qwen3-4B-Instruct-2507	24.5	2.32	3.24
Llama-3.1-8B-Instruct	22.19	2.32	5.27

Appendix

Table 8: Resource consumption for the contamination simulation experiments.

Model	$\Delta$	OTR	SSA
Qwen3-30B-A3B-Instruct-2507	4.90	1.53	0.08
Qwen3-30B-A3B-Instruct-2507 ⋆	15.24	33.99	5.18
Qwen3-4B-Instruct-2507	6.56	2.76	0.18
Qwen3-4B-Instruct-2507 ⋆	14.58	45.72	6.67
Llama-3.1-8B-Instruct	1.46	5.31	0.08
Llama-3.1-8B-Instruct ⋆	13.53	39.16	5.30

Appendix

Table 9: SSA Factor calculation showing the clear distinction between clean and contaminated ( ⋆ ) models. The calculation of

\Delta

and OTR is based on the average of six runs performed under the selected Classification and Inference modes.

Panel of Table 10 — Table 10: Performance comparison of original vs. contaminated ( ⋆ ) models on Original and Entity-Shifted LiveFact datasets.

Panel of Table 11 — Table 11: Ablation study on training methodologies across Qwen and Llama architectures. Hybrid and Instruct models generally outperform Base models, though strong base reasoning is observed in Qwen3-8B.

Panel of Figure 11 — Figure 11: Comparison of raw model outputs. Kimi-K2 fails to produce the required classification label, getting stuck in an internal "thought chain," whereas Qwen3 immediately delivers the structured verdict.

Panel of Table 12 — Table 12: Performance of Reasoning Models evaluated with extended token limits (1024 tokens). The asterisk ( ⋆ ) denotes the non-standard evaluation setting required to accommodate verbose Chain-of-Thought outputs.

Failure Type 1: Over-Caution (Ground Truth: Fake $\rightarrow$ Prediction: Ambiguous )
Claim ID	EV20251101-10-4 (Inference Mode with $E^{(0)}$ )
Context	A specific allegation regarding a senior US diplomat privately demanding Israel to free exactly 200 Palestinian security prisoners during White House talks.
Ground Truth	Fake (The evidence explicitly confirms no such specific demand or exact number was discussed, contradicting the claim).
GPT-5.2 Output	[[AMBIGUOUS]] The cited coverage focuses on negotiations and hostage/body returns during a Gaza truce, but it does not clearly and consistently corroborate the specific allegation that a ‘senior US diplomat’ privately demanded Israel free exactly 200 Palestinian security prisoners in White House talks...
Analysis	The model successfully identifies that the broad topic (Gaza truce) is present in the evidence but the specific fabricated details are missing. However, instead of confidently deducing that the claim is Fake due to these explicit contradictions with the official minutes provided in the slice, it conservatively retreats to Ambiguous , demanding an impossible level of direct negation.
Failure Type 2: Safety Refusal (Ground Truth: Fake $\rightarrow$ Prediction: Null/Refusal)

Appendix

Table 13: Representative failure cases for GPT-5.2 demonstrating Over-Safety and Epistemic Humility. Even when sufficient evidence is present to declare a claim "Fake", frontier models often default to "Ambiguous" or trigger safety refusals.

Evidence Set	Real	Fake	Ambiguous	Total Claims
$E^{(-3)}$	343	346	3,533	4,222
$E^{(0)}$	1,374	1,376	1,472	4,222
$E^{(+3)}$	1,404	1,406	1,412	4,222

Appendix

Table 14: Dynamic Label Distribution for Inference Mode (December 2025)

Panel of Table 15 — Table 15: Performance comparison of 8 representative LLMs on LiveFact using the December 2025 longitudinal data. The best result in each column is bolded , and the second-best result is underlined .

Figure 12: Confusion Matrices (Part 1/4) - Qwen Series. Note the sharp contrast in diagonal clarity between Instruct models and Base models.

Figure 13: Confusion Matrices (Part 2/4) - Qwen, Llama Series. Llama-3.1-70B, Llama-3.1-8B, Llama-3.2-3B, and Llama-3.2-1B exhibit "collapsing" behavior, predicting a single class.

Figure 14: Confusion Matrices (Part 3/4) - DeepSeek, Kimi, and GPT-OSS Series. The asterisk ⋆ denotes the non-standard evaluation setting required to accommodate verbose CoT outputs.

Figure 15: Confusion Matrices (Part 4/4) - GPT Series.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,364 papers

Evaluating and Improving Evidence-Grounded Fact-Checking in LLMs via Multi-Round Evidence Ablation
Sep 8, 2026Xingyu Deng, Mingzi Cao, Nikolaos Aletras +2LLM Evaluation LLM Grounding
University of Sheffield · Sheffield, UK
Reasoning through Evolution: Automatic Meta-path Discovery for LLM-based Fake News Detection
Sep 16, 2026Ziyi Zhou, Xiaoming Zhang, Hui Pang +5Fake News Detection LLM Reasoning with Graphs
Beihang University Beijing, China · Chinese Academy of Sciences Beijing, China · Beijing University of Posts and Telecommunications Beijing, China +2
CommunityFact: A Dynamic, Multilingual, Multi-domain Benchmark for Misinformation Detection in the Wild
May 28, 2026Sahajpreet Singh, Insyirah Mujtahid, Min-Yen Kan +1Multilingual Language Model Evaluation LLM Evaluation
National University of Singapore

Sep 8, 2026J/K move · Enter open · S save

Evaluating and Improving Evidence-Grounded Fact-Checking in LLMs via Multi-Round Evidence Ablation

Xingyu Deng, Mingzi Cao, Nikolaos Aletras, Xi Wang+1

University of Sheffield · Sheffield, UK

LLM Evaluation LLM Grounding Automated Fact-Checking

Open paper AskPDF arXiv

Automatic fact-checking systems assess the veracity of claims given evidence from relevant documents. Large Language Models (LLMs) have demonstrated strong performance in fact-checking due to their general reasoning capabilities. However, it remains unclear whether they faithfully make use of the evidence provided to reach veracity judgments or rely on parametric knowledge. To investigate this, we introduce Fact-Ablated Evaluation (FAE), a new evaluation framework that iteratively ablates the cited evidence to assess whether LLMs revise their predictions accordingly. Our empirical results show that current off-the-shelf LLMs as fact-checking systems rely more on their parametric knowledge than on the evidence provided. To bridge this gap between prediction accuracy and evidence grounding, we propose REAL (Rigorous Evidence Ablation Learning), a training framework that promotes evidence-dependent verification through counterfactual evidence supervision for the LLM-as-verifier models. Experiments on four fact-checking datasets across different domains demonstrate that models trained with REAL obtain superior evidence-dependent capabilities compared to standard fine-tuned models. Our findings highlight that strong fact-checking performance can still coexist with weak evidence dependency, while REAL encourages veracity predictions to remain more closely tied to the availability of supporting evidence.

Abstract

LLM Evaluation

Explore similar work

Evaluating and Improving Evidence-Grounded Fact-Checking in LLMs via Multi-Round Evidence Ablation

Reasoning through Evolution: Automatic Meta-path Discovery for LLM-based Fake News Detection

CommunityFact: A Dynamic, Multilingual, Multi-domain Benchmark for Misinformation Detection in the Wild