Turkish Multimodal Video Fact-Checking: A New Dataset and a Comparison of Zero-Shot Agentic and Fine-Tuned Systems
Organizations: MSc Artificial Intelligence, QMUL
Abstract
Video-based misinformation spreads rapidly on social media, but almost all existing fact-checking systems focus exclusively on English text. Tools for low-resource languages such as Turkish are severely limited. In this work, we present TeViD (Turkish Video Fact-checking Dataset), to the best of our knowledge the first multimodal video fact-checking dataset for Turkish. We compiled the dataset from articles published by Teyit.org, an IFCN-certified organisation, and enriched it with translated Snopes articles and synthetic examples to reduce class imbalance. Building on TeViD, we develop two verification pipelines that share the same open-source base model. First, in 3MFact-TR (Pipeline 1), we adapt the multi-role 3MFact framework to Turkish with the Qwen2.5-7B large language model and the Qwen2-VL-7B vision-language model, and add three modules of our own: a semantic inconsistency checker, a 5W1H question manager, and a bilingual evidence retriever that excludes fact-checking sites to prevent leakage. Second, as our Fine-Tuned Model (Pipeline 2), we fine-tune a single Qwen2.5-7B with Low-Rank Adaptation (LoRA) to produce closed-book verdicts from the claim and the video's textual description. We report accuracy, macro F1, True class F1, and True class recall for both systems, together with ablation studies.
Figures & tables
| Source | True | False | Total |
|---|---|---|---|
| Teyit.org (native) | 124 | 1,027 | 1,151 |
| Snopes (translated) | 407 | 324 | 731 |
| Ensonhaber (synthetic) | 215 | 0 | 215 |
| Total | 746 | 1,351 | 2,097 |
| Component | Receives | Produces |
| Offline, once per video | ||
| Video description | eight frames | one paragraph: what is on screen |
| Transcript | the audio | what is said |
| Before the loop | ||
| Consistency check (SIC) | claim, description, headline | a note: does the footage contradict the claim? |
| Pipeline 1: the verification loop | ||
| System | Acc | M-F1 | F1 T | R T |
|---|---|---|---|---|
| P0 untrained, closed-book | 0.690 | 0.611 | 0.437 | 0.347 |
| P1 3MFact-TR, open-book | 0.689 | 0.651 | 0.536 | 0.516 |
| P2-A fine-tuned, closed-book | 0.760 | 0.729 | 0.639 | 0.613 |
| Condition | Acc | M-F1 | F1 T | R T | |
|---|---|---|---|---|---|
| P1 full | 0.689 | 0.651 | 0.536 | 0.516 | — |
| w/o SIC advisory | 0.709 | 0.694 | 0.629 | 0.710 | 0.52 |
| P2-A full input | 0.760 | 0.729 | 0.639 | 0.613 | — |
| claim only | 0.749 | 0.721 | 0.634 | 0.629 | 0.73 |
| Condition | Train | Steps | Acc | M-F1 | R T | |
|---|---|---|---|---|---|---|
| P2-A native only | 286 | 120 | 0.760 | 0.729 | 0.613 | — |
| P2-B1 + Snopes | 1,017 | 120 | 0.760 | 0.726 | 0.589 | 1.00 |
| P2-B + all | 2,024 | 120 | 0.749 | 0.647 | 0.306 | 0.74 |
| P2-A34 epoch-matched | 286 | 34 | 0.721 | 0.680 | 0.524 | 0.06 |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Automated Fact-Checking
Automated fact-checking detects checkable claims, retrieves evidence, and assesses whether that evidence supports or refutes them.
See all 146 papers