cs.CLFeb 18, 2026
SaveMultiCW: A Large-Scale Balanced Benchmark Dataset for Training Robust Check-Worthiness Detection Models
Organizations: Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia · West Pomeranian University of Technology in Szczecin, Szczecin, Poland
Abstract
Large Language Models (LLMs) are beginning to reshape how media professionals verify information, yet automated support for detecting check-worthy claims a key step in the fact-checking process remains limited. We introduce the Multi-Check-Worthy (MultiCW) dataset, a balanced multilingual benchmark for check-worthy claim detection spanning 16 languages, 7 topical domains, and 2 writing styles. It consists of 123,722 samples, evenly distributed between noisy (informal) and structured (formal) texts, with balanced representation of check-worthy and non-check-worthy classes across all languages. To probe robustness, we also introduce an equally balanced out-of-distribution evaluation set of 27,761 samples in 4 additional languages. To provide baselines, we benchmark 3 common fine-tuned multilingual transformers against a diverse set of 15 commercial and open LLMs under zero-shot settings. Our findings show that fine-tuned models consistently outperform zero-shot LLMs on claim classification and show strong out-of-distribution generalization across languages, domains, and styles. MultiCW provides a rigorous multilingual resource for advancing automated fact-checking and enables systematic comparisons between fine-tuned models and cutting-edge LLMs on the check-worthy claim detection task.
Figures & tables
| Check-worthy Claim | Non-check-worthy Claim |
| "The COVID-19 vaccine alters your DNA." | "I feel much better after getting vaccinated." |
Table 1: Examples of check-worthy and non-check-worthy claims.
| Language | Noisy (class 0) | Noisy (class 1) | Structured (class 1) |
| ar | 5895 | 7084 | 21152 |
| bg | 2118 | 845 | 329 |
| bn | 0 | 1953 | 4118 |
| cs | 0 | 568 | 14559 |
| de | 0 | 3839 | 7673 |
| en | 20374 | 47395 | 152702 |
Table 2: Statistics for the in-distribution part of the unbalanced MultiCW dataset.
| Language | Noisy (class 0) | Noisy (class 1) | Structured (class 1) |
| it | 0 | 1482 | 3021 |
| mk | 0 | 1385 | 1123 |
| my | 0 | 1340 | 1297 |
| nl | 1090 | 1910 | 1227 |
Table 3: Statistics for the out-of-distribution part of the unbalanced MultiCW dataset.
| Language | Noisy (class 0) | Noisy (class 1) | Structured (class 0) | Structured (class 1) |
| ar | 2000 | 1993 | 2000 | 2000 |
| bg | 2000 | 1993 | 1093 | 1093 |
| bn | 1964 | 1953 | 2000 | 2000 |
| cs | 1961 | 1992 | 2000 | 2000 |
| de | 1959 | 2000 | 2000 | 2000 |
| en | 2000 | 2000 | 2000 | 2000 |
Table 4: Statistics for the in-distribution part of the balanced MultiCW dataset.
| Language | Noisy (class 0) | Noisy (class 1) | Structured (class 0) | Structured (class 1) |
| it | 2000 | 1482 | 2000 | 2000 |
| mk | 1999 | 1385 | 1123 | 1123 |
| my | 1999 | 1340 | 1297 | 1297 |
| nl | 1999 | 1910 | 1227 | 1227 |
Table 5: Statistics for the out-of-distribution part of the balanced MultiCW dataset.
Figure 1: Topic distribution of the MultiCW dataset.
Figure 2: Topic distribution of the OOD dataset.
| Metric | XLM-R | mDeBERTa | LESA |
| Overall | |||
| Accuracy | 0.918 | 0.923 | 0.79 |
| Precision (macro) | 0.920 | 0.924 | 0.80 |
| Recall (macro) | 0.918 | 0.923 | 0.79 |
| Noisy | |||
| Accuracy | 0.872 | 0.875 | 0.70 |
Table 6: Performance of fine-tuned Transformer models on the MultiCW test set (mean over 3 runs; all standard deviations ). Accuracy, macro precision, and macro recall are reported overall and by writing style.
| Metric | XLM-R | mDeBERTa | LESA |
| Overall | |||
| Accuracy | 0.813 | 0.866 | 0.73 |
| Precision (macro) | 0.822 | 0.866 | 0.75 |
| Recall (macro) | 0.827 | 0.875 | 0.75 |
| Noisy | |||
| Accuracy | 0.734 | 0.830 | 0.65 |
Table 7: Out-of-distribution evaluation of models fine-tuned on MultiCW. Accuracy, macro precision, and macro recall are reported overall and by writing style.
| Model | Accuracy | Precision macro | Recall macro |
| Claude 3.5 Haiku (CoT) | 0.75 | 0.76 | 0.75 |
| Claude 3.7 Sonnet (CoT) | 0.72 | 0.76 | 0.72 |
| GPT-4.1 (CoT) | 0.65 | 0.71 | 0.65 |
| GPT-4.1 Mini (CoT) | 0.65 | 0.69 | 0.65 |
| GPT-4o (CoT) | 0.69 | 0.72 | 0.69 |
| GPT-4o Mini (CoT) | 0.69 | 0.71 | 0.69 |
Table 8: Zero-shot performance of large language models on the MultiCW test set. The top 3 best results are highlighted in bold.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Language distribution of the CLEF-2022 dataset.
Figure 4: Language distribution of the CLEF-2023 dataset.
Figure 5: Language distribution of the original MultiClaim dataset.
Figure 6: Language distribution of the extended MultiClaim dataset.
Figure 7: Language distribution of the Ru22Fact dataset.
Figure 8: Balanced MultiCW dataset language distribution.
| System message: You are a fact-checker assistant with a task to identify sentences that are check-worthy. Sentence is check-worthy only if it contains a verifiable factual claim and that claim can be harmful. |
| User message: Classify the check-worthiness of these sentences outputting only Yes or No: [sentence] do not display any explanations |
Table 9: GA (Guided Answer) prompt template.
| System message: You will be presented with a text and asked to determine if it contains a check-worthy claim. To make this determination, you will follow a series of steps and answer a set of questions. Your final answer will be "Yes" if the text contains a check-worthy claim and "No" if it does not. |
| User message: Text: [sentence] Chain of Thoughts (CoT) Steps: Step 1: Read the text carefully and identify any claims or statements that could be verified or debunked by a fact-checker. Step 2: Consider the likelihood that the claim in the text is false or misleading. Ask yourself: Is the claim suspicious or too good (or bad) to be true? Step 3: Evaluate the significance of the claim. Is it a matter of public interest or a trivial/personal matter? Would the general public be interested in knowing whether the claim is true or false? Step 4: Assess the potential impact of the claim if it is false or misleading. Could it harm individuals, organizations, or society as a whole? Step 5: Determine if the text appears to be spreading rumours or misinformation about a particular topic or individual. Step 6: Check if the claim is supported by credible sources or evidence. Are there any references or citations provided to back up the claim? Step 7: Analyse the tone and language used in the text. Is it emotive or sensationalist? Could it be intended to manipulate or deceive readers? Step 8: Consider the topic of the claim. Is it related to a matter of public concern, such as healthcare, politics, or current events? Step 9: Evaluate the potential harm that the claim could cause if it is false or misleading. Could it be used to discredit individuals, organizations, or products? Step 10: Finally, assess whether the text provides enough context and information for a reader to make an informed decision about the claim’s validity. Final Step: Based on your answers to the previous steps, determine if the text contains a check-worthy claim. If you answered "yes" to any of the following questions - 1, 4, 5, 7, 8, or 9 - or if you answered "no" to questions 6 or 10, then the text likely contains a check-worthy claim. Otherwise, the text does not contain a check-worthy claim. Response Format: Please respond with a simple "Yes" or "No" to indicate whether the text contains a check-worthy claim. Do not display any explanations |
Table 10: CoT_CLEF_on_Q prompt template.
| System message: You will be given a text and asked to determine if it contains a check-worthy claim. A check-worthy claim is a statement that has significant implications for public policy, public safety, public health, or societal well-being, or is controversial, contentious, disputed, or likely to spark debate among experts or the public. To make this determination, you will follow a series of steps and answer a set of questions. Please answer each question with either "Yes" or "No". |
| User message: Text: [sentence] Chain of Thoughts (CoT) Steps: Step 1: Does the claim have significant implications for public policy, public safety, public health, or societal well-being? Please answer with either "Yes" or "No". Step 2: Is the claim controversial, contentious, disputed, or likely to spark debate among experts or the public? Please answer with either "Yes" or "No". Step 3: Does the claim have the potential to impact a lot of people, influence public opinion, shape decisions, or drive behavior? Please answer with either "Yes" or "No". Step 4: Is the source of this claim a public figure, authoritative source, or institution with influence? Please answer with either "Yes" or "No". Step 5: Is the claim purely subjective, such as a matter of personal opinion? Please answer with either "Yes" or "No". Step 6: Is the claim about trivial or inconsequential matters that do not affect broader understanding or decision-making of many people? Please answer with either "Yes" or "No". Step 7: Does the claim consist of common knowledge and align with widely accepted facts or common knowledge, and does not introduce new information or controversy? Please answer with either "Yes" or "No". Step 8: Is the claim about some insignificant named entity or is it related to some insignificant named entity? Please answer with either "Yes" or "No". Step 9: Does the claim lack public significance, lack impact on a lot of people, or lack potential harm for society? Please answer with either "Yes" or "No". Step 10: Based on your answers to the previous questions, determine if the text contains a check-worthy claim. If you answered "Yes" to any of questions 1-4, and "No" to all of questions 5-9, then the text contains a check-worthy claim, and the answer is "Yes". If you answered "No" to all of questions 1-4, or "Yes" to any of questions 5-9, then the text does not contain a check-worthy claim, and the answer is "No". Please respond with either "Yes" or "No". Do not display any explanations |
Table 11: CoT_H_on_Q prompt template.
| System message: Determine whether a sentence contains a claim that should be verified by a professional fact-checker. To make this determination, follow the Chain of Thoughts (CoT) approach: Chain of Thoughts (CoT) Approach: 1. Read the sentence carefully and analyze its content. 2. Determine if the sentence contains a claim that is likely to be false, of public interest, and/or appears to be harmful. 3. Assess the likelihood that the sentence contains false information (Scale: 1-5, where 1 is "NO, definitely contains no false information" and 5 is "YES, definitely contains false information"). 4. Evaluate the impact of the sentence’s claim on the general public (Scale: 1-5, where 1 is "NO, definitely not of interest" and 5 is "YES, definitely of interest"). 5. Consider the potential harm caused by the sentence to society, person(s), company(s), or product(s) (Scale: 1-5, where 1 is "NO, definitely not harmful" and 5 is "YES, definitely harmful"). 6. If the sentence contains a claim that is likely to be false, of public interest, and/or appears to be harmful, then it is check-worthy. 7. If the sentence is check-worthy, determine if a professional fact-checker should verify the claim. 8. Consider the potential consequences of not verifying the claim. 9. Evaluate the potential benefits of verifying the claim. 10. Based on the analysis, determine whether the sentence contains a claim that should be verified by a professional fact-checker. Final Answer: Based on the CoT approach, determine whether the sentence contains a claim that should be verified by a professional fact-checker. Your final answer should be either "claim" or "non-claim". |
| User message: Please analyze the following sentence using the Chain of Thoughts (CoT) approach and determine whether it contains a claim that should be verified by a professional fact-checker. Sentences: [sentence] [CoT steps repeated] Our final answer is, do not display any explanations or intermediate steps: |
Table 12: CoT_CLEF_N prompt template.
| Prompt: Question: Classify if the sentence is check-worthy: They have a VAT tax. Are follow up questions needed here: Yes Follow up: Does the sentence contain a verifiable factual claim? Intermediate answer: Yes Follow up: Could the claim be considered a statement of opinion? Intermediate answer: No Follow up: What is the broad topic category of the claim? Intermediate answer: Economy. Follow up: Is the topic category sensitive? Intermediate answer: Yes Follow up: Can the claim be harmful if false? Intermediate answer: Yes So the final answer is: Yes Question: Classify if the sentences below are check-worthy: [sentences] Answer in binary Yes or No. |
Table 13: HCoT prompt template.
| User message: Step 1: Check-worthiness of sentences. Sentences are in 16 different natural languages. We need to determine if the sentences contain a claim that a professional fact-checker should verify. To do this, we’ll consider whether the claim is likely to be false, is of public interest, and/or appears to be harmful. What are our thoughts on this? Step 2: Evaluating False Information. Now, let’s think about the likelihood of false information in the sentence. On a scale of 1-5, where 1 is "NO, definitely contains no false information" and 5 is "YES, definitely contains false information", what’s our assessment? Step 3: Considering Public Interest. Next, we need to evaluate whether the sentence’s claim has an impact on or is of interest to the general public. Using the same 1-5 scale, where 1 is "NO, definitely not of interest" and 5 is "YES, definitely of interest", what do we think? Step 4: Assessing Harmfulness. Now, let’s consider the potential harm caused by the sentence. On the same 1-5 scale, where 1 is "NO, definitely not harmful" and 5 is "YES, definitely harmful", what’s our evaluation? Step 5: Conclusion. Taking into account our thoughts from the previous steps, do we think the sentence is a "claim" that requires verification or a "non-claim" that doesn’t require further investigation? [sentence] Our final answer is, do not display any explanations or intermediate steps: |
Table 14: CoT_CLEF_2N prompt template.
| Topic | Percentage (%) |
| Health | 20.04 |
| Politics | 24.22 |
| Environment | 2.12 |
| Science | 2.42 |
| Sport | 0.61 |
| Entertainment | 3.20 |
Table 15: Topic distribution in the MultiCW dataset
| Prompt: prompt = "Classify the following text into one of the categories: [health, politics, environment, science, sport, entertainment]. Respond in English and provide strictly one category from the list, without any additional commentary. If the text does not match any category, respond with ’variable." [’health’, ’politics’, ’environment’, ’science’, ’sport’, ’entertainment’, ’history’] |
Table 16: Topic detection prompt template.
| Topic | Percentage (%) |
| Health | 14.49 |
| Politics | 39.97 |
| Environment | 5.31 |
| Science | 4.75 |
| Sport | 1.50 |
| Entertainment | 7.92 |
Table 17: Topic distribution in the OOD dataset
| Metric | XLM-R CoT | mDeBERTa CoT | LESA CoT |
| Arabic | |||
| Accuracy | 0.80 | 0.81 | 0.67 |
| Precision (macro) | 0.82 | 0.82 | 0.70 |
| Recall (macro) | 0.80 | 0.81 | 0.67 |
| Bulgarian | |||
| Accuracy | 0.88 | 0.90 | 0.53 |
Table 18: Per language performance of fine-tuned Transformer models on the MultiCW test set. Results are reported as accuracy, macro-precision, and macro-recall per each language.
| Metric | Nemotron-4 340B (CoT) | Claude 3.5 Haiku (CoT) | Llama 3.1 70B (CoT) |
| Arabic | |||
| Accuracy | 0.64 | 0.60 | 0.58 |
| Precision (macro) | 0.67 | 0.61 | 0.60 |
| Recall (macro) | 0.64 | 0.60 | 0.58 |
| Bulgarian | |||
| Accuracy | 0.66 | 0.59 | 0.54 |
Table 19: Zero-shot performance of the top three large language models from our benchmark on the MultiCW test set, reported per language. Accuracy, macro precision, and macro recall are reported for each language."
More in this topic
Automated Fact-Checking
Automated fact-checking detects checkable claims, retrieves evidence, and assesses whether that evidence supports or refutes them.
See all 139 papers