Multilingual Reference Need Assessment System for Wikipedia

Aitolkyn Baigutanova; Francisco Navas; Pablo Aragon; Mykola Trokhymovych; Muniza Aslam; Ai-Jou Chou; Miriam Redi; Diego Saez-Trumper

cs.CYMar 17, 2026

Multilingual Reference Need Assessment System for Wikipedia

Authors: Aitolkyn Baigutanova, Francisco Navas, Pablo Aragon, Mykola Trokhymovych, Muniza Aslam, Ai-Jou Chou, Miriam Redi, Diego Saez-Trumper

Organizations: Wikimedia Foundation San Francisco, USA · Wikimedia Foundation Brooklyn, USA · Wikimedia Foundation Barcelona, Spain · Pompeu Fabra University Wikimedia Foundation Barcelona, Spain ·

Abstract

Wikipedia is a critical source of information for millions of users across the Web. It serves as a key resource for large language models, search engines, question-answering systems, and other Web-based applications. In Wikipedia, content needs to be verifiable, meaning that readers can check that claims are backed by references to reliable sources. This depends on manual verification by editors, an effective but labor-intensive process, especially given the high volume of daily edits. To address this challenge, we introduce a multilingual machine learning system to assist editors in identifying claims requiring citations. Our approach is tested in 10 language editions of Wikipedia, outperforming existing benchmarks for reference need assessment. We not only consider machine learning evaluation metrics but also system requirements, allowing us to explore the trade-offs between model accuracy and computational efficiency under real-world infrastructure constraints. We deploy our system in production and release data and code to support further research.

Figures & tables

Figure 1. The Reference Need pipeline to compute the reference coverage score of an article revision.

Language	Used for training	Active users	Featured articles	Cited sentences	Uncited sentences
English (en)	Yes	122,379	6,508	682,361	511,614
French (fr)	Yes	17,728	2,145	327,240	353,463
German (de)	Yes	17,426	2,848	199,538	685,580
Spanish (es)	Yes	14,257	1,262	162,263	207,601
Russian (ru)	Yes	9,225	1,902	271,976	418,045
Japanese (ja)	No	12,427	92	22,005	14,462

Table 1. Statistics for the top 10 language editions by users.

Figure 2. Proportion of sentences accompanied by a citation in featured articles by language.

Panel of Figure 4 — Figure 4. Performance and latency comparison. The red dot in all subfigures represents the fine-tuned distil-128 model.

Base model	LM family	Quantize	AUC-ROC	Acc	F1-score	Precision	Recall	Mean latency (s)
Mistral-7B-Instruct-v0.3	LLM	FP16	0.557 $\pm$ 0.007	0.521	0.649	0.512	0.886	0.118
Meta-Llama-3.1-8B-Instruct-Turbo	LLM	FP8	0.595 $\pm$ 0.006	0.565	0.564	0.565	0.564	0.110
Llama-3-70b-chat-hf	LLM	FP16	0.607 $\pm$ 0.006	0.560	0.642	0.541	0.790	5.268
Meta-Llama-3-70B-Instruct-Lite	LLM	INT4	0.619 $\pm$ 0.006	0.566	0.643	0.546	0.783	3.390
Citation-Needed ( Redi et al., 2019 ) *	SLM	-	0.650	0.590	0.491	0.647	0.395	-
distilbert-128	SLM	torch-INT8	0.750 $\pm$ 0.006	0.676	0.696	0.655	0.743	0.023

Table 2. Reference Need model performance sorted by AUC-ROC (the final model is underlined and the best performance values are bolded).

Figure 5. Performance by language using distilbert-128.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 7. Confusion matrices by language using distil-128 model without optimizations.

Multilingual Text Classification

See all 20 papers

Multilingual and Cross-Lingual Citation Needed Detection on Wikipedia for Lower-Resource Languages
May 29, 2026Gerrit Quaremba, Amy Rechkemmer, Elizabeth Black +2Multilingual Language Model Evaluation Small Language Models
King’s College London · Wikimedia Foundation
Multilingual Fact-Checking at Scale: Fine-Tuned Compact Models vs LLMs
Jun 7, 2026Pratuat Amatya, Vinay SettyMultilingual Language Models LLM Evaluation
Factiverse AS and University of Stavanger Stavanger, Norway
Can Language Models Play the Wikipedia Game?
Oct 9, 2026Alexander Hickey, Joanna Knight, James BishopLLM Evaluation Multilingual Language Model Evaluation
The Alan Turing Institute, 96 Euston Rd., London NW1 2DB, UK

May 29, 2026J/K move · Enter open · S save

Multilingual and Cross-Lingual Citation Needed Detection on Wikipedia for Lower-Resource Languages

Gerrit Quaremba, Amy Rechkemmer, Elizabeth Black, Denny Vrandečić+1

King’s College London · Wikimedia Foundation

Multilingual Language Model Evaluation Small Language Models Automated Fact-Checking Low-Resource Language Processing Cross-Lingual Transfer

Open paper AskPDF arXiv

In automated fact-checking (AFC), check-worthiness detection identifies claims requiring verification based on domain-specific criteria. On Wikipedia, this task instantiates as Citation Needed Detection (CND), which flags claims lacking supporting citations. However, existing research has largely overlooked lower-resource languages, and recent AFC pipelines rely on large language models (LLMs), which are inaccessible to low-resource organizations. We introduce MCN, a multilingual CND corpus spanning 18 languages across three resource levels, on which we conduct an extensive study of small decoder-based language models (SLMs). Our experiments show that SLMs fine-tuned with an encoder-style objective substantially outperform prompted LLMs across languages. We further present one of the first studies on cross-lingual CND, demonstrating that SLMs fine-tuned solely on English claims surpass LLMs, even with little to no target-language adaptation. Our findings have important implications for lower-resource Wikipedia communities and suggest that compact, task-specific models are preferable to LLMs for CND. We release all data and code at https://github.com/gerritq/mcn

Abstract

Multilingual Text Classification

Explore similar work

Multilingual and Cross-Lingual Citation Needed Detection on Wikipedia for Lower-Resource Languages

Multilingual Fact-Checking at Scale: Fine-Tuned Compact Models vs LLMs

Can Language Models Play the Wikipedia Game?