WebFAQ 2.0: A Multilingual QA Dataset with Mined Hard Negatives for Dense Retrieval

Michael Dinzinger; Laura Caspari; Ali Salman; Irvin Topi; Jelena Mitrović; Michael Granitzer

cs.IRFeb 19, 2026

WebFAQ 2.0: A Multilingual QA Dataset with Mined Hard Negatives for Dense Retrieval

Authors: Michael Dinzinger, Laura Caspari, Ali Salman, Irvin Topi, Jelena Mitrović, Michael Granitzer

Organizations: University of Passau Passau, Germany · University of Passau, and IT:U Austria Passau, Germany

Abstract

We introduce WebFAQ 2.0, a new version of the WebFAQ dataset, containing 198 million FAQ-based natural question-answer pairs across 108 languages. Compared to the previous version, it significantly expands multilingual coverage and the number of bilingual aligned QA pairs to over 14.3M, making it the largest FAQ-based resource. Unlike the original release, WebFAQ 2.0 uses a novel data collection strategy that directly crawls and extracts relevant web content, resulting in a substantially more diverse and multilingual dataset with richer context through page titles and descriptions. In response to community feedback, we also release a hard negatives dataset for training dense retrievers, with 1.25M queries across 20 languages. These hard negatives were mined using a two-stage retrieval pipeline and include cross-encoder scores for 200 negatives per query. We further show how this resource enables two primary fine-tuning strategies for dense retrievers: Contrastive Learning with MultipleNegativesRanking loss, and Knowledge Distillation with MarginMSE loss. WebFAQ 2.0 is not a static resource but part of a long-term effort. Since late 2025, structured FAQs are being regularly released through the Open Web Index, enabling continuous expansion and refinement. We publish the datasets and training scripts to facilitate further research in multilingual and cross-lingual IR. The dataset itself and all related resources are publicly available on GitHub and HuggingFace.

Figures & tables

Figure 1. WebFAQ is a massive, publicly available repository of natural question-answer pairs. The second version focuses on QAs aligned across languages and hard negative mining. TODO

Language	%		Language	%
eng (English)	27.9	51.2	nld (Dutch)	4.3	2.8
spa (Spanish)	8.8	6.0	por (Portuguese)	4.0	1.7
deu (German)	7.3	6.9	ita (Italian)	3.5	2.7
fra (French)	6.5	4.8	pol (Polish)	2.5	1.7
rus (Russian)	4.4	3.8	Other	30.8	13.0

Table 1. Top 10 Languages in WebFAQ 2.0 (v1 in gray)

	%
Traveling and Hospitality	59.2	34.1
Products and Commercial Services	18.9	19.8
Healthcare Services, Wellness and Lifestyle	5.6	13.0
Entertainment, Recreation, and Leisure	4.7	9.7
Banking, Financial Services, and Insurance	4.4	6.0
General Information and Other	4.4	3.9

Table 2. Distribution of topics (v1 in gray)

Question Type	Count	%
Not-a-Question	67,870,478	34,7
Factoid	65,601,236	33,6
Experience	22,649,722	11,6
Instruction	20,179,307	10,3
Evidence-Based	8,300,537	4,3
Reason	6,503,638	3,3

Table 3. Distribution of Question Types in WebFAQ 2.0

Language Pair	v1		v2
Marathi–Telugu	0	$\rightarrow$	89,910
German–Spanish	19,739	$\rightarrow$	80,623
Russian–Ukrainian	15,251	$\rightarrow$	74,545
Italian–Portuguese	10,924	$\rightarrow$	69,469
Indonesian–Korean	1,515	$\rightarrow$	42,986

Table 4. Top language pairs for bilingual QA alignment.

	WebFAQ						MIRACL (Hard Negatives)						Mr. Tydi
	ara	eng	ind	jpn	kor	rus	ara	eng	ind	jpn	kor	rus	ara	eng	ind	jpn	kor	rus
BM25	29.7	24.4	36.1	29.0	29.6	21.2	53.2	31.6	51.6	44.7	37.6	29.8	43.4	19.9	50.5	25.0	24.8	29.7
Base	61.0	49.7	74.6	56.5	73.2	50.1	36.1	30.6	28.1	27.6	39.3	29.6	30.5	18.1	33.2	16.0	27.7	22.5
MNR (RN)	74.3	60.0	81.9	69.8	81.3	62.0	50.5	37.2	36.1	39.8	42.4	39.6	44.7	29.5	47.7	33.5	34.7	30.8
MNR Top4	71.8	48.7	78.8	63.6	81.4	51.4	40.3	23.8	26.6	31.4	35.3	27.6	34.1	13.0	36.9	25.1	26.9	19.2
MNR Den.	74.8	57.8	82.2	69.6	83.1	59.9	46.4	29.7	31.7	36.6	39.0	34.8	39.5	20.1	44.7	30.1	32.6	26.9

Table 5. Comparing retrieval performance on 3 multilingual datasets using NDCG@10 in %, including BM25 and XLM-RoBERTa-base with in-domain pre-training on MS MARCO ( Base ) as references. MNR (RN) represent the Base model fine-tuned on WebFAQ random negatives. The three bottom rows represent training with hard negatives. The top scores are marked in bold, while the second-best are underlined. Using the reranker scores directly in the loss function yields the best results ( M-MSE ).

Multilingual IR

Multilingual information retrieval (IR) finds and ranks relevant information across languages, scripts, and language-mismatched queries and documents.

See all 53 papers

Feb 16, 2026

The Wikidata Query Logs Dataset

We present the Wikidata Query Logs (WDQL) dataset, a dataset consisting of 335k question-query pairs over the Wikidata knowledge graph. It is over 11x larger than the largest existing Wikidata datasets of similar format without relying on template-generated queries. Instead, we construct it using real-world SPARQL queries sent to the Wikidata Query Service and generate questions for them. Since these log-based queries are anonymized, and therefore often do not produce results, a significant amount of effort is needed to convert them back into meaningful SPARQL queries. To achieve this, we present an agent-based method that iteratively de-anonymizes, cleans, and verifies queries against Wikidata while also generating corresponding natural-language questions. We demonstrate the benefit of this dataset for training question-answering methods. All WDQL assets, as well as the agent code, are publicly available via https://github.com/ad-freiburg/wikidata-query-logs under a permissive license.

Sebastian Walter, Hannah Bast

University of Freiburg, Department of Computer Science, Freiburg im Breisgau, Germany

Knowledge Graph Question Answering Knowledge Graphs Natural Language to SPARQL Question Answering

Ask PDF

Apr 22, 2026

RespondeoQA: a Benchmark for Bilingual Latin-English Question Answering

We introduce a benchmark dataset for question answering and translation in bilingual Latin and English settings, containing about 7,800 question-answer pairs. The questions are drawn from Latin pedagogical sources, including exams, quizbowl-style trivia, and textbooks ranging from the 1800s to the present. After automated extraction, cleaning, and manual review, the dataset covers a diverse range of question types: knowledge- and skill-based, multihop reasoning, constrained translation, and mixed language pairs. To our knowledge, this is the first QA benchmark centered on Latin. As a case study, we evaluate three large language models -- LLaMa 3, Qwen QwQ, and OpenAI's o3-mini -- finding that all perform worse on skill-oriented questions. Although the reasoning models perform better on scansion and literary-device tasks, they offer limited improvement overall. QwQ performs slightly better on questions asked in Latin, but LLaMa3 and o3-mini are more task dependent. This dataset provides a new resource for assessing model capabilities in a specialized linguistic and cultural domain, and the creation process can be easily adapted for other languages. The dataset is available at: https://github.com/slanglab/RespondeoQA

Marisa Hudspeth, Patrick J. Burns, Brendan O'Connor

Manning College of Information & Computer Sciences, University of Massachusetts Amherst · Institute for the Study of the Ancient World, New York University

Multilingual Language Model Evaluation Multilingual QA Low-Resource Language Processing Machine Translation

Ask PDF

Feb 27, 2026

LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering

Long-form question answering (LFQA) demands nuanced evaluation of multi-sentence explanatory responses, yet existing metrics often fail to reflect human judgment. We present LFQA-HP-1M, a large-scale dataset comprising 1.3M human pairwise preference annotations for LFQA. We propose nine rubrics for answer quality evaluation, and show that simple linear models based on these features perform comparably to state-of-the-art LLM evaluators. We further examine transitivity consistency, positional bias, and verbosity biases in LLM evaluators and demonstrate their vulnerability to adversarial perturbations. Overall, this work provides one of the largest public LFQA preference datasets and a rubric-driven framework for transparent and reliable evaluation.

Rafid Ishrak Jahan, Fahmid Shahriar Iqbal, Sagnik Ray Choudhury

University of North Texas, Denton, Texas, USA

LLM Evaluation LLM-as-a-Judge Bias in LLM-as-a-Judge

Ask PDF

Abstract

Multilingual IR

Explore similar work

The Wikidata Query Logs Dataset

RespondeoQA: a Benchmark for Bilingual Latin-English Question Answering

LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering