Many LLM benchmarks for languages other than English are translated, multiple-choice, or difficult to score automatically. We introduce The Shared Word (TSW-NL), a Dutch benchmark built around four core principles: it is language-specific, open-vocabulary, human-curated, and deterministically scored. Each puzzle gives three clues that describe different senses or uses of the same Dutch word. The clues are individually ambiguous, requiring the solver to combine evidence across multiple clues. We construct the benchmark from Dutch Wikipedia disambiguation pages, followed by automatic filtering and extensive manual curation. The final dataset contains 222 instances, with three independent human answers per test item. The task is clearly challenging: scores in our human study range from 49.5% to 61.0% and the highest-scoring open-weight model we tested reaches only 39.3%. Many others even score below 20%. At the same time, closed-source models show great progress: GPT-4.1 (April 2025) scores 59.2%, whereas GPT-5.6 Sol with max reasoning scores 88.8%. We believe this task provides a useful addition to Dutch model evaluation, plus a clear recipe that can be applied to construct a similar benchmark for other languages.
Figures & tables
Figure 1: Example puzzle. The three Dutch clues describe a football coach (Dick Advocaat), an alcoholic drink (Dutch-specific, similar to eggnog), and a profession (lawyer), all answered by advocaat .
Stage
Left
Removed
Operation
Automatic collection and filtering
Dutch Wikipedia disamb. pages
97,707
0
Retrieve candidate answer titles.
At least three linked pages
48,518
49,189
Require enough links for three clues.
Title in Open Dutch WordNet
5,723
42,795
Prefer lexical answers over proper-name-only pages.
Title has at least four characters
5,075
648
Remove short abbreviations and acronyms.
At least three described linked pages
4,159
916
Require three links with descriptions and categories.
Table 1: Data reduction during automatic filtering and manual annotation.
Model
Size
Type
Reference
Fietje 2 Instruct
2.7B
Dutch
Vanroy,2025
Fietje 2 Chat
2.7B
Dutch
Vanroy,2025
GEITje 7B Ultra
7B
Dutch
Vanroy,2024
ChocoLlama 3 8B Instruct
8B
Dutch
Meeus et al.,2024
Mistral 7B Instruct v0.1
7B
Multilingual
Jiang et al.,2023a
Llama 3 8B Instruct
8B
Multilingual
Grattafiori et al.,2024
Table 2: Models included in the evaluation.
System
Acc.
SD
Human annotators
54.0
—
Dutch models
Fietje 2 Instruct
7.0
1.0
Fietje 2 Chat
9.3
0.8
GEITje 7B Ultra
14.8
2.1
ChocoLlama 3 8B Instruct
15.0
1.7
Table 3: Test accuracy (%), averaged over the six permutations of the three clues. SD is the sample standard deviation of the six order-level accuracies.
Figure 2: Model accuracy by number of clues. For clarity, only a few models are highlighted in color. The other models are shown in grey. One- and two-clue scores average over the three corresponding clue subsets.
Model
0/3
1/3
2/3
3/3
Fietje 2 Instruct
4.3
7.4
5.0
10.5
Fietje 2 Chat
4.7
11.3
6.6
13.5
GEITje 7B Ultra
5.0
8.9
17.0
25.1
ChocoLlama 3 8B Instruct
5.4
12.1
15.1
24.6
Mistral 7B Instruct v0.1
3.5
6.4
1.9
8.2
Llama 3 8B Instruct
10.1
12.8
11.0
21.3
Table 4: Accuracy (%) by the number of correct human judgments, averaged over six clue orders. The four groups contain 43, 47, 53, and 57 items.
Category
Fietje
Luna
Sol
Human
Empty
1
17
7
47
Completely incorrect
21
0
0
0
Repeats (part of) a clue
41
0
0
0
Fits only one clue
90
7
3
15
Fits only two clues
35
16
13
16
Inflection variant
0
4
2
0
Table 5: Manual categories for answers initially marked incorrect in zero-shot canonical-order Fietje 2 Instruct, GPT-5.6 runs with max reasoning, and one full-test human annotator.
Feature
Models
Human
Answer frequency
.45
.34
Average clue frequency
.10
.19
Minimum clue frequency
−.07
.01
Average clue length
.11
.00
Answer length
−.13
−.14
Named-entity clues
−.03
−.01
Table 6: Item-level Spearman correlations with mean model accuracy and human success. Bold values are significant at p<.05 .
Figure 3: Number of test instances solved by at least the number of model conditions on the horizontal axis. A model condition solves an instance if it produces an accepted answer in at least one of the six clue orders. Counts are cumulative, so an instance solved by, for example, ten models also contributes to every point from one to ten.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Language
Brief description
SL
OV
HV
DE
TSW-NL (ours)
Dutch
Three clues identify one ambiguous Dutch word
✓
✓
✓
✓
JAWAHER ( Magdy et al., 2025 )
Arabic/ English
Multidialectal proverb translation and explanation (0/3 tasks pass)
∼
✓
✓
×
Nahw ( Mubarak et al., 2026 )
Arabic
Grammar analysis, correction, and explanation (2/4 tasks pass)
✓
∼
✓
∼
ALARB ( Abu Shairah et al., 2025 )
Arabic
Legal verdict prediction and statute identification (0/6 tasks pass)
✓
∼
×
∼
SAHM ( Elbadry et al., 2026 )
Arabic
Financial and Shari’ah-compliant reasoning (0/7 tasks pass)
✓
∼
✓
∼
BALSAM ( Al-Matham et al., 2025 )
Arabic
Arabic understanding and generation platform (0/78 tasks pass)
∼
∼
×
∼
Appendix
Table 7: Recent LLM benchmarks developed for non-English languages, part 1. SL : created for one target language rather than translated from another benchmark or evaluated in parallel target-language and English versions. OV : the task requires generating an answer rather than selecting from fixed candidates. HV : construction includes substantive human checking, curation and validation. DE : answers are deterministically evaluated using a fixed answer or rule without needing an LLM judge, human rating, or fuzzy metric. For multi-task benchmarks, ✓ , ∼ , and × mean that all, some, or none of the constituent tasks pass, respectively.
Benchmark
Language
Brief description
SL
OV
HV
DE
ITALIC ( Seveso et al., 2025 )
Italian
Italian culture, history, civic knowledge, and language questions from public examinations (0/12 tasks pass)
✓
×
✓
✓
SOBACO ( Yamamoto et al., 2025 )
Japanese
Paired questions testing Japanese age, gender, hierarchy, and workplace norms
✓
×
✓
✓
JPharmaBench ( Ono et al., 2025 )
Japanese
Japanese pharmaceutical licensing, terminology normalization, and consistency checking (0/3 tasks pass)
∼
×
✓
✓
Japanese Financial Benchmark ( Hirano, 2024 )
Japanese
Japanese financial sentiment, knowledge, audit, and professional-exam tasks (0/5 tasks pass)
✓
×
×
✓
JMedBench ( Jiang et al., 2025 )
Japanese
Japanese biomedical MCQA, translation, NER, classification, and similarity tasks (0/5 tasks pass)
Table 9: Dutch tasks in EuroEval, DUMB, and BEIR-NL, compared with TSW-NL . BEIR-NL is translated automatically from the English BEIR datasets, ranks documents from a fixed corpus, reports deterministic retrieval metrics, and does not include substantive human verification of its Dutch construction. Column definitions and symbols follow Table 7 .
Clue 1
Clue 2
Clue 3
Answer
Een plant
Een gemeenschap van mensen
Een term uit taalkunde
Stam
A plant
A community of people
A linguistic term
Een Nederlandse televisiepresentator
Vuur
Een Nederlandse brouwerij
Brand
A Dutch television presenter
Fire
A Dutch brewery
Een kleur
Een strafkaart
Een politieke richting
Rood
A colour
A penalty card
A political orientation
Appendix
Table 10: Ten example test items. English translations are shown in italics and are not part of the benchmark input.
Clue 1
Clue 2
Clue 3
Gold
Alternative
Who
Het terrein van beroepsmatig toegepaste kennis
Een term in sport
Het geheel van arbeidstaken
Discipline
Vak
Luna, Sol, human
A field of professionally applied knowledge
A term in sport
The set of work duties
Why accept? Vak perfectly fits clues 1 and 3. The second one is too vague to really rule it out: vak is also a term in sport.
Een automodel
Een type vliegtuig
Een wild dier
Eend
Jaguar
Luna
A car model
A type of aircraft
A wild animal
Why accept? Jaguar even fits the first and third clues better than Eend and both are relatively obscure aircraft.
Appendix
Table 11: Items for which a model or human annotator produced a correct alternative to the gold answer. These alternatives are included in the accepted-answer sets used for all reported scores.
Figure 4: Test accuracy for all models included in the evaluation, averaged over the six permutations of the three clues. Dots show the mean accuracy and lines show 95% confidence intervals obtained by bootstrapping the 200 test items while keeping the six clue orders for each item together.
Model
1
2
3
Fietje 2 Instruct
2.0
5.3
6.0
Fietje 2 Chat
2.7
7.0
9.5
GEITje 7B Ultra
3.7
9.5
15.0
ChocoLlama 3 8B Instruct
4.3
9.2
15.5
Mistral 7B Instruct v0.1
2.2
3.2
3.5
Llama 3 8B Instruct
5.5
11.7
19.5
Appendix
Table 12: Accuracy (%) by number of clues. Column 1 is the average of the single-clue conditions 1, 2, and 3. Column 2 averages the two-clue conditions 1+2, 1+3, and 2+3. Finally, column 3 uses all three clues in canonical order.
Model
0
1
3
5
10
22
Fietje 2 Instruct
6.0
7.0
7.0
6.0
6.5
3.5
Fietje 2 Chat
9.5
7.5
7.0
6.5
6.0
8.5
GEITje 7B Ultra
15.0
11.0
16.5
15.0
17.5
15.5
ChocoLlama 3 8B Instruct
15.5
15.0
17.0
17.5
16.5
16.5
Mistral 7B Instruct v0.1
3.5
10.5
9.5
10.0
11.0
11.0
Llama 3 8B Instruct
19.5
16.5
18.5
19.0
20.5
19.0
Appendix
Table 13: Accuracy (%) for a subset of models in one canonical-order run under each shot condition. Best result per model is shown in bold.
Feature
All
Open
Prop.
Human
Answer frequency
.45 ( <.001 )
.44 ( <.001 )
.42 ( <.001 )
.34 ( <.001 )
Average clue frequency
.10 (.168)
.05 (.461)
.17 (.015)
.19 (.006)
Minimum clue frequency
−.07 (.329)
−.10 (.168)
.05 (.480)
.01 (.846)
Average clue length
.11 (.117)
.12 (.099)
.10 (.160)
.00 (.945)
Answer length
−.13 (.067)
−.08 (.283)
−.17 (.014)
−.14 (.052)
Named-entity clues
−.03 (.637)
−.05 (.449)
.03 (.707)
−.01 (.880)
Appendix
Table 14: Item-level Spearman correlations. Each cell shows ρ , with the two-sided p -value in parentheses. All denotes mean accuracy across the 27 model conditions, while Open and Prop. give the corresponding open-weight and proprietary means.
We create high-quality datasets for LLM evaluation of logical reasoning skills across nine different languages, which have been manually checked by fluent speakers. The datasets consist of so-called zebra puzzles, and we analyse different ways of tuning the difficulty of the puzzles to fit modern LLMs. This includes the size of the puzzle (number of objects and number of clues), as well as a novel addition of red herring clues containing only irrelevant information. We show that presence of red herrings indeed makes the puzzles significantly harder for the models, and we find puzzle sizes 2x3 and 4x5 are sufficiently challenging for GPT-4o mini (a non-reasoning model) and o3-mini (a reasoning model), respectively. We analyse whether LLM performance of these are sensitive to the language, the cultural sensitivity of the puzzle theme, and the choice of clue types. These analyses are conducted with English and Danish, where we show that there is no significant difference for either of these three aspects, at least for the OpenAI models GPT-4o mini and o3-mini, chosen as representative non-reasoning and reasoning models, respectively. We publish the datasets for each of the nine languages for the identified sizes 2x3 and 4x5. We also publish the code used to generate the puzzles, which can be used to extend the benchmark into more languages.
Sofie Helene Bruun, Dan Saattrup Smart
The Alexandra Institute Rued Langgaards Vej 7, 5D, 2300 Copenhagen S, Denmark
This paper presents ltzGLUE, the first Natural Language Understanding (NLU) benchmark for Luxembourgish (LTZ) based on the popular GLUE benchmark for English. Although NLU tasks are available for many European languages nowadays, LTZ is one of the official national languages that is often overlooked. We construct new tasks and reuse existing ones to introduce the first official NLU benchmark and accompanying evaluation of encoder models for the language. Our tasks include common natural language processing tasks in binary and multi-class classification settings, including named entity recognition, topic classification, and intent classification. We evaluate various pre-trained language models for LTZ to present an overview of the current capabilities of these models on the LTZ language.
The output of a Language Model can be tampered with while the model is writing it. A simple test can thus be constructed by evaluating the model's perception of this external perturbation. In this spirit, a simple benchmark is built in which a single word is consistently substituted with another in the generation process. We call this method Sleight of Word. Two distinct axes are measured: metrics that relate to the model's surprise, as well as an evaluation of the textual reaction for 19 different open-weight language models.