Many LLM benchmarks for languages other than English are translated, multiple-choice, or difficult to score automatically. We introduce The Shared Word (TSW-NL), a Dutch benchmark built around four core principles: it is language-specific, open-vocabulary, human-curated, and deterministically scored. Each puzzle gives three clues that describe different senses or uses of the same Dutch word. The clues are individually ambiguous, requiring the solver to combine evidence across multiple clues. We construct the benchmark from Dutch Wikipedia disambiguation pages, followed by automatic filtering and extensive manual curation. The final dataset contains 222 instances, with three independent human answers per test item. The task is clearly challenging: scores in our human study range from 49.5% to 61.0% and the highest-scoring open-weight model we tested reaches only 39.3%. Many others even score below 20%. At the same time, closed-source models show great progress: GPT-4.1 (April 2025) scores 59.2%, whereas GPT-5.6 Sol with max reasoning scores 88.8%. We believe this task provides a useful addition to Dutch model evaluation, plus a clear recipe that can be applied to construct a similar benchmark for other languages.
Figures & tables
Figure 1: Example puzzle. The three Dutch clues describe a football coach (Dick Advocaat), an alcoholic drink (Dutch-specific, similar to eggnog), and a profession (lawyer), all answered by advocaat .
Stage
Left
Removed
Operation
Automatic collection and filtering
Dutch Wikipedia disamb. pages
97,707
0
Retrieve candidate answer titles.
At least three linked pages
48,518
49,189
Require enough links for three clues.
Title in Open Dutch WordNet
5,723
42,795
Prefer lexical answers over proper-name-only pages.
Title has at least four characters
5,075
648
Remove short abbreviations and acronyms.
At least three described linked pages
4,159
916
Require three links with descriptions and categories.
Table 1: Data reduction during automatic filtering and manual annotation.
Model
Size
Type
Reference
Fietje 2 Instruct
2.7B
Dutch
Vanroy,2025
Fietje 2 Chat
2.7B
Dutch
Vanroy,2025
GEITje 7B Ultra
7B
Dutch
Vanroy,2024
ChocoLlama 3 8B Instruct
8B
Dutch
Meeus et al.,2024
Mistral 7B Instruct v0.1
7B
Multilingual
Jiang et al.,2023a
Llama 3 8B Instruct
8B
Multilingual
Grattafiori et al.,2024
Table 2: Models included in the evaluation.
System
Acc.
SD
Human annotators
54.0
—
Dutch models
Fietje 2 Instruct
7.0
1.0
Fietje 2 Chat
9.3
0.8
GEITje 7B Ultra
14.8
2.1
ChocoLlama 3 8B Instruct
15.0
1.7
Table 3: Test accuracy (%), averaged over the six permutations of the three clues. SD is the sample standard deviation of the six order-level accuracies.
Figure 2: Model accuracy by number of clues. For clarity, only a few models are highlighted in color. The other models are shown in grey. One- and two-clue scores average over the three corresponding clue subsets.
Model
0/3
1/3
2/3
3/3
Fietje 2 Instruct
4.3
7.4
5.0
10.5
Fietje 2 Chat
4.7
11.3
6.6
13.5
GEITje 7B Ultra
5.0
8.9
17.0
25.1
ChocoLlama 3 8B Instruct
5.4
12.1
15.1
24.6
Mistral 7B Instruct v0.1
3.5
6.4
1.9
8.2
Llama 3 8B Instruct
10.1
12.8
11.0
21.3
Table 4: Accuracy (%) by the number of correct human judgments, averaged over six clue orders. The four groups contain 43, 47, 53, and 57 items.
Category
Fietje
Luna
Sol
Human
Empty
1
17
7
47
Completely incorrect
21
0
0
0
Repeats (part of) a clue
41
0
0
0
Fits only one clue
90
7
3
15
Fits only two clues
35
16
13
16
Inflection variant
0
4
2
0
Table 5: Manual categories for answers initially marked incorrect in zero-shot canonical-order Fietje 2 Instruct, GPT-5.6 runs with max reasoning, and one full-test human annotator.
Feature
Models
Human
Answer frequency
.45
.34
Average clue frequency
.10
.19
Minimum clue frequency
−.07
.01
Average clue length
.11
.00
Answer length
−.13
−.14
Named-entity clues
−.03
−.01
Table 6: Item-level Spearman correlations with mean model accuracy and human success. Bold values are significant at p<.05 .
Figure 3: Number of test instances solved by at least the number of model conditions on the horizontal axis. A model condition solves an instance if it produces an accepted answer in at least one of the six clue orders. Counts are cumulative, so an instance solved by, for example, ten models also contributes to every point from one to ten.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Language
Brief description
SL
OV
HV
DE
TSW-NL (ours)
Dutch
Three clues identify one ambiguous Dutch word
✓
✓
✓
✓
JAWAHER ( Magdy et al., 2025 )
Arabic/ English
Multidialectal proverb translation and explanation (0/3 tasks pass)
∼
✓
✓
×
Nahw ( Mubarak et al., 2026 )
Arabic
Grammar analysis, correction, and explanation (2/4 tasks pass)
✓
∼
✓
∼
ALARB ( Abu Shairah et al., 2025 )
Arabic
Legal verdict prediction and statute identification (0/6 tasks pass)
✓
∼
×
∼
SAHM ( Elbadry et al., 2026 )
Arabic
Financial and Shari’ah-compliant reasoning (0/7 tasks pass)
✓
∼
✓
∼
BALSAM ( Al-Matham et al., 2025 )
Arabic
Arabic understanding and generation platform (0/78 tasks pass)
∼
∼
×
∼
Appendix
Table 7: Recent LLM benchmarks developed for non-English languages, part 1. SL : created for one target language rather than translated from another benchmark or evaluated in parallel target-language and English versions. OV : the task requires generating an answer rather than selecting from fixed candidates. HV : construction includes substantive human checking, curation and validation. DE : answers are deterministically evaluated using a fixed answer or rule without needing an LLM judge, human rating, or fuzzy metric. For multi-task benchmarks, ✓ , ∼ , and × mean that all, some, or none of the constituent tasks pass, respectively.
Benchmark
Language
Brief description
SL
OV
HV
DE
ITALIC ( Seveso et al., 2025 )
Italian
Italian culture, history, civic knowledge, and language questions from public examinations (0/12 tasks pass)
✓
×
✓
✓
SOBACO ( Yamamoto et al., 2025 )
Japanese
Paired questions testing Japanese age, gender, hierarchy, and workplace norms
✓
×
✓
✓
JPharmaBench ( Ono et al., 2025 )
Japanese
Japanese pharmaceutical licensing, terminology normalization, and consistency checking (0/3 tasks pass)
∼
×
✓
✓
Japanese Financial Benchmark ( Hirano, 2024 )
Japanese
Japanese financial sentiment, knowledge, audit, and professional-exam tasks (0/5 tasks pass)
✓
×
×
✓
JMedBench ( Jiang et al., 2025 )
Japanese
Japanese biomedical MCQA, translation, NER, classification, and similarity tasks (0/5 tasks pass)
Table 9: Dutch tasks in EuroEval, DUMB, and BEIR-NL, compared with TSW-NL . BEIR-NL is translated automatically from the English BEIR datasets, ranks documents from a fixed corpus, reports deterministic retrieval metrics, and does not include substantive human verification of its Dutch construction. Column definitions and symbols follow Table 7 .
Clue 1
Clue 2
Clue 3
Answer
Een plant
Een gemeenschap van mensen
Een term uit taalkunde
Stam
A plant
A community of people
A linguistic term
Een Nederlandse televisiepresentator
Vuur
Een Nederlandse brouwerij
Brand
A Dutch television presenter
Fire
A Dutch brewery
Een kleur
Een strafkaart
Een politieke richting
Rood
A colour
A penalty card
A political orientation
Appendix
Table 10: Ten example test items. English translations are shown in italics and are not part of the benchmark input.
Clue 1
Clue 2
Clue 3
Gold
Alternative
Who
Het terrein van beroepsmatig toegepaste kennis
Een term in sport
Het geheel van arbeidstaken
Discipline
Vak
Luna, Sol, human
A field of professionally applied knowledge
A term in sport
The set of work duties
Why accept? Vak perfectly fits clues 1 and 3. The second one is too vague to really rule it out: vak is also a term in sport.
Een automodel
Een type vliegtuig
Een wild dier
Eend
Jaguar
Luna
A car model
A type of aircraft
A wild animal
Why accept? Jaguar even fits the first and third clues better than Eend and both are relatively obscure aircraft.
Appendix
Table 11: Items for which a model or human annotator produced a correct alternative to the gold answer. These alternatives are included in the accepted-answer sets used for all reported scores.
Figure 4: Test accuracy for all models included in the evaluation, averaged over the six permutations of the three clues. Dots show the mean accuracy and lines show 95% confidence intervals obtained by bootstrapping the 200 test items while keeping the six clue orders for each item together.
Model
1
2
3
Fietje 2 Instruct
2.0
5.3
6.0
Fietje 2 Chat
2.7
7.0
9.5
GEITje 7B Ultra
3.7
9.5
15.0
ChocoLlama 3 8B Instruct
4.3
9.2
15.5
Mistral 7B Instruct v0.1
2.2
3.2
3.5
Llama 3 8B Instruct
5.5
11.7
19.5
Appendix
Table 12: Accuracy (%) by number of clues. Column 1 is the average of the single-clue conditions 1, 2, and 3. Column 2 averages the two-clue conditions 1+2, 1+3, and 2+3. Finally, column 3 uses all three clues in canonical order.
Model
0
1
3
5
10
22
Fietje 2 Instruct
6.0
7.0
7.0
6.0
6.5
3.5
Fietje 2 Chat
9.5
7.5
7.0
6.5
6.0
8.5
GEITje 7B Ultra
15.0
11.0
16.5
15.0
17.5
15.5
ChocoLlama 3 8B Instruct
15.5
15.0
17.0
17.5
16.5
16.5
Mistral 7B Instruct v0.1
3.5
10.5
9.5
10.0
11.0
11.0
Llama 3 8B Instruct
19.5
16.5
18.5
19.0
20.5
19.0
Appendix
Table 13: Accuracy (%) for a subset of models in one canonical-order run under each shot condition. Best result per model is shown in bold.
Feature
All
Open
Prop.
Human
Answer frequency
.45 ( <.001 )
.44 ( <.001 )
.42 ( <.001 )
.34 ( <.001 )
Average clue frequency
.10 (.168)
.05 (.461)
.17 (.015)
.19 (.006)
Minimum clue frequency
−.07 (.329)
−.10 (.168)
.05 (.480)
.01 (.846)
Average clue length
.11 (.117)
.12 (.099)
.10 (.160)
.00 (.945)
Answer length
−.13 (.067)
−.08 (.283)
−.17 (.014)
−.14 (.052)
Named-entity clues
−.03 (.637)
−.05 (.449)
.03 (.707)
−.01 (.880)
Appendix
Table 14: Item-level Spearman correlations. Each cell shows ρ , with the two-sided p -value in parentheses. All denotes mean accuracy across the 27 model conditions, while Open and Prop. give the corresponding open-weight and proprietary means.