When several answers fit a riddle, exact-match scoring can penalise a valid response. CulturalCrosswords tests how answer constraints affect large language models (LLMs) solving cultural riddles, whose solutions draw on knowledge specific to a community. We constructed 53 crosswords from crowdsourced riddles in 42 languages and evaluated 11 LLM configurations on five versions of each clue set: plain batches, batches with lengths, batches with lengths and partial-answer hints, and crosswords requesting an answer list or a completed grid. On 962 clues with alternative answers, lengths improve agreement with the reference by 22.8 percentage points, compared with 8.4 points when accepted alternatives also receive credit. Part of the gain therefore appears to come from selecting a required answer form. Crossword input increases the share of all crossings with compatible answers by 9.7 points; requiring a completed grid reveals large differences between models. The benchmark distinguishes agreement with a reference from the ability to produce answers that fit together and occupy the correct cells.
Figures & tables
Condition
Short name
Information and required response
A
Plain
Full clue batch; answer list.
B
Length
A plus required answer lengths; answer list.
C
Hints
B plus correct units at one quarter of each answer’s positions, rounded down ( ⌊ℓi/4⌋ ); answer list.
D
Grid
Blank grid and clues with start coordinates and directions; completed grid. Lengths are inferred from the grid.
E
Crossword/list
Blank grid, entry paths, explicit lengths and clues; answer list.
Table 1: Task conditions. Plain batches retain the shared language and answer conventions; they omit lengths, hints and crossings.
Figure 1: A Japanese crossing. Both candidates occupy four katakana cells. Solving the school-shoes clue fixes the third Across unit and excludes the Mount Fuji form. Relative source-grid positions are preserved; clues are shortened English glosses. This illustrates compatibility given a correct crossing, rather than observed model reasoning. An incorrect crossing can propagate an error.
Figure 2: Score increases on the same 962 clues. Blue credits reference answers only; green also credits listed alternatives. Each row adds information while retaining answer-list output. Models are averaged within each of 38 equally weighted puzzles; bars show 95% language-cluster intervals. The right panel excludes Opus 4.7, Opus 5.5 and Fable 5.1, which received recovery reruns.
Measure
B
E
E–B [95% CI]
Reference agreement
70.1
72.9
+2.8 [1.3, 4.3]
Checkable crossings
81.2
74.8
-6.4 [-9.0, -3.9]
Consistent / all crossings
61.8
71.5
+9.7 [7.5, 11.9]
Table 2: Crossing analysis across all 11 models and 53 puzzles. Values are equally weighted puzzle–model percentages; changes are percentage points. Intervals resample 42 language clusters. A crossing is checkable only when both complete answers map unambiguously to the required number of cells.
Figure 3: Reference-agreement change from lengths to crossword input with answer-list output (E–B), in percentage points. Teal circles use all 53 puzzles, including recoveries. Orange squares exclude puzzles with a response selected for recovery in either condition; n is the number remaining. Bars show 95% language-cluster intervals with equal puzzle weights. Very small subsets are descriptive only.
Figure 4: Reference agreement, weighting 53 puzzles equally. A–B–C adds answer information; E supplies crossword structure; D requires placement. These are discrete tasks, not a continuous constraint scale. Missing answers remain in the denominator. Hollow markers identify means containing records selected for recovery.
Figure 5: Writing-system profiles with equal puzzle weights within groups and equal model weights within panels. Bars show language-cluster intervals. Alphabetic, abjad and abugida groups contain 27, four and ten language labels. Abjad intervals are descriptive because there are only four language clusters; Japanese has one puzzle and no population interval. D requires placement. Eight-model panels exclude Opus 4.7, Opus 5.5 and Fable 5.1, which received recovery reruns. C and E supply different information.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Display name
Recorded identifier
GPT-6 Astra (Astra)
gpt-6-astra
GPT-6.1 Sol (Sol)
gpt-6.1-sol
GPT-5.5
gpt-5.5
Gemini 4 Argon (Argon)
google/gemini-4-argon
Gemini 3.8 Flash
google/gemini-3.8-flash
Gemini 3.7 Flash
google/gemini-3.7-flash
Appendix
Table 3: Display names and recorded identifiers.
Model
A
B
C
E
D
Opus 5.5
0
5
16
43
51
Fable 5.1
8
20
31
45
51
Opus 4.7
6
38
46
39
43
Opus 4.8
0
0
0
0
0
Total / 212
14
63
93
127
145
Appendix
Table 4: Original Claude results with at least one intermediate limit stop, out of 53 per cell. Counts precede recovery and measure stop incidence rather than tokens consumed.
Model
Full: alternative/gains
Excluded: alternative/gains
Retained clues
GPT-5.5
129/272
129/272
962
GPT-6 Astra
138/293
138/293
962
GPT-6.1 Sol
140/287
140/287
962
Gemini 3.1 Pro
56/163
56/163
962
Gemini 3.7 Flash
107/237
107/237
962
Gemini 3.8 Flash
112/284
112/284
962
Appendix
Table 5: Fractions show transitions from a listed alternative to the reference divided by all transitions to the reference after adding lengths. Full samples contain 962 clues per model; exclusions remove originally flagged pairs. Losses remain in the archived data.
Model
A
B
C
E
Claude Opus 5.5
28.3 → 52.9
29.3 → 76.6
31.4 → 85.2
21.6 → 82.6
Claude Fable 5.1
46.6 → 54.6
65.4 → 77.8
73.4 → 86.1
61.2 → 84.1
Claude Opus 4.7
26.5 → 46.4
18.7 → 65.8
23.3 → 75.3
22.7 → 58.8
Appendix
Table 6: Original and recovery-integrated means for the three affected models (percent reference agreement). Each cell shows original → integrated; D is unchanged. The same 53 puzzles are retained.
Reference-agreement contrast
All 11
Without Opus 4.8
Omission range
Lengths: B–A
16.9 [14.2, 20.0]
18.5 [15.5, 21.9]
16.2–18.5
Hints: C–B
8.4 [6.5, 10.1]
8.8 [6.8, 10.7]
7.8–8.8
Crossword information: E–B
2.8 [1.3, 4.3]
4.2 [2.5, 5.8]
2.2–4.2
Answer list over grid: E–D
26.6 [24.3, 28.6]
26.8 [24.6, 28.6]
22.2–29.1
Appendix
Table 7: Model-omission sensitivity (percentage points). Brackets are pointwise 95% language-cluster intervals. The last column spans mean effects when each model is omitted in turn; it is not an interval. All four intervals remain above zero in every omission. The same 53 puzzles are used throughout.
Puzzle characteristic
All 11
Fixed eight
Without Opus 4.8
Ten more clues
-1.00 [-4.32, 1.65]
0.33 [-3.62, 3.56]
-0.77 [-4.47, 2.22]
One more answer unit
0.43 [-0.87, 1.57]
0.98 [-0.83, 2.50]
0.25 [-1.16, 1.57]
0.10 more crossing density
2.16 [-3.78, 7.22]
5.95 [-2.18, 12.88]
1.57 [-5.13, 7.41]
R2
0.032
0.038
0.017
Appendix
Table 8: Conditional associations with the puzzle-level E–B reference-agreement gain (percentage points). Each fit uses 53 equally weighted puzzles and all three predictors. Brackets give pointwise 95% intervals from 10,000 language-cluster bootstrap draws. Fixed eight excludes the three configurations selected for recovery. Answer length and crossing density are strongly correlated; these are descriptive associations, not causal effects.
Model
A
B
C
D
E
GPT-6.1 Sol
60.2
80.2
87.4
85.0
88.1
GPT-6 Astra
60.7
81.7
89.0
88.3
89.9
GPT-5.5
57.9
76.3
85.3
64.0
83.2
Gemini 4 Argon
59.8
83.1
90.0
77.4
91.1
Gemini 3.8 Flash
56.1
70.6
83.5
9.0
67.6
Gemini 3.7 Flash
53.9
66.5
80.3
5.4
75.6
Appendix
Table 9: Reference agreement (percent), weighting 53 puzzles equally and retaining missing answers in the denominator. D requires correct grid placement. Selected answer-list records for Opus 5.5, Fable 5.1 and Opus 4.7 use recovery attempts with explicit inline-output instructions.
Figure 6: Reference matches, fully specified non-reference answers, and incomplete or missing answers, using all expected clues as the denominator. Non-reference answers may be semantically acceptable. Selected list records use recovery attempts, and missing answers receive no credit.
Figure 7: Model profiles across the five task formats, using reference agreement averaged equally over 53 puzzles. All panels use the same 0–100% scale and A–B–C–D–E axis order. D also requires correct placement; missing answers remain in the denominator. Corresponding spokes show each format’s score; polygon area has no performance interpretation. Uncertainty intervals are omitted. Asterisks mark models with selected list responses replaced by recovery attempts under revised output instructions. Opus 4.8’s zero in D reflects noncompletion under this protocol.
As LLMs expand their language coverage, evaluating how well they understand and represent diverse cultures becomes increasingly important. However, many benchmarks remain English-centric, with limited cultural coverage and a focus on factual knowledge over its application in real-world cultural contexts. We propose CulturalRiddles, a benchmark of original, culturally grounded riddles created entirely by humans through a participatory community effort, spanning 62 cultures (45 countries, 51 languages). These riddles are deeply rooted in their respective cultures and require abductive and compositional reasoning, alongside detailed factual and social knowledge, language skills, and an understanding of culturally specific associations. We benchmark 28 LLM configurations using both automatic and large-scale human evaluation. Even frontier models achieve at most 22% accuracy in some communities. We publicly release all artifacts alongside an open contribution process to expand the benchmark's linguistic and cultural coverage.
My Chiffon Nguyen, Tianyi Hu, Vinod Anbalagan +161
Multilingual fluency often invites a stronger assumption: a model that can speak a user's language must also understand the culture encoded by that language. We call this the Illusion of Cultural Alignment. To test this assumption directly, we introduce MSQA, a benchmark of 1,064 natively sourced questions across 11 language groups, five cultural dimensions, and three difficulty tiers. Unlike translated benchmarks, MSQA targets locally grounded knowledge and reduces shortcuts from English-centric cross-lingual transfer. Evaluating 18 LLMs, we find substantial cultural degradation and a pronounced Locality Effect: cultural competence tracks pre-training exposure more closely than general reasoning ability. We further show that common inference-time remedies do not dissolve the illusion. Models remain overconfident on unfamiliar cultural questions, repeated sampling yields unstable rather than reliable correctness, and retrieval augmentation helps unevenly on long-tail facts. These findings indicate that cultural alignment cannot be inferred from multilingual ability alone and requires deeper intervention than calibration, sampling, or retrieval at inference time
Many LLM benchmarks for languages other than English are translated, multiple-choice, or difficult to score automatically. We introduce The Shared Word (TSW-NL), a Dutch benchmark built around four core principles: it is language-specific, open-vocabulary, human-curated, and deterministically scored. Each puzzle gives three clues that describe different senses or uses of the same Dutch word. The clues are individually ambiguous, requiring the solver to combine evidence across multiple clues. We construct the benchmark from Dutch Wikipedia disambiguation pages, followed by automatic filtering and extensive manual curation. The final dataset contains 222 instances, with three independent human answers per test item. The task is clearly challenging: scores in our human study range from 49.5% to 61.0% and the highest-scoring open-weight model we tested reaches only 39.3%. Many others even score below 20%. At the same time, closed-source models show great progress: GPT-4.1 (April 2025) scores 59.2%, whereas GPT-5.6 Sol with max reasoning scores 88.8%. We believe this task provides a useful addition to Dutch model evaluation, plus a clear recipe that can be applied to construct a similar benchmark for other languages.