When several answers fit a riddle, exact-match scoring can penalise a valid response. CulturalCrosswords tests how answer constraints affect large language models (LLMs) solving cultural riddles, whose solutions draw on knowledge specific to a community. We constructed 53 crosswords from crowdsourced riddles in 42 languages and evaluated 11 LLM configurations on five versions of each clue set: plain batches, batches with lengths, batches with lengths and partial-answer hints, and crosswords requesting an answer list or a completed grid. On 962 clues with alternative answers, lengths improve agreement with the reference by 22.8 percentage points, compared with 8.4 points when accepted alternatives also receive credit. Part of the gain therefore appears to come from selecting a required answer form. Crossword input increases the share of all crossings with compatible answers by 9.7 points; requiring a completed grid reveals large differences between models. The benchmark distinguishes agreement with a reference from the ability to produce answers that fit together and occupy the correct cells.
Figures & tables
Condition
Short name
Information and required response
A
Plain
Full clue batch; answer list.
B
Length
A plus required answer lengths; answer list.
C
Hints
B plus correct units at one quarter of each answer’s positions, rounded down ( ⌊ℓi/4⌋ ); answer list.
D
Grid
Blank grid and clues with start coordinates and directions; completed grid. Lengths are inferred from the grid.
E
Crossword/list
Blank grid, entry paths, explicit lengths and clues; answer list.
Table 1: Task conditions. Plain batches retain the shared language and answer conventions; they omit lengths, hints and crossings.
Figure 1: A Japanese crossing. Both candidates occupy four katakana cells. Solving the school-shoes clue fixes the third Across unit and excludes the Mount Fuji form. Relative source-grid positions are preserved; clues are shortened English glosses. This illustrates compatibility given a correct crossing, rather than observed model reasoning. An incorrect crossing can propagate an error.
Figure 2: Score increases on the same 962 clues. Blue credits reference answers only; green also credits listed alternatives. Each row adds information while retaining answer-list output. Models are averaged within each of 38 equally weighted puzzles; bars show 95% language-cluster intervals. The right panel excludes Opus 4.7, Opus 5.5 and Fable 5.1, which received recovery reruns.
Measure
B
E
E–B [95% CI]
Reference agreement
70.1
72.9
+2.8 [1.3, 4.3]
Checkable crossings
81.2
74.8
-6.4 [-9.0, -3.9]
Consistent / all crossings
61.8
71.5
+9.7 [7.5, 11.9]
Table 2: Crossing analysis across all 11 models and 53 puzzles. Values are equally weighted puzzle–model percentages; changes are percentage points. Intervals resample 42 language clusters. A crossing is checkable only when both complete answers map unambiguously to the required number of cells.
Figure 3: Reference-agreement change from lengths to crossword input with answer-list output (E–B), in percentage points. Teal circles use all 53 puzzles, including recoveries. Orange squares exclude puzzles with a response selected for recovery in either condition; n is the number remaining. Bars show 95% language-cluster intervals with equal puzzle weights. Very small subsets are descriptive only.
Figure 4: Reference agreement, weighting 53 puzzles equally. A–B–C adds answer information; E supplies crossword structure; D requires placement. These are discrete tasks, not a continuous constraint scale. Missing answers remain in the denominator. Hollow markers identify means containing records selected for recovery.
Figure 5: Writing-system profiles with equal puzzle weights within groups and equal model weights within panels. Bars show language-cluster intervals. Alphabetic, abjad and abugida groups contain 27, four and ten language labels. Abjad intervals are descriptive because there are only four language clusters; Japanese has one puzzle and no population interval. D requires placement. Eight-model panels exclude Opus 4.7, Opus 5.5 and Fable 5.1, which received recovery reruns. C and E supply different information.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Display name
Recorded identifier
GPT-6 Astra (Astra)
gpt-6-astra
GPT-6.1 Sol (Sol)
gpt-6.1-sol
GPT-5.5
gpt-5.5
Gemini 4 Argon (Argon)
google/gemini-4-argon
Gemini 3.8 Flash
google/gemini-3.8-flash
Gemini 3.7 Flash
google/gemini-3.7-flash
Appendix
Table 3: Display names and recorded identifiers.
Model
A
B
C
E
D
Opus 5.5
0
5
16
43
51
Fable 5.1
8
20
31
45
51
Opus 4.7
6
38
46
39
43
Opus 4.8
0
0
0
0
0
Total / 212
14
63
93
127
145
Appendix
Table 4: Original Claude results with at least one intermediate limit stop, out of 53 per cell. Counts precede recovery and measure stop incidence rather than tokens consumed.
Model
Full: alternative/gains
Excluded: alternative/gains
Retained clues
GPT-5.5
129/272
129/272
962
GPT-6 Astra
138/293
138/293
962
GPT-6.1 Sol
140/287
140/287
962
Gemini 3.1 Pro
56/163
56/163
962
Gemini 3.7 Flash
107/237
107/237
962
Gemini 3.8 Flash
112/284
112/284
962
Appendix
Table 5: Fractions show transitions from a listed alternative to the reference divided by all transitions to the reference after adding lengths. Full samples contain 962 clues per model; exclusions remove originally flagged pairs. Losses remain in the archived data.
Model
A
B
C
E
Claude Opus 5.5
28.3 → 52.9
29.3 → 76.6
31.4 → 85.2
21.6 → 82.6
Claude Fable 5.1
46.6 → 54.6
65.4 → 77.8
73.4 → 86.1
61.2 → 84.1
Claude Opus 4.7
26.5 → 46.4
18.7 → 65.8
23.3 → 75.3
22.7 → 58.8
Appendix
Table 6: Original and recovery-integrated means for the three affected models (percent reference agreement). Each cell shows original → integrated; D is unchanged. The same 53 puzzles are retained.
Reference-agreement contrast
All 11
Without Opus 4.8
Omission range
Lengths: B–A
16.9 [14.2, 20.0]
18.5 [15.5, 21.9]
16.2–18.5
Hints: C–B
8.4 [6.5, 10.1]
8.8 [6.8, 10.7]
7.8–8.8
Crossword information: E–B
2.8 [1.3, 4.3]
4.2 [2.5, 5.8]
2.2–4.2
Answer list over grid: E–D
26.6 [24.3, 28.6]
26.8 [24.6, 28.6]
22.2–29.1
Appendix
Table 7: Model-omission sensitivity (percentage points). Brackets are pointwise 95% language-cluster intervals. The last column spans mean effects when each model is omitted in turn; it is not an interval. All four intervals remain above zero in every omission. The same 53 puzzles are used throughout.
Puzzle characteristic
All 11
Fixed eight
Without Opus 4.8
Ten more clues
-1.00 [-4.32, 1.65]
0.33 [-3.62, 3.56]
-0.77 [-4.47, 2.22]
One more answer unit
0.43 [-0.87, 1.57]
0.98 [-0.83, 2.50]
0.25 [-1.16, 1.57]
0.10 more crossing density
2.16 [-3.78, 7.22]
5.95 [-2.18, 12.88]
1.57 [-5.13, 7.41]
R2
0.032
0.038
0.017
Appendix
Table 8: Conditional associations with the puzzle-level E–B reference-agreement gain (percentage points). Each fit uses 53 equally weighted puzzles and all three predictors. Brackets give pointwise 95% intervals from 10,000 language-cluster bootstrap draws. Fixed eight excludes the three configurations selected for recovery. Answer length and crossing density are strongly correlated; these are descriptive associations, not causal effects.
Model
A
B
C
D
E
GPT-6.1 Sol
60.2
80.2
87.4
85.0
88.1
GPT-6 Astra
60.7
81.7
89.0
88.3
89.9
GPT-5.5
57.9
76.3
85.3
64.0
83.2
Gemini 4 Argon
59.8
83.1
90.0
77.4
91.1
Gemini 3.8 Flash
56.1
70.6
83.5
9.0
67.6
Gemini 3.7 Flash
53.9
66.5
80.3
5.4
75.6
Appendix
Table 9: Reference agreement (percent), weighting 53 puzzles equally and retaining missing answers in the denominator. D requires correct grid placement. Selected answer-list records for Opus 5.5, Fable 5.1 and Opus 4.7 use recovery attempts with explicit inline-output instructions.
Figure 6: Reference matches, fully specified non-reference answers, and incomplete or missing answers, using all expected clues as the denominator. Non-reference answers may be semantically acceptable. Selected list records use recovery attempts, and missing answers receive no credit.
Figure 7: Model profiles across the five task formats, using reference agreement averaged equally over 53 puzzles. All panels use the same 0–100% scale and A–B–C–D–E axis order. D also requires correct placement; missing answers remain in the denominator. Corresponding spokes show each format’s score; polygon area has no performance interpretation. Uncertainty intervals are omitted. Asterisks mark models with selected list responses replaced by recovery attempts under revised output instructions. Opus 4.8’s zero in D reflects noncompletion under this protocol.