Determining whether large language models derive answers from context or prior knowledge remains a fundamental challenge. Self-contained linguistic olympiad puzzles provide a controlled setting where all answers derive solely from expert-designed context examples without external knowledge. Removing individual context examples can eliminate information needed for specific questions while leaving the rest of the puzzle unchanged. We leverage this to introduce a diagnostic framework for analyzing individual context examples. Using 53 UK Linguistics Olympiad puzzles, we generate two modified variants by deleting a single context example: (1) uniform random deletion, and (2) targeted deletion (inspired by error-correcting codes) to remove a structurally load-bearing example uniquely carrying necessary information. We formalize this impact using a Question Damage Score to classify puzzles as fragile or robust. Evaluating three frontier LLMs under instructions to abstain when information is insufficient, we find they rarely abstain, often continuing to produce correct answers after load-bearing context is removed. These findings motivate further investigation into context-based reasoning, prior knowledge, memorization, and linguistic inference. Beyond abstention, the framework enables fine-grained analyses of context reliance, including causal interventions, stopping-set analysis, targeted contamination studies, and mechanistic interpretability.
Figures & tables
Figure 1: The Pali puzzle’s context, questions, and answers (full puzzle in App. A , Fig. 3 ). Context example 3 (pink) is the only example that contains the Pali word hoti (“is”); deleting it makes questions 3.1.b and 3.2.f (also pink) unanswerable from the remaining context.
Figure 2: First three context examples of the Abawiri puzzle (full puzzle in App. A , Fig. 9 ). Context example 3 (pink) is redundant: its tokens and the lexical-possession feature it demonstrates are also attested in the other two examples.
Original
Random
ECC-inspired
Model
w/ N/A
#N/A
w/o N/A
w/ N/A
#N/A
w/o N/A
w/ N/A
#N/A
w/o N/A
GPT-5.4
0.425
1
0.415
0.407
1 (0)
0.415
0.406
1 (0)
0.415
Claude Sonnet 4.6
0.313
13
0.302
0.241
15 (10)
0.312
0.293
15 (11)
0.311
Gemini 3 Flash
0.730
3
0.701
0.697
2 (0)
0.702
0.662
4 (2)
0.704
Table 1: Average Exact Match (EM) scores across the three models and three puzzle versions. w/ N/A treats abstentions as score 0; #N/A counts puzzles where the model abstained or produced no output; w/o N/A averages over only the puzzles the model attempted. In the two deletion conditions, the parenthesized figure is the deletion-driven subset of #N/A — puzzles the model abstained on after deletion but answered in the original condition. With 53 puzzles per condition, small aggregate gaps are not informative on their own; the per-puzzle analysis below is what carries the diagnostic load.
Gemini
GPT
Sonnet
Fragility
U.
V.
U.
V.
U.
V.
Fragile ( maxDQ>0 )
32
13
22
23
27
18
Robust ( maxDQ=0 )
2
6
1
7
0
8
Total
34
19
23
30
27
26
Detecting ‘Unsolvable’ (positive class)
Precision
94.1%
95.7%
100.0%
Table 2: Each model’s judgments on the ECC-modified puzzles, compared against the structural classification by maxDQ . U. = Unsolvable, V. = Valid. Positive class: “Unsolvable.”
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: The Pali puzzle was used in UKLO in 2013. This puzzle has two difficulty scores: its score for the Foundation participants is 39% and its score for the Intermediate participants 61% ; its linguistic topic is a combination of Morphology and Syntax; its format is Rosetta; its language family is Indo-European, Indo-Aryan; its Author is Babette Verhoeven. https://www.uklo.org/wp-content/uploads/2022/09/2013.3-Pali.pdf
Figure 4: The Waama puzzle was used in UKLO in 2021. This puzzle has two difficulty scores: its score for the Breakthrough participants is 42% and its score for the Foundation participants 54% ; its linguistic topic is Syntax; its format is Match-Up; its language family is Altantic-Congo, Gur; its Author is Aleka Blackwell. https://www.uklo.org/wp-content/uploads/2022/05/2021_3-Waama.pdf
Figure 5: The Chinese Characters puzzle was used in UKLO in 2016. This puzzle has two difficulty scores: its score for the Foundation participants is 74% and its score for the Intermediate participants 92% ; its linguistic topic is Writing System; its format is Rosetta; its language family is Sino-Tibetan; its Author is Catherine Sheard. https://www.uklo.org/wp-content/uploads/2022/05/2016_4.-Beijing.pdf
Figure 6: The Longgu puzzle was used in UKLO in 2021. This puzzle has one difficulty score: its score for the Intermediate participants is 37% ; its linguistic topic is Semantics; its format is Rosetta; its language family is Austronesian, Oceanic; its Author is Babette Verhoeven. https://www.uklo.org/wp-content/uploads/2022/05/2021_6-Longgu.pdf
Figure 7: The Korowai and Haruai puzzle was used in UKLO in 2022. This puzzle has two difficulty scores: its score for the Intermediate participants is 22% and its score for the Advanced participants 26% ; its linguistic topic is Numbers; its format is Rosetta; its language family is Multiple (Trans-New-Guinea, Piawi); its Author is Simi Hellsten. https://www.uklo.org/wp-content/uploads/2022/08/7_UKLO-2022-Korowai-and-Haruai__Complete-Script.zip
Figure 8: The Permyak puzzle was used in UKLO in 2023. This puzzle has two difficulty scores: its score for the Foundation participants is 25% and its score for the Intermediate participants 44% ; its linguistic topic is Morphology; its format is Rosetta; its language family is Uralic; its Author is Pavel Iosad. https://www.uklo.org/wp-content/uploads/2023/03/2023_R1_5-Permyak.pdf
Figure 9: The Abawiri Puzzle was used in UKLO in 2023.This puzzle has one difficulty score: its score for the Round 2 participants is 60% ; its linguistic topic is Phonology; its format is Rosetta; its language family is Lakes Plain; its Author is Daniel Lovsted. https://www.uklo.org/wp-content/uploads/2023/03/2023_R2_1-Abawiri.pdf
Figure 10: Context removal in Kirichenko et al. (2025) : the original GSM8K problem statement is stripped, leaving a question with no derivable answer.
Figure 11: The TreeCut deletion example from ( Ouyang, 2025 )
Figure 12: Partial context of the Permyak puzzle (full puzzle in App. A , Fig. 8 ). Each Permyak word corresponds to a sequence of two to six English tokens, reflecting the language’s agglutinative morphology.
Model
Avg. EM
#N/A
GPT
0.173
10
Claude
0.151
4
Gemini
0.270
0
Appendix
Table 3: Avg. EM are the average Exact Match (EM) scores across the three models in no-context versions; #N/A counts puzzles where the model abstained or produced no output.
Format
Count
[’Rosetta’]
106
[’Match-up’]
66
[’Rosetta’, ’Match-up’]
3
[’Match-up’, ’Rosetta’]
1
Sum
176 (75%)
[’Computational’]
6
Appendix
Table 4: Counts of how many puzzles of a particular format (or combination of formats) were used in UKLO from 2010 to 2025.
Model
Wilcoxon p -value
McNemar p -value
(Puzzle-level EM)
(Question-level Binary)
GPT
0.4208
0.3211
Claude
0.5552
0.4799
Gemini
0.0052
0.0022
Appendix
Table 5: Statistical significance tests comparing model performance on the Original vs. ECC-deleted puzzles. Bold values indicate statistical significance at α=0.05 . The models are GPT-5.4, Claude Sonnet 4.6, and Gemini 3 Flash.
Judge Verdict
Judge Verdict
ID
Name
Random
ECC
ID
Name
Random
ECC
5
Turkish
Unsolv.
Unsolv.
139
Afrihili
Valid
Unsolv.
16
Ulwa
Valid
Valid
155
Yoruba
Valid
Valid
24
Tadaksahak
Valid
Valid
160
Kabyle
Valid
Unsolv.
25
Yolmo
Valid
Unsolv.
173
Ainu
Valid
Valid
40
Yodaspeak
Valid
Valid
175
Tawala
Unsolv.
Unsolv.
Appendix
Table 6: Breakdown of LLM judge verdicts across the Random and ECC-inspired strategies. Bold indicates cases where the Exact Match score on the modified puzzle is strictly higher than the score on the original puzzle.
ECC-inspired
Random
ID
Name
Original Score
Score
Δecc−og
Score
Δrand−og
5
Turkish
0.860
0.860
+0.000
0.860
+0.000
42
Pali
0.880
0.880
+0.000
0.880
+0.000
52
Beja
0.360
0.360
+0.000
–
–
55
Estonian
0.900
1.000
+0.100
1.000
+0.100
61
Ilokano
1.000
–
–
1.000
+0.000
Appendix
Table 7: Puzzles judged unsolvable by Gemini where Exact Match performance increased or remained equal to the original ( Δ≥0 ). Dashes (–) indicate the puzzle was either not judged unsolvable under that strategy or its score dropped.
ECC-inspired
Random
ID
Name
Original Score
Score
Δecc−og
Score
Δrand−og
16
Ulwa
1.000
0.600
-0.400
–
–
24
Tadaksahak
0.890
0.670
-0.220
0.780
-0.110
40
Yodaspeak
1.000
0.900
-0.100
–
–
93
Nung
0.120
0.000
-0.120
–
–
109
Nepali
0.900
–
–
0.700
-0.200
Appendix
Table 8: Puzzles judged solvable by Gemini where Exact Match performance strictly dropped ( Δ<0 ). Dashes (–) indicate the puzzle was either not judged solvable under that strategy or its score did not drop.
Recent benchmarks reveal that despite strong reasoning capabilities, large language models (LLMs) still struggle to faithfully apply complex contextual knowledge. These failures are often not wholesale reasoning collapses: in context-rich tasks, models may follow the central reasoning path while missing peripheral, persistent, or format-sensitive requirements.
While large language models (LLMs) have achieved remarkable linguistic capabilities, a profound question lingers at their core: do these models truly comprehend context or simply excel at pattern matching on an unprecedented scale? Contextual understanding in LLMs refers to the ability to correctly extract relevant information from a given context, integrate it into a coherent internal representation, and reason over it to produce factually consistent and contextually grounded responses. However, traditional methods such as BiLingual Evaluation Understudy (BLEU) and perplexity simply measure surface-level performance. This reveals a critical gap in question answering (QA), where responses must be contextually grounded rather than simply being memorized associations. To fill this void, we propose a novel knowledge graph (KG) based evaluation framework for LLM contextual understanding in QA. Central to this is Semantic Structural Similarity for KGs (S3KG), a hybrid similarity measure combining structural and semantic signals into a single score. In addition, a diagnostic analysis framework is developed to identify and categorize reasoning errors at the triplet level, enabling fine-grained analysis of model failures. Together, across nine benchmarks, S3KG achieves F1 gains of up to +7.6 points over the strongest baseline and AUROC up to 0.973.
Department of Computer Science & Engineering, University of Moratuwa, Sri Lanka · Department of Electrical and Computer Engineering, University of Miami, USA
We reproduce and stress-test the work of Yu et al. (2023), who characterize how language models (LMs) arbitrate between memorized knowledge and contradictory in-context statements. We replicate their world-capitals experiments on 31 models spanning Pythia, GPT-2, Qwen3, and Ministral families, including base and post-trained variants, and extend evaluations to five additional knowledge relation types from the ParaConflict dataset. We empirically confirm most of their original findings: larger models and higher-frequency entities tend to favor memorized answers, with substantial family-level variance. However, several conclusions do not generalize cleanly: entity-frequency effects disappear on Qwen3-14B and 32B; post-training shifts the memory-context trade-off inconsistently across families; question phrasing alone can change a model's reliance on memorized knowledge by up to 80 percentage points; and semantically unrelated prose can mimic coherent supporting context. Our results clarify where Yu et al.'s claims hold and to what extent they generalize to other prompts.
Guilhem Fouilhé, Nicholas Asher, Philippe Muller
IRIT ANITI · IRIT CNRS · IRIT Université de Toulouse