Load-Bearing Context: The Question Damage Score for Evaluating Context Reliance in Linguistic Reasoning

Neh Majmudar; Elena Filatova

cs.CLAug 27, 2026

Load-Bearing Context: The Question Damage Score for Evaluating Context Reliance in Linguistic Reasoning

Authors: Neh Majmudar, Elena Filatova

Organizations: CUNY

Abstract

Determining whether large language models derive answers from context or prior knowledge remains a fundamental challenge. Self-contained linguistic olympiad puzzles provide a controlled setting where all answers derive solely from expert-designed context examples without external knowledge. Removing individual context examples can eliminate information needed for specific questions while leaving the rest of the puzzle unchanged. We leverage this to introduce a diagnostic framework for analyzing individual context examples. Using 53 UK Linguistics Olympiad puzzles, we generate two modified variants by deleting a single context example: (1) uniform random deletion, and (2) targeted deletion (inspired by error-correcting codes) to remove a structurally load-bearing example uniquely carrying necessary information. We formalize this impact using a Question Damage Score to classify puzzles as fragile or robust. Evaluating three frontier LLMs under instructions to abstain when information is insufficient, we find they rarely abstain, often continuing to produce correct answers after load-bearing context is removed. These findings motivate further investigation into context-based reasoning, prior knowledge, memorization, and linguistic inference. Beyond abstention, the framework enables fine-grained analyses of context reliance, including causal interventions, stopping-set analysis, targeted contamination studies, and mechanistic interpretability.

Figures & tables

Figure 1: The Pali puzzle’s context, questions, and answers (full puzzle in App. A , Fig. 3 ). Context example 3 (pink) is the only example that contains the Pali word hoti (“is”); deleting it makes questions 3.1.b and 3.2.f (also pink) unanswerable from the remaining context.

Figure 2: First three context examples of the Abawiri puzzle (full puzzle in App. A , Fig. 9 ). Context example 3 (pink) is redundant: its tokens and the lexical-possession feature it demonstrates are also attested in the other two examples.

	Original			Random			ECC-inspired
Model	w/ N/A	#N/A	w/o N/A	w/ N/A	#N/A	w/o N/A	w/ N/A	#N/A	w/o N/A
GPT-5.4	0.425	1	0.415	0.407	1 (0)	0.415	0.406	1 (0)	0.415
Claude Sonnet 4.6	0.313	13	0.302	0.241	15 (10)	0.312	0.293	15 (11)	0.311
Gemini 3 Flash	0.730	3	0.701	0.697	2 (0)	0.702	0.662	4 (2)	0.704

Table 1: Average Exact Match (EM) scores across the three models and three puzzle versions. w/ N/A treats abstentions as score 0; #N/A counts puzzles where the model abstained or produced no output; w/o N/A averages over only the puzzles the model attempted. In the two deletion conditions, the parenthesized figure is the deletion-driven subset of #N/A — puzzles the model abstained on after deletion but answered in the original condition. With 53 puzzles per condition, small aggregate gaps are not informative on their own; the per-puzzle analysis below is what carries the diagnostic load.

Detecting ‘Unsolvable’ (positive class)
	Gemini		GPT		Sonnet
Fragility	U.	V.	U.	V.	U.	V.
Fragile ( $\max D_{Q}>0$ )	32	13	22	23	27	18
Robust ( $\max D_{Q}=0$ )	2	6	1	7	0	8
Total	34	19	23	30	27	26
Precision	94.1%		95.7%		100.0%

Table 2: Each model’s judgments on the ECC-modified puzzles, compared against the structural classification by

\max D_{Q}

. U. = Unsolvable, V. = Valid. Positive class: “Unsolvable.”

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: The Pali puzzle was used in UKLO in 2013. This puzzle has two difficulty scores: its score for the Foundation participants is $39\%$ and its score for the Intermediate participants $61\%$ ; its linguistic topic is a combination of Morphology and Syntax; its format is Rosetta; its language family is Indo-European, Indo-Aryan; its Author is Babette Verhoeven. https://www.uklo.org/wp-content/uploads/2022/09/2013.3-Pali.pdf

Figure 4: The Waama puzzle was used in UKLO in 2021. This puzzle has two difficulty scores: its score for the Breakthrough participants is $42\%$ and its score for the Foundation participants $54\%$ ; its linguistic topic is Syntax; its format is Match-Up; its language family is Altantic-Congo, Gur; its Author is Aleka Blackwell. https://www.uklo.org/wp-content/uploads/2022/05/2021_3-Waama.pdf

Figure 5: The Chinese Characters puzzle was used in UKLO in 2016. This puzzle has two difficulty scores: its score for the Foundation participants is $74\%$ and its score for the Intermediate participants $92\%$ ; its linguistic topic is Writing System; its format is Rosetta; its language family is Sino-Tibetan; its Author is Catherine Sheard. https://www.uklo.org/wp-content/uploads/2022/05/2016_4.-Beijing.pdf

Figure 6: The Longgu puzzle was used in UKLO in 2021. This puzzle has one difficulty score: its score for the Intermediate participants is $37\%$ ; its linguistic topic is Semantics; its format is Rosetta; its language family is Austronesian, Oceanic; its Author is Babette Verhoeven. https://www.uklo.org/wp-content/uploads/2022/05/2021_6-Longgu.pdf

Figure 7: The Korowai and Haruai puzzle was used in UKLO in 2022. This puzzle has two difficulty scores: its score for the Intermediate participants is $22\%$ and its score for the Advanced participants $26\%$ ; its linguistic topic is Numbers; its format is Rosetta; its language family is Multiple (Trans-New-Guinea, Piawi); its Author is Simi Hellsten. https://www.uklo.org/wp-content/uploads/2022/08/7_UKLO-2022-Korowai-and-Haruai__Complete-Script.zip

Figure 8: The Permyak puzzle was used in UKLO in 2023. This puzzle has two difficulty scores: its score for the Foundation participants is $25\%$ and its score for the Intermediate participants $44\%$ ; its linguistic topic is Morphology; its format is Rosetta; its language family is Uralic; its Author is Pavel Iosad. https://www.uklo.org/wp-content/uploads/2023/03/2023_R1_5-Permyak.pdf

Figure 9: The Abawiri Puzzle was used in UKLO in 2023.This puzzle has one difficulty score: its score for the Round 2 participants is $60\%$ ; its linguistic topic is Phonology; its format is Rosetta; its language family is Lakes Plain; its Author is Daniel Lovsted. https://www.uklo.org/wp-content/uploads/2023/03/2023_R2_1-Abawiri.pdf

Figure 10: Context removal in Kirichenko et al. (2025) : the original GSM8K problem statement is stripped, leaving a question with no derivable answer.

Figure 11: The TreeCut deletion example from ( Ouyang, 2025 )

Figure 12: Partial context of the Permyak puzzle (full puzzle in App. A , Fig. 8 ). Each Permyak word corresponds to a sequence of two to six English tokens, reflecting the language’s agglutinative morphology.

Model	Avg. EM	#N/A
GPT	0.173	10
Claude	0.151	4
Gemini	0.270	0

Appendix

Table 3: Avg. EM are the average Exact Match (EM) scores across the three models in no-context versions; #N/A counts puzzles where the model abstained or produced no output.

Format	Count
[’Rosetta’]	106
[’Match-up’]	66
[’Rosetta’, ’Match-up’]	3
[’Match-up’, ’Rosetta’]	1
Sum	176 (75%)
[’Computational’]	6

Appendix

Table 4: Counts of how many puzzles of a particular format (or combination of formats) were used in UKLO from 2010 to 2025.

Model	Wilcoxon $p$ -value	McNemar $p$ -value
	(Puzzle-level EM)	(Question-level Binary)
GPT	0.4208	0.3211
Claude	0.5552	0.4799
Gemini	0.0052	0.0022

Appendix

Table 5: Statistical significance tests comparing model performance on the Original vs. ECC-deleted puzzles. Bold values indicate statistical significance at

\alpha=0.05

. The models are GPT-5.4, Claude Sonnet 4.6, and Gemini 3 Flash.

ID	Name	Random	ECC	ID	Name	Random	ECC
		Judge Verdict				Judge Verdict
5	Turkish	Unsolv.	Unsolv.	139	Afrihili	Valid	Unsolv.
16	Ulwa	Valid	Valid	155	Yoruba	Valid	Valid
24	Tadaksahak	Valid	Valid	160	Kabyle	Valid	Unsolv.
25	Yolmo	Valid	Unsolv.	173	Ainu	Valid	Valid
40	Yodaspeak	Valid	Valid	175	Tawala	Unsolv.	Unsolv.

Appendix

Table 6: Breakdown of LLM judge verdicts across the Random and ECC-inspired strategies. Bold indicates cases where the Exact Match score on the modified puzzle is strictly higher than the score on the original puzzle.

			ECC-inspired		Random
ID	Name	Original Score	Score	$\Delta_{\text{ecc}-\text{og}}$	Score	$\Delta_{\text{rand}-\text{og}}$
5	Turkish	0.860	0.860	+0.000	0.860	+0.000
42	Pali	0.880	0.880	+0.000	0.880	+0.000
52	Beja	0.360	0.360	+0.000	–	–
55	Estonian	0.900	1.000	+0.100	1.000	+0.100
61	Ilokano	1.000	–	–	1.000	+0.000

Appendix

Table 7: Puzzles judged unsolvable by Gemini where Exact Match performance increased or remained equal to the original (

\Delta\geq 0

). Dashes (–) indicate the puzzle was either not judged unsolvable under that strategy or its score dropped.

			ECC-inspired		Random
ID	Name	Original Score	Score	$\Delta_{\text{ecc}-\text{og}}$	Score	$\Delta_{\text{rand}-\text{og}}$
16	Ulwa	1.000	0.600	-0.400	–	–
24	Tadaksahak	0.890	0.670	-0.220	0.780	-0.110
40	Yodaspeak	1.000	0.900	-0.100	–	–
93	Nung	0.120	0.000	-0.120	–	–
109	Nepali	0.900	–	–	0.700	-0.200

Appendix

Table 8: Puzzles judged solvable by Gemini where Exact Match performance strictly dropped (

\Delta<0

). Dashes (–) indicate the puzzle was either not judged solvable under that strategy or its score did not drop.

LLM Reasoning

Large language model reasoning studies how models solve multi-step problems, verify intermediate steps, and use computation or tools reliably.

See all 931 papers

ContextGuard: Structured Self-Auditing for Context Learning in Language Models
May 26, 2026Hongbo Jin, Chi Wang, Haoran Tang +5LLM Auditing Instruction Following
Peking University · SCUT · Tsinghua University
Do LLMs Understand Context? A Knowledge Graph-Based Evaluation Framework
Sep 24, 2026Subavarshana Arumugam, Mamta Nallaretnam, Kithuni Wickramasinghe +4LLM Evaluation LLM Grounding
Department of Computer Science & Engineering, University of Moratuwa, Sri Lanka · Department of Electrical and Computer Engineering, University of Miami, USA
Memory vs. Context? Influential Factors of Factual Recall in Language Models
Sep 21, 2026Guilhem Fouilhé, Nicholas Asher, Philippe MullerPrompt Sensitivity Memorization in Language Models
IRIT ANITI · IRIT CNRS · IRIT Université de Toulouse

Abstract

LLM Reasoning

Explore similar work

ContextGuard: Structured Self-Auditing for Context Learning in Language Models

Do LLMs Understand Context? A Knowledge Graph-Based Evaluation Framework

Memory vs. Context? Influential Factors of Factual Recall in Language Models