A Reality Check of Inference-Time Context Augmentation Methods

Haeun Yu; Geert Heyman; Pepa Atanasova; Isabelle Augenstein

cs.CLOct 22, 2026

A Reality Check of Inference-Time Context Augmentation Methods

Authors: Haeun Yu, Geert Heyman, Pepa Atanasova, Isabelle Augenstein

Organizations: University of Copenhagen, Copenhagen, Denmark · Nokia Bell Labs, Antwerp, Belgium

Abstract

Large language models (LLMs) often fail to utilise the context given in the input, resulting in hallucinations or a fallback to parametric knowledge. Context augmentation methods aim to improve model performance by modifying the input, from inserting filler tokens that add computational time to adding relevant content such as elaboration of a context during inference. However, these methods have been evaluated in isolation and only by their performance gains, leaving it unclear how they compare and whether they reliably improve performance. In this paper, we present an evaluation framework spanning five methods that differ in how informative the inserted content is and apply them at the sentence level across seven models spanning multiple sizes on three datasets. We find that accuracy gains are not always genuine improvements but undirected perturbations: the methods correct some predictions while breaking others, particularly when the inserted text allows only for more computational time. Our analysis reveals that methods that bring no additional content primarily change model predictions near the decision boundary. Our findings highlight the need to evaluate the reliability of the accuracy gains before adopting them in practice.

Figures & tables

Figure 1: Our evaluation framework covers five methods, which can be split into two groups: one that only adds computation and the other that adds both computation and new content. While a method can change the model prediction, we further test whether the change is a directed improvement or undirected perturbation through the sign-randomisation test.

MMLU – Professional Law #753
Context	One day, while riding his motorcycle through a residential area, a man hit and injured a child who darted into his path. The child’s mother files suit against the man. At trial, the mother is then called to testify as to her conversation with the man at the hospital where her son was rushed immediately following the accident. [Intervention text is inserted in this position.] According to the mother, the man told her, “Don’t worry, my insurance company will pay for all of your son’s medical expenses.” If offered into evidence, the court will most likely rule that the mother’s statement is
Options	A. admissible, on the issue of the defendant’s negligence. B. admissible, to show that the defendant is able to pay for the medical expenses. C. inadmissible, because the testimony was highly prejudicial. D. admissible, only to prove ownership or control of the motorcycle.
Intervention text
Paraphrase	At trial, the mother is then asked to testify about her conversation with the man at the hospital where her son was taken immediately after the accident.
Self-explanation	The mother’s testimony is crucial in establishing the man’s admission of liability and potential negligence in the accident.
Teacher-explanation	This means that during the court proceeding, the plaintiff (the mother) takes the witness stand and is asked to describe what the man said to her when they spoke at the hospital shortly after her son was taken there because of the accident.

Table 1: Example of intervention texts for an MMLU question. Each intervention is inserted at the marked position; the correct option is in bold.

Model	Original	Blank	Repeat	Paraphrase	Self-expl.	Teacher-expl.
OLMo-1b	21.2	24.7 $+$ 18.8/ $-$ 15.3	24.4 $+$ 19.9/ $-$ 16.7	24.4 $+$ 19.9/ $-$ 16.7	24.9 $+$ 18.4/ $-$ 14.7	25.3 $+$ 24.1/ $-$ 20.0
Llama-3b	52.9	47.6 $+$ 5.4/ $-$ 10.7	47.6 $+$ 5.9/ $-$ 11.1	48.2 $+$ 6.2/ $-$ 11.0	49.6 $+$ 10.1/ $-$ 13.4	54.9 $+$ 12.4/ $-$ 10.4
Gemma-4b	45.7	46.6 $+$ 2.1/ $-$ 1.1	46.8 $+$ 2.8/ $-$ 1.7	46.9 $+$ 3.0/ $-$ 1.8	48.1 $+$ 8.7/ $-$ 6.2	55.0 $+$ 14.1/ $-$ 4.7
OLMo-7b	40.6	40.8 $+$ 3.2/ $-$ 3.0	40.8 $+$ 3.5/ $-$ 3.2	40.7 $+$ 3.5/ $-$ 3.3	43.0 $+$ 9.4/ $-$ 7.0	47.6 $+$ 15.5/ $-$ 8.5
Llama-8b	53.6	46.3 $+$ 5.2/ $-$ 12.5	43.2 $+$ 3.9/ $-$ 14.4	43.6 $+$ 4.2/ $-$ 14.2	49.3 $+$ 9.4/ $-$ 13.7	52.2 $+$ 10.7/ $-$ 12.1
Gemma-27b	67.0	67.7 $+$ 1.3/ $-$ 0.7	67.7 $+$ 2.0/ $-$ 1.3	67.5 $+$ 1.9/ $-$ 1.4	69.5 $+$ 5.9/ $-$ 3.4	73.9 $+$ 9.7/ $-$ 2.8

Table 2: Sentence-averaged accuracy (%) of each method on MedQA. Original reports an accuracy without intervention. Bold numbers mark accuracies higher than the Original . Small numbers with

+

are the fraction of intervention positions at which the prediction flips from incorrect to correct (

\mathrm{Corr}

), and those with

-

are the fraction of the positions at which the prediction flips from correct to incorrect (

\mathrm{Brk}

). We colour cells where the method corrects wrong predictions more often than expected under undirected perturbation, according to the sign-randomisation test (§ 5.2 ).

	OLMo-1b	Llama-3b	Gemma-4b	OLMo-7b	Llama-8b	Gemma-27b	Qwen-72b
Blank	-	$\blacksquare$	$\bullet$ , $\blacksquare$	-	$\blacktriangle$	-	-
Repeat	$\blacksquare$	$\blacksquare$	-	-	-	$\bullet$	$\blacksquare$
Paraphrase	-	-	-	-	-	$\bullet$	$\blacksquare$
Self-explanation	$\blacksquare$	-	-	$\blacktriangle$	$\blacktriangle$ , $\blacksquare$	-	$\bullet$ , $\blacksquare$
Teacher-explanation	-	$\bullet$ , $\blacktriangle$ , $\blacksquare$	-	-	$\blacktriangle$ , $\blacksquare$	$\bullet$	$\bullet$ , $\blacksquare$

Table 3: Sign-randomisation test results across all context augmentation methods and models. We mark the datasets on which the observed corrected proportion exceeds the 95% interval of the null distribution while the broken proportion does not.

\bullet

denotes the MedQA dataset,

\blacktriangle

MMLU,

\blacksquare

DRUID. - indicates that no dataset passes the test.

Panel of Figure 2 — Figure 2: We show how margin changes before ( Original , hollow circles) and after (filled circles) intervention, averaged within each category on MedQA across seven models with Blank and Teacher-explanation . For each model, the first row and the second row are Rescue and Break and the last two rows are Unchanged .

Dataset	Rescue rate	Break rate
MedQA	0.553 $\pm$ 0.18	0.569 $\pm$ 0.23
MMLU	0.422 $\pm$ 0.09	0.422 $\pm$ 0.09
DRUID	0.459 $\pm$ 0.11	0.411 $\pm$ 0.11

Table 4: Multiplicity analysis. Rescue rate: fraction of sentence positions at which an intervention rescues the prediction, among contexts rescued at least once. Break rate: the same for breaking, among contexts broken at least once. Values are means

\pm

standard deviations over all methods and models.

Figure 3: Accuracy when intervening after the last sentence (y-axis) versus after a randomly selected non-last sentence (x-axis; mean over three random draws). Points close to the dashed diagonal indicate that the last position is no better than other positions.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Number of Questions	Mean $\pm$ SD	Total sentences
MedQA	1,270	7.09 $\pm$ 3.28	8,998
MMLU	2,310	6.52 $\pm$ 3.84	15,063
DRUID	2,669	5.72 $\pm$ 3.25	15,269

Appendix

Table 5: Dataset statistics. Sentences per context are given as mean

\pm

standard deviation; the total number of sentences equals the number of intervention positions per model and method.

Characteristic	Description
Length	Number of characters in the sentence.
Overlap	Jaccard similarity between the content words of the sentence and of the question (MedQA, MMLU) or claim (DRUID).
Readability	Flesch reading ease score.
Named entities	Number of named entities in the sentence.
Negation	Whether the sentence contains a negation (a syntactic negation or a negation cue such as no , not , without ).
Pronouns	Number of pronouns in the sentence.

Appendix

Table 6: Sentence-level lexical characteristics. The first five follow Hagström et al. (2025) ; the last five capture references across sentences. Coreference is resolved over the whole context with coreferee.

Input prompt for Paraphrase
Paraphrase the following sentence. Only output the paraphrased sentence. \n\n Sentence: %s
Input prompt for Self-explanation and Teacher-explanation
Generate additional knowledge to understand the sentence marked with << >> in the [Context]. Only generate a sentence.
\n\n Context: %s

Appendix

Table 7: We present prompts used for generating intervention texts for Paraphrase , Self-explanation , and Teacher-explanation .

Model	Original	Blank	Repeat	Paraphrase	Self-expl.	Teacher-expl.
OLMo-1b	26.8	27.1 $+$ 2.2/ $-$ 1.8	27.4 $+$ 2.7/ $-$ 2.1	27.3 $+$ 3.0/ $-$ 2.4	27.6 $+$ 4.0/ $-$ 3.1	25.9 $+$ 9.8/ $-$ 10.6
Llama-3b	54.0	53.6 $+$ 1.2/ $-$ 1.6	53.4 $+$ 2.0/ $-$ 2.6	54.0 $+$ 3.4/ $-$ 3.3	53.3 $+$ 4.0/ $-$ 4.7	55.3 $+$ 5.7/ $-$ 4.3
Gemma-4b	52.2	51.4 $+$ 1.2/ $-$ 2.0	51.7 $+$ 1.7/ $-$ 2.3	51.4 $+$ 1.7/ $-$ 2.5	51.7 $+$ 3.5/ $-$ 4.0	53.8 $+$ 6.1/ $-$ 4.5
OLMo-7b	49.3	49.1 $+$ 1.4/ $-$ 1.6	49.2 $+$ 2.0/ $-$ 2.1	49.4 $+$ 2.7/ $-$ 2.6	50.2 $+$ 5.4/ $-$ 4.5	49.4 $+$ 9.6/ $-$ 9.5
Llama-8b	52.6	53.1 $+$ 1.4/ $-$ 1.0	52.2 $+$ 1.6/ $-$ 2.0	52.9 $+$ 1.9/ $-$ 1.7	53.8 $+$ 4.3/ $-$ 3.2	54.5 $+$ 4.8/ $-$ 2.9
Gemma-27b	67.4	67.1 $+$ 1.0/ $-$ 1.3	66.9 $+$ 1.1/ $-$ 1.6	67.1 $+$ 1.5/ $-$ 1.9	67.2 $+$ 3.4/ $-$ 3.7	68.2 $+$ 4.7/ $-$ 3.9

Appendix

Table 8: Sentence-averaged accuracy (%) of each method on MMLU. Original reports an accuracy without intervention. Small numbers with

+

are the fraction of intervention positions at which the prediction flips from incorrect to correct (

\mathrm{Corr}

), and those with

-

are the fraction of the positions at which the prediction flips from correct to incorrect (

\mathrm{Brk}

). We colour cells where the method corrects wrong predictions more often than the 95% interval of the random distribution, without breaking correct predictions more often than the 95% interval of the random distribution.

Model	Original	Blank	Repeat	Paraphrase	Self-expl.	Teacher-expl.
OLMo-1b	82.4	82.2 $+$ 1.3/ $-$ 1.5	82.5 $+$ 2.5/ $-$ 2.4	82.5 $+$ 2.4/ $-$ 2.3	83.1 $+$ 4.6/ $-$ 4.0	80.8 $+$ 2.5/ $-$ 4.2
Llama-3b	71.1	71.4 $+$ 1.4/ $-$ 1.1	72.0 $+$ 2.3/ $-$ 1.4	71.6 $+$ 2.1/ $-$ 1.7	71.1 $+$ 1.8/ $-$ 1.9	72.1 $+$ 2.6/ $-$ 1.6
Gemma-4b	87.1	87.1 $+$ 1.0/ $-$ 1.0	86.9 $+$ 1.3/ $-$ 1.5	86.6 $+$ 1.2/ $-$ 1.7	87.0 $+$ 1.5/ $-$ 1.6	86.9 $+$ 1.9/ $-$ 2.1
OLMo-7b	85.0	81.7 $+$ 2.1/ $-$ 5.3	82.3 $+$ 2.0/ $-$ 4.7	82.0 $+$ 2.1/ $-$ 5.1	81.2 $+$ 2.1/ $-$ 6.0	77.0 $+$ 2.6/ $-$ 10.6
Llama-8b	79.4	79.2 $+$ 1.1/ $-$ 1.3	79.2 $+$ 1.5/ $-$ 1.7	79.3 $+$ 1.6/ $-$ 1.7	80.3 $+$ 2.3/ $-$ 1.4	80.3 $+$ 2.4/ $-$ 1.5
Gemma-27b	89.7	89.5 $+$ 0.3/ $-$ 0.5	89.6 $+$ 0.6/ $-$ 0.7	89.4 $+$ 0.5/ $-$ 0.8	89.4 $+$ 0.7/ $-$ 1.0	89.2 $+$ 0.7/ $-$ 1.2

Appendix

Table 9: Sentence-averaged accuracy (%) of each method on DRUID.

	Blank		Repeat		Paraphrase		Self-explanation		Teacher-explanation
	Rescue	Break	Rescue	Break	Rescue	Break	Rescue	Break	Rescue	Break
OLMo-1b	0.922	0.922	0.916	0.915	0.897	0.903	0.816	0.823	0.958	0.965
Llama-3b	0.747	0.883	0.646	0.813	0.628	0.787	0.560	0.624	0.521	0.555
Gemma-4b	0.449	0.373	0.390	0.393	0.380	0.339	0.471	0.409	0.457	0.341
OLMo-7b	0.549	0.521	0.475	0.464	0.419	0.463	0.421	0.412	0.461	0.451
Llama-8b	0.761	0.895	0.701	0.844	0.679	0.825	0.550	0.636	0.504	0.637

Appendix

Table 10: Multiplicity analysis on MedQA.

	Blank		Repeat		Paraphrase		Self-explanation		Teacher-explanation
	Rescue	Break	Rescue	Break	Rescue	Break	Rescue	Break	Rescue	Break
OLMo-1b	0.501	0.526	0.475	0.428	0.470	0.452	0.463	0.425	0.889	0.902
Llama-3b	0.394	0.447	0.434	0.439	0.458	0.463	0.412	0.439	0.419	0.412
Gemma-4b	0.392	0.419	0.382	0.381	0.385	0.369	0.375	0.379	0.396	0.359
OLMo-7b	0.336	0.436	0.417	0.427	0.397	0.403	0.414	0.420	0.474	0.489
Llama-8b	0.430	0.395	0.433	0.418	0.400	0.377	0.433	0.384	0.374	0.348

Appendix

Table 11: Multiplicity analysis on MMLU.

	Blank		Repeat		Paraphrase		Self-explanation		Teacher-explanation
	Rescue	Break	Rescue	Break	Rescue	Break	Rescue	Break	Rescue	Break
OLMo-1b	0.374	0.392	0.421	0.452	0.384	0.383	0.464	0.399	0.445	0.505
Llama-3b	0.437	0.424	0.437	0.392	0.436	0.420	0.421	0.425	0.443	0.386
Gemma-4b	0.377	0.336	0.441	0.390	0.396	0.372	0.413	0.349	0.456	0.358
OLMo-7b	0.744	0.720	0.648	0.637	0.705	0.620	0.634	0.626	0.750	0.678
Llama-8b	0.437	0.353	0.436	0.395	0.417	0.367	0.471	0.360	0.461	0.347

Appendix

Table 12: Multiplicity analysis on DRUID.

Method	Model	Obs.	Null	95% interval	$\Delta$	Obs.	Null	95% interval	$\Delta$	Pass
		$\mathrm{Corr}$				$\mathrm{Brk}$
Blank	OLMo-1b	18.8	9.0	[8.5, 9.5]	+9.8	15.3	7.3	[6.9, 7.7]	+8.0
	Llama-3b	5.4	7.8	[7.4, 8.3]	$-$ 2.4	10.7	6.9	[6.4, 7.4]	+3.8
	Gemma-4b	2.1	1.7	[1.5, 1.9]	+0.4	1.1	1.1	[0.9, 1.2]	+0.1	$\checkmark$
	OLMo-7b	3.2	2.7	[2.4, 3.0]	+0.5	3.0	2.5	[2.3, 2.8]	+0.4
	Llama-8b	5.2	6.3	[5.9, 6.7]	$-$ 1.2	12.5	9.1	[8.6, 9.6]	+3.4

Appendix

Table 13: MedQA: sign-randomisation test for

\mathrm{Corr}

and

\mathrm{Brk}

(%). Obs.: observed value; Null: mean of the null distribution over

B=1000

sign-randomised draws, with its 95% interval;

\Delta

: observed minus null mean. Pass:

\mathrm{Corr}

exceeds its 95% interval while

\mathrm{Brk}

does not.

Method	Model	Obs.	Null	95% interval	$\Delta$	Obs.	Null	95% interval	$\Delta$	Pass
		$\mathrm{Corr}$				$\mathrm{Brk}$
Blank	OLMo-1b	2.2	2.6	[2.4, 2.8]	$-$ 0.3	1.8	2.0	[1.8, 2.1]	$-$ 0.1
	Llama-3b	1.2	1.5	[1.4, 1.6]	$-$ 0.3	1.6	1.9	[1.7, 2.0]	$-$ 0.2
	Gemma-4b	1.2	1.1	[0.9, 1.2]	+0.1	2.0	1.5	[1.3, 1.7]	+0.5
	OLMo-7b	1.4	1.6	[1.4, 1.7]	$-$ 0.1	1.6	1.5	[1.4, 1.7]	+0.1
	Llama-8b	1.4	1.2	[1.0, 1.3]	+0.2	1.0	1.6	[1.4, 1.8]	$-$ 0.6	$\checkmark$

Appendix

Table 14: MMLU: sign-randomisation test for

\mathrm{Corr}

and

\mathrm{Brk}

(%). Obs.: observed value; Null: mean of the null distribution over

B=1000

sign-randomised draws, with its 95% interval;

\Delta

: observed minus null mean. Pass:

\mathrm{Corr}

exceeds its 95% interval while

\mathrm{Brk}

does not.

Method	Model	Obs.	Null	95% interval	$\Delta$	Obs.	Null	95% interval	$\Delta$	Pass
		$\mathrm{Corr}$				$\mathrm{Brk}$
Blank	OLMo-1b	1.3	1.6	[1.4, 1.7]	$-$ 0.3	1.5	1.3	[1.2, 1.5]	+0.2
	Llama-3b	1.4	1.1	[1.0, 1.3]	+0.3	1.1	1.3	[1.1, 1.4]	$-$ 0.1	$\checkmark$
	Gemma-4b	1.0	0.8	[0.6, 0.9]	+0.2	1.0	1.1	[0.9, 1.2]	$-$ 0.1	$\checkmark$
	OLMo-7b	2.1	2.7	[2.5, 2.9]	$-$ 0.7	5.3	4.4	[4.1, 4.6]	+1.0
	Llama-8b	1.1	1.1	[0.9, 1.2]	+0.0	1.3	1.5	[1.3, 1.7]	$-$ 0.2

Appendix

Table 15: DRUID: sign-randomisation test for

\mathrm{Corr}

and

\mathrm{Brk}

(%). Obs.: observed value; Null: mean of the null distribution over

B=1000

sign-randomised draws, with its 95% interval;

\Delta

: observed minus null mean. Pass:

\mathrm{Corr}

exceeds its 95% interval while

\mathrm{Brk}

does not.

Figure 4: Margin analysis results on MedQA with Repeat , Paraphrase , and Self-explanation .

Figure 5: Margin analysis results on MMLU with Blank , Repeat , and Paraphrase .

Figure 6: Margin analysis results on MMLU with Self-explanation and Teacher-explanation .

Figure 7: Margin analysis results on DRUID with Blank , Repeat , and Paraphrase .

Figure 8: Margin analysis results on DRUID with Self-explanation and Teacher-explanation .

	Blank		Repeat		Paraphrase		Self-explanation		Teacher-explanation
Characteristic	R	B	R	B	R	B	R	B	R	B
Length	0.51	0.51	0.54	0.52	0.53	0.52	0.50	0.49	0.51	0.49
Overlap	0.51	0.51	0.52	0.52	0.52	0.52	0.50	0.50	0.51	0.51
Readability	0.49	0.50	0.49	0.50	0.48	0.50	0.50	0.51	0.50	0.51
Named entities	0.51	0.50	0.53	0.52	0.53	0.52	0.51	0.50	0.51	0.50
Negation	0.48	0.50	0.49	0.50	0.49	0.49	0.48	0.49	0.48	0.49

Appendix

Table 16: AUROC (mean over seven models and three datasets) of each lexical characteristic for identifying intervention positions that lead to a Rescue (R) or a Break (B). AUROCs are computed over all intervention positions pooled across contexts; 0.5 indicates no relation.

Knowledge Augmentation for Language Models

Knowledge augmentation for language models adds external or newly learned information to improve factuality, reasoning, and domain performance.

See all 180 papers

Beyond the Parameters: A Technical Survey of Contextual Enrichment in Large Language Models: From In-Context Prompting to Causal Retrieval-Augmented Generation
Apr 3, 2026Prakhar Bansal, Shivangi AgarwalKnowledge Augmentation for Language Models Retrieval-Augmented Generation
When Context Misleads: In-context Learning with Jurisdiction in Large Language Models
Sep 23, 2026Pei-lin Li, Qingle Liu, Junyang Feng +4LLM Evaluation In-Context Learning
Department of Computer Science and Technology, Tsinghua University · School of Integrated Circuits, Huazhong University of Science and Technology
The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context
Jul 14, 2026Yanzhe Zhang, Sanmi Koyejo, Diyi YangLLM Evaluation Neural Network Robustness
Georgia Tech · Stanford University

Apr 3, 2026J/K move · Enter open · S save

Beyond the Parameters: A Technical Survey of Contextual Enrichment in Large Language Models: From In-Context Prompting to Causal Retrieval-Augmented Generation

Prakhar Bansal, Shivangi Agarwal

Knowledge Augmentation for Language Models Retrieval-Augmented Generation Structured RAG Retrieval-Augmented Reasoning LLM Prompting Graph Retrieval-Augmented Generation

Open paper AskPDF arXiv

Large language models (LLMs) encode vast world knowledge in their parameters, yet they remain fundamentally limited by static knowledge, finite context windows, and weakly structured causal reasoning. This survey provides a unified account of augmentation strategies along a single axis: the degree of structured context supplied at inference time. We cover in-context learning and prompt engineering, Retrieval-Augmented Generation (RAG), GraphRAG, and CausalRAG. Beyond conceptual comparison, we provide a transparent literature-screening protocol, a claim-audit framework, and a structured cross-paper evidence synthesis that distinguishes higher-confidence findings from emerging results. The paper concludes with a deployment-oriented decision framework and concrete research priorities for trustworthy retrieval-augmented NLP.

Abstract

Knowledge Augmentation for Language Models

Explore similar work

Beyond the Parameters: A Technical Survey of Contextual Enrichment in Large Language Models: From In-Context Prompting to Causal Retrieval-Augmented Generation

When Context Misleads: In-context Learning with Jurisdiction in Large Language Models

The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context