Large language models (LLMs) often fail to utilise the context given in the input, resulting in hallucinations or a fallback to parametric knowledge. Context augmentation methods aim to improve model performance by modifying the input, from inserting filler tokens that add computational time to adding relevant content such as elaboration of a context during inference. However, these methods have been evaluated in isolation and only by their performance gains, leaving it unclear how they compare and whether they reliably improve performance. In this paper, we present an evaluation framework spanning five methods that differ in how informative the inserted content is and apply them at the sentence level across seven models spanning multiple sizes on three datasets. We find that accuracy gains are not always genuine improvements but undirected perturbations: the methods correct some predictions while breaking others, particularly when the inserted text allows only for more computational time. Our analysis reveals that methods that bring no additional content primarily change model predictions near the decision boundary. Our findings highlight the need to evaluate the reliability of the accuracy gains before adopting them in practice.
Figures & tables
Figure 1: Our evaluation framework covers five methods, which can be split into two groups: one that only adds computation and the other that adds both computation and new content. While a method can change the model prediction, we further test whether the change is a directed improvement or undirected perturbation through the sign-randomisation test.
MMLU – Professional Law #753
Context
One day, while riding his motorcycle through a residential area, a man hit and injured a child who darted into his path. The child’s mother files suit against the man. At trial, the mother is then called to testify as to her conversation with the man at the hospital where her son was rushed immediately following the accident. [Intervention text is inserted in this position.] According to the mother, the man told her, “Don’t worry, my insurance company will pay for all of your son’s medical expenses.” If offered into evidence, the court will most likely rule that the mother’s statement is
Options
A. admissible, on the issue of the defendant’s negligence. B. admissible, to show that the defendant is able to pay for the medical expenses. C. inadmissible, because the testimony was highly prejudicial. D. admissible, only to prove ownership or control of the motorcycle.
Intervention text
Paraphrase
At trial, the mother is then asked to testify about her conversation with the man at the hospital where her son was taken immediately after the accident.
Self-explanation
The mother’s testimony is crucial in establishing the man’s admission of liability and potential negligence in the accident.
Teacher-explanation
This means that during the court proceeding, the plaintiff (the mother) takes the witness stand and is asked to describe what the man said to her when they spoke at the hospital shortly after her son was taken there because of the accident.
Table 1: Example of intervention texts for an MMLU question. Each intervention is inserted at the marked position; the correct option is in bold.
Model
Original
Blank
Repeat
Paraphrase
Self-expl.
Teacher-expl.
OLMo-1b
21.2
24.7 + 18.8/ − 15.3
24.4 + 19.9/ − 16.7
24.4 + 19.9/ − 16.7
24.9 + 18.4/ − 14.7
25.3 + 24.1/ − 20.0
Llama-3b
52.9
47.6 + 5.4/ − 10.7
47.6 + 5.9/ − 11.1
48.2 + 6.2/ − 11.0
49.6 + 10.1/ − 13.4
54.9 + 12.4/ − 10.4
Gemma-4b
45.7
46.6 + 2.1/ − 1.1
46.8 + 2.8/ − 1.7
46.9 + 3.0/ − 1.8
48.1 + 8.7/ − 6.2
55.0 + 14.1/ − 4.7
OLMo-7b
40.6
40.8 + 3.2/ − 3.0
40.8 + 3.5/ − 3.2
40.7 + 3.5/ − 3.3
43.0 + 9.4/ − 7.0
47.6 + 15.5/ − 8.5
Llama-8b
53.6
46.3 + 5.2/ − 12.5
43.2 + 3.9/ − 14.4
43.6 + 4.2/ − 14.2
49.3 + 9.4/ − 13.7
52.2 + 10.7/ − 12.1
Gemma-27b
67.0
67.7 + 1.3/ − 0.7
67.7 + 2.0/ − 1.3
67.5 + 1.9/ − 1.4
69.5 + 5.9/ − 3.4
73.9 + 9.7/ − 2.8
Table 2: Sentence-averaged accuracy (%) of each method on MedQA. Original reports an accuracy without intervention. Bold numbers mark accuracies higher than the Original . Small numbers with + are the fraction of intervention positions at which the prediction flips from incorrect to correct ( Corr ), and those with − are the fraction of the positions at which the prediction flips from correct to incorrect ( Brk ). We colour cells where the method corrects wrong predictions more often than expected under undirected perturbation, according to the sign-randomisation test (§ 5.2 ).
OLMo-1b
Llama-3b
Gemma-4b
OLMo-7b
Llama-8b
Gemma-27b
Qwen-72b
Blank
-
■
∙ , ■
-
▲
-
-
Repeat
■
■
-
-
-
∙
■
Paraphrase
-
-
-
-
-
∙
■
Self-explanation
■
-
-
▲
▲ , ■
-
∙ , ■
Teacher-explanation
-
∙ , ▲ , ■
-
-
▲ , ■
∙
∙ , ■
Table 3: Sign-randomisation test results across all context augmentation methods and models. We mark the datasets on which the observed corrected proportion exceeds the 95% interval of the null distribution while the broken proportion does not. ∙ denotes the MedQA dataset, ▲ MMLU, ■ DRUID. - indicates that no dataset passes the test.
Figure 2: We show how margin changes before ( Original , hollow circles) and after (filled circles) intervention, averaged within each category on MedQA across seven models with Blank and Teacher-explanation . For each model, the first row and the second row are Rescue and Break and the last two rows are Unchanged .
Dataset
Rescue rate
Break rate
MedQA
0.553 ± 0.18
0.569 ± 0.23
MMLU
0.422 ± 0.09
0.422 ± 0.09
DRUID
0.459 ± 0.11
0.411 ± 0.11
Table 4: Multiplicity analysis. Rescue rate: fraction of sentence positions at which an intervention rescues the prediction, among contexts rescued at least once. Break rate: the same for breaking, among contexts broken at least once. Values are means ± standard deviations over all methods and models.
Figure 3: Accuracy when intervening after the last sentence (y-axis) versus after a randomly selected non-last sentence (x-axis; mean over three random draws). Points close to the dashed diagonal indicate that the last position is no better than other positions.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Number of Questions
Mean ± SD
Total sentences
MedQA
1,270
7.09 ± 3.28
8,998
MMLU
2,310
6.52 ± 3.84
15,063
DRUID
2,669
5.72 ± 3.25
15,269
Appendix
Table 5: Dataset statistics. Sentences per context are given as mean ± standard deviation; the total number of sentences equals the number of intervention positions per model and method.
Characteristic
Description
Length
Number of characters in the sentence.
Overlap
Jaccard similarity between the content words of the sentence and of the question (MedQA, MMLU) or claim (DRUID).
Readability
Flesch reading ease score.
Named entities
Number of named entities in the sentence.
Negation
Whether the sentence contains a negation (a syntactic negation or a negation cue such as no , not , without ).
Pronouns
Number of pronouns in the sentence.
Appendix
Table 6: Sentence-level lexical characteristics. The first five follow Hagström et al. (2025) ; the last five capture references across sentences. Coreference is resolved over the whole context with coreferee.
Input prompt for Paraphrase
Paraphrase the following sentence. Only output the paraphrased sentence. \n\n Sentence: %s
Input prompt for Self-explanation and Teacher-explanation
Generate additional knowledge to understand the sentence marked with << >> in the [Context]. Only generate a sentence.
\n\n Context: %s
Appendix
Table 7: We present prompts used for generating intervention texts for Paraphrase , Self-explanation , and Teacher-explanation .
Model
Original
Blank
Repeat
Paraphrase
Self-expl.
Teacher-expl.
OLMo-1b
26.8
27.1 + 2.2/ − 1.8
27.4 + 2.7/ − 2.1
27.3 + 3.0/ − 2.4
27.6 + 4.0/ − 3.1
25.9 + 9.8/ − 10.6
Llama-3b
54.0
53.6 + 1.2/ − 1.6
53.4 + 2.0/ − 2.6
54.0 + 3.4/ − 3.3
53.3 + 4.0/ − 4.7
55.3 + 5.7/ − 4.3
Gemma-4b
52.2
51.4 + 1.2/ − 2.0
51.7 + 1.7/ − 2.3
51.4 + 1.7/ − 2.5
51.7 + 3.5/ − 4.0
53.8 + 6.1/ − 4.5
OLMo-7b
49.3
49.1 + 1.4/ − 1.6
49.2 + 2.0/ − 2.1
49.4 + 2.7/ − 2.6
50.2 + 5.4/ − 4.5
49.4 + 9.6/ − 9.5
Llama-8b
52.6
53.1 + 1.4/ − 1.0
52.2 + 1.6/ − 2.0
52.9 + 1.9/ − 1.7
53.8 + 4.3/ − 3.2
54.5 + 4.8/ − 2.9
Gemma-27b
67.4
67.1 + 1.0/ − 1.3
66.9 + 1.1/ − 1.6
67.1 + 1.5/ − 1.9
67.2 + 3.4/ − 3.7
68.2 + 4.7/ − 3.9
Appendix
Table 8: Sentence-averaged accuracy (%) of each method on MMLU. Original reports an accuracy without intervention. Small numbers with + are the fraction of intervention positions at which the prediction flips from incorrect to correct ( Corr ), and those with − are the fraction of the positions at which the prediction flips from correct to incorrect ( Brk ). We colour cells where the method corrects wrong predictions more often than the 95% interval of the random distribution, without breaking correct predictions more often than the 95% interval of the random distribution.
Model
Original
Blank
Repeat
Paraphrase
Self-expl.
Teacher-expl.
OLMo-1b
82.4
82.2 + 1.3/ − 1.5
82.5 + 2.5/ − 2.4
82.5 + 2.4/ − 2.3
83.1 + 4.6/ − 4.0
80.8 + 2.5/ − 4.2
Llama-3b
71.1
71.4 + 1.4/ − 1.1
72.0 + 2.3/ − 1.4
71.6 + 2.1/ − 1.7
71.1 + 1.8/ − 1.9
72.1 + 2.6/ − 1.6
Gemma-4b
87.1
87.1 + 1.0/ − 1.0
86.9 + 1.3/ − 1.5
86.6 + 1.2/ − 1.7
87.0 + 1.5/ − 1.6
86.9 + 1.9/ − 2.1
OLMo-7b
85.0
81.7 + 2.1/ − 5.3
82.3 + 2.0/ − 4.7
82.0 + 2.1/ − 5.1
81.2 + 2.1/ − 6.0
77.0 + 2.6/ − 10.6
Llama-8b
79.4
79.2 + 1.1/ − 1.3
79.2 + 1.5/ − 1.7
79.3 + 1.6/ − 1.7
80.3 + 2.3/ − 1.4
80.3 + 2.4/ − 1.5
Gemma-27b
89.7
89.5 + 0.3/ − 0.5
89.6 + 0.6/ − 0.7
89.4 + 0.5/ − 0.8
89.4 + 0.7/ − 1.0
89.2 + 0.7/ − 1.2
Appendix
Table 9: Sentence-averaged accuracy (%) of each method on DRUID.
Blank
Repeat
Paraphrase
Self-explanation
Teacher-explanation
Rescue
Break
Rescue
Break
Rescue
Break
Rescue
Break
Rescue
Break
OLMo-1b
0.922
0.922
0.916
0.915
0.897
0.903
0.816
0.823
0.958
0.965
Llama-3b
0.747
0.883
0.646
0.813
0.628
0.787
0.560
0.624
0.521
0.555
Gemma-4b
0.449
0.373
0.390
0.393
0.380
0.339
0.471
0.409
0.457
0.341
OLMo-7b
0.549
0.521
0.475
0.464
0.419
0.463
0.421
0.412
0.461
0.451
Llama-8b
0.761
0.895
0.701
0.844
0.679
0.825
0.550
0.636
0.504
0.637
Appendix
Table 10: Multiplicity analysis on MedQA.
Blank
Repeat
Paraphrase
Self-explanation
Teacher-explanation
Rescue
Break
Rescue
Break
Rescue
Break
Rescue
Break
Rescue
Break
OLMo-1b
0.501
0.526
0.475
0.428
0.470
0.452
0.463
0.425
0.889
0.902
Llama-3b
0.394
0.447
0.434
0.439
0.458
0.463
0.412
0.439
0.419
0.412
Gemma-4b
0.392
0.419
0.382
0.381
0.385
0.369
0.375
0.379
0.396
0.359
OLMo-7b
0.336
0.436
0.417
0.427
0.397
0.403
0.414
0.420
0.474
0.489
Llama-8b
0.430
0.395
0.433
0.418
0.400
0.377
0.433
0.384
0.374
0.348
Appendix
Table 11: Multiplicity analysis on MMLU.
Blank
Repeat
Paraphrase
Self-explanation
Teacher-explanation
Rescue
Break
Rescue
Break
Rescue
Break
Rescue
Break
Rescue
Break
OLMo-1b
0.374
0.392
0.421
0.452
0.384
0.383
0.464
0.399
0.445
0.505
Llama-3b
0.437
0.424
0.437
0.392
0.436
0.420
0.421
0.425
0.443
0.386
Gemma-4b
0.377
0.336
0.441
0.390
0.396
0.372
0.413
0.349
0.456
0.358
OLMo-7b
0.744
0.720
0.648
0.637
0.705
0.620
0.634
0.626
0.750
0.678
Llama-8b
0.437
0.353
0.436
0.395
0.417
0.367
0.471
0.360
0.461
0.347
Appendix
Table 12: Multiplicity analysis on DRUID.
Corr
Brk
Method
Model
Obs.
Null
95% interval
Δ
Obs.
Null
95% interval
Δ
Pass
Blank
OLMo-1b
18.8
9.0
[8.5, 9.5]
+9.8
15.3
7.3
[6.9, 7.7]
+8.0
Llama-3b
5.4
7.8
[7.4, 8.3]
− 2.4
10.7
6.9
[6.4, 7.4]
+3.8
Gemma-4b
2.1
1.7
[1.5, 1.9]
+0.4
1.1
1.1
[0.9, 1.2]
+0.1
✓
OLMo-7b
3.2
2.7
[2.4, 3.0]
+0.5
3.0
2.5
[2.3, 2.8]
+0.4
Llama-8b
5.2
6.3
[5.9, 6.7]
− 1.2
12.5
9.1
[8.6, 9.6]
+3.4
Appendix
Table 13: MedQA: sign-randomisation test for Corr and Brk (%). Obs.: observed value; Null: mean of the null distribution over B=1000 sign-randomised draws, with its 95% interval; Δ : observed minus null mean. Pass: Corr exceeds its 95% interval while Brk does not.
Corr
Brk
Method
Model
Obs.
Null
95% interval
Δ
Obs.
Null
95% interval
Δ
Pass
Blank
OLMo-1b
2.2
2.6
[2.4, 2.8]
− 0.3
1.8
2.0
[1.8, 2.1]
− 0.1
Llama-3b
1.2
1.5
[1.4, 1.6]
− 0.3
1.6
1.9
[1.7, 2.0]
− 0.2
Gemma-4b
1.2
1.1
[0.9, 1.2]
+0.1
2.0
1.5
[1.3, 1.7]
+0.5
OLMo-7b
1.4
1.6
[1.4, 1.7]
− 0.1
1.6
1.5
[1.4, 1.7]
+0.1
Llama-8b
1.4
1.2
[1.0, 1.3]
+0.2
1.0
1.6
[1.4, 1.8]
− 0.6
✓
Appendix
Table 14: MMLU: sign-randomisation test for Corr and Brk (%). Obs.: observed value; Null: mean of the null distribution over B=1000 sign-randomised draws, with its 95% interval; Δ : observed minus null mean. Pass: Corr exceeds its 95% interval while Brk does not.
Corr
Brk
Method
Model
Obs.
Null
95% interval
Δ
Obs.
Null
95% interval
Δ
Pass
Blank
OLMo-1b
1.3
1.6
[1.4, 1.7]
− 0.3
1.5
1.3
[1.2, 1.5]
+0.2
Llama-3b
1.4
1.1
[1.0, 1.3]
+0.3
1.1
1.3
[1.1, 1.4]
− 0.1
✓
Gemma-4b
1.0
0.8
[0.6, 0.9]
+0.2
1.0
1.1
[0.9, 1.2]
− 0.1
✓
OLMo-7b
2.1
2.7
[2.5, 2.9]
− 0.7
5.3
4.4
[4.1, 4.6]
+1.0
Llama-8b
1.1
1.1
[0.9, 1.2]
+0.0
1.3
1.5
[1.3, 1.7]
− 0.2
Appendix
Table 15: DRUID: sign-randomisation test for Corr and Brk (%). Obs.: observed value; Null: mean of the null distribution over B=1000 sign-randomised draws, with its 95% interval; Δ : observed minus null mean. Pass: Corr exceeds its 95% interval while Brk does not.
Figure 4: Margin analysis results on MedQA with Repeat , Paraphrase , and Self-explanation .
Figure 5: Margin analysis results on MMLU with Blank , Repeat , and Paraphrase .
Figure 6: Margin analysis results on MMLU with Self-explanation and Teacher-explanation .
Figure 7: Margin analysis results on DRUID with Blank , Repeat , and Paraphrase .
Figure 8: Margin analysis results on DRUID with Self-explanation and Teacher-explanation .
Blank
Repeat
Paraphrase
Self-explanation
Teacher-explanation
Characteristic
R
B
R
B
R
B
R
B
R
B
Length
0.51
0.51
0.54
0.52
0.53
0.52
0.50
0.49
0.51
0.49
Overlap
0.51
0.51
0.52
0.52
0.52
0.52
0.50
0.50
0.51
0.51
Readability
0.49
0.50
0.49
0.50
0.48
0.50
0.50
0.51
0.50
0.51
Named entities
0.51
0.50
0.53
0.52
0.53
0.52
0.51
0.50
0.51
0.50
Negation
0.48
0.50
0.49
0.50
0.49
0.49
0.48
0.49
0.48
0.49
Appendix
Table 16: AUROC (mean over seven models and three datasets) of each lexical characteristic for identifying intervention positions that lead to a Rescue (R) or a Break (B). AUROCs are computed over all intervention positions pooled across contexts; 0.5 indicates no relation.
Large language models (LLMs) encode vast world knowledge in their parameters, yet they remain fundamentally limited by static knowledge, finite context windows, and weakly structured causal reasoning. This survey provides a unified account of augmentation strategies along a single axis: the degree of structured context supplied at inference time. We cover in-context learning and prompt engineering, Retrieval-Augmented Generation (RAG), GraphRAG, and CausalRAG. Beyond conceptual comparison, we provide a transparent literature-screening protocol, a claim-audit framework, and a structured cross-paper evidence synthesis that distinguishes higher-confidence findings from emerging results. The paper concludes with a deployment-oriented decision framework and concrete research priorities for trustworthy retrieval-augmented NLP.
In-Context Learning (ICL) has become a cornerstone of modern LLM deployment. However, existing ICL post-training methods have a critical blind spot: they excel at extracting patterns from demonstrations while often neglecting context authority, the ability to determine whether contextual information should govern the final answer. To benchmark this capability, we introduce FakeContextBench, which contains pseudoscientific claims across seven domains. Our evaluation of commercial and open-source models shows that large-scale pre-training alone is insufficient for reliable context-authority discrimination. Moreover, prevalent ICL fine-tuning methods can increase susceptibility to misleading context, reducing reality accuracy by up to 14.95 percentage points relative to the base model. To address this trade-off, we propose Jurisdiction In-Context Learning (J-ICL), a post-training framework that incorporates context validation into the training objective. Across four model backbones, J-ICL improves ICLEval by an average of 5.84 percentage points and reality accuracy by 9.20 points over the corresponding base models. It also raises the Reality Rate by an average of 18.09 points relative to MetaICL and Symbol Tuning. These results demonstrate that ICL capability and resistance to deceptive context can be improved together. The benchmark is available at https://github.com/peilin717/FakeContext-Bench.
Pei-lin Li, Qingle Liu, Junyang Feng +4
Department of Computer Science and Technology, Tsinghua University · School of Integrated Circuits, Huazhong University of Science and Technology
As large language models (LLMs) grow more capable, they are increasingly deployed in context-rich settings where task inputs are often accompanied by long, partially irrelevant context. In a controlled setting, we find that state-of-the-art models often appear robust to task-irrelevant context at the aggregate level: prepending it to benchmark questions causes little change in overall accuracy. This aggregate stability, however, masks significant per-example instability. Even semantically meaningless pseudo-words, formed by randomly combining characters, can markedly shift model predictions on a small fraction of examples, degrading performance on some while improving it on others. This two-sided effect holds consistently across a wide range of models and datasets, yet the affected examples are largely model-specific. We further show that this instability is modulated by context type, context length, test-time compute, and model development stage. Together, our findings reveal context-induced tail risks concealed by aggregate accuracy, motivating per-example reliability evaluation of language models.