In-context learning enables large language models to perform novel tasks through few-shot demonstrations. However, demonstrations per se can naturally contain noise and conflicting examples, making this capability vulnerable. To understand how models process such conflicts, we study demonstration-dependent tasks requiring models to infer underlying patterns, a process we characterize as rule inference. We find that models suffer substantial performance degradation from a single demonstration with corrupted rule. This systematic misleading behavior motivates our investigation of how models process conflicting evidence internally. Using linear probes and logit lens analysis, we discover that under corruption models encode both correct and incorrect rules in intermediate layers but develop prediction confidence only in late layers, revealing a two-phase computational structure. We then identify attention heads for each phase underlying the reasoning failures: Vulnerability Heads in early-to-middle layers exhibit positional attention bias with high sensitivity to corruption, while Susceptible Heads in late layers significantly reduce support for correct predictions when exposed to the corrupted evidence. Targeted ablation validates our findings, with masking a small number of identified heads improving performance by over 10%.
Figures & tables
Figure 1: Left: The Operator Induction task where models are required to infer the underlying rule (+) from demonstrations, among which we introduce one corrupted example. Middle: Vulnerability heads show disproportionate attention allocation; they also show drastic output changes when the heavily-attended position corrupts. Right: Susceptible heads shows logit contributions favoring the corrupted operator ( × ) over the correct one (+) despite minority corruption.
Figure 2: Performance degradation under single-position corruption across different large language models and tasks. Each point represents the decrease in accuracy when corrupting the demonstration at that specific position.
Figure 3: Linear probe confidence across model layers under different corruption scenarios. The Correct Probe detects model’s encoding of ground-truth operator while the Corrupted Probe detects the corrupted operators.
Figure 4: Logit lens predictions across corruption scenarios. Rows show different layers, columns represent correct and corrupted rules, and values indicate prediction probability for each rule decoded from layer-wise residual streams.
Figure 5: Distribution of Vulnerability Heads across model layers. The purple line shows the average vulnerability score (A × S) among the top 5 heads per layer. Individual heads are shown as dots.
Figure 6: Weak correlation revealed between attention score allocation (A) and corruption sensitivity (S).
Figure 7: Distribution of susceptible heads across model layers. The purple line shows average susceptibility scores among top 5 heads per layer. Individual heads shown as dots.
Models
Heads
# of Ablated Heads
5
8
10
Qwen3 -0.6B
Vul.
1.58 ± 1.57
4.91 ± 1.55
8.49 ± 1.58
Susc.
10.33 ± 1.48
8.41 ± 1.18
8.21 ± 1.50
Random
-0.85 ± 0.42
-2.34 ± 0.38
-4.67 ± 0.51
Qwen3 -4B
Vul.
1.94 ± 0.87
0.88 ± 0.66
1.58 ± 0.90
Susc.
1.58 ± 0.90
1.94 ± 0.87
1.94 ± 0.87
Table 1: Attention head ablation results showing relative performance improvement (%) for Operator Induction, when masking different numbers and different types of attention heads.
Figure 8: Cross-task overlap of Susceptibility Heads between Operator Induction and Fake Word Inference tasks.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Model
0-shot
4-shot
6-shot
8-shot
Operator Induction
Qwen3-0.6B
0.35
0.84 ± 0.03
0.88 ± 0.01
0.88 ± 0.01
Qwen3-4B
0.37
0.99 ± 0.01
1.00 ± 0.00
1.00 ± 0.00
Llama-3.2-3B
0.42
0.83 ± 0.05
0.69 ± 0.08
0.75 ± 0.02
Llama-3.1-8B
0.32
0.90 ± 0.02
0.92 ± 0.02
0.91 ± 0.01
Fake Word Inference
Qwen3-0.6B
0.03
0.95 ± 0.02
0.88 ± 0.01
0.91 ± 0.02
Qwen3-4B
0.03
0.99 ± 0.01
1.00 ± 0.00
1.00 ± 0.00
Appendix
Table 2: Clean (Uncorrupted) in-context learning performance across tasks and models. Models exhibit reliable ICL capabilities on both Operator Induction and Fake Word Inference.
Figure 9: Linear probe confidence across model layers for Qwen3-0.6B, Qwen3-4B, Llama-3.1-8B-Instruct, and Llama-3.2-3B-Instruct from left to right, up to down. Red solid : Correct-rule probe confidence in all-correct scenarios. Red dashed : Correct-rule probe confidence in one-corrupted scenarios. Blue solid : Corrupted-rule probe confidence in all-correct scenarios. Blue dashed : Corrupted-rule probe confidence in one-corrupted scenarios. The baseline represents chance-level performance.
Figure 10: Logit lens results across model layers for Qwen3-0.6B, Qwen3-4B, Llama-3.1-8B-Instruct, and Llama-3.2-3B-Instruct from left to right, up to down.
Figure 11: Distribution statistics of vulnerability and susceptible heads for Qwen3-4B model.
Figure 12: Ablating top vulnerability heads significantly reduces the susceptibility score of top susceptible heads.
Condition
Mean Entropy
Std
Clean
0.0952
0.0239
Corrupted
0.1161
0.0217
Corrupted + Vulnerability Heads masked
0.1148
0.0167
Corrupted + Susceptible Heads masked
0.1120
0.0205
Appendix
Table 3: Average next-token prediction entropy during rule inference tasks under different conditions on Qwen3-4B. Lower entropy indicates higher model reasoning confidence.
In-context learning (ICL) is often motivated by the intuition that demonstrations help because they provide correct input-output examples. However, we reveal a counterintuitive phenomenon: correctness does not guarantee exemplar utility, and some correct demonstrations can even reduce ICL accuracy. To study this correctness-utility gap, we introduce task-preserving perturbations, where only the exemplar input is changed, while the example remains a correct instance of the same task. Concretely, each perturbed exemplar is assigned the target induced by the task mapping. This framework covers both label-updating perturbations, where task-relevant semantics change and targets are recomputed, and stricter target-preserving perturbations, where the original target remains valid. We formalize the resulting failure mode as contextual evidence shift: task-preserving perturbations can change the effective mixture of evidence used by the model for contextual inference, thereby separating exemplar correctness from exemplar utility. Across sentiment classification, logical reasoning, and math word problems, we find that task-preserving perturbed demonstrations can substantially degrade ICL performance, especially for smaller models, harder tasks, and higher perturbation ratios. Our results show that robust ICL requires evaluating not only whether demonstrations are correct, but also how they influence contextual inference. Code is available at https://github.com/Chenghao-Qiu/Task-Preserving-ICL.
This paper investigates context stickiness in in-context learning (ICL), a phenomenon where earlier examples in a prompt interfere with a transformer's ability to adapt to later tasks. Using synthetic regression tasks over linear and quadratic functions, we examine how models trained under sequential, mixed, and random curricula handle abrupt task switches during inference. By sweeping over structured combinations of misleading linear examples followed by recovery quadratic examples, we quantify how prior context biases prediction error and how quickly models realign. Our results show strong evidence of persistent interference: more preceding linear examples reliably degrade quadratic predictions, while additional quadratic examples reduce error but with diminishing returns. We further find that training curricula significantly modulate resilience, with sequential training on the target function class yielding the fastest recovery, and surprisingly, random training producing the least robust behavior.
Prior work has shown that in-context demonstrations can jailbreak language models, but it remains unclear how models interpret different types of compliance demonstrations. We study this by mixing benign compliance demonstrations (non-harmful request, helpful response) with harmful compliance demonstrations (harmful request, helpful response) and testing three hypotheses about how demonstration composition drives harmful compliance. Across four models, we find that benign and harmful demonstrations are not interchangeable: benign demonstrations can either reduce or increase harmful compliance depending on the model. We further show that preference optimization is the critical training stage that prevents benign demonstrations from increasing harmful compliance, that demonstration ordering exhibits strong recency bias, and that models differ in how refusal interacts with in-context learning: some adopt demonstrated formatting even when refusing, while others override all in-context signals upon refusal. Taken together, this work moves beyond showing that demonstration-based jailbreaking works to characterizing how it works: what models extract from compliance demonstrations depends on demonstration content, ordering, and training methodology.