Learning When Not to Listen: Selective Anti-Interference Pretraining for Language Models
Authors: Jinchang Zhu, Haowei He, Yi Ding, Rong Fu, Nie Xiaojian, Shuangyong Song, Zhongjiang He, Menglin Yang
Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Institute of Artificial Intelligence (TeleAI), China Telecom · University of Macau
Language models can over-condition on irrelevant preceding text: predictions already supported by local context may still change when distant, unrelated prefix tokens are perturbed. This interference is especially consequential in long, packed, or distractor-heavy contexts, where useful evidence and irrelevant spans coexist. We propose Selective Prefix Anti-Interference Regularization (SPAR), a pretraining objective for selective anti-interference. SPAR runs the original sequence and a corrupt-prefix input in which only the far prefix is changed, then uses a short-context sufficiency gate and a gated KL objective to stabilize locally supported suffix predictions. The gate operationalizes a model-based estimate of whether the far prefix supplies additional information about the target token. Mechanism analyses show that the gate identifies locally sufficient tokens and sharply reduces prefix sensitivity on gate-selected suffix tokens. In continued training on pretrained base models, SPAR improves RULER across Qwen2.5-0.5B, Qwen2.5-3B, Llama-3.2-1B, Llama-3.1-8B, and GPT2-XL under equal counted training compute; pretraining experiments further show gains on both RULER and NoLiMa. These results show that selective anti-interference is an effective objective-level signal for robust context use.
Figures & tables
Figure 1: SPAR forms a corrupt-prefix view by shuffling chunks only in the far prefix while preserving the local suffix token for token. The short-context gate selects suffix tokens with high short-context likelihood and a small full–short likelihood gain; the resulting mask restricts clean-to-corrupt-prefix KL matching while CLM remains active throughout.
Scale
Benchmark
Δ by context
Avg score
1K
2K
3K
4K
Avg
CE
SPAR
0.3B
RULER
+3.61
+3.87
+3.69
+3.33
+3.62
18.59
22.21
NoLiMa
+2.32
+2.94
+1.69
+0.98
+1.98
5.45
7.44
0.6B
RULER
+1.97
+0.74
+1.37
+0.85
+1.23
28.54
29.78
NoLiMa
+4.05
+3.05
+1.17
+1.71
+2.50
5.37
7.87
1B
RULER
+3.51
+3.96
+2.97
+2.41
+3.21
36.34
39.56
Table 1: Pretraining results averaged over three seeds. Context columns report gains (SPAR minus CE); final columns give absolute average scores. CE denotes ordinary CLM training.
Model
Δ by context
Avg score
4K
8K
12K
16K
24K
32K
Avg
CE
SPAR
Qwen2.5-0.5B
+0.24
+3.32
+1.48
+3.55
–
–
+2.15
57.83
59.98
Qwen2.5-3B
+0.37
+0.14
+0.07
+0.45
+0.38
+0.55
+0.33
79.13
79.46
Llama-3.2-1B
+1.20
+3.78
+4.98
+4.01
+3.54
+2.75
+3.38
64.16
67.53
Llama-3.1-8B
+0.62
+1.84
+2.46
+2.10
+1.72
+1.38
+1.69
78.42
80.11
GPT2-XL
+2.77
+2.03
+3.06
+3.61
+3.53
+5.59
+3.43
30.87
34.30
Table 2: Continued training on pretrained base models averaged over three seeds. Context columns report RULER gains (SPAR minus CE); final columns give absolute average scores. CE denotes ordinary CLM training.
Model
Δ by context
Avg score
8K
16K
32K
Avg
CE
SPAR
Qwen2.5-0.5B
+1.49
+1.84
+1.46
+1.60
33.83
35.43
Qwen2.5-3B
+0.71
+0.55
+0.32
+0.53
59.98
60.51
Llama-3.2-1B
+1.03
+2.21
+1.55
+1.60
50.53
52.13
Llama-3.1-8B
+0.48
+1.22
+0.86
+0.85
63.74
64.59
GPT2-XL
+1.07
+2.49
+1.31
+1.62
16.43
18.06
Table 3: HELMET long-context evaluation averaged over three seeds on NQ, TriviaQA, HotpotQA, PopQA, NIAH multikey-2/3, NIAH multivalue, JSON-KV, and MS MARCO reranking. Context columns report macro gains (SPAR minus CE); final columns give absolute average scores. CE denotes ordinary CLM training.
Model
SPAR
PIQA
SIQA
Hella
Wino
ARC-e
ARC-c
OBQA
RACE
Lambada
MMLU
Avg
0.3B
×
56.94
42.51
26.18
49.12
28.51
21.07
27.92
25.84
3.92
20.42
30.24
✓
63.72
40.62
28.54
47.92
36.86
22.41
31.08
28.48
13.92
20.11
33.37
0.6B
×
66.52
43.18
32.94
49.74
39.82
25.08
30.54
30.88
13.96
22.73
35.54
✓
69.27
41.73
35.29
49.49
46.91
26.42
32.79
31.33
20.36
23.91
37.75
1B
×
73.18
44.12
39.62
50.08
47.58
28.76
32.52
34.18
23.82
24.78
39.87
✓
73.52
42.18
40.52
50.58
54.82
29.77
32.86
32.22
25.14
26.08
40.77
Table 4: Zero-shot public benchmark accuracy (%) for from-scratch pretrained decoder-only Transformer base models.
SPAR batch fraction
4K
8K
12K
16K
Macro
10%
+0.0354
+1.6123
+0.5862
+2.7177
+1.2379
30%
-0.3477
+2.1315
+0.8277
+2.4985
+1.2775
50%
+0.2608
+3.2685
+1.1185
+3.8900
+2.1344
70%
+0.2446
+3.3231
+1.4777
+3.5515
+2.1492
90%
+0.2785
+3.5731
+1.4754
+3.0069
+2.0835
Table 5: Qwen2.5-0.5B SPAR batch fraction sensitivity at fixed λinv=0.004 . Values are RULER macro deltas against the corresponding CE baseline at each context length. CE denotes ordinary CLM training.
Figure 2: Per-prompt selectivity gains. Each point is one entity–value prompt. The horizontal axis measures robustness gain under nuisance prefix changes; the vertical axis measures gain in following changed answer evidence. The upper-right region marks prompts where SPAR improves both nuisance invariance and evidence following.
Figure 3: Gate-region analysis using an evaluation mask fixed by the frozen CE reference. Panels show (a) validation-token density, (b) SPAR minus CE prefix-distribution JS, (c) clean NLL, and (d) corrupt NLL. Blue denotes a reduction. Threshold lines mark the short-NLL and full–short-gap cutoffs. On the balanced answer-token set, the gate reaches 0.988 precision, 0.613 recall, 0.008 far-dependent false-positive rate, and 0.933 AUROC.
Model
SPAR gate
Random gate
Ungated KL
Qwen2.5-0.5B
+2.15
+0.98
+0.48
Qwen2.5-3B
+0.33
+0.12
+0.03
Llama-3.2-1B
+3.38
+1.56
+0.75
Llama-3.1-8B
+1.69
+0.74
+0.34
GPT2-XL
+3.43
+1.55
+0.85
Table 6: RULER ablation of the selection rule. Values are average gains over CE for each model, using the same evaluated context lengths as Table 2 . CE denotes ordinary CLM training.
Region
Δ JS
Δ Clean
Δ Corrupt
Δ Sens.
Gate-selected
-0.0049
+0.0186
-0.0634
-0.0820
Unselected
-0.0005
+0.0021
-0.0019
-0.0040
All suffix
-0.0014
+0.0048
-0.0096
-0.0144
Table 7: Gate-region aggregate for Figure 3 . Values are SPAR minus CE under the fixed-reference mask; Δ Sens. is the change in corrupt–clean NLL gap. CE denotes ordinary CLM training.
Figure 4: Llama-3.2-1B invariance-weight sweep at a fixed 10% SPAR batch fraction. The macro RULER gain peaks at moderate strength.
Autoregressive pretraining increasingly draws on heterogeneous data, making it important to understand how a model learns from an individual token. The next-token prediction objective naturally identifies a token's contribution with its own loss. However, each token is not only a prediction target but also context for what follows. Using controlled corruption, we decouple these two roles and find a reversal: making a noisy token easier to predict reduces its damage as a target but increases it as context. The same decoupling helps explain text generated by language models: generation selects each token by its fit to the prefix, while its role as context is never tested against an independently determined continuation, because that continuation is generated to fit it. At known corrupted positions, acting through the context can reduce damage that removing the token's own loss does not. Understanding and controlling what a model learns from a token therefore requires decoupling its roles.
Suqin Yuan, Runqi Lin, Kevin Qinghong Lin +4
University of Sydney · University of Oxford · Southeast University
This paper investigates context stickiness in in-context learning (ICL), a phenomenon where earlier examples in a prompt interfere with a transformer's ability to adapt to later tasks. Using synthetic regression tasks over linear and quadratic functions, we examine how models trained under sequential, mixed, and random curricula handle abrupt task switches during inference. By sweeping over structured combinations of misleading linear examples followed by recovery quadratic examples, we quantify how prior context biases prediction error and how quickly models realign. Our results show strong evidence of persistent interference: more preceding linear examples reliably degrade quadratic predictions, while additional quadratic examples reduce error but with diminishing returns. We further find that training curricula significantly modulate resilience, with sequential training on the target function class yielding the fastest recovery, and surprisingly, random training producing the least robust behavior.
Prefilling computational costs pose a significant bottleneck for Large Language Models (LLMs) and Large Multimodal Models (LMMs) in long-context settings. While token pruning reduces sequence length, prior methods rely on heuristics that break compatibility with hardware-efficient kernels like FlashAttention. In this work, we observe that tokens evolve toward semantic fixing points, making further processing redundant. To this end, we introduce Delta Attention Selective Halting (DASH), a training-free policy that monitors the layer-wise update dynamics of the self-attention mechanism to selectively halt stabilized tokens. Extensive evaluation confirms that DASH generalizes across language and vision benchmarks, delivering significant prefill speedups while preserving model accuracy and hardware efficiency. Code will be released at https://github.com/verach3n/DASH.git.
Yujie Chen, Tailai Chen, Yifeng Gao +4
Shanghai Jiao Tong University · University of California, San Diego · Carnegie Mellon University