Learning When Not to Listen: Selective Anti-Interference Pretraining for Language Models

Jinchang Zhu; Haowei He; Yi Ding; Rong Fu; Nie Xiaojian; Shuangyong Song; Zhongjiang He; Menglin Yang

cs.CLAug 22, 2026

Learning When Not to Listen: Selective Anti-Interference Pretraining for Language Models

Authors: Jinchang Zhu, Haowei He, Yi Ding, Rong Fu, Nie Xiaojian, Shuangyong Song, Zhongjiang He, Menglin Yang

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Institute of Artificial Intelligence (TeleAI), China Telecom · University of Macau

Abstract

Language models can over-condition on irrelevant preceding text: predictions already supported by local context may still change when distant, unrelated prefix tokens are perturbed. This interference is especially consequential in long, packed, or distractor-heavy contexts, where useful evidence and irrelevant spans coexist. We propose Selective Prefix Anti-Interference Regularization (SPAR), a pretraining objective for selective anti-interference. SPAR runs the original sequence and a corrupt-prefix input in which only the far prefix is changed, then uses a short-context sufficiency gate and a gated KL objective to stabilize locally supported suffix predictions. The gate operationalizes a model-based estimate of whether the far prefix supplies additional information about the target token. Mechanism analyses show that the gate identifies locally sufficient tokens and sharply reduces prefix sensitivity on gate-selected suffix tokens. In continued training on pretrained base models, SPAR improves RULER across Qwen2.5-0.5B, Qwen2.5-3B, Llama-3.2-1B, Llama-3.1-8B, and GPT2-XL under equal counted training compute; pretraining experiments further show gains on both RULER and NoLiMa. These results show that selective anti-interference is an effective objective-level signal for robust context use.

Figures & tables

Figure 1: SPAR forms a corrupt-prefix view by shuffling chunks only in the far prefix while preserving the local suffix token for token. The short-context gate selects suffix tokens with high short-context likelihood and a small full–short likelihood gain; the resulting mask restricts clean-to-corrupt-prefix KL matching while CLM remains active throughout.

Scale	Benchmark	$\Delta$ by context					Avg score
Scale	Benchmark	1K	2K	3K	4K	Avg	CE	SPAR
0.3B	RULER	+3.61	+3.87	+3.69	+3.33	+3.62	18.59	22.21
0.3B	NoLiMa	+2.32	+2.94	+1.69	+0.98	+1.98	5.45	7.44
0.6B	RULER	+1.97	+0.74	+1.37	+0.85	+1.23	28.54	29.78
0.6B	NoLiMa	+4.05	+3.05	+1.17	+1.71	+2.50	5.37	7.87
1B	RULER	+3.51	+3.96	+2.97	+2.41	+3.21	36.34	39.56

Table 1: Pretraining results averaged over three seeds. Context columns report gains (SPAR minus CE); final columns give absolute average scores. CE denotes ordinary CLM training.

Model	$\Delta$ by context							Avg score
Model	4K	8K	12K	16K	24K	32K	Avg	CE	SPAR
Qwen2.5-0.5B	+0.24	+3.32	+1.48	+3.55	–	–	+2.15	57.83	59.98
Qwen2.5-3B	+0.37	+0.14	+0.07	+0.45	+0.38	+0.55	+0.33	79.13	79.46
Llama-3.2-1B	+1.20	+3.78	+4.98	+4.01	+3.54	+2.75	+3.38	64.16	67.53
Llama-3.1-8B	+0.62	+1.84	+2.46	+2.10	+1.72	+1.38	+1.69	78.42	80.11
GPT2-XL	+2.77	+2.03	+3.06	+3.61	+3.53	+5.59	+3.43	30.87	34.30

Table 2: Continued training on pretrained base models averaged over three seeds. Context columns report RULER gains (SPAR minus CE); final columns give absolute average scores. CE denotes ordinary CLM training.

Model	$\Delta$ by context				Avg score
Model	8K	16K	32K	Avg	CE	SPAR
Qwen2.5-0.5B	+1.49	+1.84	+1.46	+1.60	33.83	35.43
Qwen2.5-3B	+0.71	+0.55	+0.32	+0.53	59.98	60.51
Llama-3.2-1B	+1.03	+2.21	+1.55	+1.60	50.53	52.13
Llama-3.1-8B	+0.48	+1.22	+0.86	+0.85	63.74	64.59
GPT2-XL	+1.07	+2.49	+1.31	+1.62	16.43	18.06

Table 3: HELMET long-context evaluation averaged over three seeds on NQ, TriviaQA, HotpotQA, PopQA, NIAH multikey-2/3, NIAH multivalue, JSON-KV, and MS MARCO reranking. Context columns report macro gains (SPAR minus CE); final columns give absolute average scores. CE denotes ordinary CLM training.

Model	SPAR	PIQA	SIQA	Hella	Wino	ARC-e	ARC-c	OBQA	RACE	Lambada	MMLU	Avg
0.3B	$\times$	56.94	42.51	26.18	49.12	28.51	21.07	27.92	25.84	3.92	20.42	30.24
0.3B	$\checkmark$	63.72	40.62	28.54	47.92	36.86	22.41	31.08	28.48	13.92	20.11	33.37
0.6B	$\times$	66.52	43.18	32.94	49.74	39.82	25.08	30.54	30.88	13.96	22.73	35.54
0.6B	$\checkmark$	69.27	41.73	35.29	49.49	46.91	26.42	32.79	31.33	20.36	23.91	37.75
1B	$\times$	73.18	44.12	39.62	50.08	47.58	28.76	32.52	34.18	23.82	24.78	39.87
1B	$\checkmark$	73.52	42.18	40.52	50.58	54.82	29.77	32.86	32.22	25.14	26.08	40.77

Table 4: Zero-shot public benchmark accuracy (%) for from-scratch pretrained decoder-only Transformer base models.

SPAR batch fraction	4K	8K	12K	16K	Macro
10%	+0.0354	+1.6123	+0.5862	+2.7177	+1.2379
30%	-0.3477	+2.1315	+0.8277	+2.4985	+1.2775
50%	+0.2608	+3.2685	+1.1185	+3.8900	+2.1344
70%	+0.2446	+3.3231	+1.4777	+3.5515	+2.1492
90%	+0.2785	+3.5731	+1.4754	+3.0069	+2.0835

Table 5: Qwen2.5-0.5B SPAR batch fraction sensitivity at fixed

\lambda_{\mathrm{inv}}=0.004

. Values are RULER macro deltas against the corresponding CE baseline at each context length. CE denotes ordinary CLM training.

Figure 2: Per-prompt selectivity gains. Each point is one entity–value prompt. The horizontal axis measures robustness gain under nuisance prefix changes; the vertical axis measures gain in following changed answer evidence. The upper-right region marks prompts where SPAR improves both nuisance invariance and evidence following.

Figure 3: Gate-region analysis using an evaluation mask fixed by the frozen CE reference. Panels show (a) validation-token density, (b) SPAR minus CE prefix-distribution JS, (c) clean NLL, and (d) corrupt NLL. Blue denotes a reduction. Threshold lines mark the short-NLL and full–short-gap cutoffs. On the balanced answer-token set, the gate reaches 0.988 precision, 0.613 recall, 0.008 far-dependent false-positive rate, and 0.933 AUROC.

Model	SPAR gate	Random gate	Ungated KL
Qwen2.5-0.5B	+2.15	+0.98	+0.48
Qwen2.5-3B	+0.33	+0.12	+0.03
Llama-3.2-1B	+3.38	+1.56	+0.75
Llama-3.1-8B	+1.69	+0.74	+0.34
GPT2-XL	+3.43	+1.55	+0.85

Table 6: RULER ablation of the selection rule. Values are average gains over CE for each model, using the same evaluated context lengths as Table 2 . CE denotes ordinary CLM training.

Region	$\Delta$ JS	$\Delta$ Clean	$\Delta$ Corrupt	$\Delta$ Sens.
Gate-selected	-0.0049	+0.0186	-0.0634	-0.0820
Unselected	-0.0005	+0.0021	-0.0019	-0.0040
All suffix	-0.0014	+0.0048	-0.0096	-0.0144

Table 7: Gate-region aggregate for Figure 3 . Values are SPAR minus CE under the fixed-reference mask;

\Delta

Sens. is the change in corrupt–clean NLL gap. CE denotes ordinary CLM training.

Figure 4: Llama-3.2-1B invariance-weight sweep at a fixed 10% SPAR batch fraction. The macro RULER gain peaks at moderate strength.

Language Model Pretraining

Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.

See all 422 papers

Decoupling Token Roles in Autoregressive Pretraining
Sep 27, 2026Suqin Yuan, Runqi Lin, Kevin Qinghong Lin +4Language Model Pretraining Autoregressive Language Modeling
University of Sydney · University of Oxford · Southeast University
When Context Sticks: Studying Interference in In-Context Learning
Apr 25, 2026Hanna Rød, Dagny Streit, Nils Valseth Selte +1In-Context Learning
Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefilling
Apr 20, 2026Yujie Chen, Tailai Chen, Yifeng Gao +4Self-Attention Long-Context Language Model Inference
Shanghai Jiao Tong University · University of California, San Diego · Carnegie Mellon University

Sep 27, 2026J/K move · Enter open · S save

Decoupling Token Roles in Autoregressive Pretraining

Suqin Yuan, Runqi Lin, Kevin Qinghong Lin, Junchi Yu+3

University of Sydney · University of Oxford · Southeast University

Language Model Pretraining Autoregressive Language Modeling Causal Interventions in Language Models Language Model Robustness Autoregressive Transformer Sequence Prediction

Open paper AskPDF arXiv

Autoregressive pretraining increasingly draws on heterogeneous data, making it important to understand how a model learns from an individual token. The next-token prediction objective naturally identifies a token's contribution with its own loss. However, each token is not only a prediction target but also context for what follows. Using controlled corruption, we decouple these two roles and find a reversal: making a noisy token easier to predict reduces its damage as a target but increases it as context. The same decoupling helps explain text generated by language models: generation selects each token by its fit to the prefix, while its role as context is never tested against an independently determined continuation, because that continuation is generated to fit it. At known corrupted positions, acting through the context can reduce damage that removing the token's own loss does not. Understanding and controlling what a model learns from a token therefore requires decoupling its roles.

Abstract

Language Model Pretraining

Explore similar work

Decoupling Token Roles in Autoregressive Pretraining

When Context Sticks: Studying Interference in In-Context Learning

Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefilling