Exclusive Self Attention

Shuangfei Zhai

cs.LGMar 10, 2026

Exclusive Self Attention

Authors: Shuangfei Zhai

Organizations: Apple

Abstract

We introduce exclusive self attention (XSA), a simple modification of self attention (SA) that improves Transformer's sequence modeling performance. The key idea is to constrain attention to capture only information orthogonal to the token's own value vector (thus excluding information of self position), encouraging better context modeling. Evaluated on the standard language modeling task, XSA consistently outperforms SA across model sizes up to 2.7B parameters and shows increasingly larger gains as sequence length grows.

Figures & tables

Figure 1: Visualization of the attention similarity bias of a 1.3B parameter language model of sequence length 2048 trained for 100B tokens, aggregated on 1024 random training sequences. Left : the average cosine similarity of value vectors $v_{i}$ , $v_{j}$ within a sequence; middle : the average diagonal attention value $a_{i,j}$ ; right : the average cosine similarity of attention output $y_{i}$ and the self value vector $v_{i}$ . See Eq. 1 for notations.

Figure 2: Time and memory efficiency of XSA compared to standard attention. XSA introduces minimal computational overhead across various sequence lengths and model sizes $d_{model}$ .

Model Size	$n_{\text{layers}}$	$d_{\text{model}}$	$n_{\text{heads}}$	$d_{\text{head}}$	Learning Rate
0.7B	24	1536	6	256	$5.0\times 10^{-4}$
1.4B	24	2048	24	128	$4.0\times 10^{-4}$
2.7B	32	2560	24	128	$3.0\times 10^{-4}$

Table 1: Architectures and learning rates. All models are trained with a batch size of

0.5M

tokens, for a total of 200K iterations.

Figure 3: Training and validation loss curves of XSA against the baseline Transformer for three model sizes.

0.7B
Model	ARC-E	BoolQ	HSwag	LAMBADA	OBQA	PIQA	SocIQA	WinoGr	Avg	$\Delta$ Avg
Baseline	51.26	61.07	55.68	52.82	35.00	74.05	40.02	55.88	53.22	+0.26
XSA	52.69	61.19	56.29	54.07	32.20	73.78	41.45	56.20	53.48	+0.26
1.3B
Baseline	56.19	65.47	60.69	56.24	34.60	75.90	41.40	58.80	56.16	+1.03
XSA	58.84	62.29	62.41	58.57	36.00	76.61	42.84	59.98	57.19	+1.03

Table 2: Downstream evaluation results of XSA vs baseline Transformer.

Figure 4: Training and validation loss of XSA against the baseline Transformer for various learning rates evaluated with the 1.3B model.

Figure 5: Training and validation loss of XSA against the baseline Transformer for various sequence lengths evaluated with the 1.3B model.

Figure 6: Training and validation loss of XSA against the baseline Transformer for various number of Attention Sinks evaluated with the 1.3B model.

Self-Attention

Self-attention lets sequence elements exchange context through learned query-key matches, powering Transformers and their efficient variants.

See all 222 papers

Long Context Pre-Training with Lighthouse Attention
May 7, 2026Bowen Peng, Subho Ghosh, Jeffrey QuesnelleAttention Mechanisms Self-Attention
Nous Research
Scaling Attention via Feature Sparsity
Mar 17, 2026Yan Xie, Tiansheng Wen, Tangda Huang +4Transformer Attention Efficient Attention
School of Electronic Engineering, Xidian University · Stony Brook University · TUM +2
Sessa: Selective State Space Attention
Apr 20, 2026Liubomyr HorbatkoState Space Models Long-Context Language Modeling

May 7, 2026J/K move · Enter open · S save

Long Context Pre-Training with Lighthouse Attention

Bowen Peng, Subho Ghosh, Jeffrey Quesnelle

Nous Research

Attention Mechanisms Self-Attention Long-Context Language Modeling Language Model Pretraining Efficient Attention Causal Attention+1

Open paper AskPDF arXiv

Training causal transformers at extreme sequence lengths is bottlenecked by the quadratic time and memory of scaled dot-product attention (SDPA). In this work, we propose Lighthouse Attention, a training-only symmetrical selection-based hierarchical attention algorithm that wraps around ordinary SDPA and can be easily removed towards the end of the training. Our hierarchical selection is also gradient-free, which exempts us from dealing with a complicated and potentially inefficient backward pass kernel. Our contribution is three-fold: (i) A subquadratic hierarchical pre- and post-processing step that does adaptive compression and decompression of the sequence. (ii) A symmetrical compression strategy that pools queries, keys and values at the same time, while preserving left-to-right causality, which greatly improves parallelism. (iii) A two stage training approach which we pre-train for the majority of the time with Lighthouse Attention and recover a full attention model at the end with a short training. We run preliminary small scale LLM pre-training experiments that show the effectiveness of our method compared to full attention training with all other settings matched, where we achieve a faster total training time and lower final loss after the recovery phase. Full code is available at: https://github.com/ighoshsubho/lighthouse-attention

Abstract

Self-Attention

Explore similar work

Long Context Pre-Training with Lighthouse Attention

Scaling Attention via Feature Sparsity

Sessa: Selective State Space Attention