Continuum Attention

Afshin Khadangi

cs.LGOct 14, 2026

Continuum Attention

Authors: Afshin Khadangi

Organizations: SnT, University of Luxembourg

Abstract

We introduce Continuum Attention (CA), a causal Transformer attention operator that replaces discrete key and value lookup with integration over a continuous, piecewise linear memory curve. Each query induces an exponential relevance density along the curve. Integrating the density and interpolated values yields exact normalized attention from closed form exponential moments, including half token boundary contributions. The gradients follow analytically from moments of order at most two, enabling parallel GPU implementations while retaining quadratic sequence complexity and the standard Transformer decoder. Compared with ordinary softmax and focal attention, CA reduces mean validation negative log likelihood by 0.0200 and 0.0480 nats per byte, respectively, winning all paired comparisons. An additional disjoint validation slice yields a 0.0457 nats per byte advantage over the strongest selected softmax baseline, with a paired 95% confidence interval. These findings motivate continuous integration as a promising attention alternative while leaving larger scale generalization open.

Figures & tables

Model	Attention setting	$T=192$	$T=512$
$\mathrm{B}_{0.20}$	Softmax, fixed $0.20$	1.047540	1.256038
$\mathrm{B}_{0.30}$	Softmax, fixed $0.30$	1.029903	1.174585
$\mathrm{B}_{0.50}$	Softmax, fixed $0.50$	1.039986	1.209809
$\mathrm{F}$	Softmax, learned, init. $1$	1.078360	1.264449
$\mathrm{G}$	Softmax, learned, init. $0.30$	1.026097	1.200387
$\mathrm{S}$	CA, learned, init. $1$	1.038428	1.210424

Table 1: Mean development validation NLL in nats per byte over five training seeds. Lower values are better. All configurations use an equal training token budget within each context.

Configuration	$T=192$	$T=512$
Selected softmax	1.032713	1.182236
CA $\mathrm{S}$	1.046338	1.217520
CA $\mathrm{H}$	1.020175	1.135246
CA $\mathrm{P}$	1.016345	1.136495

Table 2: Mean exploratory NLL in nats per byte over five training seeds on disjoint TinyStories validation text. The selected softmax is

\mathrm{G}

T=192

and

\mathrm{B}_{0.30}

T=512

. This evaluation portion had been examined before the additional CA ablations were conducted.

Evaluation	Context	Comparison	Mean improvement	95% confidence interval	CA wins
Development	192	$\mathrm{B}_{0.30}-\mathrm{P}$	0.020016	$[0.013779,0.026253]$	$5/5$
Development	512	$\mathrm{B}_{0.30}-\mathrm{P}$	0.048045	$[0.033504,0.062587]$	$5/5$
Development	192	$\mathrm{G}-\mathrm{H}$	0.012389	$[-0.004571,0.029349]$	$5/5$
Development	512	$\mathrm{G}-\mathrm{H}$	0.072698	$[0.030012,0.115383]$	$5/5$
Exploratory	192	$\mathrm{G}-\mathrm{P}$	0.016368	$[-0.000134,0.032870]$	$5/5$
Exploratory	192	$\mathrm{G}-\mathrm{H}$	0.012538	$[-0.004729,0.029805]$	$4/5$

Table 3: Paired NLL differences in nats per byte. Positive values favor CA. Confidence intervals use Student

t

statistics across five paired training seeds and are unadjusted for multiple comparisons. The exploratory comparisons reuse previously examined validation text.

Context	Model	Train tok/s	Forward tok/s
192	Softmax $\mathrm{B}_{0.30}$	40,547	153,136
192	CA $\mathrm{H}$	40,010	129,362
192	CA $\mathrm{P}$	40,715	131,599
512	Softmax $\mathrm{B}_{0.30}$	31,130	127,098
512	CA $\mathrm{H}$	28,192	79,975
512	CA $\mathrm{P}$	28,867	77,450

Table 4: Mean training and forward throughput over five seeds on Apple Silicon. Measurements are in tokens per second. Higher values indicate greater throughput.

Causal Attention

See all 15 papers

Long Context Pre-Training with Lighthouse Attention
May 7, 2026Bowen Peng, Subho Ghosh, Jeffrey QuesnelleAttention Mechanisms Self-Attention
Nous Research
CoWindow Attention: Full Causal Coverage Is a Collective Property
Sep 26, 2026Jingze Shi, Zhangyang Peng, Xianduo Li +6Efficient Attention Causal Attention
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China. · Beijing Academy of Artificial Intelligence, Beijing, China. · Université Paris Cité, Paris, France.
Exclusive Self Attention
Mar 10, 2026Shuangfei ZhaiSelf-Attention Transformer
Apple

May 7, 2026J/K move · Enter open · S save

Long Context Pre-Training with Lighthouse Attention

Bowen Peng, Subho Ghosh, Jeffrey Quesnelle

Nous Research

Attention Mechanisms Self-Attention Long-Context Language Modeling Language Model Pretraining Efficient Attention Causal Attention+1

Open paper AskPDF arXiv

Training causal transformers at extreme sequence lengths is bottlenecked by the quadratic time and memory of scaled dot-product attention (SDPA). In this work, we propose Lighthouse Attention, a training-only symmetrical selection-based hierarchical attention algorithm that wraps around ordinary SDPA and can be easily removed towards the end of the training. Our hierarchical selection is also gradient-free, which exempts us from dealing with a complicated and potentially inefficient backward pass kernel. Our contribution is three-fold: (i) A subquadratic hierarchical pre- and post-processing step that does adaptive compression and decompression of the sequence. (ii) A symmetrical compression strategy that pools queries, keys and values at the same time, while preserving left-to-right causality, which greatly improves parallelism. (iii) A two stage training approach which we pre-train for the majority of the time with Lighthouse Attention and recover a full attention model at the end with a short training. We run preliminary small scale LLM pre-training experiments that show the effectiveness of our method compared to full attention training with all other settings matched, where we achieve a faster total training time and lower final loss after the recovery phase. Full code is available at: https://github.com/ighoshsubho/lighthouse-attention

Abstract

Causal Attention

Explore similar work

Long Context Pre-Training with Lighthouse Attention

CoWindow Attention: Full Causal Coverage Is a Collective Property

Exclusive Self Attention