Causal Attention

Follow this topic Ask about this topic

Latest papers

15 papers3 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Continuum Attention
Oct 14, 2026Afshin KhadangiCausal Attention Transformer Attention
SnT, University of Luxembourg
CAST-ASD: Causal Attention over Speakers and Time for Online Active Speaker Detection
Oct 10, 2026Pratham Gupta, Haotian Qi, Tushar Sandhan +1Causal Attention
Department of Electrical Engineering, Indian Institute of Technology Kanpur · Department of Speech, Music and Hearing, KTH Royal Institute of Technology
Universal interpolation for deep residual self-attention networks
Oct 1, 2026Sibylle Marcotte, Joan BrunaSoftmax Attention Transformer
Department of Computer Science New York University
CoWindow Attention: Full Causal Coverage Is a Collective Property
Sep 26, 2026Jingze Shi, Zhangyang Peng, Xianduo Li +6Efficient Attention Causal Attention
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China. · Beijing Academy of Artificial Intelligence, Beijing, China. · Université Paris Cité, Paris, France.
Learning the Constitutive Behavior of Materials via Neural Operators and Causal Attention: Case Studies in Plasticity and Damage
Sep 2, 2026Rishabh Arora, Lisa Scheunemann, Tim Brepols +1Causal Attention Neural Operators
Institute of Applied Mechanics, RWTH Aachen University, Mies-van-der-Rohe-Str. 1, D-52074 Aachen, Germany · ACCESS e.V., Intzestr. 5, D-52072 Aachen, Germany
LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics
Aug 27, 2026Lukas Kuhn, Lucas Maes, Giuseppe Serra +4Causal Attention Video Representation Learning
German Cancer Research Center · German Cancer Consortium · Mila +5
Where Computation Lives Inside TabPFN: Causal Localisation of Attention Head Function
Jun 11, 2026Atharva Gupta, Dhruv Kumar, Murari Mandal +1Attention Head Specialization Attention Head Analysis
Department of Computer Science, Birla Institute of Technology and Science, Pilani, India · School of Computer Engineering, Kalinga Institute of Industrial Technology, Bhubaneswar, India · Birla AI Labs, Office of Ananya Birla, Aditya Birla Group, India
Express Language Modeling
Jun 9, 2026Albert Gong, Annabelle Michael Carrell, Raaz Dwivedi +1Long-Context Language Model Inference Attention Mechanisms
Cornell Tech · University of Cambridge · Microsoft Research
CASE-NET: Deep Spatio-Temporal Representation Learning via Causal Attention and Channel Recalibration for Multivariate Time Series Classification
May 21, 2026Fan Zhang, Yating Cui, Hua WangMultivariate Time Series Classification Time Series Classification
Shandong Technology and Business University · Ludong University
C2L-Net: A Data-Driven Model for State-of-Charge Estimation of Lithium-Ion Batteries During Discharge
May 9, 2026Khoa Tran, T. Nguyen-Thoi, Vin Nguyen-Thai +3Gated Recurrent Unit Causal Attention
Data Science Laboratory, Faculty of Information Technology, Ton Duc Thang University, Ho Chi Minh City, Vietnam. · Laboratory for Applied and Industrial Mathematics, Institute for Computational Science and Artificial Intelligence, Van Lang University, Ho Chi Minh City, 70000, Viet Nam. · Faculty of Mechanical, Electrical, and Computer Engineering, Van Lang School of Technology, Van Lang University, Ho Chi Minh City, 70000, Viet Nam. +6
Mask2Cause: Causal Discovery via Adjacency Constrained Causal Attention
May 8, 2026Omar Muhammad, Pasupuleti Dhruv Shivkant, Deepak N. SubramaniTime-Series Causal Discovery Causal Structure Learning
1Indian Institute of Science, Bengaluru, India
Long Context Pre-Training with Lighthouse Attention
May 7, 2026Bowen Peng, Subho Ghosh, Jeffrey QuesnelleAttention Mechanisms Self-Attention
Nous Research
Beyond Interleaving: Causal Attention Reformulations for Generative Recommender Systems
Mar 11, 2026Hailing ChengCausal Attention Generative Recommendation
Linkedin Inc Mountain View, California, USA
Stem: Rethinking Causal Information Flow in Sparse Attention
Mar 6, 2026Lin Niu, Xin Luo, Linchuan Xie +4Causal Attention Sparse Attention
Tencent · School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, Anhui, 230026, China · Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, Jiangsu, 215123, China +5
CMSA-Net: Causal Multi-scale Aggregation with Adaptive Multi-source Reference for Video Polyp Segmentation
Feb 26, 2026Tong Wang, Yaolei Qi, Siwen Wang +3Causal Attention Polyp Segmentation
Southeast University, China · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE

Oct 14, 2026J/K move · Enter open · S save

Continuum Attention

Afshin Khadangi

SnT, University of Luxembourg

Causal Attention Transformer Attention Softmax Attention

Open paper AskPDF arXiv

We introduce Continuum Attention (CA), a causal Transformer attention operator that replaces discrete key and value lookup with integration over a continuous, piecewise linear memory curve. Each query induces an exponential relevance density along the curve. Integrating the density and interpolated values yields exact normalized attention from closed form exponential moments, including half token boundary contributions. The gradients follow analytically from moments of order at most two, enabling parallel GPU implementations while retaining quadratic sequence complexity and the standard Transformer decoder. Compared with ordinary softmax and focal attention, CA reduces mean validation negative log likelihood by 0.0200 and 0.0480 nats per byte, respectively, winning all paired comparisons. An additional disjoint validation slice yields a 0.0457 nats per byte advantage over the strongest selected softmax baseline, with a paired 95% confidence interval. These findings motivate continuous integration as a promising attention alternative while leaving larger scale generalization open.