Efficient Attention

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 114% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 96

All topics

Beyond Pairwise Attention: Higher-Order Modular Attention for Efficient Sequence Learning
Mar 11, 2026Shirin Amiraslani, Xin GaoSelf-Attention Transformer Attention
Department of Mathematics and Statistics York University Toronto, ON M3J 1P3, Canada
SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routing
Mar 9, 2026Xuanyi Zhou, Qiuyang Mang, Shuo Yang +7Video Diffusion Models Efficient Attention
UC Berkeley, USA
FlashBlock: Attention Caching for Efficient Long-Context Block Diffusion
Feb 5, 2026Zhuokun Chen, Jianfei Cai, Bohan ZhuangVideo Diffusion Models Self-Attention
1Monash University · 2Zhejiang University
Transform Trained Transformer for Accelerating Native 4K Video Generation
Dec 15, 2025Jiangning Zhang, Junwei Zhu, Teng Hu +9Efficient Transformer Inference Efficient Attention
Youtu Lab, Tencent · Zhejiang University
Accelerating Attention with Basis Decomposition
Oct 2, 2025Jialin ZhaoSelf-Attention Transformer Attention
Department of Computer Science Tsinghua University
SEMA: a Scalable and Efficient Mamba like Attention via Token Localization and Averaging
Jun 10, 2025Nhat Thanh Tran, Fanghui Xue, Shuai Zhang +4Self-Attention Vision Transformer
Department of Mathematics; University of California, Irvine; Irvine, CA 92697, USA · Qualcomm AI Research; San Diego, CA 92121, USA

Mar 11, 2026, cs.LGJ/K move · Enter open · S save

Beyond Pairwise Attention: Higher-Order Modular Attention for Efficient Sequence Learning

Shirin Amiraslani, Xin Gao

Department of Mathematics and Statistics York University Toronto, ON M3J 1P3, Canada

Self-Attention Transformer Attention Efficient Attention Attention Mechanisms Sequence Modeling

Open paper AskPDF arXiv

Sequence modeling tasks can involve intrinsic higher-order dependencies, while standard self-attention assigns scores to token pairs and does not explicitly parameterize such interactions. We introduce Higher-Order Modular Attention (HOMA), which fuses pairwise attention with an explicit triadic attention pathway made tractable through overlapping blocks, local windows, and a low-rank projection. We compare HOMA with matched pairwise and purely triadic baselines on controlled PARITY and MATCH3 tasks, as well as TAPE benchmarks. HOMA is competitive with or outperforms the baselines, with its clearest advantages when the underlying dependencies extend beyond the explicitly modeled triadic order. These advantages are accompanied in several settings by faster convergence and improved parameter efficiency, with the learned nonlinear fusion providing an effective mechanism for combining the pairwise and triadic representations. Overall, our results provide empirical evidence that HOMA is an effective attention design when task structure extends beyond pairwise interactions.

Momentum

Related topics

Latest papers 96

Beyond Pairwise Attention: Higher-Order Modular Attention for Efficient Sequence Learning

SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routing

FlashBlock: Attention Caching for Efficient Long-Context Block Diffusion

Transform Trained Transformer for Accelerating Native 4K Video Generation

Accelerating Attention with Basis Decomposition

SEMA: a Scalable and Efficient Mamba like Attention via Token Localization and Averaging