Attention Mechanisms

Follow this topic Ask about this topic

Momentum

42 papers in the last four weeks, up 100% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 378

All topics

Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation
Jun 15, 2026Zhongzhu Zhou, Qingyang Wu, Junxiong Wang +4Neural Network Initialization Linear Attention
1The University of Sydney · 2Together AI · 5Microsoft +2
QK-Normed MLA: QK normalization without full key caching
Jun 15, 2026Yizhou Han, Yao Zhao, Jun Zhou +2KV Caching Transformer Attention
School of Data Science, The Chinese University of Hong Kong, Shenzhen.
Rethinking the Role of Efficient Attention in Hybrid Architectures
Jun 13, 2026Ziqing Qiao, Yinuo Xu, Chaojun Xiao +6Attention Head Analysis Long-Context Modeling
1Tsinghua University · 2OpenBMB
Semidirect Fourier Delta Attention: Phase-Controlled Delta Memory with Constructive Chunk-WY Kernels
Jun 12, 2026Tiantian ZhangEfficient Attention Linear Attention
Columbia University
Zero-shot generalization of transformer neural operators to larger domains
Jun 12, 2026Armand de Villeroché, Sibo Cheng, Vincent Le Guen +5Local Attention PDE Operator Learning
CEREA, ENPC, EDF R&D, Institut Polytechnique de Paris, Île-de-France, France · SINCLAIR AI Laboratory, Saclay, Île-de-France, France · EDF R&D, Île-de-France, France +1
WAM4D: Fast 4D World Action Model via Spatial Register Tokens
Jun 12, 2026Ying Li, Xiaobao Wei, Jiajun Cao +10World Models for Robotics Robotic Manipulation
1Peking University · 2The Hong Kong University of Science and Technology · 3Beijing Innovation Center of Humanoid Robotics
Context-Guided Semantic Alignment for Feature Fusion Networks
Jun 12, 2026Hyungseop Lee, Jiho Lee, Woochul KangMulti-Scale Feature Fusion Object Detection
Department of Embedded Systems Engineering, Incheon National University, Yeonsu-gu 22012, South Korea
YOLO-AMC: An Improved YOLO Architecture with Attention Mechanisms for Building Crack Detection
Jun 11, 2026Ching-Yu Tsai, Chia-Min Lin, Chih-Hsiang Yang +2Object Detection Attention Mechanisms
Department of Electrical and Computer Engineering, Tamkang University, Taiwan
LASA: A Weak Supervision Method for Open-Vocabulary Scene Sketch Semantic Segmentation
Jun 10, 2026Liwen Yi, Xianlin Zhang, Yue Zhang +2Weakly Supervised Image Segmentation Weakly Supervised Semantic Segmentation
Beijing University of Posts and Telecommunications Beijing, China
Boltzmann Attention: Learnable Ising Couplings for Cooperative Attention
Jun 10, 2026Gilhan Kim, Daniel K. ParkQuantum Annealing Ising Model
Department of Statistics and Data Science, Yonsei University, Seoul 03722, Republic of Korea · Department of Applied Statistics, Yonsei University, Seoul 03722, Republic of Korea · Department of Quantum Information, Yonsei University, Seoul 03722, Republic of Korea
Gaussian Mixture Attention: Linear-Time Sequence Mixing via Probabilistic Latent Routing
Jun 9, 2026Yongchao Huang, Hassan RazaLong-Context Language Modeling Efficient Attention
Express Language Modeling
Jun 9, 2026Albert Gong, Annabelle Michael Carrell, Raaz Dwivedi +1Long-Context Language Model Inference Attention Mechanisms
Cornell Tech · University of Cambridge · Microsoft Research
Attention Expansion: Enhancing Keyphrase Extraction from Long Documents with Attention-Augmented Contextualized Embeddings
Jun 9, 2026Roberto Martínez-Cruz, Alvaro J. López-López, José PortelaDocument Information Extraction Long-Context Modeling
Institute for Research in Technology, ICAI School of Engineering, Comillas Pontifical University, Madrid, Spain. · DD-AIM, New York, USA.
Overcoming Decoder Inconsistencies in Whisper for Dravidian and Low-Resource Languages
Jun 8, 2026Chowdam Venkata Kumar, Kumud Tripathi, Pankaj WasnikAttention Mechanisms Low-Resource Language Processing
Media Analysis Group, Sony Research India
Hasse Diagrams for Attention: A Partial Order Framework for Designing Transformer Masks
Jun 8, 2026Chentao Li, Han GuoTransformer Attention Attention Mechanisms
School of Mathematics and Statistics Northeast Petroleum University Daqing, Heilongjiang 163318, China
Spatial Priors via Space Filling Curves for Small and Limited Data Vision Transformers
Jun 8, 2026Leyla Naz Candogan, Arshia Afzal, Pol Puigdemont +1Vision Transformer Efficient ViTs
LIONS, École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland.
OmniGen-AR: AutoRegressive Any-to-Image Generation
Jun 8, 2026Junke Wang, Xun Wang, Qiushan Guo +4Conditional Image Generation Image Generation
Institute of Trustworthy Embodied AI, Fudan University · 2Shanghai Collaborative Innovation Center of Intelligent Visual Computing · 3Bytedance Seed +1
Beyond Self-Attention: Sub-Quadratic Vision Transformers for Fast Image Captioning
Jun 7, 2026Chiradeep Ghosh, Dakshina Ranjan KiskuImage Captioning Vision Transformer
Department of Computer Science and Engineering · National Institute of Technology Durgapur · Durgapur, India
TabSwift: An Efficient Tabular Foundation Model with Row-Wise Attention
Jun 5, 2026Si-Yang Liu, Han-Jia YeTabular Foundation Models Efficient Inference
School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China.
Unified Safe In-context Image Generation in Multimodal Diffusion Transformers via Restricting Unsafe Information Flows
Jun 5, 2026Xiang Yang, Feifei Li, Mi Zhang +4Diffusion Transformer Image Generation
Fudan University, Shanghai, China · East China University of Science and Technology, Shanghai, China · Shanghai University of Electric Power, Shanghai, China
Multi-Scale Feature Attention Network for Polymer Classification Using Terahertz Spectroscopy
Jun 4, 2026Roshni Mahtani, Ilán Carretero, Laura Monroy +3Multi-Scale Feature Fusion Materials Science
Instituto Universitario de Investigaci´on e Innovaci´on en Tecnolog´ıa Centarada en el Ser Humano, HumanTech, Universitat Polit`ecnica de Val`encia, Valencia, Spain · Department of Electronic Technology, Universidad Carlos III de Madrid, Legan´es, Spain · Artikode Intelligence S.L., Valencia, Spain
High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Model
Jun 4, 2026O. Duranthon, F. Boncoraglio, L. ZdeborováFine-Tuning Low-Rank Adaptation
Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) CH-1015 Lausanne
Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learning
Jun 3, 2026Niloufar Alipour Talemi, Hossein Kashiani, Fatemeh AfghahEfficient Multimodal Inference Multimodal ICL
Holcombe Department of Electrical and Computer Engineering, Clemson University, Clemson, United States.
Spatially Grounded Concept Bottleneck Models via Part-Factorized Attention
Jun 3, 2026Dhanesh RamachandramConcept Bottleneck Models Fine-Grained Image Classification
Vector Institute
Do Transformers Need Three Projections? Systematic Study of QKV Variants
Jun 1, 2026Ali Kayyam, Anusha Madan Gopal, M Anthony LewisEfficient Transformer Inference Transformer Attention
BrainChip Inc., Laguna Hills, CA, USA.
Attention Calibration for Position-Fair Dense Retrieval
Jun 1, 2026Andrianos Michail, Elias Schuhmacher, Juri Opitz +2Information Retrieval Attention Mechanisms
University of Zurich · Zurich, Switzerland
Post-Boundary Bridge: Must Local Attention Go Global Between Global Layers?
Jun 1, 2026Zhibo YangEfficient Attention Local Attention
Ocean University of China
Quantifying and Mitigating Domain Shift in Peach Leaf Damage Classification: Attention Mechanisms and Fine-Tuning Strategies
Jun 1, 2026Adrián Cánovas-Rodriguez, Miguel A. González-Illán, Maria Fernanda García-Cruz +5Agricultural Image Analysis Fine-Tuning
Department of Information and Communication Engineering, University of Murcia, Murcia, Spain · Department of Irrigation, Centro de Edafología y Biología Aplicada del Segura CEBAS-CSIC, Murcia, Spain
Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabrics
May 31, 2026Bole Ma, Jan Eitzinger, Harald Köstler +1LLM Inference KV Caching
Blurry Window Attention
May 31, 2026Axel Laborieux, Christos Sourmpis, Juan Gabriel Kostelec +1Long-Context Language Modeling Linear Attention
Huawei, Zurich, Switzerland · Huawei Advanced Computing and Storage Lab, Shenzhen, China

Jun 15, 2026, cs.LGJ/K move · Enter open · S save

Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation

Zhongzhu Zhou, Qingyang Wu, Junxiong Wang, Mayank Mishra+3

1The University of Sydney · 2Together AI · 5Microsoft+2

Neural Network Initialization Linear Attention Cross-Architecture Knowledge Distillation Language Model Distillation Attention Mechanisms Gated Delta Networks+2

Open paper AskPDF arXiv

Hybrid linear attention models offer an appealing path to faster long-context inference: they reduce the quadratic cost and KV-cache burden of full softmax attention while retaining much of the quality of Transformer models. A practical way to obtain such models is to convert a pretrained Transformer instead of pretraining a new architecture from scratch, but this conversion is still brittle. Simply copying the teacher attention projections into a Gated DeltaNet (GDN) student does not specify the new recurrent decay, write, and output-gating dynamics. As a result, the converted model often starts in a poor dynamical regime and must spend many distillation tokens repairing initialization rather than learning the remaining teacher behavior. We propose Taylor-Calibrate, a lightweight initialization method for hybrid GDN students. The method uses Taylor-guided teacher attention statistics to set the value projection, memory timescale, write gates, and output gate, then applies a short per-layer alignment step to match each converted layer to the teacher output. Across four teacher settings and three retained-layer policies, Taylor-Calibrate gives substantially stronger zero-shot students, with up to an 88x improvement in a representative ablation, and reaches matched recovery targets with 4.9x--9.2x fewer training tokens than naive conversion.

Momentum

Related topics

Latest papers 378