Attention Mechanisms

Follow this topic Ask about this topic

Momentum

42 papers in the last four weeks, up 100% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 378

All topics

Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch
Jul 4, 2026Mary Letey, Yue M. Lu, Cengiz Pehlevan +1In-Context Learning Attention Mechanisms
John A. Paulson School of Engineering and Applied Sciences, Harvard University · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University · Society of Fellows and Center for Brain Science, Harvard University
Learning to Generate Multiple Objects from Dense and Occluded Layouts
Jul 3, 2026Bach-Hoang Ngo, Si-Tri Ngo, Hieu Le +1Image Generation Compositional Image Generation
University of Science, Ho Chi Minh City, Vietnam · 2Vietnam National University, Ho Chi Minh City, Vietnam · 3UNC Charlotte, North Carolina, United States
LBTCap: A Lightweight Bilateral Transformer for Real-Time Remote Sensing Image Change Captioning
Jul 3, 2026Licheng Zhang, Siew-Kei Lam, Naveed AkhtarRemote Sensing Image Understanding Image Captioning
School of Computing and Information Systems, The University of Melbourne, Melbourne, 3010, Victoria, Australia · Nanyang Technological University, Singapore · The University of Melbourne, Melbourne, 3010, Victoria, Australia
Self-Gating Attention for Efficient Time Series Forecasting
Jul 2, 2026Dezheng Wang, Tong Chen, Wei Yuan +3Time Series Forecasting Linear Attention
School of Automation, Southeast University, 210096, China · School of Electrical Engineering and Computer Science, The University of Queensland, QLD 4072, Australia
DeepGaze3.5-VL: Modeling Scanpaths via Autoregressive Token Prediction
Jul 2, 2026Susmit Agrawal, Matthias Bethge, Matthias KümmererVisual Attention Attention Mechanisms
University of Tübingen · Tübingen AI Center · IMPRS-IS
HistoSeg++: Delving deeper with attention and multiscale feature fusion for biomarker segmentation
Jul 2, 2026Saad Wazir, Rao Faizan, Daeyoung KimU-Net Multi-Scale Feature Fusion
School of Computing, KAIST Daejeon, Republic of Korea · Kyung Hee University (Global Campus) Yongin-si, Republic of Korea
Towards Memory-Efficient Autoregressive Video Generation via Instance-Specific Parametric Absorption
Jul 1, 2026Xiaomeng Fu, Jia Li, Yiming Hu +5Long-Horizon Video Generation Streaming Video Generation
Institute of Information Engineering, Chinese Academy of Sciences · AMAP, Alibaba Group · The University of Hong Kong
RotateAttention: RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generation
Jul 1, 2026Yaofu Liu, Wanli Lan, Jinxi Li +2Mixed-Precision Quantization Rotary Positional Embeddings
The Hong Kong University of Science and Technology, Hong Kong SAR, China · Independent Researcher, China
Enhancing Oracle Bone Inscription Recognition via Multi-Scale Layer Attention
Jun 30, 2026Chaowen Yan, Kaishen Wang, Yong Wang +2Multi-Scale Feature Fusion Attention Mechanisms
aAcademy of Chinese Traditional Culture, Sichuan Normal University, Chengdu, China · bCollege of Computer Science, Sichuan University, Chengdu, China · cSchool of Artificial Intelligence, Sichuan University, Chengdu, China
MAGE: View-guided Point Cloud Completion with Efficient Modality Alignment and Adaptive Geometry Enhancement
Jun 30, 2026Weize Quan, Zhengwei Wu, Kai Wang +1Point Cloud Completion Cross-Modal Alignment
MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, China. · School of Artificial Intelligence, University of Chinese Academy Sciences, Beijing, China. · GIPSA-lab, Univ. Grenoble Alpes, CNRS, Grenoble INP, 38000 Grenoble, France.
Reweighting Framewise Attention in Video Transformers for Facial Expression Understanding
Jun 29, 2026Seongro Yoon, Donghyeon Cho, Jinsun Park +1Vision Transformer Facial Expression Recognition
Inria, Université Côte d’Azur, France · Hanyang University, South Korea · Pusan National University, South Korea
Learning High-Level Decision Making with an Interaction-Aware Attention-Based Network in Autonomous Driving
Jun 29, 2026Marcelo Contreras, Willi Poh, Christoph Stiller +1Autonomous Driving Attention Mechanisms
NODE lab, University of Alberta, Edmonton, AB, T6G 1H9 Canada · MRT Institute, Karlsruhe Institute of Technology, Karlsruhe, Germany
Adaptive Financial Transformer with Regime-Gated Attention for Stock Return Prediction
Jun 28, 2026Dishan SarkarMultivariate Time Series Forecasting Time Series Forecasting
Department of Artificial Intelligence and Machine Learning Birla Institute of Technology, Mesra Ranchi, India
NLL-Guided Full-Attention Layer Selection for Training-Free Sliding-Window Adaptation
Jun 26, 2026Qiong Tang, Xiangkun Hu, Xiangyang Liu +2Long-Context Language Model Inference Sliding-Window Attention
FARS
DnA: Denoising Attention for Visual Tasks
Jun 25, 2026Ron Campos, Subhajit Maity, Xin Li +2Visual Attention Vision Transformer
University of Central Florida, USA · University of North Carolina at Charlotte, USA
CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention
Jun 25, 2026Sayak DuttaLinear Attention Delta-Rule Linear Attention
Privacy Vulnerabilities of Attention Layers in Tabular Foundation Models and Protection of High-Risk Queries
Jun 24, 2026Tânia Carvalho, Maxime CordyTabular Foundation Models Membership Inference Attacks
SnT, University of Luxembourg · Luxembourg
Enhancing Layer Interaction Using Key-Correlated Layer Attention
Jun 24, 2026Jianlong Xiong, ChuanBo Xie, Le Yu +2Efficient Attention Linear Attention
College of Computer Science, Sichuan University, Chengdu, 610065, China · Department of Ultrasound, Zigong Maternal and Child Health Care Hospital, China
Lightweight PCGAE-Net: Parallel CrossGate Attention and Bottleneck AutoEncoder for Efficient 5G Channel Prediction
Jun 24, 2026Uma Kishore Godavarti, K. Giridhar, Vanani Prince Dharmendrabhai +2Autoencoders Attention Mechanisms
Network Modem Team, Samsung R&D India Bangalore · Department of Electrical Engineering, Indian Institute of Technology Madras, Chennai, India
Lifelong In-Context Learning with Transformers Requires Parametric Forms of Attention
Jun 24, 2026Luke McDermott, Robert W. Heath, Rahul ParhiLifelong Learning Agents Long-Context Modeling
UC San Diego
ATMA: Length-Invariant Language Modeling via Polar Attention and Gated-Delta Compression Memory
Jun 23, 2026Habibullah AkbarMemory-Augmented Language Models Long-Context Language Modeling
Kreasof AI Jakarta, Indonesia
All Routes Lead to Collapse
Jun 21, 2026K. R. BalasubramanianRepresentation Collapse Attention Sinks
Independent Research
Gated MLPs as Symmetry-Broken Rank-1 Bilinear Attention
Jun 20, 2026Nathan BreslowMultilayer Perceptrons Symmetry Breaking
CoSA: Correlation-Guided Change Attention with Learnable Residual Gating for Remote Sensing Change Detection
Jun 20, 2026Abdirashid Omar, Jonghyuk ParkRemote Sensing Change Detection Attention Mechanisms
Department of Data Science, Graduate School of Kookmin University Seoul 02707, Republic of Korea
Fidelity- and Perception-Aware Local Implicit Attention for Arbitrary-Scale Image Super-Resolution
Jun 20, 2026Yu-Syuan Xu, Hao-Lun Sun, Hao-Wei Chen +2Diffusion Models for Image Restoration Attention Mechanisms
1National Taiwan University · 2MediaTek Inc.
Low-Rank Attention Residuals
Jun 19, 2026Jonathan SuLow-Rank Attention Decoder-Only Language Models
Independent Researcher
The Token Is a Group Element: On Lie-Algebra Attention over Matrix Lie Groups
Jun 18, 2026Przemyslaw MusialskiLie Group Methods Attention Mechanisms
New Jersey Institute of Technology
Attention as Frustrated Synchronization
Jun 17, 2026Joshua NunleyOscillatory Neural Networks Autoregressive Language Modeling
Cognitive Science Program Luddy School of Informatics, Computing, and Engineering Indiana University Bloomington
Hierarchical Attention via Domain Decomposition
Jun 16, 2026Stephan Köhler, Oliver RheinbachLow-Rank Attention PDE Operator Learning
Faculty of Mathematics and Computer Science Technische Universität Bergakademie Freiberg, 09596 Freiberg, Germany
Attention Alignment Between Humans and Vision-Language Models
Jun 16, 2026Isaac R. Christian, Udith Haputhanthrige, Hanna Hornfeld +4VLM Evaluation Visual Attention
Princeton Neuroscience Institute, Princeton University · Department of Psychology, Princeton University · Department of Computer Science, Princeton University +2

Jul 4, 2026, stat.MLJ/K move · Enter open · S save

Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch

Mary Letey, Yue M. Lu, Cengiz Pehlevan, Jacob Zavatone-Veth

John A. Paulson School of Engineering and Applied Sciences, Harvard University · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University · Society of Fellows and Center for Brain Science, Harvard University

In-Context Learning Attention Mechanisms

Open paper AskPDF arXiv

Modern sequence models have a striking capacity for in-context learning (ICL); they can perform new tasks based only on examples given in the prompt. Understanding how this ability emerges requires theory that captures important properties of natural data. Linear regression has served as a useful sandbox for ICL theory, but existing work has largely focused on prompts with independent examples. In this work, we extend this setting to sequentially correlated data, a basic feature of real sequences. We present a solvable model based on linear attention and test our predictions on realistic transformer architectures. We identify two distinct effects: First, when the query token is independent of the context, within-context correlations induce an effective context length: correlated prompts behave like shorter i.i.d. prompts. Second, when the query is also correlated with its context, test error is reduced, particularly for softmax attention when compared to linear attention. These results suggest that correlated prompts alter not only the effective sample size of in-context learning, but also which attention architectures are best matched to the task.

Momentum

Related topics

Latest papers 378