Transformer Attention

Follow this topic Ask about this topic

Transformer attention uses queries, keys, and values to decide which tokens should contribute information to one another, with positional encodings and masks shaping those interactions. Work seeks both to improve this basic mechanism and to reduce its costs, using approaches such as sparse, linear, kernel, and local attention, as well as efficient GPU kernels and compressed key-value caches. Researchers also add structural or domain-specific biases and study attention’s role in learning, retrieval, stability, and interpretability across language, vision, graphs, and time series. A central tension remains: methods that reduce computation or constrain interactions can improve speed and memory use, but may lose accuracy or long-range retrieval quality, and no single design is best across settings.

Latest papers 308

41 papers in the last four weeks, up 86% on the four weeks before. 0.2% of all new papers.

All topics

Quantum Adaptive Self-Attention for Quantum Transformer Models
Apr 5, 2025Chi-Sheng Chen, En-Jui KuoHybrid Quantum-Classical ML Transformer Attention
Department of Electrophysics, National Yang Ming Chiao Tung University, Hsinchu, Taiwan
Numerical stability analysis of large language models
Mar 13, 2025Stanislav Budzinskiy, Wenyi Fang, Longbin Zeng +1Transformer Inference Transformer Attention
Faculty of Mathematics, University of Vienna, Kolingasse 14-16, 1090, Vienna, Austria · Huawei Technologies
Fewer yet critical: Reducing Redundant Token Dependencies for Transformer-based Time Series Forecasting
Mar 10, 2025Jianqi Zhang, Yuchan Liu, Zeen Song +2Time Series Forecasting Transformer Attention
National Key Laboratory of Space Integrated Information Systems, Institute of Software, Chinese Academy of Sciences, Beijing, China 1 · University of Chinese Academy of Sciences, Beijing, China 2 · Digital-Intelligent Operations Center, Guizhou Power Grid Co., Ltd., Guizhou, China 3 +1
You Do Not Fully Utilize Transformer's Representation Capacity
Feb 13, 2025Gleb Gerasimov, Yaroslav Aksenov, Nikita Balagansky +2Representation Collapse Memory-Augmented Language Models
1T-Tech · 2Moscow Institute of Physics and Technology · 3HSE University
OrientedFormer: An End-to-End Transformer-Based Oriented Object Detector in Remote Sensing Images
Sep 29, 2024Zeyu Ding, Yong Zhou, Jiaqi Zhao +4Remote Sensing Image Understanding Detection Transformer
School of Computer Science and Technology, China University of Mining and Technology, and with Mine Digitization Engineering Research Center of the Ministry of Education, and also with Innovation Research Center of Disaster Intelligent Prevention and Emergency Rescue, China University of Mining and Technology, Xuzhou 221116, China · School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, ON K1N 6N5, Canada
Efficient Concertormer for Image Deblurring and Beyond
Apr 9, 2024Pin-Hung Kuo, Jinshan Pan, Shao-Yi Chien +1Image Deblurring Transformer Attention
DiDA: Video Object Segmentation with Distillation Learning of Deformable Attention
Jan 25, 2024Quang-Trung Truong, Duc Thanh Nguyen, Binh-Son Hua +1Video Object Segmentation Transformer Attention
Hong Kong University of Science and Technology · Deakin University · Trinity College Dublin
Introduction to Transformers: an NLP Perspective
Nov 29, 2023Tong Xiao, Jingbo ZhuTransformer Transformer Attention
NLP Lab., Northeastern University, Shenyang, China · NiuTrans Research, Shenyang, China

Apr 5, 2025J/K move · Enter open · S save

Quantum Adaptive Self-Attention for Quantum Transformer Models

Chi-Sheng Chen, En-Jui Kuo

Department of Electrophysics, National Yang Ming Chiao Tung University, Hsinchu, Taiwan

Hybrid Quantum-Classical ML Transformer Attention Variational Quantum Circuits

Open paper AskPDF arXiv

A recurring weakness in quantum machine learning (QML) is that reported ``quantum advantages'' are seldom tested against a capacity-matched classical control, leaving it unclear whether a gain comes from the quantum substrate or from the architectural change that accompanies it. Our primary contribution is methodological: a protocol for attributing such gains honestly -- a capacity-matched classical bottleneck of identical parameter budget, transparent reporting of where quantum does not help, and validation on real quantum hardware -- which we develop and apply through a concrete case study. That case study is Quantum Adaptive Self-Attention (QASA), a hybrid Transformer that replaces the value projection of a single encoder layer with a 36-parameter parameterized quantum circuit (PQC), keeping all other layers classical. Across nine synthetic benchmarks and the real-world ETTh1 dataset, QASA improves on a full-capacity classical Transformer for chaotic and trend-dominated signals. To ask whether this is a genuinely quantum effect, we introduce a control rarely applied in quantum machine learning -- a capacity-matched classical bottleneck with the same parameter budget -- and find that it matches the PQC on the error metrics. The gain is therefore attributable to the low-rank value-projection bottleneck (an architectural parsimony principle), not to quantumness; adding further quantum layers only degrades performance and trainability. We accordingly position the quantum layer not as a source of accuracy advantage but as a competitive instantiation of this principle: its low-rank compression onto the signal's intrinsic dimensionality is matched by a classical bottleneck, so the gain is architectural rather than quantum.