Transformer

Ask about this topic

Overview

Transformers represent inputs as tokens and use self-attention to let each token draw on information from other positions, with feed-forward layers refining the resulting representations. Encoder-only models such as BERT, autoregressive decoder-only language models, and encoder-decoder systems are adapted for tasks ranging from text generation and image recognition to forecasting and scientific simulation. Current approaches improve efficiency and capability through local or sparse attention, retrieval and memory, hierarchical processing, and hybrids with convolutional or state-space layers; evaluation spans language-modeling, long-context, vision, and domain-specific benchmarks. A central tension is whether architectural changes and larger models deliver reliable gains in generalization and reasoning while controlling training and inference cost, and theory and interpretability work continues to probe how Transformers learn and compute.

Latest papers

CardsList

544 papers70 papers in the last four weeks, up 71% on the four weeks before. 0.4% of all new papers.

  1. Beyond Muon: MUD (MomentUm Decorrelation) for Faster Transformer Training

    Mar 18, 2026Ben S. Southworth, Stephen ThomasDeep Learning OptimizationMuon Optimizer

    Theoretical Division, Los Alamos National Laboratory, USA. · Lehigh University, Computer Science

  2. The Phasor Transformer: Resolving Attention Bottlenecks on the Unit Circle

    Mar 18, 2026Dibakar SigdelTransformerTime Series Forecasting

    Mindverse Computing LLC, WA 98087

  3. Transformers are Bayesian Networks

    Mar 17, 2026Gregory CoppolaBelief PropagationTransformer

  4. Transformers Can Learn Rules They've Never Seen: Proof of Computation Beyond Interpolation

    Mar 17, 2026Andy GrayTransformerNeural Network Generalization

    Kortical

  5. Long-Horizon Traffic Forecasting via Incident-Aware Conformal Spatio-Temporal Transformers

    Mar 17, 2026Mayur Patil, Qadeer Ahmed, Shawn Midlam-Mohler +4Spatiotemporal TransformersTraffic Forecasting

    Department of Mechanical and Aerospace Engineering and the Center for Automotive Research, The Ohio State University · Ohio Department of Transportation · Honda Research Institute USA, Inc.

  6. Diverging Transformer Predictions for Human Sentence Processing: A Comprehensive Analysis of Agreement Attraction Effects

    Mar 17, 2026Titus von der Malsburg, Sebastian PadóTransformerCognitive Modeling

    University of Stuttgart, Germany

  7. ASCENT: Transformer-Based Aircraft Trajectory Prediction in Non-Towered Terminal Airspace

    Mar 17, 2026Alexander Prutsch, David Schinagl, Horst PosseggerTrajectory PredictionMultimodal Trajectory Prediction

    Institute of Visual Computing, Graz University of Technology, Austria

  8. SignNav: Leveraging Signage for Semantic Visual Navigation in Large-Scale Indoor Environments

    Mar 17, 2026Jian Sun, Yuming Huang, He Li +6Spatiotemporal TransformersVisual Navigation

    Faculty of Science and Technology, University of Macau · Department of Robotics, University of Michigan

  9. Residual Stream Duality in Modern Transformer Architectures

    Mar 17, 2026Yifan ZhangTransformerAttention Mechanisms

  10. A Family of LLMs Liberated from Static Vocabularies

    Mar 16, 2026Aleph Alpha, :, Adnen Abdessaied +35Language ModelingByte-Level Language Model

    Aleph Alpha Research

  11. Learning to Recall with Transformers Beyond Orthogonal Embeddings

    Mar 16, 2026Nuri Mert Vural, Alberto Bietti, Mahdi Soltanolkotabi +1TransformerMemorization in Language Models

    University of Toronto and Vector Institute · Flatiron Institute · University of Southern California +1

  12. Towards Foundation Models for Consensus Rank Aggregation

    Mar 16, 2026Yijun Jin, Simon Klüttermann, Chiara Balestra +1Rank AggregationTransformer

    TU Dortmund University, Dortmund, Germany · Research Center Trustworthy Data Science and Security, University Alliance Ruhr, Dortmund, Germany

  13. Directional Routing in Transformers

    Mar 16, 2026Kevin TaylorAttention RoutingTransformer

  14. Ablate and Rescue: A Causal Analysis of Residual Stream Hyper-Connections

    Mar 16, 2026William Peng, Josheev Rai, Kevin Tseng +2TransformerHyper-Connections

    Stanford University · Georgia Institute of Technology · University of California, Berkeley +2

  15. End-to-End Spatial-Temporal Transformer for Real-time 4D HOI Reconstruction

    Mar 15, 2026Haoyu Zhang, Wei Zhai, Yuhang Yang +2Spatiotemporal Transformers4D Reconstruction

    University of Science and Technology of China

  16. Enhancing Mental Health Classification with Layer-Attentive Residuals and Contrastive Feature Learning

    Mar 14, 2026Menna Elgabry, Ali Hamdi, Khaled ShabanMental HealthTransformer

    Department of Computer Science, Modern Sciences and Arts University, Cairo, Egypt. · Computer Science and Engineering Department, College of Engineering, Qatar University, Doha, Qatar.

  17. MeTok: An Efficient Meteorological Tokenization with Hyper-Aligned Group Learning for Precipitation Nowcasting

    Mar 14, 2026Qizhao Jin, Xianhuang Xu, Yong Cao +2Precipitation NowcastingTransformer

    School of Artificial Intelligence, University of Chinese Academy of Sciences · China Meteorological Administration, National Meteorological Centre · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation

  18. GeoChemAD: Benchmarking Unsupervised Geochemical Anomaly Detection for Mineral Exploration

    Mar 13, 2026Yihao Ding, Yiran Zhang, Chris Gonzalez +2Unsupervised Anomaly DetectionAnomaly Detection

    The University of Western Australia, Perth, WA, Australia

  19. On the Computational Hardness of Transformers

    Mar 11, 2026Barna Saha, Yinzhan Xu, Christopher Ye +1TransformerAttention Mechanisms

    University of California San Diego · Columbia University

  20. Algorithmic Task Capture, Computational Complexity, and Inductive Bias of Infinite Transformers

    Mar 11, 2026Orit Davidovich, Zohar RingelAlgorithmic GeneralizationTransformer

    IBM Research, Haifa, Israel · Racah Institute of Physics, Hebrew University, Jerusalem, Israel

  21. Graph Tokenization for Bridging Graphs and Transformers

    Mar 11, 2026Zeyuan Guo, Enmao Diao, Cheng Yang +1Graph Representation LearningTransformer

    Beijing University of Posts and Telecom., Beijing 100876, China · DreamSoul

  22. Multimodal Self-Attention Network with Temporal Alignment for Audio-Visual Emotion Recognition

    Mar 11, 2026Inyong Koo, yeeun Seong, Minseok Son +2Multimodal Emotion RecognitionAudio-Visual Alignment

    School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea · Graduate School of Green Growth and Sustainability, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea

  23. Lost in the Middle at Birth: An Exact Theory of Transformer Position Bias

    Mar 10, 2026Borun D ChowdhuryLost-in-the-Middle EffectTransformer

    Meta, London, W1T 1FB, UK

  24. Correction of Transformer-Based Models with Smoothing Pseudo-Projector

    Mar 10, 2026Vitaly BulgakovNoise RobustnessTransformer

    Profiteya LLC, Boston, MA, USA · Mass General Brigham, Boston, MA, USA

  25. Routing without Forgetting

    Mar 10, 2026Alessio Masano, Giovanni Bellitto, Dipam Goswani +2Online Continual LearningClass-Incremental Learning

    University of Catania · Department of Computer Science, Universitat Autònoma de Barcelona Computer Vision Center, Barcelona

  26. An Optimal Control Approach To Transformer Training

    Mar 10, 2026Kağan Akman, Naci Saldi, Serdar YükselTransformerTransformer Training Dynamics

    Department of Mathematics, Bilkent University, Ankara, 06800, Turkey · Department of Mathematics and Statistics, Queen’s University, Kingston K7L 3N6, ON, Canada

  27. Decoding the decoder: Contextual sequence-to-sequence modeling for intracortical speech decoding

    Mar 10, 2026Michal Olak, Tommaso Boccato, Matteo FerranteSequence-to-Sequence ModelsNeural Decoding

    Tether Evo

  28. The Radio-Frequency Transformer for Signal Separation

    Mar 10, 2026Egor Lifar, Semyon Savkin, Rachana Madhukara +3Finite Scalar QuantizationTransformer

    Massachusetts Institute of Technology (MIT).

  29. Exclusive Self Attention

    Mar 10, 2026Shuangfei ZhaiSelf-AttentionTransformer

    Apple

  30. Adaptive Loops and Memory in Transformers: Think Harder or Know More?

    Mar 9, 2026Markus Frey, Behzad Shomali, Ali Hamza Bashir +3TransformerMemory-Augmented Language Models

    Lamarr Institute · Fraunhofer IAIS · University of Bonn