Transformer

Ask about this topic

Overview

Transformers represent inputs as tokens and use self-attention to let each token draw on information from other positions, with feed-forward layers refining the resulting representations. Encoder-only models such as BERT, autoregressive decoder-only language models, and encoder-decoder systems are adapted for tasks ranging from text generation and image recognition to forecasting and scientific simulation. Current approaches improve efficiency and capability through local or sparse attention, retrieval and memory, hierarchical processing, and hybrids with convolutional or state-space layers; evaluation spans language-modeling, long-context, vision, and domain-specific benchmarks. A central tension is whether architectural changes and larger models deliver reliable gains in generalization and reasoning while controlling training and inference cost, and theory and interpretability work continues to probe how Transformers learn and compute.

Latest papers

CardsList

544 papers70 papers in the last four weeks, up 71% on the four weeks before. 0.4% of all new papers.

  1. Tiny Autoregressive Recursive Models

    Mar 9, 2026Paulius Rauba, Claudio Fanconi, Mihaela van der SchaarRecursive Language ModelsIterative Refinement

    University of Cambridge

  2. The Dual-Stream Transformer: Channelized Architecture for Interpretable Language Modeling

    Mar 8, 2026J. Clayton Kerce, Alexis FoxTransformerTransformer Interpretability

    Georgia Tech Research Institute

  3. Discrete Tokenization Unlocks Transformers for Calibrated Tabular Forecasting

    Mar 8, 2026Yael S. ElmatadTransformerProbability Calibration

  4. OrthoFormer: Instrumental Variable Estimation in Transformer Hidden States via Neural Control Functions

    Mar 8, 2026Charles LuoTransformerInstrumental Variable Estimation

    Metro State University

  5. A prior information informed learning architecture for flying trajectory prediction

    Mar 6, 2026Xianda Huang, Zidong Han, Ruibo Jin +4Trajectory PredictionTransformer

    Southern University of Science and Technology, Shenzhen, China · Chinese University of Hong Kong-Shenzhen, Shenzhen, China

  6. NOBLE: Accelerating Transformers with Nonlinear Low-Rank Branches

    Mar 6, 2026Ethan SmithTransformer

    Canva Research

  7. Random Quadratic Form on a Sphere: Synchronization by Common Noise

    Mar 6, 2026Maximilian Engel, Anna ShalovaStochastic Differential EquationsDynamical Systems

    Korteweg-de Vries Institute for Mathematics, University of Amsterdam · Department of Mathematics and Computer Science, FU Berlin

  8. Structured Multidimensional Representation Learning for Large Language Models

    Mar 5, 2026Alaa El Ichi, Khalide Jbilou, Mohamed El Guide +1TransformerEmbedding Compression

    Universit´e du Littoral Cote d’Opale, LMPA, 50 rue F. Buisson, 62228 Calais-Cedex, France. · FGSES, University Mohammed VI Polytechnic, Rabat, Morocco · LISIC, 50 rue F. Buisson, Universit´e du Littoral Cote d’Opale, 62228 Calais-Cedex, France.

  9. Feature Resemblance: Towards a Theoretical Understanding of Analogical Reasoning in Transformers

    Mar 5, 2026Ruichen Xu, Wenjing Yan, Ying-Jun Angela ZhangAnalogical ReasoningAnalogical Reasoning for Language Models

    Department of Information Engineering, The Chinese University of Hong Kong, Hong Kong.

  10. Measuring the Redundancy of Decoder Layers in SpeechLLMs

    Mar 5, 2026Adel Moumen, Guangzhi Sun, Philip C WoodlandLLM PruningTransformer

    Department of Engineering, University of Cambridge, UK

  11. IntSeqBERT: Learning Arithmetic Structure in OEIS via Modulo-Spectrum Embeddings

    Mar 5, 2026Kazuhisa NakashoSequence ModelingTransformer

    Iwate Prefectural University, Takizawa, Iwate, Japan

  12. EgoPoseFormer v2: Accurate Egocentric Human Motion Estimation for AR/VR

    Mar 4, 2026Zhenyu Li, Sai Kumar Dwivedi, Filip Maric +11Human Pose Estimation3D Human Pose Estimation

    Meta · KAUST · Max Planck Institute for Intelligent Systems

  13. N-gram Injection into Transformers for Dynamic Language Model Adaptation in Handwritten Text Recognition

    Mar 4, 2026Florent Meyer, Laurent Guichard, Yann Soullard +3Inference-Time Adaptation of Language ModelsLanguage Modeling

    ANTAI, Rennes, France · Univ Rennes, CNRS, IRISA - UMR 6074, Rennes, France

  14. Retrievit: In-context Retrieval Capabilities of Transformers, State Space Models, and Hybrid Architectures

    Mar 3, 2026Georgios Pantazopoulos, Malvina Nikandrou, Ioannis Konstas +1In-Context RetrievalTransformer

    University of Edinburgh · Heriot-Watt University

  15. Detecting Transportation Mode Using Dense Smartphone GPS Trajectories and Transformer Models

    Feb 27, 2026Yuandong Zhang, Othmane Echchabi, Tianshu Feng +3Intelligent Transportation SystemsTransformer

    University of California San Diego, CA, USA · McGill University Mila - Quebec AI Institute Montréal, QC, Canada · University of Pennsylvania Philadelphia, PA, USA +1

  16. Transformers are Stateless Differentiable Neural Computers

    Feb 27, 2026Bo Tang, Weiwei XieTransformerMemory-Augmented Neural Networks

    Department of Electrical and Computer Engineering, Worcester Polytechnic Institute, Worcester, MA 01609 · Department of Data Science, Worcester Polytechnic Institute, Worcester, MA 01609

  17. Benchmarking BERT-based Models for Sentence-level Topic Classification in Nepali Language

    Feb 27, 2026Nischal Karki, Bipesh Subedi, Prakash Poudyal +2Bidirectional Encoder Representations from TransformersText Classification

    Information and Language Processing Research Lab, Kathmandu University, Kavre, Nepal

  18. Optimizer-Induced Low-Dimensional Drift and Transverse Dynamics in Transformer Training

    Feb 27, 2026Yongzhong XuTransformer Training DynamicsNeural Network Training Dynamics

  19. ThreatFormer-IDS: Robust Transformer Intrusion Detection with Zero-Day Generalization and Explainable Attribution

    Feb 26, 2026Srikumar NayakExplainable Intrusion DetectionIoT Intrusion Detection

    Incedo Inc, USA

  20. MetaOthello: A Controlled Study of Multiple World Models in Transformers

    Feb 26, 2026Aviral Chawla, Galen Hall, Juniper LovatoTransformer InterpretabilityWorld Model Learning

    Vermont Complex Systems Institute, University of Vermont, Burlington, VT · University of Michigan, Ann Arbor, MI

  21. Residual Koopman Spectral Profiling for Predicting and Preventing Transformer Training Instability

    Feb 26, 2026Bum Jun Kim, Shohei Taniguchi, Makoto Kawano +2TransformerSpectral Regularization

    Graduate School of Engineering, The University of Tokyo, Japan

  22. Trie-Aware Transformers for Generative Recommendation

    Feb 25, 2026Zhenxiang Xu, Jiawei Chen, Sirui Chen +5Generative RecommendationSequential Recommendation

    State Key Laboratory of Blockchain and Data Security, Zhejiang University. · College of Computer Science and Technology, Zhejiang University. · Zhejiang University Hangzhou, China +2

  23. Natural Language Processing Models for Robust Document Categorization

    Feb 23, 2026Radoslaw Roszczyk, Pawel Tecza, Maciej Stodolski +1Text ClassificationTransformer

    Faculty of Electrical Engineering, Warsaw University of Technology pl. Politechniki 1, 00-661 Warsaw · Student of Warsaw University of Technology, pl. Politechniki 1, 00-661 Warsaw

  24. Incremental Transformer Neural Processes

    Feb 21, 2026Philip Mortimer, Cristiana Diaconu, Tommy Rochussen +2Streaming InferenceTransformer

    University of Cambridge · Helmholtz AI · Technical University of Munich +2

  25. Toward Manifest Relationality in Transformers via Symmetry Reduction

    Feb 21, 2026J. François, L. RaveraTransformerTransformer Attention

    Mons University (UMONS), 20 Place du Parc, 7000 Mons, Belgium. · Grupo de Investigación en Física Teórica, Universidad Católica De La Santísima Concepción, Alonso de Ribera 2850, Concepción, Chile.

  26. Subgroups of U(d)U(d) Induce Natural RNN and Transformer Architectures

    Feb 20, 2026Joshua NunleyRecurrent Neural NetworksTransformer

    Cognitive Science Program & Department of Informatics, Indiana University

  27. On the "Induction Bias" in Sequence Models

    Feb 20, 2026M. Reza Ebrahimi, Michaël Defferrard, Sunny Panchal +1TransformerState Tracking

    Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.

  28. Turbo Connection: Reasoning as Information Flow from Higher to Lower Layers

    Feb 20, 2026Mohan Tang, Sidi LuTransformerLarge Reasoning Models

    UCLA.

  29. Deeper detection limits in astronomical imaging using self-supervised spatiotemporal denoising

    Feb 19, 2026Yuduo Guo, Hao Zhang, Mingyu Li +10Image DenoisingSelf-Supervised Image Denoising

    Department of Automation, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, China · Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China · Department of Astronomy, Tsinghua University, Beijing, China +6

  30. Structure-Aware Set Transformers: Temporal and Variable-Type Attention Biases for Asynchronous Clinical Time Series

    Feb 18, 2026Joohyung Lee, Kwanhyung Lee, Changhun Kim +1Set TransformerIrregular Time-Series Modeling

    AITRICS · Korea Advanced Institute of Science and Technology (KAIST)