Transformer

Ask about this topic

Overview

Transformers represent inputs as tokens and use self-attention to let each token draw on information from other positions, with feed-forward layers refining the resulting representations. Encoder-only models such as BERT, autoregressive decoder-only language models, and encoder-decoder systems are adapted for tasks ranging from text generation and image recognition to forecasting and scientific simulation. Current approaches improve efficiency and capability through local or sparse attention, retrieval and memory, hierarchical processing, and hybrids with convolutional or state-space layers; evaluation spans language-modeling, long-context, vision, and domain-specific benchmarks. A central tension is whether architectural changes and larger models deliver reliable gains in generalization and reasoning while controlling training and inference cost, and theory and interpretability work continues to probe how Transformers learn and compute.

Latest papers

CardsList

544 papers70 papers in the last four weeks, up 71% on the four weeks before. 0.4% of all new papers.

  1. The EΔΔ-MHC-Geo Transformer: Adaptive Geodesic Operations with Guaranteed Orthogonality

    May 7, 2026Arash ShahmansooriTransformerResidual Learning

    Independent Researcher

  2. Large Vision-Language Models Get Lost in Attention

    May 7, 2026Gongli Xi, Ye Tian, Mengyu Yang +5Transformer FFNsVision-Language Models

    School of Cyberspace Security, Beijing University of Posts and Telecommunications, Beijing, China · State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China · School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications, Beijing, China +3

  3. Average Attention Transformers and Arithmetic Circuits

    May 6, 2026Lena Ehrmuth, Laura StriekerTransformer ExpressivityTransformer

    Institute for Theoretical Computer Science, Leibniz University Hanover, Hanover, Germany

  4. FLUID: Continuous-Time Hyperconnected Sparse Transformer for Sink-Free Learning

    May 6, 2026Waleed Razzaq, Yun-Bo ZhaoTransformerTransformer Attention

    School of Automation University of Science and Technology China Hefei, Anhui

  5. Domain-Adaptive Arrhythmia Classification Using a Hybrid Transformer on Wearable Heart Signals

    May 5, 2026Maedeh H. Toosi, Siamak MohammadiTransformerDomain Adaptation

    School of Electrical and Computer Engineering, University of Tehran, Tehran, Iran

  6. Feature-Augmented Transformers for Robust AI-Text Detection Across Domains and Generators

    May 5, 2026Mohamed Mady, Johannes Reschke, Björn SchullerAI-Generated Text DetectionTransformer

    Chair of Health Informatics, Technical University of Munich (TUM), Germany · OTH Regensburg, Germany · GLAM – Group on Language, Audio & Music, Imperial College London, United Kingdom

  7. SATFormer: Learning When to Use Early Value Residuals in Transformers

    May 5, 2026Skye Gunasekaran, Téa Wright, Rui-Jie Zhu +1TransformerResidual Learning

    UC Santa Cruz · UC Berkeley

  8. A Systematic Benchmark of Machine Transliteration Models for the Tajik-Farsi Language Pair: A Comparative Study from Rule-Based to Transformer Architectures

    May 4, 2026Mullosharaf K. ArabovMultilingual Language ModelsTransformer

    Institute of Computational Mathematics and Information Technologies Kazan Federal University Kazan, Russia

  9. Projection-Free Transformers via Gaussian Kernel Attention

    May 4, 2026Debarshi Kundu, Archisman Ghosh, Swaroop Ghosh +1TransformerSelf-Attention

    The Pennsylvania State University, University Park, USA

  10. Linear-Time Global Visual Modeling without Explicit Attention

    May 3, 2026Ruize He, Dongchen Han, Gao HuangTransformerTransformer Attention

    Tsinghua University

  11. Characterizing the Expressivity of Local Attention in Transformers

    May 1, 2026Jiaoda Li, Ryan CotterellTransformer ExpressivityTransformer

  12. E2^2DT: Efficient and Effective Decision Transformer with Experience-Aware Sampling for Robotic Manipulation

    Apr 30, 2026Kaiyan Zhao, Borong Zhang, Yiming Wang +4Long-Horizon Robotic ManipulationTransformer

    School of Computer Science, Wuhan University, Wuhan 430072, China · State Key Lab of Internet of Things for Smart City and Department of Computer Information Science, University of Macau, Macao 999078, China

  13. Hierarchical Fault Detection and Diagnosis for Transformer Architectures

    Apr 30, 2026Sigma Jahan, Saurabh Singh Rajput, Tushar Sharma +1Software EngineeringAutomated Software Testing

    Dalhousie University, Halifax, Canada

  14. Sampling two-dimensional spin systems with transformers

    Apr 30, 2026Piotr Białas, Piotr Korcyl, Tomasz Stebel +2TransformerIsing Model

    Institute of Applied Computer Science, Jagiellonian University, ul. Łojasiewicza 11, 30-348 Kraków, Poland · Institute of Theoretical Physics, Jagiellonian University, ul. Łojasiewicza 11, 30-348 Kraków, Poland · Doctoral School of Exact and Natural Sciences, Jagiellonian University, ul. Łojasiewicza 11, 30-348 Kraków, Poland

  15. PINN-Cast: Exploring the Role of Continuous-Depth NODE in Transformers and Physics Informed Loss as Soft Physical Constraints in Short-term Weather Forecasting

    Apr 30, 2026Hira Saleem, Flora Salim, Cormac PurcellTransformerPhysics-Informed ML

    University of New South Wales, Sydney, NSW, Australia

  16. Learning Rate Transfer in Normalized Transformers

    Apr 29, 2026Boris Shigida, Boris Hanin, Andrey GromovTransformerHyperparameter Transfer

    Meta Superintelligence Labs · Princeton University

  17. Stochastic Scaling Limits and Synchronization by Noise in Deep Transformer Models

    Apr 29, 2026Andrea Agazzi, Giuseppe Bruno, Eloy Mosig García +2Dynamical SystemsTransformer

    Department of Mathematics and Statistics, University of Bern (CH) · Department of Mathematics, University of Pisa (IT)

  18. Exploring the Potential of Probabilistic Transformer for Time Series Modeling: A Report on the ST-PT Framework

    Apr 29, 2026Zhangzhi Xiong, Haoyi Wu, You Wu +3TransformerConditional Random Fields

    School of Information Science and Technology, ShanghaiTech University

  19. Folding Tensor and Sequence Parallelism for Memory-Efficient Transformer Training & Inference

    Apr 29, 2026Vasu Shyam, Anna Golubeva, Quentin AnthonyTransformer InferenceTransformer

    Zyphra

  20. Efficient and Interpretable Transformer for Counterfactual Fairness

    Apr 29, 2026Panyi Dong, Zhiyu QuanTransformer InterpretabilityTransformer

    Acturial and Risk Mangament Sciences, University of Illinois Urbana-Champaign, Champaign, Illinois, USA.

  21. Robustness of Transformer-Based Fluence Map Prediction Under Clinically Realistic Perturbations

    Apr 28, 2026Ujunwa Mgboh, Rafi Ibn Sultan, Joshua Kim +2Distribution Shift RobustnessTransformer

    Wayne State University, Detroit, MI, USA · Henry Ford Health, Detroit, MI

  22. Investigation into In-Context Learning Capabilities of Transformers

    Apr 28, 2026Rushil Chandrupatla, Leo Bangayan, Sebastian LengNeural Network GeneralizationTransformer

  23. StratFormer: Adaptive Opponent Modeling and Exploitation in Imperfect-Information Games

    Apr 28, 2026Andy Caen, Mark H. M. Winands, Dennis J. N. J. SoemersOpponent ModelingGame-Playing Agents

    Department of Advanced Computing Sciences, Maastricht University, Maastricht, The Netherlands

  24. GEGLU-Transformer for IMU-to-EMG Estimation with Few-Shot Adaptation

    Apr 28, 2026Miroljub Mihailovic, Luca Tonin, Stefano Tortora +1TransformerWearable Sensing

    Department of Information Engineering, University of Padua, 35131 Padua, Italy

  25. Scaling Probabilistic Transformer via Efficient Cross-Scale Hyperparameter Transfer

    Apr 28, 2026Penghao Kuang, Haoyi Wu, Kewei TuTransformerNeural Network Optimization

    School of Information Science and Technology, ShanghaiTech University

  26. Towards Training Transformers as a Universal Computer

    Apr 28, 2026Ruize Xu, Chenxiao Yang, Yanhong Li +1TransformerAlgorithmic Reasoning

    The University of Chicago · Toyota Technological Institute at Chicago · Allen Institute for AI

  27. ADE: Adaptive Dictionary Embeddings -- Scaling Multi-Anchor Representations to Large Language Models

    Apr 27, 2026Orhan Demirci, Sezer Aptourachman, Aydın KayaText EmbeddingsWord Embeddings

    Department of Computer Engineering Hacettepe University Ankara, Turkey

  28. A systematic literature Review for Transformer-based Software Vulnerability detection

    Apr 27, 2026Fiza Naseer, Javed Ali Khan, Muhammad Yaqoob +2TransformerSoftware Vulnerability Detection

  29. Graph Memory Transformer (GMT)

    Apr 26, 2026Nicola Zanarini, Niccolò Ferrari, Evelina LammaMemory-Augmented Neural NetworksTransformer FFNs

    Bonfiglioli Engineering s.r.l. · Department of Engineering, University of Ferrara, Via Saragat 1, 44122 Ferrara, Italy · NAIS s.r.l., Via Maria Callas 4, 40131 Bologna, Italy

  30. Transformer as an Euler Discretization of Score-based Variational Flow

    Apr 26, 2026Huadong LiaoRepresentation LearningTransformer