Transformer

Ask about this topic

Overview

Transformers represent inputs as tokens and use self-attention to let each token draw on information from other positions, with feed-forward layers refining the resulting representations. Encoder-only models such as BERT, autoregressive decoder-only language models, and encoder-decoder systems are adapted for tasks ranging from text generation and image recognition to forecasting and scientific simulation. Current approaches improve efficiency and capability through local or sparse attention, retrieval and memory, hierarchical processing, and hybrids with convolutional or state-space layers; evaluation spans language-modeling, long-context, vision, and domain-specific benchmarks. A central tension is whether architectural changes and larger models deliver reliable gains in generalization and reasoning while controlling training and inference cost, and theory and interpretability work continues to probe how Transformers learn and compute.

Latest papers

CardsList

544 papers70 papers in the last four weeks, up 71% on the four weeks before. 0.4% of all new papers.

  1. Transfer Learning for Loan Recovery Prediction under Distribution Shifts with Heterogeneous Feature Spaces

    Apr 3, 2026Christopher Gerling, Hanqiu Peng, Ying Chen +1Cross-Domain Transfer LearningDomain Adaptation

    Humboldt-Universität zu Berlin, Unter den Linden 6, 10099 Berlin, Germany · National University of Singapore, 10 Lower Kent Ridge Road, Singapore 119076, Singapore

  2. Crystalite: A Lightweight Transformer for Efficient Crystal Modeling

    Apr 2, 2026Tin Hadži Veljković, Joshua Rosenthal, Ivor Lončarić +1Diffusion TransformerTransformer

    1UvA-Bosch Delta Lab · University of Amsterdam · 3Ruđer Bošković Institute

  3. Homogenized Transformers

    Apr 2, 2026Hugo Koubbi, Borjan Geshkovski, Philippe RigolletTransformerMulti-Head Attention

  4. Physics-Informed Transformer for Multi-Band Channel Frequency Response Reconstruction

    Apr 2, 2026Anatolij Zubow, Joana Angjo, Sigrid Dimce +1Channel EstimationTransformer

    School of Electrical Engineering and Computer Science, TU Berlin, Berlin, Germany

  5. DDCL-INCRT: A Self-Organising Transformer with Hierarchical Prototype Structure (Theoretical Foundations)

    Apr 2, 2026Giansalvo CirrincioneTransformerHierarchical Representation Learning

    Laboratory LTI, Université de Picardie Jules Verne, Chemin du Thil, Amiens, 80025, France

  6. Universal YOCO for Efficient Depth Scaling

    Apr 1, 2026Yutao Sun, Li Dong, Tianzhu Ye +3Efficient Language Model InferenceInference-Time Scaling

    Microsoft Research · Tsinghua University

  7. BioCOMPASS: Integrating Biomarkers into Transformer-Based Immunotherapy Response Prediction

    Apr 1, 2026Sayed Hashim, Frank Soboczenski, Paul CairnsCancer Drug Response PredictionClinical Outcome Prediction

    University of York, UK

  8. A Benchmark of State-Space Models vs. Transformers and BiLSTM-based Models for Historical Newspaper OCR

    Apr 1, 2026Merveilles Agbeti-Messan, Pierrick Tranouez, Stéphane Nicolas +2Selective SSMsTransformer

    LITIS EA4108, University of Rouen Normandy, France · LITIS EA4108, INSA of Rouen Normandy, France

  9. ARGS: Auto-Regressive Gaussian Splatting via Parallel Progressive Next-Scale Prediction

    Apr 1, 2026Quanyuan Ruan, Kewei Shi, Jiabao Lei +2Autoregressive 3D Scene Generation3D Asset Generation

    South China University of Technology · The University of Hong Kong · The Chinese University of Hong Kong, Shenzhen +1

  10. A Dual-Stream Transformer Architecture for Illumination-Invariant TIR-LiDAR Person Tracking

    Apr 1, 2026Yuki Minase, Kanji TanakaVisual Object TrackingRobotic Perception

    Yuki Minase and Kanji Tanaka are with the Graduate School of Engineering, University of Fukui, Japan.

  11. GenoBERT: A Language Model for Accurate Genotype Imputation

    Mar 31, 2026Lei Huang, Chuan Qiu, Kuan-Jui Su +13GenomicsTransformer

    School of Computing Sciences and Computer Engineering, University of Southern Mississippi, Hattiesburg, MS, USA · Tulane Center for Biomedical Informatics and Genomics, Deming Department of Medicine, School of Medicine, Tulane University, New Orleans, LA, USA · Kennesaw State University, Atlanta, GA, USA +3

  12. Variational Neurons in Transformers for Language Modeling

    Mar 30, 2026Yves RuffenachLLM Uncertainty EstimationTransformer

    Conservatoire National des Arts et Métiers Strasbourg, France

  13. HRIR-Former: Grid-Free Time-Domain Reconstruction of Head-Related Impulse Responses with a Spatially Encoded Transformer

    Mar 30, 2026Shaoheng Xu, Chunyi Sun, Jihui Zhang +4Spatial AudioTransformer

    The Australian National University, Australia · The University of Queensland, Australia

  14. ExFusion: Efficient Transformer Training via Multi-Experts Fusion

    Mar 30, 2026Jiacheng Ruan, Daize Dong, Xiaoye Qu +5TransformerMixture of Experts

    School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, No 800, Dongchuan Road, Minhang District, Shanghai, China, 200240 · Shanghai Artificial Intelligence Laboratory, Shanghai, China, 200240 · Huazhong University of Science and Technology, Hubei, China, 430074 +3

  15. ImmSET: Sequence-Based Predictor of TCR-pMHC Specificity at Scale

    Mar 27, 2026Marco Garcia Noceda, Matthew T Noakes, Andrew FigPope +3Protein-Protein Interaction PredictionSequence Modeling

    Adaptive Biotechnologies, Seattle, WA, USA

  16. A Boltzmann-machine-enhanced Transformer For DNA Sequence Classification

    Mar 27, 2026Zhixuan Cao, Yishu Xu, Xuang WUTransformerTransformer Attention

    Tsinghua University · University of California, Berkeley

  17. Clash of the models: Comparing performance of BERT-based variants for generic news frame detection

    Mar 27, 2026Vihang JumlePolitical CommunicationTransformer

  18. Anchored-Branched Steady-state WInd Flow Transformer (AB-SWIFT): a metamodel for 3D atmospheric flow in urban environments

    Mar 26, 2026Armand de Villeroché, Rem-Sophia Mouradi, Vincent Le Guen +5Neural Surrogate ModelingPDE Surrogate Modeling

    CEREA, ENPC, EDF R&D, Institut Polytechnique de Paris, Île-de-France, France · EDF R&D, Île-de-France, France · SINCLAIR AI Laboratory, Saclay, Île-de-France, France +1

  19. Transformers in the Dark: Navigating Unknown Search Spaces via Bandit Feedback

    Mar 25, 2026Jungtaek Kim, Thomas Zeng, Ziqian Lin +5Bandit LearningTree Search

    University of Wisconsin–Madison · FuriosaAI · Yonsei University +1

  20. Transformers for Program Termination

    Mar 25, 2026Yoav Alon, Cristina DavidStatic Code AnalysisTransformer

    University of Bristol UK

  21. Circuit Complexity of Hierarchical Knowledge Tracing and Implications for Log-Precision Transformers

    Mar 25, 2026Naiming Liu, Richard Baraniuk, Shashank SonkarCircuit ComplexityKnowledge Tracing

    Rice University · University of Central Florida

  22. Transformers Trained via Gradient Descent Can Provably Learn a Class of Teacher Models

    Mar 24, 2026Chenyang Zhang, Qingyue Zhao, Quanquan Gu +1TransformerStatistical Learning Theory

    The University of Hong Kong · University of California, Los Angeles

  23. Sharper Generalization Bounds for Transformer

    Mar 23, 2026Yawen Li, Tao Hu, Zhouhui Lian +4Generalization BoundsTransformer

    School of Mathematical Sciences, Capital Normal University, 100048 Beijing, China · Wangxuan Institute of Computer Technology, Peking University, China, 100871 · Advanced Institute of Information Technology, Peking University +4

  24. Stream separation improves Bregman conditioning in transformers

    Mar 22, 2026James Clayton KerceTransformerRepresentation Geometry in Language Models

    Georgia Tech Research Institute

  25. Sonny: Breaking the Compute Wall in Medium-Range Weather Forecasting

    Mar 22, 2026Minjong CheonWeather ForecastingSpatiotemporal Forecasting

    Department of Computer Science and Engineering, Sejong University, Seoul, South Korea

  26. Neutrino Oscillation Parameter Estimation Using Structured Hierarchical Transformers

    Mar 21, 2026Giorgio Morales, Gregory Lehaut, Antonin Vacheret +2Parameter EstimationHigh-Energy Physics

    Normandie Univ, UNICAEN, ENSICAEN, CNRS, GREYC. Caen, France · Normandie Univ, ENSICAEN, UNICAEN, CNRS/IN2P3, LPC Caen. Caen, France

  27. Memory-Efficient Fine-Tuning Diffusion Transformers via Dynamic Patch Sampling and Block Skipping

    Mar 21, 2026Sunghyun Park, Jeongho Kim, Hyoungwoo Park +6Memory-Efficient Fine-TuningDiffusion Model Fine-Tuning

    Qualcomm AI Research · KAIST

  28. On the Ability of Transformers to Verify Plans

    Mar 20, 2026Yash Sarrof, Yupei Du, Katharina Stein +3TransformerSymbolic Planning

    Saarland University · Australian National University · University of Toulouse / LAAS-CNRS

  29. Anatomical Heterogeneity in Transformer Language Models

    Mar 19, 2026Tomasz WietrzykowskiTransformerEfficient Language Model Training

    Independent Researcher, Wroclaw, Poland

  30. Transformers Learn Robust In-Context Regression under Distributional Uncertainty

    Mar 19, 2026Hoang T. H. Cao, Hai D. V. Trinh, Tho Quan +1In-Context LearningTransformer

    Faculty of Computer Science and Engineering, Ho Chi Minh University of Technology (HCMUT), Vietnam National University Ho Chi Minh City (VNU-HCM), Vietnam