Overview

Transformers represent inputs as tokens and use self-attention to let each token draw on information from other positions, with feed-forward layers refining the resulting representations. Encoder-only models such as BERT, autoregressive decoder-only language models, and encoder-decoder systems are adapted for tasks ranging from text generation and image recognition to forecasting and scientific simulation. Current approaches improve efficiency and capability through local or sparse attention, retrieval and memory, hierarchical processing, and hybrids with convolutional or state-space layers; evaluation spans language-modeling, long-context, vision, and domain-specific benchmarks. A central tension is whether architectural changes and larger models deliver reliable gains in generalization and reasoning while controlling training and inference cost, and theory and interpretability work continues to probe how Transformers learn and compute.

Latest papers

CardsList

583 papers35 papers in the last four weeks, down 53% on the four weeks before. 0.3% of all new papers.

  1. Score-Informed Transformer for Refining MIDI Velocity in Automatic Piano Transcription

    Aug 11, 2025Zhanhong He, Roberto Togneri, Defeng +1Automatic Music TranscriptionTransformer

  2. Symmetry-Aware Transformer Training for Automated Planning

    Aug 11, 2025Markus Fritzsche, Elliot Gestrin, Jendrik SeippContrastive LearningTransformer

  3. MetaTT: A Global Tensor-Train Adapter for Parameter-Efficient Fine-Tuning

    Jun 10, 2025Javier Lopez-Piqueres, Pranav Deshpande, Archan Ray +3TransformerTensor Decomposition

  4. Equivariant Spherical Transformer for Efficient Molecular Modeling

    May 29, 2025Junyi An, Xinyu Lu, Yunfei Shi +6Equivariant GNNsMolecular Representation Learning

  5. Predicting Estimated Times of Restoration for Electrical Outages Using Longitudinal Tabular Transformers

    May 1, 2025Bogireddy Sai Prasanna Teja, Valliappan Muthukaruppan, Carls BenjaminEvent Time PredictionTransformer

  6. Concise One-Layer Transformers Can Perform Dictionary Lookup

    Mar 28, 2025Anton Hampe, Lena Strobl, Dana Angluin +1TransformerTransformer Expressivity

  7. On Generalisation Error Bounds for Transformers

    Oct 15, 2024Lan V. TruongTransformerStatistical Learning Theory

  8. Denoising Graph Super-Resolution towards Improved Collider Event Reconstruction

    Sep 24, 2024Nilotpal Kakati, Etienne Dreyer, Eilam GrossHigh-Energy PhysicsTransformer

  9. Clustering in pure-attention hardmax transformers and its role in sentiment analysis

    Jun 26, 2024Albert Alcalde, Giovanni Fantuzzi, Enrique ZuazuaTransformerTransformer Interpretability

  10. AdvMT: Adversarial Motion Transformer for Long-term Human Motion Prediction

    Jan 10, 2024Sarmad Idrees, Seokman Sohn, Jongeun ChoiTransformerHuman Motion Generation

  11. Introduction to Transformers: an NLP Perspective

    Nov 29, 2023Tong Xiao, Jingbo ZhuTransformerTransformer Attention

  12. Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction

    Mar 9, 2022Yuanhao Cai, Jing Lin, Xiaowan Hu +5Hyperspectral Image ReconstructionImage Reconstruction