Overview

Transformers represent inputs as tokens and use self-attention to let each token draw on information from other positions, with feed-forward layers refining the resulting representations. Encoder-only models such as BERT, autoregressive decoder-only language models, and encoder-decoder systems are adapted for tasks ranging from text generation and image recognition to forecasting and scientific simulation. Current approaches improve efficiency and capability through local or sparse attention, retrieval and memory, hierarchical processing, and hybrids with convolutional or state-space layers; evaluation spans language-modeling, long-context, vision, and domain-specific benchmarks. A central tension is whether architectural changes and larger models deliver reliable gains in generalization and reasoning while controlling training and inference cost, and theory and interpretability work continues to probe how Transformers learn and compute.

Latest papers

CardsList

583 papers35 papers in the last four weeks, down 53% on the four weeks before. 0.3% of all new papers.

  1. Multi-View Transformer with Geometry-Guided Multi-Modal ControlNet for 3D Virtual Try-On

    Nov 19, 2026Sarthak Batra, Armin Mustafa, Adrian HiltonVirtual Try-OnMulti-View Consistency

  2. NIMIS: A Structure-Agnostic, Chemically Informed Slot-Attention Framework for Demixing 1D 1^1H-NMR Mixture Spectra

    Nov 19, 2026Anna Marija Sumrova, Jonathan M. GoodmanSpectral UnmixingMagnetic Resonance Spectroscopy

  3. Antom 3-in-1 Transformer: A Payment Foundation Model

    Nov 19, 2026Weiyu Han, Haoyu Jiang, Jie Sheng +21Financial Fraud DetectionTransformer

  4. TPHC: Improving Hyper-Connections with Tensor-Parallel Grouped Updates

    Nov 17, 2026Xinying Hu, You Wu, Haoyi Wu +4Hyper-ConnectionsTransformer

  5. A Transformer-Derived Iterative Preconditioner

    Nov 17, 2026Patrick Lutz, Themistoklis Haris, Aditya Gangrade +1Transformer

  6. In-Context Estimation and Prediction for Parametric ODEs

    Nov 16, 2026Abdullah Ahmed, Yun YangParameter EstimationIn-Context Learning

  7. ACiT: Proprioceptive-Prior Action Chunking with Inverted Transformers

    Nov 16, 2026Haixiang Wu, Zhihao Guan, Yang YangRobot Policy LearningVisuomotor Policy Learning

  8. Multi-Branch Transformer

    Nov 15, 2026Gaotang Li, Lingjie Chen, Chunyuan Deng +6TransformerLanguage Model Scaling

  9. Equilibrium measures for anisotropic exponential interaction energies with applications to transformers

    Nov 14, 2026Oleksii Kachaiev, Martin Burger, Samira Kabri +2Transformer AttentionTransformer

  10. Generation of protein conformations using a simple inverted transformer

    Nov 14, 2026Luigi Leonardo PaleseProtein Structure GenerationProtein Conformational Ensemble Generation

  11. Empowering Masked Diffusion Models to Self-Correct with Leave-One-Out Transformers

    Nov 14, 2026Sofian Zalouk, Vincent Counathe, Paul Jünger +4Masked Diffusion ModelsDiffusion Language Models

  12. Match and Correct: A Repeated Computational Motif in Evoformer Layers

    Nov 13, 2026Ishan Khire, Naren Manikandan, Tyler Hayes +1Protein Structure PredictionNeural Representation Geometry

  13. PACT: A Decoder-Confidence-Guided Two-Round Receiver for Joint Channel and Phase-Noise Estimation in Sub-THz OFDM

    Nov 13, 2026Ruilin Wang, Xiaodai DongWireless CommunicationsChannel Estimation

  14. Multi-head Quantum Neural Networks: Replacing Transformer Blocks with NISQ Modules inside a 30-Billion-Parameter LLM

    Nov 11, 2026Maforikan Amoussou, Borja Aizpurua, Sukhbinder Singh +2Quantum Neural NetworksTransformer

  15. SkinAdvisor: An AI-Driven Decision Framework for Skin Production Planning via Multi-Modal Transformer Sales Forecasting

    Nov 11, 2026Ge Fan, Kai Wang, Chaoyun Zhang +3ForecastingTime Series Forecasting

  16. KiteNorm: Variance Regularisation for Stable and Scalable Post-LN Transformers

    Nov 9, 2026Leon A. Trochelmann, Sajad Movahedi, Shiwei Liu +1TransformerTransformer Training Dynamics

  17. Can small transformers learn the language of Coxeter groups?

    Nov 7, 2026J. Elisenda Grigsby, Yiyao Huang, Arushi Katyal +3Transformer InterpretabilityTransformer

  18. A Tight Hierarchy for Chain of Thought

    Nov 6, 2026Thanh Le, A. Pavan, N. V. VinodchandranCoT ReasoningStructured Reasoning in Language Models

  19. PhoPunct: A Large-Scale Benchmark of Transformer Architectures for Vietnamese Punctuation Restoration

    Nov 6, 2026Khoa D. Nguyen, Thai Bao Huynh, Binh T. NguyenTransformerConditional Random Fields

  20. Loopback Transformer: Towards a Unified View of Residual Looped Transformers

    Nov 4, 2026Xuan Ouyang, Zefan Cai, Chenlong Zhang +3TransformerRecurrent Transformers

  21. Fault-tolerant shallow recurrent decoders for sparse sensing with missing sensor measurements

    Nov 3, 2026Matteo Romilio Pasqual, Matteo Tomasetto, J. Nathan Kutz +1Learning with Missing DataRecurrent Neural Networks

  22. Library of Layers (LoL): Adaptively Ordering Transformer Layers at Runtime

    Nov 2, 2026Abdellah Rahmani, Sean McLeish, Ang Li +3TransformerAdaptive Reasoning in Language Models

  23. Metadata-Conditioned Audio Transformers for Adaptive Respiratory Sound Classification

    Nov 2, 2026George Kontogiannis, Pantelis Tzamalis, Sotiris NikoletseasRespiratory Sound ClassificationTransformer

  24. M3Former: Fine-Grained Cross-Layer Mixing for Transformers

    Nov 2, 2026Jiasong Han, Yunkai Chai, Tong Zhu +5TransformerResidual Learning

  25. RCFormer: Reliability-aware Context Transformer for Occlusion-Robust Facial Landmark Detection

    Nov 1, 2026Ashish Kumar, Prashant Kumar, Mallika GargFacial Landmark LocalizationAnatomical Landmark Detection

  26. HamFormer: A Hamiltonian Flow-based SE(3)-Transformer for Conformation Optimization

    Oct 31, 2026Minjie Cheng, Zhenghao Liu, Yitian Wang +3TransformerMolecular Optimization

  27. Hierarchical Residuals: A Minimalist Decoupling of Residual Width from Compute Width

    Oct 30, 2026Ziteng Wang, Jianqiao Lu, Xinyu Ma +4TransformerResidual Learning

  28. Transformer-inspired architectures make machine-learned electron densities accurate and transferable enough to bypass self-consistency

    Oct 29, 2026Jeheon WooDensity Functional TheoryMolecular Property Prediction