Transformer

Follow this topic Ask about this topic

Transformers represent inputs as tokens and use self-attention to let each token draw on information from other positions, with feed-forward layers refining the resulting representations. Encoder-only models such as BERT, autoregressive decoder-only language models, and encoder-decoder systems are adapted for tasks ranging from text generation and image recognition to forecasting and scientific simulation. Current approaches improve efficiency and capability through local or sparse attention, retrieval and memory, hierarchical processing, and hybrids with convolutional or state-space layers; evaluation spans language-modeling, long-context, vision, and domain-specific benchmarks. A central tension is whether architectural changes and larger models deliver reliable gains in generalization and reasoning while controlling training and inference cost, and theory and interpretability work continues to probe how Transformers learn and compute.

Latest papers

544 papers70 papers in the last four weeks, up 71% on the four weeks before. 0.4% of all new papers.

Transformer-inspired architectures make machine-learned electron densities accurate and transferable enough to bypass self-consistency
Oct 29, 2026Jeheon WooDensity Functional Theory Molecular Property Prediction
Center for AI and Natural Sciences, Korea Institute for Advanced Study, Seoul 02455, Republic of Korea
CADENCE: Distributional Next-Transaction Forecasting for Business Cash-Flow Underwriting
Oct 27, 2026Alexander Wu, Jingchao HuangNext-Event Prediction Financial Forecasting
Slope
Sometimes less is more: Investigating inductive biases of MLPs and Transformers on noisy k-parity features
Oct 27, 2026Alasdair ParenTransformer Multilayer Perceptrons
Chalmers University of Technology
HC2Robot: Hyper-Connection Transformers with Predicted Foresight for Multi-Terrain Humanoid Parkour
Oct 27, 2026Wenhao Xu, Yanheng Mai, Yifei Fu +6Humanoid Robot Locomotion Terrain-Aware Robot Locomotion
The Hong Kong University of Science and Technology (Guangzhou) · CLAI-LAB, CL-TECH · South China Agricultural University +1
An Algebra of Transformers
Oct 27, 2026Yong Wang, Zhiyan Wang, Hexiao KangTransformer Formal Verification
How Much Information Can We Fit inside a Transformer?
Oct 25, 2026Amirhossein Basareh, Rishabh Sanjay, Daniel de Mello +1Memorization in Language Models Transformer
Purdue University
STFT Frontend Mismatch in Pretrained Audio Spectrogram Transformers: Analysis and Adaptation
Oct 25, 2026Julia Wilkins, Wei-Ching Lin, Abinaya Kumar +2Audio Classification Transformer
New York University, USA · Bosch Research, USA
Putting Transformer Theories to the Test
Oct 23, 2026Awni Altabaa, Tobias Schnabel, Kiran TomlinsonTransformer Transformer Expressivity
Yale University · Microsoft Research
Scaling Transformer Energy-based Models For High Quality Image Synthesis
Oct 22, 2026Jungsoo Lee, Alexi Gladstone, Yilun DuImage Generation Energy-Based Models
MIT · UIUC · Harvard University
A Transformer-based Approach for Real-time Multi-agent Path Planning with Varying Initial/Final Conditions
Oct 21, 2026Qian Zhang, Shanqing Liu, Jerome Darbon +1Decentralized Multi-Agent Motion Planning Multi-Agent Planning
Division of Applied Mathematics, Brown University, Providence, RI 02912, USA.
Geometric knowledge spaces, Attention pairing and Architectures
Oct 21, 2026Shang XuAttention Mechanisms Transformer
This is SParTA: A modular, efficient and scalable transformer architecture for jet-tagging
Oct 19, 2026Rajneil Baruah, Subhadeep Mondal, Sunando Kumar Patra +1Transformer Jet Tagging
Department of Physics, SEAS, Bennett University, Greater Noida, Uttar Pradesh, 201310, India · Department of Physics, Bangabasi Evening College, Kolkata, 700009, West Bengal, India
Copying Under Repetition: From Prefix Matching to Position Hashing
Oct 19, 2026Anxin Guo, Tianlong Huang, Chenxiao Yang +1Transformer Repetition in Language Generation
Northwestern University · Duke University · Toyota Technological Institute at Chicago
CHSM8: A Chess Transformer that Follows Natural-Language Style Prompts
Oct 18, 2026Ilia BadaninChess Transformer
Block Recursive Transformers for Structured Parameter Sharing
Oct 18, 2026Josep Marques, Hyunyoung A. Cho, Youngeun Kim +4Transformer Recurrent Transformers
Sungkyunkwan University · Korea University
A Mixture-Gated Theory of Anisotropic Representation Dynamics in Transformers
Oct 16, 2026Nguyen Do, My T. ThaiTransformer Self-Attention
Department of Computer & Information Science & Engineering, University of Florida, Gainesville, FL, USA
ABNAT: Attention-Based Noise-Aware Training for Robust Transformers on Analog Compute-in-Memory Systems
Oct 15, 2026Wenshuai Yao, Wenyong Zhou, Hanyong Shao +5Noise Robustness Transformer
School of Integrated Circuits, Peking University, Beijing, China · Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong · School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China +1
Dynamic ordering in sequence generation in Transformers: experiments on low-resource Tagging scenarios
Oct 15, 2026Giacomo Bais, Tejaswini DeoskarTransformer Morphosyntactic Tagging
Aalborg University · Utrecht University
DuET: Note-Frame Dual Encoding for Piano Performance-to-Score Transcription
Oct 15, 2026Wei Zeng, Ye WangAutomatic Music Transcription Transformer
Integrative Sciences and Engineering Programme, NUS Graduate School · School of Computing, National University of Singapore
RTAformer: Relation-Transition-Aware Transformer for Video Scene Graph Generation
Oct 14, 2026Jaeyeon Heo, Geuntaek Lim, Yukyung ChoiVideo SGG Temporal Video Understanding
Sejong University, Seoul, Republic of Korea
Sparse Transformer: a Controlled Testbed for Mechanistic Interpretability
Oct 14, 2026Anton Korznikov, Alexey Dontsov, Elena TutubalinaMechanistic Interpretability Neuron-Level Interpretability
Self-Conditioning Enables Deep Sequential Reasoning in Flow Matching
Oct 14, 2026Zeyang Zhang, Linxiang Gao, Meiqi Gu +4Self-Conditioning Flow Matching
Shanghai Qi Zhi Institute · Institute for Interdisciplinary Information Sciences, Tsinghua University · Georgia Institute of Technology +1
DAGFormer: Global Topology Prediction for Transformer Cross-Layer Connections
Oct 14, 2026Xiaocong Yang, Yuren Hao, ChengXiang ZhaiTransformer
Asymptotic Behavior of Transformers
Oct 14, 2026Katy Blumer, Da-Cheng Juan, Jon Kleinberg +3Transformer Training Dynamics Transformer
Google Research
PairIT: Autoregressive Transformers for Low-Data Molecule Optimization
Oct 13, 2026Hannah Lawrence, Bodhi P. Vani, Ji Won Park +5Autoregressive Transformer Molecular Optimization
Genentech, South San Francisco, CA, USA · Massachusetts Institute of Technology, Cambridge, MA, USA · Prescient Design, Genentech, Basel, Switzerland +1
Bytes to Threats: Byte-Sequence Transformer For Malware Detection
Oct 13, 2026Abdelrahman Yassin Eljamal, Abdeltawab HendawiMalware Detection Transformer
Department of Computer Science, University of Rhode Island, Kingston, USA
Learning Source Structure: Bayesian Prediction with Transformers
Oct 13, 2026Meir Feder, Ruediger UrbankeTransformer Sequence Modeling
School of Electrical and Computer Engineering, Tel Aviv University, Tel Aviv, Israel · School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland
Graph Metanetworks for Transformer Based Language Models
Oct 13, 2026Yonathan Wolanowsky, Guy Bar-Shalom, Fabrizio Frasca +1Weight-Space Learning Transformer
Technion – Israel Institute of Technology · NVIDIA Research
Same Function, Different Trajectory: No Standard Optimizer Is Equivariant to Transformer Reparameterization
Oct 13, 2026Bum Jun KimDeep Learning Optimization Transformer
The University of Tokyo
Neural Dimensional Analysis: Foundations of Dimensionless and Scale-Consistent Computation
Oct 13, 2026Bum Jun Kim, Makoto Kawano, Yusuke Iwasawa +1Transformer
The University of Tokyo

Oct 29, 2026J/K move · Enter open · S save

Transformer-inspired architectures make machine-learned electron densities accurate and transferable enough to bypass self-consistency

Jeheon Woo

Center for AI and Natural Sciences, Korea Institute for Advanced Study, Seoul 02455, Republic of Korea

Density Functional Theory Molecular Property Prediction Transformer

Open paper AskPDF arXiv

We introduce machine learning models that accurately predict the electron density, the central variable of density functional theory. They are built on Transformer-inspired architectures, which require little computation per parameter and can therefore be enlarged at low cost. The proposed models, SwinUNETR-V2 and its attention-free counterpart NeXtUNETR, achieve state-of-the-art accuracy on QM9, with normalized mean absolute errors in the density averaging 0.075% and 0.046%, respectively. The predicted densities are accurate enough to bypass the self-consistent iteration: a single Kohn-Sham step reproduces the self-consistent electronic structure. For NeXtUNETR, the median errors are 0.036 meV in the total energy, 0.65 meV/angstrom in the forces and 1.8 meV in the HOMO-LUMO gap. The forces not only recover the self-consistent minima in geometry optimization but also successfully drive ab initio molecular dynamics. Fine-tuned on small datasets, both networks also transfer accurately to systems containing elements absent from QM9 or larger than its molecules.