Transformer Architectures

Follow this topic Ask about this topic

Momentum

148 papers in the last four weeks, up 164% on the four weeks before. 1.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,120

All topics

Learning Theory of Transformers: Local-to-Global Approximation via Softmax Partition of Unity
May 9, 2026Zhongjie Shi, Wenjing LiaoTransformer Architectures Spatio-Temporal Transformers
School of Mathematics, Georgia Institute of Technology, Atlanta, GA 30332, United States
Causal Dimensionality of Transformer Representations: Measurement, Scaling, and Layer Structure
May 9, 2026Nilesh Sarkar, Dawar Jyoti DekaSparse Autoencoder Features Transformer Architectures
Erdős AI Lab
Scaling Limits of Long-Context Transformers
May 8, 2026Giuseppe Bruno, Shi Chen, Zhengjiang Lin +2Linear Attention Efficient Long-Context Inference
Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regression
May 8, 2026Mingsong Yan, Dongyang Li, Charles Kulick +1Kernel Ridge Regression Transformer Attention
Department of Mathematics, University of California, Santa Barbara, CA
Context-Gated Associative Retrieval: From Theory to Transformers
May 8, 2026Moulik Choraria, Argyrios Gerogiannis, Vidhata Jayaraman +2Dense Associative Memory In-Context Learning
UIUC · University of Minnesota, Twin Cities · Stony Brook University
When Attention Beats Fourier: Multi-Scale Transformers for PDE Solving on Irregular Domains
May 8, 2026Brandon Yee, Pairie Koh, Jack Rodriguez +1Partial Differential Equations Fourier
Physics Lab, Yee Collins Research Group
NoiseGate: Learning Per-Latent Timestep Schedules as Information Gating in World Action Models
May 8, 2026Wen Huang, Haoran Sun, Yongjian Guo +8Efficient World-Action Model World Models
Tsinghua University · Peking University · JDT AI Infra +1
FlashSVD v1.5: Making Low-Rank Transformers Inference Actually Fast
May 8, 2026Wenhao Wu, Zishan Shao, Kangning Cui +5Singular Value Decomposition Large Language Model Compression
Department of Electrical & Computer Engineering, Duke University · Department of Statistical Science, Duke University · Department of Computer Science, Wake Forest University +1
Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers
May 8, 2026Noboru Isobe, Daisuke Inoue, Masaaki ImaizumiTransformer Attention Transformer Architectures
1RIKEN Center for Advanced Intelligence Project · 2Imperial College London · 3The University of Tokyo +1
Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models
May 8, 2026Victor Conchello Vendrell, Arnau Padres Masdemont, Niccolò Grillo +3Large Language Model Memory Recurrent Model
Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.
Encoder-Decoder Transformers: Logical Characterizations and Periodicity
May 8, 2026Veeti Ahvonen, Damian Heiman, Antti Kuusisto +2Transformer Encoder Transformer Architectures
Mathematics Research Centre, Tampere University, Tampere, Finland
Revisiting Transformer Layer Parameterization Through Causal Energy Minimization
May 8, 2026Jin Xu, Camille Couturier, Victor Rühle +2Transformer Architectures Multi-Head Attention
Microsoft · Microsoft Research Cambridge
Dynamic Mode Decomposition along Depth in Vision Transformers
May 8, 2026Nishant Suresh Aswani, Saif Eddin JabariSelf-Supervised Vision Transformers Vision Transformer
NYU Tandon & NYU Abu Dhabi
Approximation Error Upper and Lower Bounds for Hölder Class with Transformers
May 8, 2026Xin He, Yuling Jiao, Xiliang Lu +1Transformer Architectures Approximation
School of Mathematics and Statistics, Wuhan University, Wuhan, China · School of Artificial Intelligence, Wuhan University, Wuhan, China · Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, China
Beyond Linear Attention: Softmax Transformers Implement In-Context Reinforcement Learning
May 8, 2026Zixuan Xie, Xinyu Liu, Claire Chen +3In-Context Learning Linear Attention
University of Virginia · California Institute of Technology · Purdue University
Amortized-Precision Quantization for Early-Exit Vision Transformers
May 8, 2026Rui Fang, Hsi-Wen Chen, Ming-Syan ChenSelf-Supervised Vision Transformers Vision Transformer
National Taiwan University
Spectrum-Adaptive Generalization Bounds for Trained Deep Transformers
May 8, 2026Mana Sakai, Masaaki ImaizumiTransformer Architectures Generalization Bounds
The University of Tokyo · RIKEN Center for Advanced Intelligence Project · Kyoto University
TAS-LoRA: Transformer Architecture Search with Mixture-of-LoRA Experts
May 8, 2026Jeimin Jeon, Hyunju Lee, Bumsub HamSelf-Supervised Vision Transformers Hip-Lora
Yonsei University · Articron Inc. · Korea Institute of Science and Technology (KIST)
Geometric Asymmetry in MoE Specialization: Functional Decorrelation and Representational Overlap
May 8, 2026Feilong LiuMixture-Of-Experts Layer-Wise
Attention Transfer Is Not Universally Effective for Vision Transformers
May 8, 2026Huaiyuan Qin, Muli Yang, Gabriel James Goenawan +4Self-Supervised Vision Transformers Vision Transformer
Institute for Infocomm Research (I2R), A*STAR, Singapore · 3Shanghai Jiao Tong University · 2Sichuan University
SAFformer:Improving Spiking Transformer via Active Predictive Filtering
May 8, 2026Zequan Xie, Weiming Zeng, Yunhua Chen +3Spiking Neural Networks Transformer Architectures
School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, China · Faculty of Science, Hong Kong Baptist University, China · School of Electronic Information, Wuhan University, China
When Symbol Names Should Not Matter: A Logistic Theory of Fresh-Symbol Classification
May 8, 2026Wenjie Guan, Jelena BradicSymbol-Transformer Architectures
Department of Statistics and Data Science, Cornell University
LoopQ: Quantization for Recursive Transformers
May 8, 2026Rui Fang, Hsi-Wen Chen, Ming-Syan ChenPost-Training Quantization Transformer Architectures
National Taiwan University
Not All Tokens Need 40 Steps: Heterogeneous Step Allocation in Diffusion Transformers for Efficient Video Generation
May 7, 2026Ernie Chu, Vishal M. PatelDiffusion Transformers Spatial Tokens
Johns Hopkins University Baltimore, MD 21218
ProtSent: Protein Sentence Transformers
May 7, 2026Dan Ofer, Oriel Perets, Michal Linial +1Language Modeling Transformer Architectures
Department of Computer and Information Science Ben-Gurion University of the Negev
Transformers Efficiently Perform In-Context Logistic Regression via Normalized Gradient Descent
May 7, 2026Chenyang Zhang, Yuan CaoIn-Context Learning Transformer Architectures
School of Computing & Data Science, The University of Hong Kong
Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models
May 7, 2026Amir Rezaei Balef, Mykhailo Koshil, Katharina EggenspergerTabular Foundation Models Transformer Architectures
TU Dortmund University, Dortmund, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Dortmund, Germany · University of Tübingen, Tübingen, Germany
Cubit: Token Mixer with Kernel Ridge Regression
May 7, 2026Chuanyang Zheng, Jiankai Sun, Yihang Gao +6Transformer Attention Transformer Architectures
Patch-Effect Graph Kernels for LLM Interpretability
May 7, 2026Ruben Fernandez-Boullon, David N. OlivieriMechanistic Interpretability Transformer Architectures
Department of Computer Science, University of Vigo, Ourense 32004, Spain
Federation of Experts: Communication Efficient Distributed Inference for Large Language Models
May 7, 2026Muhammad Shahir Abdurrahman, Chun Deng, Azalia Mirhoseini +1Transformer Architectures Token Embeddings
Stanford University
Dynamic Pondering Sparsity-aware Mixture-of-Experts Transformer for Event Stream based Visual Object Tracking
May 7, 2026Shiao Wang, Xiao Wang, Duoqing Yang +5Multi-Object Tracking Self-Supervised Vision Transformers
School of Computer Science and Technology, Anhui University, Hefei 230601, China · Beijing Institute of Technology, Beijing, China · Peng Cheng Laboratory, Shenzhen, China +2
Beyond Autoregressive RTG: Conditioning via Injection Outside Sequential Modeling in Decision Transformer
May 7, 2026Yongyi Wang, Hanyu Liu, Lingfeng Li +6Transformer Architectures Sequence Modeling
School of Computer Science Peking University Beijing, China 100871
The E $Δ$ -MHC-Geo Transformer: Adaptive Geodesic Operations with Guaranteed Orthogonality
May 7, 2026Arash ShahmansooriResidual Connections Orthogonality
Independent Researcher
Transformer-Based Wildlife Species Classification from Daily Movement Trajectories
May 7, 2026Obed Irakoze, Prasenjit MitraSpecies Identification Species
Department of Electrical & Computer Engineering Carnegie Mellon University Africa Kigali, Rwanda
Training Transformers for KV Cache Compressibility
May 7, 2026Yoav Gelberg, Yam Eitan, Michael Bronstein +2Key-Value Cache Compression Efficient Long-Context Inference
University of Oxford · 2Technion – Israel Institute of Technology · 3AITHYRA +1
Transformers Provably Implement In-Context Reinforcement Learning with Policy Improvement
May 7, 2026Haodong Liang, Lifeng LaiOffline Reinforcement Learning In-Context Learning
Department of Electrical and Computer Engineering, University of California, Davis
HyperLens: Quantifying Cognitive Effort in LLMs with Fine-grained Confidence Trajectory
May 7, 2026Chengda Lu, Xiaoyu Fan, Wei XuLLM Inference Optimization Cognitive Load
1IIIS, Tsinghua University · 2Shanghai Qi Zhi Institute.
Budgeted Attention Allocation: Cost-Conditioned Compute Control for Efficient Transformers
May 7, 2026Amrit NidhiTransformer Attention Transformer Architectures
Independent Researcher
Distributional Spectral Diagnostics for Localizing Grokking Transitions
May 7, 2026Ziyue Wang, Yufeng Ying, Takafumi KanamoriGrokking Transformer Residual Streams
Institute of Science Tokyo · University of Science and Technology of China
Shortcut Solutions Learned by Transformers Impair Continual Compositional Reasoning
May 6, 2026William T. Redman, Erik C. Johnson, Brian RobinsonTransformer Architectures Replay-Based Continual Learning
Johns Hopkins Applied Physics Lab
On Semantic Loss Fine-Tuning Approach for Preventing Model Collapse in Causal Reasoning
May 6, 2026Pratik Deshmukh, Atirek GuptaCausal Reasoning Collapse
Technical University of Vienna, Austria · HCLTech, Noida, India
Taming Outlier Tokens in Diffusion Transformers
May 6, 2026Xiaoyu Wu, Yifei Wang, Tsu-Jui Fu +3Diffusion Transformers Visual Tokenizers
Rice University · Apple
Understanding In-Context Learning for Nonlinear Regression with Transformers: Attention as Featurizer
May 6, 2026Alexander Hsu, Zhaiming Shen, Wenjing Liao +1In-Context Learning Transformer Architectures
Department of Mathematics, Purdue University · School of Mathematics, Georgia Institute of Technology
Superposition Is Not Necessary: A Mechanistic Interpretability Analysis of Transformer Representations for Time Series Forecasting
May 6, 2026Alper YıldırımForecasting Backbone Time Series Forecasting
Adaptive Inverted-Index Routing for Granular Mixtures-of-Experts
May 6, 2026Klaus-Rudolf Kladny, Maximilian Mordig, Bernhard Schölkopf +1Mixture-Of-Experts Large Language Model Routing
Max Planck Institute for Intelligent Systems, Tübingen · Tübingen AI Center · ETH Zurich +1
On the (In-)Security of the Shuffling Defense in the Transformer Secure Inference
May 6, 2026Zhengyi Li, Yakai Wang, Kang Yang +6Transformer Architectures LLM Defense Mechanisms
Shanghai Jiao Tong University · State Key Laboratory of Cryptology · Shanghai Qizhi Institute
Anticipating Innovation Using Large Language Models
May 6, 2026Enrico Maria Fenoaltea, Filippo Santoro, Giordano De Marzo +2Technology Large Language Model Forecasting
Institute of Complex Systems (UBICS), Universitat de Barcelona · Centro Ricerche Enrico Fermi (CREF), Via Panisperna 89 A – 00184 Roma · University of Konstanz +1
Gyan: An Explainable Neuro-Symbolic Language Model
May 6, 2026Venkat Srinivasan, Vishaal Jatav, Anushka Chandrababu +1Large Language Model Pretraining Transformer Architectures
Innospark Ventures & Gyan AI · Gyan AI Inc.
Average Attention Transformers and Arithmetic Circuits
May 6, 2026Lena Ehrmuth, Laura StriekerTransformer Attention Transformer Encoder
Institute for Theoretical Computer Science, Leibniz University Hanover, Hanover, Germany
Learning the Channel Gain from Anywhere to Anywhere via Cross-environment Transformer Estimators
May 6, 2026Prasenjit Dhara, Daniel RomeroChannel Modeling Spatio-Temporal Transformers
Dept. of Information and Communication Technology, University of Agder, Grimstad, Norway.
FLUID: Continuous-Time Hyperconnected Sparse Transformer for Sink-Free Learning
May 6, 2026Waleed Razzaq, Yun-Bo ZhaoTransformer Architectures Replay-Based Continual Learning
School of Automation University of Science and Technology China Hefei, Anhui
Critical Windows of Complexity Control: When Transformers Decide to Reason or Memorize
May 6, 2026Sarwan AliTransformer Architectures Weight Decay
Columbia University, Irving Medical Center, NY, USA
The Scaling Properties of Implicit Deductive Reasoning in Transformers
May 5, 2026Enrico Vompa, Tanel TammetTransformer Architectures Deduction
Applied Artificial Intelligence Group Tallinn University of Technology, Estonia
UniCorrn: Unified Correspondence Transformer Across 2D and 3D
May 5, 2026Prajnan Goswami, Tianye Ding, Feng Liu +1Point Clouds 3D Generation
Northeastern University · Adobe Research
RD-ViT: Recurrent-Depth Vision Transformer for Semantic Segmentation with Reduced Data Dependence Extending the Recurrent-Depth Transformer Architecture to Dense Prediction
May 5, 2026Renjie HeSelf-Supervised Vision Transformers Dense Prediction
Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
Feature-Augmented Transformers for Robust AI-Text Detection Across Domains and Generators
May 5, 2026Mohamed Mady, Johannes Reschke, Björn SchullerMachine-Generated Text Detection Bert-Based Models
Chair of Health Informatics, Technical University of Munich (TUM), Germany · OTH Regensburg, Germany · GLAM – Group on Language, Audio & Music, Imperial College London, United Kingdom
Transformers with Selective Access to Early Representations
May 5, 2026Skye Gunasekaran, Téa Wright, Rui-Jie Zhu +1Transformer Architectures Early Layers
UC Santa Cruz · UC Berkeley
Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers
May 5, 2026Hao Yan, Haolin Yang, Yiqiao ZhongTransformer Architectures
University of Wisconsin–Madison · University of Chicago
Disentangling Shared and Task-Specific Representations from Multi-Modal Clinical Data
May 5, 2026He Lyu, Huolin Zeng, Junren Wang +7Multimodal Clinical Data Transformer Architectures
Department of Anesthesiology and West China Biomedical Big Data Center West China Hospital, Sichuan University, Chengdu, China · Med-X Center for Informatics, Sichuan University, Chengdu, China · Department of National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University, Chengdu, China +4
Benchmarking Logistic Regression, SVM, Naive Bayes, and IndoBERT Fine-Tuning for Sentiment Analysis on Indonesian Product Reviews
May 5, 2026Nabila Zakiyah Zahra, Salwa Farhanatussaidah, Nasywa Nur Afifah +3Term Frequency-Inverse Document Frequency Indonesian E-Commerce Reviews
Department of Data Science Institut Teknologi Sumatera (ITERA) · Department of Informatics Institut Teknologi Sumatera (ITERA)

May 9, 2026, stat.MLJ/K move · Enter open · S save

Learning Theory of Transformers: Local-to-Global Approximation via Softmax Partition of Unity

Zhongjie Shi, Wenjing Liao

School of Mathematics, Georgia Institute of Technology, Atlanta, GA 30332, United States

Transformer Architectures Spatio-Temporal Transformers Feed-Forward Partitioning

Open paper AskPDF arXiv

This paper investigates the learning theory of Transformer networks for regression tasks on the compact Euclidean domain

[0,1]^d

and

d

-dimensional compact Riemannian manifolds. We propose a novel constructive approximation framework for Transformers that builds local approximations of the target function and aggregates them into a global approximation via softmax partition of unity. This approach leverages the attention mechanism to achieve spatial localization through affine transformations of the input. The softmax activation plays a crucial role in aggregating local approximations to a global output. From an approximation perspective, we prove that a dense Transformer equipped with only two encoder blocks and standard single-hidden-layer point-wise feed-forward networks can achieve a uniform

\varepsilon

-approximation error for

α

-Hölder continuous functions with

α\in (0,1]

using

\mathcal{O}(\varepsilon^{-d/α})

total parameters. Building upon this approximation guarantee, we establish a near minimax-optimal generalization error bound of order

\mathcal{O}\big(n^{-\frac{2α}{2α+d}} \log n\big)

for the empirical risk minimizer, where

n

is the training data size. The Transformer architecture studied in this paper is dense, shallow and wide, and employs softmax activation and sinusoidal positional encodings, closely reflecting practical implementations.

Momentum

Related topics

Latest papers 1,120