Transformer Architectures

Follow this topic Ask about this topic

Momentum

148 papers in the last four weeks, up 164% on the four weeks before. 1.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,120

All topics

How Do Transformers Learn to Represent Symmetries?
Oct 7, 2026Eduardo Santos-Escriche, Valerie Engelmayer, Ya-Wei Eileen Lin +1Transformer Architectures
Technical University of Munich Munich Center for Machine Learning · Massachusetts Institute of Technology
Leaner Transformers Can Easily Learn to Cluster
Oct 7, 2026Charlotte Park, Kenneth L. Clarkson, Lior Horesh +2Transformer Architectures Equivariant Learning
MIT · IBM Research
Decoding Looped Transformers Better for (Almost) Free
Oct 1, 2026Weihao Liu, Huangjie Zheng, Tianrong Chen +5Transformer Encoder Transformer Architectures
Apple
Universal interpolation for deep residual self-attention networks
Oct 1, 2026Sibylle Marcotte, Joan BrunaTransformer Architectures Approximation
Department of Computer Science New York University
LAST: Looped Audio Spectrogram Transformer
Oct 1, 2026Haider Al-Tahan, Sean O'Brien, Anastasia Razdaibiedina +1Neural Audio Transformer Architectures
Georgia Institute of Technology · University of California San Diego · Google DeepMind
A foundation for systematic analysis of transformers and RNNs for tractography
Oct 1, 2026Emmanuelle Renauld, Philippe Poulin, Hugo Larochelle +2Magnetic Resonance Imaging Transformer Architectures
Computer Sciences Department, Université de Sherbrooke · Mila, Université de Montréal · Imeka Solutions Inc, Sherbrooke, QC, Canada
Stochastic Rounding in Low-Precision Transformer Inference: A Variable-Precision Emulation Study of a Small GPT-2
Oct 1, 2026Yohan Chatelain, Pablo de Oliveira CastroFull-Precision Performance Language Model Perplexity
Krembil Centre for Neuroinformatics, CAMH, Toronto, Canada · Universite Paris-Saclay, UVSQ, LI-PaRAD, Versailles, France
Decision Titan: Test-Time Training for Long-Term Memory in Offline Reinforcement Learning
Oct 1, 2026Jude Waide, Robert LieckTest-Time Training Long-Term Memory
Department of Computer Science, University of Durham, Durham, United Kingdom.
Learning Rate Transfer for Hybrid Transformer-SSM Architectures
Oct 1, 2026Jimin Seo, Gyubok Lee, Yeonsik Jo +11Hybrid Transformer Transformer Architectures
Seoul National University · SB Intuitions · LG AI Research +1
Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts
Oct 1, 2026Di He, Pengxiang Li, Da Chang +3Mixture-Of-Experts Large Language Models Mixture-Of-Experts
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Peng Cheng Laboratory · University of Chinese Academy of Sciences +5
Screw Attention: Rigid-Body Algebra Inside a Transformer
Oct 1, 2026Aly MagassoubaRigid-Body Dynamics Transformer Architectures
School of Computer Science, University of Nottingham, Nottingham, United Kingdom
Machine Translation for Sign Languages
Oct 1, 2026Ozge Mercanoglu Sincan, Anton Pelykh, Edward Fish +7Gloss-Free Sign Language Translation Sign Language Translation
Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, Guildford, UK, GU2 7XH.
Lang3DSeg: Annotation-Free Open-Vocabulary 3D Segmentation with Point Transformers
Oct 1, 2026Cigdem Kokenoz, Amir Salarpour, Alkim Domeke +5Open-Vocabulary 3D Segmentation Interactive 3D Segmentation
Department of Automotive Engineering, Clemson University, Greenville, SC, USA. · School of Computing, Clemson University, Clemson, SC, USA.
Contextual trajectory and incremental contextual displacement: Towards using LLMs to understand dynamic, utterance-specific meaning construction
Sep 30, 2026Grayson Wycliffe Storer, Julia Witte ZimmermanContextual Embeddings Single Trajectory
Department of Mathematics and Statistics. · Department of Computer Science, University of Vermont, Burlington, 05405, Vermont, United States. · Computational Story Lab. +1
Attention Kernels for Learning Maps Between Heavy-Tailed Measures
Sep 30, 2026Kailen Hargenrader, Edoardo Calvello, Bohan ChenTransformer Attention Long-Tailed Distribution
Computing and Mathematical Sciences California Institute of Technology · Lawrence Berkeley National Laboratory University of California, Berkeley ICSI
Scaling Laws for Looped Mixture of Experts
Sep 30, 2026Yanbei Chen, Anirudh Goyal, Raghuraman KrishnamoorthiMixture-Of-Experts Scaling Laws
Meta AI
Shared Weights, Selected Computations: How Looped Transformers Route What Each Loop Does
Sep 30, 2026Jiaju Wu, Yi Hu, Muhan ZhangTransformer Architectures Loop
Institute for Artificial Intelligence, Peking University. · School of Mathematical Sciences, Peking University.
FAST: Flow Any Scene Transformer
Sep 30, 2026Yongjian Zhang, Longguang Wang, Zhuo Song +3Harder Better Faster Denser Feature Matching Recent Vision Foundation Models
School of Electronics and Communication Engineering, the Shenzhen Campus of Sun Yat-sen University, Sun Yat-sen University, Shenzhen, China · Hong Kong Polytechnic University, HKSAR, China · School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China
Stable Transformers for Graph Generation
Sep 30, 2026Luca Miglior, Alessio Gravina, Davide BacciuMultiplex Graph Transformers Generative Models
Department of Computer Science, University of Pisa, Italy
Effective Does Not Mean Useful: Conditional Functional Substitutability for Redundancy and Scaling in Transformers
Sep 30, 2026Jiaheng Chen, Jiaxing Li, Yucheng Xiao +4Redundancy Scaling
Harbin Institute of Technology, Shenzhen, China · Northeastern University, Shenyang, China · Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China +2
Fiber-Resolved Microstructure Quantification from Multi-Shell Diffusion MRI using Detection Transformers
Sep 30, 2026Sebastian Endt, Marcus Wirth, Johannes Reinhold Schlund +1Magnetic Resonance Imaging Porous Media
AImotion Bavaria, Technische Hochschule Ingolstadt, Ingolstadt, Germany · TUM School of Computation, Information and Technology, Technical University of Munich, Garching, Germany · TUM School of Natural Sciences, Technical University of Munich, Garching, Germany
dattri-LLM: A Unified and Efficient Library for Training Data Attribution at LLM Scale
Sep 30, 2026Shixuan Liu, Tongli Zhou, Junwei Deng +2Large Language Model Training Large Language Models(Llms
University of Illinois at Urbana-Champaign · Google
Multi-Rate Bandwidth Extension by Token Completion in Neural Audio Codecs
Sep 29, 2026Benoît Ginies, Olivier Fercoq, Gaël RichardNeural Audio Codecs Audio Tokenizers
LTCI, Télécom Paris, Institut Polytechnique de Paris, Palaiseau, France
LoopVL: Recurrent Visual Intelligence
Sep 29, 2026Zhe Qian, Ziyang Gong, Zhongxing Xu +9Recent Vision-Language Models Transformer Architectures
Gaoling School of Artificial Intelligence, Renmin University of China · Baidu · Shanghai Jiao Tong University +5
Function-Space Transformer with Adaptive Anchors
Sep 29, 2026Guorui Sang, Pedram RooshenasSpatio-Temporal Transformers Fourier Neural Operators
University of Illinois Chicago
Pretraining Latent Information Feedback Transformers with Teacher Supervision
Sep 29, 2026Dor Tirosh, Ido Amos, Mor GevaTransformer Architectures Pretraining
Blavatnik School of Computer Science and AI, Tel Aviv University · The Hebrew University of Jerusalem
How Local Mixing Encodes Relative Position in Global NoPE Attention
Sep 29, 2026Cutter Dawes, Nick Alonso, Tom Figliolia +1Positional Encoding Transformer Architectures
Zyphra Research
DIET: Deletion-response Expert Trimming for Video Diffusion Transformers
Sep 29, 2026Jiachang Zhang, Teng Hu, Bohao Feng +4Transformer Architectures Deletion
Xi’an Jiao Tong University, China · Shanghai Jiao Tong University, China · Alibaba Token Hub, Alibaba Group, China +1
The Geometry of Inference in Transformer Residual Streams
Sep 29, 2026Timur Mudarisov, Mikhail Burtsev, Radu StateTransformer Residual Streams Transformer Architectures
University of Luxembourg, Luxembourg · London Institute for Mathematical Sciences, London, UK
Pixel-Level Transformers in Remote Sensing: A Canopy Height Case Study
Sep 29, 2026Sven Ligensa, Jan Pauls, Karsten Schrödter +2Canopy Remote Sensing
University of Münster Münster, Germany · Laboratoire des Sciences du Climat et de l’Environnement Paris, France
Predictive Geometry of Hidden Trajectories in Transformers
Sep 29, 2026Timur Mudarisov, Mikhail Burtsev, Tatiana Petrova +1Transformer Encoder Transformer Architectures
University of Luxembourg, Luxembourg
Procedural Core: A Compact Recurrent Initialization for Vision Transformers
Sep 29, 2026Zachary Shinnick, Christian Internò, Hemanth Saratchandran +2Transformer Architectures Imagenet
Australian Institute for Machine Learning (AIML), Adelaide University · Bielefeld University · Metacognition AI +1
TomoTransformer: Towards a Foundation Model for CT Reconstruction
Sep 29, 2026AmirEhsan Khorashadizadeh, Benjamín BéjarCone-Beam Computed Tomography Sparse-View
Swiss Data Science Center (SDSC) in Paul Scherrer Institute (PSI), Villigen, Switzerland.
EviViT: Evidence-Adaptive Vision Transformers for Fine-Grained Perception
Sep 29, 2026Yaoxin Niu, Zhangquan Chen, Yang Zhang +5Self-Supervised Vision Transformers Fine-Grained Perception
Tsinghua University · Peng Cheng Laboratory · The Hong Kong University of Science and Technology +3
RVQ Position Aware Speculative Decoding for On Device Text to Speech
Sep 29, 2026Berkin Durmus, Eduardo Pacheco, Zach Nagengast +1Autoregressive Text-To-Speech Autoregressive Decoding
Argmax, Inc., USA
Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It
Sep 29, 2026Zehao Jin, Ruixuan Deng, Junran WangTransformer Architectures Generative Pretrained Transformers
Georgia Institute of Technology
The Safety Operator: Modulating the Expression of Safety Instructions via Spectral Optimization
Sep 29, 2026Benoit Dherin, Michael Munn, Xavier Gonzalvo +14Safety Claims Instruction Tuning
Google Research · Cambridge University · Google DeepMind +2
LoopICL: Looping a single transformer block to solve tabular tasks
Sep 28, 2026Amir Rezaei Balef, Katharina EggenspergerTabular Foundation Models Transformer Architectures
TU Dortmund University, Dortmund, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Dortmund, Germany · University of Tübingen, Tübingen, Germany
Causal and Interpretable Structures in LLM Compositional Tasks
Sep 28, 2026Gurbir Arora, Toni J. B. Liu, Jiajun Bao +2Next-Token Prediction Transformer Architectures
Cornell University, USA · Goodfire AI, USA
How to Loop MoE: Flatten the Experts, Untie the Attention
Sep 28, 2026Shouren Wang, Chuang Ma, Mohsen Hariri +6Mixture-Of-Experts Experts
Case Western Reserve University · Kyoto University · NII LLMC
Improving Test-Time Scaling with Adaptive Looped Transformers
Sep 28, 2026Yichen You, Tianyu Fu, Aosong Feng +4Transformer Architectures Test-Time Scaling
Tsinghua University · Yale University
Copy the Same, Distill the Difference: Initializing Linear Vision Transformers
Sep 28, 2026Huaiyuan Qin, Muli Yang, Gabriel James Goenawan +9Self-Supervised Vision Transformers Transformer Architectures
Institute of Advanced Intelligence and Computing (IAIC), A*STAR, Singapore · Nanyang Technological University · ST Engineering Geo-Insights, Singapore +3
Which the Eye Fears: Writing with Read-Blindness Explains Massive Activations in Transformers
Sep 28, 2026Swagatam Mukhopadhyay, Vishal Vivek Saley, Vraj Parikh +1Transformer Architectures Residual Stream
PsiDagger · IIT Delhi
Attention Graphons: A Graph Limit Perspective on Graph Transformers
Sep 28, 2026Caio F. Deberaldini Netto, Moshe Eliasof, Luana RuizInterleaved Graph Attention Transformer Attention
Johns Hopkins University Baltimore, MD · University of Cambridge Cambridge, UK
Revisiting Risky Tackle Detection with Vision Transformers
Sep 28, 2026Syed Ahsan Masud Zaidi, Lior Shamir, Scott DietrichSelf-Supervised Vision Transformers Vision Transformer
Kansas State University, Manhattan, KS, USA · Albright College, Reading, PA, USA
Scaffold Then Internalize: Representation Injection for Diffusion Transformers
Sep 28, 2026Han Fu, Jiacheng Chen, Baoquan Zhao +4Diffusion Transformers Transformer Architectures
Sun Yat-sen University · Video Rebirth · The Hong Kong Polytechnic University
Small transformers track Bayesian evidence for latent common causes via a context-invariant mechanism
Sep 28, 2026Amir Mohammadpour, Michael FrankeTransformer Architectures Small Models
Department of Linguistics, University of Tübingen
Universality and Generalization of Causal Transformers Across Context Lengths
Sep 28, 2026Takashi Furuya, Maarten V. de Hoop, Gabriel PeyréTransformer Architectures Regularity
Doshisha University, RIKEN AIP · Rice University · CNRS, ENS, PSL Université
From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformers
Sep 28, 2026Nafiseh HosseinpourFardi, Negar Alihadi, Mahmoudreza Babaei +2Post-Training Quantization Mixed-Precision Quantization
Sharif University of Technology, Tehran, Iran · GISMA University of Applied Sciences, Potsdam, Germany · BRAINS, Brandenburg Research Center for Applied Intelligent Systems, Potsdam, Germany +3
Shallow Queries, Mature Values: Depth-Asynchronous Self-Speculation for Looped Transformers
Sep 28, 2026Guanghao Li, Zihan Su, Hao Yu +6Speculative Decoding Self-Speculative
Tsinghua University · Southern University of Science and Technology · The University of Hong Kong +2
ZonoGPT: Towards An Abstract Domain for Verifying Large GPT Models
Sep 28, 2026Hai Duong, Thanh Le, ThanhVu NguyenGenerative Pretrained Transformers Transformer Architectures
Department of Computer Science George Mason University Fairfax, VA, USA · Unaffiliated Yokosuka, Japan
Q-learning Penalized Transformer for Safe Offline Reinforcement Learning
Sep 28, 2026Shengchao Hu, Peng Wang, Jifeng Hu +5Model-Based Reinforcement Learning Q-Learning
Shanghai Jiao Tong University, China · Shenzhen Campus of Sun Yat-sen University, China · Jilin University, China +1
MegaGraph: Towards Efficient Training of Large-Scale Graph Transformers with Automated Hybrid Parallelism
Sep 28, 2026Tong Qiao, Ao Zhou, Yingjie Qi +2Multiplex Graph Transformers Cpu-Gpu Hybrid Designs
School of Computer Science and Engineering, Beihang University, China · School of Software, Beihang University, China · State Key Laboratory of Complex and Critical Software Environment, Beihang University, China +1
Query Expansion and Key Specialization in Transformer Attention Geometry
Sep 28, 2026Vidit Gupta, Siddhesh Nadkarni, Mihik Chaudhari +2Transformer Attention Transformer Architectures
Dwarkadas J. Sanghvi College of Engineering Mumbai, India
Validating Memory-Optimal Transformer Kernels on Real Hardware: From Formal Derivation to Measured Performance Across Two HPC Clusters
Sep 27, 2026Lenore M. Mullin, Gaetan HainsGraphics Processing Unit Resources High-Performance Computing
Professor Emerita, College of Nanotechnology, Science, and Engineering, University at Albany (SUNY), Albany, NY, USA.
Residual-Stream Burden Shapes Representation Learning in Diffusion Transformers
Sep 27, 2026Tongtong Liang, Siqi Kou, Ziqiao Xi +6Diffusion Transformers Transformer Residual Streams
UC San Diego · Shanghai Jiao Tong University · Aether AI
Geometric Inductive Biases for Semi-Supervised Equalization: The Constellation-Aware Transformer
Sep 27, 2026Avi CaciularuChannel Estimation Multiple-Input Multiple-Output
Google Research
EAT: Expert Account Tracker for Efficient MoE Inference
Sep 27, 2026Yuexian Li, Yifei Yang, Zouying Cao +1Mixture-Of-Experts Experts
Paris Elite Institute of Technology, Shanghai Jiao Tong University · AGI Institute, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University +1
Pretraining Transformers with Quantized Softmax in Attention
Sep 27, 2026Shangzhen Zhu, Muyan Hu, Tomasz KozlowskiTransformer Attention Gumbel-Softmax Relaxation
University of Illinois Urbana-Champaign
Approximating Softmax in Pretrained LLMs: Model Sensitivity and Kernel Acceleration
Sep 27, 2026Shangzhen Zhu, Muyan Hu, Tomasz KozlowskiGumbel-Softmax Relaxation Transformer Architectures
University of Illinois Urbana-Champaign

Oct 7, 2026, cs.LGJ/K move · Enter open · S save

How Do Transformers Learn to Represent Symmetries?

Eduardo Santos-Escriche, Valerie Engelmayer, Ya-Wei Eileen Lin, Stefanie Jegelka

Technical University of Munich Munich Center for Machine Learning · Massachusetts Institute of Technology

Transformer Architectures

Open paper AskPDF arXiv

Training Transformer-based architectures with finite data augmentation has become an increasingly popular approach in geometric machine learning. Despite its empirical success, the interplay between the Transformer architecture, invariance to different symmetries, and augmentation budgets remains underexplored. In this paper, we study the ability of a vanilla Transformer to learn various symmetries through finite data augmentation for point cloud datasets. We identify an ordering of increasing learnability across the following symmetry groups: (i) non-angle-preserving symmetries, (ii) angle-preserving symmetries, and (iii) base angle-preserving subgroups, such as translation, rotation, and scale. For the base angle-preserving groups, we further investigate the Transformer's extrapolation behavior and conduct a structural analysis of the trained models, allowing us to identify interpretable mechanisms that induce invariance. Finally, we extend our analysis to equivariant functions and show that the detected mechanisms for approximate invariance can also provide a key building block for learned equivariance. Our project page is available at https://transformers-learn-symmetries.github.io/

Momentum

Related topics

Latest papers 1,120