Transformer Architectures

Follow this topic Ask about this topic

Momentum

148 papers in the last four weeks, up 164% on the four weeks before. 1.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,120

All topics

UniT: Unified Geometry Learning with Group Autoregressive Transformer
May 20, 2026Haotian Wang, Yusong Huang, Zhaonian Kuang +4Visual Geometry Grounded Transformer Autoregressive Transformers
Intelligent Transportation Thrust of the Systems Hub, The Hong Kong University of Science and Technology (GZ), Guangzhou, P.R.China. · The National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University, Xi’an, P.R.China. · Applied Science, Amazon.com, Inc., USA.
HORST: Composing Optimizer Geometries for Sparse Transformer Training
May 20, 2026Tom Jacobs, Rohan Jain, Rebekka BurkholzSparsity Transformer Architectures
CISPA Helmholtz Center for Information Security
Towards Understanding Self-Pretraining for Sequence Classification
May 20, 2026Omar Coser, Loredana Zollo, Paolo Soda +1Self-Supervised Learning Transformer Architectures
A Sharper Picture of Generalization in Transformers
May 20, 2026Paul Lintilhac, Sair ShaikhGeneralization Bounds Sharpness-Aware Minimization
Thayer School of Engineering Dartmouth College · Dartmouth College
Markovian Circuit Tracing for Transformer State Dynamic
May 20, 2026Abdullah XTransformer Architectures Hidden Markov Models
Project AWARE and Zephara AI
TERDNet: Transformer Encoder-Recurrent Decoder Network for Scene Change Detection
May 20, 2026Jiae Yoon, Ue-Hwan KimDecoder Layers Transformer Encoder
Department of AI Convergence, Gwangju Institute of Science and Technology (GIST), Gwangju 61005, Republic of Korea
Most Transformer Modifications Still Do Not Transfer at 1-3B: A 2020-2026 Update to Narang et al. (2021) with Downstream Evaluation and a Noise Floor
May 20, 2026Yang Zhao, Jiahao Lu, Bin Huang +2Transformer Architectures Dynamic Attention
Tencent
Rethinking Cross-Layer Information Routing in Diffusion Transformers
May 20, 2026Chao Xu, Maohua Li, Qirui Li +9Diffusion Transformers Transformer Residual Streams
Alibaba Group · Nanjing University · Zhejiang University
LT2: Linear-Time Looped Transformers
May 20, 2026Chunyuan Deng, Yizhe Zhang, Rui-Jie Zhu +4Kimi Delta Attention Transformer Architectures
1Rice University · 2Apple · 3UC Santa Cruz +1
RoPeSLR: 3D RoPE-driven Sparse-LowRank Attention for Efficient Diffusion Transformers
May 20, 2026Yuxi Liu, Zekun Zhang, Yixiang Cai +3Diffusion Transformers Dynamic Sparse Attention
Peking University · University of Electronic Science and Technology of China · Beijing University of Posts and Telecommunications
Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics
May 19, 2026Lucky VermaWeight Decay Transformer Attention
Independent Researcher
AirfoilGen: A valid-by-construction and performance-aware latent diffusion model for airfoil generation
May 19, 2026Zhijie Yang, Min Tang, Peng Du +1Aerodynamics Identity-Conditioned Latent Diffusion Models
State Key Laboratory of CAD&CG, Zhejiang University, Hangzhou, 310027, China
Position: The Turing-Completeness of Autoregressive Transformers Relies Heavily on Context Management
May 19, 2026Guanyu Cui, Zhewei Wei, Kun HeAutoregressive Transformers Transformer Architectures
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · DEKE Lab, Renmin University of China, Beijing, China
Introspective X Training: Feedback Conditioning Improves Scaling Across all LLM Training Stages
May 19, 2026Brandon Cui, Ximing Lu, Jaehun Jung +7Large Language Model Training Large Language Model Pipelines
1NVIDIA · University of Washington · 3Carnegie Mellon University +1
CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs
May 19, 2026Han Guo, Jack Zhang, Arjun Menon +4Graphics Processing Unit Kernels Matrix Multiplication
1Massachusetts Institute of Technology · *Part of this work was done at Together AI. · 2Princeton University +2
Performance Monitoring of Proton Exchange Membrane Water Electrolyzer by Transformers-Based Machine Learning Model
May 18, 2026Bingqing Chen, Ivan Batalov, Qiu Chen +2Hydrogen System Components Transformer Encoder
Bosch Research & Technology Center
Resolving Representation Ambiguity in Feedforward Novel View Synthesis Transformer via Semantic-Spatial Decoupling
May 18, 2026Yihang Wu, Yihang Sun, Shaofeng Zhang +4Novel View Synthesis Spatial Tokens
Institute of Trustworthy Embodied Artificial Intelligence (TEAI), Fudan University · 3Sch. of Artificial Intelligence & Sch. of Computer Science, Shanghai Jiao Tong University · University of Science and Technology of China +1
From Division to Decision: Leveraging Temporal Cell-Stage Segmentation for Embryo Transferability Prediction
May 18, 2026Yasmine Hachani, Patrick Bouthemy, Elisa Fromont +3Early Layers Transformer Architectures
MALT · Inria center at Rennes University, France · University of Rennes, IRISA France +4
The Expressive Power of Low Precision Softmax Transformers with (Summarized) Chain-of-Thought
May 18, 2026Moritz Brösamle, Stephan EcksteinTransformer Architectures Gumbel-Softmax Relaxation
Department of Mathematics, University of Tübingen, Germany.
Protection Is (Nearly) All You Need: Structural Protection Dominates Scoring in Globally Capped KV Eviction
May 18, 2026Gabriel GarciaKey-Value Cache Eviction Transformer Architectures
Independent Researcher
Function graph transformers universally approximate operators between function spaces
May 18, 2026Takashi Furuya, David Mis, Ivan Dokmanić +2Probabilistic Operator Learning Multiplex Graph Transformers
Doshisha University, RIKEN AIP · Rice University · University of Basel +2
InfoFlow: A Framework for Multi-Layer Transformer Analysis
May 18, 2026Penghao Yu, Haotian Jiang, Zeyu Bao +1Transformer Architectures Approximation
Department of Mathematics National University of Singapore · Institute for Functional Intelligent Materials National University of Singapore · National University of Singapore +1
SNLP: Layer-Parallel Inference via Structured Newton Corrections
May 18, 2026Ligong Han, Kai Xu, Hao Wang +1Transformer Architectures Transformer Residual Streams
1Red Hat AI Innovation · 2MIT-IBM Watson AI Lab · 3Core AI, IBM
One Model, Two Roles: Emergent Specialization in a Shared Recurrent Transformer
May 18, 2026Jucheng Shen, Barbara Su, Anastasios KyrillidisRecurrent State Transformer Architectures
Rice University
Training Infinitely Deep and Wide Transformers
May 17, 2026Raphaël Barboni, Maarten V. de Hoop, Takashi Furuya +1Transformer Architectures Mean-Field Limit
Bocconi University · Rice University · Doshisha University, RIKEN AIP +1
Descriptive versus Regulatory Uncertainty in Bounded Predictive Systems
May 17, 2026Ahmed Gamal EldinUncertainty Regulation
Nova University Lisbon – Cairo Branch (NOVA IMS), Cairo, Egypt
Mamba-VGGT: Persistent Long-Sequence Video Geometry Grounded Transformer via External Sliding Window Mamba Memory
May 17, 2026Tianchen Deng, Zhenxiang Xiong, Nailin Wang +4Visual Geometry Grounded Transformer 3D Reconstruction
Shanghai Jiao Tong University · Nanyang Technological University · ETH Zurich +1
A Two-Parameter Weibull Framework for Diagnosing Transformer Weight Distributions
May 17, 2026Tiexin DingModel Weights Transformer Architectures
Independent Researcher
Cross-Subject Intracranial EEG Reconstruction from Scalp Recordings Using Multi-Scale Cross-Attention Transformers
May 17, 2026Tien-Dat Pham, Xuan-The TranNeural Recordings Transformer Architectures
HAI-Smartlink Research Lab, Anchi STE Company, Haiphong, Vietnam · School of Mechanical Engineering, Vietnam Maritime University, Haiphong, Vietnam
VISTA: Triplet-Supervised Video Style Transfer with Diffusion Transformers
May 17, 2026Yiren Song, Wangzi Yao, Haofan Wang +1Stylization Style
Show Lab, National University of Singapore · Institute of Automation, Chinese Academy of Sciences · Lovart AI
FishBack: Pullback Fisher Geometry for Optimal Activation Steering in Transformers
May 17, 2026Sihan Wang, Jiayi ZhaoLinear Activation Steering Steering
South China University of Technology
OPTNet: Ordering Point Transformer Network for Post-disaster 3D Semantic Segmentation
May 16, 2026Nhut Le, Ehsan Karimi, Maryam RahnemoonfarPoint Clouds Post-Disaster Damage Assessment
Computer Science and Engineering, Lehigh University, Bethlehem PA 18015, US · Civil and Environmental Engineering, Lehigh University, Bethlehem PA 18015, US
Propagation of Chaos in Contextual Flow Maps
May 16, 2026Shi Chen, Zhengjiang Lin, Kaizhao Liu +1Transformer Architectures Flow Map
EVA01: Unified Native 3D Understanding and Generation via Mixture-of-Transformers
May 16, 2026Zongyuan Yang, Mingjing Yi, Wanli Ma +83D Generative Models Multimodal Large Language Models
SeeleAI Team
DiRotQ: Rotation-Aware Quantization for 4-bit Diffusion Transformers
May 16, 2026Sayeh Sharify, Mahsa Salmani, Hesham MostafaTransformer Architectures Singular Value Decomposition
d-Matrix Santa Clara, CA 95054
No Free Swap: Protocol-Dependent Layer Redundancy in Transformers
May 15, 2026Gabriel GarciaTransformer Architectures Switch
Independent Researcher
Attention Dispersion in Dynamic Graph Transformers: Diagnosis and a Transferable Fix
May 15, 2026Jinhao Zhang, Kangfei Zhao, Qiuhao Zeng +1Multiplex Graph Transformers Transformer Attention
Beijing Institute of Technology · University of Toronto · Hong Kong Baptist University
Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flows
May 15, 2026Alex Massucco, Leonardo Del Grande, Marcello Carioni +2Transformer Architectures Wasserstein Gradient Flows
Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK · Department of Mathematics, University of Twente, Enschede, Netherlands
ITGPT: Generative Pretraining on Irregular Timeseries
May 15, 2026Antoine Honoré, Ming XiaoTime Series Single-Cell Foundation Models
⋆Division of Information Science and Engineering, KTH Royal Institute of Technology, Stockholm, Sweden
Grokking as Structural Inference: Transformers Need Bayesian Lottery Tickets
May 15, 2026Kai Hidajat, Solden Stoll, Joseph AnTransformer Architectures Sequence Modeling
Department of Computer Science University of Washington Seattle, WA 98195
HyperDiT: Hyper-Connected Transformers for High-Fidelity Pixel-Space Diffusion
May 15, 2026Yu He, Lichen Ma, Zipeng Guo +5Diffusion Transformers High-Fidelity
Going Beyond the Edge: Distributed Inference of Transformer Models on Ultra-Low-Power Wireless Devices
May 15, 2026Alexander Gräfe, Ding Huo, Vincent de Bakker +3Transformer Architectures Device
1RWTH Aachen University · 2TU Darmstadt
DreamSR: Towards Ultra-High-Resolution Image Super-Resolution via a Receptive-Field Enhanced Diffusion Transformer
May 15, 2026Qingji Dong, Hang Dong, Mingqin Chen +2Real-World Image Super-Resolution Diffusion Transformers
ByteDance Inc.
Transformer-like Inference from Optimal Control
May 15, 2026Aditya Kudre, Heng-Sheng Chang, Prashant G. MehtaTransformer Encoder Transformer Architectures
Coordinated Science Laboratory · Electrical and Computer Engineering · University of Illinois Urbana-Champaign +1
Gaussian Relational Graph Transformer
May 15, 2026Zezhong Ding, Jin Li, Xugang Wang +1Multiplex Graph Transformers Relational Deep Learning
School of Artificial Intelligence and Data Science, University of Science and Technology of China (USTC) · Data Darkness Lab, Suzhou Institute for Advanced Research, USTC · School of Biomedical Engineering, USTC +1
GiLT: Augmenting Transformer Language Models with Dependency Graphs
May 15, 2026Tianyu Huang, Yida Zhao, Chuyan Zhou +1Transformer Architectures Language Modeling
School of Information Science and Technology, ShanghaiTech University · Shanghai Engineering Research Center of Intelligent Vision and Imaging
3DTMDet: A Dual-Path Synergy Network of Transformer and SSM for 3D Object Detection in Point Clouds
May 15, 2026Bingwen Qiu, Yuan Liu, Junqi Bai +53D Object Detection Point Clouds
aSchool of Electronic and Optical Engineering, Nanjing University of Science and Technology, Nanjing, 210094, Jiangsu, China · bThe 28th Research Institute of China Electronics Technology Group Corporation, Nanjing, 210007, Jiangsu, China · cCollege of Astronautics, Nanjing University of Aeronautics and Astronautics, Nanjing, 210016, Jiangsu, China +2
Transformer Scalability Crisis: The First Comprehensive Empirical Analysis of Performance Walls in Modern Language Models
May 14, 2026Mahdi Naser Moghadasi, Faezeh GhaderiTransformer Architectures Model Size
Research Division, BrightMind AI, Seattle, WA · Texas Tech University, Lubbock, TX · University of Texas at Arlington, Arlington, TX
PRB-RUPFormer: A Recursive Unified Probabilistic Transformer for Residual PRB Forecasting
May 14, 2026Saad Masrur, Yuxuan Jiang, Matti Hiltunen +2Probabilistic Forecasting Mobile Networks
AT&T RAN Technology, Bedminster, NJ · Department of Electrical and Computer Engineering, North Carolina State University, Raleigh, NC
WorldParticle: Unified World Simulation of Lagrangian Particle Dynamics via Transformer
May 14, 2026Caoliwen Wang, Minghao Guo, Siyuan Chen +13Physics Simulation Particle Dynamics
University of British Columbia, Canada · MIT CSAIL, USA · Georgia Institute of Technology, USA +7
TFGN: Task-Free, Replay-Free Continual Pre-Training Without Catastrophic Forgetting at LLM Scale
May 14, 2026Anurup GanguliPretraining Catastrophic Forgetting
Independent Researcher
Representative Attention For Vision Transformers
May 14, 2026Yuntong Li, Hainuo Wang, Hengxing Liu +2Self-Supervised Vision Transformers Vision Transformer
College of Intelligence and Computing, Tianjin University, Tianjin 300350, China
Your CLIP has 164 dimensions of noise: Exploring the embeddings covariance eigenspectrum of contrastively pretrained vision-language transformers
May 14, 2026Jakub Grzywaczewski, Dawid Płudowski, Przemysław BiecekVision-Language Foundation Models Transformer Architectures
Warsaw University of Technology · Centre for Credible Artificial Intelligence · University of Warsaw
GeoViSTA: Geospatial Vision-Tabular Transformer for Multimodal Environment Representation
May 14, 2026Yuhao Liu, Sadeer Al-Kindi, Ashok Veeraraghavan +1Spatio-Temporal Transformers Modality Gap
Department of Electrical and Computer Engineering, Rice University, Houston, TX 77005 · Center for Cardiovascular Computational and Precision Health, Department of Cardiology, DeBakey Heart and Vascular Center, Houston Methodist, Houston, TX 77030
Ideology Prediction of German Political Texts
May 14, 2026Sinclair Schneider, Florian Steuber, Joao A. G. Schneider +1Political Discourse German
Bundeswehr University Munich · Research Institute CODE · Carl-Wery-Straße 22 +1
Precise Verification of Transformers through ReLU-Catalyzed Abstraction Refinement
May 14, 2026Hengjie Liu, Zhenya Zhang, Jianjun ZhaoTransformer Architectures Rectified Linear Unit
Kyushu University, Fukuoka, Japan · National Institute of Informatics, Tokyo, Japan
Diagnosing and Correcting Concept Omission in Multimodal Diffusion Transformers
May 14, 2026Kanghyun Baek, Jaihyun Lew, Chaehun Shin +2Diffusion Transformers Omissions
Interdisciplinary Program in Artificial Intelligence, Seoul National University, Seoul, South Korea · Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea · Department of Computer Science & Engineering, Korea University, Seoul, South Korea +1
Dynamics of the Transformer Residual Stream: Coupling Spectral Geometry to Network Topology
May 14, 2026Jesseba Fernando, Grigori GuitchountsTransformer Residual Streams Residual Stream
Network Science Institute, Northeastern University · Flagship Pioneering
CoReDiT: Spatial Coherence-Guided Token Pruning and Reconstruction for Efficient Diffusion Transformers
May 13, 2026Zhuojin Li, Hsin-Pai Cheng, Hong Cai +2Diffusion Transformers Visual Token Pruning
Qualcomm AI Research†
Evolving Layer-Specific Scalar Functions for Hardware-Aware Transformer Adaptation
May 13, 2026Kieran Carrigg, Sigur de Vries, Amirhossein Sadough +1Vision Transformer Transformer Architectures
Department of Machine Learning and Neural Computing Donders Institute for Brain, Cognition, and Behaviour Thomas van Aquinostraat 4, 6525 GD Nijmegen, The Netherlands

May 20, 2026, cs.CVJ/K move · Enter open · S save

UniT: Unified Geometry Learning with Group Autoregressive Transformer

Haotian Wang, Yusong Huang, Zhaonian Kuang, Hongliang Lu+3

Intelligent Transportation Thrust of the Systems Hub, The Hong Kong University of Science and Technology (GZ), Guangzhou, P.R.China. · The National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University, Xi’an, P.R.China. · Applied Science, Amazon.com, Inc., USA.

Visual Geometry Grounded Transformer Autoregressive Transformers Transformer Architectures

Open paper AskPDF arXiv

Recent feed-forward models have significantly advanced geometry perception for inferring dense 3D structure from sensor observations. However, its essential capabilities remain fragmented across multiple incompatible paradigms, including online perception, offline reconstruction, multi-modal integration, long-horizon scalability, and metric-scale estimation. We present UniT, a unified model built upon a novel Group Autoregressive Transformer, which reformulates these seemingly disparate capabilities within a single framework. The key idea is to treat groups of sensor observations as the basic autoregressive units and predict the corresponding point maps in an anchor-free and scale-adaptive manner. More specifically, diverse view configurations in both online and offline settings are naturally unified within a single group autoregression process. By varying the group size, online mode operates over multiple autoregressive steps with single-frame groups, whereas offline mode aggregates a multi-frame group in a single forward pass. Meanwhile, a queue-style KV caching mechanism ensures bounded autoregressive memory over long horizons. This is enabled by reducing long-range dependencies on early frames through anchor-free relational modeling, thereby allowing outdated memory to be discarded on the fly. To improve metric-scale generalization across scenes, a scale-adaptive geometry loss is further introduced within this framework. It couples relative geometric constraints with a partial absolute scale term, implicitly regularizing global scale and inducing a progressive transition from scale-invariant geometry to metric-scale solutions. Together with a dedicated modal attention module for integrating auxiliary modalities, UniT achieves state-of-the-art performance in unified geometry perception, as validated on ten benchmarks spanning seven representative tasks.

Momentum

Related topics

Latest papers 1,120