Transformer

Follow this topic Ask about this topic

Momentum

26 papers in the last four weeks, up 44% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 351

All topics

Dual Attention Residuals
Jul 21, 2026Xingda Yu, Yining Li, Xinzhang Liu +5Transformer Cross-Attention
TeleAI
On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers
Jul 20, 2026Sixu Li, Thomas Jacob Maranzatto, Jan Peszek +5Dynamical Systems Transformer Inference
University of Wisconsin–Madison, Department of Statistics · University of Maryland, Department of Electrical and Computer Engineering · University of Warsaw, Institute of Applied Mathematics and Mechanics +2
Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning
Jul 20, 2026Valentijn Oldenburg, Floris de Kam, Bente Zuijdam +5Transformer Hyper-Connections
University of Amsterdam, Netherlands
A Controlled Study of Attention-Only Transformers
Jul 20, 2026Henry Ndubuaku, Karen Mosoyan, Jakub Mroz +5Transformer FFNs Transformer
Cactus Compute, Inc.
Mobius Learning: Cyclic Depth Folding in Transformers
Jul 20, 2026Tongtian ZhuTransformer Decoder-Only Language Models
Zhejiang University
Brain-Aligned Multi-Stream Video Transformers with Sparse Self-Selection
Jul 20, 2026Amir Hosein Fadaei, Mahyar Maleki, Mohammad-Reza A. DehaqaniTransformer Efficient ViTs
University of Tehran, Faculty of Electrical and Computer Engineering, North Kargar st., Tehran, 1439957131, Tehran, Iran
Visible-Light Imaging Diagnosis of Neutral Particle Emission Tomography in the Tokamak Divertor: An Efficient Transformer-based Surrogate Model
Jul 19, 2026Xiao Wang, Hao Si, Qiang Chen +8Neural Surrogate Modeling Transformer
School of Computer Science and Technology, Anhui University, Hefei 230601, China. · Institute of Plasma Physics, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China. · School of Artificial Intelligence, Anhui University, Hefei 230601, China.
DynImmune-BERT: Dynamic Immune Repertoire Modeling with Neural ODE Driven Continuous Transformers
Jul 19, 2026Rong Fu, Yongtai Liu, Xiaowen Ma +5Latent Dynamics Modeling Transformer
Independent Researcher
The Geometry of Semantic Space: A Continuous Geometric Framework for the Transformer Architecture
Jul 19, 2026Zhihua LiangTransformer Transformer Attention
INFN, Sezione di Cagliari, I-09042 Monserrato (CA), Italy
Physics-Guided Masked Multi-Task Network for Edge-Friendly Battery Health Diagnostics from Sto-chastically Fragmented Charging Profiles
Jul 19, 2026Shuhao Chen, Tianyu Shi, Chengyi TuRUL Estimation Transformer
School of Economics and Management, Zhejiang Sci-Tech University, Hangzhou, 310018, China.
What does a Bayes-filtered transformer believe? A predictive Monte Carlo approach
Jul 19, 2026Afiq Abdillah Effiezal Aswadi, Haotong Ma, Susan WeiTransformer Interpretability Latent Variable Models
Department of Econometrics and Business Statistics Monash University
Boundary-Seeking GAN-Augmented TabTransformer for Adversarially Robust Intrusion Detection
Jul 17, 2026Raihan Sultan Pasha Basuki, Aliyah KurniasihSynthetic Data Augmentation Transformer
Department of Computer Science, Universitas Ary Ginanjar, Menara 165, Cilandak No.kav 1 Lt 2 & 18, Jakarta Selatan, 12560, DKI Jakarta, Indonesia
FlashDecoder: Real-Time Latent-to-Pixel Streaming Decoder with Transformers
Jul 16, 2026Minguk Kang, Suha KwakEfficient Transformer Inference Transformer
Pika Labs · POSTECH
xHC: Expanded Hyper-Connections
Jul 16, 2026Xiangdong Zhang, Xiaohan Qin, Sunan Zou +10Language Model Pretraining Transformer
School of AI, Shanghai Jiao Tong University · Dots Studio, Xiaohongshu Inc. · School of CS, Peking University +3
Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth
Jul 15, 2026Katie EverettTransformer FFNs Transformer
MIT CSAIL
When Close Enough Is Not Enough: Autoregressive Drift in Quantum Circuit Synthesis
Jul 14, 2026Mehdi Saeedi, Eddie Richter, Paul HartkeTransformer
Advanced Micro Devices, Inc. (AMD)
From Expressivity to Sample Complexity: Narrow Teachers for Transformers via C-RASP
Jul 13, 2026Michael Rizvi-Martel, Satwik Bhattamishra, Guillaume Rabusseau +1Transformer Sample Complexity
Mila & Universit´e de Montr´eal · University of Oxford · Saarland University
Comparative Analysis of GAT and BERT for Human-Like Playtesting
Jul 13, 2026Kleio Fragkedaki, Theodoros Panagiotakopoulos, Matteo Biasielli +1Graph Attention Networks Graph Neural Networks
AI Center of Excellence King Stockholm, Sweden
RepTran: Search-Based Repair of Transformer Models
Jul 13, 2026Yuta Ishimoto, Paolo Arcaini, Fuyuki Ishikawa +3Transformer
The University of Osaka, Japan · National Institute of Informatics, Japan · Kyushu University, Japan +1
Infrared Organization and Critical Cognitive Field Formation in Transformer Dynamics
Jul 12, 2026Byung Gyu ChaeTransformer Neural Network Training Dynamics
Electronics and Telecommunications Research Institute, 218 Gajeong-ro, Yuseong-gu, Daejeon 34129, Republic of Korea
AutoNorm: Understanding Adaptive Normalization in Transformers through Differentiable Gating
Jul 12, 2026Piyush Kaushik Bhattacharyya, Divyanshu Rai, Swastik Singh +3Dynamic Neural Networks Transformer
When Data Imbalance Helps: Robust Generalization Through Shortcut Saturation
Jul 11, 2026Cheng-Ting Chou, Duc Binh HoangClass-Imbalanced Learning Transformer
University of California, Los Angeles
MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers
Jul 10, 2026Roberto Garcia, Jerry Liu, Ronny Junkins +3Transformer Memorization in Language Models
Institute for Computational & Mathematical Engineering, Stanford University · Department of Computer Science, Stanford University · Department of Computer Science and Engineering, University at Buffalo
Do Transformer Temporal Heads and Post-Pooling Motion Gates Help CorrNet-based CSLR? An Empirical Study
Jul 10, 2026Lisi Wang, Zhidong Xiao, Jianjun PengTransformer Sign Language Recognition
University of New South Wales, Sydney, Australia · National Centre for Computer Animation, Faculty of Media, Science and Technology, Bournemouth University, BH12 5BB, United Kingdom · School of Information Science and Engineering, Dalian Polytechnic University, Dalian 116034, China
COBART: Controlled, Optimized, Bidirectional and Auto-Regressive Transformer for Ad Headline Generation
Jul 9, 2026Yashal Shakti Kanungo, Gyanendra Das, Pooja A +1Transformer Online Advertising
Amazon United States of America · Amazon India
Bi-PT: Bidirectional Cross-Attention Point Transformers for Four-Chamber Heart Reconstruction from Sparse Cardiac MRI Data
Jul 8, 2026Chenchuhui Hu, Shaoming Pan, Leon Axel +1Transformer 3D Medical Imaging
Department of Computer Science and Engineering, University of Texas at Arlington, Arlington, USA · School of Medicine, New York University, New York, NY, USA
Dual Attention Heads for Personalized Federated Learning in ECG Classification
Jul 7, 2026Kien Le, Joseph Lindley, Quoc Bao Phan +1Transformer Electrocardiogram Classification
Computer Science College of Arts and Sciences Florida State University Tallahassee, FL 32310, USA · Chemical & Biomedical Engineering FAMU-FSU College of Engineering Florida State University Tallahassee, FL 32310, USA · Electrical & Computer Engineering FAMU-FSU College of Engineering Florida State University Tallahassee, FL 32310, USA
At-Grok Is Not Converged:A Measurement-Validity Audit for Grokking Representation Metrics
Jul 7, 2026Truong Xuan KhanhTransformer Modular Arithmetic
H&K Research Studio / Clevix LLC, Hanoi, Vietnam
Input Pathways Shape Few-Shot, Not Zero-Shot, Binding in Tiny Transformers: A Fully-Enumerable Study
Jul 6, 2026Yoshiyuki OotaniRepresentation Learning Transformer
Independent Researcher
Structure-Specific Representational Priors Causally Control the Grokking Delay
Jul 5, 2026Gunner Levi HoweTransformer Grokking

Jul 21, 2026, cs.CLJ/K move · Enter open · S save

Dual Attention Residuals

Xingda Yu, Yining Li, Xinzhang Liu, Zhihao Yang+4

TeleAI

Transformer Cross-Attention Attention Residual Connections Residual Learning

Open paper AskPDF arXiv

Recent work extends Transformer residual pathways along two complementary axes: historical retrieval selects information from earlier depths, whereas multi-stream methods maintain multiple residual trajectories. These capabilities have largely been studied in isolation, and assigning an independent retriever to each stream still prevents one trajectory from influencing depth selection in another. We propose Dual Attention Residuals (DAR), which brings multi-stream interaction into historical retrieval through reciprocal cross-stream addressing. For each target stream, DAR computes depth weights from normalized states in the opposite stream and applies them to values from the target stream's own history. The retrieved states are combined for an unchanged Transformer branch and updated through constrained gated writes; a block-form variant operates on block-level histories to control overhead. Across dense models from 0.1B to 1B parameters and a 7B sparse-MoE model, DAR consistently improves validation loss over standard residual Transformers and Attention Residuals. Routing ablations show that the gain cannot be explained by an additional stream or value projection alone. Representation and intervention analyses further show that reciprocal cross-stream selection preserves depth-wise diversity and avoids the redundancy or functional imbalance observed in alternative two-stream designs.

Momentum

Related topics

Latest papers 351