Transformer Architectures

Follow this topic Ask about this topic

Momentum

148 papers in the last four weeks, up 164% on the four weeks before. 1.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,120

All topics

Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility
May 13, 2026Gergely Szilvasy, Manuel Faysse, Maria Lomeli +5Key-Value Cache Compression Key-Value Cache
Meta FAIR · MICS, CentraleSupélec
Attention Once Is All You Need: Efficient Streaming Inference with Stateful Transformers
May 13, 2026Victor NorgrenFast Inference Streaming
LayerScale, Inc.
Effective Context in Transformers: An Analysis of Fragmentation and Tokenization
May 13, 2026Amirmehdi Jafari Fesharaki, Mohammadamin Rami, Aslan TchamkertenTransformer Architectures Subword Tokenization
Department of Communications and Electronics Institut Polytechnique de Paris
Z-Order Transformer for Feed-Forward Gaussian Splatting
May 13, 2026Can Wang, Lei Liu, Wei Jiang +1Feed-Forward 3D Gaussian Splatting Feed-Forward 3D Gaussian
The University of Hong Kong · Futurewei Technologies Inc.
A Neural Hierarchical-Matrix Preconditioner for Real-Time GPU Solves
May 13, 2026Carl Osborne, Minghao Guo, Crystal Owens +1Spectral Preconditioning High-Performance Computing
MIT CSAIL, Cambridge, USA
Chem-GMNet: A Sphere-Native Geometric Transformer for Molecular Property Prediction
May 13, 2026Deepak Warrier, Raja Sekhar PappalaMoleculenet Molecular Property Prediction
MSTACK AI
ECG-NAT: A Self-supervised Neighborhood Attention Transformer for Multi-lead Electrocardiogram Classification
May 13, 2026Mahsa Gazeran, Sayvan Soleymanbaigi, Fatemeh Daneshfar +2Electrocardiogram Classification Self-Supervised Learning
Department of Computer Engineering, University of Kurdistan, Sanandaj, Iran · Department of Mathematics and Operational Research, University of Mons, Mons, Belgium
N-vium: Mixture-of-Exits Transformer for Accelerated Exact Generation
May 13, 2026Aleksander Lorenc, Frédéric Berdoz, Joël Mathys +1Autoregressive Transformers Transformer Architectures
ETH Zurich
Exact Linear Attention
May 13, 2026Weinuo OuKimi Delta Attention Transformer Attention
Wuyi University
Transformers Linearly Represent Highly Structured World Models
May 13, 2026Roman Kniazev, Nathanaël FijalkowTransformer Architectures Combinations
LaBRI, CNRS · University of Bordeaux · France
Precision Tracked Transformer via Kalman Filtering, Kriging and Process Noise
May 12, 2026Bo Long, Deepak Agarwal, Jelena Markovic-Voronov +2Transformer Architectures Bayesian Filtering
LinkedIn Core AI
Solve the Loop: Attractor Models for Language and Reasoning
May 12, 2026Jacob Fein-Ashley, Paria RashidinejadLarge Language Model Pretraining Transformer Architectures
University of Southern California
Geometric Factual Recall in Transformers
May 12, 2026Shauli Ravfogel, Gilad Yehudai, Joan Bruna +1Transformer Architectures Factual Recall
1New York University · 2Flatiron Institute
Recurrent Transformer-Based Near- and Far-Field THz Wideband Channel Estimation for UM-MIMO
May 12, 2026Dmitry Artemasov, Alexander Shmatok, Kirill Andreev +3Channel Estimation Beamforming
Center for Next Generation Wireless and IoT, Skolkovo Institute of Science and Technology, Moscow, Russia · Department of IEOR, Indian Institute of Technology Bombay, India · Faculty of Computer and Engineering Sciences, Innopolis University, Innopolis, Russia
Lower bounds for one-layer transformers that compute parity
May 12, 2026Daniel HsuLinear Attention Rectified Linear Unit Networks
From Index to Equity: Pre-Training Transformers for Stock Return Prediction
May 12, 2026Marie Soehl Coolsaet, Roberto Gallardo, Zhen GaoForecasting Backbone Transformer Architectures
Faculty of Engineering, McMaster University · Department* of Economics, Universidad Veracruzana, Mexico
What-Where Transformer: A Slot-Centric Visual Backbone for Concurrent Representation and Localization
May 12, 2026Ryota Yoshihashi, Masahiro Kada, Satoshi Ikehata +2Vision Transformer Object Localization
Institute of Science Tokyo · 2DENSO IT Laboratory · 3National Institute of Informatics
Rethinking Positional Encoding for Neural Vehicle Routing
May 12, 2026Chuanbo Hua, Federico Berto, Andre Hottung +8Vehicle Routing Problem Positional Encoding
KAIST · Radical Numerics · Bielefeld University +3
On Privacy-Preserving Image Transmission in Low-Altitude Networks: A Swin Transformer-Based Framework with Federated Learning
May 12, 2026Kexin Zhang, Lixin Li, Yuna Yan +5Unmanned Aerial Vehicles Semantic Communications
School of Electronics and Information, Northwestern Polytechnical University, Xi’an 710129, China · DecoreX Intelligent Technologies Co., Ltd., Xi’an 710075, China · Samsung AI Center, Cambridge CB1 2JH, UK +2
Training-Inference Consistent Segmented Execution for Long-Context LLMs
May 12, 2026Xianpeng Shang, Jiang Li, Zehua Duo +2Efficient Long-Context Inference LLM Inference Optimization
College of Computer Science, Inner Mongolia University, Hohhot 010021, China · National & Local Joint Engineering Research Center of Intelligent Information Processing Technology for Mongolian, Hohhot 010021, China · Inner Mongolia Key Laboratory of Multilingual Artificial Intelligence Technology, Hohhot 010021, China +1
Toward Stable Value Alignment: Introducing Independent Modules for Consistent Value Guidance
May 12, 2026Wenhao Chen, Sirui Sun, Shengyuan Bai +1Large Language Model Alignment Large Language Model Safety
School of Electronics Engineering and Computer Science, Peking University · Yuanpei College, Peking University · State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University
Transformer Interpretability from Perspective of Attention and Gradient
May 12, 2026Yongjin Cui, Xiaohui Fan, Huajun ChenTransformer Attention Vision Transformer
Zhejiang University
Latent Chain-of-Thought Improves Structured-Data Transformers
May 11, 2026Carson Dudley, Samet OymakTransformer Architectures Chain-of-Thought Reasoning
Variational Linear Attention: Stable Associative Memory for Long-Context Transformers
May 11, 2026Vishal Pandey, Gopal SinghKimi Delta Attention Linear Attention
Independent Researcher London, UK · Metriqual Athens, GR
HiDream-O1-Image: A Natively Unified Image Generative Foundation Model with Pixel-level Unified Transformer
May 11, 2026Qi Cai, Jingwen Chen, Chengmin Gao +22Single-Cell Foundation Models Generative Models
DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
May 11, 2026Chenyang Song, Weilin Zhao, Xu Han +3Mixture-Of-Experts Architectures Mixture-Of-Experts
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China
Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime
May 11, 2026Albert Alcalde, Leon Bungert, Konstantin Riedl +1Transformer Encoder Transformer Architectures
Department of Mathematics Friedrich-Alexander-University Erlangen-Nürnberg · Institute of Mathematics CAIDAS University of Würzburg · Mathematical Institute University of Oxford Reuben College +1
Uniform Scaling Limits in AdamW-Trained Transformers
May 11, 2026William Gibson, Christoph ReisingerTransformer Architectures Hidden States
Mathematical Institute · University of Oxford · Oxford, OX2 6GG, UK
A Single-Layer Model Can Do Language Modeling
May 11, 2026Zanmin WangLanguage Modeling Transformer Architectures
DetRefiner: Model-Agnostic Detection Refinement with Feature Fusion Transformer
May 11, 2026Soichiro Okazaki, Tatsuya Sasaki, Hiroki OhashiOpen-Vocabulary Object Detection Unsupervised Detection
Hitachi, Ltd. Research and Development Group, Japan
DynGhost: Temporally-Modelled Transformer for Dynamic Ghost Imagings
May 11, 2026Vittorio Palladino, Ahmet Enis CetinSingle-Photon Avalanche Diode Detection Transformer
Politecnico di Milano Milan, Italy · University of Illinois at Chicago Chicago, IL, USA · Department of Electrical and Computer Engineering University of Illinois at Chicago Chicago, IL, USA +1
MicroViTv2: Beyond the FLOPS for Edge Energy-Friendly Vision Transformers
May 11, 2026Novendra Setyawan, Chi-Chia Sun, Mao-Hsiu Hsu +2Self-Supervised Vision Transformers Vision Transformer
Department of Electro-Optics, National Formosa University, Taiwan · Department of Electrical Engineering, University of Muhammadiyah Malang, Indonesia · Department of Electrical Engineering, National Taipei University, Taiwan +1
Fashion130K: An E-commerce Fashion Dataset for Outfit Generation with Unified Multi-modal Condition
May 11, 2026Yu He, Ting Zhu, Yichun Liu +6Garments Multimodal Generation
Simply Stabilizing the Loop via Fully Looped Transformer
May 11, 2026Rao Fu, Zixuan Yang, Jiankun Zhang +4Transformer Architectures Loop
Hong Kong Baptist University · Jilin University
Robust Basis Spline Decoupling for the Compression of Transformer Models
May 11, 2026Joppe De Jonghe, Van Tien Pham, Mariya IshtevaTensor Decomposition Transformer Architectures
aNUMA, Department of Computer Science, KU Leuven, Kleinhoefstraat 4, Geel, 2440, Antwerpen, België · bUniversité Paris-Saclay, CEA, List, Palaiseau, F-91120, France
Attention Drift: What Autoregressive Speculative Decoding Models Learn
May 11, 2026Doğaç Eldenk, Payal Mohapatra, Yigitcan Comlek +3Speculative Decoding Drafter
Northwestern University · GE Aerospace · fal +1
Rethinking Random Transformers as Adaptive Sequence Smoothers for Sleep Staging
May 11, 2026Guisong Liu, Xin Gao, Martin Dresler +2Sleep Spatio-Temporal Transformers
School of Biological Science and Medical Engineering Southeast University Nanjing, China · University of Bath Bath, England · Donders Institute for Brain, Cognition and Behaviour Radboud University Medical Center Nijmegen, The Netherlands +1
Dissecting Jet-Tagger Through Mechanistic Interpretability
May 11, 2026Saurabh Rai, Sanmay GangulyMechanistic Interpretability Interpretability
Department of Physics, Indian Institute of Technology, Kanpur. Uttar-Pradesh 208016, India
Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memory
May 11, 2026Daniel Goldstein, Eugene CheahEfficient Long-Context Inference Transformer Architectures
Featherless AI · Eleuther AI
Continuous Latent Contexts Enable Efficient Online Learning in Transformers
May 11, 2026Emile Anand, Abdullah Ateyeh, Xinyuan Cao +1In-Context Learning Transformer Architectures
Georgia Institute of Technology · University of California, Berkeley · Columbia University
Probing Routing-Conditional Calibration in Attention-Residual Transformers
May 11, 2026Wenhao Liang, Lin Yue, Wei Emma Zhang +4Post-Hoc Transformer Architectures
Adelaide University · Australian Institute for Machine Learning (AIML), Adelaide University · The University of Queensland
The Association of Transformer-based Sentiment Analysis with Symptom Distress and Deterioration in Routine Psychotherapy Care
May 11, 2026Douglas K. Faust, Peter Awad, Alexandre Vaz +1Depression Cognitive Behavioral Therapy
Sentio University, Torrance, CA, USA · Western Washington University, Department of Mathematics, Bellingham, WA, USA
Pretraining large language models with MXFP4 on Native FP4 Hardware
May 11, 2026Musa Cim, Poovaiah Palangappa, Miro Hodak +3Large Language Model Quantization Fp8
1The Pennsylvania State University · 2Advanced Micro Devices, Inc.
TokaMind for Power Grid: Cross-Domain Transfer from Fusion Plasma
May 10, 2026JC Wu, Norton Lee, Kai Siang ChenPower Systems Transformer Architectures
TaiScience Research Group, affiliated with Fu Jen Catholic University, Taiwan · Center for Geometry and Physics, Institute for Basic Science (IBS), South Korea · Fu Jen Catholic University, Taiwan
CALYREX: Cross-Attention LaYeR EXtended Transformers for System Prompt Anchoring
May 10, 2026Li LixingInstruction-Tuned Models Cross-Attention
Cornell University Ithaca, NY 14853
One for All: A Non-Linear Transformer can Enable Cross-Domain Generalization for In-Context Reinforcement Learning
May 10, 2026Bowen He, Juncheng Dong, Lin Lin +1In-Context Learning Transformer Architectures
Duke University, Durham, U.S.A.
Benchmarking Transformer and xLSTM for Time-Series Forecasting of Heat Consumption
May 10, 2026Marja Wahl, Daniel R. Bayer, Sven Rausch +1Forecasting Backbone Time Series Forecasting
RAUSCH Technology GmbH, Würzburg, Germany · Modeling and Simulation, University of Würzburg, Würzburg, Germany
Spatial-Frequency Gated Swin Transformer for Remote Sensing Single-Image Super-Resolution
May 10, 2026Md Aminur Hossain, Parekh Valkesh, Ayush V. Patel +3Hyperspectral Image Super-Resolution Real-World Image Super-Resolution
Space Applications Centre, ISRO, Ahmedabad, India · New L J Institute of Engineering and Technology, Ahmedabad, India · Pandit Deendayal Energy University, Gandhinagar, India +2
Mixture of Layers with Hybrid Attention
May 10, 2026Ivan Ternovtsii, Yurii BilakTransformer Attention Mixture-Of-Experts
Department of Software Systems, Uzhhorod National University Narodna sq. 3, Uzhhorod, Ukraine, 88000
Spectral Transformer Neural Processes
May 10, 2026Xianhe Chen, Hao Chen, Yingzhen LiTimestep Embeddings Frequency-Aware Regularization
University of Cambridge · Tencent · Imperial College London
CTQWformer: A CTQW-based Transformer for Graph Classification
May 10, 2026Zhan Li, Wuqing Yu, Yusen Wu +1Multiplex Graph Transformers Graph Neural Networks
school of Artificial Intelligence, Beijing Normal University, Beijing, China
ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Network
May 10, 2026Qian Chen, Junqiao Zhao, Hongtu Zhou +4Soft Actor-Critic Action Chunks
Tongji University
The Transformer as a Polar State Estimator
May 10, 2026Peter RacioppoEnergy-Based Transformer Transformer Architectures
Independent Researcher Los Angeles, CA
Uncertainty-Aware Token Importance Estimation in Spiking Transformers
May 10, 2026Wenxuan Liu, Zecheng Hao, Tong Bu +2Spatial Tokens Transformer Architectures
School of Computer Science, Peking University · School of Computer Science, Peking University. · Institute for Artificial Intelligence, Peking University +2
Kinetic theory for Transformers and the lost-in-the-middle phenomenon
May 9, 2026Mitia Duerinckx, Borjan Geshkovski, Stefano RossiMean-Field Limit Causal Attention
RigidFormer: Learning Rigid Dynamics using Transformers
May 9, 2026Zhiyang Dou, Minghao Guo, Haixu Wu +3Rigid-Body Dynamics Transformer Architectures
MIT · Meta
SHIELD: Scalable Optimal Control with Certification using Duality and Convexity
May 9, 2026Hansung Kim, Siddharth H. Nair, Francesco BorrelliModel Predictive Control Optimal Control
Model Predictive Control Laboratory, UC Berkeley · Nextracker Inc.
Sparse Layers are Critical to Scaling Looped Language Models
May 9, 2026Ryan Lee, Jacob Biloki, Edward J. Hu +1Transformer Architectures Layer-Wise
1USC Information Sciences Institute · 2Netflix · 3Independent Researcher
Character-Level Transformer for Tajik-Persian Transliteration with a Parallel Lexical Corpus
May 9, 2026Mullosharaf K. ArabovPersian Text Corpora
Institute of Computational Mathematics and Information Technologies Kazan Federal University Kazan, Russia
VORT: Adaptive Power-Law Memory for NLP Transformers
May 9, 2026Nabil MlaikiTransformer Architectures Peak Memory
Department of Mathematics and Sciences, Prince Sultan University, Riyadh, Saudi Arabia

May 13, 2026, cs.LGJ/K move · Enter open · S save

Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility

Gergely Szilvasy, Manuel Faysse, Maria Lomeli, Matthijs Douze+4

Meta FAIR · MICS, CentraleSupélec

Key-Value Cache Compression Key-Value Cache Transformer Architectures Next-Token Prediction

Open paper AskPDF arXiv

Under modern test-time compute and agentic paradigms, language models process ever-longer sequences. Efficient text generation with transformer architectures is increasingly constrained by the Key-Value cache memory footprint and bandwidth. To address this limitation, we introduce Self-Pruned Key-Value Attention (SP-KV), a mechanism designed to predict future KV utility in order to reduce the size of the long-term KV cache. This strategy operates at a fine granularity: a lightweight utility predictor scores each key-value pair, and while recent KVs are always available via a local window, older pairs are written in the cache and used in global attention only if their predicted utility surpasses a given threshold. The LLM and the utility predictor are trained jointly end-to-end exclusively through next-token prediction loss, and are adapted from pretrained LLM checkpoints. Rather than enforcing a fixed compression ratio, SP-KV performs dynamic sparsification: the mechanism adapts to the input and typically reduces the KV cache size by a factor of

3

10\times

, longer sequences often being more compressible. This leads to vast improvements in memory usage and decoding speed, with little to no degradation of validation loss nor performance on a broad set of downstream tasks. Beyond serving as an effective KV-cache reduction mechanism, our method reveals structured layer- and head-specific sparsity patterns that we can use to guide the design of hybrid local-global attention architectures.

Momentum

Related topics

Latest papers 1,120