Transformers

Follow this field Ask about this field

Recent momentum

emerging

0 papers in the last 28 days · 0.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this field, kept on the site without email delivery.

Period ending 2026-09-21

42 new papers

A weekly snapshot of new work published in Transformers.

Period ending 2026-09-14

3 new papers

A weekly snapshot of new work published in Transformers.

Inside this field

Focused directions

Focused topic

Transformer Architectures

Within Transformers

684 papersExplore

Focused topic

Quadratic Attention

Within Transformers

144 papersExplore

Focused topic

Dynamic Sparse Attention

Within Transformers

109 papersExplore

Focused topic

Transformer Attention

Within Transformers

89 papersExplore

Focused topic

Transformer Encoder

Within Transformers

80 papersExplore

Focused topic

Transformer Layers

Within Transformers

80 papersExplore

Focused topic

Softmax Attention

Within Transformers

74 papersExplore

Focused topic

Unified Transformer

Within Transformers

40 papersExplore

Focused topic

Multi-Head Self-Attention

Within Transformers

35 papersExplore

Focused topic

Autoregressive Transformers

Within Transformers

33 papersExplore

Focused topic

Global Attention Mechanism

Within Transformers

20 papersExplore

1,208 papers

Latest in Transformers

Browse all topics →

Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs
May 15, 2026Vincent-Daniel Yun, Junhyuk Jo, Sai Praneeth Karimireddy +1Transformer Layers Model Activations
DualKV: Shared-Prompt Flash Attention for Efficient RL Training with Large Rollouts and Long Contexts
May 14, 2026Jiading Gai, Shuai Zhang, Xiang Song +2Quadratic Attention Efficient Long-Context Inference
Transformer Scalability Crisis: The First Comprehensive Empirical Analysis of Performance Walls in Modern Language Models
May 14, 2026Mahdi Naser Moghadasi, Faezeh GhaderiTransformer Architectures Large Models
PRB-RUPFormer: A Recursive Unified Probabilistic Transformer for Residual PRB Forecasting
May 14, 2026Saad Masrur, Yuxuan Jiang, Matti Hiltunen +2Mobile Networks Spectrum
WorldParticle: Unified World Simulation of Lagrangian Particle Dynamics via Transformer
May 14, 2026Caoliwen Wang, Minghao Guo, Siyuan Chen +13Physical Dynamics Deformable Objects
Representative Attention For Vision Transformers
May 14, 2026Yuntong Li, Hainuo Wang, Hengxing Liu +2Self-Supervised Vision Transformers Vision Transformer
Your CLIP has 164 dimensions of noise: Exploring the embeddings covariance eigenspectrum of contrastively pretrained vision-language transformers
May 14, 2026Jakub Grzywaczewski, Dawid Płudowski, Przemysław BiecekVision-Language Large Vision Language Models
HASTE: Training-Free Video Diffusion Acceleration via Head-Wise Adaptive Sparse Attention
May 14, 2026Xuzhe Zheng, Yuexiao Ma, Jing Xu +3Pre-Trained Video Diffusion Models Dynamic Sparse Attention
GeoViSTA: Geospatial Vision-Tabular Transformer for Multimodal Environment Representation
May 14, 2026Yuhao Liu, Sadeer Al-Kindi, Ashok Veeraraghavan +1Geospatial Foundation Models Spatial Representation
Where Should Diffusion Enter a Language Model? Geometry-Guided Hidden-State Replacement
May 14, 2026Injin Kong, Hyoungjoon Lee, Yohan JoDiffusion Language Models Autoregressive Transformers
Precise Verification of Transformers through ReLU-Catalyzed Abstraction Refinement
May 14, 2026Hengjie Liu, Zhenya Zhang, Jianjun ZhaoNeural Network Verification Transformer Architectures
Minimal-Intervention KV Retention via Set-Conditioned Diversity
May 14, 2026Libo Sun, Po-wei Harn, Peixiong He +1Key-Value Cache Compression Cache
Diagnosing and Correcting Concept Omission in Multimodal Diffusion Transformers
May 14, 2026Kanghyun Baek, Jaihyun Lew, Chaehun Shin +2Multimodal Diffusion Transformers Multimodal Generation
Dynamics of the Transformer Residual Stream: Coupling Spectral Geometry to Network Topology
May 14, 2026Jesseba Fernando, Grigori GuitchountsTransformer Residual Streams Residual Stream
DT-Transformer: A Foundation Model for Disease Trajectory Prediction on a Real-world Health System
May 14, 2026Yunying Zhu, Andrew R Weckstein, Kueiyu Joshua Lin +1Electronic Health Record Foundation Models Clinical Prediction
CoReDiT: Spatial Coherence-Guided Token Pruning and Reconstruction for Efficient Diffusion Transformers
May 13, 2026Zhuojin Li, Hsin-Pai Cheng, Hong Cai +2Multimodal Diffusion Transformers Visual Token Pruning
Evolving Layer-Specific Scalar Functions for Hardware-Aware Transformer Adaptation
May 13, 2026Kieran Carrigg, Sigur de Vries, Amirhossein Sadough +1Vision Transformer Sparse Deep Neural Network Accelerators
Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility
May 13, 2026Gergely Szilvasy, Manuel Faysse, Maria Lomeli +5Key-Value Cache Compression Key-Value Cache
Attention Once Is All You Need: Efficient Streaming Inference with Stateful Transformers
May 13, 2026Victor NorgrenLLM Inference Efficiency Real-Time Inference
Effective Context in Transformers: An Analysis of Fragmentation and Tokenization
May 13, 2026Amirmehdi Jafari Fesharaki, Mohammadamin Rami, Aslan TchamkertenTokenization Transformer Architectures
OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention
May 13, 2026Chenyu Zhou, Hongpei Li, Yuerou Liu +3Softmax Attention Spectral Preconditioning
Z-Order Transformer for Feed-Forward Gaussian Splatting
May 13, 2026Can Wang, Lei Liu, Wei Jiang +1Feed-Forward 3D Gaussian Splatting Feed-Forward 3D Gaussian
Exploiting Pre-trained Encoder-Decoder Transformers for Sequence-to-Sequence Constituent Parsing
May 13, 2026Daniel Fernández-González, Cristina Outeiriño CidTransformer Encoder Sequence Modeling
Hierarchical Transformer Preconditioning for Interactive Physics Simulation
May 13, 2026Carl Osborne, Minghao Guo, Crystal Owens +1Spectral Preconditioning Physics Simulation
Chem-GMNet: A Sphere-Native Geometric Transformer for Molecular Property Prediction
May 13, 2026Deepak Warrier, Raja Sekhar PappalaMolecular Structure Elucidation Molecular Property Prediction
ECG-NAT: A Self-supervised Neighborhood Attention Transformer for Multi-lead Electrocardiogram Classification
May 13, 2026Mahsa Gazeran, Sayvan Soleymanbaigi, Fatemeh Daneshfar +2Electrocardiogram Classification Transformer Architectures
N-vium: Mixture-of-Exits Transformer for Accelerated Exact Generation
May 13, 2026Aleksander Lorenc, Frédéric Berdoz, Joël Mathys +1Autoregressive Transformers Transformer Architectures
Exact Linear Attention
May 13, 2026Weinuo OuQuadratic Attention Transformer Attention
Transformers Linearly Represent Highly Structured World Models
May 13, 2026Roman Kniazev, Nathanaël FijalkowTransformer Layers Transformer Architectures
Pyramid Forcing: Head-Aware Pyramid KV Cache Policy for High-Quality Long Video Generation
May 13, 2026Jiayu Chen, Junbei Tang, Wenbiao Zhao +6Autoregressive Video Generation Dynamic Sparse Attention
Precision Tracked Transformer via Kalman Filtering, Kriging and Process Noise
May 12, 2026Bo Long, Deepak Agarwal, Jelena Markovic-Voronov +2Transformer Architectures Particle Filters
A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attention
May 12, 2026Tomohiro Hayase, Ryo KarakidaSoftmax Attention Gumbel-Softmax Relaxation
The Routing and Filtering Structure of Attention
May 12, 2026Shafayeth Jamil, Rehan KapadiaSoftmax Attention Quadratic Attention
Elastic Attention Cores for Scalable Vision Transformers
May 12, 2026Alan Z. Song, Yinjie Chen, Mu Nan +8Self-Supervised Vision Transformers Vision Transformer
Search Your Block Floating Point Scales!
May 12, 2026Tanmaey Gupta, Hayden Prairie, Xiaoxia Wu +10Quantization Floating-Point
Geometric Factual Recall in Transformers
May 12, 2026Shauli Ravfogel, Gilad Yehudai, Joan Bruna +1Transformer Architectures Factual Recall
Recurrent Transformer-Based Near- and Far-Field THz Wideband Channel Estimation for UM-MIMO
May 12, 2026Dmitry Artemasov, Alexander Shmatok, Kirill Andreev +3Joint Channel Estimation Ultra-Wideband
Lower bounds for one-layer transformers that compute parity
May 12, 2026Daniel HsuTransformer Layers Rectified Linear Unit Networks
From Index to Equity: Pre-Training Transformers for Stock Return Prediction
May 12, 2026Marie Soehl Coolsaet, Roberto Gallardo, Zhen GaoTransformer Architectures Long-Term Memory
What-Where Transformer: A Slot-Centric Visual Backbone for Concurrent Representation and Localization
May 12, 2026Ryota Yoshihashi, Masahiro Kada, Satoshi Ikehata +2Vision Transformer Localization Methods
Composition of Memory Experts for Diffusion World Models
May 12, 2026Sebastian Stapf, Pablo Acuaviva Huertos, Aram Davtyan +1Discrete Diffusion Model Memory Retrieval
On Privacy-Preserving Image Transmission in Low-Altitude Networks: A Swin Transformer-Based Framework with Federated Learning
May 12, 2026Kexin Zhang, Lixin Li, Yuna Yan +5Unmanned Aerial Vehicles Semantic Communications
Breaking Global Self-Attention Bottlenecks in Transformer-based Spiking Neural Networks with Local Structure-Aware Self-Attention
May 12, 2026Lingdong Li, Hangming Zhang, Qiang YuSpiking Neural Networks Transformer Attention
Transformer Interpretability from Perspective of Attention and Gradient
May 12, 2026Yongjin Cui, Xiaohui Fan, Huajun ChenTransformer Attention Vision Transformer
Beyond Similarity: Temporal Operator Attention for Time Series Analysis
May 11, 2026Jevon Twitty, Vinh Pham, Nitiwith Rotchanarak +4Spatial Attention Transformer Attention
Latent Chain-of-Thought Improves Structured-Data Transformers
May 11, 2026Carson Dudley, Samet OymakTransformer Architectures Chain-of-Thought Reasoning
Variational Linear Attention: Stable Associative Memory for Long-Context Transformers
May 11, 2026Vishal Pandey, Gopal SinghQuadratic Attention Dense Associative Memory
HEPA: A Self-Supervised Horizon-Conditioned Event Predictive Architecture for Time Series
May 11, 2026Jonas Petersen, Gian-Alessandro Lombardi, Riccardo Maggioni +3Time Series Transformer Encoder
HiDream-O1-Image: A Natively Unified Image Generative Foundation Model with Pixel-level Unified Transformer
May 11, 2026Qi Cai, Jingwen Chen, Chengmin Gao +22Large-Scale Image Generation Generative Models
Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime
May 11, 2026Albert Alcalde, Leon Bungert, Konstantin Riedl +1Transformer Encoder Transformer Architectures
Uniform Scaling Limits in AdamW-Trained Transformers
May 11, 2026William Gibson, Christoph ReisingerTransformer Architectures Hidden States
On the global convergence of gradient descent for wide shallow models with bounded nonlinearities
May 11, 2026Romain Petit, Clarice Poon, Gabriel PeyréTwo-Layer Neural Networks Gradient Descent
RelFlexformer: Efficient Attention 3D-Transformers for Integrable Relative Positional Encodings
May 11, 2026Byeongchan Kim, Arijit Sehanobish, Avinava Dubey +2Positional Encoding Transformer Attention
A Single-Layer Model Can Do Language Modeling
May 11, 2026Zanmin WangGraph Language Models Transformer Layers
Self-Attention as a Covariance Readout: A Unified View of In-Context Learning and Repetition
May 11, 2026Haoren Xu, Guanhua FangSoftmax Attention In-Context Learning
Empty SPACE: Cross-Attention Sparsity for Concept Erasure in Diffusion Models
May 11, 2026Nicola Novello, Andrea M. TonelloConcept Erasure Text-To-Image Diffusion Models
DetRefiner: Model-Agnostic Detection Refinement with Feature Fusion Transformer
May 11, 2026Soichiro Okazaki, Tatsuya Sasaki, Hiroki OhashiOpen-Vocabulary Object Detection Transformer Architectures
DynGhost: Temporally-Modelled Transformer for Dynamic Ghost Imaging with Quantum Detectors
May 11, 2026Vittorio Palladino, Ahmet Enis CetinSingle-Photon Avalanche Diode Transformer Architectures
MicroViTv2: Beyond the FLOPS for Edge Energy-Friendly Vision Transformers
May 11, 2026Novendra Setyawan, Chi-Chia Sun, Mao-Hsiu Hsu +2Self-Supervised Vision Transformers Vision Transformer
Complex-Valued Phase-Coherent Transformer
May 11, 2026Leona HiokiTransformer Attention Unified Transformer

May 15, 2026 · cs.LGJ/K move · Enter open · S save

Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs

Vincent-Daniel Yun, Junhyuk Jo, Sai Praneeth Karimireddy, Sunwoo Lee

University of Southern California, USA · Inha University, Republic of Korea

Layer pruning removes entire Transformer decoder blocks from large language models, but introduces a mismatch between the hidden state received by the next surviving layer and the distribution it was trained to process, leading to significant performance degradation. We propose Ghosted Layers, a training-free recovery module that addresses this issue by solving a boundary activation alignment problem. Our method derives a closed-form optimal linear operator from a small calibration set to reconstruct the activation discrepancy introduced by the pruned layers. We show that this solution corresponds to the unconstrained optimum of the alignment objective, whereas existing methods are restricted to constrained solutions over limited operator subspaces. Experiments across multiple LLM backbones and pruning strategies demonstrate that our method consistently improves accuracy and perplexity over prior training-free baselines, while preserving the efficiency gains of layer pruning. Official code repository: https://github.com/daniel-eai/ghosted_layers_official_repository/.

Transformer Layers Model Activations Transformer Encoder

Open paper Ask about it PDF