Transformer Layers

Follow this topic Ask about this topic

Recent momentum

+175%

11 papers in the last 28 days · 0.2% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-21

6 new papers

A weekly snapshot of new work published in Transformer Layers.

Period ending 2026-09-07

2 new papers

A weekly snapshot of new work published in Transformer Layers.

80 papers

Latest in Transformer Layers

Browse all topics →

GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression
Sep 22, 2026Baher Mohammad, Ammar Ali, Stamatios LefkimmiatisTransformer Layers Model Weights
Transformer Heads Looking for Order
Sep 22, 2026Jasper van Doornmalen, Alexander Kozachinskiy, Corinna Mathwieser +4Transformer Layers
Beyond Depth Truncation: Controlled Evaluation of Depth Utilization in Recursive Language Models
Sep 17, 2026Ha Van Dau, Thanh Tung Khuat, Nguyen Thanh DungSearch Depth Truncation
How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents
Sep 16, 2026Zixi Chen, Akshay Vegesna, Samip Dahal +1Scaling Laws Transformer Layers
Weakening Neurons: An Input-Output Functionality in Transformers with Outsize Influence
Sep 16, 2026Sebastian Gerstner, Hilal AlQuabeh, Kentaro Inui +1Transformer Layers Transformer Architectures
MoRE: Mixture of Reused Experts
Sep 16, 2026Eric S. Qiu, Utku Umur Acikalin, Justin Lovelace +4Mixture-of-Experts Models Experts
Persistent Recurrent Memory Between Transformer Layers - Improves Language Model Generalization
Sep 15, 2026Eduardo Novaes HeringTransformer Layers Transformer Encoder
Pre-PEFT Probing: Weight Statistics and Perturbation Robustness for Layer Selection in VLM Vision Encoders
Sep 14, 2026Qingtao Xia, Jiahua Bao, Siyao Cheng +1Parameter-Efficient Fine-Tuning Finetuning
LLM Layers Immediately Correct Each Other
Sep 7, 2026Arjun Patrawala, Jiahai Feng, Erik Jones +1Transformer Layers Model Activations
One-Layer Transformer Provably Learns Multiclass One-Nearest Neighbor in Context
Sep 1, 2026Skanda Athreya, Yutong WangTransformer Layers Multiclass Classification
Measuring Optimal Transport in Transformer Depth
Sep 1, 2026Alexandre QuemyTransformer Layers Optimal Transport
Toppling the Hierarchy in Byte-level Language Modeling
Aug 31, 2026Lukas Edman, Alexander FraserByte-Pair Encoding Transformer Layers
Late Transformer Layers Recode Syntax Canonically: Evidence from Greek Scrambling and Cross-Layer Generalisation
Aug 31, 2026Christos Nikolaos Zacharopoulos, Revekka Kyriakoglou, Chara Tsoukala +1Transformer Layers Syntactic Structure
TACTICL: Task-Aware Compression of Tabular ICL Models
Aug 11, 2026Mykhailo Koshil, Matthias Feurer, Katharina EggenspergerTabular Foundation Models Context Compression
Deployable Per-Instance Multi-Layer Activation Steering for Large Language Models
Aug 9, 2026Muhammad Faishal Adly Nelwan, Alfan Farizki WicaksonoActivation Steering Model Activations
MACRO: Markov Chain Routing of Transformer Layers
Aug 6, 2026Paweł Batorski, Abtin Pourhadi, Akylgali Aitaza +2Large Language Model Routing Transformer Layers
Role-Decoupled Attention Residuals: Separating Matching and Content Retrieval Across Depth
Aug 2, 2026Kehan WangTransformer Residual Streams Residual Connections
The Intruder Threshold: A Spectral Law for LoRA Fine-Tuning
Jul 26, 2026Peng XieLow-Rank Adaptation Catastrophic Forgetting
Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D
Jul 17, 2026Haodong Wen, Yiran Zhang, Yingfa Chen +1Positional Encoding Transformer Layers
LayerNorm as Implicit Gain Control in Looped Transformers
Jul 12, 2026Matthias M. M. BuehlmaierBatch Normalization Transformer Layers
Layer-Parallel Inference Reduces Encrypted Nonlinear Depth in Transformers
Jul 6, 2026Ligong Han, Kai Xu, Hao Wang +3Homomorphic Encryption Transformer Layers
Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training
Jul 1, 2026Zijian Zhang, Rizhen Hu, Athanasios Glentis +4Transformer Layers Reinforcement Learning Post-Training
Gradient Smoothing: Coupling Layer-wise Updates for Improved Optimization
Jun 29, 2026Haoming Meng, Anton Sugolov, Vardan PapyanGradient Flow Transformer Layers
Probing-Guided Layer Selection from Self-Supervised Speech Models for Generalizable Audio Deepfake Detection
Jun 29, 2026Marjan Beheshti, Majid Rostami, Bo ChenAudio Deepfake Detection Self-Supervised Speech Models
IG-Lens: Exact Additive Probability Attribution Across Transformer Layers via Telescoping Integrated Gradients
Jun 29, 2026Duc Anh NguyenTransformer Layers Transformer Encoder
The Context-Ready Transformer
Jun 25, 2026Mahesh GodavartiTransformer Layers Transformer Architectures
LIG: Layer-wise Integrated Gradients for Within-Layer Flow Analysis in Transformers
Jun 19, 2026Eight Suzuki, Hideitsu Hino, Noboru MurataTransformer Layers Transformer Architectures
FiLM-Coordinated Dual-Branch Transformer for Global-Local Dependency Modeling in Language Modeling
Jun 19, 2026Zhiqiang Zhou, Xu Ling, Junliang DaiTransformer Layers Softmax Attention
Learning through Internalization
Jun 18, 2026Nikolaos Tsilivis, Nirmit Joshi, Marko Medvedev +2Internalization Transformer Layers
An expressivity analysis of hierarchical modelling in deep transformers via bounded-depth grammars
Jun 16, 2026Vinoth Nandakumar, Qiang Qu, Pramod Thebe +2Transformer Layers Search Depth
Beyond Layer Importance in Layer-wise Sparsity: An Inter-Layer Perturbation-Absorption Perspective
Jun 13, 2026Tao Jing, Ningxin Wu, Chen Kang +3Sparsity Large Language Model Compression
Exploring a Layer-Wise Design Space for KV Cache Eviction
Jun 13, 2026Chao Fei, Kaihua Liang, Hanzhi Hu +4Key-Value Cache Compression Cache
A theoretical model for task routing in mixture-of-expert transformers
Jun 12, 2026Vinoth Nandakumar, Yongli Xiang, Yunzhi Yao +2Mixture-of-Experts Models Transformer Layers
Sparsely gated tiny linear experts
Jun 5, 2026Simon SchugFeed-Forward Network Transformer Layers
The Geometry of Last-Layer Model Stealing
Jun 5, 2026Snigdha Chandan KhilarTransformer Layers Transformer Architectures
Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs
Jun 3, 2026Wanhao Yu, Ziyan Wang, Zheng Wang +7Continual Fine-Tuning Transformer Layers
Parameter-Efficient Fine-Tuning with Learnable Rank
Jun 3, 2026Arpit Garg, Simon Lucey, Hemanth SaratchandranParameter-Efficient Fine-Tuning Low-Rank Adaptation
LayerRoute: Input-Conditioned Adaptive Layer Skipping via LoRA Fine-Tuning for Agentic Language Models
Jun 1, 2026Prateek Kumar SikdarLarge Language Model Backbones Transformer Layers
Refit the Probe: Single-Direction Ablation Is Not a Necessity Test
May 30, 2026Yuhang JiangTransformer Layers Sequence Modeling
Mitigating Hallucinations in Large Language Models Via Decoder Layer Skipping
May 30, 2026Hanze Li, Jinhao You, Yichen Guo +3Large Language Model Hallucination Hallucination Mitigation
Looped Transformers with Layer Normalization Provably Learn the Power Method
May 30, 2026Lyumin Wu, Chenyang Zhang, Yuan CaoTransformer Layers Transformer Architectures
Cross-Layer Subspace Coupling for LLM Compression: A Unifying Framework and Its Empirical Limits
May 29, 2026Snigdha Chandan KhilarLarge Language Model Compression Cross-Layer Interactions
A Dual-Path Architecture for Scaling Compute and Capacity in LLMs
May 28, 2026Markus Frey, Behzad Shomali, Joachim Koehler +1Transformer Layers Large Models
Transformers Provably Learn to Internalize Chain-of-Thought
May 27, 2026Yixiao Huang, Hanlin Zhu, Zixuan Wang +4Transformer Layers Transformer Architectures
ReSAE: Residualized Sparse Autoencoders for Multi-Layer Transformer Interventions
May 27, 2026Prathyush Poduval, Calvin Yeung, Neel Desai +1Improving Sparse Autoencoders Transformer Residual Streams
More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations
May 26, 2026Mingze Wang, Jinbo Wang, Yikuan Xia +2Model Activations Feed-Forward Network
Analogies between Transformer Layers and Power Method
May 25, 2026Chenglong Li, Claudio AltafiniTransformer Layers Transformer Architectures
Towards the Connection between Activation Sparsity and Flat Minima
May 25, 2026Ze Peng, Jian Zhang, Lei Qi +2Sparse Expert Activation Modern Deep Networks
Polymorphism Is Rotation: Operational Mechanistic Interpretability from a Two-Layer Transformer to Pythia-70m
May 23, 2026Jordan F. McCannTransformer Architectures Transformer Layers
Preisach Attention: A Hysteretic Model of Sequential Memory
May 22, 2026Piotr FrydrychTransformer Layers Softmax Attention
Uncovering the Latent Potential of Deep Intermediate Representations
May 21, 2026Arnesh Batra, Arush Gumber, Aniket Khandelwal +2Geometric Deep Learning Transformer Layers
One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs
May 21, 2026Di He, Songjun Tu, Keyu Wang +2Learning Rates Transformer Layers
The Expressive Power of Low Precision Softmax Transformers with (Summarized) Chain-of-Thought
May 18, 2026Moritz Brösamle, Stephan EcksteinTransformer Layers Gumbel-Softmax Relaxation
InfoFlow: A Framework for Multi-Layer Transformer Analysis
May 18, 2026Penghao Yu, Haotian Jiang, Zeyu Bao +1Transformer Layers Softmax Attention
SNLP: Layer-Parallel Inference via Structured Newton Corrections
May 18, 2026Ligong Han, Kai Xu, Hao Wang +1Transformer Layers Transformer Residual Streams
Parallel Recursive LSTM
May 16, 2026Tristan Gaudreault, Yongyi MaoParallel Recurrent Neural Networks
No Free Swap: Protocol-Dependent Layer Redundancy in Transformers
May 15, 2026Gabriel GarciaTransformer Layers Transformer Architectures
Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs
May 15, 2026Vincent-Daniel Yun, Junhyuk Jo, Sai Praneeth Karimireddy +1Transformer Layers Model Activations
Transformers Linearly Represent Highly Structured World Models
May 13, 2026Roman Kniazev, Nathanaël FijalkowTransformer Layers Transformer Architectures
Lower bounds for one-layer transformers that compute parity
May 12, 2026Daniel HsuTransformer Layers Rectified Linear Unit Networks

Sep 22, 2026 · cs.LGJ/K move · Enter open · S save

GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression

Baher Mohammad, Ammar Ali, Stamatios Lefkimmiatis

Transformer architectures exhibit cross-layer redundancies, yet post-training compression pipelines typically optimize layers in isolation or rely on heuristic grouping strategies that disregard layer-specific activation geometries. We introduce a principled, training-free framework that sequentially optimizes cross-layer weight pairings and shared-dictionary factorizations. Rather than forcing weights of adjacent layers to share a basis or heuristically merging activation statistics, our approach identifies structurally compatible projections and learns a shared representation that better preserves each layer's distinct calibration geometry. Coupled with structured sparsity, this yields highly efficient weight decompositions without sacrificing functional fidelity. Across diverse architectures, scales, and modalities, our method achieves state-of-the-art results, consistently outperforming independent structured weight decompositions and alternative pairwise weight factorizations, which operate under heuristic grouping strategies. By replacing heuristic engineering strategies with a convergent, optimization-driven pipeline, we establish a theoretically grounded foundation for scalable, transformer compression across different modalities.

Transformer Layers Model Weights

Open paper Ask about it PDF