Transformer Layers

Follow this topic Ask about this topic

Recent momentum

+175%

11 papers in the last 28 days · 0.2% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-21

6 new papers

A weekly snapshot of new work published in Transformer Layers.

Period ending 2026-09-07

2 new papers

A weekly snapshot of new work published in Transformer Layers.

80 papers

Latest in Transformer Layers

Browse all topics →

A Single-Layer Model Can Do Language Modeling
May 11, 2026Zanmin WangGraph Language Models Transformer Layers
Continuous Latent Contexts Enable Efficient Online Learning in Transformers
May 11, 2026Emile Anand, Abdullah Ateyeh, Xinyuan Cao +1In-Context Learning Transformer Architectures
Sparse Layers are Critical to Scaling Looped Language Models
May 9, 2026Ryan Lee, Jacob Biloki, Edward J. Hu +1Transformer Layers Large Models
Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models
May 8, 2026Victor Conchello Vendrell, Arnau Padres Masdemont, Niccolò Grillo +3Recurrent Model Transformer Layers
Revisiting Transformer Layer Parameterization Through Causal Energy Minimization
May 8, 2026Jin Xu, Camille Couturier, Victor Rühle +2Transformer Layers Transformer Architectures
UniPool: A Globally Shared Expert Pool for Mixture-of-Experts
May 7, 2026Minbin Huang, Han Shi, Chuanyang Zheng +5Mixture-of-Experts Models Experts
Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models
May 7, 2026Amir Rezaei Balef, Mykhailo Koshil, Katharina EggenspergerTabular Foundation Models Transformer Layers
Federation of Experts: Communication Efficient Distributed Inference for Large Language Models
May 7, 2026Muhammad Shahir Abdurrahman, Chun Deng, Azalia Mirhoseini +1Mixture-Of-Expert Inference LLM Inference Efficiency
Trust, but Verify: Peeling Low-Bit Transformer Networks for Training Monitoring
May 4, 2026Arian Eamaz, Farhang Yeganegi, Mojtaba SoltanalianTransformer Layers Transformer Architectures
Hardware-Efficient Softmax and Layer Normalization with Guaranteed Normalization for Edge Devices
Apr 26, 2026Dawon Choi, Hana Kim, Ji-Hoon KimBatch Normalization Gumbel-Softmax Relaxation
Supernodes and Halos: Loss-Critical Hubs in LLM Feed-Forward Layers
Apr 26, 2026Audrey Cherilyn, Houman SafaaiFeed-Forward Network Transformer Layers
LayerTracer: A Joint Task-Particle and Vulnerable-Layer Analysis framework for Arbitrary Large Language Model Architectures
Apr 22, 2026Yuhang Wu, Qinyuan Liu, Qiuyang Zhao +1Transformer Layers Circuit Tracing
Decomposing the Depth Profile of Fine-Tuning
Apr 19, 2026Jayadev BillaGradient Flow Transformer Layers
Equivalence of Context and Parameter Updates in Modern Transformer Blocks
Nov 22, 2025Adrian Goldwaser, Michael Munn, Javier Gonzalvo +1Transformer Layers Model Weights
Test time training enhances in-context learning of nonlinear functions
Sep 30, 2025Kento Kuwataka, Taiji SuzukiIn-Context Learning Transformer Layers
You Do Not Fully Utilize Transformer's Representation Capacity
Feb 13, 2025Gleb Gerasimov, Yaroslav Aksenov, Nikita Balagansky +2Transformer Layers Transformer Architectures
Tuning Language Models by Mixture-of-Depths Ensemble
Oct 16, 2024Haoyan Luo, Lucia SpeciaLarge Language Model Fine-Tuning Transformer Layers
On Generalisation Error Bounds for Transformers
Oct 15, 2024Lan V. TruongGeneralization Bounds Transformer Layers
SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
Date pendingShaowen Wang, Ge Zhang, Kairong Luo +6Transformer Layers Transformer Architectures
Tracing Computation Density in LLMs
Date pendingCorentin Kervadec, Iuliia Lysova, Iuri Macocco +2Large Models Transformer Layers

May 11, 2026 · cs.CLJ/K move · Enter open · S save

A Single-Layer Model Can Do Language Modeling

Zanmin Wang

Modern language models scale depth by stacking layers, each holding its own state - a per-layer KV cache in transformers, a per-layer matrix in Mamba, Gated DeltaNet (GDN), RWKV, and xLSTM. Biological systems lean heavily on recurrence rather than on stacking. We ask how far that shape can go on language modeling. We propose Grounded Prediction Networks (GPN): one state vector revisited at every step through a single recurrent block - one FFN, one shared matrix memory. At 130M parameters, a 1-layer GPN+M reaches FineWeb-Edu perplexity 18.06, within 13% of a 12-layer Transformer++ (16.05) and 18% of a 10-layer GDN (15.34); a 2-layer variant closes the gap to 6%/11%. We do not match the deep baselines. Because the working context is a single vector, we can directly inspect its geometry: a persistent default-token direction, a content-bearing horizon of tens of tokens, and memory heads that split spontaneously into fast and slow retention pools.

Graph Language Models Transformer Layers Key-Value Cache

Open paper Ask about it PDF