Multi-Layer Transformers

Ask about this topic

Recent momentum

-25%

3 papers in the last 28 days · 0.1% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Also in Transformer Encoder

Neighbouring directions

Transformer VariantTransformer EncoderUnified Transformer

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-07

2 new papers

A weekly snapshot of new work published in Multi-Layer Transformers.

56 papers

Latest in Multi-Layer Transformers

Open your feed →
CardsList

  1. Temporal Recurrence Favors Fewer Layers

    Sep 14, 2026Ivan Anokhin, Johan Obando-Ceron, Irina Rish +1Recurrent ModelMultilayer Perceptrons

  2. One-Layer Transformer Provably Learns Multiclass One-Nearest Neighbor in Context

    Sep 1, 2026Skanda Athreya, Yutong WangMulti-Layer TransformersMulticlass Classification

  3. Toppling the Hierarchy in Byte-level Language Modeling

    Aug 31, 2026Lukas Edman, Alexander FraserProtein Language ModelMulti-Layer Transformers

  4. Late Transformer Layers Recode Syntax Canonically: Evidence from Greek Scrambling and Cross-Layer Generalisation

    Aug 31, 2026Christos Nikolaos Zacharopoulos, Revekka Kyriakoglou, Chara Tsoukala +1Multi-Layer TransformersSyntactic Structure

  5. Deployable Per-Instance Multi-Layer Activation Steering for Large Language Models

    Aug 9, 2026Muhammad Faishal Adly Nelwan, Alfan Farizki WicaksonoLinear Activation SteeringModel Activations

  6. Why Knowing Both Hops Is Not Enough: Understanding Two-Hop Generalization in Language Models

    Aug 7, 2026Zili Zhang, Yilin Wang, Heng Wang +2Multi-Layer TransformersLarge Multimodal Models

  7. LoopMTP: A looped transformer guided by latent multi-token prediction

    Aug 4, 2026Behzad Shomali, Markus Frey, David Berghaus +2Multi-Layer TransformersNext-Token Prediction

  8. Indexing: the Beginning and the End

    Jul 24, 2026Alexander Kozachinskiy, Vicente Opazo, Felipe UrrutiaNeural ArchitecturesMulti-Layer Transformers

  9. Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D

    Jul 17, 2026Haodong Wen, Yiran Zhang, Yingfa Chen +1Protein Language ModelPositional Encoding

  10. T^2MLR: Transformer with Temporal Middle-Layer Recurrence

    Jul 16, 2026Ziyang Cai, Xingyu Zhu, Yihe Dong +2Multi-Layer TransformersLatent Reasoning

  11. Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training

    Jul 1, 2026Zijian Zhang, Rizhen Hu, Athanasios Glentis +4Multi-Layer TransformersReinforcement Learning Post-Training

  12. Soft Mixture-of-Recursions: Going Deeper with Recursive Vision Transformers

    Jul 1, 2026Sang In Lee, Jihun ParkSelf-Supervised Vision TransformersVision Transformer

  13. DiscoLoop: Looping Discrete Embeddings and Continuous Hidden States for Multi-hop Reasoning

    Jul 1, 2026Hengyu Fu, Tianyu Guo, Zixuan Wang +5Multi-Layer TransformersChain-Of-Thought

  14. Low-dimensional topology of deep neural networks

    Jun 30, 2026Junyu Ren, Lek-Heng LimNeural ArchitecturesDeep Neural Network

  15. Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers

    Jun 30, 2026Ying Fan, Anej Svete, Kangwook LeeLatent ReasoningMulti-Layer Transformers

  16. IG-Lens: Exact Additive Probability Attribution Across Transformer Layers via Telescoping Integrated Gradients

    Jun 29, 2026Duc Anh NguyenMulti-Layer TransformersTransformer Encoder

  17. LIG: Layer-wise Integrated Gradients for Within-Layer Flow Analysis in Transformers

    Jun 19, 2026Eight Suzuki, Hideitsu Hino, Noboru MurataMulti-Layer TransformersFeature Attribution

  18. FiLM-Coordinated Dual-Branch Transformer for Global-Local Dependency Modeling in Language Modeling

    Jun 19, 2026Zhiqiang Zhou, Xu Ling, Junliang DaiMulti-Layer TransformersProtein Language Model

  19. Learning through Internalization

    Jun 18, 2026Nikolaos Tsilivis, Nirmit Joshi, Marko Medvedev +2InternalizationMulti-Layer Transformers

  20. An expressivity analysis of hierarchical modelling in deep transformers via bounded-depth grammars

    Jun 16, 2026Vinoth Nandakumar, Qiang Qu, Pramod Thebe +2Multi-Layer TransformersSearch Depth

  21. Beyond Layer Importance in Layer-wise Sparsity: An Inter-Layer Perturbation-Absorption Perspective

    Jun 13, 2026Tao Jing, Ningxin Wu, Chen Kang +3Magnitude-Based PruningSparsity

  22. Tight Sample Complexity of Transformers

    Jun 8, 2026Chenxiao Yang, Nathan Srebro, Zhiyuan LiMulti-Layer TransformersSample Complexity

  23. Hasse Diagrams for Attention: A Partial Order Framework for Designing Transformer Masks

    Jun 8, 2026Chentao Li, Han GuoCausal AttentionMulti-Layer Transformers

  24. The Geometry of Last-Layer Model Stealing

    Jun 5, 2026Snigdha Chandan KhilarMulti-Layer TransformersData Leakage

  25. Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs

    Jun 3, 2026Wanhao Yu, Ziyan Wang, Zheng Wang +7FinetuningZeroth-Order Optimization

  26. Structure and Scale in Simplicial Sequence Modelling

    May 31, 2026Matthew Farrugia-RobertsMulti-Layer TransformersSequence Modeling

  27. Task Structure Reverses Layerwise State Encoding in Sequence Models

    May 30, 2026Yuhang JiangSequence ModelingMulti-Layer Transformers

  28. MLLM-Microscope: Unlocking Hidden Structure Within Multimodal Large Language Models

    May 30, 2026Ravil Mussabayev, Rustam MussabayevLarge Multimodal ModelsMulti-Layer Transformers

  29. Mitigating Hallucinations in Large Language Models Via Decoder Layer Skipping

    May 30, 2026Hanze Li, Jinhao You, Yichen Guo +3Large Language Model HallucinationMitigating Hallucinations

  30. Cross-Layer Subspace Coupling for LLM Compression: A Unifying Framework and Its Empirical Limits

    May 29, 2026Snigdha Chandan KhilarLarge Language Model CompressionCross-Layer Interactions

  31. A Dual-Path Architecture for Scaling Compute and Capacity in LLMs

    May 28, 2026Markus Frey, Behzad Shomali, Joachim Koehler +1Large Language Model ScaleMulti-Layer Transformers

  32. Transformers Provably Learn to Internalize Chain-of-Thought

    May 27, 2026Yixiao Huang, Hanlin Zhu, Zixuan Wang +4Multi-Layer TransformersChain-Of-Thought

  33. ReSAE: Residualized Sparse Autoencoders for Multi-Layer Transformer Interventions

    May 27, 2026Prathyush Poduval, Calvin Yeung, Neel Desai +1Improving Sparse AutoencodersSparse Autoencoders

  34. Preisach Attention: A Hysteretic Model of Sequential Memory

    May 22, 2026Piotr FrydrychMulti-Layer TransformersModel Attention

  35. Uncovering the Latent Potential of Deep Intermediate Representations

    May 21, 2026Arnesh Batra, Arush Gumber, Aniket Khandelwal +2Multi-Layer TransformersCross-Modal

  36. Towards Explainability of SLMs by investigating Token Level Activation

    May 21, 2026Sayantani Ghosh, Rajashik Datta, Amit Kumar Das +1Model ActivationsToken-Level Uncertainty

  37. One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs

    May 21, 2026Di He, Songjun Tu, Keyu Wang +2Learning RatesLarge Language Model Training

  38. The Expressive Power of Low Precision Softmax Transformers with (Summarized) Chain-of-Thought

    May 18, 2026Moritz Brösamle, Stephan EcksteinMulti-Layer TransformersGumbel-Softmax Relaxation

  39. InfoFlow: A Framework for Multi-Layer Transformer Analysis

    May 18, 2026Penghao Yu, Haotian Jiang, Zeyu Bao +1Multi-Layer TransformersSoftmax Attention

  40. Parallel Recursive LSTM

    May 16, 2026Tristan Gaudreault, Yongyi MaoRecurrent ModelMulti-Layer Transformers

  41. Transformers Linearly Represent Highly Structured World Models

    May 13, 2026Roman Kniazev, Nathanaël FijalkowMulti-Layer TransformersCombinations

  42. Lower bounds for one-layer transformers that compute parity

    May 12, 2026Daniel HsuMulti-Layer TransformersSoftmax Attention

  43. A Single-Layer Model Can Do Language Modeling

    May 11, 2026Zanmin WangProtein Language ModelMulti-Layer Transformers

  44. Sparse Layers are Critical to Scaling Looped Language Models

    May 9, 2026Ryan Lee, Jacob Biloki, Edward J. Hu +1Multi-Layer TransformersLarge Language Model Scale

  45. Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models

    May 8, 2026Victor Conchello Vendrell, Arnau Padres Masdemont, Niccolò Grillo +3Large Language Model MemoryIterative Reasoning

  46. Spectrum-Adaptive Generalization Bounds for Trained Deep Transformers

    May 8, 2026Mana Sakai, Masaaki ImaizumiMulti-Layer TransformersGeneralization Bounds

  47. Transformers Efficiently Perform In-Context Logistic Regression via Normalized Gradient Descent

    May 7, 2026Chenyang Zhang, Yuan CaoMulti-Layer TransformersIn-Context Learning

  48. Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models

    May 7, 2026Amir Rezaei Balef, Mykhailo Koshil, Katharina EggenspergerTabular Foundation ModelsMulti-Layer Transformers

  49. LayerTracer: A Joint Task-Particle and Vulnerable-Layer Analysis framework for Arbitrary Large Language Model Architectures

    Apr 22, 2026Yuhang Wu, Qinyuan Liu, Qiuyang Zhao +1Multi-Layer TransformersTraces

  50. Decomposing the Depth Profile of Fine-Tuning

    Apr 19, 2026Jayadev BillaFinetuningNeural Architectures

  51. Intermediate Layers Encode Optimal Biological Representations in Single-Cell Foundation Models

    Apr 16, 2026Vincenzo Yuto Civale, Roberto Semeraro, Andrew David Bagdanov +1Model ActivationsRepresentation Learning

  52. On the Expressive Power and Limitations of Multi-Layer SSMs

    Apr 16, 2026Nikola Zubić, Qian Li, Yuyi Wang +1State Space ModelsMultilayer Perceptrons

  53. Deductive Logic in Language Models: Horizontal vs Vertical Reasoning

    Oct 10, 2025Davide Maltoni, Matteo FerraraMulti-Layer TransformersProtein Language Model

  54. You Do Not Fully Utilize Transformer's Representation Capacity

    Feb 13, 2025Gleb Gerasimov, Yaroslav Aksenov, Nikita Balagansky +2Multi-Layer TransformersRepresentation Quality

  55. Tuning Language Models by Mixture-of-Depths Ensemble

    Oct 16, 2024Haoyan Luo, Lucia SpeciaLarge Language Model Fine-TuningMulti-Layer Transformers

  56. Tracing Computation Density in LLMs

    Date pendingCorentin Kervadec, Iuliia Lysova, Iuri Macocco +2Multi-Layer TransformersTraces