Transformer FFNs

FFN: Feed-Forward Network

Ask about this topic

Overview

Transformer FFNs, also called MLPs, transform each token’s representation between attention operations, providing much of a model’s nonlinear processing and capacity. Work studies activation choices such as GELU, SwiGLU, and mixtures of activations, as well as changes to the block itself, including sparse experts, tree routing, low-rank branches, and linear alternatives. Researchers evaluate these designs using language-model loss and downstream benchmarks, and use pruning, probing, causal interventions, and geometric analyses to understand which neurons matter and how FFNs shape representations. A central open tension is how much dense nonlinear computation is needed: sparsity and simplification may reduce cost, but must preserve useful interactions that can vary by layer, token, and task.

Latest papers

CardsList

42 papers7 papers in the last four weeks, up 40% on the four weeks before. 0.0% of all new papers.

  1. Orthogonal Quadratic Complements for Vision Transformer Feed-Forward Networks

    Apr 8, 2026Wang ZixianTransformer FFNsVision Transformer

    China Mobile Communications Group Shandong Co., Ltd.

  2. Variational Neurons in Transformers for Language Modeling

    Mar 30, 2026Yves RuffenachLLM Uncertainty EstimationTransformer

    Conservatoire National des Arts et Métiers Strasbourg, France

  3. Dynamic sparsity in tree-structured feed-forward layers at scale

    Mar 18, 2026Reza Sedghi, Robin Schiewer, Anand Subramoney +1Transformer FFNsStructured Sparsity

    CITEC, Bielefeld University, Germany · Department of Computer Science, Royal Holloway, University of London, UK

  4. The Discrete Charm of the MLP: Binary Routing of Continuous Signals in Transformer Feed-Forward Layers

    Mar 11, 2026Peter BaloghTransformer FFNsTransformer Interpretability

  5. PolyGLU: State-Conditional Activation Routing in Transformer Feed-Forward Networks

    Mar 7, 2026Daniel Nobrega MedeirosTransformer FFNsNeural Network Activation Functions

    Independent researcher.

  6. NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networks

    Mar 6, 2026Nandan Kumar Jha, Brandon ReagenTransformer FFNsNeural Network Training Dynamics

    New York University

  7. Layer by layer, module by module: Choose both for optimal OOD probing of ViT

    Mar 5, 2026Ambroise Odonnat, Vasilii Feofanov, Laetitia Chapel +2Vision TransformerOOD Generalization

    Noah’s Ark Lab, Inria · L’Institut Agro, IRISA · Univ. Rennes 2, IRISA +1

  8. Half the Nonlinearity Is Wasted: Measuring and Reallocating the Transformer's MLP Budget

    Mar 3, 2026Peter BaloghTransformer FFNsConditional Computation

    Peter Balogh

  9. GLUScope: A Tool for Analyzing GLU Neurons in Transformer Language Models

    Feb 27, 2026Sebastian Gerstner, Leonor Veloso, Hinrich SchützeNeuron-Level InterpretabilityTransformer Interpretability

    LMU Munich and Munich Center for Machine Learning (MCML), Munich, Germany

  10. Revisiting the Shape Convention of Transformer Language Models

    Feb 6, 2026Feng-Ting Liao, Guan-Ting Yi, Tzu-Quan Lin +2Efficient Transformer InferenceTransformer FFNs

    MediaTek Research · National Taiwan University

  11. Equivalence of Context and Parameter Updates in Modern Transformer Blocks

    Nov 22, 2025Adrian Goldwaser, Michael Munn, Javier Gonzalvo +1Transformer InterpretabilityTransformer FFNs

    University of Cambridge, UK · Google Research

  12. SG-Blend: Learning an Interpolation Between Improved Swish and GELU for Robust Neural Representations

    May 29, 2025Gaurav Sarkar, Syed Affan Daimi, Jay Gala +1Transformer FFNsTransformer

    IIIT · Microsoft · Intel