Transformer FFNs

FFN: Feed-Forward Network

Overview

Transformer FFNs, also called MLPs, transform each token’s representation between attention operations, providing much of a model’s nonlinear processing and capacity. Work studies activation choices such as GELU, SwiGLU, and mixtures of activations, as well as changes to the block itself, including sparse experts, tree routing, low-rank branches, and linear alternatives. Researchers evaluate these designs using language-model loss and downstream benchmarks, and use pruning, probing, causal interventions, and geometric analyses to understand which neurons matter and how FFNs shape representations. A central open tension is how much dense nonlinear computation is needed: sparsity and simplification may reduce cost, but must preserve useful interactions that can vary by layer, token, and task.

Latest papers

CardsList

45 papers3 papers in the last four weeks, down 67% on the four weeks before. 0.0% of all new papers.

  1. CP-MLPs: A Tensor-Rank Theory of Tied and Untied MLP Blocks

    Nov 10, 2026Md Rifat Arefin, Farzaneh Heidari, Irina Rish +1Multilayer PerceptronsNeural Network Expressivity

  2. Closed Gates Leak: Decomposing Adjacent-Layer Cancellation in Gated Language Models

    Oct 28, 2026Harvey Dam, Ganesh GopalakrishnanTransformer FFNsMultilayer Perceptrons

  3. Reader--Writer Coupling in Transformer MLPs

    Oct 21, 2026Giyeong Oh, Sunghee Ahn, Youngjae YuTransformer FFNsTransformer Interpretability

  4. Learning Source Structure: Bayesian Prediction with Transformers

    Oct 13, 2026Meir Feder, Ruediger UrbankeTransformerSequence Modeling

  5. On the Necessity of Attention-FFN Split in Vision Transformers

    Oct 7, 2026Junhyeok Kim, Jinyeong Kim, Jae Wan Park +1Vision TransformerAttention Mechanisms

  6. The Role of Feed-Forward Layers in Transformer Dynamics

    Sep 28, 2026Thomas Jacob Maranzatto, Semih Akkoc, Sennur UlukusTransformerTransformer FFNs

  7. Which the Eye Fears: Writing with Read-Blindness Explains Massive Activations in Transformers

    Sep 28, 2026Swagatam Mukhopadhyay, Vishal Vivek Saley, Vraj Parikh +1Transformer InterpretabilityTransformer FFNs

  8. Change the Product, Keep the Parameters: Associative Algebra Layers for Transformers

    Sep 26, 2026Ilya Koziev, Ivan OseledetsTransformerLLM Inference Acceleration

  9. Model Casting and Low-Parameter Gating: Towards More Sparsely Activated FFNs

    Sep 25, 2026Maria Lomeli, Antoine Groudiev, Matthijs Douze +7Efficient Neural Network InferenceTransformer FFNs

  10. Weakening Neurons: An Input-Output Functionality in Transformers with Outsize Influence

    Sep 16, 2026Sebastian Gerstner, Hilal AlQuabeh, Kentaro Inui +1Transformer InterpretabilityTransformer FFNs

  11. Structure is not mechanism: high-gain gated-FFN rows across text and genomic foundation models

    Sep 13, 2026Alexandros Tzanakakis, Aris Karatzikos, Ilias Georgakopoulos-SoaresGenomic Foundation ModelsTransformer FFNs

  12. Curvature Cryptanalysis of Smooth Transformer Feed-Forward Networks

    Aug 28, 2026Munawar Hasan, Apostol VassilevTransformer FFNs

  13. TANGO: Treating Tokens as Operators

    Aug 22, 2026Joshua NunleyTransformer FFNsTransformer

  14. Feed-Forward Steering in Transformer Residual Dynamics

    Aug 3, 2026Timur Mudarisov, Mikhail Burtsev, Radu StateTransformer InterpretabilityTransformer FFNs

  15. Geometry-Guided Layerwise FFN Width Allocation in Transformers

    Aug 3, 2026Timur Mudarisov, Mikhail Burtsev, Radu StateTransformer FFNsTransformer

  16. A Controlled Study of Attention-Only Transformers

    Jul 20, 2026Henry Ndubuaku, Karen Mosoyan, Jakub Mroz +5Transformer FFNsTransformer

  17. Training, Reading, and Editing Legible Transformers

    Jul 9, 2026Mark OskinTransformer InterpretabilityTransformer FFNs

  18. Sparsely gated tiny linear experts

    Jun 5, 2026Simon SchugTransformer FFNsMixture-of-Experts Language Models

  19. More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations

    May 26, 2026Mingze Wang, Jinbo Wang, Yikuan Xia +2Transformer FFNsTransformer Expressivity

  20. Large Vision-Language Models Get Lost in Attention

    May 7, 2026Gongli Xi, Ye Tian, Mengyu Yang +5Transformer FFNsVision-Language Models