Transformer FFNs

FFN: Feed-Forward Network

Transformer FFNs, also called MLPs, transform each token’s representation between attention operations, providing much of a model’s nonlinear processing and capacity. Work studies activation choices such as GELU, SwiGLU, and mixtures of activations, as well as changes to the block itself, including sparse experts, tree routing, low-rank branches, and linear alternatives. Researchers evaluate these designs using language-model loss and downstream benchmarks, and use pruning, probing, causal interventions, and geometric analyses to understand which neurons matter and how FFNs shape representations. A central open tension is how much dense nonlinear computation is needed: sparsity and simplification may reduce cost, but must preserve useful interactions that can vary by layer, token, and task.

Latest papers

42 papers7 papers in the last four weeks, up 40% on the four weeks before. 0.0% of all new papers.

Reader--Writer Coupling in Transformer MLPs
Oct 21, 2026Giyeong Oh, Sunghee Ahn, Youngjae YuTransformer FFNs Transformer Interpretability
Seoul National University
Comparing Sparsified MLPs and Transcoders: Many Transcoder Features Are Neurons in Disguise
Oct 17, 2026Stepan Shabalin, Mateusz PiotrowskiTransformer Interpretability Neuron-Level Interpretability
Anthropic · Georgia Institute of Technology
Training Dynamics of Transformers on Unbounded Inputs: Attention Learning Can Lag Output Layer Learning
Oct 14, 2026Yaomengxi Han, Debarghya GhoshdastidarTransformer Training Dynamics Neural Network Training Dynamics
Technical University of Munich
Learning Source Structure: Bayesian Prediction with Transformers
Oct 13, 2026Meir Feder, Ruediger UrbankeTransformer Sequence Modeling
School of Electrical and Computer Engineering, Tel Aviv University, Tel Aviv, Israel · School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland
On the Necessity of Attention-FFN Split in Vision Transformers
Oct 7, 2026Junhyeok Kim, Jinyeong Kim, Jae Wan Park +1Vision Transformer Attention Mechanisms
Department of Artificial Intelligence, Yonsei University, Seoul, Republic of Korea.
The Role of Feed-Forward Layers in Transformer Dynamics
Sep 28, 2026Thomas Jacob Maranzatto, Semih Akkoc, Sennur UlukusTransformer Transformer FFNs
Department of Electrical and Computer Engineering University of Maryland College Park, MD 20742, USA
Which the Eye Fears: Writing with Read-Blindness Explains Massive Activations in Transformers
Sep 28, 2026Swagatam Mukhopadhyay, Vishal Vivek Saley, Vraj Parikh +1Transformer Interpretability Transformer FFNs
PsiDagger · IIT Delhi
Change the Product, Keep the Parameters: Associative Algebra Layers for Transformers
Sep 26, 2026Ilya Koziev, Ivan OseledetsTransformer LLM Inference Acceleration
DAIMLD, Moscow, Russia · Artificial Intelligence Research Institute, Moscow, Russia
Model Casting and Low-Parameter Gating: Towards More Sparsely Activated FFNs
Sep 25, 2026Maria Lomeli, Antoine Groudiev, Matthijs Douze +7Efficient Neural Network Inference Transformer FFNs
Meta FAIR · École Normale Supérieure – PSL · École Normale Supérieure Paris-Saclay
Transformer MLP Gate Thresholds Are Couplings to a Carried Reference Direction
Sep 25, 2026Olli TuomiTransformer Interpretability Transformer FFNs
Evident Solutions Oy
Weakening Neurons: An Input-Output Functionality in Transformers with Outsize Influence
Sep 16, 2026Sebastian Gerstner, Hilal AlQuabeh, Kentaro Inui +1Transformer Interpretability Transformer FFNs
LMU Munich and Munich Center for Machine Learning · MBZUAI · Tohoku University +1
Structure is not mechanism: high-gain gated-FFN rows across text and genomic foundation models
Sep 13, 2026Alexandros Tzanakakis, Aris Karatzikos, Ilias Georgakopoulos-SoaresGenomic Foundation Models Transformer FFNs
Division of Pharmacology and Toxicology, College of Pharmacy, The University of Texas at Austin, Dell Pediatric Research Institute, Austin, TX, USA · Chandra Family Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX, USA · Department of Computer Science, College of Natural Sciences, The University of Texas at Austin, Austin, TX, USA
Curvature Cryptanalysis of Smooth Transformer Feed-Forward Networks
Aug 28, 2026Munawar Hasan, Apostol VassilevTransformer FFNs
National Institute of Standards and Technology, USA · Michigan Technological University, USA
TANGO: Treating Tokens as Operators
Aug 22, 2026Joshua NunleyTransformer FFNs Transformer
Department of Informatics Luddy School of Informatics, Computing, and Engineering Cognitive Science Program Indiana University Bloomington
Cryptanalytic Extraction of Isolated Bias-Free GLU Feed-Forward Blocks by Antipodal Separation
Aug 6, 2026Chunhui Shi, Xinwen FuTransformer FFNs Model Extraction Attacks
Feed-Forward Steering in Transformer Residual Dynamics
Aug 3, 2026Timur Mudarisov, Mikhail Burtsev, Radu StateTransformer Interpretability Transformer FFNs
University of Luxembourg · 2London Institute of Mathematical Sciences
Geometry-Guided Layerwise FFN Width Allocation in Transformers
Aug 3, 2026Timur Mudarisov, Mikhail Burtsev, Radu StateTransformer FFNs Transformer
University of Luxembourg · London Institute of Mathematical Sciences
A Controlled Study of Attention-Only Transformers
Jul 20, 2026Henry Ndubuaku, Karen Mosoyan, Jakub Mroz +5Transformer FFNs Transformer
Cactus Compute, Inc.
Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth
Jul 15, 2026Katie EverettTransformer FFNs Transformer
MIT CSAIL
Sparse Inter-Layer Dependencies of Transformer FFN Neurons
Jul 13, 2026Johannes Knittel, Hanspeter PfisterTransformer Interpretability Transformer FFNs
Harvard University
Training, Reading, and Editing Legible Transformers
Jul 9, 2026Mark OskinTransformer Interpretability Transformer FFNs
School of Computer Science and Engineering University of Washington
Explicit Fuzzy Logic in the Feed-Forward Layer: Self-Forgetting Quantifiers Discover Legible Grammatical-Licensing Detectors
Jun 30, 2026Mark OskinTransformer Interpretability Transformer FFNs
School of Computer Science and Engineering University of Washington
How Linear Is a Transformer Feed-Forward Block? Per-Block Linear Recoverability Is Learned, Not Architectural
Jun 12, 2026Stuart WhippTransformer FFNs Transformer Expressivity
Independent Research
Sparsely gated tiny linear experts
Jun 5, 2026Simon SchugTransformer FFNs Mixture-of-Experts Language Models
Department of Computer Science Princeton University
More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations
May 26, 2026Mingze Wang, Jinbo Wang, Yikuan Xia +2Transformer FFNs Transformer Expressivity
ByteDance Seed · Peking University
Sparsity Moves Computation: How FFN Architecture Reshapes Attention in Small Transformers
May 10, 2026Gabriel Smithline, Chris MascioliTransformer FFNs Transformer
University of Michigan
Large Vision-Language Models Get Lost in Attention
May 7, 2026Gongli Xi, Ye Tian, Mengyu Yang +5Transformer FFNs Vision-Language Models
School of Cyberspace Security, Beijing University of Posts and Telecommunications, Beijing, China · State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China · School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications, Beijing, China +3
Graph Memory Transformer (GMT)
Apr 26, 2026Nicola Zanarini, Niccolò Ferrari, Evelina LammaMemory-Augmented Neural Networks Transformer FFNs
Bonfiglioli Engineering s.r.l. · Department of Engineering, University of Ferrara, Via Saragat 1, 44122 Ferrara, Italy · NAIS s.r.l., Via Maria Callas 4, 40131 Bologna, Italy
Supernodes and Halos: Loss-Critical Hubs in LLM Feed-Forward Layers
Apr 26, 2026Audrey Cherilyn, Houman SafaaiTransformer Interpretability LLM Pruning
Kempner Institute at Harvard University
The Topological Trouble With Transformers
Apr 18, 2026Michael C. Mozer, Shoaib Ahmed Siddiqui, Rosanne LiuTransformer FFNs Recurrent Neural Networks
Google DeepMind

Oct 21, 2026J/K move · Enter open · S save

Reader--Writer Coupling in Transformer MLPs

Giyeong Oh, Sunghee Ahn, Youngjae Yu

Seoul National University

Transformer FFNs Transformer Interpretability Transformer Training Dynamics

Open paper AskPDF arXiv

A transformer MLP gives each hidden channel two roles: a reader that decides when the channel responds, and a writer that decides what it adds back to the residual stream. Read as a key--value memory, the layer can be measured one factor set at a time: the pairing of readers to writers is carried only by their shared index, invisible to any quantity computed from one factor set alone. Yet training couples the two factors through their joint update, so we ask what reading them together shows. The block supplies the joint instrument: the MLP writes back into the very stream it reads, so the exact Jacobian of its update is square, and its agreement with its own transpose resolves into reader--writer terms whose inner products give each channel, read as a couple, a signed pairing coordinate. Ranked by this coordinate, each block we measure has a small core of strongly coupled channels; where initialization gives either sign equally, the core is nearly all opposed: each writer points against the direction its reader responds to. This opposed core recurs from the smallest models trained from scratch to released checkpoints; its sign is in place within a few percent of training and holds while the coupling grows. Read as couples, most channels return their contribution against the reader that fired it, and the readers of later blocks take it up; few show in the logits. Training holds the couples in place: flipping the opposed writers moves the output toward common tokens, and a writer permuted onto another reader regrows to fit that reader while the reader stays.