Transformer FFNs

FFN: Feed-Forward Network

Follow this topic Ask about this topic

Momentum

2 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 24

All topics

On the Necessity of Attention-FFN Split in Vision Transformers
Oct 7, 2026Junhyeok Kim, Jinyeong Kim, Jae Wan Park +1Vision Transformer Attention Mechanisms
Department of Artificial Intelligence, Yonsei University, Seoul, Republic of Korea.
Which the Eye Fears: Writing with Read-Blindness Explains Massive Activations in Transformers
Sep 28, 2026Swagatam Mukhopadhyay, Vishal Vivek Saley, Vraj Parikh +1Transformer Interpretability Transformer FFNs
PsiDagger · IIT Delhi
Weakening Neurons: An Input-Output Functionality in Transformers with Outsize Influence
Sep 16, 2026Sebastian Gerstner, Hilal AlQuabeh, Kentaro Inui +1Transformer Interpretability Transformer FFNs
LMU Munich and Munich Center for Machine Learning · MBZUAI · Tohoku University +1
Curvature Cryptanalysis of Smooth Transformer Feed-Forward Networks
Aug 28, 2026Munawar Hasan, Apostol VassilevTransformer FFNs
National Institute of Standards and Technology, USA · Michigan Technological University, USA
TANGO: Treating Tokens as Operators
Aug 22, 2026Joshua NunleyTransformer FFNs Transformer
Department of Informatics Luddy School of Informatics, Computing, and Engineering Cognitive Science Program Indiana University Bloomington
Cryptanalytic Extraction of Isolated Bias-Free GLU Feed-Forward Blocks by Antipodal Separation
Aug 6, 2026Chunhui Shi, Xinwen FuTransformer FFNs Model Extraction Attacks
Feed-Forward Steering in Transformer Residual Dynamics
Aug 3, 2026Timur Mudarisov, Mikhail Burtsev, Radu StateTransformer Interpretability Transformer FFNs
University of Luxembourg · 2London Institute of Mathematical Sciences
Geometry-Guided Layerwise FFN Width Allocation in Transformers
Aug 3, 2026Timur Mudarisov, Mikhail Burtsev, Radu StateTransformer FFNs Transformer
University of Luxembourg · London Institute of Mathematical Sciences
A Controlled Study of Attention-Only Transformers
Jul 20, 2026Henry Ndubuaku, Karen Mosoyan, Jakub Mroz +5Transformer FFNs Transformer
Cactus Compute, Inc.
Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth
Jul 15, 2026Katie EverettTransformer FFNs Transformer
MIT CSAIL
Sparse Inter-Layer Dependencies of Transformer FFN Neurons
Jul 13, 2026Johannes Knittel, Hanspeter PfisterTransformer Interpretability Transformer FFNs
Harvard University
Training, Reading, and Editing Legible Transformers
Jul 9, 2026Mark OskinTransformer Interpretability Transformer FFNs
School of Computer Science and Engineering University of Washington
Explicit Fuzzy Logic in the Feed-Forward Layer: Self-Forgetting Quantifiers Discover Legible Grammatical-Licensing Detectors
Jun 30, 2026Mark OskinTransformer Interpretability Transformer FFNs
School of Computer Science and Engineering University of Washington
How Linear Is a Transformer Feed-Forward Block? Per-Block Linear Recoverability Is Learned, Not Architectural
Jun 12, 2026Stuart WhippTransformer FFNs Transformer Expressivity
Independent Research
Sparsely gated tiny linear experts
Jun 5, 2026Simon SchugTransformer FFNs Mixture-of-Experts Language Models
Department of Computer Science Princeton University
More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations
May 26, 2026Mingze Wang, Jinbo Wang, Yikuan Xia +2Transformer FFNs Transformer Expressivity
ByteDance Seed · Peking University
Sparsity Moves Computation: How FFN Architecture Reshapes Attention in Small Transformers
May 10, 2026Gabriel Smithline, Chris MascioliTransformer FFNs Transformer
University of Michigan
Large Vision-Language Models Get Lost in Attention
May 7, 2026Gongli Xi, Ye Tian, Mengyu Yang +5Transformer FFNs Vision-Language Models
School of Cyberspace Security, Beijing University of Posts and Telecommunications, Beijing, China · State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China · School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications, Beijing, China +3
Graph Memory Transformer (GMT)
Apr 26, 2026Nicola Zanarini, Niccolò Ferrari, Evelina LammaMemory-Augmented Neural Networks Transformer FFNs
Bonfiglioli Engineering s.r.l. · Department of Engineering, University of Ferrara, Via Saragat 1, 44122 Ferrara, Italy · NAIS s.r.l., Via Maria Callas 4, 40131 Bologna, Italy
Supernodes and Halos: Loss-Critical Hubs in LLM Feed-Forward Layers
Apr 26, 2026Audrey Cherilyn, Houman SafaaiTransformer Interpretability LLM Pruning
Kempner Institute at Harvard University
The Topological Trouble With Transformers
Apr 18, 2026Michael C. Mozer, Shoaib Ahmed Siddiqui, Rosanne LiuTransformer FFNs Recurrent Neural Networks
Google DeepMind
Revisiting the Shape Convention of Transformer Language Models
Feb 6, 2026Feng-Ting Liao, Guan-Ting Yi, Tzu-Quan Lin +2Efficient Transformer Inference Transformer FFNs
MediaTek Research · National Taiwan University
Equivalence of Context and Parameter Updates in Modern Transformer Blocks
Nov 22, 2025Adrian Goldwaser, Michael Munn, Javier Gonzalvo +1Transformer Interpretability Transformer FFNs
University of Cambridge, UK · Google Research
SG-Blend: Learning an Interpolation Between Improved Swish and GELU for Robust Neural Representations
May 29, 2025Gaurav Sarkar, Syed Affan Daimi, Jay Gala +1Transformer FFNs Transformer
IIIT · Microsoft · Intel

Oct 7, 2026, cs.CVJ/K move · Enter open · S save

On the Necessity of Attention-FFN Split in Vision Transformers

Junhyeok Kim, Jinyeong Kim, Jae Wan Park, Seong Jae Hwang

Department of Artificial Intelligence, Yonsei University, Seoul, Republic of Korea.

Vision Transformer Attention Mechanisms Transformer FFNs

Open paper AskPDF arXiv

The standard Transformer architecture relies on a rigid pattern that alternates Attention and Feed-Forward Network (FFN) layers. Despite its widespread adoption, the inductive bias imposed by this strict separation has not been systematically examined. In this work, we investigate the necessity of the Attention-FFN dichotomy in Vision Transformers (ViTs). To facilitate this analysis, we introduce the AttenFeed module, a unified component that integrates the functional properties of both Attention and FFN. Based on this module, we devise the unified Vision Transformer (uViT), which replaces the conventional alternating Attention-FFN structure with a sequence of AttenFeed modules. We then use uViT as a control group that relaxes the Attention-FFN dichotomy of the standard ViT and systematically compare the two models across multiple datasets and model scales. Our experiments reveal that the Attention-FFN dichotomy can hinder performance at smaller model scales due to the rigid parameter allocation of ViTs. The AttenFeed module and uViT serve as new analytical tools for understanding the Attention-FFN structure and offer theoretical insights into the heuristically designed architecture of conventional ViTs.

Momentum

Related topics

Latest papers 24