Revenge of Monosemanticity: Neuron Specialization as a New Form of Feature Learning in MLPs

Amirhesam Abedsoltan; Enric Boix-Adsera; Fivos Kalogiannis; Mikhail Belkin

cs.LGAug 25, 2026

Revenge of Monosemanticity: Neuron Specialization as a New Form of Feature Learning in MLPs

Authors: Amirhesam Abedsoltan, Enric Boix-Adsera, Fivos Kalogiannis, Mikhail Belkin

Abstract

Understanding how neural networks learn and organize features is central to understanding their behavior. Much existing theory of feature learning has focused on the emergence of a global low-dimensional representation. We show that this picture is incomplete. In regression problems with clustered data, we demonstrate that multilayer perceptrons (MLPs) naturally develop monosemantic specialized neurons: individual neurons become strongly aligned with a specific predictive feature relevant to a particular region of the input space. Rather than learning a single global low-dimensional representation, MLPs learn a collection of local low-dimensional representations. We show that this ability to specialize gives MLPs a provable data-efficiency advantage over feature-learning methods based on a global low-dimensional representation.

Explore similar work

From Latent Space to Training Data: Explainable Specialization in Minimal MLPs
May 25, 2026Enrique Alba, Ezequiel Lopez-RubioMultilayer Perceptrons Latent Space
Emergent Generalization by Representation Learning in Artificial Neural Networks
Jul 11, 2026Hardik Rajpal, Dan GoodmanImplicit Neural Representations Neural Network
Sparse Competition during Training For the Emergence of Specialized Modules
Aug 31, 2026Baptiste Rossigneux, Karim HarounModular Framework Emergence

May 25, 2026 · cs.LGJ/K move · Enter open · S save

From Latent Space to Training Data: Explainable Specialization in Minimal MLPs

Enrique Alba, Ezequiel Lopez-Rubio

aITIS Software, University of Malaga, Spain

We here study whether training biases can make hidden neurons specialize in minimal one-hidden-layer MLPs, and whether such specialization improves prototype-based reconstruction of the training dataset from the learned weights. We consider Gaussianactivation MLPs of width equal to dataset size and compare three structural losses that respectively encourage coverage of the training samples, separation between neuron-induced prototypes, and low overlap of hidden responses, against the standard fitting baseline. Experiments on uniformly sampled one-dimensional datasets show a stable pattern from N = 3 to N = 100 across 480 controlled runs. Coverage regularization gives the lowest mean reconstruction error at every tested size and raises the prototype-usage specialization ratio relative to the standard baseline, while separation has mixed effects and overlap penalties are systematically harmful. We show that the harm is not an optimization failure: overlap-active approaches fit the data as well as overlap-free ones but route the optimizer to a degenerate equilibrium in which prototype centers are pushed outside the convex hull of the training inputs. Coverage cannot reward this expulsion and acts as an attractor: separation admits it only at large temperature and overlap admits it at the nominal hyperparameter choice. A direct τ-sweep on the separation-only mask and a prototype-position visualization at N = 100 confirm the mechanism. The findings yield a simple design principle for prototype-recoverability-aware training: every repulsive structural loss must be compensated by a compatible attractor, or it will collapse the latent geometry it was meant to refine.

Multilayer Perceptrons Latent Space Training Data

Open paper Ask about it PDF

Abstract

Explore similar work

From Latent Space to Training Data: Explainable Specialization in Minimal MLPs

Emergent Generalization by Representation Learning in Artificial Neural Networks

Sparse Competition during Training For the Emergence of Specialized Modules