Coarse-Graining Hidden Representations: Unsupervised Neuron Selection via Mapping Entropy

Margherita Mele; Andrea Castagna; Roberto Menichetti; Raffaello Potestio; Alessandro Ingrosso

cs.LGSep 4, 2026

Coarse-Graining Hidden Representations: Unsupervised Neuron Selection via Mapping Entropy

Authors: Margherita Mele, Andrea Castagna, Roberto Menichetti, Raffaello Potestio, Alessandro Ingrosso

Organizations: Physics Department, University of Trento, via Sommarive, 14 I-38123 Trento, Italy · INFN-TIFPA, Trento Institute for Fundamental Physics and Applications, I-38123 Trento, Italy · Donders Centre for Neuroscience, Radboud University, Nijmegen, The Netherlands

Abstract

Overparameterized neural networks carry far more hidden units than a task nominally requires, raising the question of which neurons are essential and whether that distinction is legible in the representation itself, without labels or gradients. We cast neuron selection as the problem of coarse-graining the hidden layer by retaining a subset of its neurons, and score each putative selection by the mapping entropy (ME). This quantity measures the loss of discriminatory power inherent in discarding part of the network neurons, and the selection that minimises the ME is taken as particularly informative. This criterion is fully unsupervised, in that it depends only on hidden-activation statistics. In teacher-student networks, ME optimisation recovers the minimal teacher-consistent representation and retains extra units in proportion to the hidden layer's residual variability; in a non-linear Gaussian process task, it selects coherent functional-class mappings whose preferred class shifts across training. On this task and on translation-augmented MNIST, ME-selected subnetworks outperform random subsets of equal size, most clearly under strong compression - linking configurational distinguishability to predictive performance.

Figures & tables

Figure 1: TS system and ME analysis of hidden-layer representations. ( a ) Schematic representation of the teacher and student networks. The teacher has

M

hidden units with orthogonal first-layer weights

\mathbf{B}_{i}

, while the student has

K>M

hidden units labelled by

(i,\alpha)

. In the controlled construction, the student weights satisfy

\mathbf{J}_{i,\alpha}\!\cdot\!\mathbf{B}_{j}=\sqrt{1-\eta^{2}}\,\delta_{ij}

. ( b ) Training dynamics of the overlaps

\mathbf{J}_{i,\alpha}\!\cdot\!\mathbf{B}_{j}

in the M2-K10 case, shown as a function of the training epoch. Blue and green curves denote the overlaps with

\mathbf{B}_{1}

and

\mathbf{B}_{2}

, respectively, while grey curves denote the components orthogonal to the teacher directions. ( c ) Generalization error as a function of the training epoch for the same system. ( d ) Effective mismatch parameter

\eta_{\mathrm{eff}}

as a function of the training epoch. The dashed horizontal line indicates the final value

\eta_{\mathrm{final}}=0.04

. ( e , f ) Balance

\Delta

of the ME-minimising mapping as a function of the mismatch parameter

\eta

, for the M2-K10 and M5-K25 systems, respectively. Different colours identify different values of the retained CG dimensionality

n_{\mathrm{cg}}

, as indicated by the colour bar. Circular markers denote the controlled in silico construction, and square markers denote the trained networks, plotted at the corresponding effective mismatch value

\eta_{\mathrm{eff}}

. Note the different scale of the

\Delta

axis in the two plots.

Figure 2: NLGP system and ME analysis of hidden-layer representations. ( a ) Schematic of the classification task and network architecture. Input patterns from the two classes, characterised by different correlation lengths, are shown alongside the one-hidden-layer network with $K=30$ neurons. ( b ) Representative examples of hidden neurons after training: a localised neuron (left), with weights concentrated on a subset of input coordinates, and an oscillatory neuron (right), with extended alternating-sign structure. ( c ) Distribution of the network output evaluated using only localised neurons (green) or only oscillatory neurons (blue), compared across the two classes (hatched histograms). ( d ) Left: average fraction of localised neurons in the ME-minimising mapping as a function of the number of retained neurons $n_{\mathrm{cg}}$ , for different training epochs (colour-coded). Right: selection probability $p_{\mathrm{in}}$ of each neuron, shown as a function of $n_{\mathrm{cg}}$ , for two representative epochs (100 and 1000). Neurons are ordered according to their bias value. ( e ) ME $S_{\mathrm{map}}$ evaluated for mappings composed exclusively of localised neurons (green) or oscillatory neurons (blue), as a function of training epoch. ( f ) DoS of the mapping space as a function of $S_{\mathrm{map}}$ and the number of localised neurons, for different training epochs. Overlaid points indicate average optimisation trajectories.

Figure 3: NLGP reduced-network performance at different training epochs. Classification accuracy as a function of the number of retained neurons $n_{\mathrm{cg}}$ . For each $n_{\mathrm{cg}}$ , the green markers denote the accuracy of the reduced network obtained from the ME-selected subset, the orange markers the accuracies obtained from random subsets of the same size, and the dashed grey line the accuracy of the full network. After pruning, the output bias is re-optimised while keeping the hidden-layer parameters fixed. Note that the $y$ axes do not start from zero.

Figure 4: Reduced-network performance for the MNIST binary classification task ( $1$ vs $7$ ) with data augmentation by random translations. Classification accuracy as a function of the number of retained neurons $n_{\mathrm{cg}}$ . For each $n_{\mathrm{cg}}$ , the green markers denote the accuracy of the reduced network obtained from the ME-selected subset, the orange markers the accuracies obtained from random subsets of the same size, and the dashed grey line the accuracy of the full network. Note that the $y$ axes do not start from zero.

Neural Network Pruning

Neural network pruning removes weights, neurons, channels, heads, or layers to reduce model size and computation while retaining useful behavior.

See all 292 papers

Feature Superposition in Neural Networks: From Theory to Practice
Sep 6, 2026Dai Shi, Xiaoyu Li, Andi Han +1Feature Superposition Neural Network Interpretability
University of Cambridge. · University of New South Wales. · University of Sydney.
Similarity of Neural Network Representations in Superposition
Mar 31, 2026Sunny Liu, Habon Issa, André Longon +4Feature Superposition Representational Similarity Analysis
Cold Spring Harbor Laboratory · UC San Diego · Anthropic +2
Hidden Gauge Controls Feature Specialization in ReLU Networks
Aug 7, 2026Tongxi WangReLU Neural Networks Neural Network Training Dynamics
School of Future Technology Southeast University Nanjing, China

Sep 6, 2026J/K move · Enter open · S save

Feature Superposition in Neural Networks: From Theory to Practice

Dai Shi, Xiaoyu Li, Andi Han, José Miguel Hernández-Lobato

University of Cambridge. · University of New South Wales. · University of Sydney.

Feature Superposition Neural Network Interpretability Neural Representation Geometry

Open paper AskPDF arXiv

Superposition refers to neural networks representing more features than they have dimensions. It was proposed to explain polysemantic neurons, which respond to several features, and also other phenomena, such as neural scaling laws and adversarial vulnerability. A feature stored in superposition is useful to the extent that its value can be read back despite interference from the other features. The theory of superposition mainly asks how wide a layer must be for the features to be read back, and by which decoder. In theoretical models, a single linear map that reads back every combination of nonzero features needs a width nearly quadratic in their number, while sparse optimization can succeed with a width linear in that number. In practice, one may hope to interpret a trained network by recovering its features in the same way. However, its features are usually unknown, and thus they are estimated from its activations or weights by recovery methods, such as sparse autoencoders and transcoders. In this work, we organize the theory of superposition by the decoder class and its practice by the strength of the claims about a feature supported by each method's results. On the theory side, we review the lower bounds on the overlap between feature directions, the conditions under which trained toy models share dimensions among features, and the separation between storing features and computing new features. On the practice side, we categorize recovery methods into four families, from probes for a named feature to decompositions of the trained weights. We then compare their evaluations, failure modes, and applications, and grade the evidence on one scale of claims about a feature. Finally, we assess previously stated open problems and identify remaining theoretical and empirical questions about superposition in trained networks. Project page: https://eethanshi.github.io/awesome-superposition/

Abstract

Neural Network Pruning

Explore similar work

Feature Superposition in Neural Networks: From Theory to Practice

Similarity of Neural Network Representations in Superposition

Hidden Gauge Controls Feature Specialization in ReLU Networks