Organizations: Physics Department, University of Trento, via Sommarive, 14 I-38123 Trento, Italy · INFN-TIFPA, Trento Institute for Fundamental Physics and Applications, I-38123 Trento, Italy · Donders Centre for Neuroscience, Radboud University, Nijmegen, The Netherlands
Overparameterized neural networks carry far more hidden units than a task nominally requires, raising the question of which neurons are essential and whether that distinction is legible in the representation itself, without labels or gradients. We cast neuron selection as the problem of coarse-graining the hidden layer by retaining a subset of its neurons, and score each putative selection by the mapping entropy (ME). This quantity measures the loss of discriminatory power inherent in discarding part of the network neurons, and the selection that minimises the ME is taken as particularly informative. This criterion is fully unsupervised, in that it depends only on hidden-activation statistics. In teacher-student networks, ME optimisation recovers the minimal teacher-consistent representation and retains extra units in proportion to the hidden layer's residual variability; in a non-linear Gaussian process task, it selects coherent functional-class mappings whose preferred class shifts across training. On this task and on translation-augmented MNIST, ME-selected subnetworks outperform random subsets of equal size, most clearly under strong compression - linking configurational distinguishability to predictive performance.
Figures & tables
Figure 1: TS system and ME analysis of hidden-layer representations. ( a ) Schematic representation of the teacher and student networks. The teacher has M hidden units with orthogonal first-layer weights Bi , while the student has K>M hidden units labelled by (i,α) . In the controlled construction, the student weights satisfy Ji,α⋅Bj=1−η2δij . ( b ) Training dynamics of the overlaps Ji,α⋅Bj in the M2-K10 case, shown as a function of the training epoch. Blue and green curves denote the overlaps with B1 and B2 , respectively, while grey curves denote the components orthogonal to the teacher directions. ( c ) Generalization error as a function of the training epoch for the same system. ( d ) Effective mismatch parameter ηeff as a function of the training epoch. The dashed horizontal line indicates the final value ηfinal=0.04 . ( e , f ) Balance Δ of the ME-minimising mapping as a function of the mismatch parameter η , for the M2-K10 and M5-K25 systems, respectively. Different colours identify different values of the retained CG dimensionality ncg , as indicated by the colour bar. Circular markers denote the controlled in silico construction, and square markers denote the trained networks, plotted at the corresponding effective mismatch value ηeff . Note the different scale of the Δ axis in the two plots.
Figure 2: NLGP system and ME analysis of hidden-layer representations. ( a ) Schematic of the classification task and network architecture. Input patterns from the two classes, characterised by different correlation lengths, are shown alongside the one-hidden-layer network with K=30 neurons. ( b ) Representative examples of hidden neurons after training: a localised neuron (left), with weights concentrated on a subset of input coordinates, and an oscillatory neuron (right), with extended alternating-sign structure. ( c ) Distribution of the network output evaluated using only localised neurons (green) or only oscillatory neurons (blue), compared across the two classes (hatched histograms). ( d ) Left: average fraction of localised neurons in the ME-minimising mapping as a function of the number of retained neurons ncg , for different training epochs (colour-coded). Right: selection probability pin of each neuron, shown as a function of ncg , for two representative epochs (100 and 1000). Neurons are ordered according to their bias value. ( e ) ME Smap evaluated for mappings composed exclusively of localised neurons (green) or oscillatory neurons (blue), as a function of training epoch. ( f ) DoS of the mapping space as a function of Smap and the number of localised neurons, for different training epochs. Overlaid points indicate average optimisation trajectories.
Figure 3: NLGP reduced-network performance at different training epochs. Classification accuracy as a function of the number of retained neurons ncg . For each ncg , the green markers denote the accuracy of the reduced network obtained from the ME-selected subset, the orange markers the accuracies obtained from random subsets of the same size, and the dashed grey line the accuracy of the full network. After pruning, the output bias is re-optimised while keeping the hidden-layer parameters fixed. Note that the y axes do not start from zero.
Figure 4: Reduced-network performance for the MNIST binary classification task ( 1 vs 7 ) with data augmentation by random translations. Classification accuracy as a function of the number of retained neurons ncg . For each ncg , the green markers denote the accuracy of the reduced network obtained from the ME-selected subset, the orange markers the accuracies obtained from random subsets of the same size, and the dashed grey line the accuracy of the full network. Note that the y axes do not start from zero.
Superposition refers to neural networks representing more features than they have dimensions. It was proposed to explain polysemantic neurons, which respond to several features, and also other phenomena, such as neural scaling laws and adversarial vulnerability. A feature stored in superposition is useful to the extent that its value can be read back despite interference from the other features. The theory of superposition mainly asks how wide a layer must be for the features to be read back, and by which decoder. In theoretical models, a single linear map that reads back every combination of nonzero features needs a width nearly quadratic in their number, while sparse optimization can succeed with a width linear in that number. In practice, one may hope to interpret a trained network by recovering its features in the same way. However, its features are usually unknown, and thus they are estimated from its activations or weights by recovery methods, such as sparse autoencoders and transcoders. In this work, we organize the theory of superposition by the decoder class and its practice by the strength of the claims about a feature supported by each method's results. On the theory side, we review the lower bounds on the overlap between feature directions, the conditions under which trained toy models share dimensions among features, and the separation between storing features and computing new features. On the practice side, we categorize recovery methods into four families, from probes for a named feature to decompositions of the trained weights. We then compare their evaluations, failure modes, and applications, and grade the evidence on one scale of claims about a feature. Finally, we assess previously stated open problems and identify remaining theoretical and empirical questions about superposition in trained networks. Project page: https://eethanshi.github.io/awesome-superposition/
Dai Shi, Xiaoyu Li, Andi Han +1
University of Cambridge. · University of New South Wales. · University of Sydney.
Comparing internal representations is a central goal in neuroscience and machine learning, but standard linear alignment metrics (Representational Similarity Analysis, Centered Kernel Alignment, and linear regression) are frequently applied to neural activity coordinates rather than on the underlying features. We show this matters when neural systems operate in superposition, encoding more features than they have neurons via linear compression. Closed-form derivations prove that these metrics depend on the Gram matrices of each system's projection, not on the latent features themselves: alignment thus combines what a system represents with how it is encoded. For those interested in what features two systems share, this is a problem: Two networks can have identical feature content yet appear more dissimilar than networks exhibiting partial feature overlap. This apparent misalignment need not reflect lost information as compressed sensing guarantees sparse features remain recoverable from the compressed activity. We confirm this by training supervised TopK sparse autoencoders that realize solvable compressed sensing by construction, finding alignment on recovered latents restored even when raw-activation alignment remains deflated. We extend the result to unsupervised SAEs trained without ground-truth latents, and to pretrained vision and language model SAEs, where SAE-latent alignment exceeds raw-activation alignment, consistent with superposition in real systems.
Sunny Liu, Habon Issa, André Longon +4
Cold Spring Harbor Laboratory · UC San Diego · Anthropic +2
Training changes a network's predictions while allocating task-relevant structure across its internal units. In an overparameterized ReLU network, several neurons can begin with exactly the same functional role, yet one may acquire a teacher feature while the others become redundant. We call the identity of that neuron feature ownership and ask whether it can be controlled by a parameter choice invisible to the initial predictor. In a tractable Gaussian teacher--student model, we fix the complete initial function and vary only a positive-homogeneous scaling gauge. Opposite gauges produce distinct feature trajectories and a sharp Θ(D2) separation in specialization time that no global change of clock can explain. Among any fixed number of initially duplicate students, assigning the favorable gauge to one neuron deterministically selects it as the owner and drives the remaining functional contribution to zero. An exact reaction--transport decomposition attributes the effect to different mobilities for changing a feature's coefficient and direction. We prove global selection and functional pruning, extend finite-time selection to visible perturbations and small-step full-batch gradient descent, and verify the predicted loss, alignment, pruning, and dissipation trajectories in population and finite-sample training. The initial predictor therefore determines neither when the feature is learned nor which neuron learns it.
Tongxi Wang
School of Future Technology Southeast University Nanjing, China