Authors: Michael I. Ivanitskiy, John Jasper, Emily J. King, Dustin G. Mixon
Organizations: Department of Applied Mathematics and Statistics, Colorado School of Mines, Golden, CO, USA · Department of Mathematics & Statistics, Air Force Institute of Technology, Wright-Patterson AFB, OH, USA · Department of Mathematics, Colorado State University, Fort Collins, CO, USA · Department of Mathematics, The Ohio State University, Columbus, OH, USA · Translational Data Analytics Institute, The Ohio State University, Columbus, OH
We develop a mathematical theory of superposition in neural networks using tools from frame theory and compressed sensing. In our model, a sparse binary vector x of active features is encoded through an overcomplete dictionary W, and feature recovery is performed by applying ReLU(W⊤Wx+b) with an appropriate bias vector b. We prove several recovery theorems for this model. In the random-support setting, we establish high-probability support recovery for nearly tight, low-coherence dictionaries, with guarantees when the expected sparsity is up to order d/logn. In the worst-case support setting, we give a sharp and computable criterion for which sparsity levels permit support recovery. We apply this criterion to Gaussian random matrices and equiangular tight frames. For real equiangular tight frames with n>d+1, we determine the exact recovery threshold in terms of the coherence. The proof of this result for real equiangular tight frames relies on a novel characterization---which should be of independent interest to frame theorists---of the distribution of signs in the Gram matrix.
Figures & tables
Figure 1: Left: Every color represented by nonnegative linear combinations of the vectors (1,0) and (0,1) , which represent red and blue, respectively. Middle: Every color represented by nonnegative linear combinations of at most two of the vectors (1,0) , (−1/2,3/2) , and (−1/2,−3/2) , which represent red, green, and blue, respectively. Right: Different saturations of the same hue that are all represented by the same vector as a (in general not 2 -sparse) linear combination of (1,0) , (−1/2,3/2) , and (−1/2,−3/2) , namely, (1/3,0) . A 2 -sparsity assumption ensures unique representation, but also that not all colors are representable. In this case, only the darkest shade in the color bar is representable. An interactive version can be viewed at https://miv.name/web-tools/color-superposition.html
Figure 2: Intuition for superposition. Start with a nonnegative k -sparse vector x . In this case, k=1 and x∈{(1,0),(0,1)} . Left: Assume W⊤W is close to the identity matrix in some sense, e.g., the columns of W form an equiangular tight frame. Then W⊤Wx can be interpreted as a noisy version of x , i.e., it resides in the blue disk about x . Middle left: Shift this disk by adding the bias vector b , which tends to be close to a negative multiple of the all–ones vector. Middle right: At this point, we apply ReLU, which has the effect of projecting the disk onto the nonnegative orthant. In our case, the nonzero entries in x were large relative to the noise added by applying W⊤W , and the effect is that every point in this projection has the same support as x . Right: If we care about recovering x , not just its support, we can rescale the result so that it’s close to x . Since ReLU is positively homogeneous, one can avoid rescaling after the ReLU by instead taking W←α⋅W and b←α⋅b .
Figure 3: Visualization of superposition. Top left: Start with a k -sparse binary vector x . In this case, k=6 and x∈R50 . Top right: Fix a 25×50 equiangular tight frame W that is centered , meaning the column vectors sum to the zero vector. We plot the entries of W⊤Wx and the threshold θ:=6/7 as a dashed line. The entries above this threshold form the support of x . Bottom left: Taking b:=−θ⋅\mathbbm1 , then ReLU(W⊤Wx+b) has the same support as x , but the entries are smaller. Bottom right: Multiplying by α:=7/3 makes the nonzero entries closer to 1 . Since ReLU is positively homogeneous, one can avoid rescaling after the ReLU by instead taking W←α⋅W and b←α⋅b .
Neural networks can leverage feature superposition to encode more concepts than dimensions, but cross-feature interference constrains the linear accessibility of simultaneously active features. By framing linear accessibility as a compressed sensing problem, we derive high-probability bounds for fixed supports under subgaussian noise, proving the sufficient dimension scales linearly (d=Oε(klogm)) rather than prior worst-case quadratic limits. We characterize the asymmetry between active and inactive interference and the trade-off between interference and observation-noise budgets. We then validate these bounds across system parameters through Gaussian-tail approximations. We also introduce IHT-SAE, which uses learned iterative refinement to improve feature recovery beyond the limits of linear availability. These results quantify the geometric constraints of the linear representation hypothesis, providing a framework for evaluating sparse autoencoders, compositional generalization, and neural interpretability.
Enrico Vompa
Applied Artificial Intelligence Group Tallinn University of Technology, Estonia
Neural networks are thought to represent concepts as directions in their activation space, and superposition lets them encode more concepts than they have dimensions. It is natural to ask whether they can also compute more functions than they have neurons, i.e., perform computation in superposition. In this regime many functions of sparse inputs are evaluated by a layer with fewer neurons than there are functions to compute. Representation in superposition is by now fairly well understood, but computation in superposition is not, and there are few toy models of it arising through training rather than being hand designed. As a toy model of computation in superposition we study the compressed-computation setup: a single-hidden-layer ReLU network with 50 neurons that must compute the ReLU of each of 100 sparse input features. We show that training it under an L4 loss (the mean fourth power of the error), rather than the usual L2, elicits a solution that appears to compute all features in superposition. We then reverse-engineer this solution. We find that the network assigns each feature a sparse binary codeword over neurons and decodes it with a pseudoinverse of the encoder. Given these codewords, a description with only three scalars recovers most of the network's performance, and we validate it by building equivalent networks from hand-designed codes.
Superposition refers to neural networks representing more features than they have dimensions. It offers a possible explanation for polysemantic neurons and motivates methods for recovering interpretable features from neural activations. Theoretical models typically start with a given set of input features and assumptions about how their values vary across inputs, then study how a network encodes those values in a lower-dimensional hidden representation. Empirical work, by contrast, seeks to identify the features encoded in trained networks and determine their role in computation. In this survey, we review the geometry, learning, and computation of superposed representations, explaining how feature statistics and decoder choice affect the conclusions. To connect these theoretical accounts with evidence from trained networks, we compare practical methods for recovering and analyzing features and examine what their evaluations establish. Since accurate activation reconstruction alone does not establish feature identity or causal use, we discuss the methods' documented failures and applications in light of the evidence available for these different claims. Finally, we assess previously stated open problems and identify remaining theoretical and empirical questions about superposition in trained networks. We hope our work can pave the way for a deeper understanding of superposition and more reliable methods for interpreting neural networks.
Dai Shi, Xiaoyu Li, Andi Han +1
University of Cambridge · University of New South Wales · University of Sydney