Partial Soft-Matching Distance for Neural Representational Comparison with Partial Unit Correspondence
Authors: Chaitanya Kapoor, Alex H. Williams, Meenakshi Khosla
Organizations: Center for Neural Science, New York University, New York City, NY, 10003 · Center for Computational Neuroscience, Flatiron Institute, New York City, NY, 10003
Representational similarity metrics typically force all units to be matched, making them susceptible to noise and outliers common in neural representations. We extend the soft-matching distance to a partial optimal transport setting that allows some neurons to remain unmatched, yielding rotation-sensitive but robust correspondences. This partial soft-matching distance provides theoretical advantages -- relaxing strict mass conservation while maintaining interpretable transport costs -- and practical benefits through efficient neuron ranking in terms of cross-network alignment without costly iterative recomputation. In simulations, it preserves correct matches under outliers and reliably selects the correct model in noise-corrupted identification tasks. On fMRI data, it automatically excludes low-reliability voxels and produces voxel rankings by alignment quality that closely match computationally expensive brute-force approaches. It achieves higher alignment precision across homologous brain areas than standard soft-matching, which is forced to match all units regardless of quality. In deep networks, highly matched units exhibit similar maximally exciting images, while unmatched units show divergent patterns. This ability to partition by match quality enables focused analyses, e.g., testing whether networks have privileged axes even within their most aligned subpopulations. Overall, partial soft-matching provides a principled and practical method for representational comparison under partial correspondence.
Figures & tables
Figure 1: Partial Soft-Matching Distance for Matching Tuning Curves. (A) Two toy networks NA and NB ; the layer of interest for alignment is shown in red. A partial matching recovers one-to-one correspondences between units with highly similar tuning curves. Line color encodes match strength (darker = stronger). By contrast, a purely soft-matching yields a spurious pair (hatched units, red dotted line). (B) The same metric can be used to rank voxel/unit tuning-curve similarity between two subjects’ responses {r1,r2} , when exposed to the same visual stimulus.
Figure 2: Comparison of Balanced and Partial Soft-Matching. (a) We simulate two neural representations, X and Y , with 120 and 190 neurons respectively. The first 100 neurons represent pure signal , while the rest are pure noise . Red denotes all pure signal neurons in the two representations. (b) The L-curve method selects the optimal mass regularization parameter (=90/190≈0.47) , successfully discarding noisy units.
Figure 3: Model Selection Using Partial Soft-Matching. We simulate three synthetic representations to test whether partial soft-matching correctly identifies which of two candidate models— Ya or Yb —shares more signal with a reference population X (100 units). Ya contains all 100 signal units from X plus 60 noise units; Yb contains 100 units, 80 of which match X . The true fraction of shared units is known a priori , marked by a vertical gray line. With the L-curve-selected regularization, partial soft-matching yields correlation scores spartial(X,Ya)=0.715 and spartial(X,Yb)=0.645 , correctly favoring Ya . Standard soft-matching fails, with ssm(X,Ya)=0.339 and ssm(X,Yb)=0.415 , incorrectly preferring Yb due to forced matching of noise.
Figure 4: Aligning Voxel Responses Between Different Subjects in NSD. For each area, we plot the (i) partial soft-matching score at different mass regularization values and (ii) the mean noise ceilings of the voxels that were kept at that regularization. The alignment criterion consistently identifies low noise-ceiling voxels for exclusion.
Figure 5: Evaluating Methods for Identifying (Un)matched Neurons in Deep Networks. We compare three methods for ranking convolutional kernels by alignment between two ResNet- 18 models trained from different random initializations on ImageNet, across early, middle, and late layers. Removing low-alignment units identified by partial soft-matching yields alignment scores nearly identical to those obtained by removing kernels ranked least important via brute-force ablations, while correlation-based rankings perform poorly.
Figure 6: Evaluating Methods for Identifying (Un)matched Voxels in Brain Data. We evaluate three methods for ranking voxels by their degree of alignment between a subject pair from NSD across six visual areas. Removing low-alignment voxels identified by partial soft-matching yields alignment scores nearly identical to those obtained by removing voxels ranked least important via brute-force ablations, while correlation-based rankings perform poorly.
Brain Region Pair
SM Precision (↑)
ParSM Precision (↑)
ϵ=0.1(↑)
ϵ=0.3(↑)
V1v+V1d
0.839
0.905(0.76)
0.847(0.98)
0.855(0.94)
V1v+V2v
0.677
0.680(0.99)
0.680(0.96)
0.695(0.88)
V1v+V2d
0.880
0.884(0.99)
0.884(0.97)
0.894(0.91)
V1v+V3v
0.798
0.853(0.97)
0.803(0.99)
0.815(0.90)
V1v+V3d
0.882
0.890(0.98)
0.890(0.97)
0.913(0.91)
V1d+V2v
0.881
0.971(0.71)
0.889(0.96)
0.906(0.89)
Table 1: Precision of Cross-Subject Voxel Alignment Within and Across Visual Areas . Comparison of soft-matching (SM), partial soft-matching (ParSM) and noise ceiling thresholding to align voxels between visual regions in an NSD subject pair. Precision measures the fraction of matched voxels belonging to corresponding anatomical regions (higher = better specificity). We include the fraction of total voxels that contribute towards computing alignment in parenthesis. The ϵ values denote the noise ceiling threshold below which voxels are excluded. ParSM almost always yields higher precision by excluding voxels that lack clear correspondence.
Figure 8: Alignment between ResNet-18 models under original and randomly rotated coordinate systems, across early, middle and later layers and matched-mass thresholds. Rotation reduces alignment at all thresholds—including among the best-matched units—supporting convergence to a shared coordinate system even among the most aligned subpopulations.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1: Aligning Voxel Responses Between Subject IDs 5 and 7 in NSD. We plot the partial soft-matching alignment score achieved at different mass regularization values and the mean noise ceilings of the voxels that were kept at each regularization. For The alignment criterion consistently identifies low noise-ceiling voxels for exclusion for Subjects 5 and 7 .
Figure A2: Testing for Privileged Coordinate Systems Across Matched Neural Subpopulations. For all convolutional layers in a pair of ImageNet-trained ResNet- 18 models, we show that a privileged solution basis persists.
Figure A4: Partial Soft-Matching Using a Euclidean Cost Function (A) We visualize the L-curve elbow computed using a Euclidean cost function ( left ), and the corresponding transport plans ( right ). We find that the optimal regularization value s and transport plans are identical to that computed using a cosine similarity cost function. (B) Rank-ordering of neurons from 3 layers (early, middle and late) in an ImageNet-trained ResNet- 18 network using a squared Euclidean cost function. Removing units by using the Euclidean distance yields identical trends as using a cosine similarity cost. (C) We rank-order voxels from a subject pair in NSD using six visual areas by using a squared Euclidean cost function. Ordering voxels using a Euclidean distance cost function also reveals identical trends as using cosine similarity.
Activation-alignment measures such as Representational Similarity Analysis (RSA), Canonical Correlation Analysis (CCA), and Centered Kernel Alignment (CKA) are widely used to compare biological and artificial neural representations. Recent theoretical work interprets many of these methods as assessing agreement between optimal linear readouts over broad families of global tasks. However, agreement at the level of global readouts does not determine how a system uses local stimulus evidence. Specifically, representations may align in activation space yet differ in their sensitivity to small perturbations. To address this challenge, we introduce a complementary framework based on local decodable information, which focuses on a representation's ability, under noise, to discriminate small perturbations within a specified stimulus-coordinate subspace. Building on Fisher information and local representation geometry, we summarize each representation using the expected projected pullback/Fisher metric over that subspace. This formulation induces a second-moment family of local discrimination tasks, for which the resulting operator provides a minimal, complete dataset-level summary of expected discriminability. We compare these regularized signatures using a log-spectral distance on the manifold of symmetric positive definite (SPD) matrices, yielding the Spectral Riemannian Alignment Score (S-RAS) and a uniform multiplicative certificate over the corresponding family of lifted task values. Empirically, this framework enables the recovery of corresponding layers across independently trained artificial neural networks, supports transferable class-conditional probes, reveals controlled dissociations between standard and robust training, and uncovers stimulus-coordinate family effects across mouse visual cortex using the Allen Brain Observatory static gratings dataset.
Amirhossein Yavari, Farnaz Zamani Esfahlani
Stephenson School of Biomedical Engineering, University of Oklahoma · Data Science and Analytics Institute, University of Oklahoma
Neural representational dissimilarity quantifies differences between neural response distributions, and is essential for comparing neural codes across stimuli, brain areas, tasks, and models. Commonly used distance metrics involve different assumptions and are estimated with separate methods. Here, we show that a variety of distance metrics can be unified under a flow matching framework developed in deep generative models. That is, these distances arise as Jeffreys divergences under different velocity constraints. We find that flow matching has advantages for estimating distances involving complicated distributions and continuous variables. Furthermore, this framework enables the design of new distance metrics in a principled way. Together, flow matching provides a unified approach for understanding, estimating, and designing neural representational dissimilarity metrics.
Zeyuan Ye, Xue-Xin Wei
Department of Neuroscience, The University of Texas at Austin
Comparing internal representations is a central goal in neuroscience and machine learning, but standard linear alignment metrics (Representational Similarity Analysis, Centered Kernel Alignment, and linear regression) are frequently applied to neural activity coordinates rather than on the underlying features. We show this matters when neural systems operate in superposition, encoding more features than they have neurons via linear compression. Closed-form derivations prove that these metrics depend on the Gram matrices of each system's projection, not on the latent features themselves: alignment thus combines what a system represents with how it is encoded. For those interested in what features two systems share, this is a problem: Two networks can have identical feature content yet appear more dissimilar than networks exhibiting partial feature overlap. This apparent misalignment need not reflect lost information as compressed sensing guarantees sparse features remain recoverable from the compressed activity. We confirm this by training supervised TopK sparse autoencoders that realize solvable compressed sensing by construction, finding alignment on recovered latents restored even when raw-activation alignment remains deflated. We extend the result to unsupervised SAEs trained without ground-truth latents, and to pretrained vision and language model SAEs, where SAE-latent alignment exceeds raw-activation alignment, consistent with superposition in real systems.
Sunny Liu, Habon Issa, André Longon +4
Cold Spring Harbor Laboratory · UC San Diego · Anthropic +2