Partial Soft-Matching Distance for Neural Representational Comparison with Partial Unit Correspondence
Authors: Chaitanya Kapoor, Alex H. Williams, Meenakshi Khosla
Organizations: Center for Neural Science, New York University, New York City, NY, 10003 · Center for Computational Neuroscience, Flatiron Institute, New York City, NY, 10003
Representational similarity metrics typically force all units to be matched, making them susceptible to noise and outliers common in neural representations. We extend the soft-matching distance to a partial optimal transport setting that allows some neurons to remain unmatched, yielding rotation-sensitive but robust correspondences. This partial soft-matching distance provides theoretical advantages -- relaxing strict mass conservation while maintaining interpretable transport costs -- and practical benefits through efficient neuron ranking in terms of cross-network alignment without costly iterative recomputation. In simulations, it preserves correct matches under outliers and reliably selects the correct model in noise-corrupted identification tasks. On fMRI data, it automatically excludes low-reliability voxels and produces voxel rankings by alignment quality that closely match computationally expensive brute-force approaches. It achieves higher alignment precision across homologous brain areas than standard soft-matching, which is forced to match all units regardless of quality. In deep networks, highly matched units exhibit similar maximally exciting images, while unmatched units show divergent patterns. This ability to partition by match quality enables focused analyses, e.g., testing whether networks have privileged axes even within their most aligned subpopulations. Overall, partial soft-matching provides a principled and practical method for representational comparison under partial correspondence.
Figures & tables
Figure 1: Partial Soft-Matching Distance for Matching Tuning Curves. (A) Two toy networks NA and NB ; the layer of interest for alignment is shown in red. A partial matching recovers one-to-one correspondences between units with highly similar tuning curves. Line color encodes match strength (darker = stronger). By contrast, a purely soft-matching yields a spurious pair (hatched units, red dotted line). (B) The same metric can be used to rank voxel/unit tuning-curve similarity between two subjects’ responses {r1,r2} , when exposed to the same visual stimulus.
Figure 2: Comparison of Balanced and Partial Soft-Matching. (a) We simulate two neural representations, X and Y , with 120 and 190 neurons respectively. The first 100 neurons represent pure signal , while the rest are pure noise . Red denotes all pure signal neurons in the two representations. (b) The L-curve method selects the optimal mass regularization parameter (=90/190≈0.47) , successfully discarding noisy units.
Figure 3: Model Selection Using Partial Soft-Matching. We simulate three synthetic representations to test whether partial soft-matching correctly identifies which of two candidate models— Ya or Yb —shares more signal with a reference population X (100 units). Ya contains all 100 signal units from X plus 60 noise units; Yb contains 100 units, 80 of which match X . The true fraction of shared units is known a priori , marked by a vertical gray line. With the L-curve-selected regularization, partial soft-matching yields correlation scores spartial(X,Ya)=0.715 and spartial(X,Yb)=0.645 , correctly favoring Ya . Standard soft-matching fails, with ssm(X,Ya)=0.339 and ssm(X,Yb)=0.415 , incorrectly preferring Yb due to forced matching of noise.
Figure 4: Aligning Voxel Responses Between Different Subjects in NSD. For each area, we plot the (i) partial soft-matching score at different mass regularization values and (ii) the mean noise ceilings of the voxels that were kept at that regularization. The alignment criterion consistently identifies low noise-ceiling voxels for exclusion.
Figure 5: Evaluating Methods for Identifying (Un)matched Neurons in Deep Networks. We compare three methods for ranking convolutional kernels by alignment between two ResNet- 18 models trained from different random initializations on ImageNet, across early, middle, and late layers. Removing low-alignment units identified by partial soft-matching yields alignment scores nearly identical to those obtained by removing kernels ranked least important via brute-force ablations, while correlation-based rankings perform poorly.
Figure 6: Evaluating Methods for Identifying (Un)matched Voxels in Brain Data. We evaluate three methods for ranking voxels by their degree of alignment between a subject pair from NSD across six visual areas. Removing low-alignment voxels identified by partial soft-matching yields alignment scores nearly identical to those obtained by removing voxels ranked least important via brute-force ablations, while correlation-based rankings perform poorly.
Brain Region Pair
SM Precision (↑)
ParSM Precision (↑)
ϵ=0.1(↑)
ϵ=0.3(↑)
V1v+V1d
0.839
0.905(0.76)
0.847(0.98)
0.855(0.94)
V1v+V2v
0.677
0.680(0.99)
0.680(0.96)
0.695(0.88)
V1v+V2d
0.880
0.884(0.99)
0.884(0.97)
0.894(0.91)
V1v+V3v
0.798
0.853(0.97)
0.803(0.99)
0.815(0.90)
V1v+V3d
0.882
0.890(0.98)
0.890(0.97)
0.913(0.91)
V1d+V2v
0.881
0.971(0.71)
0.889(0.96)
0.906(0.89)
Table 1: Precision of Cross-Subject Voxel Alignment Within and Across Visual Areas . Comparison of soft-matching (SM), partial soft-matching (ParSM) and noise ceiling thresholding to align voxels between visual regions in an NSD subject pair. Precision measures the fraction of matched voxels belonging to corresponding anatomical regions (higher = better specificity). We include the fraction of total voxels that contribute towards computing alignment in parenthesis. The ϵ values denote the noise ceiling threshold below which voxels are excluded. ParSM almost always yields higher precision by excluding voxels that lack clear correspondence.
Figure 8: Alignment between ResNet-18 models under original and randomly rotated coordinate systems, across early, middle and later layers and matched-mass thresholds. Rotation reduces alignment at all thresholds—including among the best-matched units—supporting convergence to a shared coordinate system even among the most aligned subpopulations.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1: Aligning Voxel Responses Between Subject IDs 5 and 7 in NSD. We plot the partial soft-matching alignment score achieved at different mass regularization values and the mean noise ceilings of the voxels that were kept at each regularization. For The alignment criterion consistently identifies low noise-ceiling voxels for exclusion for Subjects 5 and 7 .
Figure A2: Testing for Privileged Coordinate Systems Across Matched Neural Subpopulations. For all convolutional layers in a pair of ImageNet-trained ResNet- 18 models, we show that a privileged solution basis persists.
Figure A4: Partial Soft-Matching Using a Euclidean Cost Function (A) We visualize the L-curve elbow computed using a Euclidean cost function ( left ), and the corresponding transport plans ( right ). We find that the optimal regularization value s and transport plans are identical to that computed using a cosine similarity cost function. (B) Rank-ordering of neurons from 3 layers (early, middle and late) in an ImageNet-trained ResNet- 18 network using a squared Euclidean cost function. Removing units by using the Euclidean distance yields identical trends as using a cosine similarity cost. (C) We rank-order voxels from a subject pair in NSD using six visual areas by using a squared Euclidean cost function. Ordering voxels using a Euclidean distance cost function also reveals identical trends as using cosine similarity.