Signal-Noise Factorization Isolates Nuisance Variation into Removable Subspaces
Authors: Sakin Kirti, Joel Zylberberg
Organizations: Bioinformatics IDP, Jules Stein Eye Institute University of California, Los Angeles Los Angeles, CA 90095, USA · Jules Stein Eye Institute University of California, Los Angeles Los Angeles, CA, 90095, USA
Recent theoretical work identified fundamental properties of representation geometry that shape inference ability of deep neural networks. These include signal-noise factorization (SNF), the ability to segregate signal from noise, and signal-signal factorization (SSF), the ability to segregate task-specific and task-irrelevant signals. Here, we built regularizers that reinforce these two properties during training. We compared networks trained with these regularizers to L2-regularized baseline networks on the CIFAR-100 classification task to understand how our regularizers shape representation geometry and impact performance on a well-known computer vision baseline. Enhancing SNF via regularization improved model performance but enhancing SSF did not. Motivated by biomedical applications, we investigated how our regularizers affected performance on the BloodMNIST dataset treated with MedMNIST-C corruptions at five severity levels, and found even larger performance gains using the SNF regularizer. To understand the mechanism by which SNF-regularization produces improved performance, we analyzed the nuisance subspaces across regularization regimes, finding that the SNF-regularized models represent noise in distinct subspaces, separate from class-relevant signal. Because this geometry is explicit, the dominant corruption-induced directions can be estimated on held-out data and projected out of the representations. This manipulation led to a substantial gain in accuracy. These results show that regularizers that enforce signal-noise factorization can produce substantial improvements on computer vision tasks that contain out-of-distribution image distortions at inference time. They also highlight how shaping representations affects model performance: isolating nuisance variables from categorical ones is more important than maintaining factorized representations of categorical variables.
Figures & tables
Figure 1: Schematic of regularization regimes and their expected effects on representation geometry. (Left) Unregularized networks have entangled representations, with non-orthogonal means and unorganized nuisance variation. (Center) SSF regularization leads to structured representations, with orthogonal class means. (Right) SNF-regularized networks displace nuisance variation into a subspace orthogonal to the class-signal directions, leaving the signal directions unconstrained.
Figure 2: Model geometry and CIFAR-100 performance. A) Theoretical f in the penultimate layer, depicting the degree to which class representations are orthogonal. B) Theoretical s in the penultimate layer, depicting the cleanliness of the signal subspace. C) Model performance on CIFAR-100 dataset, demonstrating that SNF regularization leads to small improvements in model performance. D) Difference in classification accuracy when models trained with weak vs strong augmentations are evaluated on weak vs strong augmentations, demonstrating that factorization-based regularizers improve generalization at greater corruption strengths compared to baseline.
Figure 3: Geometric regularizers applied to corrupted medical imaging data. A) Gallery of one example BloodMNIST image with 11 MedMNIST-C corruptions applied at severity 2. B) Comparison of models trained with L2 , L2 +SSF and L2 +SNF regularizers. Plot shows model accuracy and AUC performance on clean test data (severity 0) and on test data with several different corruption severity levels. C) Difference in accuracy per corruption type for models trained with L2 +SSF vs L2 and L2 +SNF vs L2 regularizers. Red is better performance than L2 , blue is worse performance than L2 .
Figure 4: SNF-regularized models evict nuisance variation into a separable subspace. A) Cumulative explained variance of corruption-induced perturbations. B) Fraction of corruption-induced variance falling inside vs. outside the signal subspace, by regularizer. C) Within- vs. cross-corruption recovery accuracy when the noise subspace is estimated and evaluated on the same test data. D) Test accuracy after projecting out a noise subspace estimated on an independent validation split, demonstrating the subspace generalizes beyond the data used to identify it.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1: Factorization-based regularizer minimization. A) L2 loss term across all training regimes. B) Lf loss term across all training regimes. C) Ls loss term across all training regimes.
Figure A.2: Sensitivity and specificity of BloodMNIST dataset by class.
Figure A.3: Breakdown of performance by corruption type projected out . For each regularization regime, Y shows the corruption type projected out, X shows the corruption type tested on. Each cell demonstrates the performance on each corruption type after a particular corruption representation is projected out.
Augmentation
Weak range
Strong range
brightness
(0.8, 1.25)
(0.5, 2.0)
contrast
(0.8, 1.25)
(0.5, 2.0)
saturation
(0.8, 1.25)
(0.5, 2.0)
hue
(-0.05, +0.05)
(-0.2, +0.2)
Appendix
Table 8
Configuration
Swept
Held fixed
Search range
Selected
Val Acc
SSF
λf
λL2=0
[ 10−5 , 10−2 ], log
10−4
0.696 ± 0.014
SNF
λs
λL2=0
[ 10−4 , 10−1 ], log
10−2
0.716 ± 0.009
L2
λL2
–
[ 10−5 , 10−2 ], log
10−4
0.722 ± 0.007
L2 +SSF
λf
λL2=10−4
[ 10−5 , 10−2 ], log
10−4
0.721 ± 0.005
L2 +SNF
λs
λL2=10−4
[ 10−4 , 10−1 ], log
10−2
0.730 ± 0.005
Appendix
Table A.1: CIFAR-100 hyperparameter search. Details of ranges searched, values held fixed, and selected hyperparameter values according to validation set performance for the CIFAR-100 task.
Configuration
Swept
Held fixed
Search range
Selected
Val Acc
L2
λL2
–
[ 10−6 , 10−2 ], log
10−6
0.984 ± 0.002
L2 +SSF
λf
λL2=10−6
[ 10−5,10−2 ], log
10−3
0.976 ± 0.004
L2 +SNF
λs
λL2=10−6
[ 10−4 , 10−1 ], log
10−1
0.981 ± 0.001
Appendix
Table A.2: BloodMNIST hyperparameter search. Details of ranges searched, values held fixed, and selected hyperparameter values according to validation set performance for the BloodMNIST task.