Subspace Learning with Outlier Exposure for Out-of-Distribution Detection
Authors: Hongxu Shen, Kun Fang, Jiahui Zheng, Jian Sun, Qinghua Tao
Organizations: School of Automation, Beijing Institute of Technology, Beijing, China · The Hong Kong Polytechnic University, Hong Kong, China · Tsinghua University, Beijing, China
Out-of-Distribution (OoD) detection is pivotal for reliable deployments of Deep Neural Networks (DNNs), aiming to distinguish anomalous samples from the In-Distribution (InD) data in training. By introducing outliers as surrogate OoD data to finetune DNNs, Outlier Exposure (OE) has greatly advanced OoD detection in recent years. Existing OE-based research remains predominantly logit-centric, either by generating diverse outliers or by regularizing them with objectives operating at the prediction-output level. Nonetheless, these approaches leave much of the rich information in feature spaces untapped. In this work, we propose an OE-based detection framework, named RecOE, that learns appropriate feature subspaces from InD data and auxiliary outliers, such that the InD-OoD discrepancy can be effectively characterized through the learned feature subspaces. Specifically, the feature subspaces are optimized to capture the dominant variations of InD and outliers by minimizing reconstruction errors therein. Besides, a sample-level objective discourages reconstruction in the opposing subspace, while an orthogonality regularizer reduces subspace overlap, further promoting discriminative subspaces. At inference, the difference in reconstruction errors between the learned subspaces serves as the OoD detection score. Extensive experiments show the superiority of RecOE on multiple benchmarks, in which significant performance gains are achieved over recent strong baselines with only slight extra overhead from two small-size matrices of subspace bases. The source code is available at https://github.com/shenhongxu123/RecOE.
Figures & tables
Figure 1: Experimental exemplification on the effectiveness of RecOE, with CIFAR-10 as InD and iSUN as OoD. We interpolate the distribution of (a) reconstruction errors from a standard PCA on the pretrained DNN and (b) reconstruction contrast score by RecOE.
Figure 2: Illustration of our RecOE for OE-based OoD detection. In training , RecOE seeks separation between InD and outliers through two subspaces spanned by Uin and Uaux , which are optimized not only to capture maximal information of InD data and OoD data respectively (i.e., Lfit ), but also to encode minimal information from the opposing data distribution (i.e., Lsep ), together with an orthogonality term (i.e., Lortho ) to further refine the two subspaces. At inference , the difference of reconstruction errors in the learned two subspaces is naturally utilized as detection scores.
Figure 3: T-SNE visualization of the learned features for InD and OoD data. RecOE achieves a clearer separation between InD (blue) and OoD (red and green) features compared to the Pretrained Model and the OE Model. By regularizing the network with subspace loss term, the network is able to output more distinguishable InD and OoD features. This distinct feature separation inherently facilitates the subsequent OoD detection based on reconstruction errors.
Method
FPR95 ↓
AUROC ↑
ID-ACC ↑
Post-hoc methods
MSP
52.21
90.61
94.84
ODIN
32.84
90.67
94.84
Mahalanobis
36.27
92.69
94.84
Energy
32.82
91.99
94.84
ASH
33.01
92.11
94.84
Table 1: OOD detection results on CIFAR-10 (%). Results are reported as mean ± standard deviation over three trials when available.
Method
FPR95 ↓
AUROC ↑
ID-ACC ↑
Post-hoc methods
MSP
75.23
76.67
68.72
ODIN
67.61
80.36
68.72
Mahalanobis
83.41
58.47
68.72
Energy
71.44
79.52
68.72
ASH
73.51
77.03
68.72
Table 2: OOD detection results on ImageNet-1K (%). Results are reported as mean ± standard deviation over three trials when available.
Method
Far-OoD
Near-OoD
FPR95 ↓
AUROC ↑
FPR95 ↓
AUROC ↑
KPCA (CoRP)
29.66
93.75
67.85
81.33
Mahalanobis++
27.93
94.78
71.16
81.51
PFS
52.23
85.62
68.16
80.92
VPC
49.75
87.10
63.25
82.30
RecOE (ours)
31.46
93.60
41.18
88.51
Table 3: OoD detection results on the detection tasks of far-OoD and near-OoD.
Method
Parameters
Time
FPR95 ↓
AUROC ↑
OE
2.244M
14.10±0.30
13.76±0.23
97.53±0.03
RecOE
2.260M
16.70±1.19
4.36±0.76
98.68±0.24
Table 4: Training cost and detection performance on CIFAR-10 with WideResNet-40-2. RecOE parameters are counted at the target rank. Time (s/epoch) and detection metrics are mean ± standard deviation.
Loss Terms
OoD Detection Results
Lfit
Lsep
Lortho
FPR95 ↓
AUROC ↑
–
✓
✓
14.89
95.83
✓
–
✓
29.19
94.57
✓
✓
–
4.77
98.40
✓
✓
✓
4.36
98.68
Table 5: Leave-one-out ablation of the subspace loss terms on CIFAR-10 (%).
Training
Testing
FPR95 ↓
AUROC ↑
InD-only
InD-only
38.18
90.58
Auxiliary-only
Auxiliary-only
76.61
70.06
Dual-subspace
InD-only
44.24
88.70
Dual-subspace
Auxiliary-only
39.05
88.74
Dual-subspace
Dual-subspace (RecOE)
36.32
91.06
Table 6: Subspace ablation on ImageNet-1K (%). Training and testing columns specify the subspaces used at each stage.
Method
CIFAR-10
ImageNet-1K
FPR95 ↓
AUROC ↑
FPR95 ↓
AUROC ↑
RecOE (fixed rank)
4.51
98.65
38.46
90.72
RecOE
4.36
98.68
36.32
91.06
Table 7: Ablation study of progressive-rank optimization on CIFAR-10 and ImageNet-1K.
Methods
FPR95 ↓
AUROC ↑
ID-ACC ↑
OE
69.17
85.78
77.26
RecOE
63.57
80.78
78.90
Table 8: OoD detection performance using 100 ImageNet-1K classes as InD data and 800 ImageNet-1K classes as surrogate OoD data.
Updated parameters
Time (s/epoch)
Trainable parameters
FPR95 ↓
AUROC ↑
ID-ACC ↑
RecOE ( U )
449
2.10M
46.78
88.18
68.72
RecOE ( h , U )
453
4.15M
44.97
88.55
71.09
RecOE
4351
27.66M
36.32
91.06
75.69
Table 9: Comparisons among variants of RecOE on ImageNet-1K. The trainable parameters and training time (s) per epoch are recorded, together with their detection results and ID-ACC.
Figure 4: Hyperparameter sensitivity analysis. From left to right: subspace-loss weight γ , orthogonality regularization weight β , separation margin m , and subspace dimension k .
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Setting
CIFAR-10
ImageNet-1K
Target rank k
64
512
Initial rank r(0)
32
256
Expansion interval K
500
1000
Initial maximum learning rate ηU
5
10
Minimum learning rate ηmin
10−3
10−4
Rank-dependent decay exponent α
0.5
0.5
Appendix
Table 1: Progressive-rank optimization settings. The expansion interval is measured in optimization steps, and the settings apply to each subspace basis.
Department of Automation, Shanghai Jiao Tong University · Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University · School of Automation, Beijing Institute of Technology +2