Subspace Learning with Outlier Exposure for Out-of-Distribution Detection
Authors: Hongxu Shen, Kun Fang, Jiahui Zheng, Jian Sun, Qinghua Tao
Organizations: School of Automation, Beijing Institute of Technology, Beijing, China · The Hong Kong Polytechnic University, Hong Kong, China · Tsinghua University, Beijing, China
Out-of-Distribution (OoD) detection is pivotal for reliable deployments of Deep Neural Networks (DNNs), aiming to distinguish anomalous samples from the In-Distribution (InD) data in training. By introducing outliers as surrogate OoD data to finetune DNNs, Outlier Exposure (OE) has greatly advanced OoD detection in recent years. Existing OE-based research remains predominantly logit-centric, either by generating diverse outliers or by regularizing them with objectives operating at the prediction-output level. Nonetheless, these approaches leave much of the rich information in feature spaces untapped. In this work, we propose an OE-based detection framework, named RecOE, that learns appropriate feature subspaces from InD data and auxiliary outliers, such that the InD-OoD discrepancy can be effectively characterized through the learned feature subspaces. Specifically, the feature subspaces are optimized to capture the dominant variations of InD and outliers by minimizing reconstruction errors therein. Besides, a sample-level objective discourages reconstruction in the opposing subspace, while an orthogonality regularizer reduces subspace overlap, further promoting discriminative subspaces. At inference, the difference in reconstruction errors between the learned subspaces serves as the OoD detection score. Extensive experiments show the superiority of RecOE on multiple benchmarks, in which significant performance gains are achieved over recent strong baselines with only slight extra overhead from two small-size matrices of subspace bases. The source code is available at https://github.com/shenhongxu123/RecOE.
Figures & tables
Figure 1: Experimental exemplification on the effectiveness of RecOE, with CIFAR-10 as InD and iSUN as OoD. We interpolate the distribution of (a) reconstruction errors from a standard PCA on the pretrained DNN and (b) reconstruction contrast score by RecOE.
Figure 2: Illustration of our RecOE for OE-based OoD detection. In training , RecOE seeks separation between InD and outliers through two subspaces spanned by Uin and Uaux , which are optimized not only to capture maximal information of InD data and OoD data respectively (i.e., Lfit ), but also to encode minimal information from the opposing data distribution (i.e., Lsep ), together with an orthogonality term (i.e., Lortho ) to further refine the two subspaces. At inference , the difference of reconstruction errors in the learned two subspaces is naturally utilized as detection scores.
Figure 3: T-SNE visualization of the learned features for InD and OoD data. RecOE achieves a clearer separation between InD (blue) and OoD (red and green) features compared to the Pretrained Model and the OE Model. By regularizing the network with subspace loss term, the network is able to output more distinguishable InD and OoD features. This distinct feature separation inherently facilitates the subsequent OoD detection based on reconstruction errors.
Method
FPR95 ↓
AUROC ↑
ID-ACC ↑
Post-hoc methods
MSP
52.21
90.61
94.84
ODIN
32.84
90.67
94.84
Mahalanobis
36.27
92.69
94.84
Energy
32.82
91.99
94.84
ASH
33.01
92.11
94.84
Table 1: OOD detection results on CIFAR-10 (%). Results are reported as mean ± standard deviation over three trials when available.
Method
FPR95 ↓
AUROC ↑
ID-ACC ↑
Post-hoc methods
MSP
75.23
76.67
68.72
ODIN
67.61
80.36
68.72
Mahalanobis
83.41
58.47
68.72
Energy
71.44
79.52
68.72
ASH
73.51
77.03
68.72
Table 2: OOD detection results on ImageNet-1K (%). Results are reported as mean ± standard deviation over three trials when available.
Method
Far-OoD
Near-OoD
FPR95 ↓
AUROC ↑
FPR95 ↓
AUROC ↑
KPCA (CoRP)
29.66
93.75
67.85
81.33
Mahalanobis++
27.93
94.78
71.16
81.51
PFS
52.23
85.62
68.16
80.92
VPC
49.75
87.10
63.25
82.30
RecOE (ours)
31.46
93.60
41.18
88.51
Table 3: OoD detection results on the detection tasks of far-OoD and near-OoD.
Method
Parameters
Time
FPR95 ↓
AUROC ↑
OE
2.244M
14.10±0.30
13.76±0.23
97.53±0.03
RecOE
2.260M
16.70±1.19
4.36±0.76
98.68±0.24
Table 4: Training cost and detection performance on CIFAR-10 with WideResNet-40-2. RecOE parameters are counted at the target rank. Time (s/epoch) and detection metrics are mean ± standard deviation.
Loss Terms
OoD Detection Results
Lfit
Lsep
Lortho
FPR95 ↓
AUROC ↑
–
✓
✓
14.89
95.83
✓
–
✓
29.19
94.57
✓
✓
–
4.77
98.40
✓
✓
✓
4.36
98.68
Table 5: Leave-one-out ablation of the subspace loss terms on CIFAR-10 (%).
Training
Testing
FPR95 ↓
AUROC ↑
InD-only
InD-only
38.18
90.58
Auxiliary-only
Auxiliary-only
76.61
70.06
Dual-subspace
InD-only
44.24
88.70
Dual-subspace
Auxiliary-only
39.05
88.74
Dual-subspace
Dual-subspace (RecOE)
36.32
91.06
Table 6: Subspace ablation on ImageNet-1K (%). Training and testing columns specify the subspaces used at each stage.
Method
CIFAR-10
ImageNet-1K
FPR95 ↓
AUROC ↑
FPR95 ↓
AUROC ↑
RecOE (fixed rank)
4.51
98.65
38.46
90.72
RecOE
4.36
98.68
36.32
91.06
Table 7: Ablation study of progressive-rank optimization on CIFAR-10 and ImageNet-1K.
Methods
FPR95 ↓
AUROC ↑
ID-ACC ↑
OE
69.17
85.78
77.26
RecOE
63.57
80.78
78.90
Table 8: OoD detection performance using 100 ImageNet-1K classes as InD data and 800 ImageNet-1K classes as surrogate OoD data.
Updated parameters
Time (s/epoch)
Trainable parameters
FPR95 ↓
AUROC ↑
ID-ACC ↑
RecOE ( U )
449
2.10M
46.78
88.18
68.72
RecOE ( h , U )
453
4.15M
44.97
88.55
71.09
RecOE
4351
27.66M
36.32
91.06
75.69
Table 9: Comparisons among variants of RecOE on ImageNet-1K. The trainable parameters and training time (s) per epoch are recorded, together with their detection results and ID-ACC.
Figure 4: Hyperparameter sensitivity analysis. From left to right: subspace-loss weight γ , orthogonality regularization weight β , separation margin m , and subspace dimension k .
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Setting
CIFAR-10
ImageNet-1K
Target rank k
64
512
Initial rank r(0)
32
256
Expansion interval K
500
1000
Initial maximum learning rate ηU
5
10
Minimum learning rate ηmin
10−3
10−4
Rank-dependent decay exponent α
0.5
0.5
Appendix
Table 1: Progressive-rank optimization settings. The expansion interval is measured in optimization steps, and the settings apply to each subspace basis.
Out-of-Distribution (OOD) detection remains a fundamental challenge for neural networks, whose predictions can be overconfident on inputs that deviate from the training distribution. Most post-hoc OOD detection methods derive scores from a single representation level (eg., logits or penultimate features) or combine multiple layers via depth selection or OOD-calibrated weighting. However, because OOD shifts are diverse, the most informative representation level can vary strongly across OOD types and domains, making fixed-layer choices and OOD-calibrated aggregation brittle. In this paper, we propose PRISM (Projected Representation with Intermediate-layer Subspace Modeling), a model-agnostic post-hoc OOD detection method that models a unified multi-layer feature representation rather than aggregating independently scored layers. PRISM fuses intermediate and deep features into a single hierarchical embedding, estimates an in-distribution (ID) principal subspace, and then combines two complementary signals: (i) a class-conditional Mahalanobis distance in the projected subspace and (ii) the residual energy orthogonal to the learned manifold. This simple design avoids OOD-tuned layer weighting while capturing both in-subspace semantic deviations and off-subspace anomalies. Across diverse benchmarks spanning natural images, medical imaging, and industrial visual inspection, PRISM achieves consistent state-of-the-art cross-domain OOD detection performance with a single default configuration across all evaluated domains and architectures. We further show that PRISM incurs minimal inference overhead, making it practical for real-world deployment.
Gerhard Krumpl, Henning Avenhaus, Horst Possegger
Institute of Visual Computing, Graz University of Technology, Austria · KESTRELEYE GmbH, Austria
Out-of-Distribution (OoD) detection is vital for the reliability of deep neural networks, the key of which lies in effectively characterizing the disparities between OoD and In-Distribution (InD) data. In this work, such disparities are exploited through a fresh perspective of non-linear feature subspace. That is, a discriminative non-linear subspace is learned from InD features to capture representative patterns of InD, while informative patterns of OoD features cannot be well captured in such a subspace due to their different distribution. Grounded on this perspective, we exploit the deviations of InD and OoD features in such a non-linear subspace for effective OoD detection. To be specific, we leverage the framework of Kernel Principal Component Analysis (KPCA) to attain the discriminative non-linear subspace and deploy the reconstruction error on such subspace to distinguish InD and OoD data. Two challenges emerge: (i) the learning of an effective non-linear subspace, i.e., the selection of kernel function in KPCA, and (ii) the computation of the kernel matrix with large-scale InD data. For the former, we reveal two vital non-linear patterns that closely relate to the InD-OoD disparity, leading to the establishment of a Cosine-Gaussian kernel for constructing the subspace. For the latter, we introduce two techniques to approximate the Cosine-Gaussian kernel with significantly cheap computations. In particular, our approximation is further tailored by incorporating the InD data confidence, which is demonstrated to promote the learning of discriminative subspaces for OoD data. Our study presents new insights into the non-linear feature subspace for OoD detection and contributes practical explorations on the associated kernel design and efficient computations, yielding a KPCA detection method with distinctively improved efficacy and efficiency.
Kun Fang, Qinghua Tao, Mingzhen He +6
Department of Automation, Shanghai Jiao Tong University · Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University · School of Automation, Beijing Institute of Technology +2
Out-of-distribution (OOD) detection is essential for deploying deep learning models reliably, yet no single method performs consistently across architectures and datasets -- a scorer that leads on one benchmark often falters on another. We attribute this inconsistency to a shared structural limitation: logit-based methods see only the classifier's confidence signal, while feature-based methods attempt to measure membership in the training distribution but do so in the full feature space where confidence and membership are entangled, inheriting architecture-sensitive failure modes. We observe that penultimate features naturally decompose into two orthogonal subspaces: a classifier-aligned component encoding confidence, and a residual the classifier discards. We discover that this residual carries a class-specific directional signature for in-distribution data -- a membership signal invisible to logit-based methods and entangled with noise in feature-based methods. We propose CORE (COnfidence + REsidual), which disentangles the two signals by scoring each subspace independently and combines them via normalized summation. Because the two signals are orthogonal by construction, their failure modes are approximately independent, producing robust detection where either view alone is unreliable. CORE achieves competitive or state-of-the-art performance across five architectures and five benchmark configurations, ranking first in three of five settings and achieving the highest grand average AUROC with negligible computational overhead.