cs.CVSep 6, 2026
SaveUnified Multi-Layer Subspace Modeling for Cross-Domain OOD Detection
Organizations: Institute of Visual Computing, Graz University of Technology, Austria · KESTRELEYE GmbH, Austria
Abstract
Out-of-Distribution (OOD) detection remains a fundamental challenge for neural networks, whose predictions can be overconfident on inputs that deviate from the training distribution. Most post-hoc OOD detection methods derive scores from a single representation level (eg., logits or penultimate features) or combine multiple layers via depth selection or OOD-calibrated weighting. However, because OOD shifts are diverse, the most informative representation level can vary strongly across OOD types and domains, making fixed-layer choices and OOD-calibrated aggregation brittle. In this paper, we propose PRISM (Projected Representation with Intermediate-layer Subspace Modeling), a model-agnostic post-hoc OOD detection method that models a unified multi-layer feature representation rather than aggregating independently scored layers. PRISM fuses intermediate and deep features into a single hierarchical embedding, estimates an in-distribution (ID) principal subspace, and then combines two complementary signals: (i) a class-conditional Mahalanobis distance in the projected subspace and (ii) the residual energy orthogonal to the learned manifold. This simple design avoids OOD-tuned layer weighting while capturing both in-subspace semantic deviations and off-subspace anomalies. Across diverse benchmarks spanning natural images, medical imaging, and industrial visual inspection, PRISM achieves consistent state-of-the-art cross-domain OOD detection performance with a single default configuration across all evaluated domains and architectures. We further show that PRISM incurs minimal inference overhead, making it practical for real-world deployment.
Figures & tables
Figure 1: Comprehensive evaluation of post-hoc OOD detection across natural, medical, and industrial benchmarks. (a) Mean AUROC ( ) and FPR95 ( ) averaged over all datasets, OOD splits, and architectures. (b) Cross-domain correlation between ImageNet-1k [ 14 ] (OpenOOD [ 83 ] ) and OpenMIBOOD [ 19 ] performance (AUROC). Our PRISM ( orange ) consistently exhibits strong performance across domains.
Figure 2: Comparison of OOD score distribution across layer depths. Histograms represent data sampled from PhaKIR (ID) and Cholec80 (Near-OOD) using a ResNet-18. (a) MDS++ applied to the first ResNet-18 block (shallow features). (b) MDS++ applied to the penultimate layer (deep features). (c) PRISM integrates the full architectural hierarchy (shallow to deep features).
Figure 3: Sensitivity of layer aggregation to OOD validation data. For three benchmark tasks: (a) MIDOG, (b) PhaKIR, and (c) ImageNet-1k, we fit linear aggregation weights over ResNet blocks 1–4 using one OOD dataset as validation and evaluate the resulting model across all OOD test sets. The stacked bar plots (top) show the normalized layer weights learned for each calibration dataset, while the bars (bottom) indicate the corresponding mean FPR95 averaged over all OOD datasets.
| Method | OpenOOD | OpenMIBOOD | ICONIC-444 | Average | ||||||||||
| C10 | C100 | IN-1k | Mean | MIDOG | PhaKIR | OASIS-3 | Mean | Almond | Wheat | Kernels | FG | Mean | ||
| MSP [ 27 ] | ||||||||||||||
| MDS [ 40 ] | ||||||||||||||
| MDSEns* [ 40 ] | 96.87 | 98.49 | 98.41 | |||||||||||
| GRAM [ 61 ] | 95.89 | 97.55 | ||||||||||||
| EBO [ 45 ] | ||||||||||||||
Table 1: AUROC ( ) performance across diverse imaging domains. Comparison of post-hoc OOD detection methods on natural (OpenOOD), medical (OpenMIBOOD), and industrial (ICONIC-444) benchmarks. Values represent the mean AUROC ( ) across the benchmark-specific OOD test sets. Standard backbones are employed: ResNet-18 for CIFAR-10/100 (C10, C100), PhaKIR, and ICONIC-444; ResNet-50 for ImageNet-1k (IN-1k) and MIDOG; and R(2+1)D for OASIS-3. The best and second-best results in each column are highlighted in bold and underlined, respectively. * Results of MDSEns should be interpreted with caution, as discussed in Sec. 3.2 and in prior benchmark analyses [ 19 ] .
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | ID Dataset | Near-OOD | Far-OOD | Extreme-OOD |
| OpenOOD [ 83 ] | CIFAR-10 [ 34 ] | CIFAR-100 [ 34 ] , Tiny ImageNet [ 69 ] | MNIST [ 39 ] , SVHN [ 53 ] , Textures [ 13 ] , Places365 [ 85 ] | MIDOG [ 7 ] , ICONIC-444 [ 36 ] |
| CIFAR-100 [ 34 ] | CIFAR-10 [ 34 ] , Tiny ImageNet [ 69 ] | MNIST [ 39 ] , SVHN [ 53 ] , Textures [ 13 ] , Places365 [ 85 ] | MIDOG [ 7 ] , ICONIC-444 [ 36 ] | |
| ImageNet-1k [ 14 ] | SSB-Hard [ 73 ] , NINCO [ 11 ] | iNaturalist [ 72 ] , Textures [ 13 ] , OpenImage-O [ 74 ] | MNIST [ 39 ] , FMNIST [ 76 ] | |
| OpenMIBOOD [ 19 ] | MIDOG [ 7 ] | other MIDOG domain shifts not in ID | CCAgT [ 3 , 6 ] , FNAC 2019 [ 60 ] | PhaKIR [ 59 ] , ImageNet [ 14 ] , iNaturalist [ 72 ] , Textures [ 13 ] |
| PhaKIR [ 59 ] | Cholec80 [ 71 ] , EndoSeg15 [ 12 ] , EndoSeg18 [ 2 ] | Kvasir [ 30 ] , CATARACTS [ 1 ] | MIDOG [ 7 ] , ImageNet [ 14 ] , iNaturalist [ 72 ] , Textures [ 13 ] | |
| OASIS-3 [ 38 ] | ATLAS [ 42 ] , BraTS [ 8 , 9 , 49 ] , OASIS-CT [ 38 ] | MDS-Heart [ 5 , 62 , 68 ] , CHAOS [ 33 , 32 ] | MIDOG [ 7 ] , MNIST [ 39 ] , FMNIST [ 76 ] , Textures [ 13 ] |
Table 4: Detailed benchmark configuration. For each benchmark and in-distribution (ID) dataset, we list the corresponding near-, far-, and extreme-OOD evaluation sets.
| Benchmark | ID Dataset / Task | Architecture | Checkpoint Source / Training |
| OpenOOD [ 83 ] | CIFAR-10 [ 34 ] | ResNet-18 [ 23 ] | Official OpenOOD checkpoint |
| CIFAR-100 [ 34 ] | ResNet-18 [ 23 ] | Official OpenOOD checkpoint | |
| ImageNet-1k [ 14 ] | ResNet-50 [ 23 ] | timm: resnet50.tv2_in1k | |
| Swin-T [ 47 ] | torchvision: Swin_T_Weights.IMAGENET1K_V1 | ||
| ViT-B/16 [ 22 ] | timm: vit_base_patch16_224.mae_in1k | ||
| OpenMIBOOD [ 19 ] | MIDOG [ 7 ] | ResNet-50 [ 23 ] | Official OpenMIBOOD checkpoint |
Table 5: Overview of architectures and their corresponding official checkpoint sources or training protocol across all benchmarks.
| Method | Cal. data | Hyperparameter search space | ||||
| ID | OOD | Param 1 | Range | Param 2 | Range | |
| MSP [ 27 ] | — | — | — | — | ||
| MDS [ 40 ] | ✓ | — | — | — | — | |
| MDSEns [ 40 ] | ✓ | ✓ | noise | [0, 0.0025, 0.0014, 0.005] | — | — |
| GRAM [ 61 ] | ✓ | — | — | — | — | |
| EBO [ 45 ] | temperature | [0.1, 0.5, 1.0, 1.5, 2.0] | — | — | ||
Table 6: Hyperparameter search space and calibration requirements for the considered OOD detection method.
| Method | Near | Far | Extreme | |||||||||||||||
| CIFAR-100 | Tiny ImageNet | MNIST | Places365 | SVHN | Textures | ICONIC-444 | MIDOG | Average | ||||||||||
| AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | |
| MSP [ 27 ] | ||||||||||||||||||
| MDS [ 40 ] | ||||||||||||||||||
| MDSEns [ 40 ] | 100.00 | 0.00 | ||||||||||||||||
Table 7: Performance of OOD detection methods for a ResNet-18 backbone trained on CIFAR-10.
| Method | Near | Far | Extreme | |||||||||||||||
| CIFAR-10 | Tiny ImageNet | MNIST | Places365 | SVHN | Textures | ICONIC-444 | MIDOG | Average | ||||||||||
| AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | |
| MSP [ 27 ] | ||||||||||||||||||
| MDS [ 40 ] | ||||||||||||||||||
| MDSEns [ 40 ] | ||||||||||||||||||
Table 8: Performance of OOD detection methods for a ResNet-18 backbone trained on CIFAR-100.
| Method | Near | Far | Extreme | ||||||||||||||
| SSB-Hard | NINCO | iNaturalist | Textures | OpenImage-O | MNIST | FashionMNIST | Average | ||||||||||
| AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | ||
| ResNet-50 | MSP [ 27 ] | ||||||||||||||||
| MDS [ 40 ] | 95.25 | 26.25 | 92.14 | ||||||||||||||
| MDSEns [ 40 ] | |||||||||||||||||
Table 9: Performance of OOD detection methods on ImageNet-1k using three backbones: ResNet-50, ViT-B/16, and Swin-T. Metrics are reported in (higher is better for AUROC; lower is better for FPR95, indicated by ). The best and second-best results for each OOD dataset ( i.e . , each column) are shown in bold or underlined, respectively.
| Method | Near | Far | Extreme | ||||||||||||||
| Mean | Mean | ImageNet-1k | iNaturalist | Textures | Places365 | Mean | Average | ||||||||||
| AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | ||
| ResNet-18 | MSP [ 27 ] | ||||||||||||||||
| MDS [ 40 ] | |||||||||||||||||
| MDSEns [ 40 ] | |||||||||||||||||
Table 10: Average OOD detection performance per method and OOD test set using three backbones: Resnet-18, CCT-7/7x2, and ConvNeXt-S. Results are reported as the mean over the four tasks (Almond, Wheat, Kernels, and Food-grade). Arrows ( / ) indicate whether higher or lower values are better. Additionally, the best and second-best results in each column are highlighted in bold and underlined, respectively. All values are reported as percentages.
Figure 6: Mean-rank comparison (AUROC) across OOD datasets. Average rank of OOD detection methods computed over OOD datasets. Error bars show CIs; lower rank is better.
Figure 7: Qualitative analysis for OpenOOD (ImageNet-1k / SSB-hard). For each column, the top row shows an OOD sample from the SSB-hard split, with its original OOD class label annotated in blue (parentheses). The two rows below show two ID ImageNet examples from the ID class predicted by the Mahalanobis component of our detector ( i.e . , the class achieving the minimum Mahalanobis distance in PCA space), with the corresponding predicted ID class label annotated in red .
Figure 8: Qualitative analysis for OpenOOD (ImageNet-1k / Textures). For each column, the top row shows an OOD sample from the Textures split, with its original OOD class label annotated in blue (parentheses). The two rows below show two ID ImageNet examples from the ID class predicted by the Mahalanobis component of our detector ( i.e . , the class achieving the minimum Mahalanobis distance in PCA space), with the corresponding predicted ID class label annotated in red .
Figure 9: Qualitative analysis for OpenOOD (ImageNet-1k / NINCO). For each column, the top row shows an OOD sample from the NINCO split, with its original OOD class label annotated in blue (parentheses). The two rows below show two ID ImageNet examples from the ID class predicted by the Mahalanobis component of our detector ( i.e . , the class achieving the minimum Mahalanobis distance in PCA space), with the corresponding predicted ID class label annotated in red .
Figure 10: Qualitative analysis on OpenMIBOOD/PhaKIR. For each column, the top row shows an OOD sample annotated with the source OOD dataset ( blue ). The two rows below show two ID examples from the Mahalanobis-predicted ID class ( red ; argmin Mahalanobis distance in PCA space), illustrating which ID category each OOD sample is mapped to.
Figure 11: ICONIC-444/almond qualitative examples. For each column, the top row shows an OOD sample with its OOD class label ( blue ), and the two rows below show two ID examples from the Mahalanobis-predicted ID class ( red ; argmin Mahalanobis distance in PCA space).
| Method | MIDOG | PhaKIR | OASIS-3 | |||||||||||||||||||||||
| Near | Far | Extreme | Mean | Near | Far | Extreme | Mean | Near | Far | Extreme | Mean | Average | ||||||||||||||
| AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | |
| MSP [ 27 ] | ||||||||||||||||||||||||||
| MDS [ 40 ] | 100.00 | 0.00 | 100.00 | 0.00 | ||||||||||||||||||||||
| MDSEns* [ 40 ] | 90.60 | 21.83 | 100.00 | 0.00 | 100.00 | 0.00 | 96.87 | 7.28 | 97.05 | 21.24 | 98.41 | 15.85 | 100.00 | 0.00 | 98.49 | 12.36 | 100.00 | 0.00 | 100.00 | 0.00 | 98.41 | 6.80 | ||||
Table 11: Average OOD detection performance for the OpenMIBOOD benchmark, measured as FPR at TPR, reported per method and OOD category (near, far, and extreme. Arrows ( / ) indicate whether higher or lower values are better. Additionally, the best and second-best results in each column are highlighted in bold and underlined, respectively. All values are reported as percentages. *: Results of MDSEns should be interpreted with caution, as discussed in the main manuscript and in prior benchmark analyses [ 19 ] .
More in this topic
OOD Detection
Out-of-distribution (OOD) detection identifies inputs unlike a model’s training data so systems can flag uncertainty, reject inputs, or adapt safely.
See all 215 papers