Circumventing Magnitude Collapse via Modular Two-Stage Open-Set Recognition
Authors: Laurin van den Bergh, Halil Bisgin, Andres Palechor, Mike Suter, Manuel Günther
Organizations: Department of Informatics University of Zurich Zurich, Switzerland · Digital Society Initiative University of Zurich Zurich, Switzerland · College of Innovation & Technology University of Michigan-Flint Michigan, USA
Open-set recognition (OSR) requires neural networks to classify known classes while rejecting unknown samples, which is critical for real-world deployment. So far, OSR research studied and developed representation learning and postprocessing methods independently and their interactions remain unexplored. In this paper, we reconsider OSR as a modular two-stage pipeline and present the first systematic study of interactions between representation learning and postprocessing methods across dataset scales and usage of known unknown data. First, we discover a failure mode we term magnitude collapse, where representation learning methods that utilize known unknown data can suffer performance degradation at large scale and irreversibly destroy discriminative information, despite excelling at small scale. Second, we study the modular interactions and reveal when they can be leveraged for modular performance gains via two-stage processing and where interaction effects amplify performance degradation due to magnitude collapse. Third, we show how these findings can be used to achieve state-of-the-art performance with a simple K+1 class classifier baseline and magnitude-aware postprocessing while circumventing magnitude collapse. Finally, our results demonstrate that small-scale evaluations with known unknown data are not predictive of large-scale performance, invalidating current best practices in OSR research. Code available at https://github.com/TwoDigitsOneNumber/Two-Stage-OSR.
Figures & tables
Class
Prediction k∗
Figure 2: Two-Stage Processing Framework for OSR . An image is presented to the backbone network F , which extracts deep features φ that are then processed with a linear layer W to logits z , and further with SoftMax to probabilities y . Solid lines indicate (potentially) learnable connections, while dashed lines highlight non-learnable connections. The postprocessor takes the deep features, logits, or probabilities as input and outputs a class prediction k∗ and a OOD score γ∗ .
Table 1: Key Characteristics . We list key characteristics of 3(a) representation learning f and 3(b) postprocessing g methods for OSR. Representation learning methods highlight which types of known unknown data they use, whether they are Magnitude-Manipulating (MM), and whether they use an additional output node K+1 for the unknown class. For postprocessors, we list whether they require training, are Magnitude-Aware (MA), and which types of inputs they operate on.
Figure 8: Swin Backbone . We show OSR performance on P2 with a Swin-B backbone, showing accuracy, AUROC, and AUOSCR metrics. Similar interaction effects between representation learning and postprocessing methods are observed as with ResNet-50 backbones.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Feature Visualization . We provide 2D feature visualizations of EMNIST experiments with different representation learning methods.
CIFAR+N
ImageNet
Method
Parameter
CIFAR+10
CIFAR+50
P1
P2
P3
f OS
λOS
10−3 / 10−2 / 10−2 / 10−2 / 10−2
10−3 / 10−2 / 10−2 / 10−3 / 10−2
10−3
10−3
10−3
ξ
10/10/10/10/10
20/10/10/10/10
30
20
10
gOpenMax∘fCE
α
3/3/3/1/3
3/2/3/3/3
5
5
2
κ
1.5/1.5/2.3/2/1.7
1.7/1.5/2.3/1.5/2.3
2.3
2.3
2
λ
100/100/250/10/100
250/100/500/100/100
750
750
100
Appendix
Table 2: Hyperparameters . This table include the optimized hyperparameter values for each method and protocol. CIFAR+N results are reported for each trial separately.
Figure 10: CIFAR+N OSR Performance . We show OSR performance for g{}_{\textrm{g}}\circ f{}_{\textrm{f}} combinations across datasets in AUOSCR. The color of each heatmap is normalized independently and centered at the gMSP∘fCE baseline, where blue shows an increase and red a decrease. Single-stage methods are surrounded by a black border. Results depict the average performance ± standard deviation over 5 trials.
Figure 11: ImageNet OSR Performance . We show OSR performance for g{}_{\textrm{g}}\circ f{}_{\textrm{f}} combinations across datasets in AUOSCR. The color of each heatmap is normalized independently and centered at the gMSP∘fCE baseline, where blue shows an increase and red a decrease. Single-stage methods are surrounded by a black border.
Table 3: Small-Scale Evaluation . This table includes a performance overview of all representation learning and postprocessing methods on the small-scale protocols. Metrics include CCR@ ζ for ζ∈10−2,10−1,10−0,101,102 , AUOSCR, AUROC, and OOSA. All scores are reported in percent and CCR@100 is the closed-set accuracy. Performance metrics are reported as mean ± standard deviation over 5 randomized trials. The best performing combination for each protocol and metric w.r.t mean score is highlighted in bold. The best performing postprocessor for each representation learning method and metric w.r.t mean score is highlighted in italic.
Table 4: Large-Scale Evaluation . This table repeats the performance overview of Table 3 on large-scale protocols. — indicates that at no threshold θζ for corresponding FPR ζ could be achieved.
Model
Class 1
Class 9
Knowns Except 1 and 9
min
max
min
max
min
max
CE
94.1
97.6
89.5
91.9
86.7
97.9
ARPL
86.7
96.9
89.5
92.2
83.8
98.0
K+1
67.0
94.8
92.4
95.6
95.2
99.5
OE
65.7
86.6
94.2
97.2
96.7
99.8
OS
58.1
88.7
93.6
95.3
97.7
99.6
Appendix
Table 5: Class-Wise CCR . EMNIST ranges of class-wise CCR are given at the operational threshold, CCRc(θ⋆) , for classes 1, 9, and rest (all other known classes) per representation learning method.
Figure 13: Absolute Performance . These heatmaps show the absolute values for evaluations of 13(a) AUOSCR, 13(b) OOSA, 13(c) AUROC, and 13(d) Accuracy. Each heatmap is normalized independently and centered around CE+MSP, where blue shows an increase and red a decrease. Baseline representation learning and postprocessing methods are surrounded by a black border. Results for CIFAR+N are averaged over 5 trials.
Figure 14: Interaction Effects . This figure shows the interaction effects of representation learning and postprocessing components as correlation between representation learning performance contribution Δf(f∣g) and postprocessor contribution Δg(g∣f) in terms of 14(a) AUOSCR, 14(b) OOSA, 14(c) AUROC, and 14(d) Accuracy. Results for CIFAR+N are averaged over 5 trials.
Figure 15: Regression Analysis . Linear regressions of class-wise CCR at the operational threshold, CCRc(θ∗) , against class-wise average feature magnitude for known classes. Regression is performed for each representation learning method independently and over all postprocessors. For CIFAR+N results are reported for the first trial only.
Open Set Recognition (OSR) aims to enable models to accurately classify known classes while rejecting samples from unseen classes. A key challenge in OSR lies in the inability to model the unbounded distribution of unknown classes during training, often leading to the misclassification of samples from these classes. Rather than modeling unknowns, recent work shapes the feature space so that known classes are compact and well separated, and spherical representation learning methods have achieved strong results this way. Label smoothing has been identified as one of the key drivers of this success, yet it applies the same coefficient to every training sample, regardless of how well each sample is already embedded. We show that the spherical representation learning objectives used in OSR share a single alignment--uniformity structure in which labels enter only through the alignment term. Label smoothing therefore acts as an alignment dial, and a fixed coefficient sets this dial to the same value for every sample. We propose a plug-in, SmoothOperator (SmoothOP), which sets the smoothing coefficient of each sample from its prominence, an embedding-space signal measuring how clearly the sample's own class stands out against its strongest competing class. Our method integrates into four existing spherical representation learning methods at minimal training overhead. SmoothOP assigns strong smoothing to samples with high prominence, which reduces their alignment and relaxes their pull. On the Semantic Shift Benchmark, SmoothOP-augmented variants generally outperform their base objectives across datasets, degrees of semantic shift, and OSR post-processors, with gains of up to 4.7% in AUROC, OSCR, and closed-set accuracy.
Open-set recognition systems face a neglected failure mode: high-confidence near-known unknowns, which lie outside the known label set but are close enough to known classes that a closed-set classifier accepts them with high confidence. We show that this failure is widespread across scalar-threshold methods, including recent post-hoc detectors, and that stronger encoders can amplify rather than remove the risk. We propose EGUR-A, which changes the decision from is this sample's score high enough?'' to does this predicted known class have sufficient evidence to accept this sample?'' EGUR-A combines class-conditional local acceptance evidence with global residual evidence, and selects their relative weight from known-sample statistics without unknown validation data. Across CUB, FGVC-Aircraft, and ImageNet-hard, EGUR-A substantially reduces high-confidence false known acceptance at matched known-rejection operating points. The result is not a stronger threshold; it is a different question: whether a known class is entitled to accept a sample.
Xi Chen, Yingjun Xiao, Gang Fang
School of Computer Science and Cyber Engineering, Guangzhou University, Guangzhou, China · School of Computer Science and Cyber Engineering, Guangzhou University, Guangzhou 510006, China · School of Artificial Intelligence, Guangzhou University, Guangzhou, China +3
Open-set recognition refers to the problem in which classes that were not seen during training appear at inference time. This requires the ability to identify instances of novel classes while maintaining discriminative capability for closed-set classification. OpenMax was the first deep neural network-based approach to address open-set recognition by calibrating the predictive scores of a standard closed-set classification network. In this paper we present MetaMax, a more effective post-processing technique that improves upon contemporary methods by directly modeling class activation vectors. MetaMax removes the need for computing class mean activation vectors (MAVs) and distances between a query image and a class MAV as required in OpenMax. Experimental results show that MetaMax outperforms OpenMax and is comparable in performance to other state-of-the-art approaches.
Zongyao Lyu, Nolan B. Gutierrez, William J. Beksi
The University of Texas at Arlington Arlington, TX, USA