Uncertainty-Aware Prototype Learning for Open-Set Test-Time Adaptation
Authors: Jinwu Hu, Langyu Bian, Jian Sun, Dongjin Yang, Chao Shuai, Yufeng Wang, Kai Zhou, Bin Xiao, Mingkui Tan, Hongmin Cai, Wei Luo
Organizations: School of Software Engineering, South China University of Technology, Guangzhou, China · Zhejiang University, Hangzhou, China · School of Future Technology, South China University of Technology, Guangzhou, China · Chongqing University of Posts and Telecommunications, Chongqing, China · School of Software Engineering, South China University of Technology, and also with the Key Laboratory of Big Data and Intelligent Robot, Ministry of Education, Guangzhou, China · South China Agricultural University
Open-set test-time adaptation (TTA) aims to adapt source-trained models to unlabeled test streams where covariate-shifted in-distribution (csID) samples and covariate-shifted out-of-distribution (csOOD) samples may coexist. Existing TTA methods are typically designed under a closed-set assumption and therefore lack explicit mechanisms for reliably distinguishing csID from csOOD samples. Consequently, csOOD samples may be mistakenly included in confidence-enhancing adaptation, producing harmful gradients that degrade known-class performance. To address this, we propose Uncertainty-Aware Prototype Learning for Open-Set Test-Time Adaptation (UAPA), a plug-and-play method for reliable open-set adaptation. Specifically, we first introduce Uncertainty-Driven ID/OOD Separation, which combines a classifier-prototype score with a mini-batch-normalized evidence-derived uncertainty score to distinguish csID from csOOD samples. We then propose a Dynamic Prototype Refinement strategy that periodically refines class-specific target prototypes using buffered csID features to progressively follow the estimated class directions. Lastly, we introduce Contrastive Prototype Entropy Adaptation, which applies entropy minimization to csID samples and entropy maximization to csOOD samples, while prototype-level contrastive learning aligns csID features with their corresponding target prototypes and reduces the maximum similarity between csOOD features and known-class prototypes. Empirically, UAPA outperforms the strongest baselines on the CIFAR benchmarks and Tiny-ImageNet-C, improving average OSCR by 2.08 and 1.28 percentage points, respectively.
Figures & tables
Fig. 1 : OSCR comparison on CIFAR-100-C before and after incorporating UAPA into representative TTA methods [ 18 , 19 , 20 , 21 ] . Values above the arrows denote absolute OSCR gains in percentage points.
Fig. 2 : Overview of UAPA. For each incoming batch Bt , UAPA combines the classifier-prototype score Sproto with the mini-batch-normalized evidence-derived uncertainty score SEU to compute a separation score S(xt) , which partitions samples into csID and csOOD subsets using threshold τ . Contrastive Prototype Entropy Adaptation applies entropy minimization to csID samples, entropy maximization to csOOD samples, and prototype-level contrastive learning to regularize feature representations. Dynamic Prototype Refinement periodically updates class-specific target prototypes using buffered csID features.
Method
CIFAR-10-C
CIFAR-100-C
Average
Acc ↑
AUROC ↑
FPR@95TPR ↓
OSCR ↑
Acc ↑
AUROC ↑
FPR@95TPR ↓
OSCR ↑
Acc ↑
AUROC ↑
FPR@95TPR ↓
OSCR ↑
Source
81.73
77.89
79.47
68.44
53.25
60.55
94.97
39.87
67.49
69.22
87.22
54.16
BN Adapt [ 57 ]
84.20
80.40
76.85
72.13
57.16
72.45
84.29
47.10
70.68
76.43
80.57
59.62
CoTTA [ 58 ]
85.74
85.81
72.77
77.17
56.50
77.06
80.98
49.00
71.12
81.44
76.88
63.09
TENT [ 18 ]
79.12
67.79
95.76
58.27
50.19
59.42
93.54
36.25
64.66
63.61
94.65
47.26
EATA [ 19 ]
76.17
55.43
92.88
46.66
61.34
87.05
56.06
57.27
68.76
71.24
74.47
51.97
TABLE I : Performance comparison on CIFAR-10-C and CIFAR-100-C using a WideResNet-40-2 backbone. ↑ indicates higher is better, ↓ lower is better. Best in bold and second best in underlined . ‘ Ours ’ denotes TENT with our UAPA.
Method
ResNet-50
ViT-Base
Average
Acc ↑
AUROC ↑
FPR@95TPR ↓
OSCR ↑
Acc ↑
AUROC ↑
FPR@95TPR ↓
OSCR ↑
Acc ↑
AUROC ↑
FPR@95TPR ↓
OSCR ↑
Source
22.36
48.11
90.80
15.81
22.09
48.45
94.85
16.29
22.23
48.28
92.83
16.05
BN Adapt [ 57 ]
33.79
56.43
87.80
25.60
-
-
-
-
-
-
-
-
CoTTA [ 58 ]
32.68
47.60
92.27
22.45
16.17
40.99
95.38
10.30
24.43
44.30
93.83
16.38
TENT [ 18 ]
25.45
52.22
94.29
17.48
26.45
52.70
95.13
19.65
25.95
52.46
94.71
18.57
EATA [ 19 ]
34.46
55.86
92.52
25.92
27.95
52.61
95.14
21.31
31.21
54.24
93.83
23.62
TABLE II : Performance comparison on Tiny-ImageNet-C across different backbones (ResNet-50 and ViT-Base). ‘-’ indicates the experiment was not applicable. Best in bold and second best in underlined . ‘ Ours ’ is TENT with our UAPA.
Method
CIFAR-10-C
CIFAR-100-C
Average
Acc ↑
AUROC ↑
FPR@95TPR ↓
OSCR ↑
Acc ↑
AUROC ↑
FPR@95TPR ↓
OSCR ↑
Acc ↑
AUROC ↑
FPR@95TPR ↓
OSCR ↑
TENT
79.12
67.79
95.76
58.27
50.19
59.42
93.54
36.25
64.66
63.61
94.65
47.26
w/o PS
78.46
85.40
36.21
69.22
53.60
90.14
27.18
50.16
66.03
87.77
31.70
59.69
w/o UE
84.87
89.88
45.13
79.10
56.96
94.33
26.07
55.51
70.92
92.11
35.60
67.31
w/o DPR
84.35
93.57
28.10
80.95
59.64
86.75
51.70
55.32
72.00
90.16
39.90
68.14
w/o CL
84.38
93.28
26.05
80.66
60.03
89.58
39.40
56.59
72.21
91.43
32.73
68.63
TABLE III : Ablation experiments on CIFAR-10-C and CIFAR-100-C using a WideResNet-40-2 backbone. ‘PS’ denotes Prototype-based Similarity, ‘UE’ denotes Uncertainty Estimation, ‘DPR’ denotes Dynamic Prototype Refinement strategy, and ‘CL’ denotes Contrastive Learning Optimization.
Method
Acc ↑
AUROC ↑
FPR@95TPR ↓
OSCR ↑
Source
66.81
46.75
93.99
36.13
BN Adapt [ 57 ]
72.70
62.06
91.30
52.15
CoTTA [ 58 ]
72.74
61.65
89.96
51.42
TENT [ 18 ]
73.68
61.16
94.77
51.55
EATA [ 19 ]
73.16
61.34
87.49
50.52
SAR [ 22 ]
73.53
62.01
91.86
52.63
TABLE IV : Results under the small-gap classes setting, where CIFAR-10/CIFAR-100 classes are split into 8:2 (ID: OOD). This setting poses a more challenging scenario.
Method
CIFAR-10-C
CIFAR-100-C
Average
Acc ↑
AUROC ↑
FPR@95TPR ↓
OSCR ↑
Acc ↑
AUROC ↑
FPR@95TPR ↓
OSCR ↑
Acc ↑
AUROC ↑
FPR@95TPR ↓
OSCR ↑
Source
81.73
77.89
79.47
68.44
53.25
60.55
94.97
39.87
67.49
69.22
87.22
54.16
TENT [ 18 ]
79.12
67.79
95.76
58.27
50.19
59.42
93.54
36.25
64.66
63.61
94.65
47.26
Ours (w/ GMM [ 15 ] )
84.45
95.67
18.16
82.12
59.20
92.30
33.64
56.72
71.83
93.99
25.90
69.42
UAPA (Ours)
84.53
96.18
13.57
82.27
60.14
93.26
27.59
58.08
72.34
94.72
20.58
70.18
TABLE V : Performance comparison of different ID/OOD separation strategies on CIFAR-10-C and CIFAR-100-C using a WideResNet-40-2 backbone. “Ours (w/ GMM)” replaces the proposed Uncertainty-Driven ID/OOD Separation with a UniEnt-style GMM gate, while keeping all other components unchanged.
Fig. 3 : t-SNE visualization of CIFAR-10-C and SVHN-C features, where SVHN-C samples are treated as csOOD. Yellow denotes csOOD samples, while the remaining colors denote different csID classes.
Fig. 4 : OSCR comparison on CIFAR-10-C under varying csOOD ratios. Values above UAPA denote its gains in percentage points over the strongest baseline at each ratio.
Method
CIFAR-10-C
CIFAR-100-C
Average
Acc ↑
AUROC ↑
FPR@95TPR ↓
OSCR ↑
Acc ↑
AUROC ↑
FPR@95TPR ↓
OSCR ↑
Acc ↑
AUROC ↑
FPR@95TPR ↓
OSCR ↑
TENT
79.12
67.79
95.76
58.27
50.19
59.42
93.54
36.25
64.66
63.61
94.65
47.26
+ UAPA
84.53
96.18
13.57
82.27
60.14
93.26
27.59
58.08
72.34
94.72
20.58
70.18
EATA
76.17
55.43
92.88
46.66
61.34
87.05
56.06
57.27
68.76
71.24
74.47
51.97
+ UAPA
84.69
92.75
27.24
80.23
61.51
92.16
34.30
59.07
73.10
92.46
30.77
69.65
COME
85.31
75.34
84.01
69.29
58.09
72.28
79.24
47.49
71.70
73.81
81.63
58.39
TABLE VI : Plug-and-play performance of UAPA on CIFAR-10-C, CIFAR-100-C, and their average results.
Method
Acc ↑
AUROC ↑
FPR@95TPR ↓
OSCR ↑
Time (ms) ↓
TENT [ 18 ]
25.45
52.22
94.29
17.48
-
+ UAPA
33.85
67.47
81.06
28.10
+0.22
EATA [ 19 ]
34.46
55.86
92.52
25.92
-
+ UAPA
35.47
56.94
88.72
27.43
+0.26
OSTTA [ 20 ]
34.75
53.00
95.29
25.54
-
+ UAPA
34.87
53.34
92.95
25.68
+0.31
TABLE VII : Results of different methods on Tiny-ImageNet-C with ResNet-50 backbone. We report Accuracy (Acc), AUROC, FPR@95TPR, OSCR, and the per-image additional inference time introduced by UAPA (in milliseconds).
Method
Acc ↑
AUROC ↑
FPR@95TPR ↓
OSCR ↑
Source
22.09
48.45
94.85
16.29
BN Adapt [ 57 ]
22.09
51.96
94.52
16.54
CoTTA [ 58 ]
15.13
42.04
95.45
9.40
TENT [ 18 ]
14.86
47.92
96.05
10.07
EATA [ 19 ]
22.26
51.98
94.64
16.64
SAR [ 22 ]
22.24
51.76
94.58
16.50
TABLE VIII : Results on Tiny-ImageNet-C with ViT-Base under the single-class-per-batch setting.
Fig. 5 : Effect of key hyperparameters on OSCR on Tiny-ImageNet-C with ResNet-50 and ViT-Base backbones. We vary one hyperparameter at a time while keeping the others fixed at their selected values.
Large-scale Vision-Language Models (VLMs) exhibit strong zero-shot recognition, yet their real-world deployment is challenged by distribution shifts. While Test-Time Adaptation (TTA) can mitigate this, existing VLM-based TTA methods operate under a closed-set assumption, failing in open-set scenarios where test streams contain both covariate-shifted in-distribution (csID) and out-of-distribution (csOOD) data. This leads to a critical difficulty: the model must discriminate unknown csOOD samples to avoid interference while simultaneously adapting to known csID classes for accuracy. Current open-set TTA (OSTTA) methods rely on hard thresholds for separation and entropy minimization for adaptation. These strategies are brittle, often misclassifying ambiguous csOOD samples and inducing overconfident predictions, and their parameter-update mechanism is computationally prohibitive for VLMs. To address these limitations, we propose Prototype-based Double-Check Separation (ProtoDCS), a robust framework for OSTTA that effectively separates csID and csOOD samples, enabling safe and efficient adaptation of VLMs to csID data. Our main contributions are: (1) a novel double-check separation mechanism employing probabilistic Gaussian Mixture Model (GMM) verification to replace brittle thresholding; and (2) an evidence-driven adaptation strategy utilizing uncertainty-aware loss and efficient prototype-level updates, mitigating overconfidence and reducing computational overhead. Extensive experiments on CIFAR-10/100-C and Tiny-ImageNet-C demonstrate that ProtoDCS achieves state-of-the-art performance, significantly boosting both known-class accuracy and OOD detection metrics. Code will be available at https://github.com/O-YangF/ProtoDCS.
Wei Luo, Yangfan Ou, Jin Deng +4
South China Agricultural University · Pazhou Laboratory · South China University of Technology +1
Open-set test-time adaptation (TTA) updates models on new data in the presence of input shifts and unknown output classes. While recent methods have made progress on improving in-distribution (InD) accuracy for known classes, their ability to accurately detect out-of-distribution (OOD) unknown classes remains underexplored. We benchmark robust and open-set TTA methods (SAR, OSTTA, UniEnt, and SoTTA) on the standard corruption benchmarks of CIFAR-10-C at the small scale and ImageNet-C at the large scale. For CIFAR-10-C, we use OOD data from SVHN and CIFAR-100 in their respective corrupted forms of SVHN-C and CIFAR-100-C. For ImageNet-C, we use OOD data from ImageNet-O and Textures in their respective corrupted forms of ImageNet-O-C and Textures-C. ImageNet-O is nearer to ImageNet, as unknown but related object classes (like ''garlic bread'' vs. ''hot dog'' for food, or ''highway'' vs. ''dam'' for infrastructure), while Textures is farther from ImageNet, as non-object patterns (like ''cracked'' mud, ''porous'' sponge, ''veined'' leaves). We evaluate the accuracy and confidence of TTA methods for InD vs. OOD recognition on CIFAR-10-C and ImageNet-C. We verify the accuracy of each method's own OOD detection technique on CIFAR-10-C. We also evaluate on ImageNet-C and report both accuracy and standard OOD detection metrics. We further examine more realistic settings, in which the proportions and rates of OOD data can vary. To explore the trade-off between InD recognition and OOD rejection, we propose a new baseline that replaces softmax/multi-class output with sigmoid/multi-label output. Our analysis shows for the first time that current open-set TTA methods struggle to balance InD and OOD accuracy and that they only imperfectly filter OOD data for their own adaptation updates.
Zefeng Li, Evan Shelhamer
University of British Columbia and Vector Institute
Test-Time Adaptation (TTA) aims to mitigate distributional shifts between training and test domains during inference time. However, existing TTA methods fall short in the realistic scenario where models face both continually changing domains and the simultaneous emergence of unknown semantic classes, a challenging setting we term Open-set Continual Test-Time Adaptation (OCTTA). The coupling of domain and semantic shifts often collapses the feature space, severely degrading both classification and out-of-distribution detection. To tackle this, we propose DOmain COmpensation (DOCO), a lightweight and effective framework that robustly performs domain adaptation and OOD detection in a synergistic, closed loop. DOCO first performs dynamic, adaptation-conditioned sample splitting to separate likely ID from OOD samples. Then, using only the ID samples, it learns a domain compensation prompt by aligning feature statistics with the source domain, guided by a structural preservation regularizer that prevents semantic distortion. This learned prompt is then propagated to the OOD samples within the same batch, effectively isolating their semantic novelty for more reliable detection. Extensive experiments on multiple challenging benchmarks demonstrate that DOCO outperforms prior CTTA and OSTTA methods, establishing a new state-of-the-art for the demanding OCTTA setting.
Yingkai Yang, Chaoqi Chen, Hui Huang
College of Computer Science and Software Engineering, Shenzhen University