Test-time adaptation (TTA) adapts pre-trained models to distribution shifts at inference using only unlabeled test data. Under the Practical TTA (PTTA) setting, where test streams are temporally correlated and non-i.i.d., memory has become an indispensable component for stable adaptation, yet existing methods universally store amples in a single unstructured pool. We show that this single-cluster design is fundamentally mismatched to PTTA: a stream clusterability analysis reveals that test streams are inherently multi-modal, with the optimal number of mixture components consistently far exceeding one. To close this structural gap, we propose Multi-Cluster Memory (MCM), a plug-and-play framework that organizes stored samples into multiple clusters using lightweight pixel-level statistical descriptors. MCM introduces three complementary mechanisms: descriptor-based cluster assignment to capture distinct distributional modes, Adjacent Cluster Consolidation (ACC) to bound memory usage by merging the most similar temporally adjacent clusters, and Uniform Cluster Retrieval (UCR) to ensure balanced supervision across all modes during adaptation. Integrated with three contemporary TTA methods on CIFAR-10-C, CIFAR-100-C, ImageNet-C, and DomainNet, MCM achieves consistent improvements across all 12 configurations, with gains up to 5.00% on ImageNet-C and 12.13% on DomainNet. Notably, these gains scale with distributional complexity: larger label spaces with greater multi-modality benefit most from multi-cluster organization. GMM-based memory diagnostics further confirm that MCM maintains near-optimal distributional balance, entropy, and mode coverage, whereas single-cluster memory exhibits persistent imbalance and progressive mode loss. These results establish memory organization as a key design axis for practical test-time adaptation.
Figures & tables
Figure 1: Motivation for multi-cluster memory. (a) Stream clusterability analysis on CIFAR-100-C (PTTA): we fit GMMs with varying K to sliding windows of the test stream and select the optimal K∗ via BIC across three descriptor types. The consistently high K∗ values ( μK∗ = 5.9–9.7) confirm that the target distribution is inherently multi-modal , far exceeding the K=1 assumption of single-cluster memory. (b) Under the same total capacity, SCM samples concentrate around similar regions of the descriptor space, whereas MCM distributes samples across distinct modes, extending coverage to under-represented regions (highlighted in red circle).
Figure 2: Overview of the TTA system with Multi-Cluster Memory (MCM). Incoming samples are assigned to clusters via pixel-level descriptors ( left ). Uniform Cluster Retrieval (UCR) draws balanced samples across all clusters for adaptation ( center ). Adjacent Cluster Consolidation (ACC) merges the closest temporally adjacent pair when capacity is reached ( right ). The three stages jointly preserve the multi-modal structure of the target stream under bounded memory.
Method
Venue
CIFAR10-C
CIFAR100-C
ImageNet-C
DomainNet
Source
–
43.50
46.40
82.00
–
BN
CoRR’20
75.20
52.90
–
–
PL
ICML’13
82.90
88.90
–
–
TENT
ICLR’21
86.00
92.80
–
–
LAME
CVPR’22
39.50
40.50
80.90
–
CoTTA
CVPR’22
83.20
52.20
98.60
–
Table 1: Practical Test-time Adaptation (PTTA) error rates (%) on CIFAR10-C, CIFAR100-C, ImageNet-C, and DomainNet (severity 5). Lower is better. MCM is integrated into three memory-based baselines (RoTTA, PeTTA, ResiTTA) and yields consistent improvements across all configurations, with an average error reduction of 2.96%. Numbers in parentheses indicate improvement over the respective baseline. † denotes our re-implementation. “–” indicates the method was not evaluated on that dataset in the original publication.
Table 4Table 5
Figure 3: Memory scaling comparison on CIFAR-100-C (PTTA). Bars denote error rate; lines denote runtime. For MCM, per-cluster capacity is fixed at 64 and total capacity is varied by the number of clusters. Across all three baselines, simply enlarging the single-cluster pool increases runtime with negligible accuracy gain, whereas MCM consistently achieves lower error at lower cost under equal total capacity. PeTTA with SCM at 256 and 320 samples encountered out-of-memory errors (middle panel).
Figure 4: Diagnostic comparison of memory quality between SCM and MCM over the CIFAR-100-C stream (PTTA, PeTTA). We fit a GMM to the evolving stream and measure three properties of the stored memory: (a) imbalance ratio (lower is better), (b) distributional entropy (higher is more uniform), and (c) cluster coverage (fraction of GMM components with > 1% representation). MCM maintains near-constant balance, entropy, and coverage throughout adaptation, whereas SCM exhibits high variance and progressive degradation.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
τ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Avg
0.1
34.8
34.4
34.5
34.6
34.7
34.6
34.8
34.7
34.7
34.5
34.6
34.4
34.3
34.4
34.2
34.3
34.4
34.3
34.1
34.1
34.5
0.3
33.3
33.1
33.3
33.3
33.4
33.2
33.2
33.2
33.0
33.0
33.0
33.1
33.0
33.0
33.0
32.9
32.9
32.9
32.9
32.8
33.1
0.5
35.1
34.4
34.4
34.5
34.4
34.4
34.4
34.4
34.1
34.2
34.1
34.2
34.0
34.1
34.2
34.3
34.3
34.1
34.0
34.2
34.3
0.7
35.5
34.6
34.9
35.0
35.0
34.9
34.9
34.8
34.9
34.9
34.8
34.9
34.8
34.7
34.9
35.0
34.8
34.8
34.8
34.7
34.9
Appendix
Table S1 : Sensitivity of the distance threshold τ on CIFAR-100-C (recurring TTA, severity 5). Columns 1–20 report per-round classification error rates (%); Avg is the mean over all 20 rounds. All results use PeTTA+MCM. Bold indicates the best value per column.
Distance Metric
Best Threshold ( τ )
Avg. Error (%)
Cosine
0.05
34.27
Mahalanobis
1.0
35.16
Manhattan ( L1 )
0.5
33.90
Euclidean ( L2 ) (Ours)
0.3
33.04
Appendix
Table S2: Ablation on distance metrics for MCM descriptor-space computation. Evaluated on CIFAR-100-C (PTTA, severity 5) with PeTTA+MCM. Each metric is paired with its best threshold from a grid search.
Figure S1 : Sensitivity of Kmax on CIFAR-100-C (PTTA, severity 5) with PeTTA+MCM. Left : average error rate (%) vs. Kmax , showing a U-shaped trend with minimum at Kmax=4 (dashed red line). Right : per-corruption error heatmap (green = lower error). Most corruptions favor moderate Kmax (3–5), while contrast and impulse noise are most sensitive to this parameter.
Method
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Avg
Source
46.5
LAME
40.5
CoTTA
53.4
58.4
63.4
67.6
71.4
74.9
78.2
81.1
84.0
86.7
88.8
90.7
92.3
93.5
94.7
95.6
96.3
97.0
97.3
97.6
83.1
EATA
88.5
95.0
96.8
97.3
97.4
97.2
97.2
97.3
97.4
97.5
97.5
97.5
97.6
97.7
97.7
97.7
97.8
97.8
97.7
97.7
96.9
RMT
50.5
48.6
47.9
47.4
47.3
47.1
46.9
46.9
46.6
46.8
46.7
46.5
46.5
46.6
46.5
46.5
46.5
46.5
46.5
46.5
47.1
MECTA
44.8
44.3
44.6
43.1
44.8
44.2
44.4
43.8
43.8
43.9
44.6
43.8
44.4
44.6
43.9
44.2
43.8
44.4
44.9
44.2
44.2
Appendix
Table S3 : CIFAR-100-C, recurring TTA (severity 5). Per-round classification error rates (%) over 20 successive revisits to the corruption stream. Results use a ResNeXt-29 backbone with RobustBench [ 3 ] preprocessing. Bold = best; underline = second best per column.
Method
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Avg
Source
43.5
LAME
31.1
CoTTA
82.2
85.6
87.2
87.8
88.2
88.5
88.7
88.7
88.9
88.9
88.9
89.2
89.2
89.2
89.1
89.2
89.2
89.1
89.3
89.3
88.3
EATA
81.6
87.0
88.7
88.7
88.9
88.7
88.6
89.0
89.3
89.6
89.5
89.6
89.7
89.7
89.3
89.6
89.6
89.8
89.9
89.4
88.8
RMT
77.5
76.9
76.5
75.8
75.5
75.5
75.4
75.4
75.5
75.3
75.5
75.6
75.5
75.5
75.7
75.6
75.7
75.6
75.7
75.8
75.8
MECTA
72.2
82.0
85.2
86.3
87.0
87.3
87.3
87.5
88.1
88.8
88.9
88.9
88.6
89.1
88.7
88.8
88.5
88.6
88.3
88.8
86.9
Appendix
Table S4 : CIFAR-10-C, recurring TTA (severity 5). Per-round classification error rates (%) over 20 revisits. All methods use a WideResNet-28 backbone from RobustBench [ 3 ] with its official preprocessing.
Figure S2 : Descriptor-space visualization of memory contents under SCM and MCM on CIFAR-100-C (PTTA, severity 5). Grey points denote the incoming test stream; coloured points are stored memory samples (blue for SCM; orange/green/red for distinct MCM clusters). SCM concentrates its budget in a narrow region, yielding high imbalance ratios (imb). MCM spreads samples across distinct clusters, reducing imbalance by 4.1× and 2.5× at t=2220 and t=1400 , respectively.
Test-time adaptation (TTA) enables a pre-trained model to adapt online to an unlabeled test stream under distribution shift. While most TTA research focuses on the adaptation objective, practical streams also depend critically on the memory used to select which test samples drive adaptation. Existing memory mechanisms are usually evaluated as components of specific TTA algorithms, making it difficult to isolate which memory design choices matter and when they matter. In this work, we provide a systematic benchmark that decouples memory from the adaptation algorithm and evaluates memory policies under unified conditions across i.i.d., non-i.i.d., continual, and practical test streams. Our study shows that effective memory management requires more than retaining recent or class-balanced samples. In particular, intra-class diversity is a key factor for avoiding redundant buffers and maintaining representative adaptation signals under temporally correlated and label-skewed streams. Motivated by this finding, we introduce Guided Observational Test-Time Adaptation (GOTTA), a family of diversity-aware memory policies that combine class-balanced allocation with feature-space diversity. GOTTA memories act as drop-in replacements for existing buffers and can be paired with different TTA objectives. Across corruption benchmarks and video-stream settings, diversity-aware memory improves adaptation most clearly under constrained memory budgets and challenging non-i.i.d. streams, while remaining competitive as memory capacity increases. These results highlight memory management as a first-class component of robust test-time adaptation and identify diversity as a central principle for practical TTA.
Conventional test-time adaptation (TTA) approaches typically adapt the model using only a small fraction of test samples, often those with low-entropy predictions, thereby failing to fully leverage the available information in the test distribution. This paper introduces DualTTA, a novel framework that improves performance under distribution shifts by utilizing a larger and more diverse set of test samples. DualTTA identifies two distinct groups: one where the model's predictions are likely consistent with the underlying semantics, and another where predictions are likely incorrect. For the first group, it minimizes prediction entropy to reinforce reliable decisions; for the second, it maximizes entropy to suppress overconfident errors and unlearn spurious behavior. These groups are adaptively selected using a new reliability criterion that measures prediction stability under both semantic-preserving and semantic-altering transformations, addressing the limitations of purely entropy-based selection. We further provide theoretical analysis and empirical justification showing that our approach enables a tighter separation between reliable and unreliable samples, in the context of their suitability for adaptation, leading to provably more effective model updates.
Nam Nguyen Phuong, Duc Nguyen The Minh, Phi Le Nguyen +2
Institute for AI Innovation and Societal Impact, Hanoi University of Science and Technology · Monash University · Adelaide University
Test-time adaptation (TTA) addresses distribution shift using only unlabeled test data. Existing methods typically adapt pretrained models by updating their parameters, limiting both what is adapted and where adaptation can occur within the network. We instead keep the pretrained network frozen and steer its intermediate representations. We introduce SPeaR (Steering Primitive for Realigning Representations), which inserts lightweight learnable modules at stage boundaries and optimizes them directly from the test stream, requiring neither source data nor supervised warm-up. Each primitive is optimized using a gated objective that reduces uncertainty only when adaptation is beneficial, along with a diversity regularizer to prevent collapse, and a multi-depth anchor to stabilize adaptation. We show that steering early representations is the most effective strategy, and that the same primitive transfers across convolutional and Transformer architectures. Across CIFAR-10-C, CIFAR-100-C, and ImageNet-C, SPeaR consistently matches or outperforms methods that adapt orders of magnitude more parameters, remains robust across a wide range of batch sizes, and preserves source-domain performance during continual adaptation.
Muhammad Sudipto Siam Dip, Ali Etemad
Department of Electrical and Computer Engineering Queen’s University, Canada