One Pool Is Not Enough: Multi-Cluster Memory for Practical Test-Time Adaptation

Yu-Wen Tseng; Xingyi Zheng; Ya-Chen Wu; I-Bin Liao; Yung-Hui Li; Hong-Han Shuai; Wen-Huang Cheng

cs.CVMar 22, 2026

One Pool Is Not Enough: Multi-Cluster Memory for Practical Test-Time Adaptation

Authors: Yu-Wen Tseng, Xingyi Zheng, Ya-Chen Wu, I-Bin Liao, Yung-Hui Li, Hong-Han Shuai, Wen-Huang Cheng

Organizations: National Taiwan University, Taiwan · Hon Hai Research Institute, Taiwan · National Yang Ming Chiao Tung University, Taiwan

Abstract

Test-time adaptation (TTA) adapts pre-trained models to distribution shifts at inference using only unlabeled test data. Under the Practical TTA (PTTA) setting, where test streams are temporally correlated and non-i.i.d., memory has become an indispensable component for stable adaptation, yet existing methods universally store amples in a single unstructured pool. We show that this single-cluster design is fundamentally mismatched to PTTA: a stream clusterability analysis reveals that test streams are inherently multi-modal, with the optimal number of mixture components consistently far exceeding one. To close this structural gap, we propose Multi-Cluster Memory (MCM), a plug-and-play framework that organizes stored samples into multiple clusters using lightweight pixel-level statistical descriptors. MCM introduces three complementary mechanisms: descriptor-based cluster assignment to capture distinct distributional modes, Adjacent Cluster Consolidation (ACC) to bound memory usage by merging the most similar temporally adjacent clusters, and Uniform Cluster Retrieval (UCR) to ensure balanced supervision across all modes during adaptation. Integrated with three contemporary TTA methods on CIFAR-10-C, CIFAR-100-C, ImageNet-C, and DomainNet, MCM achieves consistent improvements across all 12 configurations, with gains up to 5.00% on ImageNet-C and 12.13% on DomainNet. Notably, these gains scale with distributional complexity: larger label spaces with greater multi-modality benefit most from multi-cluster organization. GMM-based memory diagnostics further confirm that MCM maintains near-optimal distributional balance, entropy, and mode coverage, whereas single-cluster memory exhibits persistent imbalance and progressive mode loss. These results establish memory organization as a key design axis for practical test-time adaptation.

Figures & tables

Figure 1: Motivation for multi-cluster memory. (a) Stream clusterability analysis on CIFAR-100-C (PTTA): we fit GMMs with varying $K$ to sliding windows of the test stream and select the optimal $K^{*}$ via BIC across three descriptor types. The consistently high $K^{*}$ values ( $\mu_{K^{*}}$ = 5.9–9.7) confirm that the target distribution is inherently multi-modal , far exceeding the $K\!=\!1$ assumption of single-cluster memory. (b) Under the same total capacity, SCM samples concentrate around similar regions of the descriptor space, whereas MCM distributes samples across distinct modes, extending coverage to under-represented regions (highlighted in red circle).

Figure 2: Overview of the TTA system with Multi-Cluster Memory (MCM). Incoming samples are assigned to clusters via pixel-level descriptors ( left ). Uniform Cluster Retrieval (UCR) draws balanced samples across all clusters for adaptation ( center ). Adjacent Cluster Consolidation (ACC) merges the closest temporally adjacent pair when capacity is reached ( right ). The three stages jointly preserve the multi-modal structure of the target stream under bounded memory.

Method	Venue	CIFAR10-C	CIFAR100-C	ImageNet-C	DomainNet
Source	–	43.50	46.40	82.00	–
BN	CoRR’20	75.20	52.90	–	–
PL	ICML’13	82.90	88.90	–	–
TENT	ICLR’21	86.00	92.80	–	–
LAME	CVPR’22	39.50	40.50	80.90	–
CoTTA	CVPR’22	83.20	52.20	98.60	–

Table 1: Practical Test-time Adaptation (PTTA) error rates (%) on CIFAR10-C, CIFAR100-C, ImageNet-C, and DomainNet (severity 5). Lower is better. MCM is integrated into three memory-based baselines (RoTTA, PeTTA, ResiTTA) and yields consistent improvements across all configurations, with an average error reduction of 2.96%. Numbers in parentheses indicate improvement over the respective baseline.

\dagger

denotes our re-implementation. “–” indicates the method was not evaluated on that dataset in the original publication.

Table 4

Table 5

Figure 3: Memory scaling comparison on CIFAR-100-C (PTTA). Bars denote error rate; lines denote runtime. For MCM, per-cluster capacity is fixed at 64 and total capacity is varied by the number of clusters. Across all three baselines, simply enlarging the single-cluster pool increases runtime with negligible accuracy gain, whereas MCM consistently achieves lower error at lower cost under equal total capacity. PeTTA with SCM at 256 and 320 samples encountered out-of-memory errors (middle panel).

Figure 4: Diagnostic comparison of memory quality between SCM and MCM over the CIFAR-100-C stream (PTTA, PeTTA). We fit a GMM to the evolving stream and measure three properties of the stored memory: (a) imbalance ratio (lower is better), (b) distributional entropy (higher is more uniform), and (c) cluster coverage (fraction of GMM components with $>$ 1% representation). MCM maintains near-constant balance, entropy, and coverage throughout adaptation, whereas SCM exhibits high variance and progressive degradation.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

$\tau$	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	Avg
0.1	34.8	34.4	34.5	34.6	34.7	34.6	34.8	34.7	34.7	34.5	34.6	34.4	34.3	34.4	34.2	34.3	34.4	34.3	34.1	34.1	34.5
0.3	33.3	33.1	33.3	33.3	33.4	33.2	33.2	33.2	33.0	33.0	33.0	33.1	33.0	33.0	33.0	32.9	32.9	32.9	32.9	32.8	33.1
0.5	35.1	34.4	34.4	34.5	34.4	34.4	34.4	34.4	34.1	34.2	34.1	34.2	34.0	34.1	34.2	34.3	34.3	34.1	34.0	34.2	34.3
0.7	35.5	34.6	34.9	35.0	35.0	34.9	34.9	34.8	34.9	34.9	34.8	34.9	34.8	34.7	34.9	35.0	34.8	34.8	34.8	34.7	34.9

Appendix

Table S1 : Sensitivity of the distance threshold

\tau

on CIFAR-100-C (recurring TTA, severity 5). Columns 1–20 report per-round classification error rates (%); Avg is the mean over all 20 rounds. All results use PeTTA+MCM. Bold indicates the best value per column.

Distance Metric	Best Threshold ( $\tau$ )	Avg. Error (%)
Cosine	0.05	34.27
Mahalanobis	1.0	35.16
Manhattan ( $L_{1}$ )	0.5	33.90
Euclidean ( $L_{2}$ ) (Ours)	0.3	33.04

Appendix

Table S2: Ablation on distance metrics for MCM descriptor-space computation. Evaluated on CIFAR-100-C (PTTA, severity 5) with PeTTA+MCM. Each metric is paired with its best threshold from a grid search.

Figure S1 : Sensitivity of $K_{\max}$ on CIFAR-100-C (PTTA, severity 5) with PeTTA+MCM. Left : average error rate (%) vs. $K_{\max}$ , showing a U-shaped trend with minimum at $K_{\max}{=}4$ (dashed red line). Right : per-corruption error heatmap (green = lower error). Most corruptions favor moderate $K_{\max}$ (3–5), while contrast and impulse noise are most sensitive to this parameter.

Method	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	Avg
Source	46.5
LAME	40.5
CoTTA	53.4	58.4	63.4	67.6	71.4	74.9	78.2	81.1	84.0	86.7	88.8	90.7	92.3	93.5	94.7	95.6	96.3	97.0	97.3	97.6	83.1
EATA	88.5	95.0	96.8	97.3	97.4	97.2	97.2	97.3	97.4	97.5	97.5	97.5	97.6	97.7	97.7	97.7	97.8	97.8	97.7	97.7	96.9
RMT	50.5	48.6	47.9	47.4	47.3	47.1	46.9	46.9	46.6	46.8	46.7	46.5	46.5	46.6	46.5	46.5	46.5	46.5	46.5	46.5	47.1
MECTA	44.8	44.3	44.6	43.1	44.8	44.2	44.4	43.8	43.8	43.9	44.6	43.8	44.4	44.6	43.9	44.2	43.8	44.4	44.9	44.2	44.2

Appendix

Table S3 : CIFAR-100-C, recurring TTA (severity 5). Per-round classification error rates (%) over 20 successive revisits to the corruption stream. Results use a ResNeXt-29 backbone with RobustBench [ 3 ] preprocessing. Bold = best; underline = second best per column.

Method	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	Avg
Source	43.5
LAME	31.1
CoTTA	82.2	85.6	87.2	87.8	88.2	88.5	88.7	88.7	88.9	88.9	88.9	89.2	89.2	89.2	89.1	89.2	89.2	89.1	89.3	89.3	88.3
EATA	81.6	87.0	88.7	88.7	88.9	88.7	88.6	89.0	89.3	89.6	89.5	89.6	89.7	89.7	89.3	89.6	89.6	89.8	89.9	89.4	88.8
RMT	77.5	76.9	76.5	75.8	75.5	75.5	75.4	75.4	75.5	75.3	75.5	75.6	75.5	75.5	75.7	75.6	75.7	75.6	75.7	75.8	75.8
MECTA	72.2	82.0	85.2	86.3	87.0	87.3	87.3	87.5	88.1	88.8	88.9	88.9	88.6	89.1	88.7	88.8	88.5	88.6	88.3	88.8	86.9

Appendix

Table S4 : CIFAR-10-C, recurring TTA (severity 5). Per-round classification error rates (%) over 20 revisits. All methods use a WideResNet-28 backbone from RobustBench [ 3 ] with its official preprocessing.

Figure S2 : Descriptor-space visualization of memory contents under SCM and MCM on CIFAR-100-C (PTTA, severity 5). Grey points denote the incoming test stream; coloured points are stored memory samples (blue for SCM; orange/green/red for distinct MCM clusters). SCM concentrates its budget in a narrow region, yielding high imbalance ratios (imb). MCM spreads samples across distinct clusters, reducing imbalance by $4.1\!\times$ and $2.5\!\times$ at $t{=}2220$ and $t{=}1400$ , respectively.

Test-Time Adaptation

Test-time adaptation updates or recalibrates pretrained models at inference to handle distribution shifts, often without target labels.

See all 322 papers

GoTTA be Diverse: Rethinking Memory Policies for Test-Time Adaptation
May 19, 2026Shyma Alhuwaider, Yasmeen Alsaedy, Merey Ramazanova +2Test-Time Adaptation Continual Test-Time Adaptation
Center of Excellence in Generative AI, KAUST, Saudi Arabia
Dual Strategies for Test-Time Adaptation
Apr 19, 2026Nam Nguyen Phuong, Duc Nguyen The Minh, Phi Le Nguyen +2Distribution Shift Robustness Test-Time Adaptation
Institute for AI Innovation and Societal Impact, Hanoi University of Science and Technology · Monash University · Adelaide University
SPeaR: Test-Time Adaptation with Steering Primitives for Realigning Representations
Sep 21, 2026Muhammad Sudipto Siam Dip, Ali EtemadTest-Time Adaptation Continual Test-Time Adaptation
Department of Electrical and Computer Engineering Queen’s University, Canada

May 19, 2026J/K move · Enter open · S save

GoTTA be Diverse: Rethinking Memory Policies for Test-Time Adaptation

Shyma Alhuwaider, Yasmeen Alsaedy, Merey Ramazanova, Silvio Giancola+1

Center of Excellence in Generative AI, KAUST, Saudi Arabia

Test-Time Adaptation Continual Test-Time Adaptation

Open paper AskPDF arXiv

Test-time adaptation (TTA) enables a pre-trained model to adapt online to an unlabeled test stream under distribution shift. While most TTA research focuses on the adaptation objective, practical streams also depend critically on the memory used to select which test samples drive adaptation. Existing memory mechanisms are usually evaluated as components of specific TTA algorithms, making it difficult to isolate which memory design choices matter and when they matter. In this work, we provide a systematic benchmark that decouples memory from the adaptation algorithm and evaluates memory policies under unified conditions across i.i.d., non-i.i.d., continual, and practical test streams. Our study shows that effective memory management requires more than retaining recent or class-balanced samples. In particular, intra-class diversity is a key factor for avoiding redundant buffers and maintaining representative adaptation signals under temporally correlated and label-skewed streams. Motivated by this finding, we introduce Guided Observational Test-Time Adaptation (GOTTA), a family of diversity-aware memory policies that combine class-balanced allocation with feature-space diversity. GOTTA memories act as drop-in replacements for existing buffers and can be paired with different TTA objectives. Across corruption benchmarks and video-stream settings, diversity-aware memory improves adaptation most clearly under constrained memory budgets and challenging non-i.i.d. streams, while remaining competitive as memory capacity increases. These results highlight memory management as a first-class component of robust test-time adaptation and identify diversity as a central principle for practical TTA.

Abstract

Test-Time Adaptation

Explore similar work

GoTTA be Diverse: Rethinking Memory Policies for Test-Time Adaptation

Dual Strategies for Test-Time Adaptation

SPeaR: Test-Time Adaptation with Steering Primitives for Realigning Representations