Not Every Correction Helps: Gain-Guided Continual Test-Time Adaptation

Youjia Zhang; Huiling Liu; Soyun Choi; Jaehong Yoon; Sungeun Hong

cs.CVSep 29, 2026

Not Every Correction Helps: Gain-Guided Continual Test-Time Adaptation

Authors: Youjia Zhang, Huiling Liu, Soyun Choi, Jaehong Yoon, Sungeun Hong

Organizations: Sungkyunkwan University · Nanyang Technological University

Abstract

Continual test-time adaptation (CTTA) adapts a source model to an unlabeled test stream whose distribution may change over time. Existing TTA methods often assess prediction reliability using confidence or entropy, which primarily reflect the model's self-certainty for the current sample. In CTTA, accumulated target observations can provide complementary evidence for correcting the source prediction, but this history may become misaligned as the target distribution changes. The key question is therefore not how much the correction differs from the source prediction, but whether and how strongly it should be applied. This paper proposes Gain-Aware INtervention (GAIN), a backpropagation-free CTTA framework guided by a simple principle: history proposes, gain decides. GAIN maintains compact target statistics to form a correction proposal and a posterior-predictive evaluator that accounts for estimation uncertainty. The resulting source-relative gain estimates the proposal's benefit and determines a sample-specific intervention strength along a continuous path through efficient one-dimensional optimization. Gain-controlled predictions then update the target statistics online, limiting the propagation of unreliable corrections, all without backpropagation, sample storage, or replay. Across five benchmarks, our method achieves strong predictive performance, with favorable accuracy--calibration--efficiency trade-offs in continual adaptation. On ImageNet-C, for example, GAIN achieves 61.9% accuracy with near-source calibration. It remains stable under diverse and challenging continual shifts while running 15.9x faster than a representative optimization-based CTTA baseline.

Figures & tables

Figure 1: Gain-Guided Sample-wise Intervention. Past target batches are summarized by compact statistics to construct a history-based correction proposal (solid) and an uncertainty-aware posterior-predictive estimate (dashed) that evaluates it. The resulting source-relative gain determines a sample-specific strength along the source-to-correction path: unsupported corrections are rejected (Sample A), partially supported corrections are applied conservatively (Sample B), and well-supported corrections are fully adopted (Sample C).

Figure 2: Overview of GAIN : history proposes, gain decides. (i) The frozen source model produces the source prediction $\mathbf{s}_{t}$ and current feature $\mathbf{z}_{t}$ , while the retained target state $\mathcal{S}_{t-1}$ summarizes past batches. (ii) Together, $\mathbf{z}_{t}$ and $\mathcal{S}_{t-1}$ form the target proposal $\hat{\mathbf{q}}_{t}^{H}$ and predictive evaluator $\bar{\mathbf{q}}_{t}^{H}$ . The evaluator accounts for uncertainty to estimate the proposal’s source-relative gain $G_{t}^{\mathrm{pp}}$ . (iii) This source-relative gain determines the intervention strength $\lambda_{t}^{\star}$ along the continuous evidence path from $\mathbf{s}_{t}$ to $\hat{\mathbf{q}}_{t}^{H}$ . The final prediction $\mathbf{p}_{t}^{\star}$ then updates the target statistics $\mathcal{S}_{t}$ for future batches.

Method	BP-free	Metric	Noise			Blur				Weather				Digital				Avg.
Method	BP-free	Metric	Gauss.	Shot	Impu.	Defo.	Glas.	Moti.	Zoom	Snow	Fros.	Fog	Brig.	Cont.	Elas.	Pix.	JPEG	Avg.
		Acc. $\uparrow$	47.0	48.2	47.9	31.5	21.2	41.5	36.7	50.1	45.8	42.3	73.6	8.6	42.5	62.0	63.8	44.2
Source	–	ECE $\downarrow$	3.6	4.1	3.7	4.3	5.4	3.9	9.1	2.3	4.9	17.4	3.1	3.9	9.0	3.3	2.7	5.4
Tent ( ICLR 2021 )	✗	Acc. $\uparrow$	47.8	51.1	50.8	34.2	27.0	45.5	41.6	56.0	52.3	49.7	76.1	27.2	44.3	65.6	66.1	49.0
Tent ( ICLR 2021 )	✗	ECE $\downarrow$	5.8	8.3	10.5	12.1	18.0	13.9	18.2	12.4	14.6	13.3	7.0	19.1	20.2	10.3	8.7	12.8
CoTTA ( CVPR 2022 )	✗	Acc. $\uparrow$	47.1	48.4	48.6	31.7	21.9	42.9	38.0	51.8	47.3	44.7	74.1	10.0	43.6	63.6	64.8	45.2

Table 1: CSC results on ImageNet-C. Accuracy (Acc., %) and expected calibration error (ECE, %) with ViT-Base at severity level 5. BP-free denotes backpropagation-free adaptation. Bold indicates the best results; Source is shown for reference only.

Panel of Figure 3 — Figure 3: Accuracy and calibration under three challenging CTTA scenarios on ImageNet-C. (a) CDC evaluates recurring corruptions with irregular durations. (b) MDS interleaves samples from multiple corruption domains, with results averaged over severity levels 1–5. (c) LHA evaluates error accumulation and long-term stability over 10 repeated corruption cycles.

Figure 4: Efficiency–accuracy trade-off on ImageNet-3DCC. Inference speed is normalized to CoTTA; bubble size indicates ECE.

Table 6

Panel of Figure 5 — Figure 5: Sensitivity analysis. We study the sensitivity of GAIN to (a) the source-centered prior strength $\kappa_{0}$ and (b) the test-time batch size. For each setting, we report both accuracy and ECE.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Method	BP-free	Metric	Noise			Blur				Weather				Digital				Avg.
Method	BP-free	Metric	Gauss.	Shot	Impu.	Defo.	Glas.	Moti.	Zoom	Snow	Fros.	Fog	Brig.	Cont.	Elas.	Pix.	JPEG	Avg.
		Acc. $\uparrow$	47.0	48.2	47.9	31.5	21.2	41.5	36.7	50.1	45.8	42.3	73.6	8.6	42.5	62.0	63.8	44.2
Source	–	ECE $\downarrow$	3.6	4.1	3.7	4.3	5.4	3.9	9.1	2.3	4.9	17.4	3.1	3.9	9.0	3.3	2.7	5.4
CoTTA ( CVPR 2022 )	✗	Acc. $\uparrow$	46.1	47.4	47.5	33.3	22.3	42.7	38.6	50.3	46.1	44.2	74.0	6.6	43.3	62.4	64.6	44.6
CoTTA ( CVPR 2022 )	✗	ECE $\downarrow$	7.0	6.9	10.3	4.1	12.1	8.4	17.3	5.5	8.9	10.0	5.2	2.2	12.8	5.7	4.1	8.0
SAR ( ICLR 2023 )	✗	Acc. $\uparrow$	53.7	57.5	56.5	51.4	45.7	56.8	48.1	61.2	57.9	53.7	77.3	41.6	50.8	66.7	66.5	56.4

Appendix

Table 4: CDC results on ImageNet-C. Accuracy (Acc., %) and expected calibration error (ECE, %) with ViT-Base at severity level 5. BP-free denotes backpropagation-free adaptation. Bold indicates the best results; Source is shown for reference only. Our results are averaged over five runs.

Method	BP-free	Metric	Level 5	Level 4	Level 3	Level 2	Level 1	Avg.
		Acc. $\uparrow$	44.2	55.4	68.3	69.2	74.8	62.4
Source	–	ECE $\downarrow$	3.4	4.3	4.2	3.3	2.8	3.6
CoTTA ( CVPR 2022 )	✗	Acc. $\uparrow$	50.3	62.0	69.4	73.9	78.2	66.8
CoTTA ( CVPR 2022 )	✗	ECE $\downarrow$	5.4	4.6	3.9	3.9	3.4	4.2
SAR ( ICLR 2023 )	✗	Acc. $\uparrow$	56.1	64.9	70.9	74.0	78.3	68.8
SAR ( ICLR 2023 )	✗	ECE $\downarrow$	8.5	5.9	4.4	3.8	3.2	5.2

Appendix

Table 5: MDS results on ImageNet-C. Accuracy (Acc.,%) and expected calibration error (ECE,%) across corruption severity levels 5–1. BP-free denotes backpropagation-free adaptation. Bold indicates the best results; Source is shown for reference only. Our results are averaged over five runs.

Method	BP-free	Metric	R1	R2	R3	R4	R5	R6	R7	R8	R9	R10	Avg.
		Acc. $\uparrow$	44.2	44.2	44.2	44.2	44.2	44.2	44.2	44.2	44.2	44.2	44.2
Source	–	ECE $\downarrow$	5.4	5.4	5.4	5.4	5.4	5.4	5.4	5.4	5.4	5.4	5.4
CoTTA ( CVPR 2022 )	✗	Acc. $\uparrow$	45.2	45.3	45.9	46.3	46.6	46.4	46.0	45.7	45.4	45.3	45.8
CoTTA ( CVPR 2022 )	✗	ECE $\downarrow$	7.5	15.8	22.3	26.4	28.1	30.7	33.1	34.7	36.2	36.9	27.2
ViDA ( ICLR 2024 )	✗	Acc. $\uparrow$	56.6	57.3	57.3	57.4	57.5	57.5	57.3	54.1	54.8	54.9	56.5
ViDA ( ICLR 2024 )	✗	ECE $\downarrow$	14.7	21.0	24.1	26.3	28.0	29.3	29.9	31.1	31.9	32.9	26.9

Appendix

Table 6: LHA results on ImageNet-C. Accuracy (Acc., %) and expected calibration error (ECE, %) over 10 repeated corruption cycles with ViT-Base at severity 5. BP-free denotes backpropagation-free adaptation. Bold indicates the best results; Source is shown for reference only.

Variant	Proposal	Gain evaluation	Intervention	Acc. $\uparrow$	ECE $\downarrow$	NLL $\downarrow$
Proposal as Prediction	$\hat{\mathbf{q}}^{H}$	–	–	58.6	11.2	2.5
Evaluator as Prediction	$\bar{\mathbf{q}}^{H}$	–	–	58.9	12.8	2.4
Swapped Roles	$\bar{\mathbf{q}}^{H}$	$\hat{\mathbf{q}}^{H}$	$\tilde{\lambda}^{\star}$	61.7	8.0	2.0
GAIN (Ours)	$\hat{\mathbf{q}}^{H}$	$\bar{\mathbf{q}}^{H}$	$\lambda^{\star}$	61.9	6.0	1.9

Appendix

Table 7: Ablation of the proposal–evaluator roles under CSC. We compare direct target-side prediction, reversed proposal–evaluation roles, and the asymmetric design used by GAIN.

Figure 6: Fixed vs. adaptive intervention strength. We compare fixed

\lambda\in[0,1]

with the proposed sample-wise adaptive

\lambda^{\star}

. No single nonzero fixed intervention matches the accuracy–calibration trade-off of GAIN : the best fixed accuracy is 61.0%, while the lowest nonzero fixed ECE is 8.9%. The adaptive

\lambda^{\star}

achieves 61.9% accuracy with 6.0% ECE.

Variant	Assignment	Source Support $\bm{\zeta}$	Class Prior	CSC
				Acc. $\uparrow$	ECE $\downarrow$	NLL $\downarrow$
State Assignment
Source Assignment	$\mathbf{s}_{t}$	–	RB	58.5	13.7	2.2
		✓	RB	60.8	7.2	2.0
Proposal Assignment	$\hat{\mathbf{q}}_{t}^{H}$	–	RB	59.6	12.2	2.2
		✓	RB	61.6	6.2	2.0

Appendix

Table 8: Ablation of continual target-state updates under CSC. RB denotes the reliability-balanced historical prior.

Panel of Figure 7 — Figure 7: Correction gain analysis. (a) The estimated posterior-predictive gain is positively associated with the realized source-relative correction gain across test samples. (b) Not every correction helps: the source-relative benefit of adaptation varies throughout the stream, and target-driven corrections may yield limited or even negative gain.

Method	BP-free	Metric	Depth of field		Noise			Lighting	Weather	Video			Camera motion		Avg.
Method	BP-free	Metric	Near foc.	Far foc.	Color quant.	ISO	Low light	Flash	Fog 3D	Bit err.	H.265 abr.	H.265 crf	XY-mot.	Z-mot.	Avg.
		Acc. $\uparrow$	71.1	62.6	55.9	62.1	61.7	45.2	44.6	36.0	71.8	77.4	45.6	49.3	56.9
Source	–	ECE $\downarrow$	7.0	5.0	3.9	14.9	8.2	4.3	5.5	8.6	9.7	9.7	4.1	4.9	7.2
		Acc. $\uparrow$	70.8	62.4	57.1	61.1	62.2	45.4	44.9	35.7	72.0	77.3	46.1	49.6	57.0
CoTTA ( CVPR 2022 )	✗	ECE $\downarrow$	6.1	4.8	3.8	8.1	2.7	8.0	4.8	18.3	3.3	3.7	15.0	18.5	8.1
		Acc. $\uparrow$	70.8	62.3	57.2	60.9	62.2	45.3	44.7	35.9	72.3	77.5	46.5	50.2	57.1

Appendix

Table 9: CSC results on ImageNet-3DCC. Accuracy (Acc., %) and expected calibration error (ECE, %) with ViT-Base at severity 5. BP-free denotes backpropagation-free adaptation. Bold indicates the best results; Source is shown for reference only.

Method	BP-free	Metric	Depth of field		Noise			Lighting	Weather	Video			Camera motion		Avg.
Method	BP-free	Metric	Near foc.	Far foc.	Color quant.	ISO	Low light	Flash	Fog 3D	Bit err.	H.265 abr.	H.265 crf	XY-mot.	Z-mot.	Avg.
		Acc. $\uparrow$	71.1	62.6	55.9	62.1	61.7	45.2	44.6	36.0	71.8	77.4	45.6	49.3	56.9
Source	–	ECE $\downarrow$	7.0	5.0	3.9	14.9	8.2	4.3	5.5	8.6	9.7	9.7	4.1	4.9	7.2
		Acc. $\uparrow$	71.1	62.4	57.1	61.9	62.4	45.0	45.2	35.7	71.9	77.4	45.5	49.6	57.1
CoTTA ( CVPR 2022 )	✗	ECE $\downarrow$	3.0	5.7	6.7	3.4	3.7	14.3	4.8	15.9	3.1	5.0	9.1	10.1	7.1
		Acc. $\uparrow$	71.1	62.5	57.2	61.0	62.2	45.3	44.7	36.0	72.1	77.4	45.9	49.9	57.1

Appendix

Table 10: CDC results on ImageNet-3DCC. Accuracy (Acc.,%) and expected calibration error (ECE,%) with ViT-Base at severity 5. BP-free denotes backpropagation-free adaptation. Bold indicates the best results; Source is shown for reference only. Our results are averaged over five runs.

Method	BP-free	Metric	Level 5	Level 4	Level 3	Level 2	Level 1	Avg.
		Acc. $\uparrow$	56.9	64.1	69.4	73.6	76.6	68.1
Source	–	ECE $\downarrow$	4.3	5.9	6.7	7.7	8.3	6.6
CoTTA ( CVPR 2022 )	✗	Acc. $\uparrow$	57.1	64.3	69.6	73.8	76.7	68.3
CoTTA ( CVPR 2022 )	✗	ECE $\downarrow$	4.8	3.3	2.7	2.2	2.2	3.0
ViDA ( ICLR 2024 )	✗	Acc. $\uparrow$	57.2	64.3	69.6	73.7	76.7	68.3
ViDA ( ICLR 2024 )	✗	ECE $\downarrow$	3.5	4.2	5.0	5.9	6.4	5.0

Appendix

Table 11: MDS results on ImageNet-3DCC. Accuracy (Acc., %) and expected calibration error (ECE, %) across severity levels 5–1. BP-free denotes backpropagation-free adaptation. Bold indicates the best results; Source is shown for reference only. Our results are averaged over five runs.

Method	BP-free	Metric	R1	R2	R3	R4	R5	R6	R7	R8	R9	R10	Avg.
		Acc. $\uparrow$	56.9	56.9	56.9	56.9	56.9	56.9	56.9	56.9	56.9	56.9	56.9
Source	–	ECE $\downarrow$	7.2	7.2	7.2	7.2	7.2	7.2	7.2	7.2	7.2	7.2	7.2
CoTTA ( CVPR 2022 )	✗	Acc. $\uparrow$	57.0	57.0	57.1	56.9	56.8	56.8	56.9	57.1	57.1	57.0	57.0
CoTTA ( CVPR 2022 )	✗	ECE $\downarrow$	8.1	17.3	23.8	27.4	29.5	31.1	32.1	33.2	33.7	34.3	27.0
ViDA ( ICLR 2024 )	✗	Acc. $\uparrow$	57.1	57.9	58.3	58.6	58.8	58.9	59.0	59.0	59.1	59.1	58.6
ViDA ( ICLR 2024 )	✗	ECE $\downarrow$	6.7	6.0	6.8	8.9	11.2	13.1	14.7	16.1	17.2	18.1	11.9

Appendix

Table 12: LHA results on ImageNet-3DCC. Accuracy (Acc., %) and expected calibration error (ECE, %) over 10 repeated corruption cycles (R1–R10) with ViT-Base at severity 5. BP-free denotes backpropagation-free adaptation. Bold indicates the best results; Source is shown for reference only.

Method	BP-free	ImageNet-R	ImageNet-V2	ImageNet-Sketch	Avg.
Source	–	59.5	75.4	44.9	59.9
Tent ( ICLR 2021 )	✗	63.9	75.2	49.1	62.7
CoTTA ( CVPR 2022 )	✗	63.5	75.4	50.0	63.0
SAR ( ICLR 2023 )	✗	63.3	75.1	48.7	62.4
FOA ( ICML 2024 )	✓	63.8	75.4	49.9	63.0
REM ( ICML 2025 )	✗	64.3	75.2	49.7	63.1

Appendix

Table 13: TTA results under domain shifts. Top-1 accuracy on ImageNet-R, ImageNet-V2, and ImageNet-Sketch. BP-free denotes backpropagation-free adaptation. Bold indicates the best results.

Explore similar work

Jul 9, 2026cs.CV

Continual Test-Time Adaptation in Computer Vision: Methods, Benchmarks, and Future Directions

Deep neural nets achieve remarkable performance when training and test data share the same distribution, but this assumption frequently breaks in real-world deployment, where data undergoes continual distributional shifts. Continual Test-Time Adaptation (CTTA) addresses this challenge by adapting pretrained models to non-stationary target distributions on-the-fly, without access to source data or labeled targets, while mitigating two critical failure modes: catastrophic forgetting of source knowledge and error accumulation from noisy pseudo-labels over extended time horizons. In this comprehensive survey, we formally define the CTTA problem, analyze the diverse continual domain shift patterns that characterize different evaluation protocols, and propose a hierarchical taxonomy that categorizes existing methods into three families: optimization-based strategies (entropy minimization, pseudo-labeling, parameter restoration), parameter-efficient methods (normalization layer adaptation, adaptive parameter selection), and architecture-based approaches (teacher-student frameworks, adapters, visual prompting, masked modeling). We systematically review representative methods within each category and present comparative benchmarks and experimental results across standard evaluation settings. Finally, we discuss the limitations of current approaches and highlight emerging research directions, including the adaptation of foundation models and black-box systems, thereby providing a roadmap for future research in robust continual test-time adaptation.

Sarthak Kumar Maharana, Shambhavi Mishra, Yunbei Zhang +6

The University of Texas at Dallas, USA · LIVIA ETS Montreal, ILLS International Laboratory on Learning Systems (ILLS), McGill - ÉTS - MILA - CNRS - Université Paris-Saclay - CentraleSupélec · Tulane University, USA +2

Test-Time Adaptation Mitigating Domain Shift Computer Vision Batch Normalization

Ask PDF

Aug 30, 2026cs.CV

Continual Test-Time Adaptation via Entropy Sensitivity-Guidance in Strict Online Setting

Test-time adaptation (TTA) promises robustness under distribution shift by updating a pretrained model on unlabeled test data, but strict online TTA with batch size one and no access to source data is especially prone to drift or collapse. We introduce Sensitivity-Guided Erasing Adaptation (SEGA), a method for strict online continual TTA (CTTA) on corruption-style streams. SEGA uses a small number of structured erasures to probe how predictive entropy changes as information is removed, and uses the resulting per-sample sensitivity trajectories to coordinate recovery and sample selection rather than relying on raw entropy or batch statistics. This yields a practical feedback signal for long-horizon batch-size-one adaptation without periodic resets or model reservoirs. In experiments on ImageNet-C, CIFAR10/100-C, and corruption-generated aquaculture streams treated as controlled corruption-style proxies, SEGA yields consistent robustness and stability gains over strong CTTA baselines while reducing backward passes through sensitivity-based gating.

Chandler Timm C. Doloriel, Yunbei Zhang, Muhammad Salman Siddiqui +4

Faculty of Science and Technology (REALTEK), Norwegian University of Life Sciences (NMBU) · Tulane University

Test-Time Adaptation Concept Erasure Fish

Ask PDF

Jul 26, 2026cs.LG

Source-Free Controlled Adaptation of Teachers for Continual Test-Time Adaptation

In many real-world scenarios, encountering continual shifts in domain during inference is very common. Consequently, continual test-time adaptation (CTTA) techniques leveraging a teacher-student framework have gained prominence, allowing models to adapt continuously even after deployment. In such a framework, a weight-averaged mean teacher is used to produce pseudo-labels from test data for self-training. The mean teacher gets updated as an exponential moving average of the student parameters using a high value of momentum that is kept fixed even if different distributions of test data are encountered. To combat the resulting drift of the model, we propose a novel controlled teacher adaptation methodology that dynamically sets a proper momentum value depending on the quality of the incoming data. Additionally, we estimate class prototypes from the source pretrained model to help align the target data as they come in. Importantly, our method does not require access to source data or its statistics at any stage of the pipeline, making it truly source-free. We perform extensive experiments on benchmark datasets to demonstrate that our approach outperforms different state-of-the-art adaptation frameworks, many of which require access to source data.

Anurag Roy, Riddhiman Moulick, Vinay Kumar Verma +2

IIT Kharagpur · IBM Research · Amazon +1

Test-Time Adaptation Parameter-Efficient Adaptation Teacher

Ask PDF