cs.CVSep 29, 2026
SaveNot Every Correction Helps: Gain-Guided Continual Test-Time Adaptation
Organizations: Sungkyunkwan University · Nanyang Technological University
Abstract
Continual test-time adaptation (CTTA) adapts a source model to an unlabeled test stream whose distribution may change over time. Existing TTA methods often assess prediction reliability using confidence or entropy, which primarily reflect the model's self-certainty for the current sample. In CTTA, accumulated target observations can provide complementary evidence for correcting the source prediction, but this history may become misaligned as the target distribution changes. The key question is therefore not how much the correction differs from the source prediction, but whether and how strongly it should be applied. This paper proposes Gain-Aware INtervention (GAIN), a backpropagation-free CTTA framework guided by a simple principle: history proposes, gain decides. GAIN maintains compact target statistics to form a correction proposal and a posterior-predictive evaluator that accounts for estimation uncertainty. The resulting source-relative gain estimates the proposal's benefit and determines a sample-specific intervention strength along a continuous path through efficient one-dimensional optimization. Gain-controlled predictions then update the target statistics online, limiting the propagation of unreliable corrections, all without backpropagation, sample storage, or replay. Across five benchmarks, our method achieves strong predictive performance, with favorable accuracy--calibration--efficiency trade-offs in continual adaptation. On ImageNet-C, for example, GAIN achieves 61.9% accuracy with near-source calibration. It remains stable under diverse and challenging continual shifts while running 15.9x faster than a representative optimization-based CTTA baseline.
Figures & tables
Figure 1: Gain-Guided Sample-wise Intervention. Past target batches are summarized by compact statistics to construct a history-based correction proposal (solid) and an uncertainty-aware posterior-predictive estimate (dashed) that evaluates it. The resulting source-relative gain determines a sample-specific strength along the source-to-correction path: unsupported corrections are rejected (Sample A), partially supported corrections are applied conservatively (Sample B), and well-supported corrections are fully adopted (Sample C).
Figure 2: Overview of GAIN : history proposes, gain decides. (i) The frozen source model produces the source prediction and current feature , while the retained target state summarizes past batches. (ii) Together, and form the target proposal and predictive evaluator . The evaluator accounts for uncertainty to estimate the proposal’s source-relative gain . (iii) This source-relative gain determines the intervention strength along the continuous evidence path from to . The final prediction then updates the target statistics for future batches.
| Method | BP-free | Metric | Noise | Blur | Weather | Digital | Avg. | |||||||||||
| Gauss. | Shot | Impu. | Defo. | Glas. | Moti. | Zoom | Snow | Fros. | Fog | Brig. | Cont. | Elas. | Pix. | JPEG | ||||
| Acc. | 47.0 | 48.2 | 47.9 | 31.5 | 21.2 | 41.5 | 36.7 | 50.1 | 45.8 | 42.3 | 73.6 | 8.6 | 42.5 | 62.0 | 63.8 | 44.2 | ||
| Source | – | ECE | 3.6 | 4.1 | 3.7 | 4.3 | 5.4 | 3.9 | 9.1 | 2.3 | 4.9 | 17.4 | 3.1 | 3.9 | 9.0 | 3.3 | 2.7 | 5.4 |
| Tent ( ICLR 2021 ) | ✗ | Acc. | 47.8 | 51.1 | 50.8 | 34.2 | 27.0 | 45.5 | 41.6 | 56.0 | 52.3 | 49.7 | 76.1 | 27.2 | 44.3 | 65.6 | 66.1 | 49.0 |
| ECE | 5.8 | 8.3 | 10.5 | 12.1 | 18.0 | 13.9 | 18.2 | 12.4 | 14.6 | 13.3 | 7.0 | 19.1 | 20.2 | 10.3 | 8.7 | 12.8 | ||
| CoTTA ( CVPR 2022 ) | ✗ | Acc. | 47.1 | 48.4 | 48.6 | 31.7 | 21.9 | 42.9 | 38.0 | 51.8 | 47.3 | 44.7 | 74.1 | 10.0 | 43.6 | 63.6 | 64.8 | 45.2 |
Table 1: CSC results on ImageNet-C. Accuracy (Acc., %) and expected calibration error (ECE, %) with ViT-Base at severity level 5. BP-free denotes backpropagation-free adaptation. Bold indicates the best results; Source is shown for reference only.
Figure 3: Accuracy and calibration under three challenging CTTA scenarios on ImageNet-C. (a) CDC evaluates recurring corruptions with irregular durations. (b) MDS interleaves samples from multiple corruption domains, with results averaged over severity levels 1–5. (c) LHA evaluates error accumulation and long-term stability over 10 repeated corruption cycles.
Figure 4: Efficiency–accuracy trade-off on ImageNet-3DCC. Inference speed is normalized to CoTTA; bubble size indicates ECE.
Figure 5: Sensitivity analysis. We study the sensitivity of GAIN to (a) the source-centered prior strength and (b) the test-time batch size. For each setting, we report both accuracy and ECE.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | BP-free | Metric | Noise | Blur | Weather | Digital | Avg. | |||||||||||
| Gauss. | Shot | Impu. | Defo. | Glas. | Moti. | Zoom | Snow | Fros. | Fog | Brig. | Cont. | Elas. | Pix. | JPEG | ||||
| Acc. | 47.0 | 48.2 | 47.9 | 31.5 | 21.2 | 41.5 | 36.7 | 50.1 | 45.8 | 42.3 | 73.6 | 8.6 | 42.5 | 62.0 | 63.8 | 44.2 | ||
| Source | – | ECE | 3.6 | 4.1 | 3.7 | 4.3 | 5.4 | 3.9 | 9.1 | 2.3 | 4.9 | 17.4 | 3.1 | 3.9 | 9.0 | 3.3 | 2.7 | 5.4 |
| CoTTA ( CVPR 2022 ) | ✗ | Acc. | 46.1 | 47.4 | 47.5 | 33.3 | 22.3 | 42.7 | 38.6 | 50.3 | 46.1 | 44.2 | 74.0 | 6.6 | 43.3 | 62.4 | 64.6 | 44.6 |
| ECE | 7.0 | 6.9 | 10.3 | 4.1 | 12.1 | 8.4 | 17.3 | 5.5 | 8.9 | 10.0 | 5.2 | 2.2 | 12.8 | 5.7 | 4.1 | 8.0 | ||
| SAR ( ICLR 2023 ) | ✗ | Acc. | 53.7 | 57.5 | 56.5 | 51.4 | 45.7 | 56.8 | 48.1 | 61.2 | 57.9 | 53.7 | 77.3 | 41.6 | 50.8 | 66.7 | 66.5 | 56.4 |
Table 4: CDC results on ImageNet-C. Accuracy (Acc., %) and expected calibration error (ECE, %) with ViT-Base at severity level 5. BP-free denotes backpropagation-free adaptation. Bold indicates the best results; Source is shown for reference only. Our results are averaged over five runs.
| Method | BP-free | Metric | Level 5 | Level 4 | Level 3 | Level 2 | Level 1 | Avg. |
| Acc. | 44.2 | 55.4 | 68.3 | 69.2 | 74.8 | 62.4 | ||
| Source | – | ECE | 3.4 | 4.3 | 4.2 | 3.3 | 2.8 | 3.6 |
| CoTTA ( CVPR 2022 ) | ✗ | Acc. | 50.3 | 62.0 | 69.4 | 73.9 | 78.2 | 66.8 |
| ECE | 5.4 | 4.6 | 3.9 | 3.9 | 3.4 | 4.2 | ||
| SAR ( ICLR 2023 ) | ✗ | Acc. | 56.1 | 64.9 | 70.9 | 74.0 | 78.3 | 68.8 |
| ECE | 8.5 | 5.9 | 4.4 | 3.8 | 3.2 | 5.2 |
Table 5: MDS results on ImageNet-C. Accuracy (Acc.,%) and expected calibration error (ECE,%) across corruption severity levels 5–1. BP-free denotes backpropagation-free adaptation. Bold indicates the best results; Source is shown for reference only. Our results are averaged over five runs.
| Method | BP-free | Metric | R1 | R2 | R3 | R4 | R5 | R6 | R7 | R8 | R9 | R10 | Avg. |
| Acc. | 44.2 | 44.2 | 44.2 | 44.2 | 44.2 | 44.2 | 44.2 | 44.2 | 44.2 | 44.2 | 44.2 | ||
| Source | – | ECE | 5.4 | 5.4 | 5.4 | 5.4 | 5.4 | 5.4 | 5.4 | 5.4 | 5.4 | 5.4 | 5.4 |
| CoTTA ( CVPR 2022 ) | ✗ | Acc. | 45.2 | 45.3 | 45.9 | 46.3 | 46.6 | 46.4 | 46.0 | 45.7 | 45.4 | 45.3 | 45.8 |
| ECE | 7.5 | 15.8 | 22.3 | 26.4 | 28.1 | 30.7 | 33.1 | 34.7 | 36.2 | 36.9 | 27.2 | ||
| ViDA ( ICLR 2024 ) | ✗ | Acc. | 56.6 | 57.3 | 57.3 | 57.4 | 57.5 | 57.5 | 57.3 | 54.1 | 54.8 | 54.9 | 56.5 |
| ECE | 14.7 | 21.0 | 24.1 | 26.3 | 28.0 | 29.3 | 29.9 | 31.1 | 31.9 | 32.9 | 26.9 |
Table 6: LHA results on ImageNet-C. Accuracy (Acc., %) and expected calibration error (ECE, %) over 10 repeated corruption cycles with ViT-Base at severity 5. BP-free denotes backpropagation-free adaptation. Bold indicates the best results; Source is shown for reference only.
| Variant | Proposal | Gain evaluation | Intervention | Acc. | ECE | NLL |
| Proposal as Prediction | – | – | 58.6 | 11.2 | 2.5 | |
| Evaluator as Prediction | – | – | 58.9 | 12.8 | 2.4 | |
| Swapped Roles | 61.7 | 8.0 | 2.0 | |||
| GAIN (Ours) | 61.9 | 6.0 | 1.9 |
Table 7: Ablation of the proposal–evaluator roles under CSC. We compare direct target-side prediction, reversed proposal–evaluation roles, and the asymmetric design used by GAIN.
Figure 6: Fixed vs. adaptive intervention strength. We compare fixed with the proposed sample-wise adaptive . No single nonzero fixed intervention matches the accuracy–calibration trade-off of GAIN : the best fixed accuracy is 61.0%, while the lowest nonzero fixed ECE is 8.9%. The adaptive achieves 61.9% accuracy with 6.0% ECE.
| Variant | Assignment | Source Support | Class Prior | CSC | ||
| Acc. | ECE | NLL | ||||
| State Assignment | ||||||
| Source Assignment | – | RB | 58.5 | 13.7 | 2.2 | |
| ✓ | RB | 60.8 | 7.2 | 2.0 | ||
| Proposal Assignment | – | RB | 59.6 | 12.2 | 2.2 | |
| ✓ | RB | 61.6 | 6.2 | 2.0 | ||
Table 8: Ablation of continual target-state updates under CSC. RB denotes the reliability-balanced historical prior.
Figure 7: Correction gain analysis. (a) The estimated posterior-predictive gain is positively associated with the realized source-relative correction gain across test samples. (b) Not every correction helps: the source-relative benefit of adaptation varies throughout the stream, and target-driven corrections may yield limited or even negative gain.
| Method | BP-free | Metric | Depth of field | Noise | Lighting | Weather | Video | Camera motion | Avg. | ||||||
| Near foc. | Far foc. | Color quant. | ISO | Low light | Flash | Fog 3D | Bit err. | H.265 abr. | H.265 crf | XY-mot. | Z-mot. | ||||
| Acc. | 71.1 | 62.6 | 55.9 | 62.1 | 61.7 | 45.2 | 44.6 | 36.0 | 71.8 | 77.4 | 45.6 | 49.3 | 56.9 | ||
| Source | – | ECE | 7.0 | 5.0 | 3.9 | 14.9 | 8.2 | 4.3 | 5.5 | 8.6 | 9.7 | 9.7 | 4.1 | 4.9 | 7.2 |
| Acc. | 70.8 | 62.4 | 57.1 | 61.1 | 62.2 | 45.4 | 44.9 | 35.7 | 72.0 | 77.3 | 46.1 | 49.6 | 57.0 | ||
| CoTTA ( CVPR 2022 ) | ✗ | ECE | 6.1 | 4.8 | 3.8 | 8.1 | 2.7 | 8.0 | 4.8 | 18.3 | 3.3 | 3.7 | 15.0 | 18.5 | 8.1 |
| Acc. | 70.8 | 62.3 | 57.2 | 60.9 | 62.2 | 45.3 | 44.7 | 35.9 | 72.3 | 77.5 | 46.5 | 50.2 | 57.1 | ||
Table 9: CSC results on ImageNet-3DCC. Accuracy (Acc., %) and expected calibration error (ECE, %) with ViT-Base at severity 5. BP-free denotes backpropagation-free adaptation. Bold indicates the best results; Source is shown for reference only.
| Method | BP-free | Metric | Depth of field | Noise | Lighting | Weather | Video | Camera motion | Avg. | ||||||
| Near foc. | Far foc. | Color quant. | ISO | Low light | Flash | Fog 3D | Bit err. | H.265 abr. | H.265 crf | XY-mot. | Z-mot. | ||||
| Acc. | 71.1 | 62.6 | 55.9 | 62.1 | 61.7 | 45.2 | 44.6 | 36.0 | 71.8 | 77.4 | 45.6 | 49.3 | 56.9 | ||
| Source | – | ECE | 7.0 | 5.0 | 3.9 | 14.9 | 8.2 | 4.3 | 5.5 | 8.6 | 9.7 | 9.7 | 4.1 | 4.9 | 7.2 |
| Acc. | 71.1 | 62.4 | 57.1 | 61.9 | 62.4 | 45.0 | 45.2 | 35.7 | 71.9 | 77.4 | 45.5 | 49.6 | 57.1 | ||
| CoTTA ( CVPR 2022 ) | ✗ | ECE | 3.0 | 5.7 | 6.7 | 3.4 | 3.7 | 14.3 | 4.8 | 15.9 | 3.1 | 5.0 | 9.1 | 10.1 | 7.1 |
| Acc. | 71.1 | 62.5 | 57.2 | 61.0 | 62.2 | 45.3 | 44.7 | 36.0 | 72.1 | 77.4 | 45.9 | 49.9 | 57.1 | ||
Table 10: CDC results on ImageNet-3DCC. Accuracy (Acc.,%) and expected calibration error (ECE,%) with ViT-Base at severity 5. BP-free denotes backpropagation-free adaptation. Bold indicates the best results; Source is shown for reference only. Our results are averaged over five runs.
| Method | BP-free | Metric | Level 5 | Level 4 | Level 3 | Level 2 | Level 1 | Avg. |
| Acc. | 56.9 | 64.1 | 69.4 | 73.6 | 76.6 | 68.1 | ||
| Source | – | ECE | 4.3 | 5.9 | 6.7 | 7.7 | 8.3 | 6.6 |
| CoTTA ( CVPR 2022 ) | ✗ | Acc. | 57.1 | 64.3 | 69.6 | 73.8 | 76.7 | 68.3 |
| ECE | 4.8 | 3.3 | 2.7 | 2.2 | 2.2 | 3.0 | ||
| ViDA ( ICLR 2024 ) | ✗ | Acc. | 57.2 | 64.3 | 69.6 | 73.7 | 76.7 | 68.3 |
| ECE | 3.5 | 4.2 | 5.0 | 5.9 | 6.4 | 5.0 |
Table 11: MDS results on ImageNet-3DCC. Accuracy (Acc., %) and expected calibration error (ECE, %) across severity levels 5–1. BP-free denotes backpropagation-free adaptation. Bold indicates the best results; Source is shown for reference only. Our results are averaged over five runs.
| Method | BP-free | Metric | R1 | R2 | R3 | R4 | R5 | R6 | R7 | R8 | R9 | R10 | Avg. |
| Acc. | 56.9 | 56.9 | 56.9 | 56.9 | 56.9 | 56.9 | 56.9 | 56.9 | 56.9 | 56.9 | 56.9 | ||
| Source | – | ECE | 7.2 | 7.2 | 7.2 | 7.2 | 7.2 | 7.2 | 7.2 | 7.2 | 7.2 | 7.2 | 7.2 |
| CoTTA ( CVPR 2022 ) | ✗ | Acc. | 57.0 | 57.0 | 57.1 | 56.9 | 56.8 | 56.8 | 56.9 | 57.1 | 57.1 | 57.0 | 57.0 |
| ECE | 8.1 | 17.3 | 23.8 | 27.4 | 29.5 | 31.1 | 32.1 | 33.2 | 33.7 | 34.3 | 27.0 | ||
| ViDA ( ICLR 2024 ) | ✗ | Acc. | 57.1 | 57.9 | 58.3 | 58.6 | 58.8 | 58.9 | 59.0 | 59.0 | 59.1 | 59.1 | 58.6 |
| ECE | 6.7 | 6.0 | 6.8 | 8.9 | 11.2 | 13.1 | 14.7 | 16.1 | 17.2 | 18.1 | 11.9 |
Table 12: LHA results on ImageNet-3DCC. Accuracy (Acc., %) and expected calibration error (ECE, %) over 10 repeated corruption cycles (R1–R10) with ViT-Base at severity 5. BP-free denotes backpropagation-free adaptation. Bold indicates the best results; Source is shown for reference only.
| Method | BP-free | ImageNet-R | ImageNet-V2 | ImageNet-Sketch | Avg. |
| Source | – | 59.5 | 75.4 | 44.9 | 59.9 |
| Tent ( ICLR 2021 ) | ✗ | 63.9 | 75.2 | 49.1 | 62.7 |
| CoTTA ( CVPR 2022 ) | ✗ | 63.5 | 75.4 | 50.0 | 63.0 |
| SAR ( ICLR 2023 ) | ✗ | 63.3 | 75.1 | 48.7 | 62.4 |
| FOA ( ICML 2024 ) | ✓ | 63.8 | 75.4 | 49.9 | 63.0 |
| REM ( ICML 2025 ) | ✗ | 64.3 | 75.2 | 49.7 | 63.1 |
Table 13: TTA results under domain shifts. Top-1 accuracy on ImageNet-R, ImageNet-V2, and ImageNet-Sketch. BP-free denotes backpropagation-free adaptation. Bold indicates the best results.