Controlling Temporal Pseudo-Label Marginals for Stable Online Test-Time Adaptation
Organizations: Faculty of Engineering Bar-Ilan University Ramat Gan, Israel
Abstract
Continual Test-Time Adaptation (CTTA) methods update models online using a stream of unlabeled data to overcome distribution shifts encountered at test time. However, CTTA is often unstable, as it can reinforce biases in early predictions and increasingly predict only a few classes, resulting in drift or even collapse. In this paper, we identify narrowing pseudo-label class usage as an indicator of CTTA instability and quantify this behavior. Most existing methods have attempted to address this issue by either selecting reliable adaptation samples or re-weighting the loss to encourage diverse pseudo-labels, but they do not explicitly enforce a class marginal over a temporal window. Instead, we propose Sinkhorn Class-Marginal Regularization (SCALAR), a simple plug-and-play module that controls the pseudo-label distribution over a short temporal window. SCALAR works by maintaining recent predictions in a buffer, balancing their class distribution via the Sinkhorn algorithm to match a specified target distribution, and using the balanced soft targets for adaptation. Without changing the model architecture, SCALAR maintains the per-step runtime and increases the peak allocated GPU memory by only 10 MB on ImageNet-C. Our method is particularly effective on skewed, non-stationary data with severe distribution shifts, where SCALAR significantly alleviates drift and collapse when combined with existing stabilization methods based on entropy or diversity.
Figures & tables
| Method | CIFAR-10-C | CIFAR-100-C | ImageNet-C |
|---|---|---|---|
| Source-only | 28.1 | 35.4 | 55.8 |
| TENT [ 31 ] | 23.5 | 32.1 | 51.0 |
| CoTTA [ 32 ] | 24.6 | 34.8 | 54.8 |
| VDP [ 14 ] | 24.1 | 32.0 | 50.0 |
| VIDA [ 23 ] | 20.7 | 27.3 | 43.4 |
| Versatile [ 33 ] | 8.9 | 24.0 | 42.7 |
| High speed | Medium speed | Low speed | ||||
|---|---|---|---|---|---|---|
| Method | Avg Err | Clean Err | Avg Err | Clean Err | Avg Err | Clean Err |
| Baseline (w/o SCALAR) | 86.8 14.4 | 76.4 40.6 | 48.3 4.4 | 25.2 5.9 | 47.5 0.6 | 17.4 0.3 |
| SCALAR-U | 48.1 1.1 | 21.1 0.5 | 42.9 0.7 | 18.2 0.4 | 49.1 0.4 | 17.3 0.1 |
| SCALAR-C | 46.1 1.0 | 22.2 0.4 | 42.5 0.4 | 17.7 0.3 | 48.2 0.8 | 17.1 0.1 |
| Optimal speed | Medium speed | High speed | ||||
|---|---|---|---|---|---|---|
| Method | Avg Err | Clean Err | Avg Err | Clean Err | Avg Err | Clean Err |
| TENT-R50 [ 31 ] | 67.5 0.4 | 32.9 0.3 | 88.3 1.3 | 97.1 1.1 | 95.9 0.4 | 99.6 0.2 |
| TENT + SCALAR | 66.5 0.3 | 30.9 0.2 | 67.4 0.5 | 37.1 0.8 | 69.8 1.1 | 43.3 1.2 |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Method / Dataset | Reported | Reproduced |
|---|---|---|
| Source-only / CIFAR-10-C | 28.1 | 27.9 |
| Source-only / CIFAR-100-C | 35.4 | 36.0 |
| Source-only / ImageNet-C | 55.8 | 56.2 |
| VIDA / ImageNet-C | 43.4 | 43.35 |
| TENT / ImageNet-C | 51.0 | 50.48 |
| Augmentation | Parameters + note |
|---|---|
| RandomAffine | degrees translate scale – , bilinear, fill=0 |
| ColorJitterPro | brightness – , contrast – , saturation – , hue , gamma – |
| RandomGrayscale |
| Time | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | Ref | Gauss. | Shot | Impulse | Defocus | Glass | Motion | Zoom | Snow | Frost | Fog | Bright. | Contrast | Elastic | Pixelate | JPEG | Avg. | Gain |
| Source | 60.1 | 53.2 | 38.3 | 19.9 | 35.5 | 22.6 | 18.6 | 12.1 | 12.7 | 22.8 | 5.3 | 49.7 | 23.6 | 24.7 | 23.1 | 28.1 | 0.0 | |
| TENT | [ 31 ] | 57.7 | 56.3 | 29.4 | 16.2 | 35.3 | 16.2 | 12.4 | 11.0 | 11.6 | 14.9 | 4.7 | 22.5 | 15.9 | 29.1 | 19.5 | 23.5 | 4.6 |
| CoTTA | [ 32 ] | 58.7 | 51.3 | 33.0 | 20.1 | 34.8 | 20.0 | 15.2 | 11.1 | 11.3 | 18.5 | 4.0 | 34.7 | 18.8 | 19.0 | 17.9 | 24.6 | 3.5 |
| VDP | [ 14 ] | 57.5 | 49.5 | 31.7 | 21.3 | 35.1 | 19.6 | 15.1 | 10.8 | 10.3 | 18.1 | 4.0 | 27.5 | 18.4 | 22.5 | 19.9 | 24.1 | 4.0 |
| VIDA | [ 23 ] | 52.9 | 47.9 | 19.4 | 11.4 | 31.3 | 13.3 | 7.6 | 7.6 | 9.9 | 12.5 | 3.8 | 26.3 | 14.4 | 33.9 | 18.2 | 20.7 | 7.4 |
| Time | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | Ref | Gauss. | Shot | Impulse | Defocus | Glass | Motion | Zoom | Snow | Frost | Fog | Bright. | Contrast | Elastic | Pixelate | JPEG | Avg. | Gain |
| Source | 55.0 | 51.5 | 26.9 | 24.0 | 60.5 | 29.0 | 21.4 | 21.1 | 25.0 | 35.2 | 11.8 | 34.8 | 43.2 | 56.0 | 35.9 | 35.4 | 0 | |
| TENT | [ 31 ] | 53.0 | 47.0 | 24.6 | 22.3 | 58.5 | 26.5 | 19.0 | 21.0 | 23.0 | 30.1 | 11.8 | 25.2 | 39.0 | 47.1 | 33.3 | 32.1 | 3.3 |
| CoTTA | [ 32 ] | 55.0 | 51.3 | 25.8 | 24.1 | 59.2 | 28.9 | 21.4 | 21.0 | 24.7 | 34.9 | 11.7 | 31.7 | 40.4 | 55.7 | 35.6 | 34.8 | 0.6 |
| VDP | [ 14 ] | 54.8 | 51.2 | 25.6 | 24.2 | 59.1 | 28.8 | 21.2 | 20.5 | 23.3 | 33.8 | 7.5 | 11.7 | 32.0 | 51.7 | 35.2 | 32.0 | 3.4 |
| VIDA | [ 23 ] | 50.1 | 40.7 | 22.0 | 21.2 | 45.2 | 21.6 | 16.5 | 17.9 | 16.6 | 25.6 | 11.5 | 29.0 | 29.6 | 34.7 | 27.1 | 27.3 | 8.1 |
| Time | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | Ref | Gauss. | Shot | Impulse | Defocus | Glass | Motion | Zoom | Snow | Frost | Fog | Bright. | Contrast | Elastic | Pixelate | JPEG | Avg. | Gain |
| Source | 53.0 | 51.8 | 52.1 | 68.5 | 78.8 | 58.5 | 63.3 | 49.9 | 54.2 | 57.7 | 26.4 | 91.4 | 57.5 | 38.0 | 36.2 | 55.8 | 0.0 | |
| TENT | [ 31 ] | 52.2 | 48.9 | 49.2 | 65.8 | 73.0 | 54.5 | 58.4 | 44.0 | 47.7 | 50.3 | 23.9 | 72.8 | 55.7 | 34.4 | 33.9 | 51.0 | 4.8 |
| CoTTA | [ 32 ] | 52.9 | 51.6 | 51.4 | 68.3 | 78.1 | 57.1 | 62.0 | 48.2 | 52.7 | 55.3 | 25.9 | 90.0 | 56.4 | 36.4 | 35.2 | 54.8 | 1.0 |
| VDP | [ 14 ] | 52.7 | 51.6 | 50.1 | 58.1 | 70.2 | 56.1 | 58.1 | 42.1 | 46.1 | 45.8 | 23.6 | 70.4 | 54.9 | 34.5 | 36.1 | 50.0 | 5.8 |
| VIDA | [ 23 ] | 47.7 | 42.5 | 42.9 | 52.2 | 56.9 | 45.5 | 48.9 | 38.9 | 42.7 | 40.7 | 24.3 | 52.8 | 49.1 | 33.5 | 33.1 | 43.4 | 12.4 |
| LR | Gauss. | Shot | Impulse | Defocus | Glass | Motion | Zoom | Snow | Frost | Fog | Bright. | Contrast | Elastic | Pixelate | JPEG | Avg. | Clean | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.9990 | 0.9990 | 40.61 | 30.07 | 25.03 | 26.99 | 32.66 | 24.94 | 22.76 | 23.75 | 22.07 | 25.28 | 19.25 | 19.57 | 31.42 | 29.34 | 32.71 | 27.10 | 25.30 | |
| 0.9991 | 0.9991 | 40.00 | 28.84 | 24.43 | 26.45 | 33.75 | 25.66 | 23.52 | 25.63 | 23.19 | 26.36 | 21.08 | 20.98 | 32.70 | 29.66 | 32.57 | 27.70 | 26.66 | |
| 0.9992 | 0.9992 | 41.08 | 29.82 | 23.95 | 26.38 | 31.99 | 24.83 | 22.23 | 23.64 | 21.46 | 24.87 | 19.06 | 19.43 | 31.31 | 28.88 | 32.87 | 26.80 | 25.38 | |
| 0.9993 | 0.9993 | 41.84 | 30.24 | 25.31 | 27.02 | 34.87 | 25.77 | 22.24 | 24.21 | 22.60 | 27.14 | 19.88 | 19.97 | 31.95 | 28.05 | 31.20 | 27.50 | 24.66 | |
| 0.9994 | 0.9994 | 42.22 | 30.69 | 24.05 | 25.94 | 32.54 | 24.36 | 21.43 | 23.27 | 20.91 | 24.35 | 18.22 | 18.42 | 29.80 | 27.57 | 29.88 | 26.20 | 22.56 | |
| 0.9995 | 0.9995 | 43.48 | 31.34 | 24.91 | 27.29 | 33.43 | 24.87 | 21.60 | 23.17 | 20.85 | 24.69 | 18.13 | 18.71 | 30.59 | 31.42 | 29.85 | 27.00 | 22.63 |
| Batch size | Avg. error | Final clean | Coll. |
|---|---|---|---|
| 2 | 0/3 | ||
| 4 | 0/3 | ||
| 8 | 0/3 |
| Dominant classes | |||
|---|---|---|---|
| 25 | 50.7 / 21.6 | 43.9 / 18.5 | 45.8 / 18.0 |
| 50 | 49.6 / 19.4 | 48.4 / 18.8 | 48.7 / 18.1 |
| 200 | 48.9 / 20.3 | 46.7 / 19.2 | 46.6 / 18.3 |
| Mean | 49.7 / 20.4 | 46.3 / 18.9 | 47.0 / 18.1 |
| Method | Avg. error | Final clean | Coll. |
|---|---|---|---|
| Baseline (w/o SCALAR) | 2/3 | ||
| Sharpening | 2/3 | ||
| SCALAR-U | 0/3 |
| Component | Compute (GFLOPs) | Memory (MB) |
|---|---|---|
| ViT-B/16 backbone (forward) | ||
| SCALAR (ImageNet, FP32) | (fp32 queue) |
Continual Test-Time Adaptation
Continual test-time adaptation updates pretrained models on unlabeled, changing deployment data while retaining useful knowledge across shifts.
See all 74 papers