cs.LGNov 9, 2026

Controlling Temporal Pseudo-Label Marginals for Stable Online Test-Time Adaptation

Authors: Rani Amar, Ofir Lindenbaum

Organizations: Faculty of Engineering Bar-Ilan University Ramat Gan, Israel

Abstract

Continual Test-Time Adaptation (CTTA) methods update models online using a stream of unlabeled data to overcome distribution shifts encountered at test time. However, CTTA is often unstable, as it can reinforce biases in early predictions and increasingly predict only a few classes, resulting in drift or even collapse. In this paper, we identify narrowing pseudo-label class usage as an indicator of CTTA instability and quantify this behavior. Most existing methods have attempted to address this issue by either selecting reliable adaptation samples or re-weighting the loss to encourage diverse pseudo-labels, but they do not explicitly enforce a class marginal over a temporal window. Instead, we propose Sinkhorn Class-Marginal Regularization (SCALAR), a simple plug-and-play module that controls the pseudo-label distribution over a short temporal window. SCALAR works by maintaining recent predictions in a buffer, balancing their class distribution via the Sinkhorn algorithm to match a specified target distribution, and using the balanced soft targets for adaptation. Without changing the model architecture, SCALAR maintains the per-step runtime and increases the peak allocated GPU memory by only 10 MB on ImageNet-C. Our method is particularly effective on skewed, non-stationary data with severe distribution shifts, where SCALAR significantly alleviates drift and collapse when combined with existing stabilization methods based on entropy or diversity.

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Continual Test-Time Adaptation in Computer Vision: Methods, Benchmarks, and Future Directions

    Jul 9, 2026Sarthak Kumar Maharana, Shambhavi Mishra, Yunbei Zhang +6Source-Free Domain AdaptationDistribution Shift Robustness

  2. BP-TTA: Balanced and Prototype-Guided Test-Time Adaptation in Dynamic Scenarios

    Jun 30, 2026Shaoyang Huang, Yashi Zhu, Yichen Yu +3Class-Imbalanced LearningTest-Time Adaptation

  3. Distill Once, Adapt Life-Long: Exploring Dataset Distillation for Continual Test-Time Adaptation

    Jun 18, 2026Hyun-Kurl Jang, Jihun Kim, Hyeokjun Kweon +1Source-Free Domain AdaptationContinual Test-Time Adaptation