Calibrating Multimodal Test-Time Adaptation under Reliability and Availability Modality Shifts
Organizations: National Key Laboratory of Space Integrated Information System, Institute of Software, Chinese Academy of Sciences, and University of Chinese Academy of Sciences, Beijing, China · Wangxuan Institute of Computer Technology, Peking University, Beijing, China · Thrust of Artificial Intelligence, the Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Department of Computer Science & Engineering, the Hong Kong University of Science and Technology, Hong Kong SAR, China ·
Abstract
Multimodal test-time adaptation (TTA) enables pretrained multimodal models to accommodate distribution shifts during inference using only unlabeled test data, without recourse to the original training data. Compared with unimodal adaptation, multimodal adaptation is further complicated by the coexistence and interaction of multiple representation perspectives. This complexity is particularly pronounced under modality reliability shifts, which preserve modality availability but degrade information quality, and modality availability shifts, which render the corresponding modality entirely inaccessible. Category-conditional Gaussian modeling offers a natural starting point for handling such distributional variations, having proven effective in unimodal test-time adaptation for characterizing representation distributions, yet its extension to multimodal settings requires a more elaborate formulation. To this end, we propose Adaptive Probabilistic Gaussian Calibration (AdaPGC), which extends category-conditional Gaussian modeling across modality-specific and multimodal fusion representations for probabilistic calibration under multimodal distribution shifts. AdaPGC continuously updates category-conditional Gaussian models over modality-specific and fusion representations, using the resulting probabilistic predictions to calibrate source model predictions and guide reliability based rectification, while exploiting their joint distributions to recover unavailable fusion representations. Extensive experiments under diverse modality corruption and missingness conditions demonstrate that AdaPGC effectively improves robustness under both modality reliability and availability shifts. Our source code will be released at https://github.com/XuJinglinn/AdaPGC-plus.
Figures & tables
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Test-Time Adaptation
Test-time adaptation updates or recalibrates pretrained models at inference to handle distribution shifts, often without target labels.
See all 342 papers