Calibrating Multimodal Test-Time Adaptation under Reliability and Availability Modality Shifts
Authors: Jiangmeng Li, Jinglin Xu, Yi Li, Yumeng Liu, Jiahuan Zhou, Fanjiang Xu, Hui Xiong, Gang Hua
Organizations: National Key Laboratory of Space Integrated Information System, Institute of Software, Chinese Academy of Sciences, and University of Chinese Academy of Sciences, Beijing, China · Wangxuan Institute of Computer Technology, Peking University, Beijing, China · Thrust of Artificial Intelligence, the Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Department of Computer Science & Engineering, the Hong Kong University of Science and Technology, Hong Kong SAR, China · Amazon Alexa AI, Bellevue, WA 98004 USA
Multimodal test-time adaptation (TTA) enables pretrained multimodal models to accommodate distribution shifts during inference using only unlabeled test data, without recourse to the original training data. Compared with unimodal adaptation, multimodal adaptation is further complicated by the coexistence and interaction of multiple representation perspectives. This complexity is particularly pronounced under modality reliability shifts, which preserve modality availability but degrade information quality, and modality availability shifts, which render the corresponding modality entirely inaccessible. Category-conditional Gaussian modeling offers a natural starting point for handling such distributional variations, having proven effective in unimodal test-time adaptation for characterizing representation distributions, yet its extension to multimodal settings requires a more elaborate formulation. To this end, we propose Adaptive Probabilistic Gaussian Calibration (AdaPGC), which extends category-conditional Gaussian modeling across modality-specific and multimodal fusion representations for probabilistic calibration under multimodal distribution shifts. AdaPGC continuously updates category-conditional Gaussian models over modality-specific and fusion representations, using the resulting probabilistic predictions to calibrate source model predictions and guide reliability based rectification, while exploiting their joint distributions to recover unavailable fusion representations. Extensive experiments under diverse modality corruption and missingness conditions demonstrate that AdaPGC effectively improves robustness under both modality reliability and availability shifts. Our source code will be released at https://github.com/XuJinglinn/AdaPGC-plus.
Figures & tables
Fig. 1: Illustration of modality reliability and availability shifts in multimodal TTA. Under reliability shifts, corrupted modalities distort the corresponding modality-specific representations and consequently affect the multimodal fusion representation. Under availability shifts, missing modalities make the corresponding modality-specific representations unavailable and prevent the fusion representation from being directly constructed, motivating its recovery from the remaining modality.
Fig. 2: Overview of the proposed multimodal test-time adaptation framework. The input consists of paired video and audio streams affected by reliability shifts (modality corruption) or availability shifts (missing video/audio). For complete instances, the pretrained source model feeds the fused representation zF into the shared classifier to obtain source logits s ; for incomplete instances, it instead uses the representation of the available modality, zV or zA . The category-conditional Gaussian means and priors are initialized from the pretrained classifier, while the covariance and cross-covariance matrices are initialized as identity matrices. During adaptation, these parameters are updated using target-stream statistics to define the discriminants gV , gA , and gF and conditional estimators EV→F and EA→F . For incomplete instances, the available modality representation is used to estimate category-conditional fused representations, whose Gaussian predictions are combined with the unimodal predictions to obtain calibration logits r . For complete instances, the fused Gaussian discriminant produces the calibration logits r=gF(zF) , while discrepancies between unimodal and fused Gaussian predictions guide reliability-aware one-sided alignment.
θt←θt−1−η∇θ(wgLgt+wcLct).
Algorithm 1 Streaming Test-Time Adaptation of AdaPGC
Fig. 3: Sensitivity analysis of λ on Kinetics50. (a) Mean accuracy under audio corruption, video corruption, and missing-modality settings. (b–d) Accuracy under representative audio corruptions, video corruptions, and missing-modality conditions, respectively.
Fig. 4: Sensitivity of AdaPGC to wc and wg on Kinetics50. Rows correspond to audio corruption (a–d), video corruption (e–h), and missing-modality settings (i–l). The first column shows the accuracy averaged over all conditions in each setting, while the remaining columns present representative conditions.
Fig. 5: Sensitivity to the relative weighting of the modality-specific log-posterior and the recovery logits on Kinetics50. For incomplete samples, the Gaussian calibration logits are computed as r=(1−β)gm→F+βgm . (a) Mean accuracy over all 15 missing-modality conditions. (b) Mean accuracy under each modality-missing setting, averaged over missing rates of 50%–90%. Horizontal dashed lines indicate the results using only gm , while red stars mark the adopted equal weighting ( β=0.5 ).
Fig. 6: Grad-CAM visualizations of the backbone and AdaPGC under twelve visual corruptions on Kinetics50-C. Each example shows the clean frame, corrupted input, and prediction-specific Grad-CAM overlays for the backbone and AdaPGC. GT denotes the ground-truth category; red and green labels indicate incorrect and correct predictions, respectively. In these selected correction cases, AdaPGC generally emphasizes action-related regions more clearly, whereas backbone responses are often dispersed or concentrated on the background. Maps are independently normalized for each branch’s predicted category, with warmer colors indicating larger relative positive contributions.
Fig. 7: Cumulative accuracy during test-time adaptation on Kinetics50. Rows correspond to audio corruption (a–d), video corruption (e–h), and missing-modality settings (i–l). A-miss and V-miss denote missing audio and visual modalities, respectively, with percentages indicating the missing rates. Each curve shows the accuracy over all test samples processed so far.
Fig. 8: Visualization of fusion representation recovery under multimodal missingness. Panels (a–b) and (c–d) show the 80% and 90% missing-both settings, respectively, using the available video or audio modality as the recovery source. Colors and fill styles distinguish representation types, marker shapes denote semantic categories, and gray lines connect each recovered representation to its paired clean ground truth.
Fig. 9: Unreliable modality identification accuracy on Kinetics50-C and VGGSound-C under video and audio corruptions. The dashed horizontal line indicates the 50% chance level, and error bars represent 95% confidence intervals.
Fig. 10: Effect of batch size on Kinetics50. (a–c) Mean accuracy under audio corruption, video corruption, and modality-missing settings, respectively. Darker bars and bold values indicate the best performance in each setting, while stars mark the adopted batch size of 16.
Multi-modal test-time adaptation (TTA) enhances the resilience of benchmark multi-modal models against distribution shifts by leveraging the unlabeled target data during inference. Despite the documented success, the advancement of multi-modal TTA methodologies has been impeded by a persistent limitation, i.e., the lack of explicit modeling of category-conditional distributions, which is crucial for yielding accurate predictions and reliable decision boundaries. Canonical Gaussian discriminant analysis (GDA) provides a vanilla modeling of category-conditional distributions and achieves moderate advancement in uni-modal contexts. However, in multi-modal TTA scenario, the inherent modality distribution asymmetry undermines the effectiveness of modeling the category-conditional distribution via the canonical GDA. To this end, we introduce a tailored probabilistic Gaussian model for multi-modal TTA to explicitly model the category-conditional distributions, and further propose an adaptive contrastive asymmetry rectification technique to counteract the adverse effects arising from modality asymmetry, thereby deriving calibrated predictions and reliable decision boundaries. Extensive experiments across diverse benchmarks demonstrate that our method achieves state-of-the-art performance under a wide range of distribution shifts. The code is available at https://github.com/XuJinglinn/AdaPGC.
Jinglin Xu, Yi Li, Chuxiong Sun +3
Institute of Software Chinese Academy of Sciences · University of Chinese Academy of Sciences · National Defense University
Multimodal test-time adaptation (TTA) aims to adapt a pretrained multimodal model online to distribution shift across modalities using unlabeled test data, showing broad potential in real-world applications. However, existing methods primarily focus on adjusting fused features to bridge the source-target gap, lacking explicit control over intermediate representation misalignment, which is a key driver of performance drop under distribution shift. In this work, we tackle this challenge from the perspective of representation engineering. Unlike previous TTA methods that update fusion weights in place, we propose FourIer Representation Editor (FIRE), a novel multimodal TTA approach that directly edits semantically rich intermediate representations. Specifically, we first adopt representation editors into each intermediate layer of the unimodal encoders, enabling layer-wise calibration of unimodal representations. To further enhance the diversity and stability of the low-rank editing subspaces, each representation editor performs frequency domain mixing via the fast Fourier transform to construct structured bases. Moreover, we introduce multi-level adaptation objectives to optimize these editors, jointly promoting cross-modal semantic alignment, source-target statistical alignment, and asymmetric prediction consistency. In this way, FIRE yields aligned unimodal representations for fusion and further improves prediction reliability. Extensive experiments on two widely used multimodal benchmarks under various corruption types demonstrate the superiority of FIRE over existing multimodal TTA methods.
Longfei Huang, Xiangyu Wu, Yang Yang
Nanjing University of Science and Technology · Alibaba Group
Vision-language models transfer well in zero-shot settings, but at deployment the visual and textual branches often shift asymmetrically. Under this condition, entropy-based test-time adaptation can sharpen the fused posterior while increasing error, because an unreliable modality may still dominate fusion. We study this failure mode through a majorization view of multimodal posteriors and cast adaptation as a constrained de-mixing problem on the fused prediction. Based on this view, we propose MG-MTTA, which keeps the backbone frozen and updates only a lightweight gate or adapter. The objective combines fused-posterior entropy minimization with a reliability-aware gate prior built from anchor-based modality consistency and cross-modal conflict. Our analysis gives conditions under which entropy reduction preserves the correct ranking and a threshold that characterizes modality-dominance failure. On the ImageNet-based benchmark, MG-MTTA improves top-1 accuracy from 57.97 to 66.51 under semantics-preserving textual shift and from 21.68 to 26.27 under joint visual-textual shift, while remaining competitive in the visual-only benchmark. These results show that multimodal test-time adaptation should control modality reliability, not just prediction entropy.
Lixian Chen, Yanhui Chen, Junyi Lin
Guangdong University of Technology · Guangzhou, China