Multimodal test-time adaptation (TTA) aims to adapt a pretrained multimodal model online to distribution shift across modalities using unlabeled test data, showing broad potential in real-world applications. However, existing methods primarily focus on adjusting fused features to bridge the source-target gap, lacking explicit control over intermediate representation misalignment, which is a key driver of performance drop under distribution shift. In this work, we tackle this challenge from the perspective of representation engineering. Unlike previous TTA methods that update fusion weights in place, we propose FourIer Representation Editor (FIRE), a novel multimodal TTA approach that directly edits semantically rich intermediate representations. Specifically, we first adopt representation editors into each intermediate layer of the unimodal encoders, enabling layer-wise calibration of unimodal representations. To further enhance the diversity and stability of the low-rank editing subspaces, each representation editor performs frequency domain mixing via the fast Fourier transform to construct structured bases. Moreover, we introduce multi-level adaptation objectives to optimize these editors, jointly promoting cross-modal semantic alignment, source-target statistical alignment, and asymmetric prediction consistency. In this way, FIRE yields aligned unimodal representations for fusion and further improves prediction reliability. Extensive experiments on two widely used multimodal benchmarks under various corruption types demonstrate the superiority of FIRE over existing multimodal TTA methods.
Figures & tables
Figure 1: Overview of representation shift and editing on Kinetics50-C. (a) t-SNE visualization of multimodal representations from the source, target, and edited target domains. The star denotes the feature centroid. (b) and (c) Differences in intermediate-layer feature statistics between the source and target domains, where the mean characterizes the distribution center and the standard deviation (std) characterizes its dispersion. (d) Visualization of clean, noisy, and edited representations of the same target sample. Our method effectively mitigates the representation shift.
Figure 2: Overview of FIRE. The entire model remains frozen, with only editors being updated. The editor aims to identify an editing vector that modifies the representations only within the linear subspace spanned by the rows of U^ . “Model & Editors” is the same pretrained model and editors. In particular, we illustrate the representation editing process in a 3-dimensional space.
Noise
Blur
Weather
Digital
Method
Gauss.
Shot
Impul.
Defoc.
Glass
Motion
Zoom
Snow
Frost
Fog
Bright.
Contr.
Elast.
Pixel.
Jpeg
Avg.
Source
48.2
50.0
49.2
67.7
61.6
70.6
66.1
60.9
60.7
44.7
75.9
51.8
65.5
68.7
66.1
60.5
∙ Tent ICLR21
48.2
49.8
48.7
67.7
62.1
70.8
67.2
61.8
61.4
33.7
76.0
51.2
66.6
69.6
66.9
60.1
∙ T3A NeurIPS21
47.8
49.0
48.2
62.8
59.3
65.5
62.0
60.2
57.5
50.7
69.8
47.6
62.8
65.7
64.5
58.2
∙ EATA ICML22
48.7
50.4
49.6
67.8
63.2
70.8
67.5
62.5
62.5
47.9
76.1
52.2
66.9
69.7
67.4
61.5
∙ SAR ICLR23
48.5
50.2
49.2
67.8
63.8
70.9
67.9
63.1
62.7
38.7
76.1
52.2
67.1
69.8
67.4
61.0
Table 1: Comparison with SOTA methods on Kinetics50-C (top) and VGGSound-C (bottom) benchmarks with corrupted video modality (severity level 5).
Noise
Weather
Noise
Weather
Method
Gauss.
Traff.
Crowd
Rain
Thund.
Wind
Avg.
Gauss.
Traff.
Crowd
Rain
Thund.
Wind
Avg.
Source
74.3
65.3
68.0
70.3
68.0
70.5
69.4
37.3
21.2
16.9
21.8
27.3
25.7
25.0
∙ Tent ICLR21
74.6
67.4
69.5
70.8
67.6
71.2
70.2
10.8
2.8
1.8
2.9
5.6
3.9
4.6
∙ T3A NeurIPS21
71.4
63.0
65.4
66.8
66.1
67.8
66.8
39.8
28.2
24.8
28.6
34.7
30.6
31.1
∙ EATA ICML22
74.6
67.3
69.4
70.8
69.8
71.0
70.5
40.2
30.0
27.8
29.7
36.5
32.2
32.7
∙ SAR ICLR23
74.6
67.0
69.2
70.9
69.5
70.9
70.3
30.4
5.5
8.0
9.3
32.5
17.2
17.1
Table 2: Comparison with SOTA methods on Kinetics50-C (left) and VGGSound-C (right) benchmarks with corrupted audio modality (severity level 5).
Noise
Blur
Weather
Digital
Method
Gauss.
Shot
Impul.
Defoc.
Glass
Motion
Zoom
Snow
Frost
Fog
Bright.
Contr.
Elast.
Pixel.
Jpeg
Avg.
Source
13.1
14.1
13.3
37.2
37.4
45.3
41.8
29.4
32.6
20.4
55.2
18.3
42.5
38.8
37.8
31.8
∙ Tent ICLR21
9.1
9.7
9.1
32.5
34.1
43.5
40.2
23.2
28.3
13.2
55.1
13.7
40.7
34.7
35.0
28.1
∙ T3A NeurIPS21
12.3
13.8
12.7
34.7
35.5
42.0
39.3
29.2
28.7
20.9
51.9
17.1
40.6
37.5
37.4
30.2
∙ EATA ICML22
12.9
14.0
13.1
38.1
38.7
46.9
43.1
30.6
33.0
20.2
56.5
18.2
43.7
40.7
39.0
32.6
∙ SAR ICLR23
11.8
12.8
11.9
37.4
38.2
46.3
43.1
29.8
33.0
17.4
56.0
16.0
43.5
39.5
38.2
31.7
Table 3: Comparison with SOTA methods on Kinetics50-C benchmarks for multimodal domain shifts (severity level 5).
Noise
Weather
Method
Gauss.
Traff.
Crowd
Rain
Thund.
Wind
Avg.
Source
17.1
6.4
5.4
6.0
13.5
8.8
9.5
∙ Tent
3.2
0.9
0.8
0.9
2.8
1.3
1.6
∙ T3A
20.6
9.2
7.8
8.6
17.4
10.9
12.4
∙ EATA
21.5
7.7
7.1
7.3
17.3
11.9
12.1
∙ SAR
10.7
1.8
1.6
2.3
12.8
3.1
5.4
Table 4: Comparison with TTA baseline methods on VGGSound-C for multimodal domain shifts (severity level 5).
Table 5: Impact of rank (left) and editing depth (right) on Kinetics50-C dataset. Rank audio video both Depth audio video both 2 72.8 67.3 44.5 First Layer 71.3 63.3 36.9 4 73.3 68.2 46.3 First Half 72.7 67.7 45.2 6 73.4 68.6 47.0 Latter Half 72.7 65.7 42.2 8 73.5 69.0 47.5 Last Layer 71.2 63.3 36.5 10 73.5 69.2 48.0 All Layers 73.4 68.6 47.0 Table 6: The number of trainable parameters and performance on Kinetics50-C dataset. Method layer audio video both Params (M) Tent Normal 70.2 60.1 28.1 0.218 READ Attention 71.5 63.4 35.9 1.772 TSA Adapter 71.5 64.5 41.1 1.180 BriMPR Prompt 72.0 65.9 40.9 0.169 FIRE (Rank2) Editor 72.8 67.3 44.5 0.068
Table 5: Impact of rank (left) and editing depth (right) on Kinetics50-C dataset.
Kinetics50-C
VGGSound-C
Method
audio
video
both
audio
video
both
Source
69.4
60.5
31.8
25.0
56.2
9.5
w/o FFT
72.8
67.5
46.1
36.7
58.0
21.6
w/o Lcl
72.9
67.9
46.8
36.6
58.0
21.7
w/o Lstat
71.4
63.6
36.6
33.1
56.9
15.4
w/o Lpc
72.4
67.0
44.1
35.7
57.6
21.0
Table 7: Ablation study for different components of FIRE.
Multi-modal test-time adaptation (TTA) enhances the resilience of benchmark multi-modal models against distribution shifts by leveraging the unlabeled target data during inference. Despite the documented success, the advancement of multi-modal TTA methodologies has been impeded by a persistent limitation, i.e., the lack of explicit modeling of category-conditional distributions, which is crucial for yielding accurate predictions and reliable decision boundaries. Canonical Gaussian discriminant analysis (GDA) provides a vanilla modeling of category-conditional distributions and achieves moderate advancement in uni-modal contexts. However, in multi-modal TTA scenario, the inherent modality distribution asymmetry undermines the effectiveness of modeling the category-conditional distribution via the canonical GDA. To this end, we introduce a tailored probabilistic Gaussian model for multi-modal TTA to explicitly model the category-conditional distributions, and further propose an adaptive contrastive asymmetry rectification technique to counteract the adverse effects arising from modality asymmetry, thereby deriving calibrated predictions and reliable decision boundaries. Extensive experiments across diverse benchmarks demonstrate that our method achieves state-of-the-art performance under a wide range of distribution shifts. The code is available at https://github.com/XuJinglinn/AdaPGC.
Jinglin Xu, Yi Li, Chuxiong Sun +3
Institute of Software Chinese Academy of Sciences · University of Chinese Academy of Sciences · National Defense University
Adapting pretrained multi-modal models to evolving test-time distributions, known as multi-modal test-time adaptation, presents a significant challenge. Existing methods frequently encounter negative transfer in the unbiased modality and catastrophic forgetting in the biased modality. To address these challenges, we propose Decoupling Adaptation for Stability and Plasticity (DASP), a novel diagnose-then-mitigate framework. Our analysis reveals a critical discrepancy within the unified latent space: the biased modality exhibits substantially higher interdimensional redundancy (i.e., strong correlations across feature dimensions) compared to the unbiased modality. Leveraging this insight, DASP identifies the biased modality and implements an asymmetric adaptation strategy. This strategy employs a decoupled architecture where each modality-specific adapter is divided into stable and plastic components. The asymmetric mechanism works as follows: for the biased modality, which requires plasticity, the plastic component is activated and updated to capture domain-specific information, while the stable component remains fixed. Conversely, for the unbiased modality, which requires stability, the plastic component is bypassed, and the stable component is updated using KL regularization to prevent negative transfer. This asymmetric design enables the model to adapt flexibly to new domains while preserving generalizable knowledge. Comprehensive evaluations on diverse multi-modal benchmarks demonstrate that DASP significantly outperforms state-of-the-art methods.
Vision-language models suffer significant accuracy drops under common corruptions. While test-time adaptation (TTA) successfully mitigates these shifts in CLIP-like contrastive models, its application to generative multimodal large language models (MLLMs) remains underexplored and uniquely challenging due to their sequential token generation. In this work, we demonstrate that representation-based TTA fails to recover performance in MLLMs. Our experiments reveal that even when violating standard TTA assumptions by leveraging target data offline, sample-blind feature alignment recovers only marginal accuracy. We attribute this failure to the nature of corruption shifts: unlike CLIP-like architectures, which benefit from a shared domain-shift direction within a single pooled classification embedding, MLLM visual token sequence shifts are nearly orthogonal and span the full embedding space.These findings bound what any sample-blind, position-shared correction at a single encoder site can achieve, establishing that representation-space interventions alone might be insufficient to ensure corruption robustness in MLLMs.
Mikołaj Piórczyński, Wojciech Masarczyk, Damian Sójka +2
IDEAS Research Institute · Warsaw University of Technology · Pozna´n University of Technology +4