Decoupling Stability and Plasticity for Multi-Modal Test-Time Adaptation

Yongbo He; Zirun Guo; Tao Jin

cs.CVFeb 28, 2026

Decoupling Stability and Plasticity for Multi-Modal Test-Time Adaptation

Authors: Yongbo He, Zirun Guo, Tao Jin

Organizations: Zhejiang University

Abstract

Adapting pretrained multi-modal models to evolving test-time distributions, known as multi-modal test-time adaptation, presents a significant challenge. Existing methods frequently encounter negative transfer in the unbiased modality and catastrophic forgetting in the biased modality. To address these challenges, we propose Decoupling Adaptation for Stability and Plasticity (DASP), a novel diagnose-then-mitigate framework. Our analysis reveals a critical discrepancy within the unified latent space: the biased modality exhibits substantially higher interdimensional redundancy (i.e., strong correlations across feature dimensions) compared to the unbiased modality. Leveraging this insight, DASP identifies the biased modality and implements an asymmetric adaptation strategy. This strategy employs a decoupled architecture where each modality-specific adapter is divided into stable and plastic components. The asymmetric mechanism works as follows: for the biased modality, which requires plasticity, the plastic component is activated and updated to capture domain-specific information, while the stable component remains fixed. Conversely, for the unbiased modality, which requires stability, the plastic component is bypassed, and the stable component is updated using KL regularization to prevent negative transfer. This asymmetric design enables the model to adapt flexibly to new domains while preserving generalizable knowledge. Comprehensive evaluations on diverse multi-modal benchmarks demonstrate that DASP significantly outperforms state-of-the-art methods.

Figures & tables

Figure 1: Limitations in Multi-Modal TTA. We evaluate changes in source domain performance during continual adaptation, measured as $\Delta=\mathrm{Acc}_{\text{orignal}}-\mathrm{Acc}_{\text{adapted}}$ , for state-of-the-art methods (READ and TSA). Results indicate ongoing degradation in both multi-modal and uni-modal contexts. Performance drops in the biased modality are referred to as catastrophic forgetting , while drops in the unbiased modality are considered negative transfer .

Figure 2: Entropy and confidence statistics on the VGGSound-C with corrupted audio modality. Since audio serves as the dominant modality in this dataset, it continues to display lower entropy and greater confidence, even in the presence of distribution shifts.

Figure 3: Redundancy statistics on Kinetics50-C and VGGSound-C. The corrupted modality demonstrates increased redundancy in feature embeddings. Furthermore, the results underscore a significant correlation between redundancy and accuracy.

Figure 4: The overview of our proposed DASP features a diagnose-then-mitigate framework. It begins by diagnosing the biased modality through redundancy scores, followed by asymmetric adaptation that includes modality-specific updates guided by entropy minimization.

Method	Gauss.	Shot	Impul.	Defoc.	Glass	Mot.	Zoom	Snow	Frost	Fog	Brit.	Contr.	Elas.	Pix.	JPEG	Avg.
	Noise			Blur				Weather				Digital
Source	46.8	48.0	46.9	67.5	62.2	70.8	66.7	61.6	60.3	46.7	75.2	52.1	65.7	66.5	61.9	59.9
$\bullet$ Tent (ICLR’21)	46.3	47.0	46.3	67.2	62.5	71.0	67.6	63.1	61.1	34.9	75.4	51.6	66.8	67.2	62.7	59.4
$\bullet$ EATA (ICML’22)	46.8	47.6	47.1	67.6	62.7	70.7	66.7	62.2	63.0	49.2	75.2	52.4	65.9	67.0	59.4	60.1
$\bullet$ SAR (ICLR’23)	46.7	47.4	46.8	67.0	61.9	70.4	66.4	61.4	60.6	46.0	75.2	52.1	65.7	66.4	62.0	59.8
$\bullet$ READ (ICLR’24)	49.4	49.7	49.0	68.0	65.1	71.2	69.0	64.5	64.4	57.4	75.5	53.6	68.3	68.0	65.1	62.5

Table 1: Episodic Adaptation. Comparison with SOTA methods on Kinetics50-C with video corruptions regarding Accuracy (%,

\uparrow

) .

	Noise			Weather				Noise			Weather
Method	Gauss.	Traff.	Crowd.	Rain	Thund.	Wind	Avg.	Gauss.	Traff.	Crowd.	Rain	Thund.	Wind	Avg.
Source	73.7	65.5	67.9	70.3	67.9	70.3	69.3	37.0	25.5	16.8	21.6	27.3	25.5	25.6
$\bullet$ Tent (ICLR’21)	73.9	67.4	69.2	70.4	66.5	70.5	69.6	10.6	2.6	1.8	2.8	5.3	4.1	4.5
$\bullet$ EATA (ICML’22)	73.7	66.1	68.5	70.2	67.3	70.2	69.4	39.2	26.1	22.9	26.0	31.7	30.4	29.4
$\bullet$ SAR (ICLR’23)	73.7	65.4	68.2	69.9	67.0	70.2	69.1	37.4	9.5	9.0	12.6	23.7	20.1	20.4
$\bullet$ READ (ICLR’24)	74.1	69.0	69.7	71.1	71.8	70.7	71.1	40.4	28.9	26.0	30.9	36.0	32.6	32.4

Table 2: Episodic Adaptation. Comparison with SOTA methods on Kinetics50-C (left) and VGGSound-C (right) with audio corruptions regarding Accuracy (%,

\uparrow

) .

Method	Gauss.	Shot	Impul.	Defoc.	Glass	Mot.	Zoom	Snow	Frost	Fog	Brit.	Contr.	Elas.	Pix.	JPEG	Avg.
	Noise			Blur				Weather				Digital
	$t\;\xrightarrow{\hskip 512.1496pt}$
Source	46.8	48.0	46.9	67.5	62.2	70.8	66.7	61.6	60.3	46.7	75.2	52.1	65.7	66.5	61.9	59.9
$\bullet$ Tent (ICLR’21)	46.2	45.8	43.1	62.5	61.9	65.6	64.1	55.0	56.9	40.1	40.4	23.6	14.6	4.5	3.4	41.8
$\bullet$ EATA (ICML’22)	47.0	47.9	47.0	67.4	63.3	71.6	68.5	61.8	65.0	59.3	74.2	53.0	69.9	64.9	61.9	61.5
$\bullet$ SAR (ICLR’23)	47.0	48.0	46.8	66.2	62.2	70.7	66.8	60.2	60.6	49.8	74.9	51.0	66.5	65.0	61.1	59.8

Table 3: Continual Adaptation. Comparison with SOTA methods on Kinetics50-C with video corruptions regarding Accuracy (%,

\uparrow

) .

	Noise			Weather				Noise			Weather
Method	Gauss.	Traff.	Crowd.	Rain	Thund.	Wind	Avg.	Gauss.	Traff.	Crowd.	Rain	Thund.	Wind	Avg.
	$t\;\xrightarrow{\hskip 170.71652pt}$							$t\;\xrightarrow{\hskip 170.71652pt}$
Source	73.7	65.5	67.9	70.3	67.9	70.3	69.3	37.0	25.5	16.8	21.6	27.3	25.5	25.6
$\bullet$ Tent (ICLR’21)	73.9	67.9	71.1	70.1	71.7	70.6	70.9	10.7	1.2	0.4	0.3	0.5	0.3	2.3
$\bullet$ EATA (ICML’22)	73.8	67.6	70.2	70.5	70.3	70.2	70.4	40.4	33.0	35.5	34.9	43.6	37.1	37.4
$\bullet$ SAR (ICLR’23)	73.8	66.1	68.4	70.0	69.0	70.0	69.6	37.9	8.9	7.4	15.1	13.5	18.2	16.8

Table 4: Continual Adaptation. Comparison with SOTA methods on Kinetics50-C (left) and VGGSound-C (right) with audio corruptions regarding Accuracy (%,

\uparrow

) .

Method	Video-C	Audio-C	Interleaved-C	Avg.
Source	59.9	69.3	62.6	63.9
$\bullet$ Ours	65.3	72.5	67.3	68.4
w/o stable adapter	64.6	72.3	65.2	67.4
w/o plastic adapter	62.2	71.4	64.4	66.0
w/o asym. adaptation	54.3	71.5	61.8	62.5
w/ asym. adaptation (opposite)	52.3	71.8	61.5	61.9

Table 5: Ablation Study. Analysis of component contributions in our method on Kinetics50-C regarding Accuracy (%,

\uparrow

) .

	Gauss.	Shot	Gauss.	Impul.	Defoc.	Traff.	Glass	Mot.	Crowd.	Zoom	Snow	Frost	Rain	Fog	Brit.	Thund.	Contr.	Elas.	Wind	Pix.	JPEG	Avg.
Method	V-C		A-C	V-C		A-C	V-C		A-C	V-C			A-C	V-C		A-C	V-C		A-C	V-C
	$t\;\xrightarrow{\hskip 995.84645pt}$
Source	46.8	48.0	73.7	46.9	67.5	65.5	62.2	70.8	67.9	66.7	61.6	60.3	70.3	46.7	75.2	67.9	52.1	65.7	70.3	66.5	61.9	62.6
$\bullet$ Tent (ICLR’21)	46.3	45.8	73.2	42.9	61.6	66.1	62.5	66.6	68.5	64.1	53.7	56.9	65.5	45.6	59.1	23.6	32.6	29.7	22.5	5.1	3.6	47.4
$\bullet$ EATA (ICML’22)	47.0	47.9	74.0	47.2	67.2	66.8	65.0	70.8	69.7	68.7	59.7	64.1	69.5	58.4	73.1	68.7	53.0	69.1	69.1	64.9	61.8	63.6
$\bullet$ SAR (ICLR’23)	47.0	48.0	73.8	46.9	66.3	65.7	62.3	70.6	67.8	66.8	59.8	59.9	69.5	50.3	74.7	67.6	50.9	66.7	69.5	64.5	60.8	62.4

Table 6: Continual Adaptation. Comparison with SOTA methods on Kinetics50-C with interleaved modality corruption regarding Accuracy (%,

\uparrow

) .

	Gauss.	Shot	Gauss.	Impul.	Defoc.	Traff.	Glass	Mot.	Crowd.	Zoom	Snow	Frost	Rain	Fog	Brit.	Thund.	Contr.	Elas.	Wind	Pix.	JPEG	Avg.
Method	V-C		A-C	V-C		A-C	V-C		A-C	V-C			A-C	V-C		A-C	V-C		A-C	V-C
	$t\;\xrightarrow{\hskip 995.84645pt}$
Source	52.8	52.7	37.0	52.7	57.2	25.5	57.2	58.7	16.8	57.0	56.4	56.6	21.6	55.6	58.0	27.3	53.7	56.9	25.5	55.8	56.9	47.2
$\bullet$ Tent (ICLR’21)	52.7	52.4	5.4	49.3	52.3	0.7	51.3	52.8	0.6	52.2	50.4	51.5	0.5	51.1	51.4	1.5	47.6	47.2	0.4	36.8	11.3	34.3
$\bullet$ EATA (ICML’22)	53.1	53.4	39.7	53.0	57.1	33.5	57.1	58.9	35.9	58.0	56.2	57.3	34.2	57.4	59.6	42.9	54.4	58.2	36.3	57.2	58.4	51.0
$\bullet$ SAR (ICLR’23)	52.9	53.0	35.5	52.7	57.0	11.2	56.9	58.4	10.3	57.8	56.2	56.9	13.5	55.6	57.9	18.6	53.3	56.3	21.3	54.6	55.6	45.0

Table 7: Continual Adaptation. Comparison with SOTA methods on VGGSound-C with interleaved modality corruption regarding Accuracy (%,

\uparrow

) .

Figure 5: Sensitivity Analysis of Hyper-parameters: Batch Size ( $\mathbf{B}$ ), Redundancy Threshold ( $\mathbf{\delta}$ ) and Loss Coefficents ( $\lambda_{\text{ent}}$ , $\lambda_{\text{kl}}$ ).

Figure 6: Accuracy vs . Throughput and Memory Usage. Compared to baselines, our method demonstrates superior performance with higher efficiency (observing comparable or lower computational cost and higher inference speed) on Kinetics50-C.

Figure 7: The illustration of uni-modal continual corruption and interleaved modality continual corruption.

Figure 8: Redundancy vs . Severity Level. The redundancy score is evaluated across various severity levels of Kinetics50-C (Gaussian noise) and the original Kinetics50 validation set (source).

Figure 9: Redundancy vs . Batch Size. We investigate the correlation between redundancy score and batch size on VGGSound-C with audio corruptions.

Method	Gauss.	Shot	Impul.	Defoc.	Glass	Mot.	Zoom	Snow	Frost	Fog	Brit.	Contr.	Elas.	Pix.	JPEG	Avg.
	Noise			Blur				Weather				Digital
Source	52.8	52.7	52.7	57.2	57.2	58.7	57.0	56.4	56.6	55.6	58.0	53.7	56.9	55.8	56.9	56.0
$\bullet$ Tent (ICLR’21)	52.7	52.7	52.7	56.7	56.7	57.9	57.1	55.9	56.3	56.3	58.4	54.0	57.4	56.2	56.7	55.8
$\bullet$ EATA (ICML’22)	53.0	52.8	53.0	57.0	57.0	58.1	57.2	56.3	56.8	56.8	58.7	54.1	57.6	56.4	57.0	56.1
$\bullet$ SAR (ICLR’23)	52.9	52.8	52.9	57.1	57.1	57.7	57.6	56.6	55.7	56.7	58.6	54.0	57.1	56.3	56.9	56.0
$\bullet$ READ (ICLR’24)	53.6	53.6	53.5	57.9	57.7	59.4	58.8	56.8	57.1	56.9	59.9	53.8	58.6	57.1	57.6	56.9

Table 8: Episodic Adaptation. Comparison with SOTA methods on VGGSound-C with video corruptions (severity level 5) regarding Accuracy (%,

\uparrow

) .

Method	Gauss.	Shot	Impul.	Defoc.	Glass	Mot.	Zoom	Snow	Frost	Fog	Brit.	Contr.	Elas.	Pix.	JPEG	Avg.
	Noise			Blur				Weather				Digital
	$t\;\xrightarrow{\hskip 654.41338pt}$
Source	52.8	52.7	52.7	57.2	57.2	58.7	57.0	56.4	56.6	55.6	58.0	53.7	56.9	55.8	56.9	56.0
$\bullet$ Tent (ICLR’21)	52.7	52.4	51.5	54.4	54.0	55.2	54.8	52.1	52.9	52.8	52.8	50.6	52.7	52.2	51.8	52.9
$\bullet$ EATA (ICML’22)	53.0	53.4	53.5	56.7	57.1	58.6	58.5	55.8	56.8	57.3	58.0	55.3	57.6	57.1	57.5	56.4
$\bullet$ SAR (ICLR’23)	52.9	53.0	53.0	56.9	56.7	58.5	57.5	56.0	56.8	56.0	58.2	54.1	57.1	55.4	56.3	55.9

Table 9: Continual Adaptation. Comparison with SOTA methods on VGGSound-C with video corruptions (severity level 5) regarding Accuracy (%,

\uparrow

) .

Method	Gauss.	Shot	Impul.	Defoc.	Glass	Mot.	Zoom	Snow	Frost	Fog	Brit.	Contr.	Elas.	Pix.	JPEG	Avg.
	Noise			Blur				Weather				Digital
Source	46.8	48.0	46.9	67.5	62.2	70.8	66.7	61.6	60.3	46.7	75.2	52.1	65.7	66.5	61.9	59.9
$\bullet$ READ (ICLR’24)	49.4	49.7	49.0	68.0	65.1	71.2	69.0	64.5	64.4	57.4	75.5	53.6	68.3	68.0	65.1	62.5
$\bullet$ ABPEM (AAAI’25)	50.6	51.1	50.5	68.7	66.6	72.6	69.6	64.4	66.2	60.5	76.0	55.3	69.5	69.2	66.2	63.8
$\bullet$ SuMi (ICLR’25)	50.1	50.7	50.4	68.2	65.6	72.2	69.7	65.7	67.0	56.5	77.1	55.2	69.3	71.2	68.9	63.9
$\bullet$ TSA (ICML’25)	50.7	51.1	50.4	67.9	67.1	71.7	69.2	65.5	66.2	61.3	75.2	56.2	69.5	68.8	66.6	63.8

Table 10: Episodic Adaptation. Comparison with MM-TTA methods on Kinetics50-C with video corruptions (severity level 5) regarding Accuracy (%,

\uparrow

) .

Test-Time Adaptation

Test-time adaptation updates or recalibrates pretrained models at inference to handle distribution shifts, often without target labels.

See all 322 papers

Multi-modal Test-time Adaptation via Adaptive Probabilistic Gaussian Calibration
Apr 21, 2026Jinglin Xu, Yi Li, Chuxiong Sun +3Test-Time Adaptation Probability Calibration
Institute of Software Chinese Academy of Sciences · University of Chinese Academy of Sciences · National Defense University
Representation Editing for Multimodal Test-Time Adaptation
Sep 26, 2026Longfei Huang, Xiangyu Wu, Yang YangTest-Time Adaptation Multimodal Robustness
Nanjing University of Science and Technology · Alibaba Group
Multi-Hypothesis Test-Time Adaptation to Mitigate Underspecification
Jun 30, 2026Afshar Shamsi, Xiao-Yu Guo, Hamid Alinejad-Rokny +3Distribution Shift Robustness Test-Time Adaptation
Concordia University, Canada · Australian Institute for Machine Learning, Australia · University of New South Wales, Australia +2

Apr 21, 2026J/K move · Enter open · S save

Multi-modal Test-time Adaptation via Adaptive Probabilistic Gaussian Calibration

Jinglin Xu, Yi Li, Chuxiong Sun, Xiao Xu+2

Institute of Software Chinese Academy of Sciences · University of Chinese Academy of Sciences · National Defense University

Test-Time Adaptation Probability Calibration Multimodal Robustness

Open paper AskPDF arXiv

Multi-modal test-time adaptation (TTA) enhances the resilience of benchmark multi-modal models against distribution shifts by leveraging the unlabeled target data during inference. Despite the documented success, the advancement of multi-modal TTA methodologies has been impeded by a persistent limitation, i.e., the lack of explicit modeling of category-conditional distributions, which is crucial for yielding accurate predictions and reliable decision boundaries. Canonical Gaussian discriminant analysis (GDA) provides a vanilla modeling of category-conditional distributions and achieves moderate advancement in uni-modal contexts. However, in multi-modal TTA scenario, the inherent modality distribution asymmetry undermines the effectiveness of modeling the category-conditional distribution via the canonical GDA. To this end, we introduce a tailored probabilistic Gaussian model for multi-modal TTA to explicitly model the category-conditional distributions, and further propose an adaptive contrastive asymmetry rectification technique to counteract the adverse effects arising from modality asymmetry, thereby deriving calibrated predictions and reliable decision boundaries. Extensive experiments across diverse benchmarks demonstrate that our method achieves state-of-the-art performance under a wide range of distribution shifts. The code is available at https://github.com/XuJinglinn/AdaPGC.

Abstract

Test-Time Adaptation

Explore similar work

Multi-modal Test-time Adaptation via Adaptive Probabilistic Gaussian Calibration

Representation Editing for Multimodal Test-Time Adaptation

Multi-Hypothesis Test-Time Adaptation to Mitigate Underspecification