Representation Editing for Multimodal Test-Time Adaptation

Longfei Huang; Xiangyu Wu; Yang Yang

cs.LGSep 26, 2026

Representation Editing for Multimodal Test-Time Adaptation

Authors: Longfei Huang, Xiangyu Wu, Yang Yang

Organizations: Nanjing University of Science and Technology · Alibaba Group

Abstract

Multimodal test-time adaptation (TTA) aims to adapt a pretrained multimodal model online to distribution shift across modalities using unlabeled test data, showing broad potential in real-world applications. However, existing methods primarily focus on adjusting fused features to bridge the source-target gap, lacking explicit control over intermediate representation misalignment, which is a key driver of performance drop under distribution shift. In this work, we tackle this challenge from the perspective of representation engineering. Unlike previous TTA methods that update fusion weights in place, we propose FourIer Representation Editor (FIRE), a novel multimodal TTA approach that directly edits semantically rich intermediate representations. Specifically, we first adopt representation editors into each intermediate layer of the unimodal encoders, enabling layer-wise calibration of unimodal representations. To further enhance the diversity and stability of the low-rank editing subspaces, each representation editor performs frequency domain mixing via the fast Fourier transform to construct structured bases. Moreover, we introduce multi-level adaptation objectives to optimize these editors, jointly promoting cross-modal semantic alignment, source-target statistical alignment, and asymmetric prediction consistency. In this way, FIRE yields aligned unimodal representations for fusion and further improves prediction reliability. Extensive experiments on two widely used multimodal benchmarks under various corruption types demonstrate the superiority of FIRE over existing multimodal TTA methods.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of representation shift and editing on Kinetics50-C. (a) t-SNE visualization of multimodal representations from the source, target, and edited target domains. The star denotes the feature centroid. (b) and (c) Differences in intermediate-layer feature statistics between the source and target domains, where the mean characterizes the distribution center and the standard deviation (std) characterizes its dispersion. (d) Visualization of clean, noisy, and edited representations of the same target sample. Our method effectively mitigates the representation shift.

Figure 2: Overview of FIRE. The entire model remains frozen, with only editors being updated. The editor aims to identify an editing vector that modifies the representations only within the linear subspace spanned by the rows of $\hat{{\boldsymbol{U}}}$ . “Model & Editors” is the same pretrained model and editors. In particular, we illustrate the representation editing process in a 3-dimensional space.

Method	Gauss.	Shot	Impul.	Defoc.	Glass	Motion	Zoom	Snow	Frost	Fog	Bright.	Contr.	Elast.	Pixel.	Jpeg	Avg.
	Noise			Blur				Weather				Digital
Source	48.2	50.0	49.2	67.7	61.6	70.6	66.1	60.9	60.7	44.7	75.9	51.8	65.5	68.7	66.1	60.5
$\bullet$ Tent ICLR21	48.2	49.8	48.7	67.7	62.1	70.8	67.2	61.8	61.4	33.7	76.0	51.2	66.6	69.6	66.9	60.1
$\bullet$ T3A NeurIPS21	47.8	49.0	48.2	62.8	59.3	65.5	62.0	60.2	57.5	50.7	69.8	47.6	62.8	65.7	64.5	58.2
$\bullet$ EATA ICML22	48.7	50.4	49.6	67.8	63.2	70.8	67.5	62.5	62.5	47.9	76.1	52.2	66.9	69.7	67.4	61.5
$\bullet$ SAR ICLR23	48.5	50.2	49.2	67.8	63.8	70.9	67.9	63.1	62.7	38.7	76.1	52.2	67.1	69.8	67.4	61.0

Table 1: Comparison with SOTA methods on Kinetics50-C (top) and VGGSound-C (bottom) benchmarks with corrupted video modality (severity level 5).

Method	Gauss.	Traff.	Crowd	Rain	Thund.	Wind	Avg.	Gauss.	Traff.	Crowd	Rain	Thund.	Wind	Avg.
	Noise			Weather				Noise			Weather
Source	74.3	65.3	68.0	70.3	68.0	70.5	69.4	37.3	21.2	16.9	21.8	27.3	25.7	25.0
$\bullet$ Tent ICLR21	74.6	67.4	69.5	70.8	67.6	71.2	70.2	10.8	2.8	1.8	2.9	5.6	3.9	4.6
$\bullet$ T3A NeurIPS21	71.4	63.0	65.4	66.8	66.1	67.8	66.8	39.8	28.2	24.8	28.6	34.7	30.6	31.1
$\bullet$ EATA ICML22	74.6	67.3	69.4	70.8	69.8	71.0	70.5	40.2	30.0	27.8	29.7	36.5	32.2	32.7
$\bullet$ SAR ICLR23	74.6	67.0	69.2	70.9	69.5	70.9	70.3	30.4	5.5	8.0	9.3	32.5	17.2	17.1

Table 2: Comparison with SOTA methods on Kinetics50-C (left) and VGGSound-C (right) benchmarks with corrupted audio modality (severity level 5).

Method	Gauss.	Shot	Impul.	Defoc.	Glass	Motion	Zoom	Snow	Frost	Fog	Bright.	Contr.	Elast.	Pixel.	Jpeg	Avg.
	Noise			Blur				Weather				Digital
Source	13.1	14.1	13.3	37.2	37.4	45.3	41.8	29.4	32.6	20.4	55.2	18.3	42.5	38.8	37.8	31.8
$\bullet$ Tent ICLR21	9.1	9.7	9.1	32.5	34.1	43.5	40.2	23.2	28.3	13.2	55.1	13.7	40.7	34.7	35.0	28.1
$\bullet$ T3A NeurIPS21	12.3	13.8	12.7	34.7	35.5	42.0	39.3	29.2	28.7	20.9	51.9	17.1	40.6	37.5	37.4	30.2
$\bullet$ EATA ICML22	12.9	14.0	13.1	38.1	38.7	46.9	43.1	30.6	33.0	20.2	56.5	18.2	43.7	40.7	39.0	32.6
$\bullet$ SAR ICLR23	11.8	12.8	11.9	37.4	38.2	46.3	43.1	29.8	33.0	17.4	56.0	16.0	43.5	39.5	38.2	31.7

Table 3: Comparison with SOTA methods on Kinetics50-C benchmarks for multimodal domain shifts (severity level 5).

Method	Gauss.	Traff.	Crowd	Rain	Thund.	Wind	Avg.
	Noise			Weather
Source	17.1	6.4	5.4	6.0	13.5	8.8	9.5
$\bullet$ Tent	3.2	0.9	0.8	0.9	2.8	1.3	1.6
$\bullet$ T3A	20.6	9.2	7.8	8.6	17.4	10.9	12.4
$\bullet$ EATA	21.5	7.7	7.1	7.3	17.3	11.9	12.1
$\bullet$ SAR	10.7	1.8	1.6	2.3	12.8	3.1	5.4

Table 4: Comparison with TTA baseline methods on VGGSound-C for multimodal domain shifts (severity level 5).

Table 5: Impact of rank (left) and editing depth (right) on Kinetics50-C dataset. Rank audio video both Depth audio video both 2 72.8 67.3 44.5 First Layer 71.3 63.3 36.9 4 73.3 68.2 46.3 First Half 72.7 67.7 45.2 6 73.4 68.6 47.0 Latter Half 72.7 65.7 42.2 8 73.5 69.0 47.5 Last Layer 71.2 63.3 36.5 10 73.5 69.2 48.0 All Layers 73.4 68.6 47.0 Table 6: The number of trainable parameters and performance on Kinetics50-C dataset. Method layer audio video both Params (M) Tent Normal 70.2 60.1 28.1 0.218 READ Attention 71.5 63.4 35.9 1.772 TSA Adapter 71.5 64.5 41.1 1.180 BriMPR Prompt 72.0 65.9 40.9 0.169 FIRE (Rank2) Editor 72.8 67.3 44.5 0.068

Table 5: Impact of rank (left) and editing depth (right) on Kinetics50-C dataset.

	Kinetics50-C			VGGSound-C
Method	audio	video	both	audio	video	both
Source	69.4	60.5	31.8	25.0	56.2	9.5
w/o FFT	72.8	67.5	46.1	36.7	58.0	21.6
w/o ${\mathcal{L}}_{cl}$	72.9	67.9	46.8	36.6	58.0	21.7
w/o ${\mathcal{L}}_{stat}$	71.4	63.6	36.6	33.1	56.9	15.4
w/o ${\mathcal{L}}_{pc}$	72.4	67.0	44.1	35.7	57.6	21.0

Table 7: Ablation study for different components of FIRE.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Notation	Description
$N_{{\mathcal{S}}}$	The number of source data.
$N_{{\mathcal{T}}}$	The number of target data.
$y_{i}$	Category label of $i$ -th data.
${\boldsymbol{x}}_{i}^{a}/{\boldsymbol{x}}_{i}^{v}$	Audio / Video data point.
${\mathcal{D}}_{{\mathcal{S}}}/{\mathcal{D}}_{{\mathcal{T}}}$	Source / Target domain.
$r$	$r$ -modality.

Appendix

Table I: Notation Definition

Test-Time Adaptation

Test-time adaptation updates or recalibrates pretrained models at inference to handle distribution shifts, often without target labels.

See all 322 papers

Multi-modal Test-time Adaptation via Adaptive Probabilistic Gaussian Calibration
Apr 21, 2026Jinglin Xu, Yi Li, Chuxiong Sun +3Test-Time Adaptation Probability Calibration
Institute of Software Chinese Academy of Sciences · University of Chinese Academy of Sciences · National Defense University
Decoupling Stability and Plasticity for Multi-Modal Test-Time Adaptation
Feb 28, 2026Yongbo He, Zirun Guo, Tao JinTest-Time Adaptation Test-Time Adaptation of VLMs
Zhejiang University
On the Limits of Representation-Based Test-Time Adaptation for Multimodal LLMs
Oct 14, 2026Mikołaj Piórczyński, Wojciech Masarczyk, Damian Sójka +2Test-Time Adaptation Test-Time Adaptation of VLMs
IDEAS Research Institute · Warsaw University of Technology · Pozna´n University of Technology +4

Apr 21, 2026J/K move · Enter open · S save

Multi-modal Test-time Adaptation via Adaptive Probabilistic Gaussian Calibration

Jinglin Xu, Yi Li, Chuxiong Sun, Xiao Xu+2

Institute of Software Chinese Academy of Sciences · University of Chinese Academy of Sciences · National Defense University

Test-Time Adaptation Probability Calibration Multimodal Robustness

Open paper AskPDF arXiv

Multi-modal test-time adaptation (TTA) enhances the resilience of benchmark multi-modal models against distribution shifts by leveraging the unlabeled target data during inference. Despite the documented success, the advancement of multi-modal TTA methodologies has been impeded by a persistent limitation, i.e., the lack of explicit modeling of category-conditional distributions, which is crucial for yielding accurate predictions and reliable decision boundaries. Canonical Gaussian discriminant analysis (GDA) provides a vanilla modeling of category-conditional distributions and achieves moderate advancement in uni-modal contexts. However, in multi-modal TTA scenario, the inherent modality distribution asymmetry undermines the effectiveness of modeling the category-conditional distribution via the canonical GDA. To this end, we introduce a tailored probabilistic Gaussian model for multi-modal TTA to explicitly model the category-conditional distributions, and further propose an adaptive contrastive asymmetry rectification technique to counteract the adverse effects arising from modality asymmetry, thereby deriving calibrated predictions and reliable decision boundaries. Extensive experiments across diverse benchmarks demonstrate that our method achieves state-of-the-art performance under a wide range of distribution shifts. The code is available at https://github.com/XuJinglinn/AdaPGC.

Abstract

Test-Time Adaptation

Explore similar work

Multi-modal Test-time Adaptation via Adaptive Probabilistic Gaussian Calibration

Decoupling Stability and Plasticity for Multi-Modal Test-Time Adaptation

On the Limits of Representation-Based Test-Time Adaptation for Multimodal LLMs