Beyond Invariance: Factor-wise Selective Treatment of Shifts for Multimodal Learning

Guimin Hu; Qiwei Peng; Boyan Xu; Ruichu Cai

cs.CLOct 15, 2026

Beyond Invariance: Factor-wise Selective Treatment of Shifts for Multimodal Learning

Authors: Guimin Hu, Qiwei Peng, Boyan Xu, Ruichu Cai

Organizations: Guangdong University of Technology · University of Copenhagen

Abstract

Distribution shifts pose a fundamental challenge to robust multimodal learning. Existing methods typically promote invariance by suppressing environment-dependent variations, implicitly applying a uniform treatment to shifted information. However, shifted information can have different predictive consequences: some contributes little to prediction, some remains reliably informative, and some exhibits environment-dependent predictive effects. This raises a fundamental question: which shifted information should be suppressed, preserved, or attenuated? To address this question, we propose FaShift, a factor-wise shift framework that diagnoses latent representation factors according to their distributional changes, intervention effects, and cross-environment stability. By factorizing multimodal representations and evaluating these complementary properties, FaShift distinguishes non-predictive factors, stably predictive factors, and predictively useful but unstable factors. It then selectively suppresses, preserves, or attenuates their contributions, rather than enforcing uniform invariance. Experiments on MOSI, MOSEI, and CH-SIMS show that FaShift achieves the best ID accuracy and F1 across all three datasets. Compared with the strongest competitors reporting OOD results, it improves OOD accuracy/F1 by 1.5/3.6 points on MOSI and 1.7/2.7 points on MOSEI, while achieving the best reported OOD accuracy and F1 on CH-SIMS.

Figures & tables

Figure 1: Heterogeneous factor shifts motivate shift-aware reliance. Left: Factor-wise shift and intervention effects. Right: Shift-aware factor reliance.

Figure 2: Visualization of factor-wise shift diagnosis in the PCA space of the learned latent factors. FaShift identifies heterogeneous shift patterns across latent factors, distinguishing nuisance, unstable predictive, and stable predictive shifts.

Shift Type	Decision Criterion	Treatment
Nuisance Shift	$\Delta_{k}\leq\tau_{c}$	Suppress
Stable Predictive Shift	$\Delta_{k}>\tau_{c},\;V_{k}\leq\tau_{v}$	Preserve
Unstable Predictive Shift	$\Delta_{k}>\tau_{c},\;V_{k}>\tau_{v}$	Attenuate

Table 1: Shift categorization and corresponding selective treatment of representation factors.

\tau_{c}

and

\tau_{v}

are the thresholds for intervention effect and instability, respectively.

	MOSI				MOSEI				CH-SIMS
Method	ID		OOD		ID		OOD		ID		OOD
	Acc. $\uparrow$	F1 $\uparrow$	Acc. $\uparrow$	F1 $\uparrow$	Acc. $\uparrow$	F1 $\uparrow$	Acc. $\uparrow$	F1 $\uparrow$	Acc. $\uparrow$	F1 $\uparrow$	Acc. $\uparrow$	F1 $\uparrow$
Self-MM	84.9	84.8	76.7	76.7	85.2	85.2	74.7	74.3	80.7	80.8	70.1*	70.1*
C-MIB	87.8	87.8	77.9	77.8	86.9	86.8	75.1	75.3	-	-	-	-
DMD	84.3	84.3	77.5	77.4	84.6	84.6	75.2	75.7	-	-	-	-
DEVA	86.3	86.3	-	-	86.1	86.2	-	-	79.6	80.3	69.7*	69.8*

Table 2: Comparison of different methods under in-domain (ID) and out-of-distribution (OOD) settings on MOSI, MOSEI, and CH-SIMS. We report binary accuracy (Acc.) and F1 score. Results marked with an asterisk (*) are reproduced using the authors’ publicly available code, while “–” indicates that results and codes are unavailable. Best results are highlighted in bold.

Rep.	MOSEI		CH-SIMS
	Acc.	F1	Acc.	F1
Original $\mathbf{H}$	84.4	84.2	75.6	75.7
Factorized $\mathbf{Z}$	84.6	84.3	75.9	75.9
Gap	0.2	0.1	0.3	0.2

Table 3: Task information preservation in the factorized representation.

Figure 3: Visualization of the learned factors. Left: pairwise correlation matrix. Right: PCA projection of the learned factors.

Setting	Variant	Acc.	F1
ID: MOSEI	w/o Factorization	84.4	84.2
	w/o Shift Diagnosis	84.6	84.3
	Uniform Treatment	83.8	83.6
	FaShift	88.2	88.3
ID: CH-SIMS	w/o Factorization	75.6	75.7
	w/o Shift Diagnosis	75.9	75.9

Table 4: Ablation study of FaShift under in-domain (ID) and distribution-shifted (OOD) settings.

Figure 4: Factor-wise intervention effects and stability as a function of distribution shifts.

Figure 5: Pairwise joint intervention effects under distribution shifts. Left: Joint effects versus additive expectations. Right: Pairwise interaction effects, with positive and negative values indicating reinforcement and antagonism, respectively.

Figure 6: Cross-environment shift magnitude across the learned factors.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	K	M	$\tau_{s}$	$\tau_{c}$	$\tau_{v}$	$\lambda_{desc}$
MOSI	6	8	0.02	0.2	0.6	500
MOSEI	6	8	0.02	0.2	0.6	500
CH-SIMS	8	4	0.05	0.3	0.6	400
MUD3	8	6	0.05	0.3	0.6	200

Appendix

Table 5: Hyperparameter settings of FaShift across different multimodal datasets.

	ID		OOD
	Acc.	F1	Acc.	F1
Wang et al. (2025a)	78.3	77.3	70.9	70.7
FaShift(ours)	82.4	82.7	74.1	74.4

Appendix

Table 6: Results of MUD3 datasets.

Dataset	Environment	ID		OOD
		Acc-2 $\uparrow$	F1 $\uparrow$	Acc-2 $\uparrow$	F1 $\uparrow$
MOSEI	Observed	88.2	88.3	78.5	78.4
MOSEI	Pseudo	82.5	82.6	75.7	74.5
CH-SIMS	Observed	83.3	83.3	75.4	75.2
CH-SIMS	Pseudo	79.1	79.6	70.6	70.3

Appendix

Table 7: Comparison between observed and pseudo-environments for factor-wise shift diagnosis. For pseudo-environments, observed environmental attributes are excluded, and environments are constructed solely from Stage-I factor activation profiles.

Figure 7: Hyperparameter sensitivity of FaShift under ID and OOD settings. We vary the number of nearest neighbors

M

, intervention-effect threshold

\tau_{c}

, and stability threshold

\tau_{v}

, while keeping other hyperparameters fixed.

$K$	Avg. Factor Corr	OOD Acc-2 (%)
2	0.65	74.6
4	0.35	76.8
6	0.15	79.5
8	0.38	75.1

Appendix

Table 8: Sensitivity of FaShift to the number of latent factors

K

on MOSEI.

Distribution Shift Robustness

Distribution shift robustness studies how to keep machine-learning predictions reliable when deployment data differ from training data.

See all 280 papers

Learning Invariant Modality Representation for Robust Multimodal Learning from a Causal Inference Perspective
Apr 20, 2026Sijie Mai, Shiqin HanDisentangled Representation Learning Causal Representation Learning
School of Computer Science, South China Normal University
ShapKO: Shapley-Adaptive Modality Knockout for Robust Multimodal Learning
Jul 10, 2026Nusrat Binta Nizam, Fengbei Liu, Sunwoo Kwak +3Missing-Modality Learning Multimodal Robustness
Cornell University, New York, NY, USA · Cornell Tech, New York, NY, USA · Weill Cornell Medicine, New York, NY, USA
Learning Clinical Representations Under Systematic Distribution Shift
Mar 7, 2026Yuanyun Zhang, Shi LiOOD Generalization Invariant Representation Learning
Columbia University

Apr 20, 2026J/K move · Enter open · S save

Learning Invariant Modality Representation for Robust Multimodal Learning from a Causal Inference Perspective

Sijie Mai, Shiqin Han

School of Computer Science, South China Normal University

Disentangled Representation Learning Causal Representation Learning Multimodal Robustness Multimodal Disentangled Representation Learning Sentiment Analysis Multimodal Emotion Recognition+2

Open paper AskPDF arXiv

Multimodal affective computing aims to predict humans' sentiment, emotion, intention, and opinion using language, acoustic, and visual modalities. However, current models often learn spurious correlations that harm generalization under distribution shifts or noisy modalities. To address this, we propose a causal modality-invariant representation (CmIR) learning framework for robust multimodal learning. At its core, we introduce a theoretically grounded disentanglement method that separates each modality into causal invariant representation' and environment-specific spurious representation' from a causal inference perspective. CmIR ensures that the learned invariant representations retain stable predictive relationships with labels across different environments while preserving sufficient information from the raw inputs via invariance constraint, mutual information constraint, and reconstruction constraint. Experiments across multiple multimodal benchmarks demonstrate that CmIR achieves state-of-the-art performance. CmIR particularly excels on out-of-distribution data and noisy data, confirming its robustness and generalizability.

Abstract

Distribution Shift Robustness

Explore similar work

Learning Invariant Modality Representation for Robust Multimodal Learning from a Causal Inference Perspective

ShapKO: Shapley-Adaptive Modality Knockout for Robust Multimodal Learning

Learning Clinical Representations Under Systematic Distribution Shift