Towards Multimodal Domain Generalization with Few Labels

Hongzhao Li; Hao Dong; Hualei Wan; Shupan Li; Mingliang Xu; Muhammad Haris Khan

cs.CVFeb 26, 2026

Towards Multimodal Domain Generalization with Few Labels

Authors: Hongzhao Li, Hao Dong, Hualei Wan, Shupan Li, Mingliang Xu, Muhammad Haris Khan

Organizations: Zhengzhou University · ETH Zürich · MBZUAI

Abstract

Multimodal models ideally should generalize to unseen domains while remaining data-efficient to reduce annotation costs. To this end, we introduce and study a new problem, Semi-Supervised Multimodal Domain Generalization (SSMDG), which aims to learn robust multimodal models from multi-source data with few labeled samples. We observe that existing approaches fail to address this setting effectively: multimodal domain generalization methods cannot exploit unlabeled data, semi-supervised multimodal learning methods ignore domain shifts, and semi-supervised domain generalization methods are confined to single-modality inputs. To overcome these limitations, we propose a unified framework featuring three key components: Consensus-Driven Consistency Regularization, which obtains reliable pseudo-labels through confident fused-unimodal consensus; Disagreement-Aware Regularization, which effectively utilizes ambiguous non-consensus samples; and Cross-Modal Prototype Alignment, which enforces domain- and modality-invariant representations while promoting robustness under missing modalities via cross-modal translation. We further establish the first SSMDG benchmarks, on which our method consistently outperforms strong baselines in both standard and missing-modality scenarios. Our benchmarks and code are available at https://github.com/lihongzhao99/SSMDG.

Figures & tables

Figure 1: (a) Illustration of four related learning settings: Semi-Supervised Multimodal Learning (SSML), Multimodal Domain Generalization (MMDG), Semi-Supervised Domain Generalization (SSDG), and our proposed Semi-Supervised Multimodal Domain Generalization (SSMDG). SSML overlooks domain shifts, MMDG cannot leverage unlabeled data, and SSDG is restricted to single-modality inputs. (b) Performance comparison in the SSMDG setting, highlighting the limitations of existing paradigms and the effectiveness of our proposed method.

Figure 2: Overview of the proposed framework for SSMDG. The framework jointly leverages labeled and unlabeled multimodal data from multiple source domains through three key components: Consensus-Driven Consistency Regularization (CDCR, § 4.1 ), Disagreement-Aware Regularization (DAR, § 4.2 ), and Cross Modal Prototype Alignment (CMPA, § 4.3 ), enabling robust generalization to unseen target domains.

Method	Modality		HAC dataset				EPIC-Kitchens dataset
Method	Video	Audio	A, C $\rightarrow$ H	H, C $\rightarrow$ A	H, A $\rightarrow$ C	Mean	D2, D3 $\rightarrow$ D1	D1, D3 $\rightarrow$ D2	D1, D2 $\rightarrow$ D3	Mean
Baseline method
Source-only	$\checkmark$	$\checkmark$	42.15	47.20	37.82	42.39	25.31	32.12	30.95	29.46
Multimodal domain generalization methods (MMDG)
RNA-Net	$\checkmark$	$\checkmark$	43.82	45.63	35.44	41.63	24.14	31.87	29.32	28.44
SimMMDG	$\checkmark$	$\checkmark$	46.33	48.32	38.53	44.39	25.62	32.66	33.26	31.11

Table 1: Semi-supervised multimodal domain generalization results with 5 labels per class.

Method	Modality			10	5%	10%
Method	Video	Audio	Flow	10	5%	10%
CGMatch M [ 5 ]	$\checkmark$	$\checkmark$		59.33	58.74	59.32
STiL [ 14 ]	$\checkmark$	$\checkmark$		60.66	58.96	60.21
NIED-LR M [ 20 ]	$\checkmark$	$\checkmark$		62.92	61.08	62.86
Ours	$\checkmark$	$\checkmark$		65.00	65.67	65.90
CGMatch M [ 5 ]	$\checkmark$		$\checkmark$	58.05	59.62	60.33

Table 2: SSMDG results on the HAC dataset with 10, 5%, and 10% labels per class. We report the mean accuracy across all domains.

Figure 3: Pseudo-label accuracy and unlabeled data utilization on the HAC benchmark. Our method consistently achieves superior accuracy while maintaining a higher utilization rate compared to competitive baselines.

Method	Modality		HAC dataset				EPIC-Kitchens dataset
Method	Video	Audio	A, C $\rightarrow$ H	H, C $\rightarrow$ A	H, A $\rightarrow$ C	Mean	D2, D3 $\rightarrow$ D1	D1, D3 $\rightarrow$ D2	D1, D2 $\rightarrow$ D3	Mean
5% labels per class
Audio-only		$\checkmark$	32.82	31.06	24.36	29.41	25.62	25.01	26.62	25.75
Zero-filling	$\times$	$\checkmark$	30.86	30.29	23.66	28.27	24.91	25.04	26.31	25.42
Translation	$\times$	$\checkmark$	37.88	38.09	31.62	35.86	31.06	30.62	31.58	31.09
Video-only	$\checkmark$		51.62	52.98	48.62	51.07	35.92	35.60	34.69	35.40

Table 3: SSMDG with missing modalities.

\times

means the modality is available during training, but is missing in test time.

CDCR	DAR	CMPA	HAC dataset				EPIC-Kitchens dataset
CDCR	DAR	CMPA	A, C $\rightarrow$ H	H, C $\rightarrow$ A	H, A $\rightarrow$ C	Mean	D2, D3 $\rightarrow$ D1	D1, D3 $\rightarrow$ D2	D1, D2 $\rightarrow$ D3	Mean
			48.62	50.28	37.26	45.39	29.06	32.74	33.92	31.91
$\checkmark$			59.92	58.67	45.69	54.76	34.72	37.69	37.02	36.48
	$\checkmark$		60.69	57.95	46.03	54.89	34.63	37.62	36.97	36.41
$\checkmark$	$\checkmark$		63.37	64.26	48.28	58.64	36.02	38.63	38.91	37.85
$\checkmark$		$\checkmark$	64.37	63.69	47.58	58.55	36.72	37.16	37.09	36.99

Table 4: Ablations of each proposed module under 5 labels per class.

Method	HAC	EPIC-Kitchens
Mean-CDCR	58.26	37.61
Any2-CDCR	59.34	38.25
Strict-CDCR	59.16	37.32
Full-CDCR	60.77	39.94
CE-DAR	58.32	37.92
Weak-only DAR	60.42	38.97

Table 5: Ablation study results demonstrating the effectiveness of individual components under 5 labels per class.

Figure 4: Ablation analysis of different modules on pseudo-label accuracy and unlabeled data utilization.

Figure 5: Visualization of domain shifts in the experimental benchmarks. (a) Example frames from the EPIC-Kitchens dataset across three environments (D1, D2, D3), highlighting variations in viewpoint and illumination. (b) Samples from the HAC dataset (Human, Animal, Cartoon), illustrating stylistic differences across domains.

Explore similar work

MMDG-Bench: A Benchmark for Multimodal Domain Generalization
May 30, 2026Qianshan Zhan, Qian Wang, Da Li +2Multimodal Robustness Domain Generalization
University of Manchester · Jiyue AI · Samsung AI Centre Cambridge +1
Are We Making Progress in Multimodal Domain Generalization? A Comprehensive Benchmark Study
May 7, 2026Hao Dong, Hongzhao Li, Shupan Li +3Multimodal Robustness Domain Generalization
ETH Zürich · Zhengzhou University · MBZUAI +1
MER-DG: Modality-Entropy Regularization for Multimodal Domain Generalization
May 3, 2026Yavuz Yarici, Ghassan AlRegibDomain Generalization Cross-Domain Generalization
1OLIVES at the Center for Signal and Information Processing CSIP, School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA, USA.

May 30, 2026J/K move · Enter open · S save

MMDG-Bench: A Benchmark for Multimodal Domain Generalization

Qianshan Zhan, Qian Wang, Da Li, Xiao-Jun Zeng+1

University of Manchester · Jiyue AI · Samsung AI Centre Cambridge+1

Multimodal Robustness Domain Generalization

Open paper AskPDF arXiv

Multi-modal Domain Generalization (MMDG) seeks to leverage complementary modalities to enhance model robustness on unseen domains. Despite extensive progress in Multi-modal Learning (MML) and Domain Generalization (DG) as individual fields, their systematic integration remains under-explored. Current MMDG research is largely confined to action recognition and lacks standardized evaluation protocols. To address this, we introduce MMDG-Bench, a comprehensive benchmark featuring two foundational frameworks: DG then MML (D2M) and MML then DG (M2D). We provide unified experimental protocols across diverse tasks, including video-audio-flow action recognition and RGB-Depth-IR face anti-spoofing. By instantiating ten MMDG baselines through pairing a unified MML configuration with five DG techniques under both D2M and M2D orderings, we demonstrate that these structured combinations frequently outperform existing state-of-the-art methods, underscoring the necessity of a unified benchmarking effort. Our analysis yields three key insights: (1) Integrating DG techniques provides consistent generalization gains across various backbones, whereas non-DG methods are highly sensitive to backbone shifts; (2) The optimal framework choice depends on inter-modal stability: D2M excels when modal relations are stable across domains, while M2D is more robust to cross-domain relational variance; (3) Stronger backbones yield amplified performance dividends when integrated into our structured frameworks. MMDG-Bench provides a principled foundation and actionable design guidelines for future research in multi-modal robustness. Code is released at https://github.com/qszhan/MMDG-Bench.

Abstract

Explore similar work

MMDG-Bench: A Benchmark for Multimodal Domain Generalization

Are We Making Progress in Multimodal Domain Generalization? A Comprehensive Benchmark Study

MER-DG: Modality-Entropy Regularization for Multimodal Domain Generalization