Heterogeneous-Modal Unsupervised Domain Adaptation via Latent Space Bridging

Jiawen Yang; Shuhao Chen; Shengtao Zhang; Ke Tang; Yu Zhang

cs.CVJun 19, 2025

Heterogeneous-Modal Unsupervised Domain Adaptation via Latent Space Bridging

Authors: Jiawen Yang, Shuhao Chen, Shengtao Zhang, Ke Tang, Yu Zhang

Organizations: Southern University of Science and Technology · SZ DJI Technology Co., Ltd

Abstract

Unsupervised domain adaptation (UDA) effectively bridges the domain gap between a labeled source domain and an unlabeled target domain, but assumes that the two domains share the same modality. Heterogeneous domain adaptation (HDA) instead handles different feature spaces across domains, yet requires labeled target samples or paired data linking the source and target domains. Neither applies when a labeled source domain and a fully unlabeled target domain each hold an entirely distinct modality (e.g., 2D images and 3D point clouds). To address this limitation, we introduce a new setting termed Heterogeneous-Modal Unsupervised Domain Adaptation (HMUDA), which transfers knowledge across modalities via an unlabeled bridge domain containing paired observations from both modalities, whose distribution may deviate from those of the source and target domains. To learn under the HMUDA setting, we propose Latent Space Bridging (LSB), a dual-branch framework where a feature consistency loss on paired bridge samples closes the modality gap and a class-centroid alignment loss reduces the source-target discrepancy. Extensive experiments on eight benchmark settings covering both 2D-to-3D and 3D-to-2D transfer demonstrate that LSB achieves state-of-the-art performance.

Figures & tables

Fig. 1: An illustration of the HMUDA setting.

	Source		Target		Bridge	E-to-E
	$\mathcal{M}_{1}$	$\mathcal{M}_{2}$	$\mathcal{M}_{1}$	$\mathcal{M}_{2}$	Bridge	E-to-E
UDA	✓	✗	✓	✗	✗	✓
MM-UDA	✓	✓	✓	✓	✗	✓
HDA	✓	✗	✗	✓	✗	✗
HMUDA	✓	✗	✗	✓	✓	✓

TABLE I: Comparison between HMUDA and existing DA settings.

\mathcal{M}_{1}

and

\mathcal{M}_{2}

denote two modalities. ‘Bridge’ indicates whether a third unlabeled domain with paired modalities is available, and ‘E-to-E’ indicates whether the specific model is trained end-to-end from raw inputs.

Fig. 2: An illustration of the proposed LSB framework. Lines within different colors denote the data flow for computing different losses.

	$\mathcal{B}$		$\mathcal{S}$	$\mathcal{T}$
Scenario	Train		Train	Train	Val/Test
USA $\rightarrow$ Sing.	Sem.	18,029	15,695	9,665	2,770/2,929
Day $\rightarrow$ Night			24,745	2,779	602/602
Virt. $\rightarrow$ A2D2			2,126	24,461	808/2,426
USA $\rightarrow$ Sing.	A2D2	24,461	15,695	9,665	2,770/2,929
Day $\rightarrow$ Night			24,745	2,779	602/602

TABLE II: Statistics of the number of samples in each split of datasets for all eight settings. Each setting is evaluated in both the 2D-to-3D and 3D-to-2D directions, which share the same data splits and differ only in which modality serves as the source domain. Note that the training samples in the target and bridge domains are without labels.

		USA $\rightarrow$ Sing.	Day $\rightarrow$ Night	Virt. $\rightarrow$ A2D2	USA $\rightarrow$ Sing.	Day $\rightarrow$ Night	Virt. $\rightarrow$ Sem.	Sem. $\rightarrow$ A2D2	A2D2 $\rightarrow$ Sem.	Avg.
	Bridge Domain $\mathcal{B}$	Sem.			A2D2			Virt.		–
2D $\rightarrow$ 3D	Oracle	77.69	73.71	71.50	77.69	73.71	82.75	71.50	82.75	76.41
	xMUDA [ 65 ]	65.12	75.11	61.03	65.12	75.11	57.97	44.95	65.07	63.69
	Source-Only	51.04	57.32	19.00	49.21	55.96	36.44	43.75	43.28	44.50
	PL [ 5 ]	52.08	59.33	16.84	55.39	61.95	44.49	39.66	38.25	46.00
	CDSPP [ 32 ]	17.61	21.81	9.74	17.61	21.81	12.44	11.27	11.00	15.41

TABLE III: Testing results on HMUDA tasks in terms of mIoU. Each domain transfer is evaluated in both 2D-to-3D and 3D-to-2D modality directions. The last column reports the average over all eight settings for each transfer direction. The best performance is in bold .

$\mathcal{L}_{seg}^{s}$	$\mathcal{L}_{seg}^{b}$	$\mathcal{L}_{con}^{b}$	$\mathcal{L}_{ali}$	USA $\rightarrow$ Sing. ( Sem. )	Day $\rightarrow$ Night ( Sem. )	Virt. $\rightarrow$ A2D2 ( Sem. )	Sem. $\rightarrow$ A2D2 ( Virt. )
✓	✓	✗	✗	53.62	61.86	22.72	42.47
✓	✓	✓	✗	52.68	61.12	28.24	45.27
✓	✓	✗	✓	54.34	60.86	27.96	46.37
✓	✓	✓	✓	56.41	62.25	33.37	46.34
$\mathcal{L}_{seg}^{s}$	$\mathcal{L}_{seg}^{b}$	$\mathcal{L}_{con}^{b}$	$\mathcal{L}_{ali}$	USA $\rightarrow$ Sing. ( A2D2 )	Day $\rightarrow$ Night ( A2D2 )	Virt. $\rightarrow$ Sem. ( A2D2 )	A2D2 $\rightarrow$ Sem. ( Virt. )
✓	✓	✗	✗	49.35	60.64	35.74	42.29

TABLE IV: Effect of losses

\mathcal{L}_{seg}^{s}

\mathcal{L}_{seg}^{b}

\mathcal{L}_{con}^{b}

, and

\mathcal{L}_{ali}

in terms of mIoU for 2D-to-3D HMUDA tasks. The bridge domain

\mathcal{B}

for each task is given in parentheses. The best performance is in bold .

Panel of Figure 7 — Fig. 3: Sensitivity analysis of the LSB method with respect to hyperparameters and the number of samples in the bridge domain.

		USA	Day		A2D2
Method	$\mathcal{B}$	$\rightarrow$	$\rightarrow$	$\mathcal{B}$	$\rightarrow$
		Sing.	Night		Sem.
LSB (w/o $p_{\phi}$ )	A2D2	20.78	3.82	Virt.	15.00
LSB (w/o $p_{h}$ )		40.47	43.10		32.52
LSB		57.13	63.15		47.22

TABLE V: Effect of

p_{\phi}

and

p_{h}

on three 2D-to-3D HMUDA tasks.

		USA	Day		A2D2
Method	$\mathcal{B}$	$\rightarrow$	$\rightarrow$	$\mathcal{B}$	$\rightarrow$
		Sing.	Night		Sem.
LSB ( w. $\mathcal{L}_{ali}(\mathcal{B},\mathcal{T})$ )	A2D2	50.86	60.12	Virt.	44.44
LSB	A2D2	57.13	63.15	Virt.	47.22

TABLE VI: Effect of cross-modal alignment on three 2D-to-3D HMUDA tasks.

Fig. 4: Qualitative results on three 2D-to-3D HMUDA tasks: USA $\rightarrow$ Sing. , Day $\rightarrow$ Night , and A2D2 $\rightarrow$ Sem. . $(\cdot)$ in the vertical axis denotes the bridge domain $\mathcal{B}$ used in the HMUDA task. For example, USA $\rightarrow$ Sing.(Sem.) denotes the transfer from USA to Sing. via the bridge domain Sem. .

Panel of Figure 11 — Fig. 5: t-SNE visualization of target feature embeddings for the Source-Only and LSB methods.

Explore similar work

MUSDA: Multi-source Multi-modality Unsupervised Domain Adaptive 3D Object Detection for Autonomous Driving
May 11, 2026Xiaohu Lu, Hamed Khatounabadi, Hayder RadhaMulti-Source Domain Adaptation Unsupervised Domain Adaptation
Electrical and Computer Engineering Michigan State University East Lansing, USA
UnDA: Unpaired Domain Alignment for Cross-Modal Knowledge Transfer in Medical Imaging
Jul 23, 2026Rafsan Jany, Shadab Tanjeed Ahmad, Ahsan Bulbul +3Cross-Modal Knowledge Distillation Cross-Modality Medical Image Segmentation
Korea Institute of Oriental Medicine, Daejeon, South Korea · Islamic University of Technology, Gazipur, Bangladesh
PanDA: Unsupervised Domain Adaptation for Multimodal 3D Panoptic Segmentation in Autonomous Driving
Apr 21, 2026Yining Pan, Shijie Li, Yuchen Wu +2Unsupervised Domain Adaptation Multimodal Robustness
Singapore University of Technology and Design · Institute for Infocomm Research (I2R), A*STAR, Singapore

May 11, 2026, cs.CVJ/K move · Enter open · S save

MUSDA: Multi-source Multi-modality Unsupervised Domain Adaptive 3D Object Detection for Autonomous Driving

Xiaohu Lu, Hamed Khatounabadi, Hayder Radha

Electrical and Computer Engineering Michigan State University East Lansing, USA

Multi-Source Domain Adaptation Unsupervised Domain Adaptation Camera-LiDAR Fusion Autonomous Driving Perception 3D Object Detection

Open paper AskPDF arXiv

With the advancement of autonomous driving, numerous annotated multi-modality datasets have become available. This presents an opportunity to develop domain-adaptive 3D object detectors for new environments without relying on labor-intensive manual annotations. However, traditional domain adaptation methods typically focus on a single source domain or a single modality, limiting their effectiveness in multi-source, multi-modality scenarios. In this paper, we propose a novel framework for multi-source, multi-modality unsupervised domain adaptation in 3D object detection for autonomous driving. Given multiple labeled source domains and one unlabeled target domain, our framework first introduces hierarchical spatially-conditioned (HSC) domain classifiers, which jointly align features from both camera and LiDAR modalities at two distinct levels for each source-target domain pair. To effectively leverage information from multiple source domains, we construct a prototype graph between each pair of domains. Based on this, we develop a prototype graph weighted (PGW) multi-source fusion strategy to aggregate predictions from multiple source detection heads. Experimental results on three widely used 3D object detection datasets - Waymo, nuScenes, and Lyft - demonstrate that our proposed framework effectively integrates information across both modalities and source domains, consistently outperforming state-of-the-art methods.

Abstract

Explore similar work

MUSDA: Multi-source Multi-modality Unsupervised Domain Adaptive 3D Object Detection for Autonomous Driving

UnDA: Unpaired Domain Alignment for Cross-Modal Knowledge Transfer in Medical Imaging

PanDA: Unsupervised Domain Adaptation for Multimodal 3D Panoptic Segmentation in Autonomous Driving