Purify-then-Align: Towards Robust Human Sensing under Modality Missing with Knowledge Distillation from Noisy Multimodal Teacher

Pengcheng Weng; Yanyu Qian; Yangxin Xu; Fei Wang

cs.CVApr 7, 2026

Purify-then-Align: Towards Robust Human Sensing under Modality Missing with Knowledge Distillation from Noisy Multimodal Teacher

Authors: Pengcheng Weng, Yanyu Qian, Yangxin Xu, Fei Wang

Organizations: School of Software Engineering, Xi’an Jiaotong University, China · Institute of Computer Science, Universität Bern, Switzerland · College of Computing and Data Science, Nanyang Technological University, Singapore

Abstract

Robust multimodal human sensing must overcome the critical challenge of missing modalities. Two principal barriers are the Representation Gap between heterogeneous data and the Contamination Effect from low-quality modalities. These barriers are causally linked, as the corruption introduced by contamination fundamentally impedes the reduction of representation disparities. In this paper, we propose PTA, a novel "Purify-then-Align" framework that solves this causal dependency through a synergistic integration of meta-learning and knowledge diffusion. To purify the knowledge source, PTA first employs a meta-learning-driven weighting mechanism that dynamically learns to down-weight the influence of noisy, low-contributing modalities. Subsequently, to align different modalities, PTA introduces a diffusion-based knowledge distillation paradigm in which an information-rich clean teacher, formed from this purified consensus, refines the features of each student modality. The ultimate payoff of this "Purify-then-Align" strategy is the creation of exceptionally powerful single-modality encoders imbued with cross-modal knowledge. Comprehensive experiments on the large-scale MM-Fi and XRF55 datasets, under pronounced Representation Gap and Contamination Effect, demonstrate that PTA achieves state-of-the-art performance and significantly improves the robustness of single-modality models in diverse missing-modality scenarios.

Figures & tables

Figure 1: The overall architecture of our proposed PTA framework, built on a Purify-then-Align paradigm. The model is trained in a nested loop: (a) The Purify Stage : The Purify Stage optimizes the meta-learning weights $\mathbf{w}$ to mitigate the Contamination Effect. (b) The Align Stage : The Align Stage uses these weights $\mathbf{w}$ to construct a “clean teacher” $f_{T}$ and then uses it in a diffusion-based process ( $\mathcal{L}_{\text{DiffKD}}$ ) to refine each “student” feature $f_{S}$ , thereby bridging the Representation Gap.

Modality	Base.1	Base.2	X-Fi	Ours	Improvement $\uparrow$	Base.1	Base.2	X-Fi	Ours	Improvement $\uparrow$
	MPJPE $\downarrow$					PA-MPJPE $\downarrow$
D	102.4	102.4	96.40	84.81	+12.0%	52.7	52.7	48.76	50.72	-4.0%
L	161.5	161.5	130.06	68.30	+47.5%	103.5	103.5	89.83	44.44	+50.5%
W	227.1	227.1	210.12	182.18	+13.3%	108.0	108.0	103.91	114.63	-10.3%
D+L	111.7	108.0	89.41	64.68	+27.7%	68.8	55.8	47.97	42.63	+11.1%
D+W	141.7	155.5	95.27	95.96	-0.7%	71.4	81.5	47.71	59.77	-25.3%

Table 1: MPJPE (mm) and PA-MPJPE (mm) on MM-Fi [ 46 ] dataset.

Modality	Base.	X-Fi	Ours	Improvement $\uparrow$
R	82.1	83.9	90.03	+6.13
W	77.8	55.7	82.34	+26.64
RF	42.2	42.5	55.04	+12.54
R+W	86.8	88.2	95.23	+7.03
R+RF	71.4	86.5	92.15	+5.65
W+RF	55.6	58.1	83.14	+25.04

Table 2: Accuracy (%) on the XRF55 dataset.

Table 4

Missing-Modality Learning

Missing-modality learning develops multimodal models that remain useful when input streams are absent, corrupted, or available only during training.

See all 170 papers

Jun 9, 2026

Cross-Modal Knowledge Distillation without Paired Data: Theoretical Foundation and Algorithm

Cross-modal knowledge distillation (CMKD) studies how a (large) teacher model trained on one type of data (e.g., images) can guide a (smaller) student model building on another type of data (e.g., text/audio). Existing CMKD methods often require paired multi-modal data with aligned semantics, but obtaining such paired data are often costly and impractical. To mitigate this limitation, we develop a new CMKD framework for the more challenging setting where paired data are unavailable. In particular, we establish a cross-modal distributional relationship between teacher and student models, which reveals two fundamental quantities governing effective distillation: feature alignment and label alignment. These quantities characterize semantic discrepancy between modalities at the levels of representation and prediction distributions, respectively. Motivated by this insight, we propose a principled framework, with theoretical guarantees, that enables effective cross-modal knowledge distillation by aligning distributions rather than individual samples. Extensive experiments across a wide range of multimodal benchmarks show that our framework is highly effective in both unpaired and paired data settings, improving significantly over prior work.

Trong Khiem Tran, Anh Duc Chu, Quang Hung Pham +2

School of Electrical Engineering and Computer Science, Washington State University, Pullman, US · School of Information and Communications Technology, Hanoi University of Science and Technology, Hanoi, Vietnam

Cross-Modal Knowledge Distillation Cross-Modal Alignment Distribution Matching

Ask PDF

Mar 10, 2026

Progressive Representation Learning for Multimodal Sentiment Analysis with Incomplete Modalities

Multimodal Sentiment Analysis (MSA) seeks to infer human emotions by integrating textual, acoustic, and visual cues. However, existing approaches often rely on all modalities are completeness, whereas real-world applications frequently encounter noise, hardware failures, or privacy restrictions that result in missing modalities. There exists a significant feature misalignment between incomplete and complete modalities, and directly fusing them may even distort the well-learned representations of the intact modalities. To this end, we propose PRLF, a Progressive Representation Learning Framework designed for MSA under uncertain missing-modality conditions. PRLF introduces an Adaptive Modality Reliability Estimator (AMRE), which dynamically quantifies the reliability of each modality using recognition confidence and Fisher information to determine the dominant modality. In addition, the Progressive Interaction (ProgInteract) module iteratively aligns the other modalities with the dominant one, thereby enhancing cross-modal consistency while suppressing noise. Extensive experiments on CMU-MOSI, CMU-MOSEI, and SIMS verify that PRLF outperforms state-of-the-art methods across both inter- and intra-modality missing scenarios, demonstrating its robustness and generalization capability.

Jindi Bao, Jianjun Qian, Mengkai Yan +1

Nanjing University of Science and Technology, Nanjing 210094, China · Hohai University, Nanjing 211100, China

Multimodal Sentiment Analysis Missing-Modality Learning Cross-Modal Representation Learning Uncertainty-Aware Multimodal Fusion

Ask PDF

Jul 23, 2026

UnDA: Unpaired Domain Alignment for Cross-Modal Knowledge Transfer in Medical Imaging

Multimodal based approaches often outperform single modality approaches in downstream tasks as the different modalities provide complementary information, yet acquiring paired clinical data remains a significant challenge in real world scenarios. While cross-modal knowledge distillation addresses this, existing methods often struggle with large modality gaps and the propagation of noise from uncertain source-domain predictions. To overcome these challenges, we propose UnDA, an anchor-guided framework for unpaired cross-modal distillation. Our approach introduces a backbone-agnostic Alignment Module that extracts semantically structured class tokens via an attention based pooling mechanism. To ensure robust knowledge transfer, we propose Uncertainty-Weighted Optimal Transport (UCT-OT), which dynamically weights feature-level alignment based on prediction confidence, effectively suppressing noisy supervision. Furthermore, a per-class ProtoNCE objective maintains stable prototype memories to enforce global discriminability across unpaired batches. Evaluations on representative segmentation tasks under strictly unpaired settings show consistent improvements in accuracy and boundary precision in the target modality, demonstrating that meaningful structural knowledge can be transferred across heterogeneous data sources without paired datasets.

Rafsan Jany, Shadab Tanjeed Ahmad, Ahsan Bulbul +3

Korea Institute of Oriental Medicine, Daejeon, South Korea · Islamic University of Technology, Gazipur, Bangladesh

Cross-Modal Knowledge Distillation Cross-Modality Medical Image Segmentation Cross-Modal Alignment Optimal Transport

Ask PDF

Abstract

Missing-Modality Learning

Explore similar work

Cross-Modal Knowledge Distillation without Paired Data: Theoretical Foundation and Algorithm

Progressive Representation Learning for Multimodal Sentiment Analysis with Incomplete Modalities

UnDA: Unpaired Domain Alignment for Cross-Modal Knowledge Transfer in Medical Imaging