COMPASS: Fusion-Matched Supervision for Missing-Modality Human Sensing

Hao Wang; Yanyu Qian; Pengcheng Weng; Zixuan Xia; William Dan; Yangxin Xu; Fei Wang

cs.CVApr 2, 2026

COMPASS: Fusion-Matched Supervision for Missing-Modality Human Sensing

Authors: Hao Wang, Yanyu Qian, Pengcheng Weng, Zixuan Xia, William Dan, Yangxin Xu, Fei Wang

Abstract

Multimodal human activity recognition (HAR) and human pose estimation (HPE) must cope with modalities missing at inference. Completions generated for a fixed fusion model must preserve the fusion readout: what fusion uses from each modality. We propose COMPASS, a completion-and-fusion framework in which each modality occupies a fixed slot filled by an observed representation or a completion inferred from available inputs. Its core principle, fusion-matched supervision, supervises completions with same-sample real targets at readout granularity: token means for HAR, where fusion averages tokens, and per-joint representations for HPE, where fusion preserves joint structure. Trained with complete multimodal samples, COMPASS improves subset-averaged performance over the strongest baselines on XRF55 and MM-Fi for HAR and on MM-Fi for HPE. At fixed architecture, fusion-matched supervision outperforms full-token and centered-token matching in HAR, and joint-mean matching and no matching in HPE. The code is available at: https://github.com/haowangcoder/COMPASS.

Explore similar work

A Comparison of Fusion Techniques for Multi-Modal Human Activity Recognition on the HARMES Dataset
Jun 26, 2026Ahmed Mohamady, Robin Burchard, Kristof Van LaerhovenMultimodal Fusion Human Activity Recognition
Conformal Fusion Under Missing Modalities
Aug 7, 2026Alireza MoayedikiaMissing Modalities Multimodal Fusion
Triple Spectral Fusion for Sensor-based Human Activity Recognition
May 4, 2026Ye Zhang, Longguang Wang, Qing Gao +3Human Activity Recognition Multi-Sensor Fusion

Jun 26, 2026 · cs.LGJ/K move · Enter open · S save

A Comparison of Fusion Techniques for Multi-Modal Human Activity Recognition on the HARMES Dataset

Ahmed Mohamady, Robin Burchard, Kristof Van Laerhoven

University of Siegen, Germany

Recent advances in Human Activity Recognition (HAR) from wearable sensors have shown that multi-modal deep learning models consistently outperform their uni-modal counterparts. Modalities can include IMUs, RGB cameras, audio signals, and others. One important aspect of multi-modal deep learning is the sensor fusion approach we apply. Over recent years, multiple fusion paradigms have been proposed for multi-modal HAR. However, to the best of our knowledge, no head-to-head comparison of these paradigms exists on a common multi-modal HAR benchmark dataset. To address this research gap, we systematically compare seven state-of-the-art sensor fusion methods on the recently released HARMES dataset, which comprises 61 hours of fully labeled IMU, audio, and ambient humidity data. The chosen dataset focuses on 15 household and personal hygiene activities of daily living (ADLs). By applying the seven different fusion techniques to a state-of-the-art multi-modal model architecture, we show that Gated Multi-modal Fusion achieves the highest macro F1-score (0.82), surpassing the concatenation-based late fusion HARMES paper baseline of 0.76 by +6pp under leave-one-participant-out evaluation. All code used in our experiments is made publicly available on GitHub.

Multimodal Fusion Human Activity Recognition Multimodal Deep Learning Fusion Wearable Devices

Open paper Ask about it PDF

Abstract

Explore similar work

A Comparison of Fusion Techniques for Multi-Modal Human Activity Recognition on the HARMES Dataset

Conformal Fusion Under Missing Modalities

Triple Spectral Fusion for Sensor-based Human Activity Recognition