HeteroWear: Sensor-Aware Self-Supervised Learning for Heterogeneous Wearable Activity Recognition
Organizations: School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA
Abstract
Self-supervised learning for wearable sensing is usually confined to a single dataset or device family, leaving open whether representations transfer across heterogeneous sensor configurations and whether gains come from pretraining rather than architecture or richer sensor input. We present HeteroWear, a sensor-aware self-supervised framework pretrained on an eight-dataset wearable corpus (173 subjects, 358,261 windows) spanning inertial and physiological signals across body locations, devices, and sampling rates. HeteroWear separates within-sensor temporal modeling from cross-sensor relational modeling using sensor-context embeddings, temporal and cross-sensor Transformers, and latent cross-attention fusion. Training combines structured sensor dropout, cross-configuration consistency, and per-axis masked sensor modeling; missing sensors are omitted rather than reconstructed. Under leave-one-dataset-out pretraining and subject-disjoint evaluation on PAMAP2, WISDM, HHAR, HARTH, and UCI-HAR, HeteroWear improves mean macro-F1 from 0.688 to 0.724 over the identical architecture trained from scratch, with the largest gain on HHAR (0.823 versus 0.741), and achieves the highest macro-F1 on four of five benchmarks relative to single-accelerometer supervised baselines. Held-out-device evaluation yields a mean +0.066 macro-F1 improvement. Under 50% test-time channel removal, HeteroWear retains 73.8% of full-channel performance versus 64.1% for the same architecture trained from scratch without sensor dropout. Ablations show that structured sensor dropout primarily drives robustness, while cross-configuration consistency and sensor-context encoding contribute to transfer.
Figures & tables
| Method | Missing-modality handling | Cross-device | # datasets |
|---|---|---|---|
| CroSSL [ 23 ] | latent masking | partial | 1–2 |
| LIMU-BERT [ 21 ] | N/A (single modality) | small | 1 |
| Yuan et al. [ 24 ] | N/A (single modality) | no | 1 (very large) |
| MultiMAE [ 12 ] | masking + reconstruction | no | 1 (vision) |
| PhysioME [ 8 ] | restoration decoder | no | 2 |
| PhysioOmni [ 9 ] | decoupled codebooks | no | 6 |
| Dataset | Subj. | Windows | Modalities |
|---|---|---|---|
| PAMAP2 ∗ | 9 | 14,322 | ACC, GYRO, MAG, HR |
| WESAD | 15 | 43,363 | ACC, ECG, EDA, RESP, TEMP |
| PPG-DaLiA | 15 | 64,672 | PPG, ACC, ECG, RESP, TEMP |
| MMASH | 22 | 58,027 | ACC, HR |
| WISDM ∗ | 51 | 72,928 | ACC, GYRO |
| HHAR ∗ | 9 | 44,212 | ACC, GYRO |
| Dataset | RF | CNN | DCLSTM | Transformer | HW-Scratch | HW-LODO | |
|---|---|---|---|---|---|---|---|
| PAMAP2 | .609 | .733 | .475 | .748 | .731 | .756 | +.025 |
| WISDM | .627 | .716 | .686 | .718 | .676 | .704 | +.028 |
| HHAR | .532 | .513 | .428 | .563 | .741 | .823 | +.082 |
| HARTH | .431 | .498 | .470 | .547 | .526 | .552 | +.026 |
| UCI-HAR | .660 | .775 | .769 | .779 | .764 | .786 | +.022 |
| Mean | .572 | .647 | .566 | .671 | .688 | .724 | +.037 |
| Dataset | Transfer | Scratch | HeteroWear | |
|---|---|---|---|---|
| WISDM | phone watch | 0.491 | 0.556 | +0.065 |
| WISDM | watch phone | 0.538 | 0.591 | +0.053 |
| HHAR | held-out device A | 0.676 | 0.755 | +0.079 |
| HHAR | held-out device B | 0.703 | 0.771 | +0.068 |
| Mean | — | 0.602 | 0.668 | +0.066 |
| Dataset | S-Full | S-50 | S-Ret. | L-Full | L-50 | L-Ret. |
|---|---|---|---|---|---|---|
| PAMAP2 | .731 | .402 | 55.0% | .756 | .498 | 65.9% |
| WISDM | .676 | .447 | 66.1% | .704 | .526 | 74.7% |
| HHAR | .741 | .563 | 76.0% | .823 | .691 | 84.0% |
| HARTH | .526 | .336 | 63.8% | .552 | .409 | 74.1% |
| UCI-HAR | .764 | .456 | 59.7% | .786 | .553 | 70.4% |
| Mean | – | – | 64.1% | – | – | 73.8% |
| Configuration | F1 | F1@50% | Ret. |
|---|---|---|---|
| Scratch (no pretraining) | 0.741 | 0.563 | 76.0% |
| MSM only | 0.758 | 0.560 | 73.9% |
| MSM + consistency | 0.787 | 0.606 | 77.0% |
| MSM + struct. dropout | 0.801 | 0.650 | 81.1% |
| Full objective | 0.823 | 0.691 | 84.0% |
| Full, no context emb. | 0.781 | 0.639 | 81.8% |
| Dataset | Transf. | Transf. SSL | |
|---|---|---|---|
| PAMAP2 | 0.748 | 0.671 | |
| WISDM | 0.718 | 0.512 | |
| HHAR | 0.563 | 0.634 | |
| HARTH | 0.547 | 0.253 | |
| UCI-HAR | 0.779 | 0.706 |
Wearable Sensing
Wearable sensing uses body-worn devices to measure motion, physiology, and interaction for health, activity, and assistive applications.
See all 143 papers