QA-CMIF:Accurate and Lightweight Cross-Modal Latent Imputation and Fusion for Fault-Tolerant sEMG-Inertial Hand Gesture Recognition
Abstract
Fusing surface electromyography (sEMG) with inertial measurements is widely used for myoelectric prosthesis control and wearable human-machine interfaces. In daily wear, however, socket micro-displacement, electrode lift-off, and wire failures often corrupt an unknown subset of sEMG channels through burst artifacts, lift-off shocks, or open-circuit dropouts. Existing fusion networks mix all channels in the first convolutional layer, so a few corrupted electrodes contaminate the entire latent representation, whereas gate-only attenuation of corrupted channels discards the co-activation cues needed to separate fine-grained gestures. We propose QA-CMIF, a quality-aware cross-modal latent imputation and fusion framework that encodes each sEMG channel independently, scores its reliability with a soft gate, and protects intact channels via a confidence-weighted residual bypass, while a cross-attention imputer repairs degraded channels in latent space using the inertial stream and remaining reliable sEMG channels under a bad-channel alignment loss. Across within-subject evaluations on 29 participants from Ninapro DB3, DB2, and DB5 spanning 17-52 gesture classes, QA-CMIF outperformed the gate-only baseline under 50% channel corruption by 1.29-2.65 percentage points, denoted pp, across all three 17-class cohorts with paired-t Holm-adjusted p<0.002, widened this margin to 2.59-4.15 pp on the 40- and 52-class tasks, and surpassed corruption-augmented conventional fusion by 4.61-19.85 pp. Requiring only 0.175-0.185M parameters and 2.45-2.63M multiply-accumulate operations across the 2000 Hz and 200 Hz setups, QA-CMIF saves 46%-55% in parameters and 72%-81% in compute over recent baselines with a 1.44 ms 99th-percentile single-thread CPU latency.
Figures & tables
| Method / Paradigm | Input Modalities | Target Degradation Granularity | Processing & Repair Domain | Explicit Bad-Channel Latent Supervision | Transradial Amputee Validation | Matched Params / MACs |
|---|---|---|---|---|---|---|
| HyFusion [ 20 ] ( IEEE Sensors J. 2023) | sEMG + ACC | Nominal clean recordings | Multi-branch features with dense first-layer cross-channel mixing | No | Yes (DB3, DB7; clean only) | (HyFusion-Lite) |
| MFF-Net [ 18 ] ( IEEE JBHI 2025) | sEMG only | Sparse-channel clean / transfer | Time–frequency–spatial mixed feature fusion | No | Yes (DB3 subset) | —(Unimodal sEMG) |
| HEFF [ 8 ] ( IEEE JBHI 2026) | sEMG + FMG | Cross-day & limb-position drift | Data/feature/decision pyramid fusion | No | No (15 intact subjects) | —(Custom 8-ch bracelet) |
| TrustEMG-Net [ 11 ] ( IEEE JBHI 2025) | Single-ch sEMG | Waveform noise (PLI, BW, ECG, MOA) | Time-domain waveform reconstruction (U-Net + Transformer) | No (Waveform loss) | No (Intact DB2) | —(Front-end denoiser) |
| Prism-SQA [ 12 ] ( IEEE JBHI 2026) | Single-ch sEMG | Multi-contaminant waveform distortion | Waveform source separation + fingerprint verification | No (Quality score target) | Yes (DB3 SQA test) | —(Standalone SQA network) |
| AiFusion [ 13 ] ( IEEE TNSRE 2023) | sEMG + ACC/IMU | Complete modality absence (EMG-loss / ACC-loss) | Dense first-layer cross-channel mixing + interactive alignment | No | Yes (DB3 5-subject subset) | (AiFusion-Lite) |
| Benchmark & Population | Sensor Hardware & Sampling Rate | Input Channels & Window ( ) | Cohort / Subset Name ( , Subject IDs) | Evaluated Tasks (Classes) | QA-CMIF Parameterization |
|---|---|---|---|---|---|
| Ninapro DB3 [ 6 ] Clinical Transradial Amputees (11 enrolled; s6, s7 excluded due to 10 effective EMG channels) | Delsys Trigno Wireless Array (Raw surface EMG) | , stride Train: rep Test: rep | DB3-All9 ( : s1–s5, s8–s11) DB3-Core6 ( : s2, s4, s5, s8, s9, s11) DB3-Holdout3 ( : s1, s3, s10) | B17 (17 classes) B17 (17) & BC40 (40) B17 (17 classes) | Front end for ( , , max-pool 2) Identity embeddings on keys ( ) |
| Ninapro DB2 [ 6 ] Intact-Limb Medical Array | Delsys Trigno Wireless Array (Raw surface EMG) | , stride Train: rep Test: rep | DB2-All10 ( : s1–s10) DB2-Holdout5 ( : s6–s10) DB2-Dev5 ( : s1–s5) | B17 (17 classes) B17 (17 classes) B17 (17 classes) | Front end for ( , , max-pool 2) Identity embeddings on keys ( ) |
| Ninapro DB5 [ 3 ] Intact-Limb Wearable Armband | Dual Thalmic Myo Armbands (Pre-conditioned sEMG) | , stride Train: rep Test: rep | DB5-All10 ( : s1–s10) DB5-Holdout5 ( : s6–s10) DB5-Dev5 ( : s1–s5) | B17 (17) & ABC52 (52) B17 (17) & ABC52 (52) B17 (17) & ABC52 (52) | Front end for (strides ; no down-sampling) Identity embeddings on queries only ( ) |
| Model | clean Acc (F1) (%) | Burst-50% (%) | Dropout-50% (%) | Liftoff-50% (%) | EMG-loss (%) | ACC-loss (%) | Liftoff gain (pp) / paired a |
|---|---|---|---|---|---|---|---|
| DB3-All9 ( amputees, B17 17 classes) | |||||||
| ACC-Only | 58.67 15.01 (56.20) | 58.67 15.01 | 58.67 15.01 | 58.67 15.01 | 58.67 15.01 | 5.38 2.03 | +11.38 ( ) |
| Vanilla-Clean | 65.58 12.31 (64.44) | 37.72 11.73 | 50.27 12.77 | 32.01 10.88 | 31.01 11.13 | 20.24 3.62 | +38.03 ( ) |
| Vanilla-Aug | 68.13 12.81 (67.12) | 63.70 14.53 | 60.49 13.99 | 59.33 15.16 | 58.53 14.33 | 40.63 10.26 | +10.71 ( ) |
| HyFusion-Lite [ 20 ] | 72.89 13.04 (71.97) | 69.19 14.57 | 64.92 14.65 | 61.04 16.46 | 58.10 15.21 | 47.00 9.93 | +9.00 ( ) |
| AiFusion-Lite [ 13 ] | 68.72 14.39 (67.70) | 65.06 15.59 | 61.24 15.33 | 58.99 16.65 | 58.50 16.17 | 44.69 10.86 | +11.06 ( ) |
| Model | clean Acc (F1) (%) | Burst-50% (%) | Dropout-50% (%) | Liftoff-50% (%) | EMG-loss (%) | ACC-loss (%) | Liftoff gain (pp) / paired a |
|---|---|---|---|---|---|---|---|
| DB2-All10 ( , 12-EMG, , B17 17 cl.) | |||||||
| ACC-Only | 65.19 6.51 (63.81) | 65.19 6.51 | 65.19 6.51 | 65.19 6.51 | 65.19 6.51 | 6.80 1.76 | +16.08 ( ) |
| Vanilla-Clean | 80.92 5.83 (80.72) | 43.91 6.30 | 58.41 7.33 | 29.93 4.54 | 28.76 10.31 | 30.74 10.69 | +51.34 ( ) |
| Vanilla-Aug | 82.63 6.55 (82.64) | 76.67 8.11 | 71.03 7.05 | 66.15 6.87 | 64.56 6.94 | 59.83 9.45 | +15.12 ( ) |
| HyFusion-Lite [ 20 ] | 86.47 5.44 (86.39) | 82.10 7.05 | 75.18 6.76 | 68.71 7.61 | 63.56 7.75 | 68.84 9.14 | +12.56 ( ) |
| AiFusion-Lite [ 13 ] | 83.03 6.51 (83.13) | 77.71 7.51 | 71.16 6.61 | 66.53 6.65 | 65.32 6.53 | 66.42 9.48 | +14.73 ( ) |
| Model | clean Acc (%) | Burst- (%) | Dropout- (%) | Liftoff- (%) | Gate AUROC/F1 (Burst- ) | Clean FAR (%) | Clean disp. | MSE red. Liftoff- (%) | Learned |
| DB3-All9 ( amputees, ) | |||||||||
| Vanilla-Aug | 68.13 12.81 | 63.70 14.53 | 60.49 13.99 | 59.33 15.16 | — | — | — | — | — |
| Gate-Only | 74.93 12.39 | 68.67 13.81 | 68.65 13.77 | 68.48 13.82 | 0.9991 / 0.9904 | 0.41 | 1.405 0.194 | 0.00 0.00 | — |
| w/o Bypass | 75.23 11.51 | 69.03 13.34 | 68.98 13.55 | 69.00 13.44 | 0.9989 / 0.9863 | 0.35 | 0.384 0.038 | +37.09 11.28 | — |
| w/o Align Loss | 74.93 12.31 | 69.46 13.41 | 69.50 13.46 | 69.39 13.48 | 0.9992 / 0.9908 | 0.41 | 1.364 0.185 | 9.79 6.70 | 0.655 0.050 |
| QA-CMIF | 74.85 12.63 | 69.97 13.48 | 69.97 13.48 | 70.04 13.49 | 0.9992 / 0.9898 | 0.43 | 1.286 0.189 | +42.43 6.90 | 0.625 0.037 |
| Model | Params | Rel. params | MACs | FLOPs | Rel. MACs | CPU mean SD (P99, ms) | GPU mean SD (P99, ms) |
| Array (DB3 / DB2, , ) | |||||||
| ACC-Only | 81,233 | 1.46M | 2.92M | 0.260 0.010 (0.299) | 0.271 0.061 (0.342) | ||
| Vanilla-Clean | 174,545 | 4.07M | 8.14M | 0.553 0.012 (0.602) | 0.410 0.004 (0.422) | ||
| Vanilla-Aug | 174,545 | 4.07M | 8.14M | 0.545 0.012 (0.595) | 0.410 0.003 (0.422) | ||
| Gate-Only (Control) | 139,794 | 2.21M | 4.42M | 0.987 0.047 (1.056) | 0.800 0.022 (0.826) | ||
| QA-CMIF | 174,867 | 2.63M | 5.26M | 1.358 0.111 (1.443) | 1.446 0.386 (1.615) | ||
| Subject | Forearm (%) | PLS | Cosmetic (y) | Kinematic (y) | Myoelectric (y) | EMG | B17 | BC40 | Formal subset |
|---|---|---|---|---|---|---|---|---|---|
| s1 | 50 | 2 | 0 | 0 | 13 | 12 | Yes | No | Holdout3 |
| s2 | 70 | 5 | 6 | 0 | 0 | 12 | Yes | Yes | Core6 |
| s3 | 30 | 2 | 0 | 8 | 8 | 12 | Yes | No | Holdout3 |
| s4 | 40 | 1 | 0 | 0 | 0 | 12 | Yes | Yes | Core6 |
| s5 | 90 | 2 | 0 | 0.4 | 0 | 12 | Yes | Yes | Core6 |
| s6 | 40 | 4 | 0 | 12 | 0 | 10 | No | No | Excluded: 10 effective EMG channels |
| Model | Clean | Burst-25 | Dropout-25 | Liftoff-25 | Burst-50 | Dropout-50 | Liftoff-50 | EMG-loss | ACC-loss |
|---|---|---|---|---|---|---|---|---|---|
| DB2-All10, B17, n = 10 | |||||||||
| ACC-Only | 65.19 6.51 63.81 6.71 | 65.19 6.51 63.81 6.71 | 65.19 6.51 63.81 6.71 | 65.19 6.51 63.81 6.71 | 65.19 6.51 63.81 6.71 | 65.19 6.51 63.81 6.71 | 65.19 6.51 63.81 6.71 | 65.19 6.51 63.81 6.71 | 6.80 1.76 0.75 0.18 |
| Vanilla-Clean | 80.92 5.83 80.72 5.79 | 50.57 7.62 49.95 7.28 | 70.38 6.70 69.72 6.28 | 30.91 4.57 29.99 4.48 | 43.91 6.30 42.95 6.13 | 58.41 7.33 57.11 6.13 | 29.93 4.54 28.69 4.42 | 28.76 10.31 15.08 6.05 | 30.74 10.69 23.64 11.33 |
| Vanilla-Aug | 82.63 6.55 82.64 6.41 | 78.49 7.71 78.48 7.57 | 77.93 6.74 77.70 6.51 | 67.58 7.18 66.72 7.00 | 76.67 8.11 76.53 8.02 | 71.03 7.05 70.26 6.73 | 66.15 6.87 65.01 6.77 | 64.56 6.94 62.81 7.35 | 59.83 9.45 59.93 9.16 |
| Gate-Only | 87.58 3.75 87.43 3.69 | 84.26 4.57 84.04 4.50 | 84.05 4.54 83.78 4.48 | 84.25 4.56 84.01 4.44 | 79.61 5.20 79.20 5.06 | 79.28 5.31 78.85 5.19 | 79.57 5.33 79.15 5.14 | 63.62 6.94 61.61 7.41 | 66.68 6.59 66.40 6.49 |
| QA-CMIF | 87.72 3.66 87.56 3.59 | 85.10 4.13 84.89 4.06 | 84.97 4.14 84.71 4.07 | 85.05 4.18 84.84 4.07 | 81.28 4.72 80.93 4.59 | 81.03 4.78 80.63 4.62 | 81.26 4.86 80.88 4.66 | 64.08 7.39 62.71 7.59 | 66.06 6.09 65.84 5.93 |
| Condition | Baseline | SD | 95% CI | W/L/T | raw | raw | primary | primary | full | ||
|---|---|---|---|---|---|---|---|---|---|---|---|
| DB2-All10, B17, Accuracy | |||||||||||
| Clean | ACC-Only | 22.53 | 5.31 | [18.73, 26.33] | 4.24 | 10/0/0 | 0.0020 | – | – | – | |
| Clean | Vanilla-Clean | 6.80 | 3.30 | [4.44, 9.16] | 2.06 | 10/0/0 | 0.0020 | – | – | 0.0054 | |
| Clean | Vanilla-Aug | 5.09 | 3.58 | [2.53, 7.65] | 1.42 | 10/0/0 | 0.0015 | 0.0020 | – | – | 0.0624 |
| Clean | Gate-Only | 0.13 | 0.48 | [-0.21, 0.47] | 0.28 | 6/4/0 | 0.3987 | 0.4316 | – | – | 1.0000 |
| Clean | w/o Bypass | -1.13 | 1.39 | [-2.12, -0.13] | -0.81 | 3/7/0 | 0.0303 | 0.0371 | – | – | 0.9685 |
| Model | clean Acc | Burst-50% | Dropout-50% | Liftoff-50% | Gate AUROC/F1 Burst-50% | Clean FAR (%) | Clean dispersion | MSE reduction Liftoff-50% (%) | Learned |
|---|---|---|---|---|---|---|---|---|---|
| DB3-All9 ( amputees, ) | |||||||||
| Vanilla-Aug | 68.13 12.81 | 63.70 14.53 | 60.49 13.99 | 59.33 15.16 | — | — | — | — | — |
| Gate-Only | 74.93 12.39 | 68.67 13.81 | 68.65 13.77 | 68.48 13.82 | 0.9991 / 0.9904 | 0.41 | 1.405 0.194 | 0.00 0.00 | — |
| w/o Bypass | 75.23 11.51 | 69.03 13.34 | 68.98 13.55 | 69.00 13.44 | 0.9989 / 0.9863 | 0.35 | 0.384 0.038 | +37.09 11.28 | 0.500 (fixed) |
| w/o Align Loss | 74.93 12.31 | 69.46 13.41 | 69.50 13.46 | 69.39 13.48 | 0.9992 / 0.9908 | 0.41 | 1.364 0.185 | -9.79 6.70 | 0.655 0.050 |
| w/o Stop-Grad ( not detached) | 74.86 12.40 | 70.37 13.42 | 69.94 13.42 | 70.11 13.51 | 0.9990 / 0.9870 | 0.58 | 1.277 0.193 | +43.41 6.82 | 0.646 0.040 |
| Cohort & Expansion Pair | Condition | B17 (17 cl.) Gain QA-CMIF Gate-Only ( pp ) | 40/52-Class Gain QA-CMIF Gate-Only ( pp ) | Paired Expansion ( pp ) | 95% CI ( pp ) | Positive Subjects | Paired ( ) | Raw -value (Wilcoxon ) |
| DB3-Core6 ( amputees) BC40 (40 cl.) vs. B17 (17 cl.) | clean | +0.01 0.46 | +0.36 0.22 | +0.35 0.46 | 4/6 | +1.873 (+0.765) | 0.120 (0.156) | |
| Burst-25% | +0.49 0.56 | +1.55 0.33 | +1.06 0.47 | 6/6 | +5.529 (+2.257) | 0.00265 (0.0313) | ||
| Dropout-25% | +0.44 0.56 | +1.58 0.27 | +1.14 0.48 | 6/6 | +5.784 (+2.361) | 0.00217 (0.0313) | ||
| Liftoff-25% | +0.55 0.57 | +1.58 0.31 | +1.03 0.49 | 6/6 | +5.152 (+2.103) | 0.00361 (0.0313) | ||
| Burst-50% | +1.17 0.62 | +2.59 0.36 | +1.43 0.62 | 6/6 | +5.672 (+2.316) | 0.00237 (0.0313) | ||
| Dropout-50% | +1.22 0.69 | +2.60 0.14 | +1.37 0.57 | 6/6 | +5.868 (+2.395) | 0.00204 (0.0313) |
EMG-Based Gesture Recognition
EMG-based gesture recognition decodes hand and arm movements from muscle electrical signals for prosthetic control, wearables, and human-machine interfaces.
See all 29 papers