Federated Multi-Modal Human Activity Recognition using Multi-Agent Reinforcement Learning

Debasmita Dey; Tanmay Sen; Himel Mallick

cs.AISep 27, 2026

Federated Multi-Modal Human Activity Recognition using Multi-Agent Reinforcement Learning

Authors: Debasmita Dey, Tanmay Sen, Himel Mallick

Organizations: SQC & OR Unit Indian Statistical Institute Kolkata Kolkata, India · Department of Population Health Sciences Cornell University 425 E 61st Street New York

Abstract

Human Activity Recognition (HAR) from heterogeneous wearable sensors is fundamental to the Internet of Health Things (IoHT), supporting rehabilitation, elderly care, and smart healthcare. Existing multimodal fusion methods often assign fixed equal weights to sensor streams, overlooking differences in modality importance, acquisition cost, and sensor quality, which can vary due to movement, incorrect placement, or temporary blockage. We propose an adaptive and cost-aware multimodal HAR framework based on multi-agent reinforcement learning for centralized HAR and extend it to federated learning as FedMHAR. In the centralized setting, multimodal fusion is formulated as a cooperative Multi-Agent Reinforcement Learning (MARL) problem, where each sensing modality is assigned a PPO-based agent that learns per-sample fusion weights, enabling the model to emphasize informative modalities while down-weighting costly sensors when cheaper alternatives provide sufficient information. In the federated setting, we introduce BiFL-PPO, a bidirectional federated optimization strategy in which a server-side PPO policy learns client-specific trust weights and feeds them back to adapt local learning rates and proximal regularization. Unlike round-level optimization, BiFL-PPO uses dense batch-level rewards for more frequent feedback and stable training under heterogeneous client data. Evaluation on the MEx Rehabilitation and UTD Multimodal Human Action datasets shows that the centralized framework achieves 87.30% and 94.98% accuracy, respectively, outperforming conventional fusion methods and state-of-the-art HAR models. FedMHAR achieves 79.74% and 77.49% in the federated setting, consistently surpassing FedAvg, FedProx, FedBN, FedNova, and AdaFedProx, while providing more stable performance and reducing sensor acquisition cost.

Figures & tables

Figure 1: Overview of the centralized MAPPO-MoE architecture. Each modality’s embedding, extracted by a shared multimodal encoder, is concatenated into a joint state $s$ observed by every modality’s MAPPO agent; each agent outputs a trust weight $w_{m}$ for its own modality’s expert via a Beta policy, and these weights are combined via softmax to fuse the experts’ logits into the final prediction. A shared critic $V(s)$ estimates the value of the state, and a reward based on the classification-accuracy gain over an equal-weight baseline, penalized by sensor cost, is used to periodically update the agents and critic via a clipped-surrogate PPO objective.

\hat{\mathbf{y}}=\frac{1}{C}\sum_{c=1}^{C}g_{c}(f_{c}(\mathbf{x}_{c}))

Algorithm 1 Centralized MAPPO-MoE for Cost-Aware Multimodal HAR

Figure 2: Overview of the BiFL-PPO server-client architecture. Each client reports its local training state $S^{t}$ (validation accuracy, loss, gradient norm, data-share) to a shared server-side trunk network; per-client heads then produce trust weights $w^{t}$ , which are sent back to each client and simultaneously used to weight aggregation. Transitions $(S^{t},\log\pi,r^{t})$ are stored in a trajectory buffer and, every three rounds, used to compute Generalized Advantage Estimates and perform a PPO update on the policy and critic.

Hyperparameter	Centralized	Federated
Pretraining Epochs	40	–
MoE Training Epochs	40	–
MAPPO Training Epochs	60	–
Communication Rounds	–	50
Local Epochs (MEx)	–	3
Local Epochs (UTD-MHAD)	–	2

Table 1: Hyperparameter Settings for Centralized and Federated Learning

Panel of Figure 3 — Figure 3: (a) Loss value of the actor network of MAPPO for MEx dataset; (b) Loss value of the critic network of MAPPO for MEx dataset;

Figure 4: The Improvement of the MAPPO-Based Centralized Architecture with respect to the Baseline MoE for MEx Dataset

Exercise	Precision	Recall	F1-score
Knee-rolling	0.81	0.91	0.86
Bridging	0.83	0.80	0.82
Pelvic-tilt	0.93	0.83	0.88
The-clam	0.88	0.88	0.88
Ext-in-lying	0.98	0.94	0.96
Prone-punches	0.93	0.81	0.87

Table 2: Classification Report for MAPPO MoE

Method	Test Accuracy (%, Mean $\pm$ Std)
Baseline MoE	85.07 $\pm$ 4.04
MAPPO MoE (Ours)	87.30 $\pm$ 2.53

Table 3: Comparison of Baseline MoE and MAPPO MoE Fusion on the MEx Dataset (5-Seed Evaluation)

Panel of Figure 5 — Figure 5: (a) Loss value of the actor network of MAPPO for UTD MHAD dataset; (b) Loss value of the critic network of MAPPO for UTD MHAD dataset;

Figure 6: The Improvement of the MAPPO-Based Centralized Architecture with respect to the Baseline MoE for UTD MHAD Dataset

Class	P	R	F1	Class	P	R	F1
swipe-left	0.89	1.00	0.94	arm-curl	1.00	1.00	1.00
swipe-right	0.89	1.00	0.94	tennis-serve	1.00	1.00	1.00
wave	0.84	1.00	0.91	push	0.94	1.00	0.97
clap	0.94	1.00	0.97	knock	0.92	0.75	0.83
throw	1.00	0.88	0.93	catch	1.00	0.62	0.77
arm-cross	1.00	1.00	1.00	pickup-throw	1.00	0.88	0.93

Table 4: Classification Report for MAPPO MoE

Method	Test Accuracy (%, Mean $\pm$ Std)
Baseline MoE	82.84 $\pm$ 0.67
MAPPO MoE (Ours)	94.98 $\pm$ 0.88
Mean Improvement	+12.14

Table 5: Comparison of Baseline MoE and MAPPO MoE Fusion on the UTD-MHAD Dataset (5-Seed Evaluation)

Panel of Figure 7 — Figure 7: (a) Test Accuracy of SOTA FL methods over epochs (b) Test Accuracy of SOTA FL with PPO over epochs.

Method	Mean	Std
FedAvg	78.09	4.83
BiFL-PPO (Ours)	79.74	0.98
Mean improvement (BiFL-PPO $-$ FedAvg)	+1.64	–

Table 6: Bidirectional Server-PPO (BiFL-PPO) vs. FedAvg on MEx Dataset (5-seed, non-IID Dirichlet

\alpha{=}0.5

Figure 8: Confusion matrices of Baseline MoE and MAPPO-MoE on the MEx dataset

True $\backslash$ Pred	Knee-roll.	Bridging	Pelvic-ti.	The-clam	Ext-in-ly.	Prone-pun.	Superman	Precision	Recall	F1-score
Knee-roll.	170	0	1	5	0	0	0	1.00	0.97	0.98
Bridging	0	160	3	12	0	2	2	0.79	0.89	0.84
Pelvic-ti.	0	43	78	33	5	3	11	0.93	0.45	0.61
The-clam	0	0	2	158	0	0	0	0.76	0.99	0.86
Ext-in-ly.	0	0	0	1	182	0	0	0.96	0.99	0.98
Prone-pun.	0	0	0	0	2	143	33	0.97	0.80	0.88

Table 7: Confusion Matrix and Per-Class Metrics for MAPPO MoE (MEx)

Figure 9: Test Accuracy: SOTA FL vs. Server PPO Aggregator for MEx Dataset

Panel of Figure 10 — Figure 10: (a) Test Accuracy of SOTA FL on UTD MHAD dataset; (b) Test Accuracy of SOTA FL using PPO on UTD MHAD dataset

Figure 11: Test Accuracy: SOTA FL vs. Server PPO Aggregator for UTD MHAD Dataset

Method	Vanilla Mean $\pm$ Std	ServerPPO Mean $\pm$ Std
FedAvg	73.0 $\pm$ 4.43	77.5 $\pm$ 3.26
FedProx	74.40 $\pm$ 2.87	77.30 $\pm$ 1.59
FedBN	73.9 $\pm$ 2.65	77.3 $\pm$ 2.95
AdaFedProx	73.26 $\pm$ 2.62	77.2 $\pm$ 1.84
FedNova	73.7 $\pm$ 3.55	76.4 $\pm$ 2.27
FedDPO	73.6 $\pm$ 2.55	–

Table 9: 5-Seed Ablation Study: SOTA FL vs. Server PPO Aggregator for UTD MHAD Dataset

Category	Method	Accuracy (%)
Prior Baselines	LSTM-CNN [ 10 ]	$84.61\pm 3.09$
	MhaGNN [ 32 ]	$87.11\pm 2.62$
	TCN-Inception [ 39 ]	$81.38\pm 3.22$
	HART [ 40 ]	$80.00\pm 4.35$
	MobileHART [ 40 ]	$84.08\pm 2.74$
Centralised	Baseline MoE	$84.80\pm 2.53$

Table 10: Comparison with Baselines on MEx

Category	Method	Accuracy (%)
Prior Baselines	LSTM-CNN [ 10 ]	$92.74\pm 0.98$
	MhaGNN [ 32 ]	$92.88\pm 0.54$
	TCN-Inception [ 39 ]	$93.40\pm 0.71$
	HART [ 40 ]	$61.77\pm 2.61$
	MobileHART [ 40 ]	$75.07\pm 2.53$
Centralised	Baseline MoE	$82.84\pm 0.67$

Table 11: Comparison with Baselines on UTD-MHAD

Modalities	Baseline MoE (%)	MAPPO MoE (%)
acw	41.01 $\pm$ 0.76	35.20 $\pm$ 2.16
act	61.95 $\pm$ 1.98	62.39 $\pm$ 1.00
dc	77.74 $\pm$ 2.42	75.99 $\pm$ 2.16
pm	66.78 $\pm$ 4.30	66.89 $\pm$ 6.65
acw+act	68.31 $\pm$ 2.38	71.16 $\pm$ 1.00
acw+dc	76.21 $\pm$ 2.01	75.33 $\pm$ 0.33

Table 12: Centralized Modality Ablation on MEx Dataset

Modalities	Baseline MoE (%)	MAPPO MoE (%)
skeleton	75.67 $\pm$ 0.58	76.00 $\pm$ 0.33
inertial	86.28 $\pm$ 0.71	87.21 $\pm$ 0.93
joint_vel	71.67 $\pm$ 0.62	70.84 $\pm$ 0.84
inertial + skeleton	91.91 $\pm$ 0.86	94.19 $\pm$ 2.28
skeleton + joint_vel	80.88 $\pm$ 0.79	81.91 $\pm$ 1.02
inertial + joint_vel	91.81 $\pm$ 0.91	93.21 $\pm$ 1.40

Table 13: Centralized Modality Ablation on UTD-MHAD (5-seed).

Figure 12: Confusion matrices of Baseline MoE and MAPPO-MoE on the UTD MHAD Dataset

Figure 13: Confusion matrices of Baseline MoE and MAPPO-MoE on the UTD MHAD Dataset

Explore similar work

Jun 26, 2026cs.LG

A Comparison of Fusion Techniques for Multi-Modal Human Activity Recognition on the HARMES Dataset

Recent advances in Human Activity Recognition (HAR) from wearable sensors have shown that multi-modal deep learning models consistently outperform their uni-modal counterparts. Modalities can include IMUs, RGB cameras, audio signals, and others. One important aspect of multi-modal deep learning is the sensor fusion approach we apply. Over recent years, multiple fusion paradigms have been proposed for multi-modal HAR. However, to the best of our knowledge, no head-to-head comparison of these paradigms exists on a common multi-modal HAR benchmark dataset. To address this research gap, we systematically compare seven state-of-the-art sensor fusion methods on the recently released HARMES dataset, which comprises 61 hours of fully labeled IMU, audio, and ambient humidity data. The chosen dataset focuses on 15 household and personal hygiene activities of daily living (ADLs). By applying the seven different fusion techniques to a state-of-the-art multi-modal model architecture, we show that Gated Multi-modal Fusion achieves the highest macro F1-score (0.82), surpassing the concatenation-based late fusion HARMES paper baseline of 0.76 by +6pp under leave-one-participant-out evaluation. All code used in our experiments is made publicly available on GitHub.

Ahmed Mohamady, Robin Burchard, Kristof Van Laerhoven

University of Siegen, Germany

Multimodal Fusion Human Activity Recognition Multimodal Deep Learning Maestro Dataset

Ask PDF

Apr 2, 2026cs.CV

COMPASS: Fusion-Matched Supervision for Missing-Modality Human Sensing

Multimodal human activity recognition (HAR) and human pose estimation (HPE) must cope with modalities missing at inference. Completions generated for a fixed fusion model must preserve the fusion readout: what fusion uses from each modality. We propose COMPASS, a completion-and-fusion framework in which each modality occupies a fixed slot filled by an observed representation or a completion inferred from available inputs. Its core principle, fusion-matched supervision, supervises completions with same-sample real targets at readout granularity: token means for HAR, where fusion averages tokens, and per-joint representations for HPE, where fusion preserves joint structure. Trained with complete multimodal samples, COMPASS improves subset-averaged performance over the strongest baselines on XRF55 and MM-Fi for HAR and on MM-Fi for HPE. At fixed architecture, fusion-matched supervision outperforms full-token and centered-token matching in HAR, and joint-mean matching and no matching in HPE. The code is available at: https://github.com/haowangcoder/COMPASS.

Hao Wang, Yanyu Qian, Pengcheng Weng +4

Universität Bern, Switzerland · Xi’an Jiaotong University, China · Nanyang Technological University, Singapore

Human Activity Recognition

Ask PDF

Apr 23, 2026cs.LG

Channel-Free Human Activity Recognition via Inductive-Bias-Aware Fusion Design for Heterogeneous IoT Sensor Environments

Human activity recognition (HAR) in Internet of Things (IoT) environments must cope with heterogeneous sensor settings that vary across datasets, devices, body locations, sensing modalities, and channel compositions. This heterogeneity makes conventional channel-fixed models difficult to reuse across sensing environments because their input representations are tightly coupled to predefined channel structures. To address this problem, we investigate strict channel-free HAR, in which a single shared model performs inference without assuming a fixed number, order, or semantic arrangement of input channels, and without relying on sensor-specific input layers or dataset-specific channel templates. We argue that fusion design is the central issue in this setting. Accordingly, we propose a channel-free HAR framework that combines channel-wise encoding with a shared encoder, metadata-conditioned late fusion via conditional batch normalization, and joint optimization of channel-level and fused predictions through a combination loss. The proposed model processes each channel independently to handle varying channel configurations, while sensor metadata such as body location, modality, and axis help recover structural information that channel-independent processing alone cannot retain. In addition, the joint loss encourages both the discriminability of individual channels and the consistency of the final fused prediction. Experiments on PAMAP2, together with robustness analysis on six HAR datasets, ablation studies, sensitivity analysis, efficiency evaluation, and cross-dataset transfer learning, demonstrate three main findings...

Tatsuhito Hasegawa

Graduate School of Engineering, University of Fukui, Bunkyo, Fukui, 9108507, Fukui, Japan

Human Activity Recognition Global-Local Fusion Framework Uncertainty-Aware Fusion Internet Of Thing Channel

Ask PDF