Edge-based multimodal medical monitoring requires models that balance diagnostic accuracy with severe energy constraints. Continuous acquisition of ECG, PPG, EMG, and IMU streams rapidly drains wearable batteries, often limiting operation to under 10 hours, while existing systems overlook the high temporal redundancy present in physiological signals. We introduce Adaptive Multimodal Intelligence (AMI), an end-to-end framework that jointly learns when to sense and how to infer. AMI integrates three components: (1) a lightweight Agentic Modality Controller that uses differentiable Gumbel-Sigmoid gating to dynamically select active sensors based on model confidence and task relevance; (2) a Learned Sigma-Delta Sensing module that applies patch-wise Delta-Sigma operations with learnable thresholds to skip temporally redundant samples; and (3) a Foundation-backed Multimodal Prediction Model built on unimodal foundation encoders and a cross-modal transformer with temporal context, enabling robust fusion even under gated or missing inputs. These components are trained jointly via a multi-objective loss combining classification accuracy, sparsity regularization, cross-modal alignment, and predictive coding. AMI is hardware-aware, supporting dynamic computation graphs and masked operations, leading to real energy and latency savings. Across MHEALTH, HMC Sleep, and WESAD datasets, it reduces sensor usage by 48.8% while improving state-of-the-art accuracy by 1.9% on average.
Figures & tables
Figure 1. Illustrative battery life for uni-modal vs. multi-modal sensing (300 mWh battery). Low-power IMU-only monitoring can last for many days, but adding ECG, PPG, and EMG rapidly reduces battery life to tens of hours.
Figure 2. Our Unified Agentic Multimodal Sensing and Inference Framework for Efficient, High-Accuracy Biomedical AI.
Figure 3. Proposed AMI architecture. The Σ - Δ module performs patch-wise computation and adaptive skipping. Features are fused via early cross-attention and processed by a Tiny-Transformer backbone with context reasoning. A Gumbel-MLP agent dynamically produces per-modality masks for the next window.
Figure 4. Thresholding and skipping in Sigma–Delta Sensing.
Figure 5. Training pipeline with unrolled timesteps optimized via BPTT. Each fused state St produces a prediction loss, gating loss, predictive coding loss, and a contrastive alignment loss computed against a memory bank. The controller’s gating actions At influence future observations, and all losses jointly update the model through temporal backpropagation.
Dataset
Method
Acc. (%)
F1
Sensing (%)
MHEALTH
Almujally et al. (2025) ( Almujally and others, 2025 )
94.67
-
100
Debache et al. ( Debache and others, 2020 )
98.2
-
100
Sharma et al. ( Sharma et al., 2023 )
99.07
99.1
100
AMI (no sensing reduction)
99.10
99.13
100
AMI ( λ2=0.05 )
99.12
99.12
38.19
AMI ( λ2=0.1 )
99.04
99.04
33.3
Table 1. Performance comparison of the proposed method and prior state-of-the-art approaches on MHEALTH, HMC, and WESAD datasets. Accuracy (Acc.), F1 score (F1), and average modality sensing rate (Sensing) are reported. We also show the effect of varying the gating loss coefficient λ2 on the trade-off between predictive performance and sensor efficiency.
Figure 6. Sensing rate heatmap over patches obtained from the proposed method on ( Left ) MHEALTH and ( Right ) HMC.
Configuration
Acc.
F1
Sensing
AMI (full model)
99.12
99.12
38.19
Sensing reduction ablation
w/o AMC
98.89
99.02
78.90
w/o Σ - Δ Sensing
99.09
99.12
41.67
FMPM ablation (w/o any sensing reduction)
w/o cross-modal fusion
98.11
98.12
-
Table 2. Ablation study on MHEALTH. We report metrics for the AMI pipeline (all components turned on) and for variants with individual components removed. Sensing reduction, FMPM, and loss-function contributions are evaluated separately to isolate their effects.
p
Acc. (%)
F1 (%)
0
99.12
99.12
0.2
99.10
99.12
0.5
99.08
99.01
0.8
87.21
88.47
Table 3. (Left) Robustness under random modality dropout, and (Right) Effect of input sampling rate on MHEALTH.
Dataset
Model
Lat. (ms)
Eng. (mJ)
Savings (lat./eng.)
MHEALTH
FMPM (100% sensing)
5.28
254.74
–
AMI (38% sensing)
3.69
200.14
30.01% / 21.40%
HMC
FMPM (100% sensing)
3.17
175.39
–
AMI (73% sensing)
2.87
147.31
9.46% / 16.01%
WESAD
FMPM (100% sensing)
5.68
257.32
–
AMI (42% sensing)
2.48
161.64
56.33% / 37.1%
Table 4. Runtime latency and energy comparison between the baseline and proposed AMI during inference on an ARM CPU.
Figure 7. Per-iteration latency (left) and energy (right) as the sensing rate varies on MHEALTH. Measurements are across ARM CPU, Jetson (TensorRT) and A6000 (PyTorch and TensorRT). Latency is decomposed into AMC and FMPM and values are in of ms.
Physiological foundation models (FMs) have shown promise for biosignal representation learning, yet most remain confined to a single modality such as EEG, ECG, or PPG, largely because paired multimodal datasets are scarce. In this paper, we present PanLUNA, a compact 5.4M-parameter pan-modal FM that jointly processes EEG, ECG, and PPG within a single shared encoder. Extending LUNA's channel-unification module, PanLUNA treats multimodal channels as entries in a unified query set augmented with sensor-type embeddings, enabling efficient cross-modal early fusion while remaining inherently robust to missing modalities at inference time. Despite its small footprint, PanLUNA matches or exceeds models up to 57× larger: 81.21% balanced accuracy on TUAB abnormal EEG detection and state-of-the-art 0.7416 balanced accuracy on HMC multimodal sleep staging. Quantization-aware training with INT8 weights recovers ≥96% of full-precision performance, and deployment on the GAP9 ultra-low-power RISC-V microcontroller for wearables achieves 325.6 ms latency and 18.8 mJ per 10-second, 12-lead ECG inference, and 1.206 s latency at 68.65 mJ for multimodal 5-channel sleep staging over 30-second epochs.
Marija Zelic, Anna Tegon, Yawei Li +2
Integrated Systems Laboratory, ETH Zürich, Zürich, Switzerland · DEI, University of Bologna, Bologna, Italy
Recent advances in multimodal large language mod- els (MLLMs) have opened new opportunities for edge intelligence by enabling reasoning across heterogeneous sensor modalities, such as vision, text, and telemetry data. However, deploying these capabilities on resource-constrained edge platforms remains challenging due to the substantial computational, memory, and communication demands of modern MLLMs. Rather than transmitting raw sensor observations or partitioning neural networks at intermediate layers, Edge Multi-Modal Intelligence (EMMI) communicates a compact representation between edge devices and server resources, enabling communication-efficient edge MLLM inference. To achieve this, EMMI performs modality-specific encoding, cross-modal representation fusion, and learned compression at the edge, transmitting only a compact latent representation to server-side resources for high-capacity MLLM reasoning. This representation-centric design reduces communication overhead, preserves local data privacy, and provides a fixed-size interface between heterogeneous edge devices and server-side MLLMs. Evaluation on a representative multimodal benchmark demonstrates that EMMI can reduce the communication payload by 32x while maintaining comparable downstream accuracy, resulting in up to a 3.4x reduction in estimated end-to-end inference latency under bandwidth-constrained edge conditions.
Edge devices increasingly run multimodal sensing pipelines that must remain accurate despite fluctuating power budgets and unpredictable sensor dropout. Existing pruning methods fail under these conditions: they generally require fine-tuning after compression, consuming over 10× the deployment energy, and they assign static importance scores that are blind to which sensors are present. We present the SentryFuse framework, which addresses both challenges jointly through two key components. First, SentryGate learns modality-conditioned importance scores during training via first-order saliency supervision and then prunes attention heads and feed-forward channels at deployment without fine-tuning. Second, SentryAttend replaces dense self-attention, a key bottleneck in contemporary multimodal architectures, with sparse grouped-query attention, yielding a net 15% reduction in GFLOPs across three different multimodal architectures. Across three applications and multimodal backbones, SentryGate achieves a 12.7% average accuracy improvement over the strongest pruning baseline, and upto to 18% under modality dropout conditions. Together, SentryFuse reduces memory by 28.2% and lowers latency by up to 1.63× without further fine-tuning, establishing modality-aware zero-shot compression as a practical path to multimodal intelligence on heterogeneous edge hardware.