Organizations: the Northwestern Polytechnical University, Xi’an 710072, China · Harbin Engineering University, Harbin 150001, China · the City University of Hong Kong, Hong Kong 999077, China
Multimodal foundation models (MFMs) support generalizable inference in distributed mobile sensing applications by integrating heterogeneous data streams. However, existing MFM inference pipelines typically require complete and synchronized multimodal inputs, whereas mobile sensing streams arrive asynchronously due to network variability and heterogeneous data volumes across modalities. Waiting for all modalities increases end-to-end latency, while inference with incomplete inputs can reduce accuracy. We argue that multimodal inference should depend on the utility of available information rather than modality completeness. Based on this principle, we propose AdaModalX, a progressive multimodal inference framework that decouples inference execution from modality synchronization through runtime utility-aware orchestration. At each device, importance-aware data uploading uses a lightweight probe distilled from the edge-side MFM to allow reliable samples to exit locally while uncertain samples continue uploading. At the edge, on-demand MFM activation constructs candidates from fully received samples and, when needed, imputed features derived from partially received delayed inputs. A lightweight contribution assessor built on frozen MFM encoders estimates candidate utility and activates the MFM once the candidate meets its corresponding utility threshold. Experiments on four mobile sensing applications show that AdaModalX improves the accuracy-latency trade-off, achieving up to 26.97% accuracy improvement and up to 64.75% end-to-end latency reduction.
Figures & tables
Fig. 1: Example distributed mobile sensing applications in smart homes and intelligent vehicles.
Fig. 2: Comparison of (a) conventional deep models and (b) task-scalable MFM. The former explicitly selects models based on tasks and input modalities, whereas the latter automatically adapts to tasks and input modalities using the same model.
Fig. 3: Impact of modality asynchrony on MFM inference. (a) Illustration of asynchronous data in an audio-vision-based distributed mobile sensing system. (b) Inference accuracy with different few-shot settings. (c) Inference latency and accuracy with full audio data but partial vision data.
Fig. 4: Impact of modalities on different samples. (a) Sample-wise inference accuracy and latency with audio-only v.s. both audio and vision inputs. (b) Distribution of mutually exclusive data samples correctly processed by different modalities. (c) Theoretical upper bound of dynamic multimodal inference, illustrating the potential accuracy gain and latency reduction over existing methods.
Fig. 5: Overview of AdaModalX for MFM-based distributed sensing systems. Partial inputs are arrived data available for feature extraction; arrival states indicate per-modality upload progress, not cross-modal data-size correspondence.
Fig. 6: Overview of importance-aware data uploading at devices (Sec. IV ).
Fig. 7: Illustration of the confidence-aware uploading mechanism under multi-device coordination. Each mobile device performs local probe inference in parallel with data uploading. (a) If the estimated confidence is sufficiently high, the device triggers early exit and produces a local prediction. (b) If the estimated confidence is insufficient, the device continues uploading, and the edge-side MFM completes the inference.
Fig. 8: Overview of on-demand MFM inference activation at the edge.
Fig. 9: Overview of the MFM inference activation mechanism. Mobile devices 1 to k use a confidence-aware uploading mechanism to continuously upload multimodal data. Once the audio sample arrives completely (①), it triggers MFM inference activation (②), calculates the modality utility via assessor (③), uses partial-modality feature imputation to recover complementary multimodal information (④), predicts the imputed feature (⑤), and combines them into a candidate set to check if the utility meets its candidate-specific activation threshold (⑥). If the candidate set does not meet the threshold (⑦), AdaModalX waits for the next data arrival (①); otherwise, it triggers MFM inference (⑧).
Fig. 10: Comparison of accuracy v.s. latency between AdaModalX and other baselines on diverse tasks. Translucent markers show the five individual runs, while opaque markers show their means. The annotations report paired-test p -values for the accuracy comparison between AdaModalX and NA.
Fig. 11: Comparison of modality usage rate(%) v.s. accuracy(%) between AdaModalX and other approaches on diverse tasks.
Fig. 12: Comparison of computational cost between AdaModalX and other approaches on diverse tasks. The annotations report paired-test p -values for AdaModalX versus NA in (a) and AdaModalX versus SF in (b).
Fig. 13: Bandwidth sensitivity of AdaModalX on T2 with M1 under 10Mbps, 20Mbps, 25Mbps, and 50Mbps.
Method
Event detection ( T1 ), 10 Mbps
Driver activity recognition ( T2 ), 20 Mbps
Accuracy (%)
Latency (ms)
Accuracy (%)
Latency (ms)
w/o #D1, w/o #D2
89.35 ± 0.00
462 ± 4.05
72.82 ± 0.00
383 ± 3.72
w/ #D1, w/o #D2
87.93 ± 0.25
258 ± 5.94
73.44 ± 0.32
179 ± 4.18
w/o #D1, w/ #D2
88.73 ± 0.63
234 ± 2.86
75.25 ± 0.41
247 ± 3.67
w/ #D1, w/ #D2
88.66 ± 0.28
163 ± 3.93
73.99 ± 0.27
107 ± 3.26
TABLE I: Ablation study on importance-aware data uploading at devices (#D1) and on-demand inference at edge (#D2) on T1 and T2 tasks under different bandwidths.
Fig. 14: Comparison of accuracy v.s. offloading rate(%) between importance-aware data uploading at devices (#D1) and six baselines on T1 and T2 tasks.
Fig. 15: Effectiveness of on-demand MFM inference activation at edge (#D2) with the importance-aware data uploading at devices (#D1) disabled.
Fig. 16: Sensitivity to slow-modality arrival rates on T1 with M1 .
Distillation loss
Model acc. (%)
Overall acc. (%)
Offloading rate (%)
w/ LPTC , w/o LFA , w/o LCPD
63.55 ± 0.56
74.03 ± 0.42
46.67 ± 2.14
w/ LPTC , w/ LFA , w/o LCPD
63.77 ± 0.44
74.39 ± 0.37
44.87 ± 1.86
w/ LPTC , w/o LFA , w/ LCPD
63.23 ± 0.52
74.39 ± 0.41
44.51 ± 2.48
w/ LPTC , w/ LFA , w/ LCPD
67.69 ± 0.36
75.70 ± 0.54
38.61 ± 1.63
TABLE II: Ablation study on different distillation loss components in the IR modality.
Fig. 17: Reliability of false early-exit rate (FEER) and early-exit coverage under four device-side confidence thresholds on T1 with M1 .
Fig. 18: Reliability of device-side early exit under two confidence scores.
Method
Overall early-exit rate (%)
Overall false early-exit rate (%)
Overall edge rescue rate (%)
AdaModalX
52.62
4.24
2.69
TABLE III: Overall device-side early-exit outcomes of AdaModalX on T1 with M1 .
Method
Event detection
Average accuracy (%)
Average latency (ms)
w/o I, w/o S
75.95 ± 0.00
220 ± 5.20
w/ I, w/o S
83.53 ± 0.60
223 ± 0.41
w/o I, w/ S
86.25 ± 0.51
254 ± 6.70
w/ I, w/ S
88.73 ± 0.63
244 ± 2.86
TABLE IV: Ablation study on the effects of partial multimodal feature imputation (I) and selective inference activation (S) on the accuracy and latency of event detection.
Fig. 19: Comparison of accuracy v.s. latency under network fluctuation on T1 with M1 .
Device
Task
GFLOPs
Params.
Infer. Latency (ms)
Accuracy (%)
Offload. Rate (%)
Distill. Data
Distill. Time (min)
Calib. Time (s)
Vivo X300 Pro
T1 (Image)
1.82
12.75M
36.10 ± 2.69
67.26 ± 1.01
49.05 ± 2.07
50-shot, 28 classes
11.3 ± 0.2
14.5 ± 0.04
Jetson Orin Nano
T3 (RGB vision)
3.64
12.75M
63.72 ± 1.91
43.67 ± 1.75
67.27 ± 2.29
38-shot, 15 classes
7.2 ± 0.1
9.32 ± 0.26
Jetson Orin NX
T3 (IR vision)
3.64
12.75M
60.89 ± 1.86
55.71 ± 0.64
79.49 ± 2.82
38-shot, 15 classes
5.8 ± 0.1
8.86 ± 0.18
Xiaomi 13 Ultra
T3 (IMU)
0.13
2.19M
7.02 ± 0.75
52.02 ± 0.45
77.49 ± 0.64
38-shot, 15 classes
2.8 ± 0.1
4.65 ± 0.08
TABLE V: Per-device performance and resource costs of the device-side confidence estimators.
Module
Latency (ms)
GFLOPs
Modality contribution assessor
2.59 ± 0.05
0.019
Feature imputation module
3.79 ± 0.11
0.003
Full MFM inference
462 ± 4.05
220
TABLE VI: Latency and computational cost of the edge-side modules on T1 with M1 .
Fig. 20: Illustration of the case study with mobile devices.
Fig. 21: Comparison of accuracy v.s. inference time between AdaModalX and other baselines in the real-world case study.
Method
NA
MG
CGAN
EdgeFM
AdaModalX
GFLOPs
335.19
337.01
335.21
280.24
235.26
TABLE VII: Comparison of FLOPs on T3 with M1 .
Fig. 22: Comparison of accuracy v.s. latency in the three-modality human activity recognition case.