The rise of mobile devices equipped with numerous sensors, such as LiDAR and cameras, has driven the adoption of multi-modal deep intelligence for distributed sensing tasks, such as smart cabins and driving assistance. However, the arrival time of mobile sensory data vary due to modality size and network dynamics, which can lead to delays (if waiting for slow data) or accuracy decline (if inference proceeds without waiting). Moreover, the diversity and dynamic nature of mobile systems exacerbate this challenge. In response, we present a shift to opportunistic inference for asynchronous distributed multi-modal data, enabling inference as soon as partial data arrives. While existing methods focus on optimizing modality consistency and complementarity, known as modal affinity, they lack a computational approach to control this affinity in open-world mobile environments. % \sysname pioneers the formulation of structured cross-modality affinity in mobile contexts using a hierarchical analysis-based normalized matrix. This approach accommodates the diversity and dynamics of modalities, generalizing across different types and numbers of inputs. Employing an multi-modal lightweight Swin Transformer (MMLST), \sysname facilitates real-time and flexible data imputation, adapting to various modalities and downstream tasks without retraining. Experiments show that \sysname significantly reduces inference latency by up to 80.4% and enhances accuracy by up to 62.1%, while achieving nearly a 50% reduction in energy consumption, outperforming status quo approaches. Also, this method can enhance LLM performance to preprocess asynchronous data.
Figures & tables
Fig. 1: (a) Inference accuracy, (b) accuracy & latency of existing syn-/asynchronous methods, under varying missing rates of slow modality ( i.e. 3D point cloud).
Fig. 2: (a) Top LiDAR emphasizes complementarity and (b) Non-Top LiDAR emphasizes consistency.
Fig. 3: Workflow of AdaFlowLite .
Fig. 4: Inference latency and accuracy of BOD, BBS, DR tasks in existing a-/synchronous methods.
Fig. 6: Shifted Window applied in RGB images and cloud point.
Fig. 7: Illustration of multi-modal lightweight Swin Transformer (MMLST) for real-time data imputation on resource-limited mobile devices.
Fig. 8: Illustration of the Swin-T block in the proposed MMLST.
Fig. 9: Comparison of accuracy and inference latency between AdaFlowLite and baselines in two tasks, under varying missing rates of slow data ( i.e. lagging delay).
Models
Parameters
FLOPs(GFLOPs)
Storage Requirements(MB)
AdaFlow
354233600
4.648
1351.29
AdaFlowLite
178836346
2.256
682.21
TABLE I: Comparison of parameters between AdaFlow and AdaFlowLite .
Fig. 10: AdaFlowLite’s imputation performance in BEVFusion under different missing rates of slow data.
Fig. 11: AdaFlowLite’s imputation performance in PointPillars under different missing rates of slow data.
Fig. 12: Performance over diverse downstream tasks.
Fig. 13: Comparison between AdaFlowLite and Swin-T block.
Fig. 14: Performance on diverse modality numbers.
Fig. 15: Visualization of AdaFlowLite’s affinity matrix.
Fig. 16: Visualization of expected affinity matrix.
Fig. 17: Paradigms of AdaFlowLite and baselines.
Fig. 18: Illustration of seven sensors (six cameras and one LiDAR) on vehicle.
Fig. 19: Comparison between AdaFlowLite and asynchronous baselines (AdaFlow, KNN, PCN) in the self-collect dataset under 66% data missing rates of LiDAR.
Fig. 20: Performance with diverse bandwidths.
Fig. 21: Energy cost of AdaFlowLite on AGX Xavier.
Multimodal foundation models (MFMs) support generalizable inference in distributed mobile sensing applications by integrating heterogeneous data streams. However, existing MFM inference pipelines typically require complete and synchronized multimodal inputs, whereas mobile sensing streams arrive asynchronously due to network variability and heterogeneous data volumes across modalities. Waiting for all modalities increases end-to-end latency, while inference with incomplete inputs can reduce accuracy. We argue that multimodal inference should depend on the utility of available information rather than modality completeness. Based on this principle, we propose AdaModalX, a progressive multimodal inference framework that decouples inference execution from modality synchronization through runtime utility-aware orchestration. At each device, importance-aware data uploading uses a lightweight probe distilled from the edge-side MFM to allow reliable samples to exit locally while uncertain samples continue uploading. At the edge, on-demand MFM activation constructs candidates from fully received samples and, when needed, imputed features derived from partially received delayed inputs. A lightweight contribution assessor built on frozen MFM encoders estimates candidate utility and activates the MFM once the candidate meets its corresponding utility threshold. Experiments on four mobile sensing applications show that AdaModalX improves the accuracy-latency trade-off, achieving up to 26.97% accuracy improvement and up to 64.75% end-to-end latency reduction.
Xiangrui Xu, Sicong Liu, Zhiwen Yu +2
the Northwestern Polytechnical University, Xi’an 710072, China · Harbin Engineering University, Harbin 150001, China · the City University of Hong Kong, Hong Kong 999077, China
We consider a remote inference system with multiple modalities, where a multimodal machine learning (ML) model performs real-time inference using features collected from remote sensors. When sensor observations evolve dynamically over time, fresh features are critical for inference tasks. However, timely delivery of features from all modalities is often infeasible under limited network resources. To address this challenge, we formulate a multimodal scheduling problem to minimize the ML model's inference error. We model this error as a general function of the Age of Information (AoI) vector, where AoI quantifies data freshness. We cast the problem as a semi-Markov decision process (SMDP) and derive an equivalent reformulation with a reduced state set. We then show that the problem has fundamentally different chain structures in the two-modality and multi-modality cases. For the two-modality case, we prove that the optimal policy has an index-based threshold structure. For the general multi-modality case (two or more modalities), we develop an optimal switching-type policy (OPT-ST), which is computed using a multichain policy iteration algorithm (MPI). To further reduce complexity, we also develop two low-complexity policies: the round-robin-like switching-type policy (RR-ST) and the round-robin-like threshold policy (RR-Thres). Numerical results show that OPT-ST achieves the lowest inference error among all baselines, with reductions of up to 80.3%. In a five-modality case, RR-ST and RR-Thres achieve 5.2× and 89.8× speedups in offline computation over OPT-ST, respectively, at the cost of increasing the inference error by 23.3% and 38.6%, respectively.
Keyuan Zhang, Yin Sun, Bo Ji
Department of Computer Science, Virginia Tech · Department of Electrical and Computer Engineering, Auburn University
Multimodal systems often encode every available input, even when a subset suffices for prediction. Adaptive acquisition can reduce this cost by using predictions from incrementally fused evidence to decide which modality to encode next and when to stop. However, sequential fusion makes these predictions order-dependent, so decisions based on them may need to distinguish factorially many histories of the same acquired set. We introduce SemARC, which couples a Sequential Modality Aggregator (SeMA) with an Adaptive Runtime Controller (ARC) and uses acquired evidence to select each modality before its encoder runs. SeMA executes only selected encoder and fusion branches, updates a fixed-size state, and predicts after each acquisition without recomputing earlier branches. We supervise every acquisition prefix under randomized modality subsets and orders to encourage consistent predictions across acquisition orders. ARC combines a set-dependent marginal-utility prior with residual fitted-Q learning to select the next available modality or stop, without inspecting unacquired inputs or retaining acquisition order. Across six multimodal classification datasets and eleven baselines, SemARC achieves 3.2% higher macro-F1 and 61.4% lower total inference GFLOPs on average relative to each dataset's most accurate baseline. End-to-end latency falls by 44.0% across GPU and CPU and by 47.2% on Android INT8 relative to the fastest measured baseline, on average. Under varying runtime modality missingness, SemARC still skips available modalities, matching or exceeding the best baseline macro-F1 in 21 of 24 conditions with 14.8% lower total GFLOPs on average. SemARC thus offers a practical path toward efficient multimodal inference across heterogeneous devices.