Multimodal Remote Inference

Keyuan Zhang; Yin Sun; Bo Ji

cs.LGAug 11, 2025

Multimodal Remote Inference

Authors: Keyuan Zhang, Yin Sun, Bo Ji

Organizations: Department of Computer Science, Virginia Tech · Department of Electrical and Computer Engineering, Auburn University

Abstract

We consider a remote inference system with multiple modalities, where a multimodal machine learning (ML) model performs real-time inference using features collected from remote sensors. When sensor observations evolve dynamically over time, fresh features are critical for inference tasks. However, timely delivery of features from all modalities is often infeasible under limited network resources. To address this challenge, we formulate a multimodal scheduling problem to minimize the ML model's inference error. We model this error as a general function of the Age of Information (AoI) vector, where AoI quantifies data freshness. We cast the problem as a semi-Markov decision process (SMDP) and derive an equivalent reformulation with a reduced state set. We then show that the problem has fundamentally different chain structures in the two-modality and multi-modality cases. For the two-modality case, we prove that the optimal policy has an index-based threshold structure. For the general multi-modality case (two or more modalities), we develop an optimal switching-type policy (OPT-ST), which is computed using a multichain policy iteration algorithm (MPI). To further reduce complexity, we also develop two low-complexity policies: the round-robin-like switching-type policy (RR-ST) and the round-robin-like threshold policy (RR-Thres). Numerical results show that OPT-ST achieves the lowest inference error among all baselines, with reductions of up to 80.3%. In a five-modality case, RR-ST and RR-Thres achieve $5.2\times$ and $89.8\times$ speedups in offline computation over OPT-ST, respectively, at the cost of increasing the inference error by 23.3% and 38.6%, respectively.

Figures & tables

Fig. 1 : A multimodal remote inference system.

Goal	Source(s)	AoI Objective Function	Policy	Optimality	Work
Minimize AoI of a single source	Single source	Linear AoI function	Threshold policy	Optimal	[ 7 ]
		Non-linear AoI function	Threshold policy	Optimal	[ 8 ]
Minimize inference error of a single source	Single source	General AoI function	Index-based threshold policy	Optimal	[ 10 , 14 , 11 , 2 ]
Minimize estimation error of two correlated sources	Two correlated Gaussian processes	Minimum of two non-decreasing AoI functions	Time-shift scheduling	Optimal	[ 15 ]
Joint design for remote monitoring over random delays	Two correlated Wiener processes	Non-decreasing, non-separable function of two AoIs	Water-Filling (WF) sampler Max Age First scheduler Weighted-sum fusion estimator	Optimal	[ 16 ]
Minimize estimation error of multiple correlated sources	Multiple correlated Wiener processes	Weighted sum of AoI functions across sources	Max-Weight-like policy	Constant-factor optimal	[ 17 ]

TABLE I : Qualitative comparison with the state-of-the-art.

Fig. 2 : System model.

Fig. 3 : Illustration of switching times. At the switching time from

M_{1}

M_{2}

, the scheduler selects the decision pair

(d_{\tau},d_{s})

. In this example, we have

d_{\tau}=4

and

d_{s}=M_{3}

Fig. 4 : Illustration of AoI transitions for

M=2

with unit transmission time and zero preprocessing delay. Each circle represents an AoI state, and each arrow is labeled with the selected modality. The process has two switching states,

(1,2)

and

(2,1)

, with reformulated decisions

\tau_{1}

and

\tau_{2}

, respectively.

Algorithm	Time complexity (per-iteration)	Required parameters
Policy iteration (original problem)	$\mathcal{O}(MK^{M})$	$K^{M}$
OPT-ST	$\mathcal{O}\!\left(M(M-1)^{2}K^{M-1}\right)$	$2M(M-1)K^{M-2}$
RR-ST	$\mathcal{O}(MK^{M-1})$	$MK^{M-2}$
RR-Thres	$\mathcal{O}(MK)$	$M$

TABLE II : Summary of Complexity Comparison

Panel of Figure 7 — Fig. 5 : Experimental setup under three different cases.

Panel of Figure 8 — Fig. 6 : Inference error under different AoI vectors.

	Robot state prediction				Hand pose prediction				Vector autoregressive process prediction
Algorithm	MSE	Offline time (s)	Storage overhead	Online time ( $\mu$ s)	$L_{2}$ error (cm)	Offline time (s)	Storage overhead	Online time ( $\mu$ s)	MSE	Offline time (s)	Storage overhead	Online time ( $\mu$ s)
Round-robin	11.4	0	0 B	0.534	3.21	0	0 B	0.622	13.1	0	0 B	0.476
Uniform random	7.59	0	0 B	19.6	5.51	0	0 B	20.0	20.2	0	0 B	20.3
Greedy	8.94	0	0 B	38.0	4.57	0	0 B	104	41.1	0	0 B	46.1
RL	4.50	34.6	24.0 B	2.25	1.64	370	582 B	2.78	10.1	$1.70\times 10^{3}$	2.44 MiB	3.96
Index (ours)	4.50	$2.24\times 10^{-3}$	2.00 B	0.914	–	–	–	–	–	–	–	–

TABLE III : Comparison of scheduling algorithms under different scenarios.

Panel of Figure 10 — Fig. 7 : Average inference error and policy computation time under different AoI maximum values for $M=5$ .

Algorithm	Average error	Offline time (s)	Storage overhead	Online time ( $\mu$ s)
Round-robin	0.111	0	0 B	0.271
Uniform random	0.124	0	0 B	7.53
Greedy	0.132	0	0 B	19.2
RL	0.118	29.6	642 B	2.09
MGF-Corr	0.863	0.684	600 B	14.6
OPT-ST (ours)	0.104	0.384	300 B	2.35

TABLE IV : Comparison of approximation algorithms when the correlation parameter is

\theta=0.9

Fig. 8 : Average inference error across correlation levels.

Algorithm	$M=2$	$M=3$
Round-robin	11.5	3.31
Uniform random	7.59	5.59
Greedy	9.04	4.66
RL	4.67	1.74
Index (ours)	4.59	–
OPT-ST (ours)	4.59	1.74

TABLE V : Average inference error on the testing dataset.

Fig. 9 : Impact of perturbations to the inference error function.

Semi-Markov Decision Process

See all 10 papers

Oct 5, 2026

Efficient Multimodal Inference through Adaptive Acquisition and Sequential Fusion

Multimodal systems often encode every available input, even when a subset suffices for prediction. Adaptive acquisition can reduce this cost by using predictions from incrementally fused evidence to decide which modality to encode next and when to stop. However, sequential fusion makes these predictions order-dependent, so decisions based on them may need to distinguish factorially many histories of the same acquired set. We introduce SemARC, which couples a Sequential Modality Aggregator (SeMA) with an Adaptive Runtime Controller (ARC) and uses acquired evidence to select each modality before its encoder runs. SeMA executes only selected encoder and fusion branches, updates a fixed-size state, and predicts after each acquisition without recomputing earlier branches. We supervise every acquisition prefix under randomized modality subsets and orders to encourage consistent predictions across acquisition orders. ARC combines a set-dependent marginal-utility prior with residual fitted-Q learning to select the next available modality or stop, without inspecting unacquired inputs or retaining acquisition order. Across six multimodal classification datasets and eleven baselines, SemARC achieves 3.2% higher macro-F1 and 61.4% lower total inference GFLOPs on average relative to each dataset's most accurate baseline. End-to-end latency falls by 44.0% across GPU and CPU and by 47.2% on Android INT8 relative to the fastest measured baseline, on average. Under varying runtime modality missingness, SemARC still skips available modalities, matching or exceeding the best baseline macro-F1 in 21 of 24 conditions with 14.8% lower total GFLOPs on average. SemARC thus offers a practical path toward efficient multimodal inference across heterogeneous devices.

Payal Mohapatra, Haodong Yang, Yueyuan Sui +3

Arm Inc., USA · Northwestern University, USA

Efficient Multimodal Inference Multimodal Classification Multimodal Fusion

Ask PDF

Mar 6, 2026

Adapter-Augmented Bandits for Online Multi-Constrained Multi-Modal Inference Scheduling

Multi-modal large language model (MLLM) inference scheduling enables strong response quality under practical and heterogeneous budgets, beyond what a homogeneous single-backend setting can offer. Yet online MLLM task scheduling is nontrivial, as requests vary sharply in modality composition and latent reasoning difficulty, while execution backends incur distinct, time-varying costs due to system jitter and network variation. These coupled uncertainties pose two core challenges: deriving semantically faithful yet scheduling-relevant multi-modal task representations, and making low-overhead online decisions over irreversible multi-dimensional budgets. Accordingly, we propose M-CMAB (Multi-modal Multi-constraint Contextual Multi-Armed Bandit), a multi-adapter-enhanced MLLM inference scheduling framework with three components: (i) a CLS-attentive, frozen-backbone Predictor that extracts compact task representations and updates only lightweight adapters for action-specific estimation; (ii) a primal-dual Constrainer that maintains online Lagrange multipliers to enforce long-horizon constraints via per-round objectives; and (iii) a two-phase Scheduler that balances exploration and exploitation under irreversible budgets. We establish a regret guarantee under multi-dimensional knapsack constraints. On a composite multimodal benchmark with heterogeneous backends, M-CMAB consistently outperforms state-of-the-art baselines across budget regimes, achieving up to 14.18% higher reward and closely tracking an oracle-aided upper bound. Codes are available at https://anonymous.4open.science/r/M2CMAB/.

Xianzhi Zhang, Yue Xu, Yinlin Zhu +4

School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, 510006, China · School of Computing, Macquarie University, NSW 2109, Australia

LLM Inference Scheduling Multi-Armed Bandits Contextual Bandits Multimodal Large Language Models Combinatorial Multi-Armed Bandits

Ask PDF

Jun 22, 2026

Changing Modalities: Adapting Remote Sensing Models to New Satellites and Sensors

Machine learning models for remote sensing are trained and deployed on a static set of modalities. However, as we equip newer satellites with novel sensors and retire old ones, practitioners may wish to deploy an existing model on a substitution, superset, or subset of modalities with minimal retraining given data availability or practical computational constraints. We study the setting of updating existing models to changing modalities and identify three main scenarios: Modality Transfer (substitution), Addition (superset), and Peeking (subset). We propose DeluluNet, an architecture with modular components for all three changing modality scenarios. DeluluNet is trained end-to-end, learning a multi-modal model from a unimodal teacher and unlabeled multimodal data via modality hallucination--predicting missing modality representations from those that are present. As a result, DeluluNet can keep predicting even when input modalities change, providing a practical alternative to re-labeling and re-training in a changing world.

Tim G. Zhou, Anthony Fuller, Geoff Pleiss +1

University of British Columbia · Vector Institute · Carleton University

Missing-Modality Learning Remote Sensing

Ask PDF

Abstract

Semi-Markov Decision Process

Explore similar work

Efficient Multimodal Inference through Adaptive Acquisition and Sequential Fusion

Adapter-Augmented Bandits for Online Multi-Constrained Multi-Modal Inference Scheduling

Changing Modalities: Adapting Remote Sensing Models to New Satellites and Sensors