Multimodal Remote Inference

Keyuan Zhang; Yin Sun; Bo Ji

cs.LGAug 11, 2025

Multimodal Remote Inference

Authors: Keyuan Zhang, Yin Sun, Bo Ji

Organizations: Department of Computer Science, Virginia Tech · Department of Electrical and Computer Engineering, Auburn University

Abstract

We consider a remote inference system with multiple modalities, where a multimodal machine learning (ML) model performs real-time inference using features collected from remote sensors. When sensor observations evolve dynamically over time, fresh features are critical for inference tasks. However, timely delivery of features from all modalities is often infeasible under limited network resources. To address this challenge, we formulate a multimodal scheduling problem to minimize the ML model's inference error. We model this error as a general function of the Age of Information (AoI) vector, where AoI quantifies data freshness. We cast the problem as a semi-Markov decision process (SMDP) and derive an equivalent reformulation with a reduced state set. We then show that the problem has fundamentally different chain structures in the two-modality and multi-modality cases. For the two-modality case, we prove that the optimal policy has an index-based threshold structure. For the general multi-modality case (two or more modalities), we develop an optimal switching-type policy (OPT-ST), which is computed using a multichain policy iteration algorithm (MPI). To further reduce complexity, we also develop two low-complexity policies: the round-robin-like switching-type policy (RR-ST) and the round-robin-like threshold policy (RR-Thres). Numerical results show that OPT-ST achieves the lowest inference error among all baselines, with reductions of up to 80.3%. In a five-modality case, RR-ST and RR-Thres achieve $5.2\times$ and $89.8\times$ speedups in offline computation over OPT-ST, respectively, at the cost of increasing the inference error by 23.3% and 38.6%, respectively.

Figures & tables

Fig. 1 : A multimodal remote inference system.

Goal	Source(s)	AoI Objective Function	Policy	Optimality	Work
Minimize AoI of a single source	Single source	Linear AoI function	Threshold policy	Optimal	[ 7 ]
		Non-linear AoI function	Threshold policy	Optimal	[ 8 ]
Minimize inference error of a single source	Single source	General AoI function	Index-based threshold policy	Optimal	[ 10 , 14 , 11 , 2 ]
Minimize estimation error of two correlated sources	Two correlated Gaussian processes	Minimum of two non-decreasing AoI functions	Time-shift scheduling	Optimal	[ 15 ]
Joint design for remote monitoring over random delays	Two correlated Wiener processes	Non-decreasing, non-separable function of two AoIs	Water-Filling (WF) sampler Max Age First scheduler Weighted-sum fusion estimator	Optimal	[ 16 ]
Minimize estimation error of multiple correlated sources	Multiple correlated Wiener processes	Weighted sum of AoI functions across sources	Max-Weight-like policy	Constant-factor optimal	[ 17 ]

TABLE I : Qualitative comparison with the state-of-the-art.

Fig. 2 : System model.

Fig. 3 : Illustration of switching times. At the switching time from

M_{1}

M_{2}

, the scheduler selects the decision pair

(d_{\tau},d_{s})

. In this example, we have

d_{\tau}=4

and

d_{s}=M_{3}

Fig. 4 : Illustration of AoI transitions for

M=2

with unit transmission time and zero preprocessing delay. Each circle represents an AoI state, and each arrow is labeled with the selected modality. The process has two switching states,

(1,2)

and

(2,1)

, with reformulated decisions

\tau_{1}

and

\tau_{2}

, respectively.

Algorithm	Time complexity (per-iteration)	Required parameters
Policy iteration (original problem)	$\mathcal{O}(MK^{M})$	$K^{M}$
OPT-ST	$\mathcal{O}\!\left(M(M-1)^{2}K^{M-1}\right)$	$2M(M-1)K^{M-2}$
RR-ST	$\mathcal{O}(MK^{M-1})$	$MK^{M-2}$
RR-Thres	$\mathcal{O}(MK)$	$M$

TABLE II : Summary of Complexity Comparison

Panel of Figure 7 — Fig. 5 : Experimental setup under three different cases.

Panel of Figure 8 — Fig. 6 : Inference error under different AoI vectors.

	Robot state prediction				Hand pose prediction				Vector autoregressive process prediction
Algorithm	MSE	Offline time (s)	Storage overhead	Online time ( $\mu$ s)	$L_{2}$ error (cm)	Offline time (s)	Storage overhead	Online time ( $\mu$ s)	MSE	Offline time (s)	Storage overhead	Online time ( $\mu$ s)
Round-robin	11.4	0	0 B	0.534	3.21	0	0 B	0.622	13.1	0	0 B	0.476
Uniform random	7.59	0	0 B	19.6	5.51	0	0 B	20.0	20.2	0	0 B	20.3
Greedy	8.94	0	0 B	38.0	4.57	0	0 B	104	41.1	0	0 B	46.1
RL	4.50	34.6	24.0 B	2.25	1.64	370	582 B	2.78	10.1	$1.70\times 10^{3}$	2.44 MiB	3.96
Index (ours)	4.50	$2.24\times 10^{-3}$	2.00 B	0.914	–	–	–	–	–	–	–	–

TABLE III : Comparison of scheduling algorithms under different scenarios.

Panel of Figure 10 — Fig. 7 : Average inference error and policy computation time under different AoI maximum values for $M=5$ .

Algorithm	Average error	Offline time (s)	Storage overhead	Online time ( $\mu$ s)
Round-robin	0.111	0	0 B	0.271
Uniform random	0.124	0	0 B	7.53
Greedy	0.132	0	0 B	19.2
RL	0.118	29.6	642 B	2.09
MGF-Corr	0.863	0.684	600 B	14.6
OPT-ST (ours)	0.104	0.384	300 B	2.35

TABLE IV : Comparison of approximation algorithms when the correlation parameter is

\theta=0.9

Fig. 8 : Average inference error across correlation levels.

Algorithm	$M=2$	$M=3$
Round-robin	11.5	3.31
Uniform random	7.59	5.59
Greedy	9.04	4.66
RL	4.67	1.74
Index (ours)	4.59	–
OPT-ST (ours)	4.59	1.74

TABLE V : Average inference error on the testing dataset.

Fig. 9 : Impact of perturbations to the inference error function.

Semi-Markov Decision Process

See all 10 papers

Efficient Multimodal Inference through Adaptive Acquisition and Sequential Fusion
Oct 5, 2026Payal Mohapatra, Haodong Yang, Yueyuan Sui +3Efficient Multimodal Inference Multimodal Classification
Arm Inc., USA · Northwestern University, USA
Adapter-Augmented Bandits for Online Multi-Constrained Multi-Modal Inference Scheduling
Mar 6, 2026Xianzhi Zhang, Yue Xu, Yinlin Zhu +4LLM Inference Scheduling Multi-Armed Bandits
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, 510006, China · School of Computing, Macquarie University, NSW 2109, Australia
Changing Modalities: Adapting Remote Sensing Models to New Satellites and Sensors
Jun 22, 2026Tim G. Zhou, Anthony Fuller, Geoff Pleiss +1Missing-Modality Learning Remote Sensing
University of British Columbia · Vector Institute · Carleton University

Oct 5, 2026J/K move · Enter open · S save

Efficient Multimodal Inference through Adaptive Acquisition and Sequential Fusion

Payal Mohapatra, Haodong Yang, Yueyuan Sui, Stephen Xia+2

Arm Inc., USA · Northwestern University, USA

Efficient Multimodal Inference Multimodal Classification Multimodal Fusion

Open paper AskPDF arXiv

Multimodal systems often encode every available input, even when a subset suffices for prediction. Adaptive acquisition can reduce this cost by using predictions from incrementally fused evidence to decide which modality to encode next and when to stop. However, sequential fusion makes these predictions order-dependent, so decisions based on them may need to distinguish factorially many histories of the same acquired set. We introduce SemARC, which couples a Sequential Modality Aggregator (SeMA) with an Adaptive Runtime Controller (ARC) and uses acquired evidence to select each modality before its encoder runs. SeMA executes only selected encoder and fusion branches, updates a fixed-size state, and predicts after each acquisition without recomputing earlier branches. We supervise every acquisition prefix under randomized modality subsets and orders to encourage consistent predictions across acquisition orders. ARC combines a set-dependent marginal-utility prior with residual fitted-Q learning to select the next available modality or stop, without inspecting unacquired inputs or retaining acquisition order. Across six multimodal classification datasets and eleven baselines, SemARC achieves 3.2% higher macro-F1 and 61.4% lower total inference GFLOPs on average relative to each dataset's most accurate baseline. End-to-end latency falls by 44.0% across GPU and CPU and by 47.2% on Android INT8 relative to the fastest measured baseline, on average. Under varying runtime modality missingness, SemARC still skips available modalities, matching or exceeding the best baseline macro-F1 in 21 of 24 conditions with 14.8% lower total GFLOPs on average. SemARC thus offers a practical path toward efficient multimodal inference across heterogeneous devices.

Abstract

Semi-Markov Decision Process

Explore similar work

Efficient Multimodal Inference through Adaptive Acquisition and Sequential Fusion

Adapter-Augmented Bandits for Online Multi-Constrained Multi-Modal Inference Scheduling

Changing Modalities: Adapting Remote Sensing Models to New Satellites and Sensors