cs.LGMar 6, 2026

Adapter-Augmented Bandits for Online Multi-Constrained Multi-Modal Inference Scheduling

Authors: Xianzhi Zhang, Yue Xu, Yinlin Zhu, Di Wu, Yipeng Zhou, Miao Hu, Guocong Quan

Organizations: School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, 510006, China · School of Computing, Macquarie University, NSW 2109, Australia

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multi-modal large language model (MLLM) inference scheduling enables strong response quality under practical and heterogeneous budgets, beyond what a homogeneous single-backend setting can offer. Yet online MLLM task scheduling is nontrivial, as requests vary sharply in modality composition and latent reasoning difficulty, while execution backends incur distinct, time-varying costs due to system jitter and network variation. These coupled uncertainties pose two core challenges: deriving semantically faithful yet scheduling-relevant multi-modal task representations, and making low-overhead online decisions over irreversible multi-dimensional budgets. Accordingly, we propose M-CMAB (Multi-modal Multi-constraint Contextual Multi-Armed Bandit), a multi-adapter-enhanced MLLM inference scheduling framework with three components: (i) a CLS-attentive, frozen-backbone Predictor that extracts compact task representations and updates only lightweight adapters for action-specific estimation; (ii) a primal-dual Constrainer that maintains online Lagrange multipliers to enforce long-horizon constraints via per-round objectives; and (iii) a two-phase Scheduler that balances exploration and exploitation under irreversible budgets. We establish a regret guarantee under multi-dimensional knapsack constraints. On a composite multimodal benchmark with heterogeneous backends, M-CMAB consistently outperforms state-of-the-art baselines across budget regimes, achieving up to 14.18% higher reward and closely tracking an oracle-aided upper bound. Codes are available at https://anonymous.4open.science/r/M2CMAB/.

Figures & tables

Figure 1: Comparison of reward, latency, and monetary cost across MLLM inference backends on a mixed multi-modal task trace.
Figure 2: Overview of the M 2 -CMAB decision-making pipeline.
Figure 3: Average inference reward of M 2 -CMAB and baselines across six datasets and three budget regimes.
Figure 4: Sensitivity of M 2 -CMAB to the initial phase ratio, i.e., (A+1)T0T\frac{(A+1)T_{0}}{T} , evaluated by average inference reward (y-axis). Bars indicate the percentage of total rounds allocated to the initial phase, with three groups per subfigure corresponding to three budget regimes.
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

NotationDefinition
t / Tt\ /\ TIndex of the online scheduling round and the total number of rounds.
Ω\OmegaSet of supported input modalities (e.g., text, vision, audio).
xt={xωt}ω∈Ω\bm{x}^{t}=\{\bm{x}^{t}_{\omega}\}_{\omega\in\Omega}Multi-modal context of the task arriving at round tt , consisting of modality-specific token embeddings.
at∈Aa^{t}\in\mathcal{A}Action selected at round tt , corresponding to an MLLM inference configuration.
AACardinality of the action space.
ratr_{a}^{t}Observed reward of executing action aa under context xt\bm{x}^{t} at round tt .
Appendix
Table 1: Main notations and definitions.
HyperparametersDescription
PlocalP_{\text{local}}Average power consumption of the local device (W), 600
κ\kappaElectricity price coefficient (cost per Joule), 2.06e-8
Appendix
Table 2: Key hyperparameters used in the scheduling framework (placeholders).
Figure 6: Comparison of reward, latency, and monetary cost across MLLM inference backends on individual datasets and a mixed multi-modal task trace.
ROUGE-LReward
01
(0, 0.15)2
[0.15, 3)3
[0.3, 0.4)4
(0.4, 1]5
Appendix
Table 3: Mapping from ROUGE-L scores to discrete reward levels.
CategoryModelProvider / Deployment
LocalQwen3-VL-2B-InstructOn-device
CloudGPT-5-nanoOpenAI API
CloudQwen3-VL-32B-InstructSiliconFlow API
CloudQwen3-VL-30B-A3B-InstructOn-device
CloudGLM-4.6V-ThinkingSiliconFlow API
Appendix
Table 4: Execution models used in the scheduling action space.

Explore similar work

CardsList

  1. TCM-Serve: Modality-aware Scheduling for Multimodal Large Language Model Inference

    Mar 27, 2026Konstantinos Papaioannou, Thaleia Dimitra DoudaliLLM ServingLLM Inference Scheduling

    IMDEA Software Institute, Universidad Politécnica de Madrid

  2. Look Less, Think Faster: Joint Token-Compute Adaptation for Multimodal LLMs

    Jul 22, 2026Pengcheng Wang, Zhiquan Wang, Jayoung Lee +5Efficient Multimodal InferenceEfficient VLM Inference

    Purdue University, West Lafayette, IN, USA · University of Wisconsin–Madison, Madison, WI, USA · NVIDIA, Santa Clara, CA, USA

  3. Online LLM Selection via Constrained Bandits with Time-Varying Demand

    Jun 16, 2026Yin Huang, Qingsong Liu, Jie XuMulti-Armed BanditsCost-Aware Inference

    Department of Electrical and Computer Engineering, University of Florida · Manning College of Information and Computer Sciences, University of Massachusetts Amherst