Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity

Donglin Yu

cs.LGMar 13, 2026

Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity

Authors: Donglin Yu

Abstract

Multimodal large language model (MLLM) inference splits into two phases with opposing hardware demands: vision encoding is compute-bound, while language generation is memory-bandwidth-bound. We show that under standard transformer KV caching, the modality boundary (between vision encoder and language model) minimizes cross-device transfer among all partition points that preserve standard stage-based execution. Partitioning here reduces transfer complexity from $O(L * s_ctx)$ bytes (GB-scale KV caches under stage-level disaggregation) to $O(N_v * d)$ bytes (MB-scale embeddings), an O(L) reduction where L is the transformer depth. The result holds across attention mechanisms (MHA/GQA), dynamic vision resolutions, and model scales, and the advantage grows as models deepen. A direct implication is that existing stage-level disaggregation systems are constrained to high-bandwidth interconnects (e.g., NVLink), whereas modality-level disaggregation enables cross-tier heterogeneous serving over commodity PCIe. A closed-form cost model shows that heterogeneous deployment is cost-optimal under phase-separable workloads (predicts 31.4% savings; observed 40.6%). We build HeteroServe, a phase-aware runtime with modality-level partitioning and cross-tier scheduling, and evaluate it on LLaVA-1.5-7B and Qwen2.5-VL against vLLM v0.3.0. On identical 4xA100 hardware, engine optimizations raise throughput by up to 54%. Under a fixed budget, a heterogeneous cluster ($38k) improves Tokens/$ by 37% over a homogeneous baseline ($64k) without degrading latency.

Figures & tables

System	Partition	Transfer	Deploy	Cost	Transfer
	bound.	size	regime	model	analysis
EPD	Stage	GB (KV)	NVLink	✗	✗
ModServe	Modality	MB (emb)	NVLink	✗	✗
SpaceServe	Co-loc.	—	Intra	✗	✗
Cauchy	Stage	GB (KV)	NVLink	Heuristic	✗
Ours	Modal.	MB	PCIe	Formal	✓

Table 1: Comparison of disaggregated MLLM serving systems. Our work is the only system combining modality-level partitioning with cross-tier support and formal cost modeling.

Partition Point	Data Transferred	Size
Stage-level (PD)	KV cache	${\sim}350$ MB
Stage-level (EPD)	KV cache + tokens	${\sim}350$ MB
Modality-level (Ours)	Visual embeddings	$\mathbf{\sim 4.5}$ MB

Table 2: Transfer size per request under different disaggregation granularities for a 7B model with 576 vision tokens and 128 output tokens.

Model	$N_{v}$	$L$	$\frac{n_{\text{kv}}}{n_{h}}$	$D_{\text{KV}}$	$D_{\text{emb}}$	$R_{\text{MHA}}$	$R_{\text{GQA}}$
LLaVA-7B	576	32	32/32	352 M	4.5 M	78 $\times$	78 $\times$
LLaVA-13B	576	40	40/40	550 M	5.6 M	98 $\times$	98 $\times$
LLaVA-34B	576	60	8/56	150 M	7.3 M	147 $\times$	21 $\times$
Qwen2.5-VL-7B	1024	28	4/28	83 M	7.0 M	64 $\times$	12 $\times$
Qwen-VL-72B	576	80	8/64	215 M	9.0 M	196 $\times$	24 $\times$

Table 3: Transfer ratio

R

across representative MLLM architectures (

s_{\text{text}}{=}128

). For fixed-resolution models,

N_{v}{=}576

; for dynamic-resolution models (Qwen2.5-VL),

N_{v}{=}1024

Figure 1: (a) Cost saving

\Delta_{\text{cost}}

(Eq. 8 ) as a function of the vision-to-language time ratio

\rho

for different price ratios

\gamma

. The RTX 4090/A100 operating point (

\gamma{=}0.19

\rho{=}0.63

) is marked. (b) Transfer ratio

R

(Eq. 2 ) across model depths, confirming that modality-level disaggregation becomes increasingly advantageous for larger models.

Figure 2: HeteroServe architecture. Consumer GPUs (RTX 4090) handle vision encoding and transfer lightweight visual embeddings (

{\sim}4.5

MB) via PCIe to datacenter GPUs (A100), which perform language generation. When the consumer pool is idle, cross-type work stealing allows consumer GPUs to assist with language decoding using pre-loaded LLM weights.

Figure 3: Timeline of consumer GPU activity. After completing vision encoding, the consumer GPU transfers embeddings and then steals language generation tasks until new vision requests arrive. Pre-loaded LLM weights enable sub-100 ms role switching.

System Config	Cost	tok/s	CER	vs. vLLM
vLLM (4 $\times$ A100)	$64k	3,886	60.7	Baseline
Ours (Homo)	$64k	5,986	93.5	+54%
Ours (Hetero)	$38k	3,156	83.1	+37% CER

Table 4: LLaVA-1.5-7B results. CER = Tokens/s per $1,000 investment.

vLLM Baselines
Configuration	GPUs	Cost	tok/s	CER
TP=2 (2 $\times$ A100)	2	$32k	2,768	0.78
TP=4 (4 $\times$ A100)	4	$64k	3,619	0.51
HeteroServe
Cfg1: 4090+A100 $\times$ 2	3	$35k	3,252	0.83
Cfg2: 4090 $\times$ 2+A100 $\times$ 2	4	$38k	3,269	0.78

Table 5: Qwen2.5-VL results with tensor parallelism. All language GPUs are A100-80GB.

Phase	Time (s)	%
Vision Encoding (RTX 4090)	6.82	38.2%
Feature Transfer (PCIe)	0.45	2.5%
Language Prefill (A100)	1.24	6.9%
Language Decode (A100)	9.35	52.4%

Table 6: Latency breakdown for LLaVA-1.5-7B heterogeneous config (batch=128).

Configuration	tok/s	$\Delta$ vs. vLLM
vLLM v0.3.0 baseline	3,886	—
+ CUDA Graph capture	4,980	+28%
+ Flash Attn Varlen packed	5,500	+42%
+ Aligned batch scheduling	5,986	+54%

Table 7: Engine optimization ablation on identical 4

\times

A100 (LLaVA-1.5-7B). Each row cumulatively adds one optimization.

Configuration	tok/s	Cost	$\Delta$ CER
Homogeneous 4 $\times$ A100	5,986	$64k	—
Hetero. 2 $\times$ 4090+2 $\times$ A100	3,156	$38k	+37%
+ Work stealing	3,156	$38k	+37%

Table 8: Architectural disaggregation ablation under fixed budget (LLaVA-1.5-7B). CER = cost-efficiency ratio (Tokens/$).

Explore similar work

Look Less, Think Faster: Joint Token-Compute Adaptation for Multimodal LLMs
Jul 22, 2026Pengcheng Wang, Zhiquan Wang, Jayoung Lee +5Efficient Multimodal Inference Efficient VLM Inference
Purdue University, West Lafayette, IN, USA · University of Wisconsin–Madison, Madison, WI, USA · NVIDIA, Santa Clara, CA, USA
OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Serving
Sep 13, 2026Zikun Li, Yixuan Mei, Shiqi Pan +9Disaggregated LLM Serving Cost-Aware Inference
Carnegie Mellon University · Peking University · Carnegie Mellon University and Google
EAServe: Encode-Aware Disaggregated Serving for Multimodal Large Language Models
Sep 25, 2026Kunxiong Zhu, Zhihao Shu, Hangyu Zheng +4Disaggregated LLM Serving LLM Inference Scheduling
University of Georgia Athens, GA, USA · Western Digital San Jose, CA, USA · University of Texas at Arlington Arlington, TX, USA

Jul 22, 2026J/K move · Enter open · S save

Look Less, Think Faster: Joint Token-Compute Adaptation for Multimodal LLMs

Pengcheng Wang, Zhiquan Wang, Jayoung Lee, Zhuoyan Xu+4

Purdue University, West Lafayette, IN, USA · University of Wisconsin–Madison, Madison, WI, USA · NVIDIA, Santa Clara, CA, USA

Efficient Multimodal Inference Efficient VLM Inference Multimodal Large Language Models Multimodal Foundation Models Visual Token Pruning Adaptive Computation

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have recently demonstrated strong performance across vision-language tasks. However, their high inference cost, arising from both the large number of input visual tokens and the heavy computation of the large language model (LLM), remains a key barrier to practical deployment. Recent work attempts to reduce the cost by adaptively optimizing individual dimensions, e.g., pruning redundant visual tokens or skipping LLM layers and heads. Nonetheless, prior approaches typically treat these dimensions independently and overlook a fundamental coupling: the available compute resources must be dynamically allocated across all dimensions based on the input content. To bridge the gap, we propose SmartVL, a unified adaptive inference framework that jointly controls vision token number and model compute capability in response to varying input contents and compute budgets. SmartVL introduces a vision-side token controller that dynamically selects informative visual tokens and an LLM-side compute controller that adaptively adjusts LLM computation. Importantly, these controllers are trained to coordinate with each other so that the overall inference cost satisfies a target budget. To allow this joint scheduling, we connect the controllers using a shared budget encoding and leverage a differentiable latency estimator for end-to-end training. This design enables SmartVL to learn cross-stage allocation strategies that adapt to both input complexity and runtime compute constraints. Experiments across multiple MLLM benchmarks demonstrate that, with joint scheduling, SmartVL consistently outperforms prior adaptive methods and achieves superior accuracy-efficiency Pareto frontiers. Project page: https://www.schaterji.io/publications/2026/jointtokencompute.

Abstract

Explore similar work

Look Less, Think Faster: Joint Token-Compute Adaptation for Multimodal LLMs

OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Serving

EAServe: Encode-Aware Disaggregated Serving for Multimodal Large Language Models