EAServe: Encode-Aware Disaggregated Serving for Multimodal Large Language Models

Kunxiong Zhu; Zhihao Shu; Hangyu Zheng; Minghai Qin; Miao Yin; Gagan Agrawal; Wei Niu

cs.DCSep 25, 2026

EAServe: Encode-Aware Disaggregated Serving for Multimodal Large Language Models

Authors: Kunxiong Zhu, Zhihao Shu, Hangyu Zheng, Minghai Qin, Miao Yin, Gagan Agrawal, Wei Niu

Organizations: University of Georgia Athens, GA, USA · Western Digital San Jose, CA, USA · University of Texas at Arlington Arlington, TX, USA

Abstract

Disaggregating the two stages, Prefill and Decode, onto separate GPU pools is now a standard optimization for (text-only) LLM serving. However, multimodal LLMs (MLLMs), which add a third phase, Encode, pose new challenges for resource allocation. Encode turns images, video, or audio into embeddings that the language model can consume, yielding a three-stage Encode-Prefill-Decode (EPD) pipeline. Existing frameworks offer only partial answers: text-only PD systems lack Encode, while EPD frameworks expose it as a separate service without regulating downstream request flow. The pipeline also carries a structural resource imbalance: every request enters through Encode before downstream work can begin, yet per-request execution leaves the encode GPU severely underutilized even at high loads, starving the downstream Prefill and Decode workers. Addressing this, we reposition Encode as the control point of the EPD pipeline, exposing three tightly coupled dimensions: when work enters downstream, where prefill executes, and how the GPU is shared. We instantiate this in EAServe across two co-designed layers. Its runtime manages load-adaptive micro-batching, rate-controlled partial offload to a co-resident prefill worker, and dynamic SM partitioning for predictable co-location. The configuration layer, Hybrid Auto Selection (HAS), navigates the joint space of GPU allocation, encode batch size, and offload ratio by pruning unbalanced allocations with per-stage capacity profiling and refining the remainder through TPE-based Bayesian optimization. Evaluated on three MLLM architectures spanning image, video, and audio, EAServe delivers up to 4.3x and 1.7x higher goodput than NVIDIA Dynamo and vLLM, respectively, under identical SLO constraints, sustains more balanced and higher GPU utilization across the EPD pipeline, and reaches near-optimal configurations faster than baseline search methods.

Figures & tables

Figure 1. The disaggregated pipeline of EPD. Block diagram of the three-stage EPD serving pipeline. Incoming requests enter an Encode Pool holding GPU 0 and GPU 1. Encoded embeddings are transferred over NIXL to a Prefill Pool holding GPU 2, GPU 3 and GPU 4. The resulting KV cache is transferred over NIXL to a Decode Pool holding GPU 5, GPU 6 and GPU 7, which produces the output. Each pool is a separately provisioned group of GPUs.

Figure 2. GPU computation and HBM-bandwidth utilization on LLaVA-v1.6-34B ( Liu et al., 2023 ) under NVIDIA Dynamo ( NVIDIA, 2025 ) . Two time-series plots covering a 50-second window on the encode GPU for LLaVA-v1.6-34B. Panel (a) plots compute utilization and panel (b) plots HBM bandwidth utilization, both on a vertical axis running from 0 to 50 percent. The baseline curve is nearly flat at roughly 5 percent in both panels. The batch-size-8 curve is spiky, peaking near 35 percent for compute and near 20 percent for bandwidth but falling back between spikes. Neither configuration comes close to saturating the encode GPU.

Configuration	Throughput	Mean TTFT	Mean TPOT
	(req/s)	(s)	(ms)
Dynamo default	0.75	367.4	179
+ micro-batch	1.09	112.1	411

Table 1. A partial gain from adding micro-batching to Dynamo on LLaVA-v1.6-34B .

Figure 3. SM partitioning via libsmctrl . (a) Default time-slicing: encode (orange) and local prefill (green) alternate, leaving idle SMs (gray). (b) TPC-mask partitioning runs them concurrently on disjoint SM subsets. Schematic comparing two ways of sharing streaming multiprocessors between encode and local prefill, with time running downward. In panel (a), time-slicing, an all-orange encode row alternates with a row that mixes green local-prefill blocks and gray idle blocks, so only one kind of work runs at any instant. In panel (b), SM partitioning, orange encode blocks and green local-prefill blocks occupy disjoint SMs within the same row and run concurrently, leaving no idle blocks.

Symbol	Meaning
$PC=(\mathrm{alloc},\,B,\,s)$	Deployment configuration searched by HAS, consisting of the stage allocation, maximum encode batch size, and offload ratio.
$\mathrm{alloc}$	GPU allocation across the Encode , Prefill , and Decode stages.
$B$	Maximum encode batch size.
$s$	Target fraction of requests routed to remote prefill.
$\Lambda$	Target system-wide arrival rate.
$T_{f},\;T_{p}$	SLO thresholds on P99 TTFT and P99 TPOT, respectively.

Table 2. A summary of key symbols.

Figure 4. Overview of EAServe . Architecture diagram of the proposed system with three grouped regions. A Hybrid Auto Selection box at the top runs Stage 1 profiling, then Stage 2 TPE search, then deployment, and emits the configuration triple of stage allocation, encode batch bound B and offload ratio s. The Encode Workers region on the left passes incoming requests through a micro-batcher governed by B, then a vision tower, then a deficit-counter router. The router splits traffic between a local path carrying fraction one minus s and an offload path carrying fraction s. The local path feeds a GPU whose SMs are split between encode and local prefill. The Downstream region on the right contains remote prefill feeding remote decode through the KV cache. Both paths converge on the outputs.

Figure 5. Four dispatch policies on the same Poisson arrival trace. (a) Fixed batch bound, no timeout. (b) Timeout too small. (c) Timeout too large. (d) Our Poisson-gap threshold $g=\kappa/\lambda$ . Four timelines of the same Poisson arrival trace of sixteen requests, one per dispatch policy, with hatching marking idle waiting and arrows marking dispatches. Row (a), a fixed batch bound of eight with no timeout, issues only two dispatches and leaves long idle stretches. Row (b), a timeout of two, issues nine dispatches, several carrying only one or two requests. Row (c), a timeout of ten, issues three large dispatches but makes the earliest requests wait a long time. Row (d), the proposed Poisson-gap threshold with g equal to two, issues four dispatches whose sizes track the observed arrival density, cutting idle waiting without fragmenting batches.

Figure 6. Deficit-counter routing with offload ratio

s{=}0.7

over twenty requests. On each arrival the router increments

d

s

(value above the request). When

d{\geq}1

(purple band), the request goes to the remote path and

d

is decremented by 1 (value below the band); otherwise the request stays on the local path and

d

carries over to the next arrival. Diagram of deficit-counter routing at offload ratio 0.7 over a stream of twenty numbered requests. The middle row lists requests in arrival order, with the accumulator value after adding 0.7 printed beneath each one. A purple wrap band marks every arrival whose accumulator reaches one and subtracts one from it. The top row highlights requests 1, 4, 7, 11, 14 and 17, which stay on the local path. The bottom row highlights the remaining requests, which take the remote path, yielding roughly seven remote dispatches for every three local ones.

Figure 7. Encode latency versus SM allocation on LLaVA-v1.6-34B at

B{=}4

. The dashed line marks the 20% latency threshold defining the minimum allocation preserving encode QoS. Line chart of encode latency in milliseconds against the share of streaming multiprocessors allocated to encode, measured on LLaVA-v1.6-34B at batch size four. Latency drops steeply from about 500 milliseconds at 10 percent to about 280 milliseconds at 30 percent, then flattens to roughly 210, 190 and 185 milliseconds at 50, 70 and 90 percent. A red dashed line near 280 milliseconds marks the 20 percent latency threshold, and the shaded band above it marks allocations that violate encode quality of service. An arrow labels 30 percent as the sweet spot.

Figure 8. Dynamic SM partitioning. Left : the offline table maps each SM level to the largest batch size. Right : at runtime, the split between encode (orange) and local prefill (green) shifts with batch size while staying on disjoint SM subsets. Two-part diagram of dynamic SM partitioning. The profiling table on the left maps each encode SM level of 10, 30, 50, 70 and 90 percent to the largest batch size it sustains, namely 2, 4, 6, 8 and 8. The runtime view on the right shows three snapshots of the SM array at successive times, each split between orange encode blocks and green local-prefill blocks. Dashed lookup arrows connect batch sizes of 3, 7 and 2 to their table rows, and the orange share of the array grows and shrinks as the batch size changes.

Model	Modal.	Encoder	LLM backbone	Params
LLaVA-v1.6-34B ( Liu et al., 2023 )	Image	CLIP ViT-L ( Radford et al., 2021 )	Yi-34B ( 01.AI et al., 2024 )	34B
Qwen2.5-VL-32B ( Bai et al., 2025 )	Video	ViT (dyn.)	Qwen2.5-32B	32B
Ultravox-v0.6-27B ( Fixie AI, 2024 )	Audio	Whisper ( Radford et al., 2023 )	Gemma-2-27B ( Gemma Team et al., 2024 )	27B

Table 3. Evaluated MLLM architectures.

Model	S1 (Strict)	S2 (Tight)	S3 (Moderate)	S4 (Standard)	S5 (Relaxed)
LLaVA	5/70	40/80	50/120	90/120	140/120
Qwen	5/60	10/70	40/110	80/110	110/110
Ultravox	2/50	3/60	5/80	10/100	30/130

Table 4. SLO thresholds at the P99 level (TTFT in s / TPOT in ms) across five tiers.

Figure 9. End-to-end goodput across five SLO tiers for image, video, and audio serving. The strictest tier (S1) characterizes the SLO frontier rather than an attainable operating point at the rates we sweep. Grouped bar chart of end-to-end goodput in requests per second across five SLO tiers labeled S1 through S5, with one panel per model: LLaVA-34B on image, Qwen2.5-VL-32B on video and Ultravox-27B on audio. Each tier shows four bars for Dynamo, vLLM, the proposed system at s equal to one, and the proposed system with HAS tuning. Both of the proposed configurations exceed both baselines in every tier where any system attains goodput, with the widest margin on audio, where goodput approaches 4 requests per second against roughly 1 for Dynamo. The strictest tier S1 is unattainable for most systems at the rates swept.

Figure 10. Throughput and latency comparison across different request rates for various modalities. Dynamo’s TTFT diverges past saturation because its per-request encode (

B{=}1

) cannot match the offered rate; the encode queue grows monotonically within the run and every request inherits the accumulated wait. Grid of fifteen line charts arranged in three rows, one per model, namely LLaVA-34B on image, Qwen-32B on video and Ultravox-27B on audio, and five columns showing throughput in tokens per second, mean and P99 time to first token in seconds, and mean and P99 time per output token in milliseconds. Every chart plots request rate from 1 to 5 requests per second on the horizontal axis, with four series for Dynamo, vLLM, the proposed system at s equal to one, and the proposed system with HAS tuning. The two proposed configurations hold the highest throughput and the flattest latency curves across all three modalities, while Dynamo's latency curves climb steeply once the offered rate exceeds what its per-request encode can sustain.

Figure 11. GPU utilization comparison against baseline. Two time-series plots over a 50-second window on a vertical axis running from 0 to 100 percent, extending the earlier baseline comparison with the proposed system. In panel (a), compute utilization for the proposed system oscillates between roughly 40 and 90 percent, the batch-size-8 curve spikes to about 20 percent, and the baseline stays near zero. In panel (b), bandwidth utilization for the proposed system holds around 15 to 20 percent while both baselines remain lower. The proposed system keeps the encode GPU busy for most of the run.

System	Tput	TTFT (s)		TPOT (ms)
	(req/s)	Mean	P99	Mean	P99
RTX 6000 Ada 48 GB (PCIe)
EPDServe	0.46	448	929	119	137
HydraInfer	0.58	315	664	402	477
EAServe	1.95	66	127	71	91
A100 SXM4 80 GB (NVLink)

Table 5. Throughput and latency comparison across two eight-GPU platforms on LLaVA-v1.6-34B at

\Lambda{=}5

req/s, using the best-performing configuration of each system.

Figure 12. Wall-clock time HAS, Random, and Coordinate each take to reach a given percentile of the best-observed throughput. The per-modality reference is the maximum across all 3 methods

\times

30 trials; missing bars indicate the percentile was not reached within the 30-trial budget. Grouped bar chart of the wall-clock hours each search method needs to reach a given percentile of the best observed throughput, with one panel per modality for image, video and audio. The horizontal axis lists the 70th, 80th, 90th, 95th and 99th percentiles and the vertical axis runs from 0 to 5 hours. Three series are plotted for Random, Coordinate and HAS. HAS reaches the higher percentiles in less wall-clock time than both baselines in most cells, and it reaches percentiles that the baselines never attain. Missing bars mark percentiles a method failed to reach within its budget of 30 trials.

SM config	Tput (r/s)	P99 TTFT (s)	P99 TPOT (ms)
No partition (time-slicing)	4.78	43.7	236
Static 50/50	5.07	40.7	203
Dynamic (ours)	5.21	36.7	167

Table 6. SM partitioning ablation on Ultravox-27B with local prefill enabled. All three configurations share allocation 2E(TP1)+1P(TP2)+1D(TP4),

B{=}10

s{=}0.55

, and

\Lambda{=}10

req/s.

Arrival	Tput	M TTFT	P TTFT	M TPOT	P TPOT
P.	(req/s)	(s)	(s)	(ms)	(ms)
CV ${=}0.5$	1.77	4.16	0 6.67	67.6	71.4
CV ${=}1$	1.77	4.98	10.19	67.9	71.5
CV ${=}2$	1.71	6.77	16.52	67.0	71.0

Table 7. Sensitivity to arrival burstiness on LLaVA-34B at

\Lambda{=}2

req/s. CV

{=}1

corresponds to Poisson arrivals. M = mean, P = P99.

Explore similar work

Mar 27, 2026

TCM-Serve: Modality-aware Scheduling for Multimodal Large Language Model Inference

Multimodal Large Language Models (MLLMs) power platforms like ChatGPT, Gemini, and Copilot, enabling richer interactions with text, images, and videos. These heterogeneous workloads introduce additional inference stages, such as vision preprocessing and encoding, that inflate latency and memory demand. Existing LLM serving systems, optimized for text-only workloads, fail under multimodality: large requests (e.g., videos) monopolize resources, causing severe head-of-line blocking and performance degradation. Our key insight is that multimodal requests differ by orders of magnitude in resource demands, which we capture through a simple abstraction: videos behave like trucks, images like cars, and text like motorcycles. We design TCM-Serve, a modality-aware scheduler that lets motorcycles flow quickly through cars and trucks, ensuring interactive responsiveness while avoiding starvation. TCM-Serve classifies requests, prioritizes them dynamically, and applies aging to avoid starvation. Evaluation across state-of-the-art MLLMs shows that TCM-Serve reduces, on average, time-to-first-token (TTFT) by 54% overall, and by 78.5% for latency-critical requests, compared to current systems. TCM-Serve delivers LLM-like responsiveness for MLLMs, with modality-aware scheduling and by making the most efficient use of the available resources.

Konstantinos Papaioannou, Thaleia Dimitra Doudali

IMDEA Software Institute, Universidad Politécnica de Madrid

LLM Serving LLM Inference Scheduling Efficient Multimodal Inference Multimodal Large Language Models

Ask PDF

Mar 13, 2026

Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity

Multimodal large language model (MLLM) inference splits into two phases with opposing hardware demands: vision encoding is compute-bound, while language generation is memory-bandwidth-bound. We show that under standard transformer KV caching, the modality boundary (between vision encoder and language model) minimizes cross-device transfer among all partition points that preserve standard stage-based execution. Partitioning here reduces transfer complexity from

O(L * s_ctx)

bytes (GB-scale KV caches under stage-level disaggregation) to

O(N_v * d)

bytes (MB-scale embeddings), an O(L) reduction where L is the transformer depth. The result holds across attention mechanisms (MHA/GQA), dynamic vision resolutions, and model scales, and the advantage grows as models deepen. A direct implication is that existing stage-level disaggregation systems are constrained to high-bandwidth interconnects (e.g., NVLink), whereas modality-level disaggregation enables cross-tier heterogeneous serving over commodity PCIe. A closed-form cost model shows that heterogeneous deployment is cost-optimal under phase-separable workloads (predicts 31.4% savings; observed 40.6%). We build HeteroServe, a phase-aware runtime with modality-level partitioning and cross-tier scheduling, and evaluate it on LLaVA-1.5-7B and Qwen2.5-VL against vLLM v0.3.0. On identical 4xA100 hardware, engine optimizations raise throughput by up to 54%. Under a fixed budget, a heterogeneous cluster ($38k) improves Tokens/$ by 37% over a homogeneous baseline ($64k) without degrading latency.

Donglin Yu

Efficient Multimodal Inference LLM Serving Disaggregated LLM Serving LLM Inference Scheduling Multimodal Large Language Models

Ask PDF

Mar 3, 2026

SUN: Shared Use of Next-token Prediction for Efficient Multi-LLM Disaggregated Serving

In multi-model LLM serving, decode execution remains inefficient due to model-specific resource partitioning: since cross-model batching is not possible, memory-bound decoding often suffers from severe GPU underutilization, especially under skewed workloads. We propose Shared Use of Next-token Prediction (SUN), the first approach that enables cross-model sharing of decode execution in disaggregated multi-LLM serving. SUN decomposes a decoder-only Transformer into a prefill module and a decode module, and fine-tunes only the task-specific prefill module, enabling a frozen decode module to be shared across models. This design enables a model-agnostic decode routing policy that balances decode requests across shared workers to maximize utilization. Across diverse tasks and model families, SUN achieves accuracy comparable to full fine-tuning while maintaining system throughput with fewer decode workers. In particular, SUN improves throughput per GPU by up to 2.0x over conventional disaggregation while keeping time-per-output-token (TPOT) within 5%. SUN inherently enables and facilitates low-bit decoding; with Quantized SUN (QSUN), it achieves a 45% speedup with comparable accuracy to SUN while preserving the benefits of shared decoding.

Sunghyeon Woo, Ahreum Seo, Jaegwang Lee +6

NAVER Cloud

Disaggregated LLM Serving LLM Inference Acceleration LLM Inference Scheduling

Ask PDF