OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Serving

Zikun Li; Yixuan Mei; Shiqi Pan; Zixuan Chen; Xiaowen Zhang; Mengdi Wu; Shuhuai Lin; Yutong Yang

cs.DCSep 13, 2026

OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Serving

Authors: Zikun Li, Yixuan Mei, Shiqi Pan, Zixuan Chen, Xiaowen Zhang, Mengdi Wu, Shuhuai Lin, Yutong Yang, +4 more

Organizations: Carnegie Mellon University · Peking University · Carnegie Mellon University and Google

Abstract

LLM serving systems increasingly disaggregate inference into finer-grained stages, with recent approaches separating attention from FFN or MoE execution during decode. This operator-level disaggregated serving (ODS) can improve hardware matching and enable independent scaling, particularly across heterogeneous devices. However, existing systems fix operator boundaries and lack a unified characterization of when disaggregation reduces serving cost. We present OpWeave, an end-to-end framework for heterogeneous ODS. OpWeave provides an analytical cost model that bounds the gains of homogeneous and heterogeneous ODS over colocated serving. It jointly optimizes operator partitioning and deployment configuration through a regularity-aware planner that keeps the search tractable even for hybrid-attention models. A vLLM-based runtime executes the synthesized plans with flexible operator stages across heterogeneous device groups. In our evaluation, OpWeave reduces serving cost by up to $1.78\times$ on homogeneous and $1.89\times$ on heterogeneous GPU clusters relative to the best feasible baseline, while meeting latency SLOs.

Figures & tables

Figure 1. Colocated serving versus heterogeneous operator-level disaggregated serving (ODS). AFD fixes stage boundaries at attention–FFN interfaces, whereas OpWeave supports general operator partitions across heterogeneous device groups.

Panel of Figure 2 — Figure 2. Decode-phase bottlenecks of Gemma-3-27B motivating ODS. (a) Tensor-core utilization of attention and FFN operators across batch sizes (B200): attention remains underutilized while batching improves FFN utilization. (b) Maximum feasible batch size across sequence lengths (H100): KV-cache growth constrains batching.

Figure 3. Pipelined attention–FFN disaggregation (AFD).

Figure 4. Theoretical cost gain of homogeneous ODS over colocated serving. Solid curves show gains for Qwen3-235B and GPT-OSS-120B on H100 under different attention-TP/MoE-EP configurations; dashed lines show upper bounds.

Figure 5. Heterogeneous vs. homogeneous ODS for Qwen3-235B on H100 and H20 GPUs. The heterogeneous setup runs core attention on H20 and GEMM-dominated operators on H100. Left: normalized cost per token. Right: cost gain over the best homogeneous deployment.

Figure 6. Feasibility of balanced CAD. Case 1: the attention and rest-of-model latency ranges overlap, admitting a balanced operating point. Case 2: the ranges are disjoint, so pipeline bubbles are unavoidable.

Figure 7. CAD pipeline schedule for a hybrid-attention model. A fixed rest-of-model stage cannot balance both efficient- and full-attention layers, leaving idle pipeline slots.

Figure 8. Distributed execution plane. Black dashed arrows carry activations between stages; the orange dashed path returns next-token IDs to the first-stage workers.

GPU type	Memory BW (GB/s)	Memory (GB)	BF16 (TFLOPS)	Price ($/GPU-h)
H100 SXM	3,350	80	989	3.49
L40S	864	48	362.05	1.09
A100 SXM	2,039	80	312	1.74

Table 1. GPU specifications ( NVIDIA, 2026b ; NVIDIA, 2026c ; NVIDIA, 2026a ) . Prices are averaged across selected public clouds.

Model	MoE	Eff. Attn.	Pattern	Strict SLO (ms)	Relaxed SLO (ms)
Gemma-3-27B	✗	SWA	5:1	25	60
Qwen3-Next-80B-A3B	✓	Linear	3:1	40	80

Table 2. Evaluated models and TPOT SLOs. Eff. Attn. is the efficient attention mechanism interleaved with full attention; Pattern is the ratio of efficient- to full-attention layers.

Figure 9. Measured serving cost across context lengths and TPOT SLOs. Crosses mark infeasible configurations. Labels above each group show

g\times

cheaper, where

g

is the best feasible baseline cost divided by OpWeave cost. Four framed grouped bar charts in one horizontal row show Gemma-3-27B under 25- and 60-millisecond TPOT SLO labels, followed by Qwen3-Next-80B-A3B under 40- and 80-millisecond labels. Each panel has 8K, 32K, and 128K context groups with four slots ordered as Colocation, AFD, CAD, and \sys{}. Bar heights retain measured USD per million output tokens, with an independent zero-based linear scale for each panel. Crosses mark the eight infeasible configurations. Small labels above each context group show the cheapest feasible Colocation, AFD, or CAD baseline cost divided by \sys{} cost; a ratio below one indicates higher \sys{} cost. Panel captions are below the charts and a shared legend is above.

Figure 10. Homogeneous-machine evaluation results. OpW denotes OpWeave . (a) and (d): Mean number of pipeline stages. (b) and (e): Mean inter-node stage-to-stage payload per output token (MiB). (c) and (f): Mean price-weighted compute occupancy (Section 7.2 ). Six panels in one horizontal row, with Gemma-3-27B in the first three and Qwen3-Next-80B-A3B in the last three. Panels a and d show average full-model pipeline stage counts for CAD, AFD, and \sys{}: 124, 124, and 20.67 for Gemma; 96, 96, and 24 for Qwen3-Next. Panels b and e show mean inter-node stage-to-stage payload in MiB per output token: 6.54, 8.84, and 0.39 for Gemma; 2.29, 2.16, and 0.35 for Qwen3-Next. Panels c and f show price-weighted compute occupancy: 49.71, 47.01, and 88.08 percent for Gemma; 51.45, 54.37, and 81.74 percent for Qwen3-Next.

Panel of Figure 11 — Figure 11. Mean per-GPU batch size at the relaxed TPOT SLO, normalized to OpWeave within each context and averaged across contexts. Higher is better. Two bar charts in one horizontal row show mean normalized per-GPU batch size at the relaxed TPOT SLO for Colocation, AFD, CAD, and \sys{}. The Gemma-3-27B values are 0.68, 0.38, 0.39, and 1.00. The Qwen3-Next-80B-A3B values are 0.36, 0.98, 0.98, and 1.00.

Figure 12. Simulated and measured TPOT for AFD, CAD, and OpWeave , normalized by measured TPOT. Labels show signed simulation errors. Two grouped bar charts in one horizontal row compare simulation and real-machine TPOT on the same zero-based normalized scale, with Gemma-3-27B on the left and Qwen3-Next-80B-A3B on the right, and one shared legend centered above the panels. Each pair is normalized by its real-machine TPOT, so all real-machine bars equal 1. For AFD, CAD, and \sys{} respectively, Gemma simulation bars are 0.935, 0.939, and 0.952; Qwen3-Next simulation bars are 0.990, 0.914, and 1.033. Labels above the groups show signed percentage differences: -6.5%, -6.1%, and -4.8% for Gemma, and -1.0%, -8.6%, and +3.3% for Qwen3-Next.

Figure 13. Simulated serving cost across context lengths and TPOT SLOs, normalized to heterogeneous OpWeave within each context. Labels show

g\times

cheaper, where

g

is the best feasible non- OpWeave baseline cost divided by heterogeneous OpWeave cost.

Figure 14. Network sensitivity of OpWeave , AFD and CAD. Two line plots compare AFD, CAD, and \sys{}. Latency is swept over 5, 20, 50, 200, and 500 microseconds at 32 GB/s; bandwidth is swept over 1, 2, 4, 8, 16, and 32 GB/s at 20 microseconds. \sys{} has the lowest reported cost at every comparable setting. AFD and CAD have no reported winner at 500 microseconds.

HW	Method	Candidates	Simulations	Time (s)
H100	Full	4.78M (1.00 $\times$ )	4.78M (1.00 $\times$ )	228.3 (1.00 $\times$ )
	No-PF	8.83M (1.85 $\times$ )	8.83M (1.85 $\times$ )	264.4 (1.16 $\times$ )
	No-B&B	219.79M (45.96 $\times$ )	4.78M (1.00 $\times$ )	290.3 (1.27 $\times$ )
	Neither	293.24M (61.31 $\times$ )	8.83M (1.85 $\times$ )	330.5 (1.45 $\times$ )
A100 +H100	Full	13.89M (1.00 $\times$ )	13.89M (1.00 $\times$ )	446.7 (1.00 $\times$ )
	No-PF	52.14M (3.75 $\times$ )	52.14M (3.75 $\times$ )	776.4 (1.74 $\times$ )

Table 3. Planner search ablation. Full enables Pareto-frontier (PF) pruning and branch-and-bound (B&B); No-PF , No-B&B , and Neither disable the corresponding optimizations. Parentheses show degradation relative to Full on the same hardware. Lower is better; all methods find the same best plan.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
Model and workload
$\mathbf{o}$	Operator sequence $(o_{1},\ldots,o_{N_{\mathrm{model}}})$ of one decode step
$N_{\mathrm{model}}$	Total number of operators
$L_{\mathrm{model}}$	Total number of transformer layers
$P_{\mathrm{act}}$	GEMM-side parameters activated per request after model-parallel sharding (routed experts only for MoE)
$M_{\mathrm{weights}}$	Resident model-weight footprint after sharding, including inactive experts

Appendix

Table 4. Unified notation used throughout the paper.

Explore similar work

Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity
Mar 13, 2026Donglin YuEfficient Multimodal Inference LLM Serving
Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs
May 5, 2026Yixuan Mei, Zikun Li, Zixuan Chen +5LLM Serving Cost-Aware Inference
Carnegie Mellon University · Peking University
Frontier: Towards Comprehensive and Accurate LLM Inference Simulation
May 20, 2026Yicheng Feng, Xin Tan, Yangtao Deng +3LLM Serving Disaggregated LLM Serving
The Chinese University of Hong Kong · Anuttacon · StepFun

Mar 13, 2026J/K move · Enter open · S save

Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity

Donglin Yu

Efficient Multimodal Inference LLM Serving Disaggregated LLM Serving LLM Inference Scheduling Multimodal Large Language Models

Open paper AskPDF arXiv

Multimodal large language model (MLLM) inference splits into two phases with opposing hardware demands: vision encoding is compute-bound, while language generation is memory-bandwidth-bound. We show that under standard transformer KV caching, the modality boundary (between vision encoder and language model) minimizes cross-device transfer among all partition points that preserve standard stage-based execution. Partitioning here reduces transfer complexity from

O(L * s_ctx)

bytes (GB-scale KV caches under stage-level disaggregation) to

O(N_v * d)

bytes (MB-scale embeddings), an O(L) reduction where L is the transformer depth. The result holds across attention mechanisms (MHA/GQA), dynamic vision resolutions, and model scales, and the advantage grows as models deepen. A direct implication is that existing stage-level disaggregation systems are constrained to high-bandwidth interconnects (e.g., NVLink), whereas modality-level disaggregation enables cross-tier heterogeneous serving over commodity PCIe. A closed-form cost model shows that heterogeneous deployment is cost-optimal under phase-separable workloads (predicts 31.4% savings; observed 40.6%). We build HeteroServe, a phase-aware runtime with modality-level partitioning and cross-tier scheduling, and evaluate it on LLaVA-1.5-7B and Qwen2.5-VL against vLLM v0.3.0. On identical 4xA100 hardware, engine optimizations raise throughput by up to 54%. Under a fixed budget, a heterogeneous cluster ($38k) improves Tokens/$ by 37% over a homogeneous baseline ($64k) without degrading latency.

Abstract

Explore similar work

Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity

Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs

Frontier: Towards Comprehensive and Accurate LLM Inference Simulation