cs.LGAug 9, 2025

PrismServe: Efficient LLM Serving on Heterogeneous Accelerator Clusters

Authors: Jinhao Zhang, Zicheng Yan, Boyang Zhang, Zeyu Liu, Daning Cheng, Yunquan Zhang, Guangming Tan, Xiang Gao

Organizations: Beijing University of Posts and Telecommunications, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Peng Cheng Laboratory, Shenzhen, China · North University of China, Taiyuan, China · 

Abstract

Serving large language models across heterogeneous accelerators is challenging because partitioning a replica across device types introduces communication overhead during autoregressive decoding, while deploying complete quantized replicas can degrade model quality. We present PrismServe, a serving system that combines request-level collaboration across heterogeneous hardware with a mixture of quantized experts derived from the same foundation model. Each expert executes within a group of same-type accelerators in a single node, while a two-level router selects an expert based on predicted response quality and then selects a replica according to load and prefix-cache locality. Implemented on unmodified vLLM, PrismServe achieves 2.0--2.6×\times the throughput of cross-node or cross-vendor full-precision pipelined deployments and reduces P99 time per output token by 55--65% when serving Qwen3-14B on 48 accelerators from three vendors. Across Qwen3-8B, Qwen3-14B, and Qwen3-32B on five downstream tasks, PrismServe outperforms the best single quantization method by 2.0--4.3 points while remaining within 1.4 points of full-precision accuracy. Our code is available at https://anonymous.4open.science/r/LLM-Inference-System-B000/

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters

    Apr 7, 2025Zonghang Li, Tao Li, Wenjiao Feng +8On-Device Language Model InferenceLLM Inference

  2. A Few GPUs, A Whole Lotta Scale: Faithful LLM Training Emulation with PrismLLM

    May 15, 2026Shaoke Xi, ChonLam Lao, Boyi Jia +11High-Performance ComputingDistributed Training

  3. Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs

    May 5, 2026Yixuan Mei, Zikun Li, Zixuan Chen +5LLM ServingCost-Aware Inference