Organizations: Beijing University of Posts and Telecommunications, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Peng Cheng Laboratory, Shenzhen, China · North University of China, Taiyuan, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
Serving large language models across heterogeneous accelerators is challenging because partitioning a replica across device types introduces communication overhead during autoregressive decoding, while deploying complete quantized replicas can degrade model quality. We present PrismServe, a serving system that combines request-level collaboration across heterogeneous hardware with a mixture of quantized experts derived from the same foundation model. Each expert executes within a group of same-type accelerators in a single node, while a two-level router selects an expert based on predicted response quality and then selects a replica according to load and prefix-cache locality. Implemented on unmodified vLLM, PrismServe achieves 2.0--2.6× the throughput of cross-node or cross-vendor full-precision pipelined deployments and reduces P99 time per output token by 55--65% when serving Qwen3-14B on 48 accelerators from three vendors. Across Qwen3-8B, Qwen3-14B, and Qwen3-32B on five downstream tasks, PrismServe outperforms the best single quantization method by 2.0--4.3 points while remaining within 1.4 points of full-precision accuracy. Our code is available at https://anonymous.4open.science/r/LLM-Inference-System-B000/
Figures & tables
Figure 1: Complementary reasoning errors under AWQ and GPTQ quantization. (a) AWQ correctly solves the age problem, while GPTQ reaches an incorrect answer due to a subtraction error. (b) GPTQ correctly computes the total fabric area, while AWQ overlooks the requirement for two pieces. These examples illustrate that neither method consistently preserves correctness across problems.
Figure 2: Conceptual illustration of input-dependent expert complementarity. The experts have different loss profiles across the input space. The dashed curve sketches an idealized routing objective, and the bottom labels indicate the preferred expert in each region.
Figure 3: Offline training of the PrismServe router. Routing labels are determined by the expert with the lowest prediction loss on each training sample. Quantized experts remain frozen, while the embedding encoder and MLP are jointly trained using cross-entropy loss.
Figure 4: Comparison of inference architectures. (a) Homogeneous FP16 inference partitions a model across nodes from the same hardware vendor. (b) Heterogeneous FP16 inference partitions a model across vendors, requiring cross-vendor activation transfers. (c) PrismServe deploys complete quantized experts across heterogeneous devices. The Gateway obtains an expert-selection decision from the quality router and forwards the request to the corresponding vLLM Router, which selects a replica for complete-model inference.
Method
Qwen3-8B
Qwen3-14B
Qwen3-32B
GSM
MM
IF
HE
MB
W2
C4
GSM
MM
IF
HE
MB
W2
C4
GSM
MM
IF
HE
MB
W2
C4
FP16
88.9
74.7
84.8
87.2
63.5
9.7
13.3
91.4
78.5
86.3
86.6
64.3
8.6
12.0
93.6
81.2
87.4
87.2
66.9
7.6
10.8
AWQ
83.7
67.8
77.3
79.9
56.3
12.4
16.3
84.9
71.8
79.3
79.3
57.1
10.8
14.6
87.3
74.8
80.2
80.5
60.1
9.4
13.0
GPTQ
84.3
68.9
78.2
81.1
57.4
11.5
15.4
86.1
73.0
80.6
80.5
58.5
10.0
13.7
88.7
76.0
81.5
81.7
61.4
8.7
12.2
FlatQuant
85.1
70.0
79.1
81.7
58.2
10.9
14.8
86.8
73.8
81.5
81.1
59.0
9.5
13.2
89.4
76.8
82.3
82.3
62.2
8.3
11.8
QuaRot
84.5
69.3
78.0
80.5
56.9
11.2
15.1
85.7
73.2
80.2
79.9
57.9
9.8
13.5
88.5
76.2
81.1
81.1
60.8
8.5
12.0
Table 1: Quality comparison across three Qwen3 model scales. PrismServe uses five experts. GSM: GSM8K; MM: MMLU; IF: IFEval; HE: HumanEval; MB: MBPP; W2: WikiText-2. Higher task scores and lower W2/C4 perplexities are better.
Figure 5: Effect of expert pool size on six benchmarks across three Qwen3 model scales. Starting with GPTQ and AWQ, we sequentially add FlatQuant, QuaRot, and SmoothQuant to form pools of three, four, and five experts.
Deployment
Hardware
TP/PP
C=1
C=2
C=4
C=8
Thr. ↑
TTFT ↓
TPOT ↓
Thr. ↑
TTFT ↓
TPOT ↓
Thr. ↑
TTFT ↓
TPOT ↓
Thr. ↑
TTFT ↓
TPOT ↓
Single-node FP16
Ascend 910C
2/1
28.64
56.38
36.97
51.32
109.51
40.92
101.34
164.68
41.14
189.34
267.77
43.16
MTT S5000
2/1
39.73
84.01
26.65
66.55
124.04
31.69
118.88
177.59
35.31
222.94
314.85
36.13
BI-V150
2/1
32.11
99.76
33.02
58.77
231.32
34.70
110.18
369.43
36.18
205.13
635.11
35.96
Same-vendor cross-node
Ascend 910C
2/2
18.12
178.57
62.23
32.76
301.35
68.69
60.32
424.66
73.94
118.58
526.38
75.11
MTT S5000
2/2
27.98
183.71
39.59
52.50
338.06
41.10
95.89
548.37
43.64
168.58
961.96
47.80
Table 2: 8 GPUs deployment comparison under fixed64. TP/PP denotes tensor and pipeline parallelism per replica. Single-node FP16 uses four replicas; cross-node baselines use two. PrismServe uses four quantized experts with one replica each. C denotes total concurrency. Thr. is output throughput in token/s; TTFT and TPOT are P99 latencies in ms and ms/token.
Deployment
Thr. ↑
TTFT ↓
TPOT ↓
fixed64
FP16 same-vendor
606.79
2053.71
77.43
FP16 cross-vendor
492.47
3235.74
99.96
FP16 same-node
1102.70
1364.97
52.76
Quantized replicas
1255.56
995.09
33.28
PrismServe
1231.80
1042.28
34.58
Table 3: Performance with 48 GPUs across six nodes (two per vendor) at concurrency 48. The quantized-replica baseline serves FlatQuant models, with each replica executing entirely within one node. Throughput: token/s; P99 TTFT: ms; P99 TPOT: ms/token.
Arrival rate (req/s)
Goodput ↑ (req/s)
PrismServe
Same-vendor
Cross-vendor
18
18.00
4.29
1.02
24
18.80
4.17
0.87
32
17.37
4.09
0.95
48
16.59
3.96
0.96
64
16.44
3.86
1.00
Table 4: Goodput on 48 accelerators under the mixed workload at different request arrival rates. The TPOT SLO threshold is 100 ms/token.
Figure 6: Scalability under the mixed workload. (a) PrismServe’s load response at different accelerator counts. (b) Goodput of the three deployments as the accelerator count increases, with the arrival rate fixed at 32 requests/s. The TPOT SLO threshold is 100 ms/token.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Hardware
GPUs per node
Memory
Ascend 910C
8
128 GB
MTT S5000
8
80 GB
BI-V150
8
32 GiB
Appendix
Table 5: Hardware configurations. Memory is reported per board for Ascend 910C and MTT S5000, and per runtime-visible logical device for BI-V150.
Node
PyTorch
Extension/runtime
vLLM
ascend910c-01
2.7.1
torch-npu 2.7.1
0.11.0
s5000-01
2.5.0
torch-musa 2.5.0
0.9.3.dev0
iluvatar150-01
2.10.0+CoreX 4.5
CoreX 4.5
0.23.0
Appendix
Table 6: Software environments of representative nodes.
Total GPUs
Deployment
TP
PP
GPUs/replica
Replicas
Placement
8
Single-node FP16
2
1
2
4
Four replicas within one node
8
Same-vendor cross-node FP16
2
2
4
2
Each replica spans two same-vendor nodes
8
Cross-vendor cross-node FP16
2
2
4
2
Each replica spans nodes from different vendors
8
PrismServe
2
1
2
4
Four experts, one replica per expert
48
Same-vendor cross-node FP16
2
2
4
12
Four replicas per vendor group
48
Cross-vendor cross-node FP16
2
2
4
12
Pipeline stages placed across vendors
Appendix
Table 7: Deployment configurations for the eight-GPU and forty-eight-GPU experiments. TP and PP are specified per replica.
Total
Ascend
Moore
Iluvatar
Nodes
Replicas
48
16
16
16
6
24
64
24
16
24
8
32
80
32
16
32
10
40
88
40
16
32
11
44
Appendix
Table 8: Scaling configurations. Ascend, Moore, and Iluvatar denote the numbers of Ascend 910C, MTT S5000, and BI-V150 accelerators, respectively. Replicas denote the total number of PrismServe expert replicas.
Quantized expert
Configuration
AWQ
W4A16
GPTQ
W4A16
FlatQuant
W4A16
QuaRot
W4A16
SmoothQuant
W8A8
Calibration parameter
Setting
Appendix
Table 9: Expert precision configurations and shared calibration settings. W and A denote weight and activation bit widths, respectively.
Category
Proportion
Data sources
Problem-solving
80%
GSM8K, MATH, and MBPP training sets; MMLU auxiliary training set
Conversation
10%
ShareGPT90K
Document
10%
WikiText-2 and C4 training splits
Appendix
Table 10: Sources and composition of the router training data. Proportions are measured by sample count.
Parameter
Setting
Input encoder
Qwen3-Embedding-0.6B
Training strategy
Joint encoder and MLP optimization
Quantized expert parameters
Frozen
Pooling
Last non-padding token
Semantic vector normalization
L2 normalization
Semantic vector dimension
1024
Appendix
Table 11: Router architecture and training hyperparameters.
Benchmark
Few-shot
Metric
GSM8K
8-shot CoT
Accuracy
MMLU
5-shot
Accuracy
IFEval
0-shot
Instruction-following rate
HumanEval
0-shot
pass@1
MBPP
3-shot
pass@1
WikiText-2
–
Token-level PPL
Appendix
Table 12: Evaluation settings and metrics. CoT denotes chain-of-thought prompting.
Figure 7: Expert selection shares under the mixed synthetic workload. Each model group contains PrismServe configurations with two, three, and four experts. Each bar sums to 100%, and segment labels indicate the percentage of requests assigned to each expert.
Method
GSM8K ↑
MMLU ↑
IFEval ↑
HumanEval ↑
MBPP ↑
FP16
91.4
78.5
86.3
86.6
64.3
AWQ
84.9
71.8
79.3
79.3
57.1
GPTQ
86.1
73.0
80.6
80.5
58.5
FlatQuant
86.8
73.8
81.5
81.1
59.0
QuaRot
85.7
73.2
80.2
79.9
57.9
SmoothQuant
88.3
75.8
80.2
82.9
61.1
Appendix
Table 13: Quality comparison of the full-precision model, individual quantized experts, and routing strategies on Qwen3-14B. All three routing strategies use the same four-expert pool: AWQ, GPTQ, FlatQuant, and QuaRot. SmoothQuant is excluded from routing and serves only as an additional fixed-expert reference. All scores are percentages; higher is better.
Metric
Latency (ms)
Mean latency
47.19
P95 latency
52.90
P99 latency
55.94
Appendix
Table 14: Routing service latency, including queueing, gateway forwarding, and router computation.