HyCoInfer: Cloud-Edge Collaborative Inference for Large Language Models on Heterogeneous GPU-FPGA Architectures
Organizations: Tsinghua University, China
Abstract
Large language model (LLM) serving relies predominantly on cloud GPUs, yet autoregressive decoding is memory-bound and often leaves their arithmetic units underutilized. Edge FPGAs offer attractive decoding efficiency, but are ill suited to sustaining a high-concurrency serving workload alone. This paper presents HyCoInfer, a heterogeneous cloud-edge serving system that assigns prefill to cloud GPUs and distributes decoding across cloud GPUs and edge FPGAs. The design addresses three barriers to practical offloading: capacity balancing across heterogeneous accelerators, the large KV cache crossing a constrained wide-area link, and the latency added by heterogeneous execution. HyCoInfer incorporates FPGA decode capacity into GPU prefill/decode role partitioning and places requests according to queue load, prompt length, and network state. It further combines layer-wise pipelined transfer, decoder-side precomputation, and bandwidth-adaptive KV cache compression based on residual vector quantization. We implement HyCoInfer using two NVIDIA V100 GPUs and an AMD Xilinx Alveo U280 and evaluate it over 100-1000 Mbps links. At 500 Mbps, HyCoInfer delivers up to 4.11 times the request goodput of cloud-only vLLM and reduces mean end-to-end latency by up to 61.1%. Across different network conditions and workloads, the system improves overall serving capacity, with the largest gains on communication-intensive tasks, and reduces per-request energy.
Figures & tables
| Component | Configuration |
|---|---|
| Cloud accelerators | 2–4 Tesla V100-PCIe-32GB |
| Edge accelerator | AMD Xilinx Alveo U280 |
| Cloud runtime | vLLM with extended KV Connector |
| Edge runtime | FlightLLM |
| Link rates | 100, 500, 1000 Mbps |
| Dataset | Input | Output | Rate | Conc. | TTFT SLO |
|---|---|---|---|---|---|
| (tokens) | (tokens) | (qps) | (ms) | ||
| IMDB | 100–200 | 1–5 | 1 | 8 | 200 |
| CNN/DailyMail | 800 | 50 | 0.5 | 4 | 2,000 |
| Qasper | 5K–16K | 10–100 | 0.1 | 4 | 10,000 |
| GovReport | 9K | 500 | 0.05 | 4 | 20,000 |
LLM Serving
Large language model serving studies how to deliver model responses with low latency, high throughput, and efficient use of compute and memory.
See all 247 papers