Cerium: A Multi-GPU Framework for Terabyte-Scale Encrypted Inference

Siddharth Jayashankar; Joshua Kim; Michael B. Sullivan; Wenting Zheng; Dimitrios Skarlatos

cs.CRDec 12, 2025

Cerium: A Multi-GPU Framework for Terabyte-Scale Encrypted Inference

Authors: Siddharth Jayashankar, Joshua Kim, Michael B. Sullivan, Wenting Zheng, Dimitrios Skarlatos

Organizations: Carnegie Mellon University · UT Austin · NVIDIA

Abstract

Encrypted AI using fully homomorphic encryption (FHE) enables inference directly over encrypted queries, providing strong privacy guarantees. However, its computational and memory overheads have limited practical deployment. Custom FHE accelerators improve performance, but rely on advanced manufacturing technologies that limit their accessibility. GPUs offer a more widely available alternative, yet achieving ASIC-class performance on GPUs is challenging. Large models such as LLMs compound these challenges by requiring optimized kernels, terabyte-scale memory management, and efficient execution across multiple devices. We present Cerium, a multi-GPU framework for large-scale FHE inference. Cerium combines a domain-specific language, an optimizing compiler, and a runtime system to automatically generate optimized GPU kernels, compress plaintext data, manage terabyte-scale memory footprints, and optimize multi-GPU communication. Implemented on NVIDIA GPUs, Cerium outperforms expert-written GPU libraries by up to 2.25x and achieves performance competitive with state-of-the-art FHE ASICs, matching the CraterLake ASIC. It is the first GPU system to execute CKKS bootstrapping in under 10 milliseconds, achieving 7.5 milliseconds, and is the first to demonstrate encrypted inference for BERT-Base and Llama3-8B prefill in 8 seconds and 43 seconds, respectively. The Cerium framework is open-source and available at https://github.com/CMU-CAOS/Cerium

Figures & tables

Figure 1. Encoding and Encrypting Values in RNS-CKKS

Figure 2. FHE Memory Footprint vs Model Parameters

Figure 3. An Overview of the Cerium Framework.

Figure 4. A Code Sample with the Cerium DSL.

Figure 5. Limb IR Fusion and Code Generation

Figure 6. Speedup vs Vertical Fusion Threshold for a

128\times 4096

Ciphertext Matrix with a

4096\times 1024

Plaintext Matrix. Labels indicate aggregate kilobytes spilled per thread

Figure 8. Merge aggregate scatter and all reduce across output aggregation and input broadcast keyswitches

Figure 9. Showing how Cerium manages memory and creates reusable CudaGraphs for a transformer inference

Benchmark	GPU	Cerium Execution Time				Related	Unencrypted
Benchmark	GPU	$1\times$	$2\times$	$4\times$	$8\times$	Work	CPU	$1\times$ GPU
Bootstrap	B200	14.5 ms	9.9 ms	8.2 ms	7.5 ms	C: 21 ms	-	-
	H100	19.5 ms	13.7 ms	10.6 ms	9.7 ms	C: 23 ms
	A100	34.2 ms	22.5 ms	17.9 ms	16.5 ms	C: 40 ms
ResNet-20	B200	456 ms	340 ms	310 ms	298 ms	-	5 ms	2 ms
	H100	620 ms	464 ms	408 ms	386 ms	-		3 ms

Table 1. Benchmark Execution Times

Figure 10. Comparing Cerium to FHE ASICs 3 3 3 CraterLake and ARK results are omitted for BERT-Base because the published papers do not evaluate this benchmark and their implementations are not publicly available.

Benchmark	Cerium Compile Time	CudaGraph Creation Time
Bootstrap	5.5 s	350 ms
ResNet-20	13.9 s	1.75 s
BERT-Base	2 min 47 s	4.75 s
Llama3-8B	11 min 29 s	24.45 s

Table 2. Compilation and CudaGraph Creation Times

Panel of Figure 11 — Figure 11. Cerium Techniques: (a) Kernel Optimization; Memory Optimizations for (b) BERT-Base, (c) Llama3-8B

Figure 12. CDF for Bootstrapping Kernels

Panel of Figure 13 — Figure 13. (a) Cerium’s Multi-GPU Optimizations; (b) Llama3-8B Transformer Block Execution Timeline

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 15. Matrix Multiplication with Periodic Plaintexts

Homomorphic Encryption

Homomorphic encryption lets data be computed on while encrypted, supporting privacy-preserving machine learning, analytics, and inference.

See all 41 papers

AEGIS: Scaling Long-Sequence Homomorphic Encrypted Transformer Inference via Hybrid Parallelism on Multi-GPU Systems
Apr 3, 2026Zhaoting Gong, Ran Ran, Fan Yao +1Efficient Transformer Inference Homomorphic Encryption
North Carolina State University, Raleigh, NC, USA · University of Central Florida, Orlando, FL, USA
HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation
Sep 1, 2026Alessandro Zirilli, Davide Marincione, Evgenios M. Kornaropoulos +2Privacy-Preserving Language Models Homomorphic Encryption
Sapienza University of Rome · George Mason University · Paradigma
FEnc $^2$ : Unifying Data Packing for Efficient Private Inference via Convolution and Architecture-Aware Fragment Encoding
Jun 15, 2026Ran Ran, Zhaoting Gong, Nuo Xu +3Efficient Neural Network Inference Homomorphic Encryption
North Carolina State University · University of Minnesota · University of California, Santa Cruz +1

Apr 3, 2026J/K move · Enter open · S save

AEGIS: Scaling Long-Sequence Homomorphic Encrypted Transformer Inference via Hybrid Parallelism on Multi-GPU Systems

Zhaoting Gong, Ran Ran, Fan Yao, Wujie Wen

North Carolina State University, Raleigh, NC, USA · University of Central Florida, Orlando, FL, USA

Efficient Transformer Inference Homomorphic Encryption Privacy-Preserving ML GPU Acceleration Tensor Parallelism

Open paper AskPDF arXiv

Fully Homomorphic Encryption (FHE) enables privacy-preserving Transformer inference, but long-sequence encrypted Transformers quickly exceed single-GPU memory capacity because encoded weights are already large and encrypted activations grow rapidly with sequence length. Multi-GPU execution therefore becomes unavoidable, yet scaling remains challenging because communication is jointly induced by application-level aggregation and encryption-level RNS coupling. Existing approaches either synchronize between devices frequently or replicate encrypted tensors across devices, leading to excessive communication and latency. We present AEGIS, an Application-Encryption Guided Inference System for scalable long-sequence encrypted Transformer inference on multi-GPU platforms. AEGIS derives device placement from ciphertext dependencies jointly induced by Transformer dataflow and CKKS polynomial coupling, co-locating modulus-coherent and token-coherent data so that communication is introduced only when application dependencies require it, while reordering polynomial operators to overlap the remaining collectives with computation. On 2048-token inputs, AEGIS reduces inter-GPU communication by up to 57.9% in feed-forward networks and 81.3% in self-attention versus prior state-of-the-art designs. On four GPUs, it achieves up to 96.62% scaling efficiency, 3.86x end-to-end speedup, and 69.1% per-device memory reduction. These results establish coordinated application-encryption parallelism as a practical foundation for scalable homomorphic Transformer inference.

Abstract

Homomorphic Encryption

Explore similar work

AEGIS: Scaling Long-Sequence Homomorphic Encrypted Transformer Inference via Hybrid Parallelism on Multi-GPU Systems

HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation

FEnc2^22: Unifying Data Packing for Efficient Private Inference via Convolution and Architecture-Aware Fragment Encoding

FEnc $^2$ : Unifying Data Packing for Efficient Private Inference via Convolution and Architecture-Aware Fragment Encoding