GPU Acceleration of Sparse Fully Homomorphic Encrypted DNNs

Lara D'Agata; Carlos Agulló-Domingo; Óscar Vera-López; Kaustubh Shivdikar; Ardhi W. B. Yudha; Ferhat Yaman; David Kaeli; José L. Abellán

cs.CRApr 13, 2026

GPU Acceleration of Sparse Fully Homomorphic Encrypted DNNs

Authors: Lara D'Agata, Carlos Agulló-Domingo, Óscar Vera-López, Kaustubh Shivdikar, Ardhi W. B. Yudha, Ferhat Yaman, David Kaeli, José L. Abellán,

Organizations: University of Glasgow, UK · University of Murcia, Spain · AMD · Northeastern University, USA

Abstract

Fully homomorphic encryption (FHE) has recently attracted significant attention as both a cryptographic primitive and a systems challenge. Given the latest advances in accelerated computing, FHE presents a promising opportunity for progress, with applications ranging from machine learning to information security. We target the most computationally intensive operation in deep neural networks from a hardware perspective, matrix multiplication (matmul), and adapt it for execution on AMD GPUs. We propose a new optimized method that improves the runtime and complexity of ciphertext matmul by using FIDESlib, a recent open-source FHE library designed specifically for GPUs. By exploiting sparsity in both operands, our sparse matmul implementation outperforms its CPU counterpart by up to $3.0\times$ and reduces the time complexity from cubic to semi-linear, demonstrating an improvement over existing FHE matmul implementations.

Figures & tables

Figure 1. Visualization of a request to a third-party cloud environment with and without FHE; with FHE, data remains encrypted throughout processing.

Figure 2. Example of a matrix represented in the CSR sparsity representation format.

Figure 3. Overview of SIMD operations used for the FHE CKKS matmul computation, with example value matrices A and B. The encrypted vectors are represented using a blue outline, the ellipses ( ... ) represent the ciphertext padding.

Panel of Figure 4 — Figure 4. Runtime comparison of the different algorithms used to multiply two 16x16 matrices on (a) an AMD Radeon RX 7900 GPU, (b) a comparison with OpenFHE on the AMD Radeon GPU, and (c) runtimes of each method on the AMD Radeon and AMD MI300 GPUs. The matrix sparsity is represented as a fraction, and the runtimes are normalized.

Sparsity	CSR/C			VCSR/C
Sparsity	Open	FIDES	Speed	Open	FIDES	Speed
	FHE	lib	up	FHE	lib	up
0%	101.234	38.266	2.65 $\times$	118.151	48.743	2.42 $\times$
10%	85.261	31.605	2.70 $\times$	101.569	40.641	2.50 $\times$
20%	67.349	25.025	2.69 $\times$	82.270	32.730	2.51 $\times$
30%	49.806	19.190	2.60 $\times$	61.147	26.275	2.33 $\times$

Table 1. Overall time (in seconds) and speedup comparison of CSR/C and VCSR/C using FIDESlib vs OpenFHE on the AMD Radeon GPU for 16x16 matrices across sparsity levels.

Sparsity	8x8 (Error e-10)		16x16 (Error e-10)
Sparsity	CSR/C	VCSR/C	CSR/C	VCSR/C
0%	33.54	38.02	985.58	1758.36
10%	28.47	23.43	894.49	526.77
20%	22.62	22.98	945.95	773.00
30%	22.41	15.44	591.93	528.83
40%	14.93	19.01	451.01	267.18

Table 2. Frobenius Norm of the CSR/C and VCSR/C methods comparing plaintext matmul with FIDESlib matmul across sparsity levels for matrix sizes 8x8 and 16x16.

Sparse Matrix Multiplication

See all 10 papers

FEnc $^2$ : Unifying Data Packing for Efficient Private Inference via Convolution and Architecture-Aware Fragment Encoding
Jun 15, 2026Ran Ran, Zhaoting Gong, Nuo Xu +3Efficient Neural Network Inference Homomorphic Encryption
North Carolina State University · University of Minnesota · University of California, Santa Cruz +1
Fiona: Accelerating FHE Inference with Packing-Aware Ternary Weights
Sep 28, 2026Yiteng Peng, Zhibo Liu, Dongwei Xiao +1Mixed-Precision Quantization Efficient Neural Network Inference
The Hong Kong University of Science and Technology · State Key Laboratory of Novel Software Technology, Nanjing University
HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation
Sep 1, 2026Alessandro Zirilli, Davide Marincione, Evgenios M. Kornaropoulos +2Privacy-Preserving Language Models Homomorphic Encryption
Sapienza University of Rome · George Mason University · Paradigma

Jun 15, 2026J/K move · Enter open · S save

FEnc $^2$ : Unifying Data Packing for Efficient Private Inference via Convolution and Architecture-Aware Fragment Encoding

Ran Ran, Zhaoting Gong, Nuo Xu, Yuanchao Xu+2

North Carolina State University · University of Minnesota · University of California, Santa Cruz+1

Efficient Neural Network Inference Homomorphic Encryption Privacy-Preserving ML

Open paper AskPDF arXiv

Fully Homomorphic Encryption (FHE) enables privacy-preserving machine learning but incurs extreme computational and memory overhead. These costs come not only from expensive low-level primitives, including Number Theoretic Transform (NTT), rotation, and key-switching, but also from inefficient ciphertext packing at the application level. Existing packing strategies typically preserve either neighboring data elements or feature grouping, but not both, leading to wasted ciphertext slots, excessive rotations, and inflated ciphertext counts. We propose FEnc2, a unified and principled fragment-based encoding framework for CKKS-based private convolutional neural network inference. FEnc2 optimizes slot utilization, rotation complexity, and ciphertext density through two components: 1)Conv-aware Encoding, which analytically selects an optimal fragment size to decouple spatial dependencies and jointly minimize inner-outer rotations across layers, and 2)Arch-aware Ct Compression, which restores ciphertext density after feature- or channel-reduction layers. Together, these transformations reshape encrypted workload structure and reduce homomorphic operations by one to two orders of magnitude. With full memory capacity utilized, i.e., at maximum batch size, FEnc2 achieves end-to-end latency speedups over the state-of-the-art Orion of up to 228.83x on GPU and 226.06x on CPU for LeNet on MNIST, and up to 4.55x on GPU and 9.43x on CPU for MobileNet on ImageNet. FEnc2 is hardware-agnostic yet architecturally transformative: by optimizing encrypted tensor layout before execution, it reduces ciphertext count and workload pressure on hardware, complementing primitive-level optimizations such as NTT and keyswitch accelerators. These results show that application-level data layout is a first-order architectural design dimension for encrypted inference and an important enabler for next-generation FHE systems.

Abstract

Sparse Matrix Multiplication

Explore similar work

FEnc2^22: Unifying Data Packing for Efficient Private Inference via Convolution and Architecture-Aware Fragment Encoding

Fiona: Accelerating FHE Inference with Packing-Aware Ternary Weights

HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation

FEnc $^2$ : Unifying Data Packing for Efficient Private Inference via Convolution and Architecture-Aware Fragment Encoding