Pacifo: An Efficient Fully Homomorphic Encryption-based Lightweight ResNet for Trustworthy Machine Learning as a Service

Yang Li; Gang He; Ruichen Zhang; Guangyuan Liu; Yinqiu Liu

cs.CROct 24, 2026

Pacifo: An Efficient Fully Homomorphic Encryption-based Lightweight ResNet for Trustworthy Machine Learning as a Service

Authors: Yang Li, Gang He, Ruichen Zhang, Guangyuan Liu, Yinqiu Liu

Organizations: School of Communication and Information Engineering, Shanghai University, Shanghai, China · College of Computing and Data Science, Nanyang Technological University, Singapore

Abstract

Machine Learning as a Service (MLaaS) has become a popular paradigm for deploying AI applications. However, it raises significant privacy concerns that the raw client data needs to be uploaded to the server for inference services. Private inference using Fully Homomorphic Encryption (FHE) offers a solution by allowing computations directly on encrypted data, with only the final results decrypted on a client side. However, current practical implementations still suffer from large computational overhead and high latency due to the cost of FHE operations. In this paper, we propose Pacifo, a framework that adapts customized fine-tuning strategies to efficiently evaluate the ResNet20 model in the context of FHE. Specifically, Pacifo introduces a public-private layer partitioning scheme and an efficient ReLU approximation, significantly reducing the number of private layers and costly FHE operations during inference. It also offers flexible security configurations to accommodate varying client privacy requirements. Compared to state-of-the-art FHE implementations of ResNet20, Pacifo achieves a speedup of $3.7-21.7\times$ in runtime while effectively preserving $95\%-99\%$ of the original plaintext model accuracy across downstream tasks. Pacifo is open-sourced at https://github.com/BryceYangLi/Pacifo.

Figures & tables

Fig. 1 : MLaaS without privacy protection

Panel of Table 2 — TABLE I : Comparison of privacy-preserving schemes for private neural network inference.

Fig. 2 : Performance comparison between existing HE-based methods and ours on ResNet20 using the CIFAR-10 dataset.

Fig. 3 : The architecture of a ResNet20 model.

Fig. 4 : A toy example of our utilized vector encoding method for 2D convolutional computation with an input feature map

I

of size

3\times 3

and a kernel

K

of size

k=2

. Specifically, the input

I

needs to be packed and encrypted before computation.

Fig. 5 : Placement of bootstrappings in a block of ResNet20.

Fig. 6 : Leverage layer freezing techniques on an open-source pretrained ResNet20 model.

Security	Personal computer	Supercomputer	Quantum computer
80-bit	$1000\text{\,}\mathrm{y}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{s}$	$14\text{\,}\mathrm{d}\mathrm{a}\mathrm{y}\mathrm{s}$	$1\text{\,}\mathrm{s}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{d}\mathrm{s}$
96-bit	$1\text{\times}{10}^{6}\text{\,}\mathrm{y}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{s}$	$2500\text{\,}\mathrm{y}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{s}$	$10\text{\,}\mathrm{s}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{d}\mathrm{s}$
128-bit	$1\text{\times}{10}^{20}\text{\,}\mathrm{y}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{s}$	$1\text{\times}{10}^{13}\text{\,}\mathrm{y}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{s}$	$16\text{\,}\mathrm{h}\mathrm{o}\mathrm{u}\mathrm{r}\mathrm{s}$
192-bit	$1\text{\times}{10}^{36}\text{\,}\mathrm{y}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{s}$	$1\text{\times}{10}^{25}\text{\,}\mathrm{y}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{s}$	$10\text{\,}\mathrm{y}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{s}$
256-bit	$1\text{\times}{10}^{53}\text{\,}\mathrm{y}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{s}$	$1\text{\times}{10}^{40}\text{\,}\mathrm{y}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{s}$	$1\text{\times}{10}^{6}\text{\,}\mathrm{y}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{s}$

TABLE II : Cracking costs for different bit-security levels using various computing systems.

Scheme	Ring ( $N$ )	Moduli chain $log_{2}(PQ)$	Noise distribution ( $\chi_{e}$ )	Security level ( $\lambda$ )
$1$	$2^{16}$	$1228$ bits	$\mathcal{N}_{\mathbb{Z}}(0,\,3.19^{2})$	$128$ -bit
$2$	$2^{16}$	$1188$ bits	$\mathcal{N}_{\mathbb{Z}}(0,\,3.19^{2})$	$192$ -bit
$3$	$2^{17}$	$1289$ bits	$\mathcal{N}_{\mathbb{Z}}(0,\,3.19^{2})$	$256$ -bit

TABLE III : Key parameters for our RNS-CKKS schemes to achieve different HE security levels.

System	Method	Interval	Degree	Depth
Lee et al. [ 25 ]	Remez. [ 33 ]	[-1,1]	50	12
Lee et al. [ 26 ]	Remez. [ 33 ]	[-1,1]	50	12
Kim et al. [ 28 ]	Lee. [ 59 ]	[-1,1]	28	10
Rovida et al. [ 29 ]	Chebyshev. [ 60 ]	[-1,1]	59	5
Ours	Remez. [ 48 ]	[-1,1]	2	2

TABLE IV : Comparison of the computational efficiency for different ReLU approximations in FHE.

Dataset	Model	Acc. (ReLU)	Acc. (Approx)	Loss $\downarrow$
MNIST	$\textnormal{ResNet20}_{\textnormal{frozen-6}}$	99.59%	99.60%	-0.01%
MNIST	$\textnormal{ResNet20}_{\textnormal{frozen-8}}$	99.40%	99.39%	0.01%
CIFAR-10	$\textnormal{ResNet20}_{\textnormal{frozen-6}}$	92.31%	92.58%	-0.27%
CIFAR-10	$\textnormal{ResNet20}_{\textnormal{frozen-8}}$	88.11%	88.14%	-0.03%

TABLE V : Comparison of model accuracy with ReLU and approximated ReLU activations on different datasets.

Framework	Scheme	#Private Params	Processor	#Threads	Runtime (s)	Accuracy
Lee et al. [ 25 ]	CKKS	298.8K	Xeon Platinum 8280 CPU	64	10602	90.67%
Lee et al. [ 26 ]	CKKS	278.3K	Threadripper PRO [email protected]	1	2271	91.31%
Benamira et al. [ 27 ]	TFHE	278.3K	[email protected]	4	570	74.10%
Kim et al. [ 28 ]	CKKS	278.3K	EYPC [email protected]	4	368	90.32%
Rovida et al. [ 29 ]	CKKS	278.3K	Xeon Gold 5218R [email protected]	8	426.5	91.67%
$\textnormal{Pacifo}_{\{\textnormal{L}\}}$ (Ours)	CKKS	206.3K	Xeon Gold 5218R [email protected]	8	135.1	92.58% †

TABLE VI : Comparison of the inference performance for HE-based ResNet20 on CIFAR-10.

Operation	Available Depth	Time (s)	%
Conv+BatchNorm	2 $\rightarrow$ 1	4.60	24.86%
Bootstrapping	1 $\rightarrow$ 4	4.11	22.21%
ReLU	4 $\rightarrow$ 2	0.23	0.01%
Conv+BatchNorm	2 $\rightarrow$ 1	4.79	25.89%
Residual Connection	1 $\rightarrow$ 1	0.12	0.01%
Bootstrapping	1 $\rightarrow$ 4	3.86	20.86%

TABLE VII : End-to-end runtime breakdown of a basic block in layer 3 of

\textnormal{Pacifo}_{\{\textnormal{L}\}}

Fig. 7 : Flattened output size of different block in ResNet20 and the corresponding FHE ciphertext size (MB).

System	Dataset	Plaintext	Ciphertext	Loss $\downarrow$
$\textnormal{Pacifo}_{\{\textnormal{L}\}}$	MNIST	99.65%	99.63%	0.02%
$\textnormal{Pacifo}_{\{\textnormal{L}\}}$	CIFAR10	92.58%	92.58%	0.00%
$\textnormal{Pacifo}_{\{\textnormal{B}\}}$	MNIST	99.39%	99.40%	-0.01%
	CIFAR10	88.14%	88.19%	-0.05%

TABLE VIII : Inference accuracy on different datasets.

Fig. 8 : Comparision of relative precision of Pacifo with plaintext model.

Security level	Ring size	Moduli chain size	Runtime (s)
$128$ -bit	$2^{16}$	$1228$ bits	$135.1$
$192$ -bit	$2^{16}$	$1188$ bits	$149.7$
$256$ -bit	$2^{17}$	$1289$ bits	$256.5$

TABLE IX : Runtime performance of

\textnormal{Pacifo}_{\{\textnormal{L}\}}

under different security levels.

Privacy-Preserving ML

Privacy-preserving machine learning develops methods to learn, share, and run models while limiting exposure of sensitive data.

See all 433 papers

Practical and Private Hybrid ML Inference with Fully Homomorphic Encryption
Sep 1, 2025Sayan Biswas, Philippe Chartier, Akash Dhasade +7Privacy-Preserving ML Homomorphic Encryption
EPFL · Inria · IRMAR +3
Towards Deep Encrypted Training: Low-Latency, Memory-Efficient, and High-Throughput Inference for Privacy-Preserving Neural Networks
Apr 18, 2026Nges Brian Njungle, Eric Jahns, Michel A. KinsyEfficient Neural Network Inference Memory-Efficient Inference
STAM Center, Ira A. Fulton Schools of Engineering · Arizona State University, Tempe, AZ 85281, USA
FEnc $^2$ : Unifying Data Packing for Efficient Private Inference via Convolution and Architecture-Aware Fragment Encoding
Jun 15, 2026Ran Ran, Zhaoting Gong, Nuo Xu +3Efficient Neural Network Inference Homomorphic Encryption
North Carolina State University · University of Minnesota · University of California, Santa Cruz +1

Sep 1, 2025J/K move · Enter open · S save

Practical and Private Hybrid ML Inference with Fully Homomorphic Encryption

Sayan Biswas, Philippe Chartier, Akash Dhasade, Tom Jurien+6

EPFL · Inria · IRMAR+3

Privacy-Preserving ML Homomorphic Encryption Efficient Neural Network Inference

Open paper AskPDF arXiv

In contemporary cloud-based services, protecting users' sensitive data and ensuring the confidentiality of the server's model are critical. Fully homomorphic encryption (FHE) enables inference directly on encrypted inputs, but its practicality is hindered by expensive bootstrapping and inefficient approximations of non-linear activations. We introduce Safhire, a hybrid inference framework that executes linear layers under encryption on the server while offloading non- linearities to the client in plaintext. This design eliminates bootstrapping, supports exact activations, and significantly reduces computation cost. To safeguard model confidentiality despite client access to intermediate outputs, Safhire applies randomized shuffling, which obfuscates intermediate values and makes it practically impossible for a client to reconstruct the model. To further reduce inference latency, Safhire incorporates advanced optimizations such as fast ciphertext packing and partial extraction. Evaluations on multiple standard models and datasets show that Safhire achieves up to 7.6x and 7.4x lower inference latency than Orion and Concrete-ML, respectively, which are two state-of-the-art FHE baselines, with manageable communication overhead and comparable accuracy. These results underline the practicality of our hybrid FHE inference scheme.

Abstract

Privacy-Preserving ML

Explore similar work

Practical and Private Hybrid ML Inference with Fully Homomorphic Encryption

Towards Deep Encrypted Training: Low-Latency, Memory-Efficient, and High-Throughput Inference for Privacy-Preserving Neural Networks

FEnc2^22: Unifying Data Packing for Efficient Private Inference via Convolution and Architecture-Aware Fragment Encoding

FEnc $^2$ : Unifying Data Packing for Efficient Private Inference via Convolution and Architecture-Aware Fragment Encoding