Practical and Private Hybrid ML Inference with Fully Homomorphic Encryption

Sayan Biswas; Philippe Chartier; Akash Dhasade; Tom Jurien; David Kerriou; Anne-Marie Kerrmarec; Mohammed Lemou; Franklin Tranie

cs.CRSep 1, 2025

Practical and Private Hybrid ML Inference with Fully Homomorphic Encryption

Authors: Sayan Biswas, Philippe Chartier, Akash Dhasade, Tom Jurien, David Kerriou, Anne-Marie Kerrmarec, Mohammed Lemou, Franklin Tranie,

Organizations: EPFL · Inria · IRMAR · Université de Rennes ·

Abstract

In contemporary cloud-based services, protecting users' sensitive data and ensuring the confidentiality of the server's model are critical. Fully homomorphic encryption (FHE) enables inference directly on encrypted inputs, but its practicality is hindered by expensive bootstrapping and inefficient approximations of non-linear activations. We introduce Safhire, a hybrid inference framework that executes linear layers under encryption on the server while offloading non- linearities to the client in plaintext. This design eliminates bootstrapping, supports exact activations, and significantly reduces computation cost. To safeguard model confidentiality despite client access to intermediate outputs, Safhire applies randomized shuffling, which obfuscates intermediate values and makes it practically impossible for a client to reconstruct the model. To further reduce inference latency, Safhire incorporates advanced optimizations such as fast ciphertext packing and partial extraction. Evaluations on multiple standard models and datasets show that Safhire achieves up to 7.6x and 7.4x lower inference latency than Orion and Concrete-ML, respectively, which are two state-of-the-art FHE baselines, with manageable communication overhead and comparable accuracy. These results underline the practicality of our hybrid FHE inference scheme.

Figures & tables

Figure 1: Latency breakdown for bootstrapping and other operations during a single sample inference request using Orion , a state-of-the-art FHE inference framework. We evaluate three models and datasets and show the E2E inference duration in minutes on the right.

Figure 2: Existing FHE -based ML inference (top) performs all computation on the server in the encrypted domain; our hybrid scheme (bottom) offloads computationally expensive operations to the client to significantly improve efficiency.

Primitive	Why Safhire needs it	Where it is used in Safhire ’s workflow ( Figure 3 )
LWE ( LWE ) and RLWE encryption ( Section 2.1 )	Hides the client’s data from the server. An RLWE ciphertext carries up to $N$ values, which keeps communication low. Since the client decrypts after every layer, the noise is reset without bootstrapping.	Key generation; encryption, steps 2-4; decryption, step 12.
Homomorphic linear operations ( Section 2.1 )	Let the server evaluate linear layers on ciphertexts, provided that the weights are integers, hence the quantized model.	Partial extraction, step 6, and linear layer, step 8.
Automorphisms and key switching ( Section 2.2 )	Needed to evaluate traces homomorphically: an automorphism changes the key under which a ciphertext decrypts, and key switching restores it.	Keys generated at setup; fast packing, step 10; avoided in step 6 thanks to the powers computed by the client in step 3.
Trace ( Section 2.3 )	Isolates one coefficient of an encrypted polynomial. Computed level by level along a tower of subfields, which reduces its cost.	Power computation, step 3; partial trace extraction, step 6; fast packing, step 10.
Packing ( Section 2.4 )	Gathers the outputs of a layer, each carried by its own ciphertext, into a few RLWE ciphertexts before they are returned.	Fast packing, step 10, and transfer, step 11.

Table 1: FHE primitives used by Safhire , why they are needed, and where they appear in the workflow of Figure 3 .

Figure 3: The workflow of the Safhire during an inference request using a CNN model. The box on the left and right shows the operations performed on the client and server, respectively.

Key size	Trace extraction level ( $\gamma$ )
$M=t^{\alpha}$	$\gamma=0$ (Base)	$\gamma=1$	$\gamma=2$
$3^{7}$	2.50	1.50	0.50
$5^{5}$	3.25	1.25	0.25
$7^{4}$	4.16	1.16	0.16

Table 2: Key switches per output coefficient for different start partial trace extraction level

(\gamma)

Approach	FHE scheme	Linear layers	Non-linear layers	Bootstrapping required	Client-server rounds	Offline phase	What the client learns	Client threat model
Full encryption	CKKS / BFV	Plaintext–ciphertext products with slot rotations	Polynomial approximations, server-side	Yes	1	No	Final output only	Malicious a
Full encryption (TFHE)	TFHE	Scalar MACs with plaintext weights	Exact, via programmable bootstrapping (lookup tables)	Yes	1	No	Final output only	Malicious a
HE + 2PC	HE + authenticated secret sharing	HE on authenticated shares	Garbled circuits with consistency checks	No	$O(L)$	Yes	Final output only	Malicious a
Client-aided FHE	CKKS / TFHE	Server-side under FHE	Client-side, in plaintext, exact	No	$O(L)$	No	Intermediate pre-activations in the clear b	Not addressed b
Safhire (ours)	RLWE	Scalar MACs with plaintext weights	Client-side, in plaintext, exact	No	$L$	No c	Shuffled pre-activations (see Section 5 )	Malicious (crafted inputs)

Table 3: Design-level comparison of Safhire with existing approaches to private inference.

L

denotes the number of linear blocks bounded by non-linearities.

Precision ( $b$ )	Key size	RLWE Noise ( $\Delta$ )	$\gamma=0$		$\gamma\in[1,2]$
	$(M=t^{\alpha})$		$D$	$l$	$D$	$l$
8 bits	$3^{7}$	$2^{-36}$	512	3	512	3
12 bits	$\mathbf{7^{4}}$	$2^{-51}$	$2^{12}$	3	$2^{16}$	2
16 bits	$\mathbf{5^{5}}$	$2^{-51}$	310	5	310	4

Table 4: Parameterization of Safhire for different precision levels

b

and levels of partial trace extraction level

\gamma

			Precision ( $b$ )
Model	Orion	Extraction level ( $\gamma$ )	8 bit		12 bit		16 bit
			Safhire	Speedup	Safhire	Speedup	Safhire	Speedup

ResNet-20 (CIFAR-10)	866.5 s	0	175.2 s	4.9 $\times$	277.1 s	3.1 $\times$	425.7 s	2.0 $\times$
		1	135.6 s	6.4 $\times$	131.3 s	6.6 $\times$	247.4 s	3.5 $\times$
		2	114.0 s	7.6 $\times$	252.8 s	3.4 $\times$	189.1 s	4.6 $\times$

Table 5: The E2E latencies of Safhire over Orion and associated speedups, for different trace extraction levels and precisions. To compute the duration of network communication in Safhire , we assume a conservative network speed of

1.25\text{\,}\mathrm{MB}\text{\,}{\mathrm{s}}^{-1}

. For each model and precision, we mark the largest speedup in bold.

Dataset	Model	Concrete-ML	Orion (SiLU)	Safhire
				16 bit	12 bit

CIFAR-10	ResNet-20	91.46%	91.70%	90.00%	89.63%
Tiny	ResNet-18	52.37%	57.00%	53.19%	53.16%
ImageNet	ResNet-34	64.06%	N/A	70.16%	49.47%

Table 6: Top-1 accuracy of Safhire across configurations.

Figure 4: Total inference time for Safhire under varying the network speed between the client and server. We consider networks speeds of 0.1, 0.3, 1, 3, 10, 30, 100, and 300

\mathrm{MB}

		Extraction level ( $\gamma$ )
		0	1	2
	Client	0.3 s	0.4 s	0.8 s
8 bits	Server	591.4 s	465.6 s	356.5 s
	Client	0.3 s	1.2 s	5.7 s
12 bits	Server	888.8 s	415.2 s	371.6 s
	Client	0.3 s	1.8 s	3.8 s

Table 7: Breakdown of total inference time between client and server on Tiny with ResNet-18.

Figure 5: Breakdown of the server-side execution time for a single sample inference in Safhire across various settings.

Figure 6: Breakdown of the client-server communication for a single sample inference in Safhire across various settings.

				Precision ( $b$ )
Model	Concrete-ML		Extraction level ( $\gamma$ )	8 bit			12 bit			16 bit
	Time (s)	Data (GB)		Safhire (s)	Data (MB)	Speedup	Safhire (s)	Data (MB)	Speedup	Safhire (s)	Data (MB)	Speedup

ResNet-20 (CIFAR-10)	86.8	1.8	0	60.9	18.0	1.4 $\times$	109.7	27.2	0.8 $\times$	147.4	22.7	0.6 $\times$
			1	35.6	22.7	2.4 $\times$	22.2	44.6	3.9 $\times$	36.9	35.3	2.4 $\times$
			2	19.7	41.7	4.4 $\times$	16.0	169.9	5.4 $\times$	20.5	102.5	4.2 $\times$

Table 8: The computation latencies of Safhire over Concrete-ML on multi-threaded execution and associated speedups, for different trace extraction levels and precisions. Both solutions are run using 32 threads. Latencies cover the client and server computation without network transfer, and Data is the total amount of data exchanged between the client and the server per inference. For each model and precision, we mark the largest speedup in bold.

Figure 7: The breakdown of server-side execution time of Safhire in function of the number of utilized threads.

Figure 8: Breakdown of the duration of individual server operations when increasing the number of threads, for ResNet-18 with the Tiny dataset, while fixing

b=16

Figure 9: The breakdown of server-side execution time on A100 GPU for a single sample inference using a ResNet-20 model on CIFAR-10 dataset.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Param.	Description
$\mu$	A cleartext message
$\operatorname{LWE}_{s}(\mu)$	LWE encryption of $\mu$ with key s
$\operatorname{RLWE}_{s(X)}(\mu(X))$	RLWE encryption of polynomial $\mu(X)$ with key s(X)
$M$	Cyclotomic polynomial index; size of RLWE key
$N$	$\phi(M)$ s.t. $\phi$ is Euler’s totient function
$\alpha$	Exponent in $M=t^{\alpha}$

Appendix

Table 9: Relevant RLWE parameters

Figure 10: The distribution of noise from the fast packing operation for varying precision and extraction levels.

Privacy-Preserving ML

Privacy-preserving machine learning develops methods to learn, share, and run models while limiting exposure of sensitive data.

See all 432 papers

Sep 8, 2026

Encrypt What Matters: When Selective Homomorphic Inference Is Efficient

Fully homomorphic encryption (FHE) enables inference on private data without revealing it to the server, but evaluating an entire input under FHE is expensive. We study selective homomorphic inference, where only a sensitive region of interest (ROI) is encrypted, and computations independent of that region are performed in plaintext. Selective evaluation produces the same output as full FHE on the same model, without retraining. Its efficiency depends on how quickly encrypted dependencies spread through the network. For small encrypted ROIs, locality-preserving architectures can achieve order-of-magnitude homomorphic-evaluation speedups, whereas architectures with early global mixing provide essentially no speedup. These results identify locality as the key architectural property governing the benefit of selective homomorphic inference.

Ali Backour, Juan Reyes, Jaime Punyed +1

Massachusetts Institute of Technology Cambridge, MA 02139

Efficient Inference Selective Inference Homomorphic Encryption Privacy-Preserving ML

Ask PDF

Sep 1, 2026

HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation

Fully homomorphic encryption (FHE) allows a server to run a language model directly on encrypted user prompts, but current approaches remain prohibitively slow. Ciphertexts natively support only addition, multiplication, and rotation, and multiplications may be composed only to a bounded depth before a costly bootstrapping operation is required to continue. Every nonlinearity must therefore be approximated by an iterative method; each iteration increasing the number of multiplications. A higher iteration count buys precision but exhausts the available depth more frequently and thus triggers more bootstraps, which dominate latency. We introduce Homomorphic Encryption-Aware Training (HEAT), a fine-tuning method that makes the per-nonlinearity iteration counts learnable, enabling them and the model weights to co-adapt during training. HEAT optimizes iterations with respect to the task objective, allowing the model to adapt to approximation errors encountered during inference without architectural changes or retraining from scratch. We further relate iteration count to quantization bit width and bound, at fixed weights, the gap between our objective and quantization-aware training. On encrypted GPT-2 decoding, HEAT reduces iterations by

3.1\times

, bootstraps by

1.6\times

, and end-to-end latency by

1.4\times

, while improving decode agreement over the calibrated encrypted baseline.

Alessandro Zirilli, Davide Marincione, Evgenios M. Kornaropoulos +2

Sapienza University of Rome · George Mason University · Paradigma

Privacy-Preserving Language Models Homomorphic Encryption Privacy-Preserving ML Efficient Language Model Inference

Ask PDF

Jun 15, 2026

FEnc $^2$ : Unifying Data Packing for Efficient Private Inference via Convolution and Architecture-Aware Fragment Encoding

Fully Homomorphic Encryption (FHE) enables privacy-preserving machine learning but incurs extreme computational and memory overhead. These costs come not only from expensive low-level primitives, including Number Theoretic Transform (NTT), rotation, and key-switching, but also from inefficient ciphertext packing at the application level. Existing packing strategies typically preserve either neighboring data elements or feature grouping, but not both, leading to wasted ciphertext slots, excessive rotations, and inflated ciphertext counts. We propose FEnc2, a unified and principled fragment-based encoding framework for CKKS-based private convolutional neural network inference. FEnc2 optimizes slot utilization, rotation complexity, and ciphertext density through two components: 1)Conv-aware Encoding, which analytically selects an optimal fragment size to decouple spatial dependencies and jointly minimize inner-outer rotations across layers, and 2)Arch-aware Ct Compression, which restores ciphertext density after feature- or channel-reduction layers. Together, these transformations reshape encrypted workload structure and reduce homomorphic operations by one to two orders of magnitude. With full memory capacity utilized, i.e., at maximum batch size, FEnc2 achieves end-to-end latency speedups over the state-of-the-art Orion of up to 228.83x on GPU and 226.06x on CPU for LeNet on MNIST, and up to 4.55x on GPU and 9.43x on CPU for MobileNet on ImageNet. FEnc2 is hardware-agnostic yet architecturally transformative: by optimizing encrypted tensor layout before execution, it reduces ciphertext count and workload pressure on hardware, complementing primitive-level optimizations such as NTT and keyswitch accelerators. These results show that application-level data layout is a first-order architectural design dimension for encrypted inference and an important enabler for next-generation FHE systems.

Ran Ran, Zhaoting Gong, Nuo Xu +3

North Carolina State University · University of Minnesota · University of California, Santa Cruz +1

Efficient Neural Network Inference Homomorphic Encryption Privacy-Preserving ML

Ask PDF

Abstract

Privacy-Preserving ML

Explore similar work

Encrypt What Matters: When Selective Homomorphic Inference Is Efficient

HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation

FEnc2^22: Unifying Data Packing for Efficient Private Inference via Convolution and Architecture-Aware Fragment Encoding

FEnc $^2$ : Unifying Data Packing for Efficient Private Inference via Convolution and Architecture-Aware Fragment Encoding