Privacy concerns have been raised in Large Language Models (LLM) inference when models are deployed in Cloud Service Providers (CSP). Homomorphic encryption (HE) offers a promising solution by enabling secure inference directly over encrypted inputs. However, the high computational overhead of HE remains a major bottleneck. To address this challenge, we propose MOAI, an efficient HE-based, non-interactive framework for secure transformer inference. MOAI gains significant efficiency improvement from: (1) a novel evaluation flow that combines column and diagonal packing with consistent strategies across all layers, eliminating expensive format conversions. (2) rotation-free algorithms for Softmax and LayerNorm that significantly reduce the number of costly HE rotations, removing 2,448 HE rotations in BERT-base inference. (3) Column packing removes rotations in plaintext-ciphertext matrix multiplications and interleaved batching further reduces the rotations in ciphertext-ciphertext matrix multiplications. MOAI uses at least 1.7x fewer HE rotations compared to the state-of-the-art works across all matrix multiplications of BERT-base. As a result, we achieve a 52.8% reduction in evaluation time compared to the state-of-the-art HE-based non-interactive secure transformer inference, THOR (Moon et al., CCS 25). We then apply MOAI on the Powerformer's framework and achieve a 55.7% reduction in evaluation time compared to Powerformer (Park et al., ACL 25), which approximates Softmax and LayerNorm with simpler functions in transformer and proposes HE-based non-interactive transformer inference. We report an amortized time of 2.36 minutes per input on a single GPU environment. We show the extendibility by applying MOAI in LLaMA-3-8B. Our implementation is publicly available as open source.
Table 1: Comparison of number of HE rotations in matrix multiplications. For short, we use σ to denote Softmax and omit d′ . t is the intermediate size of the feed-forward layer. H is the number of heads. c=16 is from THOR. k=64 is from Powerformer.
Function
NEXUS # RotHE
THOR # RotHE
MOAI (ours) # RotHE
softmax(X∈Rm×m)
⌈log2(m)⌉4m⋅2⌈log2(m)⌉/N
⌈log2(m)⌉4m⋅⋅2⌈log2(m)⌉/N
0
\layernorm(X∈Rm×d)
⌈log2(d)⌉4d⋅2⌈log2(d)⌉/N
⌈log2(d)⌉4d⋅2⌈log2(d)⌉/N
0
Total (Substitute BERT’s parameters into the equation: N=216 , m=128 , d=768 , H=12 , 12 layers)
2448
2448
0
Table 2: Comparison with NEXUS Zhang et al. (2024a) , THOR Moon et al. (2024) and MOAI Softmax and LayerNorm.
Function
NEXUS # RotHE
MOAI (Ours) # RotHE
softmax
⌈log2(m)⌉4m⋅2⌈log2(m)⌉/N
0
\layernorm
⌈log2(d)⌉4d⋅2⌈log2(d)⌉/N
0
# RotHE for BERT
2448
0
Table 3: Comparison with NEXUS Zhang et al. (2024a) , THOR Moon et al. (2024) and MOAI Softmax and LayerNorm.
Figure 1: Our rotation-free Softmax evaluation whose input and output are both in diagonal packing.
Figure 2: The full diagram of MOAI. The parameters are from the BERT-base-uncased model, which has 12 transformer layers and 12 heads in each layer. c is the number of classes.
Component
Operation in each layer
Depth
Packing
(Data dimensions) × (Number of operations)
CPU time(s)
GPU time(s)
Attention
Pt-ct MatrixMul
15→14
Col → Col
(R128×768×R768×64)×36
37.4
10.2
Ct-ct MatrixMul
14→13
Col → Diag
(R128×64×R64×128)×12
40.3
8.55
Softmax
13→3
Diag → Diag
R128×128×12
53.3
0.74
Ct-ct MatrixMul
3→2
Diag → Col
(R128×128×R128×64)×12
1.4
0.26
Self Output
Pt-ct MatrixMul
2→1
Col → Col
(R128×768×R768×768)
1.7
0.41
Bootstrapping
1→21
Col → Col
R128×768
95.4
26.1
Table 4: Breakdown of MOAI encrypted inference, amortized over 256 inputs
Algorithm 7 \multdiag,col(\encdiag(diag(C)),\enccol(V~)) using interleaved batching.
Function
Scheme
Key switch
Equation
Number
{XW∗(h)}h∈[H]X∈Rm×d , W∗∈Rd×d′∗=Q,K,V
NEXUS
2md/(N/(2m))
768
THOR
2m+3(H−2)cd′+c3d′
192
Powerformer
≈95⋅2N/2md(3d)
122
MOAI (ours)
0
0
{Q(h)(K(h))⊺}h∈[H]Q(h)∈Rm×d′K(h)∈Rm×d′
NEXUS
m(d′+1)H/(N/(4m))
780
Appendix
Table 8: Comparison of HE matrix multiplications in transformer architecture. The concrete parameters are based on BERT-base-uncased model: m=128 , d=768 , d′=64 , t=3072 , H=12 . N is the CKKS ring dimension. c=16 is from THOR. k=64 is from Powerformer.
Figure 4: The flow of HE evaluation of LayerNorm.
\enc(d⋅mean(X)):=∑i∈[d]\enc(xi⊺),
Appendix
Algorithm 8 HE evaluation of LayerNorm.
Figure 5: Absolute error of Goldschmidt division algorithm in Softmax
Scheme
Softmax
LayerNorm
Size
CPU time(s)
GPU time(s)
Size
CPU time(s)
GPU time(s)
MOAI
128×128
0.62
0.005
128×768
0.38
0.0039
NEXUS (NDSS’25)
128×128
2.53
0.032
128×768
46.7
0.34
THOR (CCS’25)
128×128
-
0.11
128×768
-
0.59
Appendix
Table 9: Comparison between Softmax and LayerNorm implementations
Operation in each layer
Description
Depth
(Data dimensions) × (Number of operations)
GPU time(s)
Pt-ct MatrixMul
Q=XWQ
17→16
(R8×4096×R4096×4096)
20.2
Pt-ct MatrixMul
K=XWK , V=XWV
17→16
(R8×4096×R4096×1024)×2
10.1
Pt-ct position-wise Mul
RoPE(Q)
16→15
R8×4096
0.11
Pt-ct position-wise Mul
RoPE(K)
16→15
R8×1024
0.025
Ct-ct MatrixMul
QK⊺
15→14
(R8×128×R128×8)×32
8.6
Casual masked SoftMax
σ′(QK⊺)
14→3
R8×8×32
0.6
Appendix
Table 10: Amortized breakdown of MOAI encrypted inference of LLaMA-3-8B with 8 input tokens
Model
Dataset
SST-2
QNLI
RTE
LLaMA-3-8B
Unencrypted input
94.9
94.2
84.2
Encrypted input
94.5
93.6
82.2
Appendix
Table 11: Scores on the GLUE benchmarks
Operation in each layer
Description
(Data dimensions) × (Number of operations)
GPU time(s)
Attention Heads
X→σ′(QK⊺)V
R1×4096→(R1×128×32)
1.57
Pt-ct MatrixMul
XWselfoutput
(R1×4096×R4096×4096)
0.18
Bootstrapping
-
R1×4096
1.18
Bootstrapping after LayerNorm
-
R1×4096
1.15
Pt-ct MatrixMul
u=XWu,v=XWv
(R1×4096×R4096×14336)×2
12.22
SiLU
h=SiLU(u)
R1×14336
0.82
Appendix
Table 12: Amortized breakdown of MOAI encrypted inference of LLaMA-3-8B with input token length =1
Figure 6: Box and whisker figure of absolute error in one layer
Large Language Models (LLMs) have made it exceptionally convenient for users to send their private data to model providers, which process them in plaintext. Fully Homomorphic Encryption (FHE) promises to secure remote inference by computing directly on encrypted data, yet it has remained prohibitively slow. The effort to make it practical remains fragmented, with most existing frameworks being closed-source, which keeps progress from compounding. We present Perseus, a free and open Python software library that supports the full stack of encrypted model inference, from the cryptographic backend and data packing, to encrypted nonlinearities and end-to-end model execution. Perseus ships with built-in novel optimizations, one at each level of the stack: in-kernel regeneration of key-switching keys at the (i) backend level, a complex variant of Cachemir packing at the (ii) data level, and a state-of-the-art bootstrap placer that is both packing- and magnitude-aware at the (iii) model level. Perseus achieves state-of-the-art latency across multiple models and modalities, within 3% of the plaintext models' accuracy and perplexity. Remarkably, it is 3.96× faster at single-token decoding than any other implementation on Llama-3-8B. We initiate, for the first time, a public speedrun on encrypted GPT-2, so that progress can compound.
Alessandro Zirilli, Davide Marincione, Evgenios M. Kornaropoulos +2
Sapienza University of Rome · Paradigma · George Mason University
Fully homomorphic encryption (FHE) allows a server to run a language model directly on encrypted user prompts, but current approaches remain prohibitively slow. Ciphertexts natively support only addition, multiplication, and rotation, and multiplications may be composed only to a bounded depth before a costly bootstrapping operation is required to continue. Every nonlinearity must therefore be approximated by an iterative method; each iteration increasing the number of multiplications. A higher iteration count buys precision but exhausts the available depth more frequently and thus triggers more bootstraps, which dominate latency. We introduce Homomorphic Encryption-Aware Training (HEAT), a fine-tuning method that makes the per-nonlinearity iteration counts learnable, enabling them and the model weights to co-adapt during training. HEAT optimizes iterations with respect to the task objective, allowing the model to adapt to approximation errors encountered during inference without architectural changes or retraining from scratch. We further relate iteration count to quantization bit width and bound, at fixed weights, the gap between our objective and quantization-aware training. On encrypted GPT-2 decoding, HEAT reduces iterations by 3.1×, bootstraps by 1.6×, and end-to-end latency by 1.4×, while improving decode agreement over the calibrated encrypted baseline.
Alessandro Zirilli, Davide Marincione, Evgenios M. Kornaropoulos +2
Sapienza University of Rome · George Mason University · Paradigma
Fully homomorphic encryption (FHE) provides strong cryptographic guarantees for private inference, but deploying transformer models under FHE remains prohibitively expensive. A key bottleneck is that non-linear operations such as softmax, normalization, and activation must be replaced with polynomial approximations compatible with the CKKS scheme, and the multiplicative depth consumed by these approximations dominates inference cost. Recent frameworks have advanced approximation techniques, yet all rely on manually configured approximation hyperparameters (e.g., number of iterations, polynomial degree), applied uniformly across all layers. While convenient, this uniform-configuration approach is overly rigid: different layers can tolerate different levels of approximation error without degrading predictive accuracy, and uniform configurations cannot exploit this variability to reduce latency. Allowing each layer to adopt its own configuration, however, causes the search space to explode with model depth, reaching roughly 1084 configurations for BERT/ViT (12 layers) and 10225 for LLaMA3 (32 layers), rendering manual exploration practically impossible. We present ATLAS, an automated framework that configures per-layer approximation settings by formulating the problem as a multi-objective optimization over latency and predictive accuracy. The resulting problem is inherently difficult: 1) competing objectives over a large decision space (120 or 320 variables for BERT/ViT or LLaMA3); 2) expensive evaluation, as each configuration takes 70-1,000 seconds even in cleartext; and 3) sparse optimization signals, as 35-50% of candidate configurations yield numerically invalid solutions. ATLAS addresses these challenges through a two-stage optimization strategy that progressively relaxes layer-wise constraints, combined with surrogate models to accelerate evaluation.