Privacy concerns have been raised in Large Language Models (LLM) inference when models are deployed in Cloud Service Providers (CSP). Homomorphic encryption (HE) offers a promising solution by enabling secure inference directly over encrypted inputs. However, the high computational overhead of HE remains a major bottleneck. To address this challenge, we propose MOAI, an efficient HE-based, non-interactive framework for secure transformer inference. MOAI gains significant efficiency improvement from: (1) a novel evaluation flow that combines column and diagonal packing with consistent strategies across all layers, eliminating expensive format conversions. (2) rotation-free algorithms for Softmax and LayerNorm that significantly reduce the number of costly HE rotations, removing 2,448 HE rotations in BERT-base inference. (3) Column packing removes rotations in plaintext-ciphertext matrix multiplications and interleaved batching further reduces the rotations in ciphertext-ciphertext matrix multiplications. MOAI uses at least 1.7x fewer HE rotations compared to the state-of-the-art works across all matrix multiplications of BERT-base. As a result, we achieve a 52.8% reduction in evaluation time compared to the state-of-the-art HE-based non-interactive secure transformer inference, THOR (Moon et al., CCS 25). We then apply MOAI on the Powerformer's framework and achieve a 55.7% reduction in evaluation time compared to Powerformer (Park et al., ACL 25), which approximates Softmax and LayerNorm with simpler functions in transformer and proposes HE-based non-interactive transformer inference. We report an amortized time of 2.36 minutes per input on a single GPU environment. We show the extendibility by applying MOAI in LLaMA-3-8B. Our implementation is publicly available as open source.
Table 1: Comparison of number of HE rotations in matrix multiplications. For short, we use σ to denote Softmax and omit d′ . t is the intermediate size of the feed-forward layer. H is the number of heads. c=16 is from THOR. k=64 is from Powerformer.
Function
NEXUS # RotHE
THOR # RotHE
MOAI (ours) # RotHE
softmax(X∈Rm×m)
⌈log2(m)⌉4m⋅2⌈log2(m)⌉/N
⌈log2(m)⌉4m⋅⋅2⌈log2(m)⌉/N
0
\layernorm(X∈Rm×d)
⌈log2(d)⌉4d⋅2⌈log2(d)⌉/N
⌈log2(d)⌉4d⋅2⌈log2(d)⌉/N
0
Total (Substitute BERT’s parameters into the equation: N=216 , m=128 , d=768 , H=12 , 12 layers)
2448
2448
0
Table 2: Comparison with NEXUS Zhang et al. (2024a) , THOR Moon et al. (2024) and MOAI Softmax and LayerNorm.
Function
NEXUS # RotHE
MOAI (Ours) # RotHE
softmax
⌈log2(m)⌉4m⋅2⌈log2(m)⌉/N
0
\layernorm
⌈log2(d)⌉4d⋅2⌈log2(d)⌉/N
0
# RotHE for BERT
2448
0
Table 3: Comparison with NEXUS Zhang et al. (2024a) , THOR Moon et al. (2024) and MOAI Softmax and LayerNorm.
Figure 1: Our rotation-free Softmax evaluation whose input and output are both in diagonal packing.
Figure 2: The full diagram of MOAI. The parameters are from the BERT-base-uncased model, which has 12 transformer layers and 12 heads in each layer. c is the number of classes.
Component
Operation in each layer
Depth
Packing
(Data dimensions) × (Number of operations)
CPU time(s)
GPU time(s)
Attention
Pt-ct MatrixMul
15→14
Col → Col
(R128×768×R768×64)×36
37.4
10.2
Ct-ct MatrixMul
14→13
Col → Diag
(R128×64×R64×128)×12
40.3
8.55
Softmax
13→3
Diag → Diag
R128×128×12
53.3
0.74
Ct-ct MatrixMul
3→2
Diag → Col
(R128×128×R128×64)×12
1.4
0.26
Self Output
Pt-ct MatrixMul
2→1
Col → Col
(R128×768×R768×768)
1.7
0.41
Bootstrapping
1→21
Col → Col
R128×768
95.4
26.1
Table 4: Breakdown of MOAI encrypted inference, amortized over 256 inputs
Algorithm 7 \multdiag,col(\encdiag(diag(C)),\enccol(V~)) using interleaved batching.
Function
Scheme
Key switch
Equation
Number
{XW∗(h)}h∈[H]X∈Rm×d , W∗∈Rd×d′∗=Q,K,V
NEXUS
2md/(N/(2m))
768
THOR
2m+3(H−2)cd′+c3d′
192
Powerformer
≈95⋅2N/2md(3d)
122
MOAI (ours)
0
0
{Q(h)(K(h))⊺}h∈[H]Q(h)∈Rm×d′K(h)∈Rm×d′
NEXUS
m(d′+1)H/(N/(4m))
780
Appendix
Table 8: Comparison of HE matrix multiplications in transformer architecture. The concrete parameters are based on BERT-base-uncased model: m=128 , d=768 , d′=64 , t=3072 , H=12 . N is the CKKS ring dimension. c=16 is from THOR. k=64 is from Powerformer.
Figure 4: The flow of HE evaluation of LayerNorm.
\enc(d⋅mean(X)):=∑i∈[d]\enc(xi⊺),
Appendix
Algorithm 8 HE evaluation of LayerNorm.
Figure 5: Absolute error of Goldschmidt division algorithm in Softmax
Scheme
Softmax
LayerNorm
Size
CPU time(s)
GPU time(s)
Size
CPU time(s)
GPU time(s)
MOAI
128×128
0.62
0.005
128×768
0.38
0.0039
NEXUS (NDSS’25)
128×128
2.53
0.032
128×768
46.7
0.34
THOR (CCS’25)
128×128
-
0.11
128×768
-
0.59
Appendix
Table 9: Comparison between Softmax and LayerNorm implementations
Operation in each layer
Description
Depth
(Data dimensions) × (Number of operations)
GPU time(s)
Pt-ct MatrixMul
Q=XWQ
17→16
(R8×4096×R4096×4096)
20.2
Pt-ct MatrixMul
K=XWK , V=XWV
17→16
(R8×4096×R4096×1024)×2
10.1
Pt-ct position-wise Mul
RoPE(Q)
16→15
R8×4096
0.11
Pt-ct position-wise Mul
RoPE(K)
16→15
R8×1024
0.025
Ct-ct MatrixMul
QK⊺
15→14
(R8×128×R128×8)×32
8.6
Casual masked SoftMax
σ′(QK⊺)
14→3
R8×8×32
0.6
Appendix
Table 10: Amortized breakdown of MOAI encrypted inference of LLaMA-3-8B with 8 input tokens
Model
Dataset
SST-2
QNLI
RTE
LLaMA-3-8B
Unencrypted input
94.9
94.2
84.2
Encrypted input
94.5
93.6
82.2
Appendix
Table 11: Scores on the GLUE benchmarks
Operation in each layer
Description
(Data dimensions) × (Number of operations)
GPU time(s)
Attention Heads
X→σ′(QK⊺)V
R1×4096→(R1×128×32)
1.57
Pt-ct MatrixMul
XWselfoutput
(R1×4096×R4096×4096)
0.18
Bootstrapping
-
R1×4096
1.18
Bootstrapping after LayerNorm
-
R1×4096
1.15
Pt-ct MatrixMul
u=XWu,v=XWv
(R1×4096×R4096×14336)×2
12.22
SiLU
h=SiLU(u)
R1×14336
0.82
Appendix
Table 12: Amortized breakdown of MOAI encrypted inference of LLaMA-3-8B with input token length =1
Figure 6: Box and whisker figure of absolute error in one layer