cs.CRNov 10, 2026

MOAI: Module-Optimizing Architecture for non-interactive secure transformer Inference

Authors: Linru Zhang, Xiangning Wang, Jun Jie Sim, Zhicong Huang, Jiahao Zhong, Huaxiong Wang, Pu Duan, Kwok-Yan Lam

Organizations: Nanyang Technological University · Ant International · Ant Group

Abstract

Privacy concerns have been raised in Large Language Models (LLM) inference when models are deployed in Cloud Service Providers (CSP). Homomorphic encryption (HE) offers a promising solution by enabling secure inference directly over encrypted inputs. However, the high computational overhead of HE remains a major bottleneck. To address this challenge, we propose MOAI, an efficient HE-based, non-interactive framework for secure transformer inference. MOAI gains significant efficiency improvement from: (1) a novel evaluation flow that combines column and diagonal packing with consistent strategies across all layers, eliminating expensive format conversions. (2) rotation-free algorithms for Softmax and LayerNorm that significantly reduce the number of costly HE rotations, removing 2,448 HE rotations in BERT-base inference. (3) Column packing removes rotations in plaintext-ciphertext matrix multiplications and interleaved batching further reduces the rotations in ciphertext-ciphertext matrix multiplications. MOAI uses at least 1.7x fewer HE rotations compared to the state-of-the-art works across all matrix multiplications of BERT-base. As a result, we achieve a 52.8%52.8 \% reduction in evaluation time compared to the state-of-the-art HE-based non-interactive secure transformer inference, THOR (Moon et al., CCS 25). We then apply MOAI on the Powerformer's framework and achieve a 55.7%55.7 \% reduction in evaluation time compared to Powerformer (Park et al., ACL 25), which approximates Softmax and LayerNorm with simpler functions in transformer and proposes HE-based non-interactive transformer inference. We report an amortized time of 2.36 minutes per input on a single GPU environment. We show the extendibility by applying MOAI in LLaMA-3-8B. Our implementation is publicly available as open source.

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Perseus: A Full-Stack System for Homomorphic Model Inference

    Nov 3, 2026Alessandro Zirilli, Davide Marincione, Evgenios M. Kornaropoulos +2Privacy-Preserving Language ModelsHomomorphic Encryption

  2. HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation

    Sep 1, 2026Alessandro Zirilli, Davide Marincione, Evgenios M. Kornaropoulos +2Privacy-Preserving Language ModelsHomomorphic Encryption

  3. ATLAS: Automated Approximation of Transformers for Efficient Homomorphic Inference in One Hour

    Jul 26, 2026Jianhang Xie, Sicheng Tan, Vishnu Naresh Boddeti +1Efficient Transformer InferenceHomomorphic Encryption