cs.AINov 19, 2026

Ask in the Crowd: Differentially Private LLM Inference via Dummy-Augmented Shuffling

Authors: Zhihao Liu, Zixiong Guo, Shuo Shao, Yu He, Wenli Wang, Meihui Chen, Di Wang, Yuke Hu

Organizations: 1Zhejiang University · 2Xiaomi Corporation · 3King Abdullah University of Science and Technology

Abstract

Large language models are increasingly deployed as LLM-as-a-Service (LLMaaS), where user queries are sent to remote LLMs for inference, raising severe privacy concerns about the exposure of sensitive user inputs. Differentially private inference mitigates these risks by injecting noise into input embeddings. However, existing methods struggle to balance privacy and utility, as stronger privacy protection often leads to substantial accuracy degradation in LLM inference. In this work, we propose DP-SPD, a differentially private LLM inference framework that protects user queries by hiding the real query among client-generated dummy queries. All queries are randomly shuffled before being sent to the server, preventing direct identification of the user's true intent. To improve inference efficiency, we exploit the shared-prefix structure among queries and reuse transformer key--value (KV) caches during decoding. We analyze the privacy guarantee of DP-SPD and evaluate its utility on classification and generation tasks, as well as its privacy protection under QIA, EIA, and GPT-based attacks. Experimental results show that our method achieves strong inference-time privacy protection and near-plaintext utility while improving inference efficiency through shared-prefix KV-cache reuse.

Figures & tables

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. SharedRequest: Privacy-Preserving Model-Agnostic Inference for Large Language Models

    Jun 3, 2026Peihua Mai, Xuanrong Gao, Youlong Ding +3Privacy-Preserving Language ModelsLLM Inference

  2. SpliTEE: Fast and Private LLM Inference by Coupling GPU-Assisted Trusted Execution Environments with Differential Privacy

    Sep 14, 2026Shashie Dilhara Batan Arachchige, Robin Carpentier, Hassan Jameel Asghar +1Privacy-Preserving Language ModelsSplit Inference

  3. DP-MIND: Differentially private memory-guided Long context Inference

    Oct 9, 2026Zhihao Liu, Yitian Cheng, Shuang Liu +3Privacy-Preserving Language ModelsLong-Context Language Model Inference