cs.CRFeb 16, 2026

Differentially Private Retrieval-Augmented Generation

Authors: Tingting Tang, James Flemings, Yongqin Wang, Murali Annavaram

Organizations: University of Southern California Los Angeles, California, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Retrieval-augmented generation (RAG) is a widely used framework for reducing hallucinations in large language models (LLMs) on domain-specific tasks by retrieving relevant documents from a database to support accurate responses. However, when the database contains sensitive corpora, such as medical records or legal documents, RAG poses serious privacy risks by potentially exposing private information through its outputs. Prior work has demonstrated that one can practically craft adversarial prompts that force an LLM to regurgitate the augmented contexts. A promising direction is to integrate differential privacy (DP), a privacy notion that offers strong formal guarantees, into RAG systems. However, naively applying DP mechanisms into existing systems often leads to significant utility degradation. Particularly for RAG systems, DP can reduce the usefulness of the augmented contexts leading to increase risk of hallucination from the LLMs. Motivated by these challenges, we present DP-KSA, a novel privacy-preserving RAG algorithm that integrates DP using the propose-test-release paradigm. DP-KSA follows from a key observation that most question-answering (QA) queries can be sufficiently answered with a few keywords. Hence, DP-KSA first obtains an ensemble of relevant contexts, each of which will be used to generate a response from an LLM. We utilize these responses to obtain the most frequent keywords in a differentially private manner. Lastly, the keywords are augmented into the prompt for the final output. This approach effectively compresses the semantic space while preserving both utility and privacy. We formally show that DP-KSA provides formal DP guarantees on the generated output with respect to the RAG database. We evaluate DP-KSA on two QA benchmarks using three instruction-tuned LLMs, and our empirical results demonstrate that DP-KSA achieves a strong privacy-utility tradeoff.

Figures & tables

Figure 1. Overview of the DP RAG problem setting. The adversarial bound illustrates what capabilities the adversary has. In this case, the adversary can only query the RAG system and access the answer. The generator model has been pre-trained on publicly available data, which the adversary has access to. However, we are not concerned about preserving the privacy of the pre-training data.
Figure 2. The proposed DP-KSA framework consists of three steps. First, it retrieves the top- NN documents most relevant to the query x\mathbf{x} from the private database DD . Each retrieved document DixD^{\mathbf{x}}_{i} is paired with query x\mathbf{x} and passed to the generator model FF to produce responses. Next, DP-KSA applies a differentially private mechanism to extract the most frequent keywords from the ensemble of responses. Finally, the selected keywords are combined with query x\mathbf{x} and fed back into the generator FF to produce the final output y\mathbf{y} .
Figure 3. Histogram of token lengths of ground truth answers in NQ dataset.
Figure 4. Histogram of token lengths of ground truth answers in TQA dataset.
Figure 5. Results of DP-KSA on NQ dataset with different generator LLMs: Qwen 2.5 (3B), Llama 3.2 (3B), and Llama 3.1 (8B). We use three baselines including non-RAG ( ϵ=0\epsilon=0 ), non-private RAG with top-2 retrieved documents ( ϵ=∞\epsilon=\infty ), and non-private KSA ( ϵ=∞\epsilon=\infty ).
Figure 6. Results of DP-KSA on TQA dataset with different generator LLMs: Qwen 2.5 (3B), Llama 3.2 (3B), and Llama 3.1 (8B). We use three baselines including non-RAG ( ϵ=0\epsilon=0 ), non-private RAG with top-2 retrieved documents ( ϵ=∞\epsilon=\infty ), and non-private KSA ( ϵ=∞\epsilon=\infty ).
Figure 7. Propose-test-release (PTR) test pass rate of DP-KSA with varying privacy parameters ϵ\epsilon on NQ and TQA datasets. We report the results with three different LLMs: Qwen 2.5 (3B), Llama 3.2 (3B), and Llama 3.1 (8B).
Figure 8. Ablation studies on model sizes with Qwen 2.5 model family on NQ dataset. Model size axis is in log scale. Plot showing ablation studies on model sizes with Qwen 2.5 model family on NQ dataset. Model size axis is in log scale.
Figure 9. Ablation studies on number of ensembles with Llama 3.2 (3B) model and ϵ=3\epsilon=3 on NQ dataset.
More in this topic

Privacy-Preserving RAG

See all 13 papers

Explore similar work

CardsList
Oct 15, 2026

Private Retrieval Augmented Generation via Random Projection

Retrieval-Augmented Generation (RAG) enhances the capabilities of large language models (LLMs) by querying external structured knowledge. However, it can also introduce privacy risks by leaking sensitive information from the retrieval datastore. We propose a simple method to preserve datastore privacy in RAG systems via random projection. By applying the same projection to both datastore embeddings and query embeddings, our method provably preserves semantic similarity between queries and retrieved items while substantially mitigating data extraction attacks. Across multiple RAG architectures and datasets, we show that this lightweight approach achieves superior retrieval and generation performance compared to prior methods which leverage formal differential privacy (DP) guarantees, while exhibiting comparable empirical privacy under strong attack models. Our results for the first time suggest that random projection can serve as a competitive and practical baseline for privacy-preserving RAG systems.

Dixi Yao, Tahseen Rabbani, Tian Li

University of Chicago

Privacy-Preserving RAGRAG SecurityRetrieval-Augmented GenerationPrivacy Leakage in Language Models
AskPDF
Jun 23, 2026

Privacy-Preserving RAG via Multi-Agent Semantic Rewriting: Achieving Confidentiality Without Compromising Contextual Fidelity

Retrieval-Augmented Generation enhances large language models by incorporating external knowledge, but deploying it in sensitive scenarios risks privacy leakage via malicious prompts. To address this, we propose a multi-agent framework that sanitizes retrieved content through semantic rewriting. By employing three specialized agents for privacy extraction, semantic analysis, and reconstruction, our approach collaboratively removes sensitive identifiers while preserving the semantic core. We evaluate the framework on the ChatDoctor and Wiki-PII datasets across six large language models. Experimental results demonstrate a significant reduction in privacy leakage under targeted attacks. For instance, we reduced targeted information exposure in LLaMA-3-8B from 144 instances in the baseline to just 1. Furthermore, we maintain strong contextual fidelity with a BLEU-1 score of 0.122, outperforming the existing SAGE method's 0.117. Finally, the framework operates as an asynchronous preprocessing module, introducing no additional latency to online inference, as all rewriting is executed as a one-time offline preprocessing step. To promote reproducibility, the source code of this work is publicly available at https://github.com/foursoils/Privacy-Preserving-RAG.

Yuanhe Zhao, Tianyu Zhang, Huafei Xing +3

School of Control and Computer Engineering, North China Electric Power University, Beijing, 102206, China · Department of Computer & Information Science & Engineering, University of Florida, Gainesville, Florida, 32611, USA · NLP2CT Lab, Department of Computer and Information Science, University of Macau, Macao, 999078, Macao SAR +1

Retrieval-Augmented GenerationPrivacy Leakage in Language ModelsPrivacy-Preserving RAGRAG Security
AskPDF
Mar 16, 2026

P2^2RAG: Efficient Privacy-Preserving RAG Service Supporting Arbitrary Top-kk Retrieval

Retrieval-Augmented Generation (RAG) enables large language models to use external knowledge, but outsourcing the RAG service raises privacy concerns for both data owners and users. Privacy-preserving RAG systems address these concerns by performing secure top-kk retrieval, which is typically implemented using secure sorting to identify relevant documents. However, existing systems face challenges supporting arbitrary kk due to their inability to change kk, new security issues, and in particular, efficiency degradation with large kk. This is a significant limitation because applications such as finance, law, and healthcare require a kk that is large enough to cause huge overhead for existing systems. Also, modern long-context models generally achieve higher accuracy with larger retrieval sets. We propose P2^2RAG, an efficient privacy-preserving RAG service that supports arbitrary top-kk retrieval. Unlike existing systems, P2^2RAG avoids sorting candidate documents. Instead, it uses an interactive bisection method to determine the set of top-kk documents. For security, P2^2RAG uses secret sharing on two semi-honest non-colluding servers to protect the data owner's database and the user's prompt. It enforces restrictions and verification to defend against malicious users and tightly bounds the information leakage of the database. The experiments show that P2^2RAG is 3--300×\times faster than the state-of-the-art PRAG for k=16k = 16--10241024.

Yulong Ming, Mingyue Wang, Jijia Yang +4

City University of Hong Kong · Peng Cheng Laboratory

Privacy-Preserving RAGRAG SecurityRetrieval-Augmented Generation
AskPDF