cs.LGOct 21, 2026

Privacy in Context Is Being Overlooked in the Age of General-Purpose LLMs

Authors: Prakhar Ganesh, Yash More, Marco Romanelli, Ferdinando Fioretto, Golnoosh Farnadi

Organizations: McGill University · Mila - Quebec AI Institute · Hofstra University · University of Virginia

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

As large language models (LLMs) continue to grow in scope and capability, privacy concerns are on the rise. Since users increasingly interact with a single general-purpose system across many distinct contexts, there is a natural tendency to pursue a one-size-fits-all solution to privacy. Unfortunately, this overlooks a crucial insight from decades of privacy research: privacy is inherently contextual. In this paper, we argue that existing work in prompt privacy is drifting towards attempts to design a unified framework, which is highly misleading and can foster a false sense of security. Furthermore, we argue that even future work in this direction needs to wrestle with fundamental tensions across various desirable properties of prompt privacy. Thus, such continued attempts towards a unified framework need to be reconsidered. We recommend three critical paths forward: application-specific deployment of LLMs, context-specific evaluation of privacy frameworks, and better privacy literacy.

Figures & tables

Figure 1: Five desirable properties of a prompt privacy framework ( § 2 ), coverage of various existing frameworks in the literature ( § 6 ), and several non-trivial underlying tensions ( § 7 ).
Figure 2: Prompt Privacy Pipeline and a Glossary of terms used throughout the paper. The user interacts with a privacy tool/transform before their query is sent to the LLMs . All privacy interventions occur within a local trusted environment, ensuring that only the protected query is transmitted. We assume both the LLMs and the communication channel can have malicious actors. The final reconstructed output is compared against the hypothetical output of the original query to make sure the LLM utility is preserved.
Figure 3: (Left) Detailed annotations of literature in prompt privacy, highlighting the evidence of a drift away from privacy in context. (Right) Detailed annotations of publicly available tools for prompt privacy, highlighting similar trends of drift as the literature.
Figure 4: Four commonly adopted frameworks for prompt privacy: sanitization, differential privacy, cryptography, and contextual integrity.
Figure 5: Motivating use cases as defined in § 5 , along with our recommendations based on the preceding discussions in the paper.
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

TitlePrivacy Target/Adversary GoalsUse CaseExplicit Mention of LimitationsTechnique Category
CoGenesis: A Framework Collaborating Large and Small Language Models for Secure Context-Aware Instruction Following [ Zhang et al., 2024b ]Additional ContextContext-Aware InstructionsNoContextual Integrity
Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory [ Mireshghallah et al., 2024b ]Input PromptNoYesContextual Integrity
CipherGPT: Secure Two-Party GPT Inference [ Hou et al., 2023 ]Input PromptNoNoCryptography
Iron: Private Inference on Transformers [ Hao et al., 2022 ]Input PromptNoNoCryptography
THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption [ Chen et al., 2022 ]Input PromptNoNoCryptography
A Verified Confidential Computing as a Service Framework for Privacy Preservation [ Chen et al., 2023a ]Input PromptNoYesCryptography
Appendix
Table 1: Annotations for all 31 papers.
TitleUse CasePriceTechnique Category
JanGeneral PurposeFreeLocal Deployment
llama.cppGeneral PurposeFreeLocal Deployment
LocalAIGeneral PurposeFreeLocal Deployment
OllamaGeneral PurposeFreeLocal Deployment
PasteGuardGeneral PurposeFreeSanitization/Obfuscation
ShimmyGeneral PurposeFreeLocal Deployment
Appendix
Table 2: Annotations for all 11 tools.
More in this topic

Privacy-Preserving Language Models

Privacy-preserving language models protect sensitive data during training, inference, personalization, and information sharing.

See all 178 papers

Explore similar work

CardsList

  1. Say Something Else: Rethinking Contextual Privacy as Information Sufficiency

    Apr 7, 2026Yunze Xiao, Wenkai Li, Xiaoyuan Wu +3Privacy-Preserving Language ModelsPrivacy Protection for LLM Agents

    Carnegie Mellon University · Massachusetts Institute of Technology · 2077AI +1

  2. Long Context, Less Focus: A Scaling Gap in LLMs Revealed through Privacy and Personalization

    Feb 16, 2026Shangding GuPrivacy Leakage in Language ModelsLong-Context Language Model Evaluation

    University of California, Berkeley

  3. Personalized Privacy Control in LLMs via Attention Head Intervention

    Aug 21, 2026Junseok Kim, Nakyeong Yang, Kyomin JungPrivacy-Preserving Language ModelsPersonalized Language Model Alignment

    IPAI, Seoul National University · Max Planck Institute for Software Systems