As large language models (LLMs) continue to grow in scope and capability, privacy concerns are on the rise. Since users increasingly interact with a single general-purpose system across many distinct contexts, there is a natural tendency to pursue a one-size-fits-all solution to privacy. Unfortunately, this overlooks a crucial insight from decades of privacy research: privacy is inherently contextual. In this paper, we argue that existing work in prompt privacy is drifting towards attempts to design a unified framework, which is highly misleading and can foster a false sense of security. Furthermore, we argue that even future work in this direction needs to wrestle with fundamental tensions across various desirable properties of prompt privacy. Thus, such continued attempts towards a unified framework need to be reconsidered. We recommend three critical paths forward: application-specific deployment of LLMs, context-specific evaluation of privacy frameworks, and better privacy literacy.
Figures & tables
Figure 1: Five desirable properties of a prompt privacy framework ( § 2 ), coverage of various existing frameworks in the literature ( § 6 ), and several non-trivial underlying tensions ( § 7 ).
Figure 2: Prompt Privacy Pipeline and a Glossary of terms used throughout the paper. The user interacts with a privacy tool/transform before their query is sent to the LLMs . All privacy interventions occur within a local trusted environment, ensuring that only the protected query is transmitted. We assume both the LLMs and the communication channel can have malicious actors. The final reconstructed output is compared against the hypothetical output of the original query to make sure the LLM utility is preserved.
Figure 3: (Left) Detailed annotations of literature in prompt privacy, highlighting the evidence of a drift away from privacy in context. (Right) Detailed annotations of publicly available tools for prompt privacy, highlighting similar trends of drift as the literature.
Figure 4: Four commonly adopted frameworks for prompt privacy: sanitization, differential privacy, cryptography, and contextual integrity.
Figure 5: Motivating use cases as defined in § 5 , along with our recommendations based on the preceding discussions in the paper.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Title
Privacy Target/Adversary Goals
Use Case
Explicit Mention of Limitations
Technique Category
CoGenesis: A Framework Collaborating Large and Small Language Models for Secure Context-Aware Instruction Following [ Zhang et al., 2024b ]
Additional Context
Context-Aware Instructions
No
Contextual Integrity
Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory [ Mireshghallah et al., 2024b ]
LLM agents increasingly draft messages on behalf of users, yet users routinely overshare sensitive information and disagree on what counts as private. Existing systems support only suppression (omitting sensitive information) and generalization (replacing information with an abstraction), and are typically evaluated on single isolated messages, leaving both the strategy space and evaluation setting incomplete. We formalize privacy-preserving LLM communication as an Information Sufficiency (IS) task, introduce free-text pseudonymization as a third strategy that replaces sensitive attributes with functionally equivalent alternatives, and propose a conversational evaluation protocol that assesses strategies under realistic multi-turn follow-up pressure. Across 792 scenarios spanning three power-relation types (institutional, peer, intimate) and three sensitivity categories (discrimination risk, social cost, boundary), we evaluate seven frontier LLMs on privacy at two granularities, covertness, and utility. Pseudonymization yields the strongest privacy\textendash utility tradeoff overall, and single-message evaluation systematically underestimates leakage, with generalization losing up to 16.3 percentage points of privacy under follow-up.
Yunze Xiao, Wenkai Li, Xiaoyuan Wu +3
Carnegie Mellon University · Massachusetts Institute of Technology · 2077AI +1
Large language models (LLMs) are increasingly deployed in privacy-critical and personalization-oriented scenarios, yet the role of context length in shaping privacy leakage and personalization effectiveness remains largely unexplored. We introduce a large-scale benchmark, PAPerBench, to systematically study how increasing context length influences both personalization quality and privacy protection in LLMs. The benchmark comprises approximately 29,000 instances with context lengths ranging from 1K to 256K tokens, yielding a total of 377K evaluation questions. It jointly evaluates personalization performance and privacy risks across diverse scenarios, enabling controlled analysis of long-context model behavior. Extensive evaluations across state-of-the-art LLMs reveal consistent performance degradation in both personalization and privacy as context length increases. We further provide a theoretical analysis of attention dilution under context scaling, explaining this behavior as an inherent limitation of soft attention in fixed-capacity Transformers. The empirical and theoretical findings together suggest a general scaling gap in current models -- long context, less focus. We release the benchmark to support reproducible evaluation and future research on scalable privacy and personalization. Code and data are available at https://github.com/SafeRL-Lab/PAPerBench
The rise of agentic AI enables LLMs to access diverse user data, raising critical privacy concerns. Prior work on contextual privacy studies whether LLMs regulate information disclosure according to context-dependent norms. However, acceptable disclosure boundaries may vary across users even within the same context. To address this limitation, we introduce personalized privacy, which incorporates user-specific disclosure preferences into privacy control. We further present P3Bench~(Personalized Privacy Preservation Benchmark), a novel benchmark extending contextual privacy policies with personalized disclosure policies. Experiments show that prompt-based policies fail to reliably enforce personalized privacy policies, with Qwen2.5-7B and Gemma3-4B showing average policy ignorance ratios of 51.25% and 74.28%, respectively. Finally, to address this problem, we propose Repair, a robust inference-time attention head intervention method that adjusts disclosure behavior toward policy-consistent responses. Our method significantly improves adherence to user-specific privacy preferences by reducing cases where the model fails to follow the given policy.
Junseok Kim, Nakyeong Yang, Kyomin Jung
IPAI, Seoul National University · Max Planck Institute for Software Systems