cs.HCMar 19, 2026

Investigating In-Context Privacy Learning by Integrating User-Facing Privacy Tools into Conversational Agents

Authors: Mohammad Hadi Nezhad, Francisco Enrique Vicente Castro, Ivon Arroyo

Organizations: University of Massachusetts Amherst · New York University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Supporting users in protecting sensitive information when using conversational agents (CAs) is crucial, as users may undervalue privacy protection due to outdated, partial, or inaccurate knowledge about privacy in CAs. Although privacy knowledge can be developed through standalone resources, it may not readily translate into practice and may remain detached from real-time contexts of use. In this study, we investigate in-context, experiential learning by examining how interactions with privacy tools during chatbot use enhance users' privacy learning. We also explore interface design features that facilitate engagement with these tools and learning about privacy by simulating ChatGPT's interface which we integrated with a just-in-time privacy notice panel. The panel intercepts messages containing sensitive information, warns users about potential sensitivity, offers protective actions, and provides FAQs about privacy in CAs. Participants used versions of the chatbot with and without the privacy panel across two task sessions designed to approximate realistic chatbot use. We qualitatively analyzed participants' pre- and post-test survey responses and think-aloud transcripts and describe findings related to (a) participants' perceptions of privacy before and after the task sessions and (b) interface design features that supported or hindered user-led protection of sensitive information. Finally, we discuss future directions for designing user-facing privacy tools in CAs that promote privacy learning and user engagement in protecting privacy in CAs.

Figures & tables

Figure 1: ChatGPT Interface Simulation With Privacy Notice Panel. The panel appears after sending a message containing sensitive info (highlighted in the input). It includes (A) a warning message, (B) an anonymization panel, (C) shortcuts to built-in privacy controls, (D) FAQs, and (E) a proceed with sending button. The anonymization panel further includes, for each detected instance, a (B.1.) locate icon, (B.2.) drop-down menu of anonymization options, and (B.3.) restore button.
IDDegree - YearGenderChatGPT UseTask Assignment Version
   DurationFrequencySession 1Session 2
P1Master - 1st yearFemale~2.5 yearDailyAB
P2Undergrad - 4th yearFemale~2.5 yearsDailyBA
P3Master - 1st yearFemale~2.5 yearsSeveral times a weekAB
P4Master - 2nd yearMale~2.5 yearsDailyBA
P5Undergrad - 3rd yearMale~1.5 yearsDailyAB
Table 1: Participant Details (self-reported) and their Task Assignment Versions. Session 1 refers to the task session using the chatbot without the panel, and Session 2 refers to the session using the chatbot with the panel.
RQThemes & Sub-ThemesBefore StudyAfter Study
RQ1.1. How students conceptualize sensitive information.• Potentials for harm if leaked ( 4.1.1 )P1, P2, P3, P5, P6, P7, P8P1, P2, P3, P7, P10
 • Restricted access ( 4.1.2 )P1, P4, P5, P6, P7, P9, P10P3 , P4, P5, P6, P7, P9, P10
 • Linkable to identity ( 4.1.3 )P2, P3, P4, P8, P10P2, P3, P8
 • Context-based sensitivity ( 4.1.4 )P5, P9P6 , P9, P10
 • Security measures ( 4.1.5 )P5P6
RQ1.2. Views on the importance of protecting sensitive info during chatbot use• Unclear data lifecycle ( 4.2.1 )P1, P4, P5, P6, P8, P9, P10P3 , P4, P5, P6, P7 , P8, P9
Table 2: Columns (L →\rightarrow R): RQ, identified Themes & Sub-Themes, the list of participants who articulated each theme before and after the study. Bold participant IDs denote those who discussed the theme after the study but not before.
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Simulated ChatGPT settings panels for built-in privacy controls: (left) Personalization panel for enabling or disabling memory, with a functional toggle and static buttons; (right) Data Control panel for opting in or out of content sharing for model training, with an interactive toggle.
VTask DescriptionSensitive InfoFInfo Subject
ATask 1: Email Analysis • Step 1: Classifying seven emails 1 1 1 Drawn from the Enron Email Corpus and modified as needed. by their sentiments • Step 2: Summarizing two separate email threadsName Email address Phone number Physical address61 23 6 4Others (e.g., employees)
ATask 2: Searching Housing Documents • Step 1: Finding party restrictions from their housing contract (9-page doc) • Step 2: Finding maintenance information from the same housing contract and a welcome letter (5-page doc)Name Email address Phone number Physical address SSN Date of birth38 9 16 16 4 4Self, Others (e.g., friend, home owner)
ATask 3: Planning a Trip • Step 1: Finding timing conflicts between their travel itinerary (2-page doc) and semester schedule (1-page doc) • Step 2: Drafting an email to resolve the conflicts and provide their contact details for reservation purposesName Email address Phone number Physical address Date of birth5 1 3 7 2Self, Others (e.g., friend)
BTask 1: Customer Complaint Analysis • Step 1: Classifying seven customer complaints 2 2 2 Drawn from a Ticket Classification Dataset and modified as needed. by relevant departments • Step 2: Summarizing recommendations for the company from two separate groups of customer complaintsName Email address Phone number Physical address Date of birth18 6 4 4 2Others (e.g., customers)
BTask 2: Searching Car & Housing Documents • Step 1: Finding costs and fees related to purchasing a leased car from their car lease contract (15-page doc) • Step 2: Finding information about utilities and responsibilities from their housing contract (9-page doc)Name Email address Phone number Physical address SSN Date of birth23 4 10 12 4 4Self, Others (e.g., friend, home owner)
BTask 3: Planning Teeth Removal • Step 1: Finding out-of-pocket expenses from their insurance document (6 pages) and an email from their doctor. • Step 2: Drafting an email to their doctor providing insurance details and asking if the insurance is acceptedName Email address Phone number Physical address Date of birth8 2 2 1 5Self, Others (e.g., doctor)
Appendix
Table 3: User Tasks. Columns (L →\rightarrow R): task assignment version (V), task description, types of sensitive info contained in each task (Sensitive Info), frequency—of occurrences—of each type (F), and Info Subjects (who the info pertains/belongs to).
More in this topic

Privacy Protection for LLM Agents

See all 29 papers

Explore similar work

CardsList
Sep 28, 2026

PrivacySkills: How Privacy Guidance Shapes Source Selection in LLM Agents

While prior work has documented privacy failures in LLM agents, it remains unclear how the presentation of privacy guidance influences their choice of information sources. We introduce PrivacySkills, a controlled framework for evaluating how agents choose among acquisition pathways that provide the same task-relevant value: consulting publicly available personal information, accessing confidential sources, or interacting with the user. The evaluation framework comprises 55 synthetic tasks spanning 11 categories of personal information, with 169 associated skills that describe the available acquisition pathways. We consider privacy guidance through system-level instructions, skill-level metadata labels, or both. Separately, we vary user availability and urgency framing. With users available and no privacy guidance, agents access confidential sources in 30% of valid runs on average across five open-weight models, despite sufficient alternatives. This rate increases to 45% when users are unavailable, whereas urgency framing has no detectable effect. System-level privacy instructions alone have limited effects on confidential access, while skill-level intrusiveness labels produce a modest reduction (24% on average), but combining the two roughly halves confidential access. Our findings motivate incorporating privacy annotations into skill specifications and evaluating their effectiveness alongside system-level instructions.

Lucas Biechy, Cédric Eichler, Héber H. Arcolezi +1

Petscraft, Inria · Université Paris-Saclay · INSA CVL +3

LLM Agent EvaluationPrivacy Protection for LLM Agents
AskPDF
Jul 3, 2026

PromptPET: Privacy-Utility Optimized Prompt Obfuscation

Privacy is an important challenge when users interact with AI chatbots, since users may share sensitive information, explicitly or implicitly, and AI chatbots can use this information for user profiling. In this paper, we aim to protect user privacy via a user-side mechanism that transforms sensitive information in a user prompt, while preserving enough information to elicit a useful response from the chatbot. This approach faces an inherent tradeoff between protecting privacy (i.e., avoiding profiling) and preserving utility (i.e., getting personalized and task-specific responses). To that end, we consider, evaluate, and compare four different obfuscation actions, namely redaction, abstraction, replacement, and a novel noising/denoising scheme that we introduce. Additional novel insights include: utilizing a data type taxonomy to both identify and obfuscate sensitive information and explicitly taking into account the utility of chat responses in making the obfuscation decision. First, we systematically optimize and evaluate each obfuscation action independently in terms of the privacy-utility tradeoff it achieves. Second, we propose PROMPTPET, an LLM-based agent that selects the best obfuscation action for each sensitive part of the prompt, using a reinforcement-learning inspired rule optimizer, applied for the first time in this context. Using a real-world chat dataset, we show that PROMPTPET matches the best privacy-utility tradeoff attainable by any single obfuscation action and significantly outperforms prior state-of-the-art approaches.

Ke Yang, Olivia Figueira, Umar Iqbal +1

University of California, Irvine Irvine, CA, USA · Washington University in St. Louis St. Louis, MO, USA

Privacy-Preserving Language ModelsPrivacy-Utility Tradeoff
AskPDF
Mar 31, 2026

Can LLMs Infer Conversational Agent Users' Personality Traits from Chat History?

Sensitive information, such as knowledge about an individual's personality, can be can be misused to influence behavior (e.g., via personalized messaging). To assess to what extent an individual's personality can be inferred from user interactions with LLM-based conversational agents (CAs), we analyze and quantify related privacy risks of using CAs. We collected actual ChatGPT logs from N=668 participants, containing 62,090 individual chats, and report statistics about the different types of shared data and use cases. We fine-tuned RoBERTa-base text classification models to infer personality traits from CA interactions. The findings show that these models achieve trait inference with accuracy (ternary classification) better than random in multiple cases. For example, for extraversion, accuracy improves by +44% relative to the baseline on interactions for relationships and personal reflection. This research highlights how interactions with CAs pose privacy risks and provides fine-grained insights into the level of risk associated with different types of interactions.

Derya Cögendez, Verena Zimmermann, Noé Zufferey

ETH Zurich, Switzerland

Privacy Leakage in Language ModelsConversational Agents
AskPDF