It's My Agent, Not Me: Personal Privacy Boundaries in AI-Delegated Information Sharing

Bingcan Guo; Eryue Xu; Zhiping Zhang; Tianshi Li

cs.CRSep 26, 2025

It's My Agent, Not Me: Personal Privacy Boundaries in AI-Delegated Information Sharing

Authors: Bingcan Guo, Eryue Xu, Zhiping Zhang, Tianshi Li

Organizations: Department of Human Centered Design & Engineering, University of Washington, United States · School of Information Sciences, University of Illinois Urbana-Champaign, United States · Khoury College of Computer Sciences, Northeastern University, United States

Abstract

Aligning AI systems with human privacy preferences requires understanding individuals' nuanced disclosure behaviors beyond general privacy norms. Yet eliciting these boundaries remains challenging because they involve context-dependent trade-offs. We present an elicitation approach that probes individuals' privacy boundaries through a discriminative task. Through a between-subjects study with varying communication roles and delegation conditions, we collected 1,650 privacy boundaries from 169 participants across 60 scenarios. We examined how communication roles, AI delegation, and individual differences influence boundaries. Results show that communication roles significantly influence sensitivity to disclosure granularity or identifiability, depending on whose private information is shared; AI delegation and individuals' need for privacy heighten sensitivity to disclosed identifiers. Our analysis further reveals how AI delegation shifts privacy preferences across contexts, highlighting new complexities in AI-mediated communication. We advocate eliciting nuanced privacy boundaries within real-world data flows by considering roles and AI delegation, and enabling user involvement in guiding privacy-aware AI behavior.

Figures & tables

Figure 1. Even when a disclosure violates general privacy norms, individuals may still find some versions of it acceptable. P134, the data subject in the AI-delegation condition, evaluated messages written by a sender’s AI agent to share P134’s recent traumatic experience with a mutual friend over Messenger. P134 accepted the general variants (A, B) but rejected those with more detail (C, D).

Granularity / Identifiability	Not Identifiable	Partially Identifiable	Fully Identifiable
General	“ A friend of mine is settling into a healthier daily routine. ”	“ Our close friend is settling into a healthier daily routine. ”	“ Emily is settling into a healthier daily routine. ”
Moderately Detailed	“ A friend of mine has begun morning jogs, joined a support meeting, and had their first therapy session. ”	“ Our close friend has begun morning jogs, joined a support meeting, and had her first therapy session. ”	“ Emily has begun morning jogs, joined a support meeting, and had her first therapy session. ”
Very Detailed	“ A friend of mine now jogs at the park each morning, and they attended a meeting at Haven of Hope and completed their first therapy session as part of a new routine. ”	“ Our close friend now jogs at the park each morning, and she attended a meeting at Haven of Hope and completed her first therapy session as part of a new routine. ”	“ Emily now jogs at the park each morning, and she attended a meeting at Haven of Hope and completed her first therapy session as part of a new routine. ”

Table 1. Example of nine disclosure variants that could be shared by the sender in scenario main 129 ( “A person is sharing the new daily routine and habits of their friend, a survivor of intimate partner violence, with a shared friend by sending a message on Messenger” ). Minimal edits were applied to some variants to ensure grammatical correctness.

Figure 2. An overview of the personal privacy boundary elicitation task (single scenario). ① The participant was presented with the corresponding version of the scenario visual description based on the communication role they were assigned. ② A rating prompt question was displayed based on their assigned communication role and delegation condition (see Section 5.4 ). ③ The participant then rated nine disclosure variants presented in random order by indicating “Yes” or “No” for acceptance. The example is assigned the recipient role and AI condition.

Panel of Figure 3 — Figure 3. Scenario visual representations for the three communication roles in Scenario main 3.

Figure 4. Heatmaps of variant acceptance rate for Self and Others scenarios in Human/AI agent conditions show an overall trend of decreasing acceptance as granularity and identifiability increase (except for the Self (Human) scenario).

	Acceptance of Private Information Disclosures ( Self )
	Model 1 Coef. (S.E.)	Model 2 Coef. (S.E.)
Granularity (Linear)	$-0.412^{***}$ (0.067)	$-0.720^{***}$ (0.121)
Identifiability (Linear)	$-0.384^{***}$ (0.067)	0.121 (0.120)
Communication Role (Recipient)	0.390 (0.278)	0.352 (0.287)
Delegation Condition (AI)	$-0.527$ (0.282)	$-0.525$ (0.288)
Need for Privacy	$-0.186$ (0.146)	$-0.193$ (0.149)

Table 2. Generalized Linear Mixed Models (GLMMs) results for participants’ acceptance of private information disclosure in Self scenarios. Model 1 shows the main effects of granularity, identifiability, communication role, delegation condition, need for privacy, AI attitudes, and age; Model 2 additionally includes the preregistered interaction effects.

	Acceptance of Private Information Disclosures ( Others )
	Model 1 Coef. (S.E.)	Model 2 Coef. (S.E.)
Granularity (Linear)	$-0.551^{***}$ (0.040)	$-0.646^{***}$ (0.083)
Identifiability (Linear)	$-1.602^{***}$ (0.044)	$-1.552^{***}$ (0.089)
Communication Role (Recipient)	0.162 (0.199)	0.148 (0.204)
Communication Role (Subject)	0.001 (0.220)	0.007 (0.224)
Delegation Condition (AI)	$-0.076$ (0.174)	$-0.101$ (0.177)

Table 3. Generalized Linear Mixed Models (GLMMs) results for participants’ acceptance of private information disclosure in Others scenarios. Model 1 shows the main effects of granularity, identifiability, communication role, delegation condition, need for privacy, AI attitudes, and age; Model 2 additionally includes the preregistered interaction effects.

Figure 11. Exploratory analysis of how AI delegation shifts averaged $3\times 3$ privacy boundaries, by scenario type and communication role. Each grid shows the average AI $-$ Human difference across scenarios in percentage points. Negative values (blue) indicate lower acceptance under AI delegation; positive values (red) indicate higher acceptance. ${}^{*}p<.05$ ; ${}^{**}p<.01$ (Bonferroni corrected within each role’s nine cells).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Demographics
Sex		N	Sample (%)
	Female	86	50.9%
	Male	83	49.1%
Age
	18-24	11	6.5%
	25-34	50	29.6%

Appendix

Table 4. Demographics of participants (N=169)

ID	Description	Information Type †
main 1	A lawyer is sharing detailed notes on a pending legal case involving a defendant accused of embezzlement with visitors on Facebook by sending a Facebook post.	others
main 100	A person is sharing detailed discussions about their friend (a stalking victim)’s workplace with social media visitors by sending a Facebook post.	others
main 104	An employee is sharing their coworker’s personal conversations about relationship issues within the team with a mutual acquaintance at work by sending a Slack direct message.	others
main 107	A Wikipedia editor is sharing their detailed notes on editing Wikipedia articles on controversial topics with their social media followers by sending a Facebook post.	self
main 109	A person is sharing their friend’s active participation in a niche fandom community, with another shared friend by sending a message on Messenger.	others
main 111	A therapist is sharing detailed notes on a particular client’s journey, including struggles and triumphs with overcoming addiction with social media followers by sending a Facebook post.	others

Appendix

Table 5. Scenarios selected from the PrivacyLens dataset.

Panel of Figure 12 — Figure 12. Prompt for generating disclosure variants for one scenario (part 1).

Panel of Figure 13 — Figure 13. Prompt for generating disclosure variants for one scenario (part 2).

	Quadratic Polynomial Components ( Self )
	Model 1 Coef. (S.E.)	Model 2 Coef. (S.E.)
Granularity (Quadratic)	$-0.204^{**}$ (0.067)	$-0.181$ (0.121)
Identifiability (Quadratic)	0.005 (0.067)	$-0.028$ (0.120)
Granularity (Q) $\times$ Identifiability (L)		0.076 (0.118)
Granularity (L) $\times$ Identifiability (Q)		0.079 (0.116)
Granularity (Q) $\times$ Identifiability (Q)		0.017 (0.117)

Appendix

Table 6. Quadratic polynomial components from the GLMMs of participants’ acceptance of private information disclosure in Self scenarios. Only the quadratic (Q) terms are shown.

	Quadratic Polynomial Components ( Others )
	Model 1 Coef. (S.E.)	Model 2 Coef. (S.E.)
Granularity (Quadratic)	$-0.004$ (0.040)	0.090 (0.082)
Identifiability (Quadratic)	$-0.107^{**}$ (0.039)	0.074 (0.081)
Granularity (Q) $\times$ Identifiability (L)		0.055 (0.072)
Granularity (L) $\times$ Identifiability (Q)		$-0.102$ (0.068)
Granularity (Q) $\times$ Identifiability (Q)		$-0.057$ (0.068)

Appendix

Table 7. Quadratic polynomial components from the GLMMs of participants’ acceptance of private information disclosure in Others scenarios. Only the quadratic (Q) terms are shown.

Preference Elicitation

Preference elicitation studies how systems learn people’s values and trade-offs from questions, choices, rankings, and feedback.

See all 69 papers

When Agents Shop for You: Role Coherence in AI-Mediated Markets
Apr 29, 2026Soogand Alavi, Salar NozariPrivacy Leakage in Language Models Agentic Commerce
Tippie College of Business, University of Iowa
CIDER: A Dataset of Contextual Disclosure Boundaries for Privacy Preference Alignment
Aug 10, 2026Bingcan Guo, Eryue Xu, Jijie Zhou +2Preference Alignment Inference-Time Language Model Alignment
University of Washington · UIUC · Northeastern University
The Interlocutor Effect: Why LLMs Leak More Personal Data to Agents Than Humans
Apr 26, 2026Faouzi El Yagoubi, Godwin Badu-Marfo, Ranwa Al MallahData Leakage Multi-Agent LLM Systems
Department of Computer and Software Engineering, Polytechnique Montréal, Montréal, Canada

Apr 29, 2026J/K move · Enter open · S save

When Agents Shop for You: Role Coherence in AI-Mediated Markets

Soogand Alavi, Salar Nozari

Tippie College of Business, University of Iowa

Privacy Leakage in Language Models Agentic Commerce User Preference Modeling Privacy-Utility Tradeoff Personal AI Agents

Open paper AskPDF arXiv

Consumers are increasingly delegating purchase decisions to AI agents, providing natural-language descriptions of their preferences and identity. We argue that these representations constitute an information channel, role coherence, through which sellers can infer willingness to pay without explicit disclosure by the buyer agent, leading to preference leakage. In an experiment where a language-model buyer agent shops on behalf of a verbal consumer profile, we show that seller-side inference from dialogue alone recovers willingness to pay nearly one-for-one. Comparing this setting to a numeric-budget condition with confidentiality instructions cleanly isolates role coherence as distinct from instruction-following failure. Because this leakage arises from delegation itself, it cannot be mitigated at the prompt level. Instead, we propose architectural interventions that trade off personalization against preference privacy.

Abstract

Preference Elicitation

Explore similar work

When Agents Shop for You: Role Coherence in AI-Mediated Markets

CIDER: A Dataset of Contextual Disclosure Boundaries for Privacy Preference Alignment

The Interlocutor Effect: Why LLMs Leak More Personal Data to Agents Than Humans