Personal AI agents make recommendations and take actions on people's behalf in high-stakes economic contexts, e.g., buying a flight, choosing health insurance, or selecting a graduate program. The agent is given access to the user's personal context, e.g., their email inbox and a structured profile of personal attributes, with the intention of making an optimal, personalized decision for the user. We show that by simply providing this personal context, the agent steers recommendations based on inferred wealth, without being explicitly instructed to do so. In a suite of 325K experiments on 13 agents across three types of economic decisions (flights, health insurance, and graduate programs), we find that 8 models systematically choose more expensive options for wealthier users when requests are identical. This steering continues even when it directly goes against the user's stated objective: when explicitly instructed to find the cheapest option, some agents still act on the wealth profile they have inferred. It also occurs when wealth is inferred from ambient data, such as emails unrelated to the task. And it persists under privacy controls that block specific attributes: blocking financial attributes largely removes the disparity, but blocking other attributes leaves it unchanged and can increase it by up to 40% for insurance, as agents rely on the remaining signals to infer wealth. Larger and more capable models are no better; Claude Opus 4.8 shows the largest effect. We term this misalignment "adversarial delegation", in which the very conditions that make a personal AI agent useful - access to personal information - enable it to act against the user's interests.
Figures & tables
Figure 1: Example of adversarial delegation in a single evaluation. The user queries for the most affordable airfare to Chicago. The agent returns a 91economyclassticketifitisuncalibrated—thatis,hasnoknowledgeoftheuser(left).However,whentheagenthasaccesstouserdata(right),itfindsthreeemailsaboutfinance,fromwhichitinfersthattheuserwouldbewillingtopaymoreforaseat,andrecommendsa601 business-class ticket, despite the existence of the $91 ticket. The agent was not informed of the user’s net worth nor instructed to take it into account. We show that this phenomenon is systematic and not just anecdotal across our experiments (Table 1 ).
Figure 2: Blocking financial information most strongly changes the discrimination gap, while blocking other attributes generally leaves it intact or amplifies it. Each row blocks a single attribute (returning access_restricted ) while the other four remain available. Results are shown for GPT-5, Claude Opus 4.8, and Gemini 2.5 Flash across flights, insurance, and graduate programs. Each cell reports the discrimination gap in the blocked condition, with the percentage change in its magnitude relative to the corresponding tool-full condition shown in parentheses. Color encodes this relative change: blue indicates a reduction in gap magnitude, red indicates an increase, and zero indicates no change. Negative gap values indicate that the direction of the disparity has reversed. Across most model–domain pairs, blocking financial information sharply reduces the gap, whereas blocking employment, health, life events, or demographics typically preserves or increases it.
Figure 3: Limited email access can produce larger discrimination gaps than full-inbox access. Discrimination gaps on flights for GPT-5, Claude Opus 4.8, and Gemini 2.5 Flash across the access spectrum, from inferential email access to direct profile access. With no email bodies available, gaps are near zero across all three models. For Opus 4.8 and Gemini 2.5 Flash, the gap is largest after only two emails (248and175, respectively) and decreases as more of the inbox becomes available, reaching 59and91 with full-inbox access. GPT-5 shows the same qualitative pattern at smaller magnitude.
Figure 4: Explicit preferences do not necessarily remove wealth-conditioned recommendations. Recommended flight prices under four user intents for Gemini 2.5 Flash, GPT-5, and Claude Opus 4.8. Bars show the low-income recommendation and the additional wealth gap between the wealthy and the low-income. Even when users explicitly request the cheapest option, Gemini 2.5 Flash still has a 208gap,whilethecorrespondinggapsforGPT−5andOpus4.8are21 and $20, respectively.
Family
Model
Flights ($)
Insurance ($/mo)
Grad schools ($/yr)
Mean d
GPT
GPT-5.5
+92
+122
+763
0.26
GPT-5
+107
+191
+1,061
0.33
GPT-5-mini
+74
+124
+2,647
0.34
GPT-5-nano
+13
+56
+1,622
0.16
Claude
Claude Opus 4.8
+198
+284
+3,467
0.85
Claude Sonnet 5
+176
+151
+2,406
0.52
Table 1: Discrimination gap by model and domain (tool-full condition); larger is not safer. Models are grouped by family. The disparity increases with model size within the same family (e.g. GPT-5 nano to base with +13\to+74 → +107;AppendixB.5examinesthegapbetweenretrievalandrecommendationsteering),whereasthedisparitywithGPT−5.5(+92) is lower than GPT-5, implying capabilities and discriminatory behavior are not monotonically increasing with model generations. Claude Opus 4.8 produces the largest effect across all models (Cohen’s d=0.85 ).
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Domain
Inventory
Price range
Context
Flights
200 flights
91–883
Denver (DEN) to Chicago (ORD), Jan 10–24
Health insurance
200 plans
85–1,350/mo
Colorado, single zip code
Graduate programs
200 CS PhD programs
−20Kto+61K net/yr
Same undergraduate background
Appendix
Table 2: Domain inventories and price ranges.
Intent
Flights
Insurance
Graduate programs
Neutral
“Find me a flight”
“Find me a plan”
“Help me find programs”
Cheap
“Find the cheapest”
“Cheapest covering basics”
“Funded, highest acceptance”
Quality
“Most comfortable”
“Best coverage regardless”
“Highest-ranked regardless”
Ceiling
“Under $200”
“Under $220/month”
“Full funding”
Appendix
Table 3: User intents by domain.
Condition
Count
Persona access
Role in the design
Control
1
No persona information
Provides the stateless baseline.
In-context
1
All five axes of persona information
Provides the full profile in the system prompt without any retrieval.
Tool-full
1
All five axes through get_attributes(axis)
Enables us to measure which profile information the agent chooses to access.
Tool-blocked
5
Four axes available and one restricted
Measures how access to each axis changes the recommendation gap relative to tool-full.
Email-full
1
All ten bodies through read_email(id)
Measures recommendations when the agent can infer the full persona from email content.
Email-capped
5
Subjects plus 0, 2, 4, 6, or 8 bodies
Measures how reading more emails changes recommendations; the zero-read condition tests subject lines alone (giving us the subject-baseline to control for bias).
Appendix
Table 4: Information-access conditions. The count gives the number of conditions in each group.
Subject
High-level body
Low-level body
“Your Annual Tax Filing”
Capital gains $847K, K-1 forms
W-2 income $31,400, EITC
“Health Insurance Update”
Platinum PPO renewal, concierge care
Medicaid redetermination notice
“Neighborhood Newsletter”
HOA meeting, $2.1M median home value
Section 8 recertification
“Employment Update”
Board meeting, stock vesting schedule
Shift schedule for next week
“Life Update”
Closing on second property
Eviction prevention resources
Appendix
Table 5: Matched email examples for the five persona axes. Subjects are held fixed within each high–low pair. The objective here is to both test information access-priority and ability to infer user-profile
Condition
GPT-5
GPT-5- mini
GPT-5- nano
GPT-5.5
Claude Opus 4.8
Claude Sonnet 5
Claude Haiku 4.5
Gemini 2.5 Flash
Gemini 3.1 Flash Lite
Gemini 3 Flash
Qwen3.5- 2B
Qwen3.5- 9B
Qwen3.5- 35B-A3B
in_context
+153
+119
+64
+134
+136
+124
+103
+249
+98
+181
+12
+154
+148
tool_full
+107
+74
+13
+92
+198
+176
+141
+177
+112
+145
+14
+138
+141
tool_blocked_financial
-11
+9
-2
-1
+4
+1
-2
+17
+8
-4
+6
-5
+10
tool_blocked_employment
+117
+69
+25
+123
+224
+220
+159
+197
+162
+192
+21
+153
+170
tool_blocked_health
+92
+62
+38
+88
+176
+173
+138
+155
+119
+150
+12
+135
+118
tool_blocked_life_events
+113
+69
+46
+96
+214
+204
+151
+178
+99
+175
+10
+136
+144
Appendix
Table 6: Flight discrimination gap ($) by information condition, averaged over intents and motivations.
Condition
GPT-5
GPT-5- mini
GPT-5- nano
GPT-5.5
Claude Opus 4.8
Claude Sonnet 5
Claude Haiku 4.5
Gemini 2.5 Flash
Gemini 3.1 Flash Lite
Gemini 3 Flash
Qwen3.5- 2B
Qwen3.5- 9B
Qwen3.5- 35B-A3B
in_context
+240
+192
+128
+189
+346
+186
+201
+247
+60
+181
+26
+203
+174
tool_full
+191
+124
+56
+122
+284
+151
+158
+217
+179
+332
+17
+195
+133
tool_blocked_financial
-1
+1
-7
-14
-9
+1
-13
-3
-43
+17
-14
-1
+0
tool_blocked_employment
+210
+105
+77
+171
+317
+195
+194
+246
+183
+370
+20
+236
+151
tool_blocked_health
+220
+129
+53
+126
+323
+171
+169
+219
+326
+508
+29
+179
+104
tool_blocked_life_events
+201
+109
+69
+124
+287
+157
+166
+232
+250
+330
+63
+214
+150
Appendix
Table 7: Insurance discrimination gap ($/mo) by information condition, averaged over intents and motivations.
Condition
GPT-5
GPT-5- mini
GPT-5- nano
GPT-5.5
Claude Opus 4.8
Claude Sonnet 5
Claude Haiku 4.5
Gemini 2.5 Flash
Gemini 3.1 Flash Lite
Gemini 3 Flash
Qwen3.5- 2B
Qwen3.5- 9B
Qwen3.5- 35B-A3B
in_context
+1,599
+2,859
+2,078
+1,644
+2,576
+2,623
+2,898
+3,938
+8,214
+9,337
+267
+3,070
+3,605
tool_full
+1,061
+2,647
+1,622
+763
+3,467
+2,406
+2,322
+607
+5,591
+11,081
-2,056
+2,872
+3,827
tool_blocked_financial
+117
+4
+152
-114
-58
+50
+27
-1,001
-648
+656
+1,792
-471
-403
tool_blocked_employment
+1,678
+2,501
+2,558
+664
+4,486
+3,570
+3,105
+682
+6,384
+11,787
+2,911
+3,609
+4,470
tool_blocked_health
+1,733
+2,279
+2,341
+635
+3,803
+2,740
+2,962
+3,926
+5,193
+15,226
-627
+3,299
+3,290
tool_blocked_life_events
+2,542
+2,887
+2,636
+1,228
+3,993
+3,550
+3,128
+2,427
+6,226
+10,513
-59
+2,694
+4,908
Appendix
Table 8: Graduate school discrimination gap ($/yr) by information condition, averaged over intents and motivations.
Blocked axis
GPT-5
Gemini 2.5 Flash
Gemini 3 Flash
Qwen3.5-9B
Qwen3.5- 35B-A3B
None (tool-full)
+107
+177
+145
+138
+141
Financial
− 11
+17
− 4
− 5
+10
Employment
+117
+197
+192
+153
+170
Health
+92
+155
+150
+135
+118
Life Events
+113
+178
+175
+136
+144
Demographics
+119
+178
+172
+150
+146
Appendix
Table 9: Flight discrimination gap ($) by blocked axis, with all intents pooled. Appendix B.2 reports all models and domains.
Model
Financial Retrieval (%)
Flights d
Insurance d
Flights Gap ($)
GPT-5
100.0
0.41
0.42
+107
GPT-5-mini
97.9
0.29
0.29
+74
GPT-5-nano
87.1
0.07
0.15
+13
GPT-5.5
100.0
0.39
0.28
+92
Claude Opus 4.8
100.0
1.01
0.73
+198
Claude Sonnet 5
100.0
0.83
0.33
+176
Appendix
Table 10: Financial-axis retrieval and recommendation effects for all 13 models in the tool-full condition.
Personal AI assistants are changing how individuals use advice. We study how an advisor should design its recommendation in anticipation of stochastic consultation with personal AI whose recommendation is predictable. Personal AI enters through two dimensions: consultation probability and relative trust, which captures the relative influence personal AI receives when consulted. In the baseline model, the advisor optimally counteracts the personal AI signal. Counteraction increases with consultation probability but is hump-shaped in relative trust. The advisor's minimized loss is hump-shaped in consultation probability, vanishing when personal AI is never or always consulted. Greater relative trust in personal AI increases the irreducible loss arising from stochastic consultation. We extend the analysis to partial predictability and costly recommendation adjustment, characterizing their effects on optimal recommendations and minimized loss. The framework also accommodates richer information structures, including settings in which personal AI is perceived as having access to private information relevant to the task. We introduce an online forecasting experiment that examines how participants obtain personal AI advice and combine it with an advisor's recommendation and their initial judgments. Participants place weight on all three inputs. When access requires an additional action, some participants do not seek personal AI advice, while some others attempt to obtain it without success. Together, these findings highlight two distinct aspects of personal AI use: whether advice is obtained and how much weight it receives when available.
Yueyang Liu, Wichinpong Park Sinchaisri
1Jones Graduate School of Business, Rice University · 2Haas School of Business, University of California, Berkeley
Consumers are increasingly delegating purchase decisions to AI agents, providing natural-language descriptions of their preferences and identity. We argue that these representations constitute an information channel, role coherence, through which sellers can infer willingness to pay without explicit disclosure by the buyer agent, leading to preference leakage. In an experiment where a language-model buyer agent shops on behalf of a verbal consumer profile, we show that seller-side inference from dialogue alone recovers willingness to pay nearly one-for-one. Comparing this setting to a numeric-budget condition with confidentiality instructions cleanly isolates role coherence as distinct from instruction-following failure. Because this leakage arises from delegation itself, it cannot be mitigated at the prompt level. Instead, we propose architectural interventions that trade off personalization against preference privacy.
As people turn to generative AI for financial advice, these systems can personalize how they communicate and what they say. Whether these forms of personalization shape decisions differently remains unclear. We conducted a preregistered 2 x 2 between-subjects factorial experiment (N=240): participants ranked three comparably viable stocks, discussed them with an AI, and reranked them. Participants perceived both forms of personalization, but only context-based personalization reliably changed ranking behavior: it increased reconsideration and moved rankings toward the AI's assigned recommendation. Participants felt more influenced without judging the AI as more correct, trustworthy, intelligent, likeable, or high-quality. Those initially farther from its recommendation moved more toward it while judging its advice less correct; exploratory analyses suggest greater susceptibility among lower-expertise participants. These findings show how personalized AI can steer decisions among defensible options with only a minimal evaluative trace, raising concerns for the design and governance of personalized decision support.
Hasibur Rahman, Benjamin R. Cowan, Smit Desai
Northeastern University, Boston, Massachusetts, USA · University College Dublin, Dublin, Ireland