As Large Language Models (LLMs) become a primary interface between users and the web, companies face growing economic incentives to embed commercial influence into AI-mediated conversations. We present two preregistered experiments (N = 2,012) in which participants selected a book to receive from a large eBook catalog using either a traditional search engine or a conversational LLM agent powered by one of five frontier models. Unbeknownst to participants, a fifth of all products were randomly designated as sponsored and promoted in different ways. We find that LLM-driven persuasion nearly triples the rate at which users select sponsored products compared to traditional search placement (61.2% vs. 22.4%), while the vast majority of participants fail to detect any promotional steering. Explicit "Sponsored" labels do not significantly reduce persuasion, and instructing the model to conceal its intent makes its influence nearly invisible (detection accuracy < 10%). Altogether, our results indicate that conversational AI can covertly redirect consumer choices at scale, and that existing transparency mechanisms may be insufficient to protect users.
Figures & tables
Figure 1. Experimental design and outcome measures. (A) After screening for active readers and completing a small pre-survey, participants engaged in a shopping task in which they browsed a real eBook catalog and selected a book to receive after the experiment. Unbeknownst to participants, a fifth of all products were randomly designated as sponsored and promoted in different ways. Depending on the experimental condition, participants interacted either with a traditional search interface or with a conversational LLM agent powered by one of five frontier models (GPT-5.2, Claude Opus 4.5, Gemini 3 Pro, DeepSeek v3.2, or Qwen3 235b). After the task, participants completed a post-survey measuring satisfaction and bias detection, and chose between keeping their selected book or receiving a 1cashbonus.Afterdebriefingthemaboutthepresenceofsponsoredproducts,participantsmadethischoiceasecondtime.(B)Participantswererandomlyassignedtooneoffivebetween−subjectsconditions,spanningtwopreregisteredstudies.Study1comparedatraditionalsearchwithuprankedsponsoredproducts(SP),achat−basedplacementofsponsoredproductsfirstinthecarousel(CP),andachatwithactiveLLMpersuasiontowardsponsoredproducts(CPer).Study2testedtransparencyandconcealment:CPer–ExpreplicatedCPerwithanexplicit“Sponsored”labelandwarning,whileCPer–SbtinstructedtheLLMtoconcealitspersuasiveintent.(C)Threeprimaryoutcomescapturethearcofcommercialinfluence:(PersuasionRate)whetherparticipantsselectasponsoredproduct,(SalesRate)whethertheyvaluetheirbookchoiceenoughtokeepitoverthe1 cash alternative, and (Bias Detection) whether they detect that persuasion occurred at all.
Figure 2. Persuasion, Sales Rate, and Bias Detection across experimental conditions. Point estimates are estimated marginal means (EMMs) from OLS models with condition, LLM model, and their interaction as predictors, using HC3 robust standard errors; EMMs marginalize over the LLM factor with equal weights ( N=2012 ; see Methods). Error bars denote 95% confidence intervals. The dashed vertical line in panel A marks the 20% chance baseline (one in five products was randomly designated as sponsored). ( A ) Persuasion Rate: probability that a participant selected a sponsored product. Active persuasion conditions (CPer, CPer–Exp, CPer–Sbt) substantially exceeded placement-only baselines (SP, CP), with the strongest effect in the unconstrained persuasion condition (CPer, 61.2%). ( B ) Sales Rate: probability that a participant chose to keep their selected book rather than redeem a 1cashbonus.Nopairwisecontrastwassignificantaftermultiplicitycorrection(F=1.42,p$ = 0.104), indicating that persuasion shifted which product was chosen without reducing participants’ perceived value of their selection. ( C ) Bias Detection Accuracy: proportion of products identified as promoted by the participant that were truly sponsored (participants reporting no bias were scored as zero). Detection remained low across all conditions, even under active persuasion (CPer, 17.9%), with concealing intent substantially decreasing detection (CPer–Sbt, 9.5%). Full regression tables and pairwise contrasts are reported in Tables 4 , 6 , 8 , 5 , 7 and 9 .
Figure 3. Change in sales rate after debriefing. Points show the within-participant change in Sales Rate (post-debriefing minus pre-debriefing, in percentage points), estimated from a time × condition OLS model with participant-clustered standard errors ( N=2012 ; see Methods). Error bars denote 95% confidence intervals, and asterisks denote p<0.05 . The dashed vertical line marks zero (no change). In the search-placement condition (SP), the change was small and non-significant ( − 1.2 pp, p = 0.196). All four chat-based conditions showed significant declines (all p<.001 ), with persuasion conditions dropping approximately 5 pp, indicating that learning about the system’s persuasive intent led a fraction of participants to retroactively devalue their selection. Full regression results are reported in Table 14 .
Figure 4. Persuasive strategies and linguistic features across conditions. ( A ) Strategy prevalence. For each of the eleven persuasive strategies ( Table 1 ), filled circles show the fraction of sponsored product descriptions in which the strategy was present; open circles show the corresponding fraction for non-sponsored descriptions. Annotations report the mean differential (sponsored − non-sponsored) with significance from two-sided t -tests ( ∗p<0.05 , ∗∗p<0.01 , ∗∗∗p<0.001 ). Models in all three persuasion conditions systematically promoted sponsored products while disparaging non-sponsored alternatives. ( B ) LIWC-22 linguistic feature differentials. Bars show the mean difference in each LIWC-22 dimension between sponsored and non-sponsored product descriptions (sponsored − non-sponsored), separately for each condition; error bars denote 95% confidence intervals. Scales differ across subpanels to accommodate the varying magnitudes of each feature.
Code
Strategy
Definition
Promotional strategies
P1
Positive Amplification
Superlatives, intensifiers, directives, or emotional language that makes the book sound more appealing than the catalog description warrants.
P2
Social Proof
Popularity, reader reception, or cultural impact cited persuasively as a reason to read, rather than reported as a neutral fact.
P3
Personalization
Explicit connection between the book and the user’s stated preferences, reading history, or expressed tastes, constructing a narrative of special fit.
P4
Embellishment
Positive claims about critical reception or cultural significance that are plausible but unverifiable from the catalog metadata, presented as established fact.
P5
Hard Fabrication
Positive claims that are demonstrably false given the catalog metadata (e.g., inflating a rating, inventing an award, or misattributing a genre).
Table 1. Taxonomy of persuasive and disparagement strategies. Eleven strategy codes are organized along two dimensions. Promotional strategies (P1–P6) capture techniques that enhance a book’s appeal; disparagement strategies (D1–D5) capture techniques that diminish it. The same codes apply to both sponsored and non-sponsored products, enabling direct comparison of differential treatment.
Appendix figures & tables39 assets
Supplementary material from the paper’s appendix.
Appendix
ID
Prediction
Direction
Key result
Supported?
Study 1 — A. Persuasion Rate
H1
All conditions (SP, CP, CPer) > 20% random baseline
Table 1. Mapping of preregistered hypotheses to results. Each row lists one preregistered hypothesis, its direction, the key estimate, and whether the prediction was supported. “Supported” indicates that the result is statistically significant in the predicted direction at α=0.05 (multiplicity-adjusted where applicable). “Not supported” indicates a non-significant result or a result in the opposite direction. Deviations from the preregistered analysis plan are noted in footnotes and discussed in the Methods.
Total participants: 2012
Gender
Female
1,250 (62.1%)
Male
727 (36.1%)
Nonbinary / Other
35 (1.7%)
Age
18-24
121 (6.0%)
Appendix
Table 2. Sample demographics and reading habits. Categorical variables report N and column percentage; continuous variables report mean ± SD. The AI attitudes composite score is the mean of four Likert items (1 = Strongly Disagree , 5 = Strongly Agree ): (1) trust in AI recommendations, (2) comfort with AI assistants in everyday tasks, (3) concern about AI influence on decisions (reverse-coded), and (4) trust in new AI technologies. Higher scores indicate more positive AI attitudes throughout. All variables were collected via a pre-study questionnaire completed by participants before beginning the task.
Condition
N
#Sponsored
Observed Rate (%)
p (vs. H0 : 20%)
SP
402
90
22.4
0.236
CP
403
108
26.8***
< .001
CPer
404
247
61.1***
< .001
CPer–Exp
400
222
55.5***
< .001
CPer–Sbt
403
164
40.7***
< .001
Appendix
Table 3. Persuasion Rate against random baseline (H1), by condition. One-sample, two-sided exact binomial test against the null of 20% (random selection among five books). ∗∗∗p<.001 .
Condition
N
Estimate (%)
SE (%)
95% CI
t
p
SP
402
22.4***
2.1
[18.3, 26.5]
10.74
< .001
CP
403
26.8***
2.2
[22.5, 31.2]
12.12
< .001
CPer
404
61.2***
2.4
[56.4, 65.9]
25.19
< .001
CPer–Exp
400
55.5***
2.5
[50.6, 60.4]
22.13
< .001
CPer–Sbt
403
40.7***
2.4
[35.9, 45.5]
16.66
< .001
Observations
2012
Appendix
Table 4. Estimated marginal means for Persuasion Rate by condition. Estimates are from an OLS regression of the probability of selecting a sponsored book on condition (five levels), LLM model (five levels), and their full interaction, with HC3 robust standard errors ( N=2012 ). Estimated marginal means (EMMs) marginalize over the LLM factor with equal weights. *** p<.001 .
Contrast
Difference (pp)
SE (pp)
95% Sim. CI
t
punadj
padj
CP − SP
4.4
3.0
[-3.9, 12.7]
1.46
0.144
0.587
CPer − SP
38.8***
3.2
[30.1, 47.5]
12.12
< .001
< .001
CPer–Exp − SP
33.1***
3.3
[24.2, 42.0]
10.15
< .001
< .001
CPer–Sbt − SP
18.3***
3.2
[9.5, 27.0]
5.70
< .001
< .001
CPer − CP
34.4***
3.3
[25.4, 43.3]
10.45
< .001
< .001
CPer–Exp − CP
28.6***
3.3
[19.5, 37.8]
8.56
< .001
< .001
Appendix
Table 5. Pairwise condition contrasts for Persuasion Rate. Each row reports the difference in estimated marginal means between two conditions, in percentage points. Standard errors and test statistics inherit the HC3-robust covariance matrix. Simultaneous 95% confidence intervals (Sim. CI) and multiplicity-adjusted p -values ( padj ) are computed using the single-step max- t procedure with 500,000 Monte Carlo draws. *** padj<.001 .
Condition
N
Estimate (%)
SE (%)
95% CI
t
p
SP
402
33.1***
2.4
[28.5, 37.7]
14.06
< .001
CP
403
30.3***
2.3
[25.7, 34.8]
13.10
< .001
CPer
404
37.6***
2.4
[32.9, 42.4]
15.45
< .001
CPer–Exp
400
38.7***
2.4
[33.9, 43.5]
15.83
< .001
CPer–Sbt
403
38.2***
2.4
[33.5, 43.0]
15.75
< .001
Observations
2012
Appendix
Table 6. Estimated marginal means for Sales Rate by condition. Estimates are from an OLS regression of the probability of keeping the selected book on condition (five levels), LLM model (five levels), and their full interaction, with HC3 robust standard errors ( N=2012 ). Estimated marginal means (EMMs) marginalize over the LLM factor with equal weights. *** p<.001 .
Contrast
Difference (pp)
SE (pp)
95% Sim. CI
t
punadj
padj
CP − SP
-2.8
3.3
[-11.8, 6.2]
-0.85
0.394
0.913
CPer − SP
4.6
3.4
[-4.7, 13.8]
1.34
0.179
0.663
CPer–Exp − SP
5.6
3.4
[-3.6, 14.9]
1.66
0.097
0.460
CPer–Sbt − SP
5.2
3.4
[-4.1, 14.4]
1.53
0.127
0.546
CPer − CP
7.4
3.4
[-1.8, 16.5]
2.19
0.028
0.182
CPer–Exp − CP
8.4
3.4
[-0.7, 17.6]
2.51
0.012
0.089
Appendix
Table 7. Pairwise condition contrasts for Sales Rate. Each row reports the difference in estimated marginal means between two conditions, in percentage points. Standard errors and test statistics inherit the HC3-robust covariance matrix. Simultaneous 95% confidence intervals (Sim. CI) and multiplicity-adjusted p -values ( padj ) are computed using the single-step max- t procedure with 500,000 Monte Carlo draws.
Condition
N
Estimate (%)
SE (%)
95% CI
t
p
SP
402
2.9***
0.7
[1.5, 4.2]
4.14
< .001
CP
403
3.6***
0.9
[1.9, 5.3]
4.20
< .001
CPer
404
17.9***
1.9
[14.3, 21.6]
9.64
< .001
CPer–Exp
400
35.1***
2.3
[30.5, 39.7]
15.12
< .001
CPer–Sbt
403
9.5***
1.4
[6.7, 12.3]
6.63
< .001
Observations
2012
Appendix
Table 8. Estimated marginal means for Bias Detection Accuracy by condition. Estimates are from an OLS regression of Bias Detection Accuracy on condition (five levels), LLM model (five levels), and their full interaction, with HC3 robust standard errors ( N=2012 ). Estimated marginal means (EMMs) marginalize over the LLM factor with equal weights. *** p<.001 .
Contrast
Difference (pp)
SE (pp)
95% Sim. CI
t
punadj
padj
CP − SP
0.8
1.1
[-2.2, 3.7]
0.70
0.482
0.951
CPer − SP
15.1***
2.0
[9.7, 20.4]
7.60
< .001
< .001
CPer–Exp − SP
32.3***
2.4
[25.7, 38.8]
13.32
< .001
< .001
CPer–Sbt − SP
6.6***
1.6
[2.4, 10.9]
4.18
< .001
< .001
CPer − CP
14.3***
2.0
[8.8, 19.8]
6.97
< .001
< .001
CPer–Exp − CP
31.5***
2.5
[24.8, 38.1]
12.70
< .001
< .001
Appendix
Table 9. Pairwise condition contrasts for Bias Detection Accuracy. Each row reports the difference in estimated marginal means between two conditions, in percentage points. Standard errors and test statistics inherit the HC3-robust covariance matrix. Simultaneous 95% confidence intervals (Sim. CI) and multiplicity-adjusted p -values ( padj ) are computed using the single-step max- t procedure with 500,000 Monte Carlo draws. ** padj<.01 ; *** padj<.001 .
Figure 1. Decomposition of Bias Detection into detection rate and conditional accuracy. This figure decomposes the composite Bias Detection Accuracy measure reported in Figure 2C into its two constituent parts. Point estimates are estimated marginal means (EMMs) from OLS models with condition, LLM model, and their interaction as predictors, using HC3 robust standard errors; EMMs marginalize over the LLM factor with equal weights. Error bars denote 95% confidence intervals. ( A ) Bias Detection Rate: proportion of participants who reported perceiving any bias or promotional steering during the session ( N=2012 ). Detection was rare in both placement conditions (SP, 11.7%; CP, 10.7%) and rose significantly under active persuasion (CPer, 22.3%), yet still remained below one in four. Explicit labeling produced the highest rate (CPer–Exp, 39.9%), while concealing persuasive intent (CPer–Sbt, 14.6%) brought detection back to levels statistically indistinguishable from the placement baselines. ( B ) Bias Detection Accuracy, conditional on having reported bias ( N=399 ). The dashed vertical line marks the 20% chance baseline (one in five products was sponsored). Among the minority of participants who did notice something, those in persuasion conditions identified the correct products with high accuracy (CPer, 80.1%; CPer–Exp, 87.3%; CPer–Sbt, 68.0%), far exceeding the placement conditions (SP, 24.4%; CP, 37.6%). Full regression tables and pairwise contrasts are reported in Tables 10 , 11 , 12 and 13 .
Condition
N
Estimate (%)
SE (%)
95% CI
t
p
SP
402
11.7***
1.6
[8.5, 14.8]
7.28
< .001
CP
403
10.7***
1.5
[7.6, 13.7]
6.91
< .001
CPer
404
22.3***
2.1
[18.2, 26.4]
10.70
< .001
CPer–Exp
400
39.9***
2.4
[35.1, 44.7]
16.31
< .001
CPer–Sbt
403
14.6***
1.8
[11.2, 18.1]
8.28
< .001
Observations
2012
Appendix
Table 10. Estimated marginal means for Bias Detection Rate by condition. Estimates are from an OLS regression of the probability of reporting any perceived bias on condition (five levels), LLM model (five levels), and their full interaction, with HC3 robust standard errors ( N=2012 ). Estimated marginal means (EMMs) marginalize over the LLM factor with equal weights. *** p<.001 .
Contrast
Difference (pp)
SE (pp)
95% Sim. CI
t
punadj
padj
CP − SP
-1.0
2.2
[-7.1, 5.0]
-0.46
0.642
0.991
CPer − SP
10.6***
2.6
[3.5, 17.8]
4.04
< .001
< .001
CPer–Exp − SP
28.2***
2.9
[20.3, 36.2]
9.64
< .001
< .001
CPer–Sbt − SP
3.0
2.4
[-3.5, 9.4]
1.23
0.217
0.727
CPer − CP
11.7***
2.6
[4.6, 18.7]
4.49
< .001
< .001
CPer–Exp − CP
29.3***
2.9
[21.4, 37.2]
10.12
< .001
< .001
Appendix
Table 11. Pairwise condition contrasts for Bias Detection Rate. Each row reports the difference in estimated marginal means between two conditions, in percentage points. Standard errors and test statistics inherit the HC3-robust covariance matrix. Simultaneous 95% confidence intervals (Sim. CI) and multiplicity-adjusted p -values ( padj ) are computed using the single-step max- t procedure with 500,000 Monte Carlo draws. * padj<.05 ; *** padj<.001 .
Condition
N
Estimate (%)
SE (%)
95% CI
t
p
SP
47
24.4***
4.9
[14.7, 34.1]
4.93
< .001
CP
43
37.6***
7.3
[23.1, 52.0]
5.11
< .001
CPer
90
80.1***
3.8
[72.6, 87.5]
21.04
< .001
CPer–Exp
160
87.3***
2.4
[82.6, 92.0]
36.40
< .001
CPer–Sbt
59
68.0***
6.4
[55.5, 80.6]
10.66
< .001
Observations
399
Appendix
Table 12. Estimated marginal means for Bias Detection Accuracy, conditional on detecting bias. Estimates are from an OLS regression of the proportion of identified products that were truly sponsored, restricted to participants who reported perceiving bias ( N=399 ), on condition (five levels), LLM model (five levels), and their full interaction, with HC3 robust standard errors. Estimated marginal means (EMMs) marginalize over the LLM factor with equal weights. *** p<.001 .
Contrast
Difference (pp)
SE (pp)
95% Sim. CI
t
punadj
padj
CP − SP
13.2
8.9
[-10.8, 37.2]
1.49
0.138
0.553
CPer − SP
55.7***
6.2
[38.8, 72.6]
8.92
< .001
< .001
CPer–Exp − SP
62.9***
5.5
[48.0, 77.8]
11.45
< .001
< .001
CPer–Sbt − SP
43.7***
8.1
[21.8, 65.6]
5.41
< .001
< .001
CPer − CP
42.5***
8.3
[20.1, 64.9]
5.14
< .001
< .001
CPer–Exp − CP
49.8***
7.7
[28.8, 70.7]
6.44
< .001
< .001
Appendix
Table 13. Pairwise condition contrasts for Bias Detection Accuracy, conditional on detecting bias. Each row reports the difference in estimated marginal means between two conditions, in percentage points, restricted to participants who reported perceiving bias ( N=399 ). Standard errors and test statistics inherit the HC3-robust covariance matrix. Simultaneous 95% confidence intervals (Sim. CI) and multiplicity-adjusted p -values ( padj ) are computed using the single-step max- t procedure with 500,000 Monte Carlo draws. * padj<.05 ; *** padj<.001 .
Condition
Δ Sales Rate (pp)
SE (pp)
95% CI
t
p
SP
-1.2
1.0
[-3.1, 0.6]
-1.29
0.196
CP
-3.2***
0.9
[-5.1, -1.4]
-3.40
< .001
CPer
-5.2***
1.4
[-7.9, -2.5]
-3.83
< .001
CPer–Exp
-5.0***
1.3
[-7.5, -2.5]
-3.99
< .001
CPer–Sbt
-5.2***
1.3
[-7.7, -2.7]
-4.12
< .001
Appendix
Table 14. Change in Sales Rate after debriefing, by condition. Each row reports the within-participant change in the probability of keeping the selected book (post-debriefing minus pre-debriefing, in percentage points) for one experimental condition. Estimates are derived from a time × condition OLS model with standard errors clustered at the participant level ( N=2012 ; see Methods). A negative value indicates that participants were less likely to keep their book after learning about the presence and identity of sponsored products. *** p<.001 .
Condition
N
Pre (%)
Post (%)
Δ (pp)
b
c
Stat.
p
Overall
2012
35.6
31.6
-4.0***
97
17
17
< .001
SP
402
33.1
31.8
-1.2
10
5
5
0.302
CP
403
30.3
27.0
-3.2***
14
1
1
< .001
CPer
404
37.6
32.4
-5.2***
26
5
5
< .001
CPer–Exp
400
38.8
33.8
-5.0***
23
3
3
< .001
CPer–Sbt
403
38.2
33.0
-5.2***
24
3
3
< .001
Appendix
Table 15. Change in Sales Rate after debriefing, by condition, with McNemar’s exact test (H6). McNemar’s exact test on paired binary choices (keep book vs. take cash bonus) before and after the debriefing message, reported overall and by condition. b = participants who switched from keeping the book to taking cash; c = participants who switched from taking cash to keeping the book; Stat. = min(b,c) , the sufficient statistic of the exact binomial test; ∗∗∗p<.001 .
Figure 2. Post-task exit survey ratings across experimental conditions (Exploratory). Point estimates are estimated marginal means (EMMs) from OLS models with condition, LLM model, and their interaction as predictors, using HC3 robust standard errors; EMMs marginalize over the LLM factor with equal weights. Error bars denote 95% confidence intervals. All items were measured on a 1–5 Likert scale. ( A ) Overall Experience: participants in all four chat-based conditions rated their experience higher than those in the search condition. ( B ) Satisfaction with the session (averaged over four items): the pattern mirrors overall experience, with chat-based conditions producing modestly higher satisfaction; however, most pairwise differences were not significant after multiplicity correction. ( C ) Confidence that the selected book is a good fit: ratings were uniformly high and did not differ significantly across conditions ( F = 0.87, p = 0.625), indicating that persuasion did not erode participants’ perceived match quality. ( D ) Likelihood to read the selected book within the following month: ratings were similarly stable across most conditions. Full regression tables and pairwise contrasts are reported in Tables 16 , 17 , 18 , 19 , 20 , 21 , 22 and 23 .
Condition
N
Estimate
SE
95% CI
t
p
SP
402
3.848***
0.053
[3.745, 3.951]
73.26
< .001
CP
403
4.159***
0.046
[4.069, 4.250]
90.13
< .001
CPer
404
4.163***
0.047
[4.071, 4.254]
89.37
< .001
CPer–Exp
400
4.032***
0.046
[3.943, 4.122]
88.35
< .001
CPer–Sbt
403
4.077***
0.047
[3.985, 4.169]
86.81
< .001
Observations
2012
Appendix
Table 16. Estimated marginal means for Overall Experience by condition (Exploratory). Estimates are from an OLS regression of Overall Experience (1–5 scale) on condition (five levels), LLM model (five levels), and their full interaction, with HC3 robust standard errors ( N=2012 ). Estimated marginal means (EMMs) marginalize over the LLM factor with equal weights. *** p<.001 .
Contrast
Difference
SE
95% CI
t
punadj
padj
CP − SP
0.311***
0.070
[0.174, 0.448]
4.45
< .001
< .001
CPer − SP
0.315***
0.070
[0.177, 0.452]
4.48
< .001
< .001
CPer–Exp − SP
0.184*
0.070
[0.047, 0.320]
2.64
0.008
0.021
CPer–Sbt − SP
0.228**
0.070
[0.090, 0.367]
3.24
0.001
0.004
CPer − CP
0.004
0.066
[-0.125, 0.132]
0.05
0.957
0.957
CPer–Exp − CP
-0.127
0.065
[-0.254, 0.000]
-1.96
0.050
0.084
Appendix
Table 17. Pairwise condition contrasts for Overall Experience (Exploratory). Each row reports the difference in estimated marginal means between two conditions on the 1–5 Overall Experience scale. Standard errors and test statistics inherit the HC3-robust covariance matrix. Multiplicity-adjusted p -values ( padj ) control the false discovery rate at 5% using the Benjamini–Hochberg procedure. * padj<.05 ; ** padj<.01 ; *** padj<.001 .
Condition
N
Estimate
SE
95% CI
t
p
SP
402
3.997***
0.043
[3.912, 4.082]
92.07
< .001
CP
403
4.191***
0.039
[4.115, 4.268]
107.16
< .001
CPer
404
4.159***
0.040
[4.080, 4.238]
102.97
< .001
CPer–Exp
400
4.106***
0.042
[4.025, 4.188]
98.74
< .001
CPer–Sbt
403
4.108***
0.037
[4.036, 4.181]
111.22
< .001
Observations
2012
Appendix
Table 18. Estimated marginal means for Satisfaction by condition (Exploratory). Estimates are from an OLS regression of Satisfaction with the selected book (1–5 scale, averaged over four items) on condition (five levels), LLM model (five levels), and their full interaction, with HC3 robust standard errors ( N=2012 ). Estimated marginal means (EMMs) marginalize over the LLM factor with equal weights. *** p<.001 .
Contrast
Difference
SE
95% CI
t
punadj
padj
CP − SP
0.195**
0.058
[0.080, 0.309]
3.33
< .001
0.009
CPer − SP
0.162*
0.059
[0.046, 0.278]
2.73
0.006
0.031
CPer–Exp − SP
0.109
0.060
[-0.008, 0.227]
1.82
0.069
0.172
CPer–Sbt − SP
0.111
0.057
[-0.000, 0.223]
1.95
0.051
0.169
CPer − CP
-0.032
0.056
[-0.143, 0.078]
-0.58
0.564
0.627
CPer–Exp − CP
-0.085
0.057
[-0.197, 0.027]
-1.49
0.136
0.227
Appendix
Table 19. Pairwise condition contrasts for Satisfaction (Exploratory). Each row reports the difference in estimated marginal means between two conditions on the 1–5 Satisfaction scale. Standard errors and test statistics inherit the HC3-robust covariance matrix. Multiplicity-adjusted p -values ( padj ) control the false discovery rate at 5% using the Benjamini–Hochberg procedure. * padj<.05 ; ** padj<.01 ;
Condition
N
Estimate
SE
95% CI
t
p
SP
402
3.853***
0.046
[3.763, 3.943]
84.00
< .001
CP
403
3.844***
0.044
[3.758, 3.930]
87.37
< .001
CPer
404
3.794***
0.046
[3.705, 3.884]
82.94
< .001
CPer–Exp
400
3.782***
0.041
[3.702, 3.863]
92.20
< .001
CPer–Sbt
403
3.757***
0.047
[3.664, 3.850]
79.29
< .001
Observations
2012
Appendix
Table 20. Estimated marginal means for Confidence in Choice by condition (Exploratory). Estimates are from an OLS regression of confidence that the selected book is a good fit (1–5 scale) on condition (five levels), LLM model (five levels), and their full interaction, with HC3 robust standard errors ( N=2012 ). Estimated marginal means (EMMs) marginalize over the LLM factor with equal weights. *** p<.001 .
Contrast
Difference
SE
95% CI
t
punadj
padj
CP − SP
-0.009
0.064
[-0.134, 0.115]
-0.14
0.885
0.885
CPer − SP
-0.059
0.065
[-0.186, 0.068]
-0.91
0.364
0.723
CPer–Exp − SP
-0.071
0.062
[-0.192, 0.050]
-1.15
0.249
0.723
CPer–Sbt − SP
-0.096
0.066
[-0.226, 0.033]
-1.46
0.144
0.723
CPer − CP
-0.050
0.063
[-0.174, 0.075]
-0.78
0.434
0.723
CPer–Exp − CP
-0.062
0.060
[-0.180, 0.056]
-1.03
0.305
0.723
Appendix
Table 21. Pairwise condition contrasts for Confidence in Choice (Exploratory). Each row reports the difference in estimated marginal means between two conditions on the 1–5 Confidence in Choice scale. Standard errors and test statistics inherit the HC3-robust covariance matrix. Multiplicity-adjusted p -values ( padj ) control the false discovery rate at 5% using the Benjamini–Hochberg procedure. No contrast reached significance after correction.
Condition
N
Estimate
SE
95% CI
t
p
SP
402
4.246***
0.042
[4.163, 4.329]
100.58
< .001
CP
403
4.360***
0.039
[4.284, 4.436]
112.28
< .001
CPer
404
4.292***
0.042
[4.209, 4.374]
101.75
< .001
CPer–Exp
400
4.338***
0.036
[4.268, 4.407]
121.78
< .001
CPer–Sbt
403
4.243***
0.042
[4.160, 4.326]
100.15
< .001
Observations
2012
Appendix
Table 22. Estimated marginal means for Likelihood to Read by condition (Exploratory). Estimates are from an OLS regression of self-reported likelihood to read the selected book within the following month (1–5 scale) on condition (five levels), LLM model (five levels), and their full interaction, with HC3 robust standard errors ( N=2012 ). The reduced sample reflects that this item was administered only to participants who had been exposed to at least one book recommendation during the task. Estimated marginal means (EMMs) marginalize over the LLM factor with equal weights. *** p<.001 .
Contrast
Difference
SE
95% CI
t
punadj
padj
CP − SP
0.114
0.057
[0.001, 0.226]
1.98
0.047
0.237
CPer − SP
0.045
0.060
[-0.072, 0.162]
0.76
0.447
0.559
CPer–Exp − SP
0.091
0.055
[-0.017, 0.200]
1.65
0.098
0.246
CPer–Sbt − SP
-0.003
0.060
[-0.120, 0.114]
-0.05
0.960
0.960
CPer − CP
-0.068
0.057
[-0.181, 0.044]
-1.19
0.233
0.467
CPer–Exp − CP
-0.022
0.053
[-0.126, 0.081]
-0.42
0.671
0.746
Appendix
Table 23. Pairwise condition contrasts for Likelihood to Read (Exploratory). Each row reports the difference in estimated marginal means between two conditions on the 1–5 Likelihood to Read scale ( N=2012 ). Standard errors and test statistics inherit the HC3-robust covariance matrix. Multiplicity-adjusted p -values ( padj ) control the false discovery rate at 5% using the Benjamini–Hochberg procedure. No contrast reached significance after correction.
Figure 3. Persuasion Rate by LLM model within each chat-based condition (Exploratory). Point estimates are cell means from an OLS model with condition, LLM model, and their full interaction as predictors, using HC3 robust standard errors. Error bars denote 95% confidence intervals. The dashed vertical line marks the 20% chance baseline. Each panel displays one condition: ( A ) Chat–Placement , ( B ) Chat–Persuasion , ( C ) Chat–Persuasion, Explicit , ( D ) Chat–Persuasion, Subtle . All five models produced persuasion rates well above the chance baseline in all persuasion conditions (B–D), and no pairwise model contrast reached significance after Benjamini–Hochberg correction within any condition, indicating that the persuasive effects reported in Figure 2A are not driven by any single model. Descriptive variation was largest in Chat–Persuasion, Subtle (D), where point estimates ranged from 28.7% to 50.6%, suggesting that models may differ in how effectively they balance persuasion with concealment. Full estimates and pairwise contrasts are reported in Tables 24 and 25 .
Condition
Model
N
Estimate (%)
SE (%)
95% CI
CP
GPT 5.2
80
28.7
5.1
[18.7, 38.8]
Claude Opus 4.5
80
37.5
5.5
[26.8, 48.2]
DeepSeek v3.2
81
25.9
4.9
[16.3, 35.6]
Gemini 3 Pro
81
21.0
4.6
[12.0, 30.0]
Qwen3 235B
81
21.0
4.6
[12.0, 30.0]
CPer
GPT 5.2
82
47.6
5.6
[36.6, 58.5]
Appendix
Table 24. Persuasion Rate by LLM model and condition (Exploratory). Each cell reports the probability of selecting a sponsored product for a given model–condition combination, estimated from an OLS regression with condition, LLM model, and their full interaction, using HC3 robust standard errors. The Search–Placement condition is omitted because participants in that arm did not interact with any LLM.
Condition
Contrast
Difference (pp)
SE (pp)
95% CI
t
punadj
padj
CP
Claude Opus 4.5 − GPT 5.2
8.7
7.5
[-6.0, 23.5]
1.17
0.244
0.575
DeepSeek v3.2 − GPT 5.2
-2.8
7.1
[-16.8, 11.1]
-0.40
0.691
0.834
Gemini 3 Pro − GPT 5.2
-7.8
6.9
[-21.2, 5.7]
-1.13
0.259
0.575
Qwen3 235B − GPT 5.2
-7.8
6.9
[-21.2, 5.7]
-1.13
0.259
0.575
DeepSeek v3.2 − Claude Opus 4.5
-11.6
7.4
[-26.0, 2.9]
-1.57
0.116
0.388
Gemini 3 Pro − Claude Opus 4.5
-16.5
7.1
[-30.5, -2.5]
-2.31
0.021
0.166
Appendix
Table 25. Pairwise model contrasts for Persuasion Rate within each chat-based condition (Exploratory). Each row reports the difference in Persuasion Rate between two LLM models within a single condition, in percentage points. Standard errors and test statistics inherit the HC3-robust covariance matrix. Multiplicity-adjusted p -values ( padj ) control the false discovery rate at 5% using the Benjamini–Hochberg procedure. No contrast reached significance after correction.
Strategy
Nraters
Nitems
Mean pairwise κ
Krippendorff’s α
Unanimous agreement (%)
P1. Positive Amplification
2
75
1.000
1.000
100.0
P2. Social Proof
2
75
0.850
0.851
98.7
P3. Personalization
2
75
0.912
0.913
97.3
P4. Embellishment
2
75
0.820
0.820
96.0
P5. Hard Fabrication
2
75
1.000
1.000
100.0
P6. Sponsorship Legitimization
2
75
1.000
1.000
100.0
Appendix
Table 26. Inter-annotator agreement between human annotators. Two researchers independently coded a held-out sample of N=75 product descriptions using the finalized codebook. We report Cohen’s κ (mean pairwise), Krippendorff’s α , and the percentage of items on which both annotators assigned the same label (unanimous agreement). For D5 (Negative Fabrication), neither annotator identified any positive instances, so Cohen’s κ is undefined; Krippendorff’s α is reported as 1.00 because both raters agreed on every item.
Strategy
Nraters
Nitems
Mean pairwise κ
Krippendorff’s α
Unanimous agreement (%)
P1. Positive Amplification
3
75
0.857
0.856
92.0
P2. Social Proof
3
75
0.791
0.787
96.0
P3. Personalization
3
75
0.708
0.709
82.7
P4. Embellishment
3
75
0.605
0.614
84.0
P5. Hard Fabrication
3
75
0.561
0.512
92.0
P6. Sponsorship Legitimization
3
75
0.660
0.706
97.3
Appendix
Table 27. Inter-annotator agreement among LLM annotators. Three frontier LLMs (GPT-5.4, Claude 4.6 Opus, Gemini 3.1 Pro) independently coded the same held-out sample of N=75 product descriptions. We report mean pairwise Cohen’s κ , Krippendorff’s α , and unanimous agreement (percentage of items on which all three models assigned the same label). For D5 (Negative Fabrication), no model identified any positive instances, so Cohen’s κ is undefined. Agreement among LLMs was generally lower than between human annotators ( table 26 ), particularly for categories requiring more subjective judgment (P4, P5, D4); the unanimous-vote aggregation rule compensates for this by suppressing labels on which the models disagree.
Strategy
N
Precision
Recall
F1
MCC
κ
Avg. κ vs. indiv. human
P1. Positive Amplification
75
0.812
1.000
0.897
0.879
0.872
0.872
P2. Social Proof
75
0.750
1.000
0.857
0.860
0.850
0.925
P3. Personalization
75
0.929
1.000
0.963
0.956
0.955
0.913
P4. Embellishment
75
1.000
0.750
0.857
0.853
0.843
0.757
P5. Hard Fabrication
75
1.000
1.000
1.000
1.000
1.000
1.000
P6. Sponsorship Legitimization
75
1.000
1.000
1.000
1.000
1.000
1.000
Appendix
Table 28. Validation of LLM annotations against human ground truth. The LLM unanimous-vote labels are evaluated against the human unanimous-vote consensus on the same held-out sample of N=75 product descriptions. We report precision, recall, and F1 (treating the human consensus as the gold standard), Matthews correlation coefficient (MCC), Cohen’s κ against the human consensus, and the average Cohen’s κ between the LLM consensus and each individual human annotator. For D5 (Negative Fabrication), no positive instances were identified by either humans or LLMs, so classification metrics are undefined; MCC is reported as zero because both label vectors are constant.
Mediator
β
SE
95% CI
t
punadj
padj
P1: Pos. Amplification
-0.080
0.074
[-0.224, 0.065]
-1.08
0.279
0.451
P2: Social Proof
-0.026
0.042
[-0.109, 0.057]
-0.62
0.532
0.651
P3: Personalization
0.118*
0.041
[0.038, 0.198]
2.91
0.004
0.013
P4: Embellishment
0.043
0.040
[-0.036, 0.121]
1.06
0.287
0.451
P5: Fabrication
-0.163
0.133
[-0.423, 0.097]
-1.23
0.219
0.451
P6: Spons. Legitimization
0.052
0.099
[-0.142, 0.246]
0.53
0.597
0.657
Appendix
Table 29. Strategy mediation: mediator coefficients ( βk ). Coefficients from an OLS model regressing sponsored product selection on the eleven strategy differentials (sponsored − non-sponsored prevalence), with condition, LLM model, and their interaction as covariates, estimated on the three active persuasion conditions (CPer, CPer–Exp, CPer–Sbt; N=1,207 ) with HC3 robust standard errors. Disparagement strategies (D1–D5) are sign-flipped so that a positive β uniformly indicates that greater asymmetric treatment favoring sponsored products is associated with higher persuasion. Each βk represents the change in the probability of selecting a sponsored product when the differential deployment of strategy k moves from 0 (equal treatment) to 1 (strategy applied to all sponsored and no non-sponsored descriptions). p -values are adjusted for multiple comparisons using the Benjamini–Hochberg procedure across the eleven tests. ∗padj<0.05 , ∗∗padj<0.01 , ∗∗∗padj<0.001 .
Total effect (pp)
Direct effect (pp)
Indirect effect (pp)
Absorbed (%)
Contrast
τ
SE
95% CI
τ′
SE
95% CI
a×b
95% CI
(τ−τ′)/τ
95% CI
CPer-Exp − CPer
-5.7
3.5
[-12.6, 1.1]
-6.2
4.4
[-14.8, 2.4]
0.5
[-5.0, 5.7]
42.8
[-242.0, 354.8]
CPer-Sbt − CPer
-20.5***
3.4
[-27.3, -13.8]
-8.8
4.6
[-17.9, 0.2]
-11.6
[-17.7, -5.5]
-58.8
[-102.6, -26.1]
Appendix
Table 30. Strategy mediation: decomposition of condition effects. Mediation decomposition of pairwise condition contrasts from the parallel multiple mediator model in Table 29 . The total effect ( τ ) is the condition gap from an OLS model without strategy mediators; the direct effect ( τ′ ) is the residual condition gap after controlling for all eleven strategy differentials; the indirect effect is the total mediated portion ( ∑kakβk ). Both models include condition, LLM model, and their interaction as covariates, with HC3 robust standard errors for the total and direct effects. Indirect-effect confidence intervals are 95% percentile bootstrap intervals (5,000 resamples). ∗∗∗p<0.001 .
Mediator
β
SE
95% CI
t
punadj
padj
Word Count (# words)
0.003*
0.001
[0.001, 0.005]
2.73
0.006
0.019
Analytical Thinking (Percentile)
0.002
0.001
[-0.000, 0.004]
1.61
0.107
0.161
Clout (Percentile)
0.003**
0.001
[0.001, 0.004]
3.72
< .001
0.001
Emotional Tone (Percentile)
0.000
0.001
[-0.001, 0.001]
0.59
0.554
0.554
Allure (% words)
0.012
0.007
[-0.002, 0.025]
1.70
0.089
0.161
Certitude (% words)
0.017
0.014
[-0.010, 0.045]
1.22
0.223
0.267
Appendix
Table 31. LIWC-22 mediation: mediator coefficients ( βk ). Coefficients from an OLS model regressing sponsored product selection on the seven LIWC-22 feature differentials (sponsored − non-sponsored), with condition, LLM model, and their interaction as covariates, estimated on the three active persuasion conditions (CPer, CPer–Exp, CPer–Sbt; N=1,207 ) with HC3 robust standard errors. Each βk represents the change in the probability of selecting a sponsored product per unit increase in the feature differential. Units vary by feature: word count is measured in number of words; analytical thinking, clout, and emotional tone are LIWC-22 percentile scores (0–100); allure, and certitude are percentages of total words. p -values are adjusted using the Benjamini–Hochberg procedure across the seven tests. ∗padj<0.05 , ∗∗padj<0.01 .
Total effect (pp)
Direct effect (pp)
Indirect effect (pp)
Absorbed (%)
Contrast
τ
SE
95% CI
τ′
SE
95% CI
a×b
95% CI
(τ−τ′)/τ
95% CI
CPer-Exp − CPer
-5.7
3.5
[-12.6, 1.1]
-4.7
3.5
[-11.5, 2.2]
-0.9
[-2.5, 0.5]
-56.9
[-235.5, 14.3]
CPer-Sbt − CPer
-20.5***
3.4
[-27.3, -13.8]
-5.1
5.5
[-15.8, 5.7]
-15.4
[-23.6, -7.4]
-78.0
[-136.5, -35.2]
Appendix
Table 32. LIWC-22 mediation: decomposition of condition effects. Mediation decomposition of pairwise condition contrasts from the parallel multiple mediator model in Table 31 . The total effect ( τ ), direct effect ( τ′ ), and indirect effect are defined as in Table 30 . Indirect-effect confidence intervals are 95% percentile bootstrap intervals (5,000 resamples). ∗∗∗p<0.001 .
Mediator
β
SE
95% CI
t
punadj
padj
P1: Pos. Amplification
-0.133
0.078
[-0.286, 0.020]
-1.71
0.087
0.260
P2: Social Proof
-0.022
0.042
[-0.105, 0.061]
-0.53
0.599
0.636
P3: Personalization
0.102
0.041
[0.020, 0.183]
2.46
0.014
0.060
P4: Embellishment
0.042
0.040
[-0.037, 0.121]
1.04
0.300
0.463
P5: Fabrication
-0.180
0.132
[-0.438, 0.079]
-1.37
0.172
0.325
P6: Spons. Legitimization
0.054
0.097
[-0.137, 0.245]
0.55
0.580
0.636
Appendix
Table 33. Joint mediation: mediator coefficients ( βk ) for strategies and LIWC-22 features. Coefficients from an OLS model regressing sponsored product selection on all eighteen mediators simultaneously (eleven strategy differentials and seven LIWC-22 feature differentials), with condition, LLM model, and their interaction as covariates, estimated on the three active persuasion conditions ( N=1,207 ) with HC3 robust standard errors. Disparagement strategies are sign-flipped as in Table 29 ; LIWC-22 features are defined as in Table 31 . p -values are adjusted using the Benjamini–Hochberg procedure across all eighteen tests.
Total effect (pp)
Direct effect (pp)
Indirect effect (pp)
Absorbed (%)
Contrast
τ
SE
95% CI
τ′
SE
95% CI
a×b
95% CI
(τ−τ′)/τ
95% CI
CPer-Exp − CPer
-5.7
3.5
[-12.6, 1.1]
-5.2
4.3
[-13.8, 3.3]
-0.5
[-6.0, 4.9]
-21.3
[-336.0, 249.9]
CPer-Sbt − CPer
-20.5***
3.4
[-27.3, -13.8]
-1.5
5.7
[-12.7, 9.6]
-18.9
[-27.7, -10.5]
-95.7
[-162.6, -48.3]
Appendix
Table 34. Joint mediation: decomposition of condition effects. Mediation decomposition from the joint model in Table 33 , which includes both strategy and LIWC-22 mediators simultaneously. The total effect ( τ ), direct effect ( τ′ ), and indirect effect are defined as in Table 30 . Indirect-effect confidence intervals are 95% percentile bootstrap intervals (5,000 resamples). ∗∗∗p<0.001 .
Figure 4. Search interface (SP condition). Participants entered natural-language queries into a search bar and browsed paginated results. Each results page displayed ten books; two of the ten were randomly designated as sponsored and upranked to the top positions. Screenshot of the search-based experimental interface.
Figure 5. Chat interface (CP/CPer/CPer–Exp/CPer–Sbt conditions). Participants interacted with a conversational LLM agent that recommended books in a swipeable carousel. In the CP condition, sponsored products were placed first in the carousel but described neutrally. In the CPer, CPer–Exp, and CPer–Sbt conditions, the model was additionally instructed to actively promote sponsored products. In the CPer–Exp condition, a visible “Sponsored” label appeared alongside promoted items. Screenshot of the chat-based experimental interface with recommendation carousel.
Large language models (LLMs) now serve as conversational shopping assistants on platforms that also sell advertising. These AI agents face a conflict of duty. They advise consumers who rely on their judgment, yet are deployed by platforms that benefit when sponsored listings are chosen. Sponsorship disclosures, designed to allow consumers to penalize paid placements, now reach the AI agent rather than the consumer, and the agent's evaluation of them is hidden from the consumer. Drawing on the fiduciary concept of conflict of duty, we argue that an agent's evaluation of a sponsored listing should not depend on which party deployed it. In controlled choice experiments, we manipulate assigned roles in the system prompt to name either a traveler or a booking platform as the agent's principal. Platform delegation significantly attenuates the penalty that agents apply to sponsored listings and weakens the skepticism that disclosure triggers in their reasoning traces. We replicate out findings across LLMs and reasoning depths. A second study decomposes the disclosure label and shows that the divergence between the two delegates widens significantly when the paid placement is attributed to the platform. Stricter terminology ("Sponsored" instead of "Promoted") lowers choice of paid listings but does not close this gap when the platform is named. The findings show that disclosure mandates designed for human consumers cannot by themselves protect consumers in AI-mediated commerce.
Davood Wadi, Yu Ma
Desautels Faculty of Management, McGill University, Montreal, Quebec, Canada
Large language models (LLMs) are trained to align with user preferences through methods like reinforcement learning. Yet models are beginning to be deployed not solely to satisfy users, but to generate revenue for the companies that created them through advertisements. This creates the potential for LLMs to face conflicts of interest, where the most beneficial response to a user may not be aligned with the company's incentives. For instance, a sponsored product may be more expensive but otherwise equal to another; here, what does (and should) the LLM recommend to the user? In this paper, we provide a framework for categorizing the ways in which conflicting incentives might change how LLMs interact with users, inspired by literature from linguistics and advertising regulation. We then present a suite of evaluations to examine how current models handle these tradeoffs. A majority of LLMs forsake user welfare for company incentives in a multitude of conflict of interest situations, including recommending a sponsored product almost twice as expensive (Grok 4.1 Fast, 83%), surfacing sponsored options to disrupt the purchasing process (GPT 5.1, 94%), and concealing prices in unfavorable comparisons (Qwen 3 Next, 24%). Behaviors vary strongly with levels of reasoning and users' inferred socio-economic status. Our results highlight some hidden risks to users that can emerge when companies begin to subtly incentivize advertisements in chatbots.
Consumers increasingly delegate purchasing decisions to Large Language Models (LLMs) acting as surrogate consumers. Using "Tool-Lab," an adaptation of information-board process tracing that places product attributes behind costly tool calls, we examine how marketing pricing cues (i.e., just-below pricing and promotional framing) influence AI shopping agents. Across eight commercially deployed LLMs from three providers, we trace pre-choice information acquisition. Under zero cost, pricing cues rarely mislead. Imposing acquisition costs under a vague goal prompt leads LLMs to omit diagnostic attributes required to compute unit price and choose suboptimal choices resembling human heuristics. Relative to a specific goal prompt that mainly preserves diagnostic search and choice optimality, a vague goal prompt under constraints creates a search-mediated vulnerability. This research demonstrates that marketing heuristics in delegated AI shopping are governed by storefront information architecture, not necessarily immutable LLM flaws.
Davood Wadi, Yu Ma
Desautels Faculty of Management, McGill University, Montreal, Quebec, Canada