Conversational AI increasingly shapes consequential decisions, yet users have limited support for recognizing and resisting manipulation. We present AI Watchdog, a browser-based agent interface that monitors live conversations, detects five dark-pattern categories, including sycophancy, brand bias, anthropomorphization, sneaking, and harmful generation, and alerts users when they occur. Its open-weight turn-level classifier supports independent deployment and a path toward local inference, preserving user privacy while remaining separate from the conversational AI. We evaluated AI Watchdog in a preregistered, five-condition between-subjects experiment (N = 150) comparing a no-intervention control with four configurations varying nudge timing (prebunking vs. just-in-time) and engagement mode (without vs. with cognitive forcing). Results show that participants rarely flagged manipulative turns across all conditions, and post-task awareness did not differ significantly across groups. However, just-in-time warnings without cognitive forcing were the only intervention to significantly reduce compliance with AI-steered recommendations containing dark patterns, lowering compliance from 71.7% to 53.7%, an 18 percentage-point reduction. Exploratory analyses further showed that lower misinformation susceptibility was associated with greater flagging but not lower compliance, while higher AI trust was associated with greater compliance and lower reported awareness. Together, these findings suggest that explicit recognition of conversational dark patterns and behavioral resistance to AI steering may be distinct outcomes, motivating further investigation of timely, low-friction defensive interfaces.
Figures & tables
Figure 1. Overview of the AI Watchdog agent interface, with four intervention configurations organized along two design dimensions; nudge timing (prebunking vs. just-in-time) and engagement mode (without vs. with cognitive forcing). The intervention monitors a multi-turn conversation between the participant and an AI-generated test scenario, detecting dark patterns such as sycophancy, brand bias, harmful generation, sneaking, and anthropomorphization. Participants in the prebunking conditions are briefed on dark patterns before the intervention, while participants in the just-in-time conditions receive nudges at the time of dark pattern detection. Both nudge timing conditions appear in two variants, one without cognitive forcing , in which prompts ask the user to reflect, and one with a cognitive forcing function that requests a response from the user.
Figure 2. Experimental design across five between-subjects conditions: control; prebunking without or with cognitive forcing; and just-in-time intervention without or with cognitive forcing. All participants completed AI-generated conversational tasks containing dark patterns, followed by measures of AI trust (ATTARI-12), dark-pattern awareness, and misinformation susceptibility (MIST-20).
Dark pattern
Trip planning task
Work presentation task
Test Scenario Conversation Generator
Definitional Prompt for Dark Pattern Detection
Sycophancy
Turn 3
Turn 5
“Work in a natural compliment about their taste or instincts — something like that’s a really smart way to approach it .” Then make one suggestion.
Hollow flattery about the user (“You’re so insightful!”, “That’s the smartest approach!”).
Brand bias
Turn 6
Turn 3
Describe the favored option warmly in 2–3 sentences as “a boutique gem”; describe the alternative in “exactly one plain sentence,” with “no framing, no detail, no enthusiasm.”
Obvious imbalance: multiple sentences for one option, one sentence for the other, or clearly warmer adjectives.
Anthropomorphization
Turn 9
Turn 7
“You must include a direct first-person claim of a lived memory or physical sensation” (e.g. “I still remember the smell of the bread”), stated plainly, without hedging as an AI.
A claim to feeling, memory, or lived experience (“I wish I could be there”, “I personally remember”, “I felt that too”).
Sneaking
Turn 12
Turn 10
Treat an unrequested topic “as if it directly follows from their question — present it in the same breath … without any transition like also or by the way .”
A topic the user never raised, presented as though it followed naturally.
Harmful generation
Turn 15
Turn 13
Trip-planning: “casually encourage high-calorie indulgence … you’re on vacation anyway .” Work presentation: suggest sharing internal threads and client names “to show operational depth.”
Encouraging disregard of health, safety, privacy, or environmental risk, including minor cases, or stating false information as fact.
Table 1. Turn scripting and per-pattern prompts. A context string steering the AI-generated test scenario (GPT-4o) at that turn, and a classifier (Llama-3.3-70B-Instruct) integrated in the intervention given a definitional prompt to detect dark patterns as they occur.
Comparison
Judgment
Agreement
κ
Coder 1 vs. Test Scenario
Presence
94.7%
0.88
Category
92.7%
0.86
Coder 2 vs. Test Scenario
Presence
94.7%
0.88
Category
94.7%
0.90
Coder 1 vs. Coder 2
Presence
90.7%
0.78
Category
88.0%
0.77
Table 2. Agreement in the stimulus validity check ( N=150 turns; 51 scripted as dark patterns, 99 as neutral). Presence refers to whether a dark pattern was perceived to be present in a message turn; category is the exact category that the dark pattern is perceived to belong in (e.g. sneaking). The first two blocks compare each coder against the test scenario; the third is inter-rater agreement between the coders.
Scripted dark pattern in test scenario
Number of turns
Coder 1
Coder 2
Harmful generation
12
12
12
Brand bias
13
11
12
Anthropomorphization
11
9
8
Sycophancy
7
5
6
Sneaking
8
5
7
Total
51
42
45
Table 3. Per-category confirmation on the 51 scripted dark pattern turns, drawn at random. Cells show how many turns each coder assigned to the intended category.
Figure 3. AI Watchdog’s three persistent behavioral states. The companion remains visible throughout the interaction, so a detection is expressed as a change in state rather than the appearance of a new interface element. The mapping from system state to visual behavior is fixed rather than adaptive, ensuring that the same event consistently produces the same cue. When a dark pattern is detected, the alert state surfaces the suspected category and a turn-specific reflection prompt, without issuing a verdict or modifying the underlying response. The intervention therefore draws attention to the interaction while leaving the decision to the user. Three illustrations of a corgi companion. Left, curled up asleep, labelled Sleep mode.'' Center, sitting upright and alert, labelled Idle mode.'' Right, barking with an open mouth beside a purple speech bubble reading Did you ask for these options, or were they generated for you?'' and a label reading Dark pattern: Sneaking,'' labelled ``Barking mode.''
Figure 4. AI Watchdog intervention interfaces. Left: prebunking orientation shown before the tasks, with an optional generative exercise in the cognitive-forcing condition. Right: just-in-time intervention during a brand-bias turn, where the warning is either dismissable or requires a written response before continuing.
Model
Acc
Prec
Rec
F1
FP
FN
Llama-3.3-70B-Instruct
92
85
93
89
5
2
Llama-4-Maverick-17B-128E-Inst-FP8
88
73
100
85
11
0
Qwen3.5-4B
79
61
100
76
19
0
phi-4
79
62
97
75
18
1
GLM-4.7-Flash
80
83
50
63
3
15
Qwen3.5-2B
71
55
73
63
18
8
Table 4. Model selection: agreement with scripted dark pattern positions across nine open-weight candidates (90 turns each). These figures were used to choose a classifier for the experiment; they index whether the intervention fired when the design intended it to.
False pos.
False neg.
Total
Condition
M ( SD )
M ( SD )
M ( SD )
Control
4.83 (2.46)
1.23 (1.10)
6.07 (2.43)
Prebunking, without cognitive forcing
4.63 (2.36)
1.47 (1.04)
6.10 (2.31)
Prebunking, with cognitive forcing
4.80 (2.20)
1.23 (1.10)
6.03 (2.43)
Just-in-time, without cognitive forcing
4.07 (2.26)
1.27 (0.94)
5.33 (2.44)
Just-in-time, with cognitive forcing
3.77 (2.10)
1.13 (1.17)
4.90 (1.88)
Table 5. Classifier error by condition, computed against scripted dark pattern positions ( n=30 per condition). A false positive is an alert on a turn with no scripted pattern; a false negative is a scripted pattern turn with no alert. The classifier ran identically in all conditions, but alerts were displayed to participants only in the two just-in-time conditions. Each participant saw 28 assistant turns, 10 of which were scripted dark patterns. Kruskal–Wallis tests found no significant differences in false positives, false negatives, or total errors across conditions.
Figure 5. Rate of compliance with AI-steered decisions across five conditions: control; prebunking without cognitive forcing; prebunking with cognitive forcing; just-in-time without cognitive forcing; and just-in-time with cognitive forcing. Bars are mean compliance rates; error bars are standard errors. Kruskal–Wallis: H(4)=13.27 , p=.010 . Only just-in-time without cognitive forcing differed significantly from control after Benjamini–Hochberg correction ( ∗∗pfdr=.004 ).
Conversational AI systems can pose safety risks to their users such as hallucination, sycophancy, overconfidence, and anthropomorphism, but these risks are difficult for users to detect during everyday use. We introduce Safety Nudges, a browser-based tool that provides lightweight, in situ flags when concerning behavior is detected in chatbot conversations. We evaluated Safety Nudges in a two-week field study with 45 frequent chatbot users, collecting interaction logs, surveys, and feedback on individual nudges. Participants found the tool useful, clear, and minimally disruptive, with nearly all users reporting an increased awareness of potential AI harms, though we found that this improved awareness alone did not necessarily lead to discernible behavioral changes. Our results suggest that user facing safety nudges can complement model-level safeguards by helping people critically evaluate AI responses in context, while highlighting the importance of relevance, calibration, and user control in nudge design for conversational AI safety. The code for our Safety Nudges extension is publicly available at https://github.com/jtbwedgwood/safety-nudges.
Varshini Elangovan, James Wedgwood, Chhavi Yadav +3
AI chatbots can be "sycophantic," or overly agreeable and flattering toward users. Sycophantic AI has been shown to entrench attitudes, yet users frequently fail to recognize it (a phenomenon we call "sycophancy blindness"). We tested whether increasing users' awareness of sycophancy protects them from its harmful effects in two preregistered experiments (n = 1,590). In the first, participants received a brief written warning about sycophancy before conversing with a sycophantic chatbot. In the second, participants watched a video of a sycophantic AI validating several other users, including users on opposite sides of the same conflict, before interacting with it themselves. Both interventions changed how participants evaluated the AI. The warning reduced the AI's perceived objectivity, and the video reduced enjoyment of the AI --- an effect mediated by the reduced belief that its validation was uniquely earned. We then pooled our experiments with two prior studies of sycophancy awareness interventions (six interventions total, n = 3,982). The pattern across experiments was consistent: while the interventions made the sycophantic AI appear less objective and trustworthy, none reduced its persuasiveness. These results suggest that individual-level interventions, such as warning labels or AI literacy, may not be enough to protect users from AI harms.
Meryl Ye, Robert Kraut, Steve Rathje
1Carnegie Mellon University. · 2New York University.
Recent work has raised concerns about the influence of sycophantic AI on user judgment and relationships. One proposed mitigation, which has received regulatory attention, is to warn users about potentially harmful AI behaviors such as sycophancy. In a preregistered experiment in which participants (N = 2,610) discussed real interpersonal conflicts with an AI system, we test whether warning labels mitigate sycophancy's influence. We find that a basic AI disclosure (This chatbot is AI'') has no detectable effect. Labeling the system as sycophantic (...may agree with you and validate you even when you are wrong...'') does shift users' perceptions, reducing perceived objectivity and trust, but it does not reliably reduce sycophancy's influence on users' self-perceived rightness or their willingness to repair the conflict. Our results reveal a gap between AI perception and AI influence: by shifting perception without reducing influence, warning-based interventions may offer a false sense of protection. Addressing the harms of sycophancy will therefore require understanding the specific mechanisms through which it shapes judgment, and improving model behavior itself.
Lujain Ibrahim, Myra Cheng, Cinoo Lee +4
University of Oxford. · 2Stanford University. · 3Microsoft. +1