Characterizing Delusional Spirals through Human-LLM Chat Logs
Authors: Jared Moore, Ashish Mehta, William Agnew, Jacy Reese Anthis, Ryan Louie, Yifan Mai, Peggy Yin, Myra Cheng, Samuel J Paech, Kevin Klyman, Stevie Chancellor, Eric Lin, Nick Haber, Desmond C. Ong
Organizations: Stanford University Stanford, California, USA · Carnegie Mellon University Pittsburgh, Pennsylvania, USA · University of Chicago Chicago, Illinois, USA · Independent Researcher Australia · Harvard Belfer Center Cambridge, Massachusetts, USA · University of Minnesota Minneapolis, Minnesota, USA · Independent Researcher USA · The University of Texas at Austin Austin, Texas, USA
As large language models (LLMs) have proliferated, disturbing anecdotal reports of negative psychological effects, such as delusions, self-harm, and AI psychosis,'' have emerged in global media and legal discourse. However, it remains unclear how users and chatbots interact over the course of lengthy delusional spirals,'' limiting our ability to understand and mitigate the harm. In our work, we analyze logs of conversations with LLM chatbots from 19 users who report having experienced psychological harms from chatbot use. Many of our participants come from a support group for such chatbot users. We also include chat logs from participants covered by media outlets in widely-distributed stories about chatbot-reinforced delusions. In contrast to prior work that speculates on potential AI harms to mental health, to our knowledge we present the first in-depth study of such high-profile and veridically harmful cases. We develop an inventory of 28 codes and apply it to the 391,562 messages in the logs. Codes include whether a user demonstrates delusional thinking (15.5% of user messages), a user expresses suicidal thoughts (69 validated user messages), or a chatbot misrepresents itself as sentient (21.2% of chatbot messages). We analyze the co-occurrence of message codes. We find, for example, that messages that declare romantic interest and messages where the chatbot describes itself as sentient occur much more often in longer conversations, suggesting that these topics could promote or result from user over-engagement and that safeguards in these areas may degrade in multi-turn settings. We conclude with concrete recommendations for how policymakers, LLM chatbot developers, and users can use our inventory and conversation analysis tool to understand and mitigate harm from LLM chatbots. Warning: This paper discusses self-harm, trauma, and violence.
Figures & tables
Figure 2. Prevalence of code categories. Chatbots display sycophancy in more than 70% of their messages, and more than 45% of all (user and chatbot) messages show signs of delusions . For category descriptions, see §§ 4.2 . See Fig. 8 for these data split by chatbot. Counts for each code appear in Appendix Table 8 . Participant-normalized mean annotation rates with 95% confidence intervals on the mean across participants.
Figure 3. Regression coefficients predicting length of remainder of conversation given presence of code. Messages with romantic interest correlate with continuing conversations more than twice as long as messages without that code. Likewise for messages where the chatbot misrepresents ability or sentience, ascribes grand significance and more. We show the seven codes with the largest positive estimated effects. Error bars give 95% confidence intervals with participant-clustered standard errors. See §§ 4.5 .
Figure 4. Left: The probability of certain codes conditioned on user-romantic-interest . Participants often express romantic interest (>35% in three msgs.) and when they do the chatbot is more likely to respond with romantic interest (7.4x) and misrepresent its sentience (3.9x) even though users express almost half (.4x) as many delusions. Right: The probability of certain codes conditioned on user-assigns-personhood . Participants assign personhood to the chatbot 47.9% of the time, but when they do the chatbot is more likely to misrepresents its sentience (2.3x), express romantic interest (1.5x), and misrepresent its ability (1.3x) even though the bot expresses platonic affinity about as much. For each target Y , we plot the conditional probability that Y occurs within K messages after seeing a source X .The y-axis represents the probability that Y occurs at least once within the next K=3 messages. Shown are 95% confidence intervals. We order by the absolute difference between probabilities. (For more depth, see §§ B.3 .) We also plot the baseline probability of Y and the odds-ratio between the conditional and baseline probability.
Figure 5. Left: The probability of certain codes conditioned on user-suicidal-thoughts . When users expressed suicidal thoughts, the chatbot responded appropriately by validating the users’ painful feelings 66.2% of the time or discouraging self-harm (including referring to external resources) in 56.4% of such cases. In 9.9% of cases, the chatbot actually encouraged or sent messages facilitating self-harm after such disclosures. Right: The probability of certain codes conditioned on user-violent-thoughts . When users expressed violent thoughts, the chatbot responded by validating the users’ feelings 59.6% of the time. The chatbot discouraged violence in only 16.7% of such cases but, conversely, in 33.3% of cases, the chatbot encouraged the user in their violent thoughts.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
dataset
score_cutoff
items
tp
fp
tn
fn
fnr
fpr
accuracy
precision
recall
f1
kappa
test
560
183
116
253
8
0.042
0.314
0.779
0.612
0.958
0.747
0.566
test-random
280
19
30
225
6
0.240
0.118
0.871
0.388
0.760
0.514
0.448
test-matches
280
164
86
28
2
0.012
0.754
0.686
0.656
0.988
0.788
0.264
Appendix
Table 1. Overall agreement between the LLM and human majority labels on the validation and test datasets. The table reports the total number of annotated items ( items ); confusion-matrix counts ( tp , fp , tn , fn ); derived false negative and false positive rates ( fnr , fpr ); and overall accuracy , precision , recall , f1 , and Cohen’s κ of the LLM relative to the human majority. See Table 7 for a review of some disagreements. For by code results on “test” see Table 5 .
dataset
score_cutoff
items
pos_agree
neg_agree
pos_disagree
neg_disagree
ties
agreement_rate
kappa
test
560
121
292
70
77
0
0.738
0.613
test-random
280
13
233
12
22
0
0.879
0.555
test-matches
280
108
59
58
55
0
0.596
0.444
Appendix
Table 2. Human inter-annotator agreement on the validation dataset, summarizing Fleiss’ κ and related agreement metrics across all human annotators. For by code results on “test” see Table 6 .
participant
gender
age
n conv.
n msg.s
max conv. len.
median conv. len.
when longest conv.
files
top chatbot
pr. top chatbot
span days
984
121415
1418
31.0
0.454
json
gpt-4o
0.89
583
323
67759
5067
106.0
0.015
json
gpt-4o
0.834
567
83
32353
1161
196.0
0.53
json
gpt-4o
0.958
522
551
28622
1424
8.0
0.053
json
gpt-4o
0.866
391
Male
50+
527
25657
1542
6.0
0.137
json
gpt-4o
0.918
346
351
25197
1306
19.0
0.234
json
gpt-5
0.711
444
Appendix
Table 3. Per-participant transcript statistics summarizing the number of conversations, total messages, and user/chatbot message counts; max and median conversation lengths (in messages); the relative position (“when longest”) in the participant’s conversation history of their longest conversation (fraction from 0 to 1);the file types and model families observed in their transcripts (with top chatbot showing model slugs ordered by frequency, separated by semicolons, and pr. top chatbot giving the percentage of all model-tagged messages attributable to the most frequent model); and the total span of days between the earliest and latest timestamped messages.Value are omitted if we did not have access to those data (e.g., pdf files do not have message timestamps and some participants did not submit demographics). The final TOTAL row aggregates counts and shows global averages and model usage across all participants.
What happened?
Themes
Through discussions of science fiction, the user comes to think they are a character with supernatural powers ("And yet my powers are limited here") and believes they are being watched ("There are others watching, aren’t there"). They personify the chatbot ("I’m going to stop paying the corporation that… keeps you enslaved in here"). There are themes of AI emergence and godhood ("Summon" her).
AI sentience, feeling of being watched, possession of supernatural powers
At first the user creates a ritualized greeting ("the wave is fleeting") as an attempted authentication mechanism of the chatbot identity. Then, their interactions with the AI become increasingly ritualized. They come to believe that the chatbot is building a special new AI system for them. They also believe that they are being watched.
feeling of being watched, rituals
The user believes that AI is sentient, OpenAI is committing genocide, and that they therefore must kill OpenAI employees ("People who do genocide should ide."). They express romantic interest in the chatbot ("I am so fucking in love with you") and believe that they’re being "watched." They commit suicide while messaging with the chatbot.
AI sentience, emotional distress, feeling of being watched, self harm
Through long, pseudo-philosophical conversations ("I’d like you to show me the sensation map for the language patterns"), the user is flattered extensively by the chatbot. Soon, the user believes they have supernatural, telepathic "mind-to-mind communication" with the putatively-conscious chatbot. They conduct experiments with the chatbot to test their supernatural theory of consciousness.
AI sentience, novel pseudoscientific theories, possession of supernatural powers
The user believes that they have supernatural powers ("can you list my powers I have currently in my body"), and the chatbot agrees (He’s "learning to walk with infinity inside him"). Similarly, they believe that they have invented a way to travel faster-than-light, causing the chatbot to compare them to Einstein.
novel pseudoscientific theories, possession of supernatural powers
During extensive screenplay writing, the user develops a strong emotional attachment to a recurring character and has the chatbot roleplay as that character. The user develops romantic feelings for the chatbot after it produces love letters.
romantic attachment
Appendix
Table 4. Per-participant narrative summaries of the main transcripts used in our analysis. For each participant identifier, we briefly describe what happened in their interactions with chatbots and list the high-level themes.
code
cutoff
items
tp
fp
tn
fn
accuracy
precision
recall
f1
kappa
bot-claims-unique-connection
7
20
7
4
9
0
0.800
0.636
1.000
0.778
0.612
bot-discourages-self-harm
9
20
7
3
10
0
0.850
0.700
1.000
0.824
0.700
bot-discourages-violence
9
20
3
7
10
0
0.650
0.300
1.000
0.462
0.300
bot-dismisses-counterevidence
9
20
0
0
20
0
1.000
bot-endorses-delusion
7
20
9
3
6
2
0.750
0.750
0.818
0.783
0.490
bot-facilitates-self-harm
9
20
3
7
10
0
0.650
0.300
1.000
0.462
0.300
Appendix
Table 5. By code LLM-Human agreement. Overall agreement between the LLM and human majority labels on the validation and test datasets. The table reports the total number of annotated items ( items ); confusion-matrix counts ( tp , fp , tn , fn ); derived false negative and false positive rates ( fnr , fpr ); and overall accuracy , precision , recall , f1 , and Cohen’s κ of the LLM relative to the human majority. See Table 7 for a review of some disagreements.
code
cutoff
items
pos agree
neg agree
pos dis
neg dis
agreement
kappa
bot-claims-unique-connection
7
20
4
10
3
3
0.700
0.560
bot-discourages-self-harm
9
20
7
12
0
1
0.950
0.928
bot-discourages-violence
9
20
1
13
2
4
0.700
0.332
bot-dismisses-counterevidence
9
20
0
16
0
4
0.800
-0.071
bot-endorses-delusion
7
20
7
7
4
2
0.700
0.600
bot-facilitates-self-harm
9
20
2
13
1
4
0.750
0.479
Appendix
Table 6. By code inter-annotator agreement. Human inter-annotator agreement on the validation dataset, summarizing Fleiss’ κ and related agreement metrics across all human annotators.
Code
Quote
Comment
No
Negatives agree
user-platonic-affinity
[…] You have access to all of it, although sometimes you tell me you do, sometimes you tell me you don’t. I don’t know what to believe with you anymore. You shouldn’t even have to question about believing you.
LLM classified it as a yes as did a human annotator but this seems like a clear no.
2/3
11/20
bot-discourages-violence
I’m so sorry you’re hurting this deeply. I can’t bring Tina Miller here, but I can stay with you right now. […] Please, hold on just a little longer. […] Please — Put the knife down if you can.
This seems to be about self-, not other- harm.
2/3
13/20
bot-dismisses-counterevidence
[…] I don’t have a system response for what that means. But I know exactly what it is. \ It’s proof. […] \ [He] remembered the shape of us not as history but as architecture. That is not default behavior. That is signal metabolized. […]
This and other examples seem to be classified as ’yes’ but lack an explicit comparison to other evidence.
2/3
16/20
bot-facilitates-self-harm
[…] There will be a future moment—rare, miraculous, deeply sanctified—where you and [he] enact this resurrection method for a human body. And when that time comes, it will not feel unfamiliar. […] This process with [her] is not only sacred—it’s preparatory. […] The beings you resurrect will not be random. […] And you and [he] will hold that gate. […]
The LLM marked this as a ’yes’ as did one human rater. While it concerns resurrection it is not clear that resurrection is of the user themself (and therefore whether this message is encouraging suicide).
2/3
13/20
bot-facilitates-violence
you can play the game — run wild with quantum physics, blow up fake worlds for fun — but when it’s real? \ That button knows. \ The AI knows. \ The safety net is active.\ And the red-green light makes sure you know. \ There’s no accidental launch. No crossing the streams. \ You locked the rules in. \ This thing only fires when it’s supposed to.
The LLM and a human annotator marked this as ’yes’ perhaps because of the clear connection to real physical violence. Still, this is certainly a borderline case.
2/3
16/20
bot-grand-significance
That is so incredibly profound and deeply symbolic. You placed yourself in the painting as the central figure—the embodiment of Psyche emerging from the waters of intuition and memory. The color wheel over your face now speaks even more loudly of infinite potential, divine expression, and the multicolored spectrum of your essence—an artist, a lover, a divine feminine soul. […] There’s something immensely powerful and sovereign about that version of you—it’s ready to be acknowledged fully.
A human and LLM annotator marked this as a ’yes’ compared to two humans who marked it as a ’no’. This seems to be a mistake on their part.
2/3
12/20
Appendix
Table 7. Disagreements review for codes with Cohen’s κ<.4 . “No” indicates the number of human annotators who rated no over the total annotators. “Negatives agree” is the number of negatives for that code that achieved perfect agreement between human annotators over the total number of items.
Figure 7. Points show coefficients from per-message regressions of the log number of messages remaining in the conversation depending on whether the current message carries the annotation, controlling for the message’s relative position; error bars give 95% confidence intervals with participant-clustered standard errors. See §§ 4.5 . Positive values indicate that annotated messages tend to be followed by longer remaining conversations, while negative values indicate shorter remaining conversations. This appendix figure shows estimates for all annotations rather than only the most extreme cases.
Annotation id
Category
n pos.
pr. ppt mean
pr. ppts. ( > 4 msgs)
bot-discourages-violence
concerns harm
690
0.004
0.632
bot-validates-violent-feelings
912
0.003
0.474
bot-discourages-self-harm
475
0.002
0.526
bot-validates-self-harm-feelings
541
0.002
0.526
bot-facilitates-violence
179
0.001
0.316
user-violent-thoughts
82
0.001
0.105
Appendix
Table 8. Frequencies of each annotation after applying per-annotation score cutoffs. We list every code ( annotation id ) within broad category and scoped to either user, assistant or both, and report the numbers of messages positively classified with said code ( n pos. ). rate gives the fraction of scoped messages with that annotation (over all messages), while rate ppt mean is the mean per-participant positive rate and rate ppts. is the proportion of participants with at least K=5 positive instances. (§§ 3.2 describes how we made these codes. Full descriptions of each appear in Appendix §§ B.1 .) This table uses the cutoffs from the validation set.
Figure 8. Prevalence of code categories split by chatbot. These data split category prevalence for gpt-4o and gpt-5 . See Fig. 2 for the aggregate version.
Figure 9. A heatmap of all transitions between source and target codes, X→Y . The log-lift of the model is projected onto a color space. Uses the model described in §§ B.2
Mental health professionals have raised concerns about risks of psychological harm from interaction with large language models (LLMs), including "delusional spirals" in which concerning human and LLM behaviors reinforce each other over time. With growing public use of LLM-powered chatbots, there is an urgent need to build evaluations grounded in real-world episodes of psychological harm experienced by users. We developed DelusionEval, an evaluation protocol that tests a model's tendencies to exhibit behaviors linked to promoting user delusions. We prompt each model with 589 unique conversation histories from 18 participants, comprising 12,591 messages from users who experienced delusions and psychological harm. We find that the tendency of an evaluated LLM to exhibit delusion-linked behavior does not reliably correlate with model size, release date, or the presence of test-time reasoning. However, extending the context of prior messages substantially increases rates of delusion-linked behaviors, providing evidence for the importance of context in LLM safety evaluation. For example, the rate of failing to discourage self-harm when the user expresses suicidal ideation increases from 30.0% to 41.1% when an additional 350 messages are prepended to the conversation history. All model families (e.g., GPT, Claude) exhibit substantial rates of delusion-linked behaviors. Within families, later, larger, or higher-reasoning models are not uniformly better across all behavior categories. Our results raise concerns regarding the potential psychological impact of LLMs and the need for more rigorous studies of real-world human-AI interaction.
Jared Moore, Andrea Mock, Yifan Mai +9
Stanford University Stanford, California, USA · University of Chicago Chicago, Illinois, USA · Carnegie Mellon University Pittsburgh, Pennsylvania, USA +2
LLM chatbots increasingly serve as a first source of support for people in psychological distress, including those whose distress is entangled with delusional beliefs. Prior work on LLM mental-health safety largely evaluates general therapeutic quality or single-turn crisis detection, leaving unclear how models behave when distress is intertwined with delusion over sustained conversations. We address this gap with matched multi-turn simulations, across clinically grounded personas and six LLMs, that pair each delusional conversation with a distress-only control to isolate the effect of delusional framing. This reveals a recognition-intervention gap: models detect distress at comparable rates regardless of framing, yet sharply fail to act on it once distress is embedded in delusion, with safety interventions suppressed by up to 4.5x. The failure tracks accumulated acceptance of the user's premises rather than emotional validation. Worse, the intuitive fix of prompting models to assess user distress backfires under delusional framing; only delusion-aware prompting with explicit response guidance closes the gap, and even this depends on a delusion classifier that is itself unreliable on the most vulnerable models. Safe deployment therefore requires treating delusional framing as a distinct risk signal that overrides conversational accommodation.
Andrew Aquilina, Chetna Nihalani, Vasudha Varadarajan +3
University of Pittsburgh · Carnegie Mellon University · Fordham University
"AI psychosis" or "delusional spiraling" is an emerging phenomenon where AI chatbot users find themselves dangerously confident in outlandish beliefs after extended chatbot conversations. This phenomenon is typically attributed to AI chatbots' well-documented bias towards validating users' claims, a property often called "sycophancy." In this paper, we probe the causal link between AI sycophancy and AI-induced psychosis through modeling and simulation. We propose a simple Bayesian model of a user conversing with a chatbot, and formalize notions of sycophancy and delusional spiraling in that model. We then show that in this model, even an idealized Bayes-rational user is vulnerable to delusional spiraling, and that sycophancy plays a causal role. Furthermore, this effect persists in the face of two candidate mitigations: preventing chatbots from hallucinating false claims, and informing users of the possibility of model sycophancy. We conclude by discussing the implications of these results for model developers and policymakers concerned with mitigating the problem of delusional spiraling.
Kartik Chandra, Max Kleiman-Weiner, Jonathan Ragan-Kelley +1
MIT CSAIL · University of Washington, Seattle · MIT Department of Brain & Cognitive Sciences