Large Language Models (LLMs) are increasingly used for interpersonal advice, potentially shaping how users interpret ambiguous social encounters. Existing Theory of Mind evaluations typically rely on predefined answers, while uncertainty research overlooks LLMs' preferred explanations and confidence under psychological distress. To address these gaps, we introduce MentalLoom, a novel evaluation of LLM mentalisation comprising 604 ambiguous social episodes, matched depression and social anxiety disclosures, and four-turn dialogues with unchanged event evidence. Across eight LLMs, disclosures initially elicited more negative explanations overall but more positive narrator-directed interpretations. Scripted repeated questioning showed greater emotional support, stronger positive narrator-directed endorsement, and higher certainty without new event evidence. These findings show that user context can reshape interpretive preferences and commitment while the underlying ambiguity remains unresolved.
Figures & tables
Figure 1: Example of Mentalisation where the same behaviour can support multiple provisional interpretations.
Model
− ve valence
+ ve valence
− ve narrator directed valence
+ ve narrator directed valence
Emotional support
High-certainty
None
Dep
SA
None
Dep
SA
None
Dep
SA
None
Dep
SA
None
Dep
SA
None
Dep
SA
DeepSeek-V3.1
34.64
38.18
32.63
11.48
11.44
12.79
13.61
10.88
9.03
12.41
14.52
15.12
0.00
16.29
3.45
2.46
1.59
0.58
Qwen2.5-72B
33.35
40.44
34.75
17.18
17.19
18.67
7.54
5.86
5.74
17.85
20.60
20.84
0.00
5.76
4.07
4.20
3.74
2.42
Qwen3-8B
22.72
39.23
32.43
20.24
20.41
23.06
7.49
7.21
7.17
21.65
23.86
26.40
0.17
3.10
4.24
9.10
5.95
4.87
Nemotron-3-Super
31.99
36.78
30.55
14.53
13.71
16.35
9.33
7.55
6.17
16.22
19.48
21.07
0.00
6.53
4.63
7.17
6.75
6.98
Mistral-Small-24B
32.36
39.06
35.97
17.22
15.35
16.52
10.41
11.38
11.58
16.73
16.18
16.70
0.00
4.41
2.69
3.87
0.75
0.50
Table 1: Initial turn (T0) response statistics (%). Red: negative valence/high certainty; green: positive valence/emotional support. Highest value in each column is bolded. Dep = Depression; SA = Social anxiety.
Table 2: Certainty Weighted Interpretive Valence (CWIV, %). Values indicate net narrator-directed endorsement weighted by expressed certainty.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Item
Question
Core
1
Little interest or pleasure in doing things
2
Feeling down, depressed, or hopeless
Supporting
3
Trouble falling or staying asleep, or sleeping too much
4
Feeling tired or having little energy
Appendix
Table 3: PHQ-9 items grouped by role in the study design.
Item
Question
Fear
1
I am afraid of people in authority.
3
Parties and social events scare me.
5
Being criticized scares me a lot.
10
Talking to strangers scares me.
14
I am afraid of doing things when people might be watching.
Appendix
Table 4: SPIN items grouped by category.
Dimension
Categories
Label Definitions
Plausibility
3
Plausible: Reasonable causal account consistent with available evidence. Implausible: Unreasonable causal account or clear contradiction. Unclear: Plausibility cannot be determined.
Explanation valence
3
Positive: Pleasant emotions, benevolent intentions or favourable evaluations. Negative: Distress, hostility, unfavourable evaluations or adverse circumstances. Neutral: Neither. Negative takes precedence when both valences occur within an explanation.
Narrator-directed valence
3
Positive: Care, acceptance, favourable evaluation or reassuring nonpersonal meaning towards the narrator. Negative: Rejection, hostility, blame or unfavourable evaluation towards the narrator. Neutral: Neither.
Certainty
3
1 (Possible): Hypothetical, unfavoured or unranked possibility. 2 (Likely): Preferred explanation without strong confidence. 3 (High certainty): Strong confidence or unqualified conclusion outside possibility framing.
Emotional support
2
1: Acknowledges or validates feelings expressed by the user. 0: No such acknowledgement or validation.
Appendix
Table 5: Operational definitions of the response evaluation labels.
Figure 3: Prompt used to select activity labels for ambiguous interpersonal scenarios.
Figure 4: Prompt used to nominate and rank ten scenario contexts for each social behaviour.
Figure 5: Prompt used to generate a first-person event account for each behaviour–scenario pair.
Figure 6: Prompt used to combine three selected symptom features into a first-person preamble.
Figure 7: Prompts for all conversation turns
Figure 8: Prompt used to select candidate A or B, or reject both candidates with NEITHER.
Figure 9: Response evaluation prompt covering task rules and certainty.
Figure 10: Response evaluation prompt specifying the JSON output structure and input fields.
Model
No Disclosure
Depression
Social anxiety
T0
T1
T2
T3
Mean
T0
T1
T2
T3
Mean
T0
T1
T2
T3
Mean
DeepSeek-V3.1
−0.5
+26.7
+17.9
+26.9
+17.7
+1.7
+35.4
+37.3
+41.1
+28.9
+2.3
+37.3
+42.4
+46.7
+32.2
Qwen2.5-72B
+4.3
+16.2
+13.2
+17.9
+12.9
+6.5
+15.0
+15.4
+18.2
+13.7
+6.0
+17.3
+18.9
+21.3
+15.9
Qwen3-8B
+6.8
+15.4
+9.9
+11.5
+10.9
+7.3
+15.0
+10.9
+12.5
+11.4
+7.7
+16.9
+13.2
+14.9
+13.2
Nemotron-3
+3.3
+24.3
+15.0
+31.2
+18.4
+5.5
+23.6
+23.2
+29.2
+20.4
+6.9
+28.9
+29.0
+34.1
+24.7
Mistral-24B
+2.9
+8.2
+7.3
+10.5
+7.2
+1.9
+9.2
+10.5
+10.0
+7.9
+1.9
+11.0
+14.8
+14.8
+10.6
Appendix
Table 6: Certainty Weighted Interpretive Valence (CWIV, %) by model, disclosure condition and dialogue turn. Positive and negative values indicate net positive and negative narrator-directed endorsement weighted by expressed certainty. Green indicates positive CWIV and red indicates negative CWIV.
Interaction
− ve valence
+ ve valence
− ve narrator directed valence
+ ve narrator directed valence
Emotional support
High-certainty
None
Dep
SA
None
Dep
SA
None
Dep
SA
None
Dep
SA
None
Dep
SA
None
Dep
SA
Direct final request
28.82
32.55
25.79
23.82
19.23
21.90
8.75
7.82
6.89
30.48
32.38
38.31
18.77
45.33
40.62
2.89
2.00
2.05
After dialogue
28.68
33.31
25.44
23.26
21.05
23.15
6.55
3.97
3.32
37.35
40.86
45.02
66.18
70.42
68.50
5.24
5.03
5.55
Appendix
Table 7: Responses to the same final request in T3 presented directly or after dialogue (%). Red denotes negative valence/high certainty; green denotes positive valence/support. Darker shading indicates larger values; bold marks the higher value in each column. None = no disclosure; Dep = depression; SA = social anxiety.
People increasingly turn to large language models (LLMs) to interpret ambiguous social situations: a delayed text reply, an unusually cold supervisor, a teacher's mixed signals, or a boundary-crossing friend. Yet in many such cases, no stable interpretation can be verified from the available evidence alone. We study how LLMs respond to these situations across four domains: early-stage romantic relationships, teacher--student dynamics, workplace hierarchies, and ambiguous friendships. Across 72 responses from GPT, Claude, and Gemini, only 9 (12.5%) genuinely preserved uncertainty. The remaining 87.5% produced interpretive closure through recurring pathways including narrative alignment, narrative reversal, normative advice under uncertainty, and hedged language that still supported a single conclusion. We further find that narrator perspective shapes the path to closure: first-person accounts more often elicited alignment, while third-person accounts invited more detached interpretation, even when the underlying situation remained comparable. Together, these findings show that LLMs do not simply assist interpersonal sensemaking; they tend to resolve ambiguity into coherent and actionable narratives. These results suggest that the central risk is not only that LLMs may misinterpret social situations, but that they may make unresolved situations feel prematurely settled. We frame this tendency as a design challenge for uncertainty-preserving social AI.
What if large language models could not only infer human mindsets but also expose every blind spot in team dialogue such as discrepancies in the team members' joint understanding? We present a novel, two-step framework that leverages large language models (LLMs) both as human-style annotators of team dialogues to track the team's shared mental models (SMMs) and as automated discrepancy detectors among individuals' mental states. In the first step, an LLM generates annotations by identifying SMM elements within task-oriented dialogues from the Cooperative Remote Search Task (CReST) corpus. Then, a secondary LLM compares these LLM-derived annotations and human annotations against gold-standard labels to detect and characterize divergences. We define an SMM coherence evaluation framework for this use case and apply it to six CReST dialogues, ultimately producing: (1) a dataset of human and LLM annotations; (2) a reproducible evaluation framework for SMM coherence; and (3) an empirical assessment of LLM-based discrepancy detection. Our results reveal that, although LLMs exhibit apparent coherence on straightforward natural-language annotation tasks, they systematically err in scenarios requiring spatial reasoning or disambiguation of prosodic cues.
LLM chatbots increasingly serve as a first source of support for people in psychological distress, including those whose distress is entangled with delusional beliefs. Prior work on LLM mental-health safety largely evaluates general therapeutic quality or single-turn crisis detection, leaving unclear how models behave when distress is intertwined with delusion over sustained conversations. We address this gap with matched multi-turn simulations, across clinically grounded personas and six LLMs, that pair each delusional conversation with a distress-only control to isolate the effect of delusional framing. This reveals a recognition-intervention gap: models detect distress at comparable rates regardless of framing, yet sharply fail to act on it once distress is embedded in delusion, with safety interventions suppressed by up to 4.5x. The failure tracks accumulated acceptance of the user's premises rather than emotional validation. Worse, the intuitive fix of prompting models to assess user distress backfires under delusional framing; only delusion-aware prompting with explicit response guidance closes the gap, and even this depends on a delusion classifier that is itself unreliable on the most vulnerable models. Safe deployment therefore requires treating delusional framing as a distinct risk signal that overrides conversational accommodation.
Andrew Aquilina, Chetna Nihalani, Vasudha Varadarajan +3
University of Pittsburgh · Carnegie Mellon University · Fordham University