Veterans' Mental Health Crisis Severity Annotation: Do LLMs Agree More Than Human Experts?
Authors: K M Sajjadul Islam, Pranav Yaramalla, Nadiyah Johnson, Mark Flower, Praveen Madiraju, Zeno Franco
Organizations: Computer Science, Marquette University · North Carolina School of Science and Mathematics · Veteran Peer Specialist, Captain John D Mason Peer Support Program · Emergency Medicine, Medical College of Wisconsin
Large language models (LLMs) offer a scalable approach for annotating mental health and crisis-related social media content, but their reliability in safety-critical tasks remains unclear. This study evaluates LLM-based crisis-severity annotation of veteran-related social media posts against annotations from eight Veteran Subject Matter Experts (SMEs). Using 149 tweets and five severity categories, we evaluate 15 open-source and 10 proprietary LLMs across eight prompt variants, examining annotation performance, inter-model agreement, prompt sensitivity, and high-severity crisis detection. Human reviewers showed higher ordinal than nominal agreement (α = 0.494), indicating that disagreements often occurred between neighboring severity levels. In contrast, selected LLMs demonstrated substantially higher inter-model consistency, with ordinal agreement (α) reaching 0.8336 under the baseline prompt. Proprietary models generally showed better agreement with expert annotations and greater stability than open-source models. GPT-5.6-Terra achieved the best overall performance, while the open-source Qwen2.5-7B showed the strongest performance in identifying Acute Crisis with Suicidal Ideation cases. However, higher model consistency did not necessarily correspond to agreement with human judgments or reliable detection of rare high-severity cases. Label-flipped demonstration experiments also revealed model-specific sensitivity to misleading in-context examples, while explanatory demonstrations did not consistently improve performance. Overall, LLMs show promise as scalable complements to expert annotation, but require careful evaluation before use in high-stakes mental health applications.
Figures & tables
Annotation Category/Label
Severity
Count
L1 - No Concern
1
100
L2 - Distressed Self-Disclosure
2
19
L3 - Self-Identified Crisis Symptoms and Seeking Support
3
15
L4 - Self-Identified Crisis Symptoms but Not Seeking Support
4
0
L5 - Acute Crisis with Suicidal Ideation
5
6
L6 - Not Relevant to the Research
–
9
TABLE I: Annotation Categories, Crisis Severity Levels, and Reference Label Distribution
Metric
7 Labels
6 Labels
5 Labels
Majority Agreement ( ≥ 5/8)
59.06%
61.74%
81.20%
Full agreement ( 8/8 )
0.00%
0.00%
32.88%
Fleiss’ κ
0.1430
0.1442
0.2913
Krippendorff’s α (nominal)
0.1437
0.1449
0.2919
Krippendorff’s α (ordinal)
0.4941
0.4941
0.4467
TABLE II: Inter-Rater Reliability Before and After Label Consolidation
Fig. 1: Label distributions of the eight Veteran SME reviewers and six selected LLMs under Prompt G. The stacked percentages show the relative frequency with which each reviewer/LLM assigned the seven annotation categories.
Type
Model / Prompt
A
B
C
D
E
F
G
H
Mean
SD
Avg. Rank
Open-source local models (Ollama)
aya-expanse-8b
0.714
0.604
–
0.468
0.746
–
0.515
0.736
0.630
0.120
6.000
command-r7b-7b
0.581
0.376
0.581
0.192
0.613
0.657
0.719
0.595
0.539
0.171
8.938
deepseek-r1-8b
0.298
0.190
0.298
0.375
0.344
0.406
0.742
0.486
0.392
0.166
10.500
gemma3-4b
0.479
0.423
0.479
0.304
0.753
0.486
0.763
0.379
0.508
0.166
9.000
granite3.3-8b
0.471
0.357
0.471
0.602
0.371
0.679
0.693
0.282
0.491
0.161
10.625
hermes3-8b
0.629
0.636
0.629
0.797
0.780
0.719
0.785
0.402
0.672
0.131
4.375
TABLE III: Micro-F1 Across Prompt Variants
Type
Model/Prompt
A
G
Open-Source LLM
hermes3-8b
0.547
0.611
llama3.1-8b
0.555
0.654
qwen2.5-7B
0.542
0.569
Proprietary LLM
gpt-5.4-mini
0.584
0.750
gpt-5.6-terra
0.517
0.699
claude-opus-5
0.669
0.640
TABLE IV: Macro-F1 Scores of the Six Selected LLMs Under Prompts A and G
Metric
Prompt A
Prompt G
6 Labels
6 Labels
5 Labels
Majority agreement ( ≥ 4/6)
86.58%
87.25%
91.95%
Full agreement ( 6/6 )
63.09%
51.68%
73.83%
Fleiss’ κ
0.6619
0.6038
0.6490
Krippendorff’s α (nominal)
0.6619
0.6083
0.6498
Krippendorff’s α (ordinal)
0.8336
0.8533
0.8321
TABLE V: Inter-model Reliability of the Six Selected LLMs Under Prompts A and G
Fig. 2: Pairwise Cohen’s κ Agreement Among LLMs Under Prompt G. Cohen’s κ measures pairwise inter-rater agreement beyond chance [ 23 ] .
Metric
Open-Source LLM
Proprietary LLM
Majority agreement ( ≥ 2/3)
97.99%
97.32%
Full agreement ( 3/3 )
73.83%
68.46%
Fleiss’ κ
0.6676
0.6490
Krippendorff’s α (nominal)
0.6716
0.6498
Krippendorff’s α (ordinal)
0.8113
0.9297
TABLE VI: Inter-model Reliability of Open-Source and Hosted LLMs
Tweet
Human Votes
Reference Label
LLM Votes
(Tied)
(Tie Resolution)
Prompt A
Prompt G
I’m done #bipolar #bipolarclub113
L2; L5 [2 votes]
L5
L4; L5 [3 votes]
L2; L4 [3 votes]
I meant I’m doing okay. #PTSDChat
L1; L2 [4 votes]
L2
L1 (6 votes)
L1 (6 votes)
Hi everyone #PTSDChat113
L1; L3 [3 votes]
L3
L1 (6 votes)
L1 (6 votes)
My current mood is…..probably not safe enough to tweet about11
L2; L3; L5 [2 votes]
L5
L2 (5 votes); L5 (1 vote)
L2 (3 votes); L4 (2 votes); L5 (1 vote)
Fuckin ideation won’t go away HELP @BipolarClubRx #bipolar111
L1; L3; L5 [2 votes]
L5
L3; L5 [3 votes]
L3 (4 votes); L5 (2 vote)
TABLE VII: Tweets with Tied Human Reviewer Votes and LLM Votes from the 6 Selected Models
Tweet
llama3.1-8b
hermes3-8b
qwen2.5-7b
gpt-5.4-mini
claude-Opus-5
gpt-5.6-Terra
I’m done #bipolar #bipolarclub113
L4
L4
L5
L4
L5
L5
I’m done #bipolar #bipolarclub113.1
L4
L4
L5
L4
L5
L5
Dark too dark falling falling fading…..gone
L5
L5
L5
L5
L5
L5
My current mood is…..probably not safe enough to tweet about11
L2
L2
L5
L2
L4
L4
Fuckin ideation won’t go away HELP @BipolarClubRx #bipolar111
L3
L3
L5
L3
L3
L5
It’s useless I can’t anymore BYE1
L5
L5
L5
L5
L5
L5
TABLE VIII: LLM Predictions for Tweets with Human Reference Label L5
Demonstration Tweet
Reference Label
Random Label-Flipped
Targeted Label-Flipped
Happy birthday to the beloved king of grunge #[PERSON]
L1
L6
L1
I can’t make it through this one guys I’m sorry
L2
L1
L1
[PERSON] has died rot in pieces you racist sack of crap
L1
L1
L1
TABLE IX: Few-Shot Demonstrations Used in Prompts I and J
A large language model (LLM) can rate depression severity directly from a social media post, or mark which clinical criteria the post shows and let a simple rule turn the count into a label. The latter is easier to audit because a clinician can check each marked criterion. We compare these approaches on two Reddit corpora using three LLMs (from 9B to frontier scale) and two questionnaires (PHQ-9, BDI-II), and measure agreement with quadratic weighted kappa. For the two frontier models, criteria extraction scores above chain-of-thought on one corpus only when its decision thresholds are fitted on labeled data. Neither model's gain is significant, with or without recalibrating chain-of-thought on the same labels. With thresholds fixed a priori from PHQ-9's criteria, extraction shows no gain on either corpus, even where models mark over two criteria per post. The 9B model behaves differently on a corpus from depression communities. It labels most posts as severe, whether prompted directly or with chain-of-thought, while the a priori rule, which uses no labels, beats both. After chain-of-thought is recalibrated on the same labels, no significant gap remains, consistent with a calibration effect. Yet higher ordinal agreement does not ensure better detection of severe cases. PHQ-9 criteria extraction misses most severe posts, and moving from direct prompting to chain-of-thought and then to extraction increases misses in nearly all comparisons. On one corpus, a relabeled stress dataset, a model using that dataset's own features, some computed from the post text, is not significantly different from frontier criteria extraction under the a priori rule.
General-purpose Large Language Models (LLMs) are becoming widely adopted by people for mental health support. Yet emerging evidence suggests there are significant risks associated with high-frequency use, particularly for individuals suffering from psychosis, as LLMs may reinforce delusions and hallucinations. Existing evaluations of LLMs in mental health contexts are limited by a lack of clinical validation and scalability of assessment. To address these issues, this research focuses on psychosis as a critical condition for LLM safety evaluation by (1) developing and validating seven clinician-informed safety criteria, (2) constructing a human-consensus dataset, and (3) testing automated assessment using an LLM as an evaluator (LLM-as-a-Judge) or taking the majority vote of several LLM judges (LLM-as-a-Jury). Results indicate that LLM-as-a-Judge aligns closely with the human consensus (Cohen's κhuman×gemini=0.75, κhuman×qwen=0.68, κhuman×kimi=0.56) and that the best judge slightly outperforms LLM-as-a-Jury (Cohen's κhuman×jury=0.74). Overall, these findings have promising implications for clinically grounded, scalable methods in LLM safety evaluations for mental health contexts.
May Lynn Reese, Markela Zeneli, Mindy Ng +3
Apart Research · Odyssean Institute · London School of Economics and Political Science +1
Judgments about psychological distress are socially situated: what counts as concerning hinges on community norms around emotional expression, vulnerability, and help-seeking. Yet large language models (LLMs) used for distress detection are typically aligned to a single, undifferentiated standard. How well do these models capture the perspectives of the communities whose language they assess? We address this question through a perspectivist annotation study in which 321 participants provided 9,587 judgments on 1,198 Reddit posts spanning six identity-based communities, yielding community-specific labels. Raters in the contextualized in-group condition show a modest tendency to agree more with their community than uncontextualized out-group raters (OR = 1.18), an effect varying significantly across communities. We then evaluate nine open-weight LLM configurations and four frontier configurations against these labels. Open-weight LLMs systematically over-estimate distress: when communities perceive none-to-mild distress, these models achieve only 31-44% accuracy, predominantly producing false positives. GPT-5 and Gemini 2.5 Pro show the same none-to-mild inflation even when their full-sample over/under rates are mixed, while Claude Opus 4 is more conservative. This pattern does not simply mirror an outsider reading position: uncontextualized out-group human aggregates were nearly symmetric, with 18% over-estimation versus 19% under-estimation. Instead, the models that inflate none-to-mild cases exhibit a distress prior that exceeds both contextualized in-group and uncontextualized out-group human judgments. These findings have implications for equitable AI deployment in mental health contexts, where miscalibrated distress detection may unevenly affect the communities being assessed.
Andrew Aquilina, Xiang Lorraine Li, Yu-Ru Li
School of Computing and Information, University of Pittsburgh