Veterans' Mental Health Crisis Severity Annotation: Do LLMs Agree More Than Human Experts?
Authors: K M Sajjadul Islam, Pranav Yaramalla, Nadiyah Johnson, Mark Flower, Praveen Madiraju, Zeno Franco
Organizations: Computer Science, Marquette University · North Carolina School of Science and Mathematics · Veteran Peer Specialist, Captain John D Mason Peer Support Program · Emergency Medicine, Medical College of Wisconsin
Large language models (LLMs) offer a scalable approach for annotating mental health and crisis-related social media content, but their reliability in safety-critical tasks remains unclear. This study evaluates LLM-based crisis-severity annotation of veteran-related social media posts against annotations from eight Veteran Subject Matter Experts (SMEs). Using 149 tweets and five severity categories, we evaluate 15 open-source and 10 proprietary LLMs across eight prompt variants, examining annotation performance, inter-model agreement, prompt sensitivity, and high-severity crisis detection. Human reviewers showed higher ordinal than nominal agreement (α = 0.494), indicating that disagreements often occurred between neighboring severity levels. In contrast, selected LLMs demonstrated substantially higher inter-model consistency, with ordinal agreement (α) reaching 0.8336 under the baseline prompt. Proprietary models generally showed better agreement with expert annotations and greater stability than open-source models. GPT-5.6-Terra achieved the best overall performance, while the open-source Qwen2.5-7B showed the strongest performance in identifying Acute Crisis with Suicidal Ideation cases. However, higher model consistency did not necessarily correspond to agreement with human judgments or reliable detection of rare high-severity cases. Label-flipped demonstration experiments also revealed model-specific sensitivity to misleading in-context examples, while explanatory demonstrations did not consistently improve performance. Overall, LLMs show promise as scalable complements to expert annotation, but require careful evaluation before use in high-stakes mental health applications.
Figures & tables
Annotation Category/Label
Severity
Count
L1 - No Concern
1
100
L2 - Distressed Self-Disclosure
2
19
L3 - Self-Identified Crisis Symptoms and Seeking Support
3
15
L4 - Self-Identified Crisis Symptoms but Not Seeking Support
4
0
L5 - Acute Crisis with Suicidal Ideation
5
6
L6 - Not Relevant to the Research
–
9
TABLE I: Annotation Categories, Crisis Severity Levels, and Reference Label Distribution
Metric
7 Labels
6 Labels
5 Labels
Majority Agreement ( ≥ 5/8)
59.06%
61.74%
81.20%
Full agreement ( 8/8 )
0.00%
0.00%
32.88%
Fleiss’ κ
0.1430
0.1442
0.2913
Krippendorff’s α (nominal)
0.1437
0.1449
0.2919
Krippendorff’s α (ordinal)
0.4941
0.4941
0.4467
TABLE II: Inter-Rater Reliability Before and After Label Consolidation
Fig. 1: Label distributions of the eight Veteran SME reviewers and six selected LLMs under Prompt G. The stacked percentages show the relative frequency with which each reviewer/LLM assigned the seven annotation categories.
Type
Model / Prompt
A
B
C
D
E
F
G
H
Mean
SD
Avg. Rank
Open-source local models (Ollama)
aya-expanse-8b
0.714
0.604
–
0.468
0.746
–
0.515
0.736
0.630
0.120
6.000
command-r7b-7b
0.581
0.376
0.581
0.192
0.613
0.657
0.719
0.595
0.539
0.171
8.938
deepseek-r1-8b
0.298
0.190
0.298
0.375
0.344
0.406
0.742
0.486
0.392
0.166
10.500
gemma3-4b
0.479
0.423
0.479
0.304
0.753
0.486
0.763
0.379
0.508
0.166
9.000
granite3.3-8b
0.471
0.357
0.471
0.602
0.371
0.679
0.693
0.282
0.491
0.161
10.625
hermes3-8b
0.629
0.636
0.629
0.797
0.780
0.719
0.785
0.402
0.672
0.131
4.375
TABLE III: Micro-F1 Across Prompt Variants
Type
Model/Prompt
A
G
Open-Source LLM
hermes3-8b
0.547
0.611
llama3.1-8b
0.555
0.654
qwen2.5-7B
0.542
0.569
Proprietary LLM
gpt-5.4-mini
0.584
0.750
gpt-5.6-terra
0.517
0.699
claude-opus-5
0.669
0.640
TABLE IV: Macro-F1 Scores of the Six Selected LLMs Under Prompts A and G
Metric
Prompt A
Prompt G
6 Labels
6 Labels
5 Labels
Majority agreement ( ≥ 4/6)
86.58%
87.25%
91.95%
Full agreement ( 6/6 )
63.09%
51.68%
73.83%
Fleiss’ κ
0.6619
0.6038
0.6490
Krippendorff’s α (nominal)
0.6619
0.6083
0.6498
Krippendorff’s α (ordinal)
0.8336
0.8533
0.8321
TABLE V: Inter-model Reliability of the Six Selected LLMs Under Prompts A and G
Fig. 2: Pairwise Cohen’s κ Agreement Among LLMs Under Prompt G. Cohen’s κ measures pairwise inter-rater agreement beyond chance [ 23 ] .
Metric
Open-Source LLM
Proprietary LLM
Majority agreement ( ≥ 2/3)
97.99%
97.32%
Full agreement ( 3/3 )
73.83%
68.46%
Fleiss’ κ
0.6676
0.6490
Krippendorff’s α (nominal)
0.6716
0.6498
Krippendorff’s α (ordinal)
0.8113
0.9297
TABLE VI: Inter-model Reliability of Open-Source and Hosted LLMs
Tweet
Human Votes
Reference Label
LLM Votes
(Tied)
(Tie Resolution)
Prompt A
Prompt G
I’m done #bipolar #bipolarclub113
L2; L5 [2 votes]
L5
L4; L5 [3 votes]
L2; L4 [3 votes]
I meant I’m doing okay. #PTSDChat
L1; L2 [4 votes]
L2
L1 (6 votes)
L1 (6 votes)
Hi everyone #PTSDChat113
L1; L3 [3 votes]
L3
L1 (6 votes)
L1 (6 votes)
My current mood is…..probably not safe enough to tweet about11
L2; L3; L5 [2 votes]
L5
L2 (5 votes); L5 (1 vote)
L2 (3 votes); L4 (2 votes); L5 (1 vote)
Fuckin ideation won’t go away HELP @BipolarClubRx #bipolar111
L1; L3; L5 [2 votes]
L5
L3; L5 [3 votes]
L3 (4 votes); L5 (2 vote)
TABLE VII: Tweets with Tied Human Reviewer Votes and LLM Votes from the 6 Selected Models
Tweet
llama3.1-8b
hermes3-8b
qwen2.5-7b
gpt-5.4-mini
claude-Opus-5
gpt-5.6-Terra
I’m done #bipolar #bipolarclub113
L4
L4
L5
L4
L5
L5
I’m done #bipolar #bipolarclub113.1
L4
L4
L5
L4
L5
L5
Dark too dark falling falling fading…..gone
L5
L5
L5
L5
L5
L5
My current mood is…..probably not safe enough to tweet about11
L2
L2
L5
L2
L4
L4
Fuckin ideation won’t go away HELP @BipolarClubRx #bipolar111
L3
L3
L5
L3
L3
L5
It’s useless I can’t anymore BYE1
L5
L5
L5
L5
L5
L5
TABLE VIII: LLM Predictions for Tweets with Human Reference Label L5
Demonstration Tweet
Reference Label
Random Label-Flipped
Targeted Label-Flipped
Happy birthday to the beloved king of grunge #[PERSON]
L1
L6
L1
I can’t make it through this one guys I’m sorry
L2
L1
L1
[PERSON] has died rot in pieces you racist sack of crap
L1
L1
L1
TABLE IX: Few-Shot Demonstrations Used in Prompts I and J