cs.AINov 5, 2026

Veterans' Mental Health Crisis Severity Annotation: Do LLMs Agree More Than Human Experts?

Authors: K M Sajjadul Islam, Pranav Yaramalla, Nadiyah Johnson, Mark Flower, Praveen Madiraju, Zeno Franco

Organizations: Computer Science, Marquette University · North Carolina School of Science and Mathematics · Veteran Peer Specialist, Captain John D Mason Peer Support Program · Emergency Medicine, Medical College of Wisconsin

Abstract

Large language models (LLMs) offer a scalable approach for annotating mental health and crisis-related social media content, but their reliability in safety-critical tasks remains unclear. This study evaluates LLM-based crisis-severity annotation of veteran-related social media posts against annotations from eight Veteran Subject Matter Experts (SMEs). Using 149 tweets and five severity categories, we evaluate 15 open-source and 10 proprietary LLMs across eight prompt variants, examining annotation performance, inter-model agreement, prompt sensitivity, and high-severity crisis detection. Human reviewers showed higher ordinal than nominal agreement (α = 0.494), indicating that disagreements often occurred between neighboring severity levels. In contrast, selected LLMs demonstrated substantially higher inter-model consistency, with ordinal agreement (α) reaching 0.8336 under the baseline prompt. Proprietary models generally showed better agreement with expert annotations and greater stability than open-source models. GPT-5.6-Terra achieved the best overall performance, while the open-source Qwen2.5-7B showed the strongest performance in identifying Acute Crisis with Suicidal Ideation cases. However, higher model consistency did not necessarily correspond to agreement with human judgments or reliable detection of rare high-severity cases. Label-flipped demonstration experiments also revealed model-specific sensitivity to misleading in-context examples, while explanatory demonstrations did not consistently improve performance. Overall, LLMs show promise as scalable complements to expert annotation, but require careful evaluation before use in high-stakes mental health applications.

Figures & tables

Explore similar work

CardsList
  1. Using LLM-as-a-Judge/Jury to Advance Scalable, Clinically-Validated Safety Evaluations of Model Responses to Users Demonstrating Psychosis

    Mar 20, 2026May Lynn Reese, Markela Zeneli, Mindy Ng +3LLM SafetyLLM Safety Evaluation

  2. Whose Assessment of Distress? Community Perspectives and LLM Alignment on Well-Being Posts

    Aug 29, 2026Andrew Aquilina, Xiang Lorraine Li, Yu-Ru LiSocial Bias in Language ModelsLanguage Model Bias Evaluation