Large Language Models (LLMs) have shown remarkable capabilities for complex tasks, yet adaptation in medical domain, specifically mental health, poses specific challenges. Mental health is a rising concern globally with LLMs having large potential to help address the same. We highlight three primary challenges for LLMs in mental health - lack of high quality interpretable and knowledge grounded training data; training paradigms restricted to core capabilities, and evaluation of multi turn dialogue settings. Addressing it, we present oMind framework which includes training and aligning LLM agents for diverse capabilities including conversations; high quality ~164k multi-task SFT dataset, as a result of our generation pipeline based on Structured Knowledge retrieval, LLM based pruning, and review actions. We also introduce oMind-Chat - a novel multi turn benchmark dataset with expert annotated turn level and conversation level rubrics. Our diverse experiments on both core capabilities and conversations shows oMind LLMs consistently outperform baselines. oMind-LLM also shows significantly better reasoning with up to 80% win rate.
Figures & tables
Figure 1: Complete pipeline for oMind SFT and model training. (A) Generation Pipeline consisting of retrieval stage, pruning & generation stage, and review stage. (B) Generation of support QA and conversations using LLM. (c) LLM finetuning and preference learning.
Type
Dataset
# Train
# Test
MCQA
MHQA Racha et al. (2025)
25,000
500
MedMCQA Pal et al. (2022)
2,750
594
MMLU-hsp Hendrycks et al. (2021)
441
150
MMLU-pp Hendrycks et al. (2021)
462
150
USMLE-Mental Jin et al. (2023)
498
219
Disorder
ANGST Hengle et al. (2024)
6,990
500
Table 1: Datasets statistics used for SFT
Figure 2: Illustration of oMind-Chat with rubrics.
Model
Method
Classification
MCQA
ANGST
dreaddit
CAMS
SAD
AVG
MHQA
MedMCQA
MMLU-hsp
MMLU-pp
USMLE
AVG
Llama 3.1 8B
Zero Shot
31.0
63.8
30.7
49.3
43.7
63.6
84.8
89.9
62.8
60.5
72.3
LLM+KG
35.4
71.3
33.8
63.8
51.1
75.4
85.5
88.5
69.0
63.1
76.3
LLM+Books
39.4
75.1
25.5
60.0
50.0
70.2
84.4
83.3
62.9
34.9
67.1
MentaLLaMA 13B
Zero Shot
40.7
75.8
36.9
63.7
54.3
63.2
52.4
75.9
45.1
30.1
53.3
LLM+KG
39.2
66.8
26.2
60.0
48.1
66.0
59.2
75.4
47.1
28.0
55.2
Table 2: Performance comparison (% F1 scores) of oMind models and baselines across Classification and MCQA tasks. AVG is the average score. Best and second best performance are in bold and underline respectively. We also report average % gain/dip for oMind models compared with base models. LLM+KG and LLM+Books is baseline with retrieved KG and Books Chunks respectively.
Model
Turn Level
Conversation Level
Binary
Scale 0-10
Likert 1-5
Binary
Scale 0-10
Likert 1-5
Llama 3.1 8B
0.64
5.2
3.7
0.76
6.2
3.9
MentaLLaMA 13B
0.36
2.8
2.6
0.45
3.4
2.8
Mistral 7B
0.61
4.9
3.7
0.76
6.0
3.9
Qwen 2.5 7B
0.60
4.8
3.7
0.74
5.9
3.9
PMC-Llama 13B
0.40
3.0
2.7
0.51
3.8
2.8
Table 3: Conversational performance on oMind-Chat benchmark. Models are evaluated at turn-level and conversation-level using three metrics: Binary rubric Coverage, 0-10 Scale, and Likert (1-5).
Figure 3: Impact of Supporting QA removal (wosq) and DPO alignment on Qwen-2.5-7B and Mistral-7B performance across classification, MCQA, and conversational tasks.
Models
Win (%)
Tie (%)
Lose (%)
oMind-Llama vs Llama
77.0
10.5
12.5
oMind-Mistral vs Mistral
76.0
23.0
1.0
oMind-Qwen vs Qwen
80.0
4.0
16.0
oMind-Mistral vs wosq
24.5
73.5
2.0
oMind-Qwen vs wosq
6.0
66.0
28.0
oMind-Mistral-dpo vs oMind-Mistral
14.0
71.5
14.5
Table 4: Pairwise win-rate comparison for generated explanations.
Factual Richness
Faithfulness
Clarity
oMind Win
0.67/ 0.58
0.49 / 0.32
0.38 / 0.31
Tie
0.23/ 0.28
0.36/ 0.56
0.29/ 0.24
GPT Win
0.10/ 0.14
0.15/ 0.12
0.33/ 0.45
Table 5: Win Rate scores by annotator 1/annotator 2 for oMind and only GPT.
Division of Digital Psychiatry, Beth Israel Deaconess Medical Center, Boston, MA · Department of Psychiatry, Beth Israel Deaconess Medical Center, Boston, MA