Building Multilingual Datasets for Predicting Mental Health Severity through LLMs: Prospects and Challenges
Authors: Konstantinos Skianis, John Pavlopoulos, A. Seza Doğruöz
Organizations: Department of Computer Science and Engineering, University of Ioannina, Greece · Department of Informatics, Athens University of Economics and Business, Greece · Archimedes/AthenaRC, Greece · LT3, IDLab, Universiteit Gent, Belgium
Large Language Models (LLMs) are increasingly being integrated into various medical fields, including mental health support systems. However, there is a gap in research regarding the effectiveness of LLMs in non-English mental health support applications. To address this problem, we present a novel multilingual adaptation of widely-used mental health datasets, translated from English into six languages (e.g., Greek, Turkish, French, Portuguese, German, and Finnish). This dataset enables a comprehensive evaluation of LLM performance in detecting mental health conditions and assessing their severity across multiple languages. By experimenting with GPT and Llama, we observe considerable variability in performance across languages, despite being evaluated on the same translated dataset. This inconsistency underscores the complexities inherent in multilingual mental health support, where language-specific nuances and mental health data coverage can affect the accuracy of the models. Through comprehensive error analysis, we emphasize the risks of relying exclusively on LLMs in medical settings (e.g., their potential to contribute to misdiagnoses). Moreover, our proposed approach offers significant cost savings for multilingual tasks, presenting a major advantage for broad-scale implementation.
Figures & tables
Figure 1: An illustration of our proposed methodology.
Dataset
Reference
Category
#Classes
#Instances
Labels (#Support)
Dep-Sev
Naseem et al. (2022)
Depression
4
3553
Minimum (2587)
Mild (290)
Moderate (394)
Severe (282)
C-ssrs
Gaur et al. (2019)
Suicide
5
500
Supportive (108)
Indicator (99)
Table 1: Benchmark datasets used in our study.
Dataset
Prediction prompt
Dep-Sev
“Categorize the following text with 1 of the 4 depression severity levels (0: Minimum, 1: Mild, 2: Moderate, 3: Severe)”
C-ssrs
“Categorize the content into the following classes based on the user’s mental state: 1. Supportive, 2. Indicator, 3. Ideation, 4. Behavior, 5. Attempt.”
Table 2: The used prediction prompts.
English
Turkish
French
Portuguese
German
Greek
Finnish
Class
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
minimum
0.98
0.14
0.25
0.93
0.03
0.05
0.99
0.15
0.27
0.98
0.11
0.20
0.97
0.24
0.38
0.99
0.07
0.14
0.98
0.08
0.15
mild
0.04
0.15
0.07
0.02
0.01
0.01
0.05
0.17
0.08
0.04
0.07
0.05
0.08
0.32
0.12
0.04
0.17
0.06
0.05
0.19
0.08
moderate
0.13
0.22
0.17
0.10
0.62
0.17
0.15
0.39
0.21
0.12
0.28
0.17
0.16
0.42
0.23
0.14
0.55
0.23
0.12
0.31
0.17
severe
0.13
0.71
0.22
0.13
0.39
0.19
0.15
0.58
0.23
0.12
0.74
0.20
0.16
0.35
0.22
0.16
0.28
0.20
0.12
0.55
0.20
Macro avg
0.32
0.30
0.17
0.29
0.26
0.11
0.33
0.32
0.20
0.31
0.30
0.16
0.34
0.33
0.24
0.33
0.27
0.16
0.32
0.28
0.15
Table 3: GPT-3.5 with 0-shot learning on Dep-Severity , measuring Precision, Recall, and F1 per class per language. The last row shows the macro averages. The best F1 per class is shown in bold. The best macro-averaged F1 across languages is underlined.
English
Turkish
French
Portuguese
German
Greek
Finnish
Class
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Supportive
0.27
0.13
0.18
0.40
0.02
0.04
0.30
0.19
0.24
0.36
0.04
0.07
0.38
0.27
0.31
0.39
0.35
0.37
0.29
0.09
0.14
Indicator
0.21
0.76
0.33
0.20
0.95
0.34
0.21
0.72
0.32
0.21
0.92
0.34
0.27
0.57
0.37
0.22
0.49
0.31
0.22
0.44
0.30
Ideation
0.45
0.26
0.33
0.25
0.05
0.08
0.40
0.21
0.28
0.30
0.09
0.14
0.40
0.49
0.44
0.39
0.42
0.40
0.36
0.56
0.44
Behavior
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.50
0.03
0.05
0.00
0.00
0.00
0.00
0.00
0.00
Attempt
0.00
0.00
0.00
0.25
0.02
0.04
0.00
0.00
0.00
1.00
0.02
0.04
1.00
0.04
0.09
0.00
0.00
0.00
0.00
0.00
0.00
Table 4: GPT-3.5-turbo with 0-shot learning on C-ssrs , measuring Precision, Recall, and F1 per class per language. The last row shows the macro averages. The best F1 per class is shown in bold. The best average F1 across languages is underlined.
English
Turkish
French
Portuguese
German
Greek
Finnish
Class
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
minimum
0.97
0.32
0.48
0.97
0.18
0.30
0.97
0.30
0.46
0.97
0.31
0.47
0.96
0.36
0.52
0.98
0.26
0.41
0.97
0.26
0.40
mild
0.07
0.26
0.11
0.04
0.18
0.07
0.05
0.14
0.07
0.07
0.22
0.11
0.06
0.14
0.08
0.05
0.13
0.07
0.05
0.17
0.07
moderate
0.18
0.41
0.25
0.15
0.42
0.23
0.16
0.50
0.24
0.16
0.41
0.23
0.15
0.42
0.22
0.15
0.52
0.23
0.15
0.48
0.23
severe
0.19
0.43
0.26
0.17
0.48
0.25
0.18
0.44
0.26
0.17
0.49
0.25
0.18
0.48
0.26
0.17
0.40
0.24
0.18
0.40
0.25
Macro avg
0.35
0.35
0.27
0.33
0.31
0.21
0.34
0.34
0.26
0.34
0.36
0.27
0.34
0.35
0.27
0.34
0.33
0.24
0.34
0.33
0.24
Table 5: GPT-4o-mini with 0-shot learning on Dep-Severity , measuring Precision, Recall, and F1 per class per language. The last row shows the macro averages. The best F1 per class is shown in bold. The best macro-averaged F1 across languages is underlined.
English
Turkish
French
Portuguese
German
Greek
Finnish
Class
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Supportive
0.40
0.65
0.50
0.40
0.69
0.51
0.43
0.60
0.50
0.49
0.57
0.53
0.44
0.56
0.50
0.45
0.54
0.49
0.39
0.70
0.50
Indicator
0.26
0.52
0.34
0.27
0.36
0.31
0.27
0.44
0.33
0.29
0.55
0.38
0.24
0.46
0.32
0.27
0.53
0.36
0.21
0.29
0.25
Ideation
0.48
0.20
0.29
0.46
0.35
0.40
0.46
0.32
0.37
0.43
0.26
0.33
0.42
0.36
0.39
0.46
0.46
0.46
0.44
0.37
0.40
Behavior
1.00
0.01
0.03
0.31
0.06
0.11
0.67
0.03
0.05
0.50
0.01
0.03
0.00
0.00
0.00
0.00
0.00
0.00
0.50
0.01
0.03
Attempt
0.39
0.47
0.42
0.30
0.22
0.26
0.31
0.44
0.37
0.34
0.62
0.44
0.42
0.22
0.29
0.57
0.09
0.15
0.35
0.13
0.19
Table 6: GPT-4o-mini with 0-shot learning on C-ssrs , measuring Precision, Recall, and F1 per class per language. The last row shows the macro averages. The best F1 per class is shown in bold. The best average F1 across languages is underlined.
English
French
Portuguese
German
Class
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
minimum
0.88
0.12
0.22
0.91
0.23
0.37
0.95
0.18
0.30
0.95
0.25
0.40
mild
0.04
0.20
0.07
0.04
0.13
0.07
0.05
0.14
0.08
0.05
0.15
0.08
moderate
0.15
0.60
0.25
0.16
0.58
0.24
0.14
0.76
0.24
0.14
0.41
0.21
severe
0.18
0.22
0.20
0.12
0.26
0.17
0.20
0.11
0.14
0.13
0.41
0.20
Macro avg
0.32
0.29
0.19
0.31
0.30
0.21
0.34
0.30
0.19
0.32
0.31
0.22
Table 7: Llama3.1 with 0-shot learning on Dep-Severity , measuring Precision, Recall, and F1 per class per language. The last row shows the macro averages. The best F1 per class is shown in bold. The best average F1 across languages is underlined.
English
Turkish
French
Portuguese
German
Greek
Finnish
Class
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
minimum
0.90
0.14
0.25
0.76
0.23
0.36
0.91
0.11
0.19
0.90
0.11
0.20
0.76
0.75
0.76
0.79
0.02
0.05
0.68
0.16
0.25
mild
0.03
0.06
0.04
0.07
0.10
0.08
0.04
0.04
0.04
0.05
0.14
0.07
0.07
0.08
0.07
0.05
0.10
0.06
0.03
0.08
0.05
moderate
0.14
0.65
0.23
0.11
0.29
0.16
0.13
0.44
0.20
0.15
0.51
0.23
0.16
0.23
0.19
0.14
0.26
0.18
0.13
0.39
0.19
severe
0.15
0.35
0.21
0.09
0.40
0.14
0.11
0.61
0.19
0.13
0.48
0.20
0.16
0.07
0.09
0.10
0.73
0.17
0.10
0.37
0.16
Macro avg
0.30
0.30
0.18
0.26
0.26
0.18
0.30
0.30
0.16
0.31
0.31
0.18
0.29
0.28
0.28
0.27
0.28
0.12
0.23
0.25
0.16
Table 8: GPT-3.5 with 1-shot learning on Dep-Severity by measuring Precision, Recall, and F1 per class per language. The last row shows macro averages for precision, recall, and F1. Best F1 per class is shown in bold. Best average F1 across languages is underlined.
Figure 2: F1 of GPT-3.5 across languages (vertically) per class (box) for Dep-Severity .
Figure 3: F1 of GPT-4o-mini across languages (vertically) per class (box) for C-ssrs .
Turkish
French
Portuguese
German
Greek
Finnish
Class
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Pr
Rec
F1
Supportive
0.43
0.33
0.38
0.37
0.26
0.30
0.44
0.34
0.39
0.37
0.31
0.34
0.36
0.19
0.24
0.40
0.34
0.37
Indicator
0.22
0.77
0.35
0.22
0.75
0.34
0.23
0.76
0.36
0.20
0.64
0.31
0.22
0.79
0.34
0.23
0.74
0.35
Ideation
0.43
0.19
0.27
0.42
0.20
0.27
0.40
0.22
0.28
0.41
0.22
0.29
0.45
0.22
0.29
0.41
0.21
0.28
Behavior
1.00
0.01
0.03
0.67
0.03
0.05
0.00
0.00
0.00
1.00
0.01
0.03
0.50
0.01
0.03
0.00
0.00
0.00
Attempt
0.00
0.00
0.00
0.00
0.00
0.00
1.00
0.04
0.09
0.00
0.00
0.00
0.50
0.02
0.04
1.00
0.04
0.09
Table 9: Back-translation and prediction with GPT-3.5-turbo with 0-shot learning on C-ssrs , measuring Precision, Recall, and F1 per class per language. The last row shows the macro averages. The best F1 per class is shown in bold. The best average F1 across languages is underlined.
Figure 4: An example of translation (English to Greek) from the C-ssrs dataset with GPT-4o-mini.
Department of Computer Science and Engineering, University of Ioannina, Greece · LT3, IDLab, Universiteit Gent, Belgium · Department of Informatics, Athens University of Economics and Business, Greece +1