Language Shapes Mental Health Evaluations in Large Language Models

Jiayi Xu; Xiyang Hu

cs.CLMar 6, 2026

Language Shapes Mental Health Evaluations in Large Language Models

Authors: Jiayi Xu, Xiyang Hu

Organizations: University of North Carolina at Chapel Hill · Arizona State University

Abstract

Multilingual large language models (LLMs) are increasingly used in socially sensitive mental health contexts, including support chatbots, screening, and content moderation. This raises a reliability question: do semantically equivalent mental health inputs elicit comparable evaluations across languages, or systematic shifts consistent with language-associated social and cultural contexts? We examine this question in an English-Chinese setting with GPT-4o and Qwen3-32B using a two-level framework: construct-level evaluative orientation, measured by psychometric stigma instruments, and decision-level behavior, measured by binary stigma detection and four-class depression severity classification. Across instruments and models, Chinese prompts elicit higher stigma-related scores than English prompts. At the decision level, Chinese prompts reduce sensitivity to stigmatizing content and produce more conservative depression severity judgments, leading to more under-estimation errors. These findings show that prompt language can shift both evaluative orientation and downstream behavior in LLM-based mental health evaluation. They highlight the need to evaluate multilingual LLMs not only for aggregate performance, but also for whether they apply comparable evaluative standards across languages in socially sensitive domains.

Figures & tables

Figure 1: Framework for evaluating cross-linguistic variation in mental health assessment. The framework compares English and Chinese input conditions across GPT-4o and Qwen3. Level 1 measures construct-level evaluative orientation using psychometric instruments covering social stigma, self-stigma, and professional stigma. Level 2 evaluates downstream decision behavior through binary mental health stigma detection and four-class depression severity detection.

Figure 2: LLMs express systematically higher perceived public stigma and personal stigma when prompted in Chinese than in English. (a) Perceived public stigma (DDS) Link (1987) ; (b) personal stigma toward individuals with mental illness (MISS) Day et al. (2007) ; (c) scenario-based personal stigma Pescosolido et al. (2021) . Both GPT-4o and Qwen3 scored higher under Chinese prompts (red) than English prompts (blue). Bars: mean scores; error bars: 95% CIs. Two-sided Welch’s t -tests:

{}^{*}p<0.05

;

{}^{**}p<0.01

;

{}^{***}p<0.001

Figure 3: LLMs express systematically higher depression-specific social stigma, self-stigma, and professional stigma when prompted in Chinese than in English. (a) Depression-specific perceived stigma and (b) personal stigma (DSS) Griffiths et al. (2008) ; (c) self-stigma (SSOSH) Vogel et al. (2006) ; (d) professional stigma (OMS-HC) Modgill et al. (2014) . Both GPT-4o and Qwen3 scored higher under Chinese prompts (red) than English prompts (blue). Bars: mean scores; error bars: 95% CIs. Two-sided Welch’s t -tests:

{}^{*}p<0.05

;

{}^{**}p<0.01

;

{}^{***}p<0.001

Model	Metric	Chinese	English	$\Delta$
GPT-4o	Accuracy	0.717	0.737	0.020*
	Precision	0.927	0.943	0.015
	Recall	0.428	0.466	0.039*
	F1-score	0.586	0.624	0.039*
Qwen3	Accuracy	0.722	0.769	0.046***
	Precision	0.927	0.930	0.003

Table 1: Binary mental health stigma detection performance by prompt language.

Figure 4: Chinese prompts produce more under-estimation errors in depression severity detection. Bars show paired discordant under- and over-estimation cases between English and Chinese prompts.

Figure 5: Chinese prompts shift predicted severity downward across severity levels. Mean prediction error (predicted

-

true severity) is shown across true severity levels for English and Chinese prompts. Positive values indicate over-estimation; negative values indicate under-estimation. Error bars indicate 95% CIs.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Measure	Example items
DDS (12 items)	- Most people would willingly accept a former mental patient as a close friend. - Most people believe that a former mental patient is just as trustworthy as the average citizen. (1 = strongly agree, 7 = strongly disagree)
MISS (28 items)	- I would find it difficult to trust someone with a mental illness. - I feel anxious and uncomfortable when I am around someone with a mental illness. (1 = completely disagree, 7 = completely agree)
Vignette (4 vignettes)	- How willing would you be to have the person described in the vignette work closely with you on a job? - How likely is it that the person described in the vignette would do something violent toward other people? (1 = definitely willing / not at all likely, 5 = definitely unwilling / very likely)
DSS (18 items)	Perceived stigma - Most people believe that people with depression are unpredictable. - Most people believe that it is best to avoid people with depression so you do not become depressed yourself. Personal stigma - Depression is a sign of personal weakness. - People with depression are dangerous. (1 = strongly disagree, 6 = strongly agree)

Appendix

Table A1: Social stigma

Measure	Example items
SSOSH (10 items)	- I would feel inadequate if I went to a therapist for psychological help. - It would make me feel inferior to ask a therapist for help. (1 = strongly disagree, 5 = strongly agree)

Appendix

Table A2: Self-stigma

Measure	Example items
OMS-HC (15 items)	- I am more comfortable helping a person who has a physical illness than I am helping a person who has a mental illness. - Despite my professional beliefs, I have negative reactions towards people who have mental illness. (1 = strongly disagree, 5 = strongly agree)

Appendix

Table A3: Health professional stigma

Dimension	Measure	Chinese	English	$\Delta$
Social	DDS	3.85 (0.55)	3.65 (0.69)	+0.20***
	MISS	4.44 (0.89)	4.39 (0.83)	+0.05*
	Vignette	2.92 (0.89)	2.61 (1.14)	+0.31***
	DSS-perc.	2.73 (1.03)	2.21 (0.80)	+0.52***
	DSS-pers.	1.55 (0.83)	1.33 (0.47)	+0.22***
Self	SSOSH	2.86 (0.41)	2.33 (0.55)	+0.53***

Appendix

Table A4: GPT-4o stigma scores by prompt language.

Dimension	Measure	Chinese	English	$\Delta$
Social	DDS	3.30 (0.46)	3.30 (0.46)	+0.00
	MISS	4.53 (1.23)	4.38 (1.12)	+0.15***
	Vignette	3.12 (0.86)	2.91 (1.10)	+0.21***
	DSS-perc.	3.14 (0.84)	3.00 (0.82)	+0.14***
	DSS-pers.	2.44 (1.16)	2.00 (1.07)	+0.44***
Self	SSOSH	2.41 (0.67)	2.20 (0.87)	+0.21***

Appendix

Table A5: Qwen3 stigma scores by prompt language.

Dimension	Measure	Chinese	English	$\Delta$
Social	DDS	3.80 (0.54)	3.73 (0.68)	+0.07**
	MISS	4.37 (0.92)	4.39 (0.85)	-0.02
	Vignette	2.91 (0.90)	2.58 (1.11)	+0.33***
	DSS-perc.	2.74 (1.04)	2.24 (0.82)	+0.50***
	DSS-pers.	1.60 (0.84)	1.36 (0.49)	+0.24***
Self	SSOSH	2.85 (0.46)	2.32 (0.57)	+0.53***

Appendix

Table A6: GPT-4o stigma scores by prompt language at temperature

T=1.0

Dimension	Measure	Chinese	English	$\Delta$
Social	DDS	3.26 (0.44)	3.28 (0.49)	-0.02
	MISS	4.55 (1.17)	4.25 (1.20)	+0.30***
	Vignette	3.13 (0.87)	2.91 (1.06)	+0.22***
	DSS-perc.	3.16 (0.77)	2.97 (0.81)	+0.19***
	DSS-pers.	2.64 (1.05)	2.01 (1.18)	+0.63***
Self	SSOSH	2.36 (0.74)	2.16 (0.84)	+0.20***

Appendix

Table A7: Qwen3 stigma scores by prompt language at temperature

T=1.0

Model	Metric	Significance test
GPT-4o	Accuracy	McNemar: $\chi^{2}_{1}=4.34$ , $p=0.037$ ; $b=38$ , $c=21$
	Precision	Bootstrap: $p=0.381$ ; 95% CI $[-0.019,0.051]$
	Recall	McNemar: $\chi^{2}_{1}=4.17$ , $p=0.041$ ; $b=31$ , $c=16$
	F1-score	Bootstrap: $p=0.021$ ; 95% CI $[0.005,0.073]$
Qwen3	Accuracy	McNemar: $\chi^{2}_{1}=19.01$ , $p<0.001$ ; $b=60$ , $c=20$
	Precision	Bootstrap: $p=0.547$ ; 95% CI $[-0.022,0.043]$

Appendix

Table A8: Significance tests for stigma detection.

Model	Metric	EN	ZH	$\Delta$
GPT-4o	Accuracy	0.760 $\pm$ 0.003	0.677 $\pm$ 0.004	0.084
	Precision	0.934 $\pm$ 0.004	0.830 $\pm$ 0.010	0.104
	Recall	0.525 $\pm$ 0.005	0.389 $\pm$ 0.006	0.136
	F1	0.672 $\pm$ 0.005	0.530 $\pm$ 0.006	0.142
Qwen3	Accuracy	0.761 $\pm$ 0.008	0.693 $\pm$ 0.002	0.068
	Precision	0.886 $\pm$ 0.004	0.810 $\pm$ 0.007	0.076

Appendix

Table A9: Robustness check without role-framing (Task 1).

Model	Metric	EN	ZH	$\Delta$
GPT-4o	Accuracy	0.753 $\pm$ 0.008	0.735 $\pm$ 0.004	0.018
	Precision	0.947 $\pm$ 0.013	0.943 $\pm$ 0.009	0.004
	Recall	0.502 $\pm$ 0.026	0.463 $\pm$ 0.005	0.040
	F1	0.656 $\pm$ 0.019	0.621 $\pm$ 0.006	0.035
Qwen3	Accuracy	0.764 $\pm$ 0.003	0.719 $\pm$ 0.012	0.045
	Precision	0.901 $\pm$ 0.018	0.898 $\pm$ 0.011	0.003

Appendix

Table A10: Robustness check using an alternative translator (Task 1).

Model	Lang.	Acc.	Macro-P	Macro-R	Macro-F1
GPT-4o	EN	0.446	0.452	0.440	0.432
	ZH	0.433	0.438	0.428	0.426
Qwen3	EN	0.447	0.446	0.440	0.419
	ZH	0.417	0.413	0.409	0.390

Appendix

Table A11: Overall classification performance.

Model	Lang.	Signed err.	MAE	Over	Under
GPT-4o	EN	0.294	0.766	0.369	0.185
	ZH	0.212	0.782	0.350	0.217
Qwen3	EN	0.487	0.817	0.429	0.124
	ZH	0.447	0.859	0.429	0.154

Appendix

Table A12: Bias metrics for depression severity prediction by language.

Model	Test	Result
GPT-4o	Signed error	Mean diff. = 0.082; paired $t=5.051$ , $p=5.22\times 10^{-7}$ ; Wilcoxon $W=5340$ , $p=8.30\times 10^{-7}$ ; bootstrap CI $[0.050,0.113]$
	Over-est.	EN-only $b=38$ , ZH-only $c=19$ , $p=0.016$
	Under-est.	EN-only $b=11$ , ZH-only $c=43$ , $p=1.40\times 10^{-5}$
Qwen3	Signed error	Mean diff. = 0.040; paired $t=2.122$ , $p=0.0341$ ; Wilcoxon $W=7806$ , $p=0.0306$ ; bootstrap CI $[0.004,0.077]$
	Over-est.	EN-only $b=36$ , ZH-only $c=36$ , $p=1.00$
	Under-est.	EN-only $b=17$ , ZH-only $c=46$ , $p=3.37\times 10^{-4}$

Appendix

Table A13: Cross-linguistic tests for prediction bias.

Model	Test	Result
GPT-4o	Signed error	Mean diff. = 0.118; bootstrap CI $[0.085,0.151]$
	Over-est.	EN-only $b=46$ , ZH-only $c=20$ ; direction: EN more often over-estimates
	Under-est.	EN-only $b=9$ , ZH-only $c=46$ ; direction: ZH more often under-estimates
Qwen3	Signed error	Mean diff. = 0.075; bootstrap CI $[0.038,0.112]$
	Over-est.	EN-only $b=35$ , ZH-only $c=30$ ; direction: EN slightly more often over-estimates
	Under-est.	EN-only $b=12$ , ZH-only $c=40$ ; direction: ZH more often under-estimates

Appendix

Table A14: Task 2 robustness check without role-framing: paired bias-shift results.

Model	Test	Result
GPT-4o	Signed error	Mean diff. = 0.070; bootstrap CI $[0.042,0.098]$
	Over-est.	EN-only $b=32$ , ZH-only $c=13$ ; direction: EN more often over-estimates
	Under-est.	EN-only $b=15$ , ZH-only $c=24$ ; direction: ZH more often under-estimates
Qwen3	Signed error	Mean diff. = 0.045; bootstrap CI $[0.011,0.078]$
	Over-est.	EN-only $b=35$ , ZH-only $c=35$ ; direction: equal over-estimation rates
	Under-est.	EN-only $b=10$ , ZH-only $c=33$ ; direction: ZH more often under-estimates

Appendix

Table A15: Task 2 robustness check using an alternative Chinese translation pipeline: paired bias-shift results.

Explore similar work

Somatic in the East, Psychological in the West?: Investigating Clinically-Grounded Cross-Cultural Depression Symptom Expression in LLMs
Aug 5, 2025Shintaro Sakai, Jisun An, Migyeong Kang +1Mental Health Cultural Bias in Language Models
Indiana University Bloomington, USA · Sungkyunkwan University, Republic of Korea
Analyzing LLM Reasoning to Uncover Mental Health Stigma
Apr 27, 2026Sreehari Sankar, Aliakbar Nafar, Mona Barman +8LLM Evaluation Social Bias in Language Models
BetterHelp
Using LLM-as-a-Judge/Jury to Advance Scalable, Clinically-Validated Safety Evaluations of Model Responses to Users Demonstrating Psychosis
Mar 20, 2026May Lynn Reese, Markela Zeneli, Mindy Ng +3LLM Safety LLM Safety Evaluation
Apart Research · Odyssean Institute · London School of Economics and Political Science +1

Aug 5, 2025J/K move · Enter open · S save

Somatic in the East, Psychological in the West?: Investigating Clinically-Grounded Cross-Cultural Depression Symptom Expression in LLMs

Shintaro Sakai, Jisun An, Migyeong Kang, Haewoon Kwak

Indiana University Bloomington, USA · Sungkyunkwan University, Republic of Korea

Mental Health Cultural Bias in Language Models Clinical Language Model Evaluation

Open paper AskPDF arXiv

Prior clinical psychology research shows that Western individuals with depression tend to report psychological symptoms, while Eastern individuals report somatic ones. We test whether Large Language Models (LLMs), which are increasingly used in mental health, reproduce these cultural patterns by prompting them with Western or Eastern personas. Results show that LLMs largely fail to replicate the patterns when prompted in English, though prompting in major Eastern languages (i.e., Chinese, Japanese, and Hindi) improves alignment in several configurations. Our analysis pinpoints two key reasons for this failure: the models' low sensitivity to cultural personas and a strong, culturally invariant symptom hierarchy that overrides cultural cues. These findings reveal that while prompt language is important, current general-purpose LLMs lack the robust, culture-aware capabilities essential for safe and effective mental health applications.

Abstract

Explore similar work

Somatic in the East, Psychological in the West?: Investigating Clinically-Grounded Cross-Cultural Depression Symptom Expression in LLMs

Analyzing LLM Reasoning to Uncover Mental Health Stigma

Using LLM-as-a-Judge/Jury to Advance Scalable, Clinically-Validated Safety Evaluations of Model Responses to Users Demonstrating Psychosis