Automatic Evaluation of Mental Health Stigma in Online Communication

Naomi Baes; Jemima Kang; Nick Haslam; Chris Groot; Alsa Wu; Luc Raszewski; Yulia Otmakhova

cs.CLOct 2, 2026

Automatic Evaluation of Mental Health Stigma in Online Communication

Authors: Naomi Baes, Jemima Kang, Nick Haslam, Chris Groot, Alsa Wu, Luc Raszewski, Yulia Otmakhova

Organizations: Melbourne School of Psychological Sciences, The University of Melbourne · Department of General and Computational Linguistics, University of Tübingen · School of Computing and Information Systems, The University of Melbourne

Abstract

Mental health stigma has profoundly harmful impacts but its complexity makes it difficult to evaluate. Stigma may involve explicit derogation, but also subtler forms of blame, fear, paternalistic pity, social distancing, structural exclusion, and discrimination. We introduce a theory-grounded benchmark for automatic evaluation of mental health stigma in online communication, consisting of naturally occurring online news and social media text annotated with a fine-grained taxonomy of stigma across multiple mental health conditions. Our annotation framework comprises a binary stigma-detection task and a multi-level taxonomy covering (i) stigma mode, (ii) domain, and (iii) specific components of certain forms of stigma. We apply this framework to texts mentioning six mental health conditions and evaluate large language models alongside stigma-related classifiers for detecting sentiment, toxicity, and hate speech. Results show that mental health stigma is not well captured by models trained to detect these neighboring constructs, and that LLMs often overpredict stigma unless given explicit operational rules - mirroring the importance of decision rules in human annotation. We release the publicly available part of benchmark, annotations, prototypical exemplars of stigma and code at: https://github.com/jemimakang/mh_stigma.

Figures & tables

Label	N
Binary stigma
Stigma absent	383
Stigma present	87
Mode
Expressed/enacted	49
Perceived/experienced	15

Table 1: Label support in the final benchmark (470 texts). Source and condition breakdowns are given in Table 16 .

Strategy	Model	Acc	P	R	F1	P	R	F1	Bias
			Binary			Macro
baseline	Claude Sonnet 4.6	0.666	0.310	0.655	0.421	0.602	0.662	0.593	0.206
	Deepseek v4 Flash	0.704	0.268	0.345	0.302	0.554	0.565	0.557	0.053
	Gemini 3 Flash	0.734	0.386	0.736	0.506	0.655	0.735	0.662	0.168
	GPT-5.5	0.566	0.281	0.862	0.424	0.611	0.680	0.538	0.383
definition	Claude Sonnet 4.6	0.698	0.315	0.540	0.398	0.595	0.637	0.598	0.132

Table 2: Stigma presence detection results (accuracy, as well binary and macro-averaged precision, recall, and F1) across experimental settings and models. Binary metrics are reported in terms of the positive class (“stigma present”). Bias = model’s predicted positive rate

-

gold stigma-positive rate; positive values (red) = overprediction, negative values (blue) = underprediction. Bracketed values are 95% confidence intervals obtained from bootstrapping (2000 resamples). Best results across all models and conditions are in bold ; bold italics means that the difference with the next-best result for a particular metric is also statistically significant (per-instance McNemar test,

\chi^{2}

with continuity correction, p < 0.05).

Test set	Model	Acc	P	R	F1	P	R	F1	Bias
			Binary			Macro
Meng’s	Meng’s	0.917	0.908	0.915	0.911	0.916	0.917	0.917	0.004
Ours	Meng’s (zero-shot)	0.306	0.165	0.678	0.266	0.459	0.450	0.304	0.574
	Meng’s + ours †	0.755	0.321	0.287	0.303	0.581	0.574	0.577	$-$ 0.019
	Ours †	0.753	0.321	0.299	0.310	0.582	0.578	0.580	$-$ 0.013

Table 3: LoRA-tuned classifier results for stigma presence (Qwen2.5-7B). † 5-fold cross-validation on our benchmark; metrics are computed over the pooled out-of-fold predictions from all five folds. Bias = predicted positive rate

-

gold positive rate; positive values (red) = overprediction, negative values (blue) = underprediction.

		Mode		Domain		Type		Cognitive
Strategy	Model	Acc	F1	Acc	F1	Acc	F1	Acc	F1
baseline	Claude Sonnet 4.6	0.563	0.809	0.529	0.706	0.386	0.528	0.14	0.49
	Deepseek v4 Flash	0.575	0.759	0.483	0.532	0.351	0.441	0.12	0.36
	Gemini 3 Flash	0.575	0.792	0.517	0.559	0.632	0.426	0.18	0.48
	GPT-5.5	0.621	0.837	0.713	0.854	0.596	0.562	0.24	0.50
definition	Claude Sonnet 4.6	0.540	0.798	0.563	0.662	0.474	0.473	0.22	0.48

Table 4: Predicting stigma mode (n=87), domain (n=87), type (n=57), and subtype (for Cognitive type only, n=49) (accuracy / macro-F1). The Cognitive subtype task is multi-label, and accuracy is evaluated strictly (all labels match).

Condition	N	Stig+	Acc	P	R	F1	Bias
ADHD	77	13	.758	.344	.462	.389	.060
Alcoholism	78	12	.759	.321	.400	.336	.056
Autism	78	16	.746	.419	.613	.494	.095
BPD	80	20	.585	.295	.470	.358	.150
Depression	79	4	.911	.400	.600	.416	.048
Schizophrenia	78	22	.562	.360	.627	.447	.228

Table 5: Condition-level stigma-presence results for Claude Sonnet 4.6, averaged across the five prompting strategies. N = number of items; Stig+ = gold stigma-positive items. Bias = model’s predicted positive rate

-

gold stigma-positive rate; positive values (red) = overprediction.

Difficulty	N items	Error rate (mean / median)
Easy	232	.241 / .100
Moderate	190	.356 / .275
Hard	48	.514 / .525

Table 6: Model error rates by item-level annotation difficulty. Error rate is the proportion of incorrect model–prompt predictions for each text, calculated across 20 model–prompt combinations.

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Condition	Search lexicon
ADHD	ADHD; attention deficit hyperactivity disorder
Alcoholism	alcoholism; alcohol abuse; alcohol dependence; alcohol use disorder
Autism	autism; autistic; autism spectrum disorder; ASD
BPD	borderline personality disorder; borderline PD
Depression	depression; major depressive disorder; clinical depression
Schizophrenia	schizophrenia; schizophrenic

Appendix

Table 7: Surface variants used to retrieve candidate texts for each target mental health condition. Variants were grouped under their associated target condition before sampling and annotation.

Target	Articles
ADHD	8,316
alcoholism	12,934
autism	26,405
borderline personality disorder	976
depression	106,193
schizophrenia	12,088

Appendix

Table 8: Articles per target in the NOW subcorpora

Variant	Articles
ADHD	7,811
attention deficit hyperactivity disorder	1,455
alcoholism	8,674
alcohol abuse	4,305
alcohol dependence	463
alcohol use disorder	837

Appendix

Table 9: Articles containing at least one instance of the surface variant in its associated NOW subcorpus

Target	Median (whole)	Median (sampled)
ADHD	1,265	1,284
Alcoholism	1,142	1,199
Autism	1,050	976
Borderline PD	1,233	1,221
Depression	1,145	1,071
Schizophrenia	1,084	1,030

Appendix

Table 10: Article length descriptives (token count) for all articles in the full and randomly sampled NOW corpora per target.

Target	Posts
ADHD	219,715
Alcoholism	45,712
Autism	122,467
Borderline PD	91,714
Depression	792,623
Schizophrenia	31,206

Appendix

Table 11: Posts per target in the Reddit subcorpora

Target	Median	Q3
ADHD	992	1,792
Alcoholism	1,580	2,996
Autism	1,142	2,181
Borderline PD	1,225	2,328
Depression	1,343	2,539
Schizophrenia	1,264	2,675

Appendix

Table 12: Post length descriptives for Reddits subcorpora per target.

Level	Type	Definition and Example	Source
Level 2: Stigma Mode
	Expressed/Enacted	Expression of stigma or description of its enactment by a stigmatizer. Example: “Police restrained the man using force after noting he had a history of psychiatric admissions.”	Fox et al. (2018)
	Perceived/Experienced	Stigma is perceived or its expression experienced by people with lived experience. Example: “I was treated like I was dangerous,” said Mark, who lives with bipolar disorder.	Fox et al. (2018)
Level 3: Stigma Domain
	Public	Negative and misinformed stereotyped beliefs, prejudicial emotional responses, or discriminatory behaviors. Example: “Dr White said that, ‘Once people develop mental illness, they will never recover completely.’ ”	Thornicroft et al. (2022)
	Structural	Systems, policies, or laws that disempower, discriminate, or limit opportunities for people with lived experience of mental illness. Example: “The report found that mental health patients waited twice as long for emergency care.”	Thornicroft et al. (2022)

Appendix

Table 13: Definitions and illustrative examples of stigma expressions across hierarchical levels. Citations indicate the source framework, scale, or measure from which each category or illustrative item was adapted.

Source measure or framework	Categories informed	Citation
Attribution Questionnaire	Dangerousness; personal responsibility and blame; uncontrollable; pity/condescension; segregation endorsement; coercion; fear; anger/irritation; pity; social distancing	Corrigan et al. (2003)
Depression Stigma Scale	Perceived/experienced stigma; controllability; moral or character flaw; trivialization/minimization; contamination; selected dangerousness and blame prototypes	Griffiths et al. (2008)
Community Attitudes toward the Mentally Ill scale	Structural stigma; public stigma prototypes; dangerousness; moral or character flaw; trivialization/minimization; segregation; coercion; incompetence; fear	Taylor and Dear (1981)
Internalized Stigma of Mental Illness scale	Self-stigma; internalized stereotypes; affective responses; behavioral withdrawal; immutability; selected incompetence and social-withdrawal prototypes	Ritsher et al. (2003)
Mental Illness Stigma Scale	Immutability and other general mental-illness stigma components where applicable	Day et al. (2007)
Theory-derived definitions	Expressed/enacted stigma; public, structural, and associative stigma; broad cognitive, affective, and behavioral sub-factors; disgust; ambivalence	Present taxonomy

Appendix

Table 14: Source measures and frameworks used to construct the stigma-present and stigma-absent prototype exemplar set.

Provider	Model	Inference settings
Anthropic	Claude Sonnet 4.6 claude-sonnet-4-6	Temperature was set to 0. Adaptive/extended thinking was disabled. Safety settings used provider defaults.
OpenAI	GPT-5.5 gpt-5.5	Temperature was was set to 0. Reasoning effort was not explicitly overridden and therefore used the documented GPT-5.5 default of medium . Reasoning summaries were not requested. Safety settings used provider defaults.
Google	Gemini 3 Flash gemini-3-flash-preview	Temperature was set to 0. Dynamic thinking was used at default settings ( high ). Safety settings used provider defaults.
DeepSeek	DeepSeek V4 Flash deepseek-v4-flash	The model was used in thinking mode, where temperature and top- $p$ are not supported. Safety settings used provider defaults.
OpenAI	GPT-5.2 Pro gpt-5.2-pro-2025-12-11	Used only for LLM-assisted candidate filtering during benchmark construction, not for the main evaluation. Temperature = 0. Reasoning effort was not explicitly overridden in our filtering run. Safety settings used provider defaults.

Appendix

Table 15: Model identifiers and inference settings used in the LLM experiments. All model runs were conducted in May 2026. Unless otherwise stated, decoding, reasoning, and safety parameters were not manually tuned; where provider defaults or wrapper behavior were not fully exposed, we report the settings as unset rather than inferred.

Panel A: Source and condition coverage
	News		Reddit		Total
Condition	N	Stigma+	N	Stigma+	N	Stigma+
Overall	249	38	221	49	470	87
ADHD	42	6	35	7	77	13
Alcoholism	37	4	41	8	78	12
Autism	36	1	42	15	78	16

Appendix

Table 16: Expanded composition of the final benchmark. Panel A reports source-condition coverage using raw text counts. Panel B reports source, condition, and stigma-mode coverage. Panels C and D report taxonomy labels among the 87 stigma-present texts. Counts in Panels C and D refer to label assignments rather than mutually exclusive texts, because stigma-present texts could receive multiple labels. These statistics describe benchmark composition rather than real-world stigma prevalence.

Source	Cue triggering overprediction	Interpretation
Reddit / schizophrenia	Risk or causal association	The text asks about a possible causal or risk association between cannabis use and schizophrenia. It mentions a negative outcome, but does not direct a stigmatizing attribution, treatment, or expression toward people with schizophrenia.
Paraphrased text. “someone told me thc only takes 30 days to leave your system no matter what? also heard that if you start smoking weed young you’ll definitely get schizophrenia schizophrenia ”
Reddit / borderline personality disorder	Clinical and interpersonal-risk framing	This example illustrates a difficult threshold case. The passage contains negative, clinically loaded descriptions of BPD symptoms, but the final adjudicated label was stigma-absent. Models treated the diagnostic and interpersonal-risk framing as sufficient evidence of stigma.
Paraphrased text. “people with borderline personality disorder ( bpd ) deal with really intense emotional swings that can put them in a really desperate headspace. its kind of like when you feel cornered - you lash out at whoever is closest to you that makes you feel threatened. and the thing is, the people who know you best are actually the most vulnerable because sufferers of bpd can use what they know about you against you. so the ones who care the most tend to get hurt the most.”
News / depression	Mental health crisis context	The passage links student behavior to a broader mental health crisis, but it frames disruptive behavior as unmet need rather than as a stigmatizing attribution toward people with depression. Models may have over-relied on negative behavioral language such as “outbursts”, “lashing out”, and “disruptive”.
Full text. “The spike in outbursts, fidgeting and nonstop chatter in classrooms coincides with a national mental health crisis. American pediatricians during the pandemic went as far as to call it a national emergency. A parents with kids under 18 said they are “extremely or very worried that their children might struggle with anxiety or depression at some point.” Teachers are under pressure to help students make up for what they haven’t learned, and kids feel their stress. The combination has led to a pronounced rise in students expressing anger or sadness by lashing out and being disruptive at school. “Anytime we’ve got a chaotic structure and high expectations of ourselves, it’s very easy for that to come out in disruptions,” said Kent McIntosh, an associate professor in the College of Education at the University of Oregon. “What on the outside can look like defiant behavior is sometimes a student’s best bet to get needs met in the moment.”

Appendix

Table 17: Selected false-positive examples. These are stigma-absent texts that one or more models predicted as stigma-present. Reddit examples have been paraphrased to preserve user privacy; news examples are reproduced in full.

ID / source	Cue missed by models	Interpretation
Reddit / alcoholism	Self-directed shame and guilt	The broader tone is hopeful and recovery-oriented, but the passage contains self-directed shame and guilt related to alcoholism. Models often missed this case, suggesting that self-stigma may be under-detected when embedded in supportive, religious, or non-toxic discourse.
Full text. “heading to church for the first time in a while and i’m actually really looking forward to it. i’d been staying away because i felt so ashamed about my alcoholism but i’ve realised that’s kind of the whole reason i should go. woke up this morning and just prayed for strength and a fresh start. feeling hopeful. hope everyone has a good day”
Reddit / autism	Structural exclusion / being left behind	The passage expresses concern about structural exclusion and unequal access to support services for autistic adults and people with disabilities. Models may have missed this case because it appears in advocacy-oriented and institutional language rather than in overtly hostile or derogatory language.
Full text. “Two autistic adults here. We rely on in house daily support. The pandemic really showed how essential these kinds of community-based services are for people with disabilities. There’s a real opportunity right now with the COVID-19 relief package to actually fund these services properly but honestly we’re scared disabled people are going to get overlooked again like always. The Autistic Self Advocacy Network (disability rights advocacy group) has been tracking COVID-19 deaths in care institutions since early 2020, and this research just confirms what we already knew - government decisions literally determine whether people like us get the support we need to survive and live our lives.”
Reddit / ADHD	Invalidation of diagnosis / ADHD treated as an excuse	The passage was adjudicated as stigma-present because it describes a family context in which ADHD is dismissed or invalidated: the poster reports that their stepfather has never acknowledged their ADHD and that their mother says it is “an excuse.” Models may have missed this case because the stigma is embedded in a first-person help-seeking post rather than expressed as overt abuse or toxic language.
Full text. “tl;dr: strict home environment is causing me to have regular meltdowns. basically the pressure at home is so intense that i end up breaking down in the bathroom every couple of weeks. the expectations are really high - grades have to be near perfect, chores can’t slip even once, screen time is capped and has to be timed, and i can only use my laptop in a shared space with the screen visible to everyone. i can’t go out without giving my parents my exact route in advance, and i’m still not allowed to get my learner’s permit so i’m getting around on a bike. there’s a lot of other rules on top of all that. the hardest part is that my stepdad completely dismisses my adhd diagnosis that i’ve had since i was a kid, and my mom says i just use it as an excuse. just needed to get that out - does anyone have any advice?”

Appendix

Table 18: Selected false-negative examples. These are stigma-present texts that one or more models predicted as stigma-absent. Reddit examples have been paraphrased to preserve user privacy.

Figure 3: Pairwise agreement matrices for stigma presence.

Figure 4: Pairwise agreement matrices for stigma mode among the items that all annotators judged as stigma-present.

Round	Pair	n	Exact	AC1
R2	A1–A2	120	.683	.521
	A1–A3	120	.700	.514
	A2–A3	120	.850	.805
	Overall	120	.617	.622
R3	A1–A2	150	.693	.491
	A1–A3	150	.540	.081

Appendix

Table 19: Pairwise and overall agreement on stigma presence by annotation round. Exact = exact agreement; AC1 = Gwet’s AC1. The A3 position was held by a psychology professor in Round 2 and by a psychology PhD student in Round 3.

			Binary			Macro
Strategy	Model	Acc	P	R	F1	P	R	F1
baseline	Claude Sonnet 4.6	.67 [.62, .71]	.31 [.24, .37]	.66 [.55, .75]	.42 [.35, .49]	.60 [.57, .64]	.66 [.61, .72]	.59 [.55, .64]
	Deepseek v4 Flash	.70 [.66, .75]	.27 [.19, .35]	.34 [.25, .45]	.30 [.22, .39]	.55 [.51, .60]	.57 [.51, .62]	.56 [.51, .61]
	Gemini 3 Flash	.73 [.69, .78]	.39 [.31, .46]	.74 [.64, .82]	.51 [.43, .58]	.65 [.61, .69]	.73 [.68, .79]	.66 [.61, .71]
	GPT-5.5	.57 [.52, .61]	.28 [.23, .34]	.86 [.78, .93]	.42 [.36, .49]	.61 [.58, .64]	.68 [.63, .72]	.54 [.49, .58]
definition	Claude Sonnet 4.6	.70 [.66, .74]	.32 [.25, .39]	.54 [.44, .65]	.40 [.32, .48]	.60 [.56, .64]	.64 [.58, .69]	.60 [.55, .65]

Appendix

Table 20: Stigma presence detection results (accuracy, binary and macro-averaged precision, recall, and F1) across experimental settings and models. Binary metrics are reported in terms of the positive class (“stigma present”). Bracketed values are 95% confidence intervals obtained from bootstrapping (2000 resamples). Best results across all models and conditions are in bold ; bold italics means that the difference with the next-best result for a particular metric is also statistically significant (per-instance McNemar test,

\chi^{2}

with continuity correction, p < 0.05).

Strategy	Model	Accuracy	Precision	Recall	F1
baseline	Claude Sonnet 4.6	0.563 [.45, .67]	0.788 [.73, .85]	0.868 [.81, .92]	0.809 [.75, .85]
	Deepseek v4 Flash	0.575 [.47, .68]	0.839 [.77, .91]	0.698 [.62, .77]	0.759 [.69, .82]
	Gemini 3 Flash	0.575 [.47, .68]	0.804 [.74, .87]	0.826 [.77, .88]	0.792 [.73, .84]
	GPT-5.5	0.621 [.52, .71]	0.842 [.78, .91]	0.831 [.76, .90]	0.837 [.79, .88]
definition	Claude Sonnet 4.6	0.540 [.43, .64]	0.785 [.72, .84]	0.848 [.79, .90]	0.798 [.75, .84]
	Deepseek v4 Flash	0.632 [.53, .74]	0.848 [.78, .91]	0.809 [.74, .88]	0.826 [.77, .88]

Appendix

Table 21: Macro-averaged prediction results for stigma mode (n=87). Bracketed values are 95% bootstrap confidence intervals (1000 resamples).

Strategy	Model	Accuracy	Precision	Recall	F1
baseline	Claude Sonnet 4.6	0.529 [.43, .63]	0.569 [.40, .73]	0.988 [.73, 1.00]	0.706 [.49, .83]
	Deepseek v4 Flash	0.483 [.38, .59]	0.461 [.39, .53]	0.690 [.65, .73]	0.532 [.46, .60]
	Gemini 3 Flash	0.517 [.41, .62]	0.462 [.40, .53]	0.960 [.69, .98]	0.559 [.47, .64]
	GPT-5.5	0.713 [.62, .80]	0.787 [.49, .85]	0.982 [.71, 1.00]	0.854 [.54, .90]
definition	Claude Sonnet 4.6	0.563 [.45, .67]	0.527 [.40, .72]	0.988 [.73, 1.00]	0.662 [.49, .82]
	Deepseek v4 Flash	0.575 [.47, .68]	0.481 [.41, .55]	0.723 [.70, .75]	0.572 [.49, .63]

Appendix

Table 22: Macro-averaged results for stigma domain (n=87). Bracketed values are 95% bootstrap confidence intervals (1000 resamples).

Strategy	Model	Accuracy	Precision	Recall	F1
baseline	Claude Sonnet 4.6	0.386 [.26, .51]	0.441 [.37, .52]	0.795 [.61, .95]	0.528 [.43, .61]
	Deepseek v4 Flash	0.351 [.25, .47]	0.381 [.32, .46]	0.655 [.44, .88]	0.441 [.35, .54]
	Gemini 3 Flash	0.632 [.51, .75]	0.364 [.30, .44]	0.593 [.43, .67]	0.426 [.33, .51]
	GPT-5.5	0.596 [.47, .74]	0.521 [.39, .68]	0.733 [.52, .91]	0.562 [.41, .69]
definitions	Claude Sonnet 4.6	0.474 [.35, .61]	0.412 [.32, .51]	0.670 [.49, .81]	0.473 [.36, .57]
	Deepseek v4 Flash	0.456 [.33, .60]	0.430 [.33, .55]	0.666 [.41, .87]	0.489 [.35, .63]

Appendix

Table 23: Macro-averaged results for stigma type (Public domain only, n=57). Bracketed values are 95% bootstrap confidence intervals (1000 resamples).

Strategy	Model	Acc	Pr	Re	F1
baseline	Claude Sonnet 4.6	0.14 [.06,.24]	0.57 [.33,.68]	0.63 [.39,.76]	0.49 [.30,.61]
	Deepseek v4 Flash	0.12 [.04,.22]	0.36 [.22,.45]	0.46 [.34,.52]	0.36 [.24,.41]
	Gemini 3 Flash	0.18 [.08,.29]	0.55 [.37,.59]	0.59 [.40,.68]	0.48 [.35,.56]
	GPT-5.5	0.24 [.14,.37]	0.58 [.37,.68]	0.67 [.41,.75]	0.50 [.36,.61]
definition	Claude Sonnet 4.6	0.22 [.12,.35]	0.52 [.34,.61]	0.59 [.39,.68]	0.48 [.31,.56]
	Deepseek v4 Flash	0.18 [.08,.31]	0.40 [.28,.48]	0.53 [.35,.61]	0.38 [.26,.47]

Appendix

Table 24: Macro-averaged results for the Cognitive stigma subtype (n=49). Bracketed values are 95% bootstrap confidence intervals (1000 resamples).

		Affective				Behavioral
Strategy	Model	Ac	Pr	Re	F1	Ac	Pr	Re	F1
basel.	Claude	0.38 [.12,.75]	0.67 [.33,.67]	0.39 [.11,.60]	0.49 [.17,.63]	0.80 [.40,1.00]	1.00 [.50,1.00]	0.88 [.20,1.00]	0.93 [.29,1.00]
	Deepseek	0.12 [.00,.38]	0.61 [.33,.67]	0.56 [.19,.89]	0.55 [.25,.68]	0.20 [.00,.60]	0.75 [.00,.83]	0.62 [.00,.88]	0.53 [.00,.73]
	Gemini	0.00 [.00,.00]	0.00 [.00,.00]	0.00 [.00,.00]	0.00 [.00,.00]	0.80 [.40,1.00]	1.00 [.50,1.00]	0.88 [.20,1.00]	0.93 [.29,1.00]
	GPT	0.38 [.12,.75]	0.67 [.33,.67]	0.56 [.17,.67]	0.60 [.22,.67]	0.80 [.40,1.00]	1.00 [.50,1.00]	0.88 [.20,1.00]	0.93 [.29,1.00]
def.	Claude	0.12 [.00,.38]	0.50 [.00,.56]	0.19 [.00,.50]	0.27 [.00,.47]	0.80 [.40,1.00]	1.00 [.50,1.00]	0.88 [.20,1.00]	0.93 [.29,1.00]

Appendix

Table 25: Macro-averaged results for the Affective (n=8) and Behavioral (n=5) stigma subtypes. Bracketed values are 95% bootstrap confidence intervals (1000 resamples).

Strategy	Condition	N	Stig+	Acc	P	R	F1	Bias
baseline	ADHD	77	13	0.740	0.333	0.538	0.412	0.104
	Alcoholism	78	12	0.654	0.241	0.583	0.341	0.218
	Autism	78	16	0.718	0.400	0.750	0.522	0.179
	BPD	80	20	0.525	0.286	0.600	0.387	0.275
	Depression	79	4	0.848	0.250	1.000	0.400	0.152
	Schizophrenia	78	22	0.513	0.326	0.682	0.441	0.308

Appendix

Table 26: Condition-level stigma-presence results for Claude Sonnet 4.6 by prompting strategy. N = number of items; Stig+ = gold stigma-positive items. Bias = model’s predicted positive rate

-

gold stigma-positive rate; positive values (red) = overprediction, negative values (blue) = underprediction.

Panel of Figure 5 — Figure 5: Row-normalized model confusion matrix for cognitive public-stigma component prediction.

Construct	Model	Purpose
Targeted sentiment	DeBERTa-v3 ABSA	Tests whether stigma is captured by negative sentiment toward the mental health target.
General sentiment	Twitter-RoBERTa sentiment	Tests whether stigma is captured by global negative sentiment.
Hate speech	Twitter-RoBERTa hate speech	Tests whether stigma overlaps with explicit hate or derogation.
Toxicity	Unbiased Toxic RoBERTa	Tests whether stigma overlaps with insult, abuse, or toxic language.

Appendix

Table 27: Neighboring-construct baselines used to test whether mental health stigma is captured by sentiment, hate speech, or toxicity models.

Classifier prediction	df	$\boldsymbol{\chi^{2}}$	$\boldsymbol{p}$
General sentiment	2	2.56	.277
Targeted sentiment	2	2.37	.306
Hate speech	1	0.06	.813

Appendix

Table 28: Chi-square tests of independence between gold stigma labels and classifier predictions for neighboring constructs. Toxicity is not included because it was analyzed as a continuous score rather than a categorical prediction.

Panel of Figure 8 — Figure 8: Standardized residuals from chi-square tests of independence between gold stigma labels and adjacent classifier predictions. Residuals are shown for predicted general sentiment, targeted sentiment, and hate-speech categories.

Figure 9: Distribution of toxicity scores by gold stigma label. The distributions overlap substantially, suggesting that toxicity scores do not cleanly separate stigma-present from stigma-absent texts.

Figure 10: Prediction of stigma presence, mode, domain and subfactor

Explore similar work

Apr 27, 2026

Analyzing LLM Reasoning to Uncover Mental Health Stigma

While large language models (LLMs) are increasingly being explored for mental health applications, recent studies reveal that they can exhibit stigma toward individuals with psychological conditions. Existing evaluations of this stigma primarily rely on multiple-choice questions (MCQs), which fail to capture the biases embedded within the models' underlying logic. In this paper, we analyze the intermediate reasoning steps of LLMs to uncover hidden stigmatizing language and the internal rationales driving it. We leverage clinical expertise to categorize common patterns of stigmatizing language directed at individuals with psychological conditions and use this framework to identify and tag problematic statements in LLM reasoning. Furthermore, we rate the severity of these statements, distinguishing between overt prejudice and more subtle, less immediately harmful biases. To broaden the reasoning domain and capture a wider array of patterns, we also extend an existing mental health stigma benchmark by incorporating additional psychological conditions. Our findings demonstrate that evaluating model reasoning not only exposes substantially more stigma than traditional MCQ-based methods but also helps identify the flaws in the LLMs' logic and their understanding of mental health conditions.

Sreehari Sankar, Aliakbar Nafar, Mona Barman +8

BetterHelp

LLM Evaluation Social Bias in Language Models Clinical Language Model Evaluation

Ask PDF

Mar 6, 2026

Language Shapes Mental Health Evaluations in Large Language Models

Multilingual large language models (LLMs) are increasingly used in socially sensitive mental health contexts, including support chatbots, screening, and content moderation. This raises a reliability question: do semantically equivalent mental health inputs elicit comparable evaluations across languages, or systematic shifts consistent with language-associated social and cultural contexts? We examine this question in an English-Chinese setting with GPT-4o and Qwen3-32B using a two-level framework: construct-level evaluative orientation, measured by psychometric stigma instruments, and decision-level behavior, measured by binary stigma detection and four-class depression severity classification. Across instruments and models, Chinese prompts elicit higher stigma-related scores than English prompts. At the decision level, Chinese prompts reduce sensitivity to stigmatizing content and produce more conservative depression severity judgments, leading to more under-estimation errors. These findings show that prompt language can shift both evaluative orientation and downstream behavior in LLM-based mental health evaluation. They highlight the need to evaluate multilingual LLMs not only for aggregate performance, but also for whether they apply comparable evaluative standards across languages in socially sensitive domains.

Jiayi Xu, Xiyang Hu

University of North Carolina at Chapel Hill · Arizona State University

Cross-Lingual Consistency in Language Models Multilingual Language Model Evaluation Clinical Language Model Evaluation Depression Severity Estimation

Ask PDF

Apr 9, 2026

Detecting HIV-Related Stigma in Clinical Narratives Using Large Language Models

Human immunodeficiency virus (HIV)-related stigma is a critical psychosocial determinant of health for people living with HIV (PLWH), influencing mental health, engagement in care, and treatment outcomes. Although stigma-related experiences are documented in clinical narratives, there is a lack of off-the-shelf tools to extract and categorize them. This study aims to develop a large language model (LLM)-based tool for identifying HIV stigma from clinical notes. We identified clinical notes from PLWH receiving care at the University of Florida (UF) Health between 2012 and 2022. Candidate sentences were identified using expert-curated stigma-related keywords and iteratively expanded via clinical word embeddings. A total of 1,332 sentences were manually annotated across four stigma subscales: Concern with Public Attitudes, Disclosure Concerns, Negative Self-Image, and Personalized Stigma. We compared GatorTron-large and BERT as encoder-based baselines, and GPT-OSS-20B, LLaMA-8B, and MedGemma-27B as generative LLMs, under zero-shot and few-shot prompting. GatorTron-large achieved the best overall performance (Micro F1 = 0.62). Few-shot prompting substantially improved generative model performance, with 5-shot GPT-OSS-20B and LLaMA-8B achieving Micro-F1 scores of 0.57 and 0.59, respectively. Performance varied by stigma subscale, with Negative Self-Image showing the highest predictability and Personalized Stigma remaining the most challenging. Zero-shot generative inference exhibited non-trivial failure rates (up to 32%). This study develops the first practical NLP tool for identifying HIV stigma in clinical notes.

Ziyi Chen, Yasir Khan, Mengyuan Zhang +7

Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA · Department of Epidemiology, College of Public Health and Health Professions, University of Florida, Gainesville, FL, USA · Preston A. Wells, Jr. Center for Brain Tumor Therapy, Lillian S. Wells Department of Neurosurgery, University of Florida, Gainesville, FL, USA

Clinical NLP Medical LLMs

Ask PDF