cs.CLOct 2, 2026

Automatic Evaluation of Mental Health Stigma in Online Communication

Authors: Naomi Baes, Jemima Kang, Nick Haslam, Chris Groot, Alsa Wu, Luc Raszewski, Yulia Otmakhova

Organizations: Melbourne School of Psychological Sciences, The University of Melbourne · Department of General and Computational Linguistics, University of Tübingen · School of Computing and Information Systems, The University of Melbourne

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Mental health stigma has profoundly harmful impacts but its complexity makes it difficult to evaluate. Stigma may involve explicit derogation, but also subtler forms of blame, fear, paternalistic pity, social distancing, structural exclusion, and discrimination. We introduce a theory-grounded benchmark for automatic evaluation of mental health stigma in online communication, consisting of naturally occurring online news and social media text annotated with a fine-grained taxonomy of stigma across multiple mental health conditions. Our annotation framework comprises a binary stigma-detection task and a multi-level taxonomy covering (i) stigma mode, (ii) domain, and (iii) specific components of certain forms of stigma. We apply this framework to texts mentioning six mental health conditions and evaluate large language models alongside stigma-related classifiers for detecting sentiment, toxicity, and hate speech. Results show that mental health stigma is not well captured by models trained to detect these neighboring constructs, and that LLMs often overpredict stigma unless given explicit operational rules - mirroring the importance of decision rules in human annotation. We release the publicly available part of benchmark, annotations, prototypical exemplars of stigma and code at: https://github.com/jemimakang/mh_stigma.

Figures & tables

LabelN
Binary stigma 
Stigma absent383
Stigma present87
Mode 
Expressed/enacted49
Perceived/experienced15
Table 1: Label support in the final benchmark (470 texts). Source and condition breakdowns are given in Table 16 .
   BinaryMacro 
StrategyModelAccPRF1PRF1Bias
baselineClaude Sonnet 4.60.6660.3100.6550.4210.6020.6620.5930.206
Deepseek v4 Flash0.7040.2680.3450.3020.5540.5650.5570.053
Gemini 3 Flash0.7340.3860.7360.5060.6550.7350.6620.168
GPT-5.50.5660.2810.8620.4240.6110.6800.5380.383
definitionClaude Sonnet 4.60.6980.3150.5400.3980.5950.6370.5980.132
Table 2: Stigma presence detection results (accuracy, as well binary and macro-averaged precision, recall, and F1) across experimental settings and models. Binary metrics are reported in terms of the positive class (“stigma present”). Bias = model’s predicted positive rate −- gold stigma-positive rate; positive values (red) = overprediction, negative values (blue) = underprediction. Bracketed values are 95% confidence intervals obtained from bootstrapping (2000 resamples). Best results across all models and conditions are in bold ; bold italics means that the difference with the next-best result for a particular metric is also statistically significant (per-instance McNemar test, χ2\chi^{2} with continuity correction, p < 0.05).
   BinaryMacro 
Test setModelAccPRF1PRF1Bias
Meng’sMeng’s0.9170.9080.9150.9110.9160.9170.9170.004
OursMeng’s (zero-shot)0.3060.1650.6780.2660.4590.4500.3040.574
Meng’s + ours †0.7550.3210.2870.3030.5810.5740.577−- 0.019
Ours †0.7530.3210.2990.3100.5820.5780.580−- 0.013
Table 3: LoRA-tuned classifier results for stigma presence (Qwen2.5-7B). † 5-fold cross-validation on our benchmark; metrics are computed over the pooled out-of-fold predictions from all five folds. Bias = predicted positive rate −- gold positive rate; positive values (red) = overprediction, negative values (blue) = underprediction.
  ModeDomainTypeCognitive
StrategyModelAccF1AccF1AccF1AccF1
baselineClaude Sonnet 4.60.5630.8090.5290.7060.3860.5280.140.49
Deepseek v4 Flash0.5750.7590.4830.5320.3510.4410.120.36
Gemini 3 Flash0.5750.7920.5170.5590.6320.4260.180.48
GPT-5.50.6210.8370.7130.8540.5960.5620.240.50
definitionClaude Sonnet 4.60.5400.7980.5630.6620.4740.4730.220.48
Table 4: Predicting stigma mode (n=87), domain (n=87), type (n=57), and subtype (for Cognitive type only, n=49) (accuracy / macro-F1). The Cognitive subtype task is multi-label, and accuracy is evaluated strictly (all labels match).
ConditionNStig+AccPRF1Bias
ADHD7713.758.344.462.389.060
Alcoholism7812.759.321.400.336.056
Autism7816.746.419.613.494.095
BPD8020.585.295.470.358.150
Depression794.911.400.600.416.048
Schizophrenia7822.562.360.627.447.228
Table 5: Condition-level stigma-presence results for Claude Sonnet 4.6, averaged across the five prompting strategies. N = number of items; Stig+ = gold stigma-positive items. Bias = model’s predicted positive rate −- gold stigma-positive rate; positive values (red) = overprediction.
DifficultyN itemsError rate (mean / median)
Easy232.241 / .100
Moderate190.356 / .275
Hard48.514 / .525
Table 6: Model error rates by item-level annotation difficulty. Error rate is the proportion of incorrect model–prompt predictions for each text, calculated across 20 model–prompt combinations.
Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

ConditionSearch lexicon
ADHDADHD; attention deficit hyperactivity disorder
Alcoholismalcoholism; alcohol abuse; alcohol dependence; alcohol use disorder
Autismautism; autistic; autism spectrum disorder; ASD
BPDborderline personality disorder; borderline PD
Depressiondepression; major depressive disorder; clinical depression
Schizophreniaschizophrenia; schizophrenic
Appendix
Table 7: Surface variants used to retrieve candidate texts for each target mental health condition. Variants were grouped under their associated target condition before sampling and annotation.
TargetArticles
ADHD8,316
alcoholism12,934
autism26,405
borderline personality disorder976
depression106,193
schizophrenia12,088
Appendix
Table 8: Articles per target in the NOW subcorpora
VariantArticles
ADHD7,811
attention deficit hyperactivity disorder1,455
alcoholism8,674
alcohol abuse4,305
alcohol dependence463
alcohol use disorder837
Appendix
Table 9: Articles containing at least one instance of the surface variant in its associated NOW subcorpus
TargetMedian (whole)Median (sampled)
ADHD1,2651,284
Alcoholism1,1421,199
Autism1,050976
Borderline PD1,2331,221
Depression1,1451,071
Schizophrenia1,0841,030
Appendix
Table 10: Article length descriptives (token count) for all articles in the full and randomly sampled NOW corpora per target.
TargetPosts
ADHD219,715
Alcoholism45,712
Autism122,467
Borderline PD91,714
Depression792,623
Schizophrenia31,206
Appendix
Table 11: Posts per target in the Reddit subcorpora
TargetMedianQ3
ADHD9921,792
Alcoholism1,5802,996
Autism1,1422,181
Borderline PD1,2252,328
Depression1,3432,539
Schizophrenia1,2642,675
Appendix
Table 12: Post length descriptives for Reddits subcorpora per target.
LevelTypeDefinition and ExampleSource
Level 2: Stigma Mode
 Expressed/EnactedExpression of stigma or description of its enactment by a stigmatizer. Example: “Police restrained the man using force after noting he had a history of psychiatric admissions.”Fox et al. (2018)
 Perceived/ExperiencedStigma is perceived or its expression experienced by people with lived experience. Example: “I was treated like I was dangerous,” said Mark, who lives with bipolar disorder.Fox et al. (2018)
Level 3: Stigma Domain
 PublicNegative and misinformed stereotyped beliefs, prejudicial emotional responses, or discriminatory behaviors. Example: “Dr White said that, ‘Once people develop mental illness, they will never recover completely.’ ”Thornicroft et al. (2022)
 StructuralSystems, policies, or laws that disempower, discriminate, or limit opportunities for people with lived experience of mental illness. Example: “The report found that mental health patients waited twice as long for emergency care.”Thornicroft et al. (2022)
Appendix
Table 13: Definitions and illustrative examples of stigma expressions across hierarchical levels. Citations indicate the source framework, scale, or measure from which each category or illustrative item was adapted.
Source measure or frameworkCategories informedCitation
Attribution QuestionnaireDangerousness; personal responsibility and blame; uncontrollable; pity/condescension; segregation endorsement; coercion; fear; anger/irritation; pity; social distancingCorrigan et al. (2003)
Depression Stigma ScalePerceived/experienced stigma; controllability; moral or character flaw; trivialization/minimization; contamination; selected dangerousness and blame prototypesGriffiths et al. (2008)
Community Attitudes toward the Mentally Ill scaleStructural stigma; public stigma prototypes; dangerousness; moral or character flaw; trivialization/minimization; segregation; coercion; incompetence; fearTaylor and Dear (1981)
Internalized Stigma of Mental Illness scaleSelf-stigma; internalized stereotypes; affective responses; behavioral withdrawal; immutability; selected incompetence and social-withdrawal prototypesRitsher et al. (2003)
Mental Illness Stigma ScaleImmutability and other general mental-illness stigma components where applicableDay et al. (2007)
Theory-derived definitionsExpressed/enacted stigma; public, structural, and associative stigma; broad cognitive, affective, and behavioral sub-factors; disgust; ambivalencePresent taxonomy
Appendix
Table 14: Source measures and frameworks used to construct the stigma-present and stigma-absent prototype exemplar set.
Figure 2: Annotation interface example.
ProviderModelInference settings
AnthropicClaude Sonnet 4.6 claude-sonnet-4-6Temperature was set to 0. Adaptive/extended thinking was disabled. Safety settings used provider defaults.
OpenAIGPT-5.5 gpt-5.5Temperature was was set to 0. Reasoning effort was not explicitly overridden and therefore used the documented GPT-5.5 default of medium . Reasoning summaries were not requested. Safety settings used provider defaults.
GoogleGemini 3 Flash gemini-3-flash-previewTemperature was set to 0. Dynamic thinking was used at default settings ( high ). Safety settings used provider defaults.
DeepSeekDeepSeek V4 Flash deepseek-v4-flashThe model was used in thinking mode, where temperature and top- pp are not supported. Safety settings used provider defaults.
OpenAIGPT-5.2 Pro gpt-5.2-pro-2025-12-11Used only for LLM-assisted candidate filtering during benchmark construction, not for the main evaluation. Temperature = 0. Reasoning effort was not explicitly overridden in our filtering run. Safety settings used provider defaults.
Appendix
Table 15: Model identifiers and inference settings used in the LLM experiments. All model runs were conducted in May 2026. Unless otherwise stated, decoding, reasoning, and safety parameters were not manually tuned; where provider defaults or wrapper behavior were not fully exposed, we report the settings as unset rather than inferred.
Panel A: Source and condition coverage
 NewsRedditTotal
ConditionNStigma+NStigma+NStigma+
Overall249382214947087
ADHD4263577713
Alcoholism3744187812
Autism36142157816
Appendix
Table 16: Expanded composition of the final benchmark. Panel A reports source-condition coverage using raw text counts. Panel B reports source, condition, and stigma-mode coverage. Panels C and D report taxonomy labels among the 87 stigma-present texts. Counts in Panels C and D refer to label assignments rather than mutually exclusive texts, because stigma-present texts could receive multiple labels. These statistics describe benchmark composition rather than real-world stigma prevalence.
SourceCue triggering overpredictionInterpretation
Reddit / schizophreniaRisk or causal associationThe text asks about a possible causal or risk association between cannabis use and schizophrenia. It mentions a negative outcome, but does not direct a stigmatizing attribution, treatment, or expression toward people with schizophrenia.
Paraphrased text. “someone told me thc only takes 30 days to leave your system no matter what? also heard that if you start smoking weed young you’ll definitely get schizophrenia schizophrenia ”
Reddit / borderline personality disorderClinical and interpersonal-risk framingThis example illustrates a difficult threshold case. The passage contains negative, clinically loaded descriptions of BPD symptoms, but the final adjudicated label was stigma-absent. Models treated the diagnostic and interpersonal-risk framing as sufficient evidence of stigma.
Paraphrased text. “people with borderline personality disorder ( bpd ) deal with really intense emotional swings that can put them in a really desperate headspace. its kind of like when you feel cornered - you lash out at whoever is closest to you that makes you feel threatened. and the thing is, the people who know you best are actually the most vulnerable because sufferers of bpd can use what they know about you against you. so the ones who care the most tend to get hurt the most.”
News / depressionMental health crisis contextThe passage links student behavior to a broader mental health crisis, but it frames disruptive behavior as unmet need rather than as a stigmatizing attribution toward people with depression. Models may have over-relied on negative behavioral language such as “outbursts”, “lashing out”, and “disruptive”.
Full text. “The spike in outbursts, fidgeting and nonstop chatter in classrooms coincides with a national mental health crisis. American pediatricians during the pandemic went as far as to call it a national emergency. A parents with kids under 18 said they are “extremely or very worried that their children might struggle with anxiety or depression at some point.” Teachers are under pressure to help students make up for what they haven’t learned, and kids feel their stress. The combination has led to a pronounced rise in students expressing anger or sadness by lashing out and being disruptive at school. “Anytime we’ve got a chaotic structure and high expectations of ourselves, it’s very easy for that to come out in disruptions,” said Kent McIntosh, an associate professor in the College of Education at the University of Oregon. “What on the outside can look like defiant behavior is sometimes a student’s best bet to get needs met in the moment.”
Appendix
Table 17: Selected false-positive examples. These are stigma-absent texts that one or more models predicted as stigma-present. Reddit examples have been paraphrased to preserve user privacy; news examples are reproduced in full.
ID / sourceCue missed by modelsInterpretation
Reddit / alcoholismSelf-directed shame and guiltThe broader tone is hopeful and recovery-oriented, but the passage contains self-directed shame and guilt related to alcoholism. Models often missed this case, suggesting that self-stigma may be under-detected when embedded in supportive, religious, or non-toxic discourse.
Full text. “heading to church for the first time in a while and i’m actually really looking forward to it. i’d been staying away because i felt so ashamed about my alcoholism but i’ve realised that’s kind of the whole reason i should go. woke up this morning and just prayed for strength and a fresh start. feeling hopeful. hope everyone has a good day”
Reddit / autismStructural exclusion / being left behindThe passage expresses concern about structural exclusion and unequal access to support services for autistic adults and people with disabilities. Models may have missed this case because it appears in advocacy-oriented and institutional language rather than in overtly hostile or derogatory language.
Full text. “Two autistic adults here. We rely on in house daily support. The pandemic really showed how essential these kinds of community-based services are for people with disabilities. There’s a real opportunity right now with the COVID-19 relief package to actually fund these services properly but honestly we’re scared disabled people are going to get overlooked again like always. The Autistic Self Advocacy Network (disability rights advocacy group) has been tracking COVID-19 deaths in care institutions since early 2020, and this research just confirms what we already knew - government decisions literally determine whether people like us get the support we need to survive and live our lives.”
Reddit / ADHDInvalidation of diagnosis / ADHD treated as an excuseThe passage was adjudicated as stigma-present because it describes a family context in which ADHD is dismissed or invalidated: the poster reports that their stepfather has never acknowledged their ADHD and that their mother says it is “an excuse.” Models may have missed this case because the stigma is embedded in a first-person help-seeking post rather than expressed as overt abuse or toxic language.
Full text. “tl;dr: strict home environment is causing me to have regular meltdowns. basically the pressure at home is so intense that i end up breaking down in the bathroom every couple of weeks. the expectations are really high - grades have to be near perfect, chores can’t slip even once, screen time is capped and has to be timed, and i can only use my laptop in a shared space with the screen visible to everyone. i can’t go out without giving my parents my exact route in advance, and i’m still not allowed to get my learner’s permit so i’m getting around on a bike. there’s a lot of other rules on top of all that. the hardest part is that my stepdad completely dismisses my adhd diagnosis that i’ve had since i was a kid, and my mom says i just use it as an excuse. just needed to get that out - does anyone have any advice?”
Appendix
Table 18: Selected false-negative examples. These are stigma-present texts that one or more models predicted as stigma-absent. Reddit examples have been paraphrased to preserve user privacy.
Figure 3: Pairwise agreement matrices for stigma presence.
Figure 4: Pairwise agreement matrices for stigma mode among the items that all annotators judged as stigma-present.
RoundPairnExactAC1
R2A1–A2120.683.521
 A1–A3120.700.514
 A2–A3120.850.805
 Overall120.617.622
R3A1–A2150.693.491
 A1–A3150.540.081
Appendix
Table 19: Pairwise and overall agreement on stigma presence by annotation round. Exact = exact agreement; AC1 = Gwet’s AC1. The A3 position was held by a psychology professor in Round 2 and by a psychology PhD student in Round 3.
   BinaryMacro
StrategyModelAccPRF1PRF1
baselineClaude Sonnet 4.6.67 [.62, .71].31 [.24, .37].66 [.55, .75].42 [.35, .49].60 [.57, .64].66 [.61, .72].59 [.55, .64]
Deepseek v4 Flash.70 [.66, .75].27 [.19, .35].34 [.25, .45].30 [.22, .39].55 [.51, .60].57 [.51, .62].56 [.51, .61]
Gemini 3 Flash.73 [.69, .78].39 [.31, .46].74 [.64, .82].51 [.43, .58].65 [.61, .69].73 [.68, .79].66 [.61, .71]
GPT-5.5.57 [.52, .61].28 [.23, .34].86 [.78, .93].42 [.36, .49].61 [.58, .64].68 [.63, .72].54 [.49, .58]
definitionClaude Sonnet 4.6.70 [.66, .74].32 [.25, .39].54 [.44, .65].40 [.32, .48].60 [.56, .64].64 [.58, .69].60 [.55, .65]
Appendix
Table 20: Stigma presence detection results (accuracy, binary and macro-averaged precision, recall, and F1) across experimental settings and models. Binary metrics are reported in terms of the positive class (“stigma present”). Bracketed values are 95% confidence intervals obtained from bootstrapping (2000 resamples). Best results across all models and conditions are in bold ; bold italics means that the difference with the next-best result for a particular metric is also statistically significant (per-instance McNemar test, χ2\chi^{2} with continuity correction, p < 0.05).
StrategyModelAccuracyPrecisionRecallF1
baselineClaude Sonnet 4.60.563 [.45, .67]0.788 [.73, .85]0.868 [.81, .92]0.809 [.75, .85]
Deepseek v4 Flash0.575 [.47, .68]0.839 [.77, .91]0.698 [.62, .77]0.759 [.69, .82]
Gemini 3 Flash0.575 [.47, .68]0.804 [.74, .87]0.826 [.77, .88]0.792 [.73, .84]
GPT-5.50.621 [.52, .71]0.842 [.78, .91]0.831 [.76, .90]0.837 [.79, .88]
definitionClaude Sonnet 4.60.540 [.43, .64]0.785 [.72, .84]0.848 [.79, .90]0.798 [.75, .84]
Deepseek v4 Flash0.632 [.53, .74]0.848 [.78, .91]0.809 [.74, .88]0.826 [.77, .88]
Appendix
Table 21: Macro-averaged prediction results for stigma mode (n=87). Bracketed values are 95% bootstrap confidence intervals (1000 resamples).
StrategyModelAccuracyPrecisionRecallF1
baselineClaude Sonnet 4.60.529 [.43, .63]0.569 [.40, .73]0.988 [.73, 1.00]0.706 [.49, .83]
Deepseek v4 Flash0.483 [.38, .59]0.461 [.39, .53]0.690 [.65, .73]0.532 [.46, .60]
Gemini 3 Flash0.517 [.41, .62]0.462 [.40, .53]0.960 [.69, .98]0.559 [.47, .64]
GPT-5.50.713 [.62, .80]0.787 [.49, .85]0.982 [.71, 1.00]0.854 [.54, .90]
definitionClaude Sonnet 4.60.563 [.45, .67]0.527 [.40, .72]0.988 [.73, 1.00]0.662 [.49, .82]
Deepseek v4 Flash0.575 [.47, .68]0.481 [.41, .55]0.723 [.70, .75]0.572 [.49, .63]
Appendix
Table 22: Macro-averaged results for stigma domain (n=87). Bracketed values are 95% bootstrap confidence intervals (1000 resamples).
StrategyModelAccuracyPrecisionRecallF1
baselineClaude Sonnet 4.60.386 [.26, .51]0.441 [.37, .52]0.795 [.61, .95]0.528 [.43, .61]
Deepseek v4 Flash0.351 [.25, .47]0.381 [.32, .46]0.655 [.44, .88]0.441 [.35, .54]
Gemini 3 Flash0.632 [.51, .75]0.364 [.30, .44]0.593 [.43, .67]0.426 [.33, .51]
GPT-5.50.596 [.47, .74]0.521 [.39, .68]0.733 [.52, .91]0.562 [.41, .69]
definitionsClaude Sonnet 4.60.474 [.35, .61]0.412 [.32, .51]0.670 [.49, .81]0.473 [.36, .57]
Deepseek v4 Flash0.456 [.33, .60]0.430 [.33, .55]0.666 [.41, .87]0.489 [.35, .63]
Appendix
Table 23: Macro-averaged results for stigma type (Public domain only, n=57). Bracketed values are 95% bootstrap confidence intervals (1000 resamples).
StrategyModelAccPrReF1
baselineClaude Sonnet 4.60.14 [.06,.24]0.57 [.33,.68]0.63 [.39,.76]0.49 [.30,.61]
Deepseek v4 Flash0.12 [.04,.22]0.36 [.22,.45]0.46 [.34,.52]0.36 [.24,.41]
Gemini 3 Flash0.18 [.08,.29]0.55 [.37,.59]0.59 [.40,.68]0.48 [.35,.56]
GPT-5.50.24 [.14,.37]0.58 [.37,.68]0.67 [.41,.75]0.50 [.36,.61]
definitionClaude Sonnet 4.60.22 [.12,.35]0.52 [.34,.61]0.59 [.39,.68]0.48 [.31,.56]
Deepseek v4 Flash0.18 [.08,.31]0.40 [.28,.48]0.53 [.35,.61]0.38 [.26,.47]
Appendix
Table 24: Macro-averaged results for the Cognitive stigma subtype (n=49). Bracketed values are 95% bootstrap confidence intervals (1000 resamples).
  AffectiveBehavioral
StrategyModelAcPrReF1AcPrReF1
basel.Claude0.38 [.12,.75]0.67 [.33,.67]0.39 [.11,.60]0.49 [.17,.63]0.80 [.40,1.00]1.00 [.50,1.00]0.88 [.20,1.00]0.93 [.29,1.00]
Deepseek0.12 [.00,.38]0.61 [.33,.67]0.56 [.19,.89]0.55 [.25,.68]0.20 [.00,.60]0.75 [.00,.83]0.62 [.00,.88]0.53 [.00,.73]
Gemini0.00 [.00,.00]0.00 [.00,.00]0.00 [.00,.00]0.00 [.00,.00]0.80 [.40,1.00]1.00 [.50,1.00]0.88 [.20,1.00]0.93 [.29,1.00]
GPT0.38 [.12,.75]0.67 [.33,.67]0.56 [.17,.67]0.60 [.22,.67]0.80 [.40,1.00]1.00 [.50,1.00]0.88 [.20,1.00]0.93 [.29,1.00]
def.Claude0.12 [.00,.38]0.50 [.00,.56]0.19 [.00,.50]0.27 [.00,.47]0.80 [.40,1.00]1.00 [.50,1.00]0.88 [.20,1.00]0.93 [.29,1.00]
Appendix
Table 25: Macro-averaged results for the Affective (n=8) and Behavioral (n=5) stigma subtypes. Bracketed values are 95% bootstrap confidence intervals (1000 resamples).
StrategyConditionNStig+AccPRF1Bias
baselineADHD77130.7400.3330.5380.4120.104
Alcoholism78120.6540.2410.5830.3410.218
Autism78160.7180.4000.7500.5220.179
BPD80200.5250.2860.6000.3870.275
Depression7940.8480.2501.0000.4000.152
Schizophrenia78220.5130.3260.6820.4410.308
Appendix
Table 26: Condition-level stigma-presence results for Claude Sonnet 4.6 by prompting strategy. N = number of items; Stig+ = gold stigma-positive items. Bias = model’s predicted positive rate −- gold stigma-positive rate; positive values (red) = overprediction, negative values (blue) = underprediction.
Figure 5: Row-normalized model confusion matrix for cognitive public-stigma component prediction.
ConstructModelPurpose
Targeted sentimentDeBERTa-v3 ABSATests whether stigma is captured by negative sentiment toward the mental health target.
General sentimentTwitter-RoBERTa sentimentTests whether stigma is captured by global negative sentiment.
Hate speechTwitter-RoBERTa hate speechTests whether stigma overlaps with explicit hate or derogation.
ToxicityUnbiased Toxic RoBERTaTests whether stigma overlaps with insult, abuse, or toxic language.
Appendix
Table 27: Neighboring-construct baselines used to test whether mental health stigma is captured by sentiment, hate speech, or toxicity models.
Classifier predictiondfχ2\boldsymbol{\chi^{2}}p\boldsymbol{p}
General sentiment22.56.277
Targeted sentiment22.37.306
Hate speech10.06.813
Appendix
Table 28: Chi-square tests of independence between gold stigma labels and classifier predictions for neighboring constructs. Toxicity is not included because it was analyzed as a continuous score rather than a categorical prediction.
Figure 8: Standardized residuals from chi-square tests of independence between gold stigma labels and adjacent classifier predictions. Residuals are shown for predicted general sentiment, targeted sentiment, and hate-speech categories.
Figure 9: Distribution of toxicity scores by gold stigma label. The distributions overlap substantially, suggesting that toxicity scores do not cleanly separate stigma-present from stigma-absent texts.
Figure 10: Prediction of stigma presence, mode, domain and subfactor

Explore similar work

CardsList

  1. Analyzing LLM Reasoning to Uncover Mental Health Stigma

    Apr 27, 2026Sreehari Sankar, Aliakbar Nafar, Mona Barman +8LLM EvaluationSocial Bias in Language Models

    BetterHelp

  2. Language Shapes Mental Health Evaluations in Large Language Models

    Mar 6, 2026Jiayi Xu, Xiyang HuCross-Lingual Consistency in Language ModelsMultilingual Language Model Evaluation

    University of North Carolina at Chapel Hill · Arizona State University

  3. Detecting HIV-Related Stigma in Clinical Narratives Using Large Language Models

    Apr 9, 2026Ziyi Chen, Yasir Khan, Mengyuan Zhang +7Clinical NLPMedical LLMs

    Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA · Department of Epidemiology, College of Public Health and Health Professions, University of Florida, Gainesville, FL, USA · Preston A. Wells, Jr. Center for Brain Tumor Therapy, Lillian S. Wells Department of Neurosurgery, University of Florida, Gainesville, FL, USA