Building Trustworthy Mental Health Benchmarks on Bluesky: A Validation-Aware Weak-Supervision Framework

Gaurab Chhetri; Anandi Dutta; Subasish Das

cs.AISep 19, 2026

Building Trustworthy Mental Health Benchmarks on Bluesky: A Validation-Aware Weak-Supervision Framework

Authors: Gaurab Chhetri, Anandi Dutta, Subasish Das

Organizations: Department of Computer Science Texas State University San Marcos, Texas, USA · Ingram School of Engineering Texas State University San Marcos, Texas, USA

Abstract

Decentralized social media platforms create new opportunities and challenges for computational mental health research because data access, moderation, labeling, and deployment responsibilities are distributed across multiple technical and governance layers. This paper presents a validation-aware weak-supervision system for constructing and evaluating suicidal ideation (SI) and broader mental health (MH) disclosure benchmarks on Bluesky, a decentralized social media platform built on the AT Protocol. The system integrates public firehose collection, task-specific lexicon filtering, Llama-3-8B-assisted binary annotation, human-adjudicated validation subsets, and transformer-based model benchmarking. Using this pipeline, we construct two task-specific corpora containing 8,346 SI-labeled posts and 9,988 MH-labeled posts. The evaluation shows that model performance depends strongly on both task definition and validation protocol. BERT+LSTM achieves the highest SI stratified cross-validation F1-score, RoBERTa achieves the strongest SI holdout F1-score, and DistilRoBERTa achieves the best MH cross-validation F1-score. Human validation reveals different weak-label failure modes across tasks, with SI labels dominated by false negatives and MH labels dominated by false positives. These findings show that decentralized social media can support reproducible mental health benchmarking, but only when system design, label provenance, validation strategy, and deployment constraints are evaluated together.

Figures & tables

Figure 1: System architecture for validation-aware weak-supervision benchmarking on Bluesky.

Measure	SI	MH
Validation sample	276	250
Human-human Cohen’s ( $\kappa$ )	0.762	0.595
LLM-human observed agreement	88.8%	55.6%
LLM-human Cohen’s ( $\kappa$ )	0.636	0.222
LLM-human precision	0.947	0.276
LLM-human recall	0.554	0.977

Table 1: Human validation of LLM-assisted weak labels.

Category	Positive examples in scope	Excluded cases
SI	First-person desire to die, passive death wish, current self-harm ideation, planning language, or recent self-directed suicidal intent.	News about suicide, advocacy posts, condolences, quotations, jokes, hypotheticals, and third-person discussion without self-directed ideation.
MH	First-person disclosure of depression, anxiety, panic, hopelessness, distress-related insomnia, help-seeking, or self-harm-related distress.	General awareness posts, clinical information sharing, political commentary, metaphors, sarcasm, and statements about another person’s mental health.
Ambiguous content	Labeled positive only when self-directed disclosure was explicit enough to support the task definition.	Labeled negative when context was insufficient, quoted, purely rhetorical, or not attributable to the author.

Table 2: Annotation boundary rules used for human validation.

Design element	SI	MH
Primary validation	Holdout + CV	5-fold CV
Optimization metric	F1	F1
Model emphasis	broad architecture set	compact encoders
Transformer epochs	3	up to 5
Early stopping	hybrid models	all models
Train batch size	16	8

Table 3: Task-specific evaluation design.

Property	SI	MH
Collection window	14 days	2 months
Labeled posts	8,346	9,988
Positive posts	2,180	4,793
Negative posts	6,166	5,195
Positive rate	26.1%	48.0%
Unique users	6,648	not reported

Table 4: Dataset summary for the two Bluesky benchmarks.

Model	Accuracy	Precision	Recall	F1	Train time (s)
RoBERTa-base	0.892	0.799	0.784	0.792	841
ELECTRA-base	0.891	0.813	0.757	0.784	834
ALBERT-base-v2	0.886	0.789	0.768	0.778	908
BERT-base	0.883	0.785	0.763	0.774	835
DistilBERT-base	0.883	0.780	0.766	0.773	423
XLNet-base	0.884	0.797	0.745	0.770	1359

Table 5: SI holdout validation performance.

Model	Stratified 5-fold		Repeated 5-fold
	Mean F1	SD	Mean F1	SD
BERT+LSTM	0.9162	0.007	0.9148	0.009
RoBERTa-base	0.9017	0.012	0.9024	0.010
BERT-base	0.8986	0.006	0.8993	0.010
XLNet-base	0.8904	0.008	0.8896	0.014
ELECTRA-base	0.8863	0.006	0.8860	0.016

Table 6: SI cross-validation performance.

Figure 2: SI holdout F1 versus stratified cross-validation F1.

Figure 3: SI efficiency summary. Bars show holdout F1; right-side labels report F1, training time, and recall for every model.

Model	F1	Acc.	AUC	Prec.	Time (s)
DistilRoBERTa	0.905	0.909	0.931	0.903	731.78
DistilBERT	0.903	0.907	0.930	0.901	521.05
MiniLM	0.901	0.904	0.928	0.891	263.84
ELECTRA	0.898	0.898	0.913	0.865	272.00
TinyBERT	0.887	0.890	0.908	0.877	171.51

Table 7: MH cross-validation performance (5-fold mean).

Figure 4: MH efficiency summary. Bars show F1-score; right-side labels report F1, training time, and AUC for every model.

Dimension	SI benchmark	MH benchmark
Construct boundary	Narrower; requires self-directed suicidal ideation or passive desire to die.	Broader; includes self-directed mental health symptoms and distress.
Dominant validation risk	False negatives for subtle, indirect, or context-dependent SI.	False positives for awareness, advocacy, metaphor, humor, and third-person discussion.
Best performance signal	BERT+LSTM under stratified CV; RoBERTa under holdout.	DistilRoBERTa under stratified 5-fold CV.
Efficiency signal	DistilBERT gives strong holdout performance at lowest transformer training time.	TinyBERT is fastest, while MiniLM offers a strong speed-performance balance.
Deployment implication	Requires conservative interpretation and high-sensitivity review workflows if used for screening research.	Requires calibration to avoid over-classifying general discourse as self-disclosure.

Table 8: Interpretive comparison of SI and MH benchmark outcomes.

Explore similar work

A Benchmark Suite of Reddit-Derived Datasets for Mental Health Detection
Apr 25, 2026Khalid Hasan, Jamil SaquerMedical Diagnosis Mental Health
Missouri State University, Springfield, MO, USA
HealthBench-Psych: A Mental Health Subset of OpenAI's HealthBench
Aug 25, 2026Matthew Flathers, Phuong Anh Nguyen, Jill Noorily +7Clinical Language Model Evaluation LLM Evaluation
Division of Digital Psychiatry, Beth Israel Deaconess Medical Center, Boston, MA · Department of Psychiatry, Beth Israel Deaconess Medical Center, Boston, MA
FedMental: Evaluating Federated Learning for Mental Health Detection from Social Media Data
May 18, 2026Nuredin Ali Abdelkadir, Anjali Ratnam, Zeerak Talat +1Depression Detection Mental Health
University of Minnesota · University of Edinburgh

Apr 25, 2026J/K move · Enter open · S save

A Benchmark Suite of Reddit-Derived Datasets for Mental Health Detection

Khalid Hasan, Jamil Saquer

Missouri State University, Springfield, MO, USA

Medical Diagnosis Mental Health

Open paper AskPDF arXiv

The growing availability of online support groups has opened up new windows to study mental health through natural language processing (NLP). However, it is hindered by a lack of high-quality, well-validated datasets. Existing studies have a tendency to build task-specific corpora without collecting them into widely available resources, and this makes reproducibility as well as cross-task comparison challenging. In this paper, we present a uniform benchmark set of four Reddit-based datasets for disjoint but complementary tasks: (i) detection of suicidal ideation, (ii) binary general mental disorder detection, (iii) bipolar disorder detection, and (iv) multi-class mental disorder classification. All datasets were established upon diligent linguistic inspection, well-defined annotation guidelines, and human-judgmental verification. Inter-annotator agreement metrics always exceeded the baseline agreement score of 0.8, ensuring the labels' trustworthiness. Previous work's evidence of performance on both transformer and contextualized recurrent models demonstrates that these models receive excellent performances on tasks (F1 ~ 93-99%), further validating the usefulness of the datasets. By combining these resources, we establish a unifying foundation for reproducible mental health NLP studies with the ability to carry out cross-task benchmarking, multi-task learning, and fair model comparison. The presented benchmark suite provides the research community with an easy-to-access and varied resource for advancing computational approaches toward mental health research.

Abstract

Explore similar work

A Benchmark Suite of Reddit-Derived Datasets for Mental Health Detection

HealthBench-Psych: A Mental Health Subset of OpenAI's HealthBench

FedMental: Evaluating Federated Learning for Mental Health Detection from Social Media Data