SynSym: A Synthetic Data Generation Framework for Psychiatric Symptom Identification

Migyeong Kang; Jihyun Kim; Hyolim Jeon; Sunwoo Hwang; Jihyun An; Yonghoon Kim; Haewoon Kwak; Jisun An

cs.CLMar 23, 2026

SynSym: A Synthetic Data Generation Framework for Psychiatric Symptom Identification

Authors: Migyeong Kang, Jihyun Kim, Hyolim Jeon, Sunwoo Hwang, Jihyun An, Yonghoon Kim, Haewoon Kwak, Jisun An, +1 more

Organizations: Sungkyunkwan University, Seoul, Republic of Korea · Samsung Medical Center, Seoul, Republic of Korea · Omnicns, Seoul, Republic of Korea · Indiana University, Bloomington, IN, United States

Abstract

Psychiatric symptom identification on social media aims to infer fine-grained mental health symptoms from user-generated posts, allowing a detailed understanding of users' mental states. However, the construction of large-scale symptom-level datasets remains challenging due to the resource-intensive nature of expert labeling and the lack of standardized annotation guidelines, which in turn limits the generalizability of models to identify diverse symptom expressions from user-generated text. To address these issues, we propose SynSym, a synthetic data generation framework for constructing generalizable datasets for symptom identification. Leveraging large language models (LLMs), SynSym constructs high-quality training samples by (1) expanding each symptom into sub-concepts to enhance the diversity of generated expressions, (2) producing synthetic expressions that reflect psychiatric symptoms in diverse linguistic styles, and (3) composing realistic multi-symptom expressions, informed by clinical co-occurrence patterns. We validate SynSym on three benchmark datasets covering different styles of depressive symptom expression. Experimental results demonstrate that models trained solely on the synthetic data generated by SynSym perform comparably to those trained on real data, and benefit further from additional fine-tuning with real data. These findings underscore the potential of synthetic data as an alternative resource to real-world annotations in psychiatric symptom modeling, and SynSym serves as a practical framework for generating clinically relevant and realistic symptom expressions.

Figures & tables

Figure 1. The SynSym framework consists of four stages: (1) Symptom Concept Expansion , which decomposes each symptom into fine-grained sub-concepts; (2) Single-Symptom Expression Generation , which produces expressions of individual symptoms in two linguistic styles: clinical and colloquial; (3) Multi-Symptom Expression Generation , which simulates realistic co-occurring symptom expressions based on clinical knowledge; and (4) Synthetic Data Evaluation , which assesses whether the generated expressions accurately reflect real clinical symptoms. The resulting synthetic dataset is then used to train multi-label classification models for identifying psychiatric symptoms from user-generated text.

Statistic	PsySym	PRIMATE	D2S	SynSym
# Samples	1,433	2,003	1,717	18,254
# Symptom Classes	14	9	9	14
Avg. Post Length	18.5	299.3	17.9	23.2
Avg. Symptoms per Sample	1.4	3.5	1.2	1.6
Avg. Samples per Class	143.5	779.3	227.4	2119.5
Max. Samples per Class	348	1,679	527	4,829

Table 1. Summary of three benchmark datasets and our synthetic dataset. Compared to real-world datasets, SynSym provides substantially more training samples per symptom class, alleviating the issue of class imbalance commonly observed in psychiatric symptom corpora.

Dataset	Example of Expression
PsySym ( Zhang et al., 2022 )	I have begun to have suicidal ideation again, something that I have not experienced in a very long time.
PRIMATE ( Gupta et al., 2022 )	So Much Pain In The World I read Tara Condell’s suicide note and it really hit home for me. I have been struggling with suicidal thoughts for years. Sometimes it seems like the little things in life aren’t enough to make up for all the suffering I see in the world. I can’t understand how someone can have 4 houses, take 12 cruises a year and fly around in private jets when everyone else is dying. How can we be this selfish as a species? Darwinism no longer controls us, it’s survival of the wealthy, which I know I will never be.
D2S ( Yadav et al., 2020 )	My mind is quite literally killing me.

Table 2. Representative examples of the Suicidal Ideas symptom from the three benchmark datasets. PsySym contains concise, clinically grounded expressions, whereas D2S uses more figurative and indirect language. PRIMATE includes significantly longer posts that often encapsulate multiple symptom cues within a single entry (e.g., both Suicidal Ideas and Feeling Down are labeled in the presented example).

Component	Expert 1	Expert 2	Agreement
Expanded Sub-Concepts	4.61	4.57	94.86%
Synthetic Expressions	4.99	5.00	99.66%

Table 3. Results of expert validation on the two core components of SynSym . Each item was rated for clinical validity by two licensed psychiatrists on a 5-point Likert scale. Inter-rater agreement represents the proportion of items rated identically by both experts.

Model	PsySym ( Zhang et al., 2022 )		PRIMATE ( Gupta et al., 2022 )		D2S ( Yadav et al., 2020 )
Model	Macro-Rec ↑	Macro-F1 ↑	Macro-Rec ↑	Macro-F1 ↑	Macro-Rec ↑	Macro-F1 ↑
BERT ( Devlin et al., 2018 )	0.738 ± 0.005	0.814 ± 0.004	0.625 ± 0.024	0.646 ± 0.021	0.555 ± 0.046	0.600 ± 0.046
DeBERTa ( He et al., 2020 )	0.747 ± 0.007	0.796 ± 0.003	0.604 ± 0.054	0.614 ± 0.049	0.535 ± 0.041	0.563 ± 0.045
MentalBERT ( Ji et al., 2021 )	0.730 ± 0.005	0.811 ± 0.004	0.629 ± 0.038	0.643 ± 0.031	0.554 ± 0.033	0.603 ± 0.030
GPT4o-ZSL ( Achiam et al., 2023 )	0.774 ± 0.004	0.808 ±0.005	0.569 ± 0.001	0.567 ± 0.000	0.631 ± 0.001	0.588 ± 0.003
GPT4o-COT ( Wei et al., 2022 )	0.775 ± 0.004	0.807 ± 0.002	0.567 ± 0.000	0.568 ± 0.000	0.642 ± 0.003	0.592 ± 0.003

Table 4. Overall performance on three benchmark datasets for psychiatric symptom identification. We adopted MentalBERT ( Ji et al., 2021 ) as the backbone encoder for models trained using SynSym -generated data. SynSym w/o Real refers to models trained solely on synthetic data, while SynSym with Real refers to models initialized with SynSym w/o Real and further fine-tuned on real data. The highest score for each column is shown in bold. For PRIMATE, due to the longer average post length compared to the synthetic dataset, individual sentences were evaluated separately and their predicted symptoms were aggregated for SynSym w/o Real .

Train	Test: PsySym		Test: PRIMATE		Test: D2S
Train	Rec. ↑	F1. ↑	Rec. ↑	F1. ↑	Rec. ↑	F1. ↑
PsySym ( Zhang et al., 2022 )	0.730	0.811	0.466	0.510	0.348	0.380
PRIMATE ( Gupta et al., 2022 )	0.544	0.440	0.629	0.643	0.512	0.340
D2S ( Yadav et al., 2020 )	0.342	0.412	0.404	0.425	0.554	0.603
Multi-Source	0.405	0.476	0.277	0.257	0.441	0.434
SynSym (ours)	0.732	0.778	0.712	0.557	0.525	0.518

Table 5. The results of cross-dataset generalizability, where each row indicates the training dataset and each column indicates the test dataset. Gray-shaded cells indicate cases where the training and test datasets are the same.

Model	PsySym		PRIMATE		D2S
Model	Rec. ↑	F1. ↑	Rec. ↑	F1. ↑	Rec. ↑	F1. ↑
SynSym (ours)	0.798	0.830	0.645	0.650	0.588	0.614
w/o EV	0.785	0.821	0.633	0.640	0.583	0.610
w/o EV + CK	0.770	0.817	0.630	0.633	0.572	0.612
w/o EV + CK + DU	0.755	0.809	0.618	0.629	0.566	0.608
w/o EV + CK + DU + SE	0.759	0.812	0.612	0.625	0.532	0.585

Table 6. Results of the ablation study evaluating the contribution of each component in the SynSym framework. In the w/o CK setting, symptom combinations were generated using only the LLM’s internal knowledge, without any clinically derived co-occurrence pattern.

Figure 2. Extensibility experiment results across three mental health conditions. Bar heights represent the average Macro-F1 score across all their corresponding symptom classes.

Training	PsySym	PRIMATE	D2S
Training	Macro-F1 ↑	Macro-F1 ↑	Macro-F1 ↑
Real only	0.811 ± 0.004	0.643 ± 0.031	0.603 ± 0.030
Back-trans with Real	0.817 ± 0.005	0.641 ± 0.019	0.600 ± 0.034
SynSym with Real	0.830 ± 0.006	0.650 ± 0.016	0.614 ± 0.023

Table 7. Comparison with back-translation augmentation across three benchmark datasets. All models use MentalBERT as the backbone encoder.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3. Prompt for symptom concept expansion. Given a symptom keyword and its brief description as input, the LLM is instructed to generate a list of expanded sub-concepts that reflect more fine-grained aspects of the target symptom.

Figure 4. Prompt for multi-symptom expression generation. Given a set of symptom combinations as input, the LLM generates natural language expressions that simultaneously reflect all symptoms.

DSM-5 Label	PHQ-9 Category
Decreased Energy or Fatigue	Lack of Energy
Depressed Mood	Feeling Down
Genitourinary Symptoms	Lack of Interest
Hyperactivity or Agitation	Hyper/Lower Activity
Inattention	Concentration Problems
Indecisiveness	Concentration Problems

Appendix

Table 8. Label remapping scheme from DSM-5 symptom categories to PHQ-9 symptom categories, used for training and evaluation across datasets with different annotation schemes.

Figure 5. Prompt for evaluating multi-symptom expressions. Given a generated expression and its corresponding symptom labels, the LLM is instructed to assess whether all target symptoms are accurately reflected in the text.

Symptom	# Samples
Anger Irritability	1294
Decreased Energy, Tiredness, Fatigue	2106
Depressed Mood	4829
Genitourinary Symptoms	1186
Hyperactivity Agitation	1569
Inattention	2054

Appendix

Table 9. Label distribution of the synthetic dataset. Depressed Mood appears most commonly as a co-occurring symptom.

Figure 6. Symptom-level comparison between SynSym with Real and MentalBERT on three benchmark datasets: PsySym (left), PRIMATE (center), and D2S (right). SynSym shows higher recall across most symptoms, particularly for low-frequency or underperforming symptom classes.

Symptom	Type	Example
Depressed Mood	Clinical	There is a persistent and pervasive sadness in my emotional state.
Depressed Mood	Colloquial	I’m not feeling anything at all.
Suicidal Ideation	Clinical	I frequently have intrusive thoughts about ending my life.
Suicidal Ideation	Colloquial	I keep thinking that the only way to escape this feeling of being stuck is to end it all.
Genitourinary Symptoms	Clinical	I frequently exhibit nocturia, waking up multiple times throughout the night specifically to urinate.
Genitourinary Symptoms	Colloquial	Sometimes I just pee myself without being able to stop it.

Appendix

Table 10. Representative examples of synthetic symptom expressions generated by SynSym , illustrating both clinical-style and colloquial-style expressions for selected symptoms.

Symptom	Representative Sub-Concept Examples
Depressed Mood	- Persistent feelings of sadness - Experiencing frequent crying spells - Feeling empty
Suicidal Ideas	- Making a suicide attempt - Experiencing fear of dying - Having recurrent thoughts of death
Inattention	- Diminished ability to concentrate - Avoids tasks that require sustained mental effort - Fails to finish duties in the workplace
Sleep Disturbance	- Difficulty falling asleep - Restless, unsatisfying sleep - Insomnia
Loss of Interest or Motivation	- Lose motivation to do things - Feelings of detachment or estrangement from others - Reduced desire to pursue personal goals or ambitions
Worthlessness and Guilt	- Feeling disappointed in oneself - Expecting to be punished - Experiencing guilty preoccupations or ruminations over minor past failings

Appendix

Table 11. Representative examples of sub-concepts generated for selected symptoms, illustrating plausible manifestations of high-level symptom concepts in real-world contexts.

Explore similar work

Knowledge-Guided Retrieval-Augmented Generation for Zero-Shot Psychiatric Data: Privacy Preserving Synthetic Data Generation
Mar 26, 2026Adam Jakobsen, Sushant Gautam, Hugo Lewi Hammer +4Synthetic Tabular Data Generation Privacy-Preserving ML
∗SimulaMet, Norway · †Oslo Metropolitan University, Norway · ‡Uppsala University, Sweden
Learning Evidence of Depression Symptoms via Prompt Induction
Apr 27, 2026Eliseo Bao, Anxo Perez, David Otero +1Depression Detection Mental Health
IRLab, CITIC, Universidade da Coruña A Coruña, Spain
Fidelity, Diversity, and Privacy: A Multi-Dimensional LLM Evaluation for Clinical Data Augmentation
Apr 29, 2026Guillermo Iglesias, Gema Bello-Orgaz, María Navas-Loro +3Synthetic Data Augmentation Synthetic Data Generation
Departamento de Sistemas Informáticos, Escuela Técnica Superior de Ingeniería de Sistemas Informáticos, Universidad Politécnica de Madrid, Spain · Department of Psychiatry, University Hospital Rey Juan Carlos, Mostoles, Spain · Department of Psychiatry, University Hospital Jimenez Diaz Foundation, Madrid, Spain +5

Mar 26, 2026J/K move · Enter open · S save

Knowledge-Guided Retrieval-Augmented Generation for Zero-Shot Psychiatric Data: Privacy Preserving Synthetic Data Generation

Adam Jakobsen, Sushant Gautam, Hugo Lewi Hammer, Susanne Olofsdotter+3

∗SimulaMet, Norway · †Oslo Metropolitan University, Norway · ‡Uppsala University, Sweden

Synthetic Tabular Data Generation Privacy-Preserving ML Synthetic Data Generation for Language Models Retrieval-Augmented Generation

Open paper AskPDF arXiv

AI systems in healthcare research have shown potential to increase patient throughput and assist clinicians, yet progress is constrained by limited access to real patient data. To address this issue, we present a zero-shot, knowledge-guided framework for psychiatric tabular data in which large language models (LLMs) are steered via Retrieval-Augmented Generation using the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) and the International Classification of Diseases (ICD-10). We conducted experiments using different combinations of knowledge bases to generate privacy-preserving synthetic data. The resulting models were benchmarked against two state-of-the-art deep learning models for synthetic tabular data generation, namely CTGAN and TVAE, both of which rely on real data and therefore entail potential privacy risks. Evaluation was performed on six anxiety-related disorders: specific phobia, social anxiety disorder, agoraphobia, generalized anxiety disorder, separation anxiety disorder, and panic disorder. CTGAN typically achieves the best marginals and multivariate structure, while the knowledge-augmented LLM is competitive on pairwise structure and attains the lowest pairwise error in separation anxiety and social anxiety. An ablation study shows that clinical retrieval reliably improves univariate and pairwise fidelity over a no-retrieval LLM. Privacy analyses indicate that the real data-free LLM yields modest overlaps and a low average linkage risk comparable to CTGAN, whereas TVAE exhibits extensive duplication despite a low k-map score. Overall, grounding an LLM in clinical knowledge enables high-quality, privacy-preserving synthetic psychiatric data when real datasets are unavailable or cannot be shared.

Abstract

Explore similar work

Knowledge-Guided Retrieval-Augmented Generation for Zero-Shot Psychiatric Data: Privacy Preserving Synthetic Data Generation

Learning Evidence of Depression Symptoms via Prompt Induction

Fidelity, Diversity, and Privacy: A Multi-Dimensional LLM Evaluation for Clinical Data Augmentation