Why Are We Lonely? Leveraging LLMs to Measure and Understand Loneliness in Caregivers and Non-caregivers
Authors: Michelle Damin Kim, Ellie S. Paek, Yufen Lin, Emily Mroz, Jane Chung, Jinho D. Choi
Organizations: Department of Computer Science, Emory University, Atlanta, GA, USA · Nell Hodgson Woodruff School of Nursing, Emory University, Atlanta, GA, USA
This paper presents an LLM-driven approach for constructing diverse social media datasets to measure and compare loneliness in the caregiver and non-caregiver populations. We introduce an expert-developed loneliness evaluation framework and an expert-informed typology for categorizing causes of loneliness for analyzing social media text. Using a human-validated data processing pipeline, we apply GPT-4o, GPT-5-nano, and GPT-5 to build a high-quality Reddit corpus and analyze loneliness across both populations. The loneliness evaluation framework achieved average accuracies of 76.09% and 79.78% for caregivers and non-caregivers, respectively. The cause categorization framework achieved micro-aggregate F1 scores of 0.825 and 0.80 for caregivers and non-caregivers, respectively. Across populations, we observe substantial differences in the distribution of types of causes of loneliness. Caregivers' loneliness were predominantly linked to caregiving roles, identity recognition, and feelings of abandonment, indicating distinct loneliness experiences between the two groups. Demographic extraction further demonstrates the viability of Reddit for building a diverse caregiver loneliness dataset. Overall, this work establishes an LLM-based pipeline for creating high quality social media datasets for studying loneliness and demonstrates its effectiveness in analyzing population-level differences in the manifestation of loneliness.
Figures & tables
Caregiver (n=29)
Non-caregiver(n=31)
Item
Accuracy (%)
Accuracy (%)
1
86.21
93.55
2
75.86
83.87
3
82.76
83.87
4
68.97
83.87
5
68.97
83.87
Table 1: Loneliness evaluation accuracy by item for samples of the caregiver and non-caregiver subreddits.
Caregiver Sample (n=29)
Non-caregiver Sample (n=28)
Type
Precision
Recall
F1
Accuracy
Count
Precision
Recall
F1
Accuracy
Count
Social
0.6
0.75
0.667
0.897
4
0.92
1
0.958
0.929
23
Emotional
0.667
0.8
0.727
0.897
5
0.75
0.857
0.8
0.786
14
Physical
0.667
0.857
0.75
0.862
7
0
0
0
0.893
1
Mental Health
0
0
0
0.931
0
1
0.333
0.5
0.714
12
Relational
0.923
0.923
0.923
0.931
13
0.333
0.5
0.4
0.893
2
Table 2: Metrics for cause of loneliness categorization on the caregiver and non-caregiver sample, with macro-aggregate and micro-aggregate metrics, rounded to the nearest thousandth. Due to an error, 28 of the 31 posts in the non-caregiver sample were utilized.
Type, Related to Caregiving
Precision
Recall
F1
Accuracy
Count
Social
0.5
0.667
0.571
0.897
3
Emotional
0
0
0
0.793
5
Physical
0.333
1
0.5
0.793
3
Mental Health
0
0
0
0.931
0
Relational
0.923
0.857
0.889
0.897
14
Network
0.933
0.824
0.875
0.862
17
Table 3: Metrics for cause of loneliness categorization for identifying the correct type of cause and if a cause is related to the author being a caregiver, on the caregiver sample rounded to the nearest thousandth.
Figure 1: Proportion of all posts with a identified cause(s) of a given type in the caregiver and non-caregiver datasets.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Item
Item
Coding Guidelines
1
The author is unhappy doing so many things alone.
To rate this as "yes," they clearly express feeling alone and unhappy, often mentioning multiple activities or situations they are facing alone in the given context.
14
It is difficult for the author to make friends.
This item pertains to evidence that the caregiver is attempting to reach out and make new connections. Existing friendships or family relationship changes are irrelevant. Connecting with other caregivers is relevant.
15
People are around the author but not with them.
Two issues: does the author have people in their lives, and are they supportive? Being around: physically/socially present. Being “with”: spending quality time, alignment with caregiving approach, and/or feeling emotionally close.
Appendix
Table 4: Loneliness evaluation items and their corresponding coding guidelines.
Type
Criteria
Additional Guidelines
Social
The author’s cause of loneliness MUST be dissatisfaction with their quantity of relationships and/or frequency of social contact.
Often mentioning a lack of people around them or lacking a social network. “Around them” can include physical distance, such as a dissatisfaction with the number of friends nearby.
Emotional
The author’s cause of loneliness MUST be dissatisfaction with the closeness or intimacy of their relationship(s).
Pertaining to a lack of close and intimate relationships. Often mentioning loneliness as a result of an actual and/or perceived loss of a close relationship (death can also be considered the loss of a relationship). Often mentioning loneliness as a result of a desire for a close or intimate relationship and a perceived inability to fulfill that desire.
Appendix
Table 5: Annotation items and corresponding coding guidelines.
Label
Accuracy (%)
Caregiver Gender
89.66
Caregiver Age
91.38
Caregiving Duration
84.48
Patient Gender
94.83
Patient Age
96.55
Diagnosis
84.48
Appendix
Table 6: The accuracy of GPT-4o on the caregiver subreddit sample(n=29) against the expert demographic annotation for each demographic label.
Subreddit
Total
Relevance
Loneliness ≥ 7
AgingParents
5391
355
9
cancer
5346
401
1
CancerCaregivers
1578
970
41
caregivers
1307
548
27
caregiversofreddit
271
41
2
CaregiverSupport
5957
3407
236
Appendix
Table 7: Post counts after each stage of the pipeline for the caregiver subreddits, resulting in the final dataset.
Subreddit
Total
Sample
Relevance
Loneliness ≥ 7
alone
2644
238
217
112
ForeverAlone
1473
152
122
32
loneliness
3132
320
279
101
lonely
9723
839
783
298
lonelywomen
348
28
23
10
mentalhealth
17377
2333
857
274
Appendix
Table 8: Post counts after each stage of pipeline for the non-caregiver subreddits, resulting in the final dataset.
Figure 2: Confusion matrix showing the aggregate accu- racy of GPT-5 for answering the loneliness evaluation framework’s items for the caregiver subreddit sample. Depicts out of all annotator’s labels of a certain label, what proportion was labeled as each label by GPT-5.
Figure 3: Confusion matrix showing the aggregate accu- racy of GPT-5 for answering the loneliness evaluation framework’s items for the non-caregiver subreddit sam- ple. Depicts out of all annotator’s labels of a certain label, what proportion was labeled as each label by GPT- 5.
Figure 4: Distribution of posts, by caregiver age, among known labels.
Figure 5: Distribution of posts, by caregiving duration, among known labels.
Figure 6: Distribution of posts, by caregiving gender, among known labels.
Figure 7: Distribution of caregiver relationship to patient among known labels, by patient.
Figure 8: Distribution of patient ages, among known labels, by patient.
Figure 9: Distribution caregiver categories based on patient diagnosis, among known labels.
Label
Known (%)
Unknown (%)
Caregiver Age
25.3
74.7
Caregiving Duration
43.9
56.1
Caregiver Gender
25.3
74.7
Caregiver to Patient Relationship
95.8
4.2
Patient Age
32.7
67.3
Caregiver Category by Patient Diagnosis
78.6
21.4
Appendix
Table 9: The proportion of posts in the caregiver dataset with the demographic attribute known versus unknown.