Curation and Extraction of Drug-Related Entities from Reddit Platform
Organizations: Population Health Sciences, Weill Cornell Medicine, New York City, USA · School of Computing and Information Systems, University of Melbourne, Melbourne, Australia · Emergency Medicine, Weill Cornell Medicine, New York City, USA
Abstract
Physicians learn primarily about illicit drugs from clinical overdose cases, limiting their understanding of real-world usage. Meanwhile, drug users share first-hand experiences online, offering insights into dosage and effects of drugs. To bridge this gap, we introduce ReDose (REddit Drug DOSe and Effect), a dataset of 6,435 Reddit posts on substance use. A board-certified toxicologist primarily annotated both the training and test sets, while two medical science students contributed to the test set, labeling DRUG, DOSE, and EFFECT entities. We benchmarked 6,267 annotations using BERT-based, large language model (LLM)-based, and Retrieval-Augmented Generation (RAG) models. BiomedBERT achieved an F1-score of 0.843 for DRUG, while Llama-3 70B outperformed GPT-4 (F1 = 0.79 vs. 0.72). EFFECT extraction remains challenging, with GPT-4 achieving a recall of 0.41. ReDose captures patient-curated narratives to advance medical data extraction from social media.
Figures & tables
| Dataset Name | Source | Annotated Drugs | Open source | Social media |
|---|---|---|---|---|
| Graves et al. [ 3 ] | Public posts from the /r/suboxone subreddit of Reddit | Buprenorphine–naloxone | ||
| Leas et al. [ 13 ] | r/CBD (2014–2019) | Cannabidiol | ||
| Spadaro et al. [ 10 ] | r/fentanyl, r/heroin, r/microdosing, r/opiates, r/OpiatesRecovery, r/OurOverUsedVeins, r/suboxone | Buprenorphine and fentanyl | ||
| BC5CDR [ 12 ] | PubMed Articles | Chemicals | ||
| ADE [ 11 ] | MIMIC–III clinical care database | Prescription Medication | ||
| Reddit–Impacts [ 14 ] | Opioid–related subreddits on Reddit | None (impact only) |
| Subreddit | Description | #Posts |
|---|---|---|
| r/fentanyl | Subreddit for the discussion of fentanyl and its many analogues. We focus on harm reduction, and hope to dispel some common myths about these substances. | 10,816 |
| r/heroin | For the junkies and ex-junkies of Reddit: A subreddit for all things heroin. Topics include heroin appreciation, harm reduction, withdrawal, recovery, lifestyle discussions, and more. | 31,322 |
| r/microdosing | This is a community for discussion pertaining to microdosing research, experiments, regimens, and experiences. The most probable candidates for microdosing are psychedelics, but we encourage dialogue on the effects of any drugs at sub-threshold dosage. No sourcing of drugs allowed! Please have a look at the r/microdosing Sidebar. | 42,766 |
| r/opiates | Discussion of all things related to the narcotics known as opiates, from harm-reduction to pharmacology. We are participating in the blackout! We will not be adding approved users to the community as we will return to normal functions tomorrow | 62,138 |
| r/OpiatesRecovery | You are not alone in this fight. We are a group of people dedicated to helping each other stop and stay stopped. | 76,888 |
| r/OurOverUsedVeins | For those of us with an appreciation for heroin, to soothe, help, and give information on the subject of heroin use, and getting past our heroin addictions. Also, share your true stories! As heroin addicts, we’ve been in more terrifying situations than most people. OurOverUsedVeins wants to hear about them! | 79,581 |
| Training | Test | Total | |
| Document | 6,068 | 367 | 6,435 |
| Sentence | 14,260 | 1,209 | 15,469 |
| Entity | |||
| DRUG | 4,004 | 833 | 4,784 |
| EFFECT | 674 | 128 | 750 |
| DOSE | 647 | 98 | 733 |
| Precision | Recall | F1-score | |
| BaseBERT | |||
| DRUG | 0.891 (0.889, 0.893) | 0.873 (0.871, 0.875) | 0.882 (0.880, 0.884) |
| EFFECT | 0.424 (0.411, 0.437) | 0.231 (0.222, 0.240) | 0.298 (0.288, 0.308) |
| DOSE | 0.649 (0.640, 0.658) | 0.752 (0.742, 0.761) | 0.696 (0.688, 0.705) |
| micro avg | 0.843 (0.840, 0.845) | 0.810 (0.808, 0.812) | 0.826 (0.824, 0.829) |
| BioBERT | |||
| Precision | Recall | F1-score | |
|---|---|---|---|
| Llama-3 Instruct 8B | |||
| DRUG | 0.71 | 0.44 | 0.55 |
| EFFECT | 0.03 | 0.11 | 0.05 |
| micro avg | 0.50 | 0.41 | 0.45 |
| Llama-3 Instruct 70B | |||
| DRUG | 0.74 | 0.84 | 0.79 |