A Rubric-Guided Large Language Model Solution for Opioid Use Disorder Computable Phenotyping
Authors: Mengxian Lyu, Paredes Pardo, Cheng Peng, Ziyi Chen, Mengyuan Zhang, Jieting Li Lu, Gary M Reisfield, William M Greene, +2 more
Organizations: Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA · Department of Engineering Education, Herbert Wertheim College of Engineering, University of Florida, Gainesville, FL, USA · Department of Psychiatry, College of Medicine, University of Florida, Gainesville, FL, USA · Division of General Internal Medicine, Department of Medicine, University of Pittsburgh School of Medicine, Pittsburgh, PA, USA · Preston A. Wells, Jr. Center for Brain Tumor Therapy, Lillian S. Wells Department of Neurosurgery, University of Florida, Gainesville, Florida, USA
Opioid use disorder (OUD) remains a public health crisis in the United States, yet it is difficult to identify from electronic health records (EHRs) because missing diagnosis codes and supporting evidence are buried in clinical narratives. Accurate OUD identification is critical to support interventions and improve health outcomes. This study developed a rubric-guided large language model (LLM) that incorporated Optimization by PROmpting (OPRO) for OUD computable phenotyping (CP). The framework used an 18-item, expert-identified rubric to instruct LLMs to automatically extract critical text with supporting evidence to determine OUD flags. Two UF Health physicians (GMR and WMG) chart-reviewed 253 patients, including 68 OUD-positive cases. Our LLM-based computable phenotype (CP) achieved the best F1 score of 0.774 and an AUROC of 0.934, outperforming the machine learning-based CP using EHR and natural language processing-extracted variables, and zero-shot LLMs by relative F1 improvements of 12.8% and 44.4%, respectively. The proposed LLM-based CP could link LLM-extracted evidence to OUD phenotyping for better explainability.
Department of Applied Mathematics & Statistics, Stony Brook University, Stony Brook, NY, USA · Department of Computer Science, Stony Brook University, Stony Brook, NY, USA · Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, USA +1
Manning College of Information and Computer Sciences, UMass Amherst, Amherst, MA, USA · Center for Healthcare Organization and Implementation Research, VA Bedford Health Care, Bedford, MA, USA · Miner School of Computer and Information Sciences, UMass Lowell, Lowell, MA, USA