A Dataset and Resources for Identifying Patient Health Literacy Information from Clinical Notes
Authors: Madeline Bittner, Dina Demner-Fushman, Yasmeen Shabazz, Davis Bartels, Dukyong Yoon, Brad Quitadamo, Rajiv Menghrajani, Leo Celi, Sarvesh Soni
Organizations: National Library of Medicine, Bethesda, MD, USA · Massachusetts Institute of Technology (MIT), Cambridge, MA, USA · Yonsei University College of Medicine, Seoul, Republic of Korea · Beth Israel Deaconess Medical Center, Boston, Massachusetts, USA · NYC Health + Hospitals - Lincoln, Bronx, New York, USA
Health literacy is a critical determinant of patient outcomes, yet current screening tools are not always feasible and differ considerably in the number of items, question format, and dimensions of health literacy they capture, making documentation in structured electronic health records difficult to achieve. Automated detection from unstructured clinical notes offers a promising alternative, as these notes often contain richer, more contextual health literacy information, but progress has been limited by the lack of annotated resources. We introduce HEALIX, the first publicly available annotated health literacy dataset derived from real clinical notes, curated through a combination of social worker note sampling, keyword-based filtering, and LLM-based active learning. HEALIX contains 589 notes across 9 note types, annotated with three health literacy labels: low, normal, and high. To demonstrate its utility, we benchmarked zero-shot and few-shot prompting strategies across four open source large language models (LLMs).
Figures & tables
Figure 1: Workflow for dataset development. First, clinical notes were collected from the MIMIC-III database via three sampling strategies: random sampling of social worker notes, keyword-based filtering, and LLM-based active learning. Next, the sampled notes were independently annotated and reconciled to produce the HEALIX gold standard dataset. Finally, zero-shot and few-shot prompting strategies were applied to establish baseline model performance.
adherence
insight (lack of/poor/little/minimal)
refusal
awareness
knowledge / limited knowledge
understands / comprehends
compliance / (non-)compliant
medical advice
understanding (good/poor/limited)
difficulty understanding
misunderstanding (of/the)
unaware (of diagnosis/history)
health literacy / literacy / literate
non(-)adherence / non(-)adherent
well(-)aware
inadequacy
patient choice / patient preference
Table 1: Keywords used in health literacy dataset curation.
Table 2: LLaMA 3-8B Instruct prompt template for health literacy note level classification in the active learning workflow.
Level
Definition
Examples
Low
Patient is unable to understand, interpret, and use health information to make decisions
The patient does not understand the need to follow up with physician.
Normal
An individual is able to obtain, understand, and use health/medical information to make informed health related decisions
The patient is aware of the treatment plan.
High
Patient demonstrates an above average understanding of medical related information and is able to use that information to make informed decisions
The patient demonstrated their medication regimen.
Table 3: Definitions of health literacy levels and examples.
Table 4: Prompt template used for zero-shot and few-shot clinical note strict classification during model evaluation.
Model
Setting
Strict
Lenient
P
R
F1
P
R
F1
SVM
–
0.64
0.39
0.41
0.56
0.49
0.50
LLaMA 3-8B-Instruct
Zero-Shot
0.46
0.47
0.46
0.55
0.54
0.54
Few-Shot
0.45
0.48
0.45
0.55
0.55
0.55
LLaMA 3.3-70B-Instruct
Zero-Shot
0.52
0.51
0.51
0.64
0.62
0.63
Few-Shot
0.47
0.49
0.46
0.61
0.62
0.60
Table 5: Model Performance Evaluation at the strict, 4 classes level, and lenient, 3 classes level.
Figure 2: Confusion matrices for SVM and LLaMA 70B (zero-/few-shot). SVM matrix uses a different color bar than the LLaMA matrices.
Category
Value
Low (N=75)
Normal (N=186)
High (N=45)
Not Related (N=283)
Total (N=589)
Sex
Male
49 (65.3)
112 (60.2)
29 (64.4)
143 (50.5)
333 (56.5)
Female
26 (34.7)
74 (39.8)
16 (35.6)
140 (49.5)
256 (43.5)
Ethnicity
White
47 (62.7)
124 (66.7)
37 (82.2)
196 (69.3)
404 (68.6)
Black/African American
16 (21.3)
17 (9.1)
1 (2.2)
28 (9.9)
62 (10.5)
Hispanic/Latino
3 (4.0)
16 (8.6)
3 (6.7)
17 (6.0)
39 (6.6)
Asian
1 (1.3)
8 (4.3)
3 (6.7)
11 (3.9)
23 (3.9)
Table 6: Patient demographics, encounter characteristics, and clinical note categories in the dataset, stratified by health literacy label (Low, Normal, High, Not Related) and overall (Total). Values are shown as count (%) within column, with percentages rounded to one decimal place.
Figure 3: Annotated clinical note excerpts illustrating examples of varying model classification difficulty. Highlighted text represents evidence of patient health literacy within the note. Any misspellings or abbreviations are reported as they appear in the original note.