PVminer: A Domain-Specific Tool to Detect the Patient Voice in Patient Generated Data
Authors: Samah Fodeh, Linhai Ma, Yan Wang, Srivani Talakokkul, Ganesh Puthiaraju, Afshan Khan, Ashley Hagaman, Sarah Lowe, Aimee Roundtree
Organizations: Department Of Emergency Medicine, Yale School of Medicine, 464 Congress Ave, 06519, CT, USA · Department of Biomedical Informatics & Data Science, Yale School of Medicine, 100 College Street, 06510, CT, USA · Department of Social and Behavioral Sciences, Yale School of Public Health, 60 College Street, 06520, CT, USA · Division of Research, Texas State University, 601 University Dr., 78666, TX, USA
Patient-generated text such as secure messages, surveys, and interviews contains rich expressions of the patient voice (PV), reflecting communicative behaviors and social determinants of health (SDoH). Traditional qualitative coding frameworks are labor intensive and do not scale to large volumes of patient-authored messages across health systems. Existing machine learning (ML) and natural language processing (NLP) approaches provide partial solutions but often treat patient-centered communication (PCC) and SDoH as separate tasks or rely on models not well suited to patient-facing language. We introduce PVminer, a domain-adapted NLP framework for structuring patient voice in secure patient-provider communication. PVminer formulates PV detection as a multi-label, multi-class prediction task integrating patient-specific BERT encoders (PV-BERT-base and PV-BERT-large), unsupervised topic modeling for thematic augmentation (PV-Topic-BERT), and fine-tuned classifiers for Code, Subcode, and Combo-level labels. Topic representations are incorporated during fine-tuning and inference to enrich semantic inputs. PVminer achieves strong performance across hierarchical tasks and outperforms biomedical and clinical pre-trained baselines, achieving F1 scores of 82.25% (Code), 80.14% (Subcode), and up to 77.87% (Combo). An ablation study further shows that author identity and topic-based augmentation each contribute meaningful gains. Pre-trained models, source code, and documentation will be publicly released, with annotated datasets available upon request for research use.
Figures & tables
Figure 1: Overview of the PVminer framework. (1) A BERT baseline model is domain-adapted by pre-training on 6M unlabeled patient-authored messages using a masked language modeling objective, producing PV-BERT-base and PV-BERT-large. (2) A BERTopic model is trained on 500K unlabeled messages to learn latent thematic structure and generate topic keywords, producing PV-Topic-BERT. (3) During fine-tuning, PV-BERT models are augmented with topic cues from the pre-trained PV-Topic-BERT model and trained on annotated patient voice data to predict structured labels (Code, Subcode, Combo). (4) During inference, given a new message, topic cues are again retrieved from the PV-Topic-BERT model, and the fine-tuned PV-BERT produces the structured patient voice representation.
Dataset
Sex (M)
Sex (F)
Race(W)
Race(B)
Race(A)
Race(O)
Ethnicity(Hispanic)
YNHH
81 (38%)
132 (62%)
152 (71%)
35 (16%)
11 (5%)
15 (7%)
30 (14%)
TXACC Woven
29 (48%)
32 (52%)
8 (13%)
6 (10%)
4 (7%)
43 (70%)
24 (39%)
TXACC Bethesda
71 (47%)
79 (53%)
20 (13%)
15 (10%)
9 (6%)
106 (71%)
60 (40%)
Survey
38 (26%)
109 (74%)
109 (74%)
8 (5%)
19 (13%)
11 (8%)
8 (5%)
Total
219 (38%)
352 (62%)
289 (51%)
64 (11%)
43 (8%)
175 (31%)
122 (21%)
Table 1: Demographic Distribution across Datasets. Some short terms are used: For the Sex, M is Male and F is Famale; For the Race, W is White, B is Black, A is Asian and O is Others.
Item
Information
#Messages
1137
#Provider’s Message
380
#Patient’s Message
757
#Total Words
46038
Ave. Words ± std
40.49 ± 32.84
Max Words
261
Table 2: The statistics of the annotated secure message
Figure 2: Distribution of Code categories across all annotated patient messages. Each message may contain multiple non-overlapping labels.
Figure 3: Distribution of Subcodes representing fine-grained PV categories.
Figure 4: Distribution of Combo labels (Code–Subcode pairs) in the annotated dataset. Absent Subcodes are labeled as None.
Table 3: Representative topics learned by the PV-Topic-BERT model. Topics reflect clinical, communication, and social themes commonly found in patient-authored messages.
Model
Precision mean
Precision std
Recall mean
Recall std
F1 mean
F1 std
dmis-lab/biobert- v1.1
86.95
0.24
71.78
0.70
78.64
0.49
allenai/scibert_ scivocab_uncased
85.96
0.45
72.50
0.95
78.65
0.38
emilyalsentzer/ Bio_ClinicalBERT
86.52
0.44
70.64
0.86
77.77
0.39
cambridgeltl/SapBERT- from-PubMedBERT-fulltext
84.93
0.30
71.90
0.52
77.87
0.42
bert-base-uncased
88.52
0.64
74.61
0.47
80.97
0.05
bert-large-uncased
86.01
0.70
75.15
1.77
80.21
1.25
Table 4: Model performance on Code-level classification. Each reported result is averaged over five independent runs. Mean and standard deviation are provided to indicate the variability of each method. The best mean value in each column is highlighted in bold. All models are inside the PVminer.
Model
Precision mean
Precision std
Recall mean
Recall std
F1 mean
F1 std
dmis-lab/biobert- v1.1
81.65
0.39
72.85
1.47
76.99
0.83
allenai/scibert_ scivocab_uncased
81.07
0.48
73.27
0.75
76.97
0.59
emilyalsentzer/ Bio_ClinicalBERT
81.50
0.96
71.63
1.05
76.24
0.84
cambridgeltl/SapBERT- from-PubMedBERT-fulltext
80.09
0.30
72.20
0.42
75.94
0.37
bert-base-uncased
81.90
0.93
74.80
1.33
78.18
0.95
bert-large-uncased
80.88
1.48
75.64
3.63
78.15
2.63
Table 5: Model performance on Subcode-level classification. Each reported result is averaged over five independent runs. Mean and standard deviation are provided to indicate the variability of each method. The best mean value in each column is highlighted in bold. All models are inside the PVminer.
Model
Precision mean
Precision std
Recall mean
Recall std
F1 mean
F1 std
dmis-lab/biobert- v1.1
79.67
0.46
70.32
1.17
74.70
0.68
allenai/scibert_ scivocab_uncased
79.16
0.32
70.55
0.69
74.61
0.50
emilyalsentzer/ Bio_ClinicalBERT
79.45
0.64
69.63
0.70
74.21
0.49
cambridgeltl/SapBERT- from-PubMedBERT-fulltext
78.01
0.07
69.76
0.00
73.66
0.03
bert-base-uncased
80.74
0.36
72.63
0.92
76.47
0.52
bert-large-uncased
78.63
1.25
72.56
3.82
75.45
2.64
Table 6: Model performance on Combo-level classification. Each reported result is averaged over five independent runs. Mean and standard deviation are provided to indicate the variability of each method. The best mean value in each column is highlighted in bold. All models are inside the PVminer.
Class Label
Precision mean
Precision std
Recall mean
Recall std
F1 mean
F1 std
All
89.47
0.76
76.11
1.16
82.25
0.41
CareCoordinationPatient
86.44
7.40
62.39
1.48
72.33
1.75
CareCoordinationProvider
76.60
1.56
68.57
2.86
72.35
2.02
PartnershipPatient
93.16
0.71
88.81
1.21
90.93
0.30
PartnershipProvider
97.97
1.86
86.81
1.10
92.04
0.62
SDOH
89.63
2.22
77.60
0.47
83.17
0.68
Table 7: Class-level performance metrics of the PV-BERT-large model on the Code-level classification task. Each reported result is averaged over five independent runs. Mean and standard deviation are provided to indicate the variability across runs.
Class Label
Precision mean
Precision std
Recall mean
Recall std
F1 mean
F1 std
All
82.58
0.65
77.28
0.46
79.84
0.23
Appreciation/Gratitude
95.51
0.30
68.82
4.93
79.93
3.41
Approval/Reinforcement
0.00
0.00
0.00
0.00
0.00
0.00
ApprovalofDecision/Reinforcement
0.00
0.00
0.00
0.00
0.00
0.00
Clinical Care
82.91
1.59
91.61
0.65
87.03
0.59
EconomicStability
91.88
1.94
77.78
3.92
84.23
3.09
Table 8: Detailed class-level performance of the PV-BERT-large model on Subcode categories. Each reported result is averaged over five independent runs. Mean and standard deviation are provided to indicate the variability across runs.
Table 9: Detailed class-level performance of the PV-BERT-large model on Combo categories. Each reported result is averaged over five independent runs. Mean and standard deviation are provided to indicate the variability across runs.
Method
Code F1 Mean
Std
Subcode F1 Mean
Std
Combo F1 Mean
Std
PV-BERT-large
78.62
1.30
76.21
1.75
73.94
1.75
PV-BERT-large + author
79.21
0.66
77.01
0.59
75.33
0.86
PV-BERT-large + author + topic
82.25
0.33
79.84
0.19
77.87
0.17
Table 10: Ablation study of PVminer using the PV-BERT-large encoder. Results show mean and standard deviation across five runs with different random seeds.
Department of Emergency Medicine, Yale School of Medicine, 464 Congress Ave, 06519, CT, USA · Department of Social and Behavioral Sciences, Yale School of Public Health, 60 College Street, 06520, CT, USA · Division of Research, Texas State University, 601 University Dr., 78666, TX, USA
Yale University, New Haven, CT, USA · Cleveland Clinic Lerner College of Medicine of Case Western Reserve University, Cleveland Clinic, Cleveland, OH, USA · Medical Oncology, Yale School of Medicine, New Haven, CT, USA