CLiGNet: Clinical Label-Interaction Graph Network for Medical Specialty Classification from Clinical Transcriptions
Authors: Pronob Kumar Barman, Pronoy Kumar Barman
Organizations: University of Maryland, Baltimore County, Department of Information Systems, Baltimore, MD, USA · Jagannath University, Department of Statistics, Dhaka, Bangladesh
Automated classification of clinical transcriptions into medical specialties is essential for routing, coding, and clinical decision support, yet prior work on the widely used MTSamples benchmark suffers from severe data leakage caused by applying SMOTE oversampling before train test splitting. We first document this methodological flaw and establish a leakage free benchmark across 40 medical specialties (4966 records), revealing that the true task difficulty is substantially higher than previously reported. We then introduce CLiGNet (Clinical Label Interaction Graph Network), a neural architecture that combines a Bio ClinicalBERT text encoder with a two layer Graph Convolutional Network operating on a specialty label graph constructed from semantic similarity and ICD 10 chapter priors. Per label attention gates fuse document and label graph representations, trained with focal binary cross entropy loss to handle extreme class imbalance (181 to 1 ratio). Across seven baselines ranging from TF IDF classifiers to Clinical Longformer, CLiGNet without calibration achieves the highest macro F1 of 0.279, with an ablation study confirming that the GCN label graph provides the single largest component gain (increase of 0.066 macro F1). Adding per label Platt scaling calibration yields an expected calibration error of 0.007, demonstrating a principled trade off between ranking performance and probability reliability. We provide comprehensive failure analysis covering pairwise specialty confusions, rare class behaviour, document length effects, and token level Integrated Gradients attribution, offering actionable insights for clinical NLP system deployment.
Figures & tables
Statistic
Value
Note
Raw records
4,999
After cleaning
4,966
33 empty removed
Medical specialties
40
Single-label
Training set
3,476 (70%)
Stratified
Validation set
745 (15%)
Stratified
Test set
745 (15%)
Stratified
Table 1. MTSamples dataset summary after preprocessing.
Model
Ma-F1 ↑
Mi-F1 ↑
Acc ↑
Ham ↓
B1: TF-IDF + LR
0.173
0.105
0.105
0.045
B2: TF-IDF + SVC
0.099
0.152
0.152
0.042
B3: ClinBERT + OvR
0.211
0.344
0.344
0.033
B4: BioBERT + OvR
0.172
0.318
0.318
0.034
B7: Longformer + BR
0.262
0.336
0.336
0.033
B6: CLiGNet (no cal.)
0.279
0.362
0.362
0.032
Table 2. Test-set performance across all models. Best result per metric is bolded . B6 achieves the highest macro-F1; B8 additionally provides calibrated probabilities (ECE = 0.007).
Baseline
n01
n10
χ2
p
Sig.
B1: TF-IDF + LR
42
216
116.0
< 0.001
B2: TF-IDF + SVC
54
193
77.1
< 0.001
B3: ClinBERT
120
116
0.04
0.845
ns
B4: BioBERT
100
115
0.91
0.340
ns
B6: no calib.
130
112
1.19
0.274
ns
B7: Longformer
123
125
0.00
0.949
ns
Table 3. McNemar’s test results: B8 vs. each baseline. Bonferroni-corrected α=0.0083 . n01 : samples correct only in baseline; n10 : correct only in B8.
ID
Removed Component
Ma-F1
Mi-F1
Δ Ma-F1
B6
(Full, no calibration)
0.279
0.362
—
A1
GCN label graph
0.214
0.336
− 0.066
A2
Focal loss
0.266
0.366
− 0.013
A4
Sliding window
0.271
0.379
− 0.008
A5
Per-label thresholds
0.270
0.338
− 0.009
Table 4. Ablation study: macro-F1 and micro-F1 on the test set. Each variant removes one component from B6 (macro-F1 = 0.279).
True Label
Predicted
Count
% of True
General Medicine
Consult - Hist. & Phy.
26
66.7
Surgery
Gastroenterology
24
14.7
Surgery
Orthopedic
23
14.1
Surgery
Cardiovascular/Pulm.
21
12.9
Urology
Surgery
16
69.6
Table 5. Top-5 confused specialty pairs for B8 on the test set.
Training Samples
Mean F1
Num. Classes
0–20
0.271
16
20–50
0.316
6
50–100
0.447
7
100–500
0.410
8
500+
0.596
1 (Surgery)
Table 6. Mean per-label F1 by training set class size bin.
Clinical domain classification plays an important role in organizing and analyzing large volumes of unstructured medical text. However, medical transcription datasets are often highly imbalanced, which can substantially degrade classification performance, particularly for underrepresented clinical specialties. In this work, we present a comparative study of machine learning and transformer-based approaches for clinical domain classification from medical transcriptions. We evaluate six traditional machine learning classifiers---Naive Bayes, Support Vector Machine (SVM), Decision Tree, Random Forest, K-Nearest Neighbors (KNN), and XGBoost---along with two pretrained transformer models, BERT and XLNet, and a few-shot large language model prompting approach. Experiments are conducted on medical transcription data collected from MTSamples, comprising 5,013 samples across 40 clinical specialties. To address severe class imbalance, we investigate two balancing strategies: text augmentation using NLP-based synonym replacement and Synthetic Minority Over-sampling Technique (SMOTE). Experimental results demonstrate that data balancing substantially improves classification performance across the evaluated models. In particular, BERT achieves the highest F1-score of 0.996 on the SMOTE-balanced dataset while requiring lower training time than XLNet. The results highlight the effectiveness of transformer-based representations combined with appropriate data balancing strategies for clinical domain classification and provide a systematic comparison of classical machine learning, transformer models, and few-shot prompting for medical transcription analysis.
Sravani Pottipati, Lakshmikar R. Polamreddy
Independent Researcher, New York City, USA · Yeshiva University, New York City, USA
Clinical text classification requires choosing between specialized fine-tuned models (BERT variants) and general-purpose large language models (LLMs), yet neither dominates across all instances. We introduce Learning to Defer for clinical text (L2D-Clinical), a framework that learns when a BERT classifier should defer to an LLM based on uncertainty signals and text characteristics. Unlike prior L2D work that defers to human experts assumed universally superior, our approach enables adaptive deferral-improving accuracy when the LLM complements BERT. We evaluate on two English clinical tasks: (1) ADE detection (ADE Corpus V2), where BioBERT (F1=0.911) outperforms the LLM (F1=0.765), and (2) treatment outcome classification (MIMIC-IV with multi-LLM consensus ground truth), where GPT-5-nano (F1=0.967) outperforms ClinicalBERT (F1=0.887). On ADE, L2D-Clinical achieves F1=0.928 (+1.7 points over BERT) by selectively deferring 7% of instances where the LLM's high recall compensates for BERT's misses. On MIMIC, L2D-Clinical achieves F1=0.980 (+9.3 points over BERT) by deferring only 16.8% of cases to the LLM. The key insight is that L2D-Clinical learns to selectively leverage LLM strengths while minimizing API costs.
Rishik Kondadadi, John E. Ortega
University of Minnesota, Northeastern University, Minneapolis, Boston, USA
Clinical notes contain valuable unstructured information. Named entity recognition (NER) enables the automatic extraction of medical concepts; however, benchmarks for Portuguese remain scarce. In this study, we aimed to evaluate BERT-based models and large language models (LLMs) for clinical NER in Portuguese and to test strategies for addressing multilabel imbalance. We compared BioBERTpt, BERTimbau, ModernBERT, and mmBERT with LLMs such as GPT-5 and Gemini-2.5, using the public SemClinBr corpus and a private breast cancer dataset. Models were trained under identical conditions and evaluated using precision, recall, and F1-score. Iterative stratification, weighted loss, and oversampling were explored to mitigate class imbalance. The mmBERT-base model achieved the best performance (micro F1 = 0.76), outperforming all other models. Iterative stratification improved class balance and overall performance. Multilingual BERT models, particularly mmBERT, perform strongly for Portuguese clinical NER and can run locally with limited computational resources. Balanced data-splitting strategies further enhance performance.
Vinicius Anjos de Almeida, Sandro Saorin da Silva, Josimar Chire +8
Spesia, Curitiba - PR, Brazil · Faculdade de Medicina, Universidade de São Paulo, São Paulo - SP, Brazil · Pontifícia Universidade Católica do Paraná (PUCPR), Curitiba - PR, Brazil +2