cs.CLApr 12, 2026

Specialty-Specific Medical Language Model for Immune-Mediated Diseases

Authors: Veysel Kocaman, Gursev Pirge, Yigit Gul, Ace Vo, Zhenya Nargizyan, David Talby

Organizations: John Snow Labs Inc., Delaware, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Extracting detailed clinical information from free-text medical narratives remains a practical challenge for researchers and healthcare systems. Terminology for immune-mediated and infectious diseases is especially inconsistent across sources, which often limits the ability of general-purpose Natural Language Processing (NLP) systems to capture the relevant biomedical concepts with sufficient granularity. We developed a domain-specific Named Entity Recognition (NER) model tailored to identify disease-related entities occurring in immunology and infectious disease contexts. We assembled and manually annotated a dataset of 371 case reports in collaboration with two clinical specialists, defining twelve entity classes covering immune-mediated and infectious conditions as well as related symptoms and clinical descriptors. We evaluated several modeling strategies, including the MedicalNER architecture with multiple healthcare-specific embeddings, a BERT-based token classification model, and zero-shot NER systems. The strongest performance was obtained with a transformer-based model trained on clinical-domain embeddings, which reached an F1 score of 0.89, consistently outperforming baseline and zero-shot approaches. The combination of specialized embeddings and expert annotation proved particularly valuable for capturing nuanced disease terminology and improving generalization across heterogeneous biomedical text. The prompted LLM baseline achieved substantially lower performance under the same evaluation protocol, reflecting difficulties in producing span-consistent outputs for fine-grained entity boundaries despite detailed prompting. The resulting model provides a structured way to analyze case reports and can support downstream tasks such as cohort identification, disease monitoring, and clinical decision support.

Figures & tables

CategoryPrecisionRecallF1-ScoreSupport
Bacterial_Infection0.930.950.94439
Biomarker0.800.910.85183
Fungal_Infection0.960.980.97109
Geographical_Location0.960.970.96377
Immune_Mediated_Disease0.920.870.89330
Other_Disease_Disorder0.870.740.80627
Table 1: IMD NER model accuracy results.
Figure 1: Progress of model performance across annotation batches.
Embeddings / ModelPrecisionRecallF1-score
BERT For Token Classifier0.790.830.81
embeddings_clinical0.880.900.89
bert_l2_h256_uncased0.820.820.82
bert_l2_h512_uncased0.820.840.83
bert_l6_h256_uncased0.810.840.82
bert_l10_h768_uncased0.800.870.83
Table 2: Performance metrics.
CategoryPrecisionRecallF1-ScoreSupport
Bacterial_Infection0.580.620.60439
Biomarker0.380.290.33183
Fungal_Infection0.630.760.69109
Geographical_Location0.890.830.86377
Immune_Mediated_Disease0.650.720.68330
Other_Disease_Disorder0.550.590.57627
Table 3: ChatGPT-5.1 performance metrics.
Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Example of entity recognition performed by the IMD NER model on a clinical note describing Systemic Lupus Erythematosus (SLE).
Figure 3: Automatically generated SLE knowledge graph showing relationships between Immune_Mediated_Disease and associated entities.
Figure 4: Clinical NER training workflow (full Iteration Loop)
Figure 5: Example annotated case report in Generative AI Lab showing multiple entity types.
BatchSource(s)CountFocus / Notes
1PubMed52Initial corpus of IMD-related cases.
2Synthetic (perplexity.ai, DeepSeek)20Generated to augment entity diversity.
3PubMed22Supplementary disease-specific texts.
4PubMed, ScienceDirect, MedRxiv32Mostly PubMed; added variation in style.
5PubMed, ScienceDirect, CDC.org30Broader coverage of infectious comorbidities.
6Synthetic (claude.ai, perplexity.ai)58Short clinical narratives: IMD entity focus.
Appendix
Table 4: Overview of data sources and batches for the IMD dataset.
EntityDefinition
Bacterial_InfectionAn infection caused by pathogenic bacteria, identified by the bacterial organism or the disease it produces.
BiomarkerA measurable biological molecule indicating a normal or abnormal process, condition, or disease.
Fungal_InfectionAn infection caused by pathogenic fungi, identified by the fungal organism or the disease resulting from fungal infection.
Geographical_LocationA specific place or area mentioned in the text, such as a country, city, or region.
Immune_Mediated_DiseaseA condition in which the immune system abnormally targets the body’s own tissues, leading to inflammation and tissue damage.
Other_Disease_DisorderA disease or disorder not caused by infectious agents or immune system dysfunction.
Appendix
Table 5: Entity Definitions of the Model.
HyperparameterOptimal Value (Values Tested)
Batch Size8 (4, 16, 32, 64)
Epoch16 (16 - 60)
Learning Rate0.001 (0.01, 0.0001)
Dropout Rate0.5 (0.3 – 0.7)
OptimizerAdam
Embedding Size200
Appendix
Table 6: Hyperparameters for training – optimum parameter (range of values).
EntityCount
Symptom4,734
Other_Test4,098
Test_Result3,548
Treatment3,337
Other_Disease_Disorder3,058
Bacterial_Infection1,433
Appendix
Table 7: Distribution of entities in the dataset.
Annotation MetricValue
Average Task Length (characters)2,367
Average Time Annotators Spent on One Task (seconds)1,353
Average Edit Time Annotators Spent on One Task (seconds)225
Average Number of Edits per Task2
Average Number of Tokens for Submitted/Reviewed Tasks86.94
Total Number of Tokens Across Submitted/Reviewed Tasks31,298
Appendix
Table 8: Annotation process statistics.
AnnotatorArchitectureEmbeddings / Backbone / Model
MedicalNERBiLSTM-CNN-Charembeddings_clinical John Snow Labs (2020c)
  biobert_pubmed_base_cased John Snow Labs (2020b)
  biobert_v1.1_pubmed John Snow Labs (2023)
  biobert_pubmed_base_cased_v1.2 John Snow Labs (2022)
  biobert_clinical_base_cased John Snow Labs (2020a)
  bert_l2_h256_uncased John Snow Labs (2025b)
Appendix
Table 9: Architectures and embeddings.
More in this topic

Clinical Information Extraction

Clinical Information Extraction turns clinical notes, reports, and patient narratives into structured, searchable facts and relations.

See all 106 papers

Explore similar work

CardsList
Apr 19, 2026

Beyond the Basics: Leveraging Large Language Model for Fine-Grained Medical Entity Recognition

Extracting clinically relevant information from unstructured medical narratives such as admission notes, discharge summaries, and emergency case histories remains a challenge in clinical natural language processing (NLP). Medical Entity Recognition (MER) identifies meaningful concepts embedded in these records. Recent advancements in large language models (LLMs) have shown competitive MER performance; however, evaluations often focus on general entity types, offering limited utility for real-world clinical needs requiring finer-grained extraction. To address this gap, we rigorously evaluated the open-source LLaMA3 model for fine-grained medical entity recognition across 18 clinically detailed categories. To optimize performance, we employed three learning paradigms: zero-shot, few-shot, and fine-tuning with Low-Rank Adaptation (LoRA). To further enhance few-shot learning, we introduced two example selection methods based on token- and sentence-level embedding similarity, utilizing a pre-trained BioBERT model. Unlike prior work assessing zero-shot and few-shot performance on proprietary models (e.g., GPT-4) or fine-tuning different architectures, we ensured methodological consistency by applying all strategies to a unified LLaMA3 backbone, enabling fair comparison across learning settings. Our results showed that fine-tuned LLaMA3 surpasses zero-shot and few-shot approaches by 63.11% and 35.63%, respectivel respectively, achieving an F1 score of 81.24% in granular medical entity extraction.

Nwe Ni Win, Jim Basilakis, Steven Thomas +6

Western Sydney University, Sydney, Australia · Western Sydney University, Australia · South Western Emergency Research Institute, Sydney, Australia +7

Named Entity RecognitionHealthcareLLM Fine-TuningFew-Shot LearningClinical Information ExtractionLLM Information Extraction
AskPDF
Jun 11, 2026

AAbAAC: An Annotated Corpus for Autoimmunity Information Extraction

Despite advances in information extraction driven by deep learning and large language models, performance gaps remain in highly specialized biomedical fields, where domainspecific complexity poses challenges for generalist models. In this work, we focus on the domain of autoimmunity, where the main entities of interest are autoimmune diseases, autoantibodies (i.e., molecules that may mark or cause these diseases), their molecular targets, their location in the body, and their associated clinical signs. Herein, we present AAbAAC (AutoAntibodies and Autoimmunity Annotated Corpus), a corpus of 115 abstracts selected from PubMed, where we manually annotated entities and their relationships. First, AAbAAC was used to evaluate several methods on the task of named entity recognition (NER), and secondly, to fine-tune NER models. Our study demonstrates the utility of AAbAAC for information extraction in the domain of autoimmunity, showing expected improvement in NER performance after finetuning. This illustrates the value of small-scale annotation efforts for specialized domains and contributes to the computational study of autoimmunity. The AAbAAC corpus is available at https://github.com/f-maury/AAbAAC.

Fabien Maury, Solène Grosdidier, Maud de Dieuleveult +1

Imagine - U1163, HeKA | U1346 · Inserm, Université Paris Cité, U1163 Institut Imagine, Paris, France · Imagine - U1163 +3

Named Entity RecognitionDocument Information ExtractionBiomedical IR
AskPDF
Sep 24, 2026

Language Specificity vs. Domain Diversity: Benchmarking Transformers for Bangla Medical NER

Medical Named Entity Recognition (NER) for low-resource languages remains a challenging task due to high linguistic variability and a scarcity of domain-specific annotated corpora. This work presents a comprehensive empirical benchmark evaluating three fine-tuned transformer encoders-BanglaBERT, multilingual BERT (mBERT), and XLM-RoBERTa-against GPT-4o mini under zero-shot and few-shot prompting configurations for Bangla medical NER. In contrast to prior studies that evaluated large language models on limited subsets of only 50 samples, we conduct a large-scale evaluation across the full test set of 3,179 samples, providing statistically robust and reproducible baselines. Our fine-tuned XLM-RoBERTa model achieves an F1- score of 0.5959, establishing a new state-of-the-art and surpassing the previously reported best result of 0.5848. Crucially, we demonstrate that the language-specific BanglaBERT model consistently underperforms its multilingual counterparts with an F1-score of 0.4937, indicating that pretraining domain diversity can outweigh language specificity in highly specialized clinical settings. Furthermore, we present a detailed per-entity-type analysis for this task, revealing that Medicine and Specialist categories are recognized with high reliability, achieving F1- scores above 0.83, while the Symptom category remains the most challenging with an F1-score of 0.4367 despite being the most frequent training class. Finally, fine-tuned transformer models outperform the optimal prompting configuration by a factor of 3.76, confirming that prompt-only pipelines remain inadequate for structured clinical entity extraction in low-resource language environments.

Rakib Abdullah, Md. Maruful Islam Maruf

Department of Artificial Intelligence and Data Science Green University of Bangladesh Dhaka, Bangladesh

Named Entity RecognitionMultilingual Language Model EvaluationFew-Shot PromptingLow-Resource Language ProcessingClinical Language Model Evaluation
AskPDF