Small Language Models for Privacy-Preserving Clinical Information Extraction in Low-Resource Languages

Mohammadreza Ghaffarzadeh-Esfahani; Nahid Yousefian; Ebrahim Heidari-Farsani; Ali Akbar Omidvarian; Sepehr Ghahraei; Atena Farangi; AmirBahador Boroumand

cs.CLFeb 24, 2026

Small Language Models for Privacy-Preserving Clinical Information Extraction in Low-Resource Languages

Authors: Mohammadreza Ghaffarzadeh-Esfahani, Nahid Yousefian, Ebrahim Heidari-Farsani, Ali Akbar Omidvarian, Sepehr Ghahraei, Atena Farangi, AmirBahador Boroumand

Organizations: Student Research Committee, Isfahan University of Medical Sciences, Isfahan, Iran · Department of Emergency Medicine, Isfahan University of Medical Sciences, Isfahan, Iran

Abstract

Extracting clinical information from medical transcripts in low-resource languages remains a significant challenge in healthcare natural language processing (NLP). This study evaluates a two-step pipeline combining Aya-expanse-8B as a Persian-to-English translation model with five open-source small language models (SLMs) -- Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct, Qwen2.5-1.5B-Instruct, and Gemma-3-1B-it -- for binary extraction of 13 clinical features from 1,221 anonymized Persian transcripts collected at a cancer palliative care call center. Using a few-shot prompting strategy without fine-tuning, models were assessed on macro-averaged F1-score, Matthews Correlation Coefficient (MCC), sensitivity, and specificity to account for class imbalance. Qwen2.5-7B-Instruct achieved the highest overall performance (median macro-F1: 0.899; MCC: 0.797), while Gemma-3-1B-it showed the weakest results. Larger models (7B--8B parameters) consistently outperformed smaller counterparts in sensitivity and MCC. A bilingual analysis of Aya-expanse-8B revealed that translating Persian transcripts to English improved sensitivity, reduced missing outputs, and boosted metrics robust to class imbalance, though at the cost of slightly lower specificity and precision. Feature-level results showed reliable extraction of physiological symptoms across most models, whereas psychological complaints, administrative requests, and complex somatic features remained challenging. These findings establish a practical, privacy-preserving blueprint for deploying open-source SLMs in multilingual clinical NLP settings with limited infrastructure and annotation resources, and highlight the importance of jointly optimizing model scale and input language strategy for sensitive healthcare applications.

Figures & tables

Figure 1: Schematic overview of the study. The upper panel shows the dataset preprocessing, inference generation, and postprocessing, starting from 1,221 Persian palliative care phone-call transcripts, followed by translation into English, prompt construction with input–output examples, and inference using multiple small language models (SLMs). The models’ structured outputs are then post-processed to extract tabular data. The lower panel illustrates the multi-facet analysis framework, comparing manual extraction of 13 reference features with model-derived features through performance metrics (accuracy, sensitivity, specificity, precision, F1-score), assessment of translation effects, crobustness analysis (Matthews correlation coefficient (MCC), missing values), and sensitivity–specificity trade-offs.

Figure 2: Comparative performances of different models on validation metrics. (A) The median value for 5 metrics of accuracy, sensitivity, specificity, macro-averaged F1 score, and precision among 13 extracted features compared to the manually-extracted ground truth. (B) Matthews Correlation Coefficient (MCC) values for each evaluated model across the 13 extracted clinical features, comparing model-generated outputs with the manually extracted ground-truth annotations. (C) Total number of missing counts for each model among different extracted features.

Feature	Llama-3.1	Qwen2.5	Llama-3.2	Qwen2.5	Gemma-3	Aya-expanse-8B
Feature	-8B-Inst.	-7B-Inst.	-3B-Inst.	-1.5B-Inst.	-1B-it	English	Persian
Doctor’s visit request	0.832	0.678	0.723	0.733	0.633	0.493	0.411
Psychological complaints	0.825	0.832	0.818	0.800	0.782	0.835	0.859
Sleep disorders	0.804	0.838	0.766	0.788	0.740	0.739	0.753
Loss of appetite	0.905	0.915	0.895	0.874	0.732	0.865	0.874
Seizures	0.869	0.908	0.880	0.819	0.790	0.869	0.798

Table 1: Macro-averaged F1 scores for the binary extraction of 13 clinical features from Persian palliative care transcripts across the evaluated small language models. Values are reported for Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, Llama-3.2-3B-Instruct, Qwen2.5-1.5B-Instruct, Gemma-3-1B-it, and Aya-expanse-8B (English-translated and direct Persian variants).

Figure 3: Sensitivity–specificity trade-offs across evaluated small language models. Each panel depicts sensitivity (y-axis) versus specificity (x-axis) for 13 binary clinical features (points labeled 1–13; see legend) for (A) Llama-3.1-8B-Instruct, (B) Qwen2.5-7B-Instruct, (C) Llama-3.2-3B-Instruct, (D) Qwen2.5-1.5B-Instruct, (E) Gemma-3-1B-it, (F) Aya-expanse-8B (English), and (G) Aya-expanse-8B (Persian). The dashed diagonal line denotes the locus where sensitivity equals specificity.

Clinical Information Extraction

Clinical Information Extraction turns clinical notes, reports, and patient narratives into structured, searchable facts and relations.

See all 106 papers

Jun 19, 2026

Clinical Term Extraction using Open-Source Small Language Models

Clinical information for amyotrophic lateral sclerosis (ALS) care documented in unstructured clinical notes limits downstream analysis without extraction into structured formats. Open-source small language models with few-shot prompting for detecting the presence of ALS-relevant clinical terms in patient documentation were evaluated without task-specific training data. The detection task targeted 17 categories spanning functional scores, respiratory measures, medications, and related clinical and non-clinical attributes. Clinical note content was normalized from JSON-encoded discharge summaries and processed with a prompt template having structured JSON outputs. We compared 26 open-source models using aggregate, label-level, and manual-validation multilabel classification metrics. Manual validation showed that a regex rule baseline had higher overall micro-F1 and lower Hamming loss than any single SLM or TF-IDF baseline, while Qwen3-4B-Instruct-2507 was the highest-performing SLM by micro-F1. Model rankings varied by metric and label category, with the TF-IDF baseline showing high recall but low precision, some SLMs showing higher precision but lower recall, and Hammer2.1-7b showing strong performance for ALSFRS-R subscore detection. These findings support targeted hybrid extraction workflows rather than replacement of existing rule-based methods.

Noah Marchal, William E. Janes, Mihail Popescu +1

University of Missouri, Columbia, Missouri, USA · Department of Occupational Therapy; University of Missouri, Columbia, Missouri, USA

Small Language Models Clinical Information Extraction Classification Clinical NLP

Ask PDF

Aug 2, 2026

Gaokerena: A Small Persian Medical Language Model Family

Persian medical NLP is limited by scarce domain-specific resources and insufficient characterization of their coverage, quality, and limitations. We present Gaokerena, a suite of Persian medical resources, and evaluate two Aya-Expanse-8B-derived models as case studies. The suite includes an 85.5-million-word corpus of 101,954 web articles; MF3QA, 24,000 patient-doctor QA pairs selected from 186,029 raw pairs plus 201 translated K-QA items; 1,878 Iranian medical-exam questions (KOPP); 1,089 manually translated medical MMLU questions (FA_MED_MMLU); and approximately 18,000 machine-translated MedMCQA training items filtered by two language-model evaluators. We characterize source composition, orthographic variation, redundancy, topical coverage, QA selection effects, privacy indicators, answer-key balance, extraction artifacts, and translation limitations. On translated medical MMLU, Gaokerena-V achieves 49.31% under direct prompting vs. 46.65% for Aya-Expanse-8B; the unpaired difference is not significant (p=0.21), and a paired exact McNemar test on 1,088 matched questions yields p=0.0502, just above 0.05 and thus inconclusive. Gaokerena-R reaches 52.80% with five-sample self-consistency plus a verifier, but this higher-compute configuration is unmatched to the directly prompted backbone; under matched five-sample chain-of-thought self-consistency without a verifier, it scores 48.58% (529/1,089) vs. 46.92% (511/1,089), with paired exact McNemar p=0.190. English evaluation also reveals a substantial gap relative to Persian. Overall, this work contributes a characterized suite of Persian medical resources; the model results are protocol-specific case studies and do not establish an isolated training advantage or clinical readiness.

Mehrdad Ghassabi, Hamidreza Baradaran Kashani, Pedram Rostami +5

University of Isfahan, Isfahan, Iran · University of Tehran, Tehran, Iran · University of Windsor, Windsor, ON, Canada +3

RL for Language Model Reasoning Small Language Models Medical QA Medical LLMs

Ask PDF

Mar 25, 2026

Evaluating Fine-Tuned LLM Model For Medical Transcription With Small Low-Resource Languages Validated Dataset

Clinical documentation is a critical factor for patient safety, diagnosis, and continuity of care. The administrative burden of EHRs is a significant factor in physician burnout. This is a critical issue for low-resource languages, including Finnish. This study aims to investigate the effectiveness of a domain-aligned natural language processing (NLP); large language model for medical transcription in Finnish by fine-tuning LLaMA 3.1-8B on a small validated corpus of simulated clinical conversations by students at Metropolia University of Applied Sciences. The fine-tuning process for medical transcription used a controlled preprocessing and optimization approach. The fine-tuning effectiveness was evaluated by sevenfold cross-validation. The evaluation metrics for fine-tuned LLaMA 3.1-8B were BLEU = 0.1214, ROUGE-L = 0.4982, and BERTScore F1 = 0.8230. The results showed a low n-gram overlap but a strong semantic similarity with reference transcripts. This study indicate that fine-tuning can be an effective approach for translation of medical discourse in spoken Finnish and support the feasibility of fine-tuning a privacy-oriented domain-specific large language model for clinical documentation in Finnish. Beside that provide directions for future work.

Mohammed Nowshad Ruhani Chowdhury, Mohammed Nowaz Rabbani Chowdhury, Sakari Lukkarinen

School of ICT and Industrial Management, Metropolia University of Applied Sciences, Helsinki, Finland · Department of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute, Troy, NY, USA

LLM Fine-Tuning Medical LLMs Clinical Language Model Evaluation Low-Resource Language Processing

Ask PDF

Abstract

Clinical Information Extraction

Explore similar work

Clinical Term Extraction using Open-Source Small Language Models

Gaokerena: A Small Persian Medical Language Model Family

Evaluating Fine-Tuned LLM Model For Medical Transcription With Small Low-Resource Languages Validated Dataset