cs.CLMar 15, 2026

Selective Fine-Tuning of GPT Architectures for Parameter-Efficient Clinical Text Classification

Authors: Fariba Afrin Irany, Sampson Akwafuo

Organizations: Clarksville, TN, USA · California State University Fullerton, California, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The rapid expansion of electronic health record (EHR) systems has generated large volumes of unstructured clinical narratives that contain valuable information for disease identification, patient cohort discovery, and clinical decision support. Extracting structured knowledge from these free-text documents remains challenging because clinical language is highly specialized, labeled datasets are limited, and full fine-tuning of large pretrained language models can require substantial computational resources. Efficient adaptation strategies are therefore essential for practical clinical natural language processing applications. This study proposes a parameter-efficient selective fine-tuning framework for adapting GPT-2 to clinical text classification tasks. Instead of updating the entire pretrained model, the majority of network parameters are frozen, and only the final Transformer block, the final layer normalization module, and a lightweight classification head are updated during training. This design substantially reduces the number of trainable parameters while preserving the contextual representation capabilities learned during pretraining. The proposed approach is evaluated using radiology reports from the MIMIC-IV-Note dataset with automatically derived CheXpert-style labels. Experiments on 50,000 radiology reports demonstrate that selective fine-tuning achieves approximately 91% classification accuracy while updating fewer than 6% of the model parameters. Comparative experiments with head-only training and full-model fine-tuning show that the proposed method provides a favorable balance between predictive performance and computational efficiency. These results indicate that selective fine-tuning offers an efficient and scalable framework for clinical text classification.

Figures & tables

Fig. 1: Selective fine-tuning workflow for a GPT-based classification model. Pretrained parameters are loaded and all layers are frozen except the final Transformer block, final LayerNorm, and the task-specific classification head. Fine-tuning proceeds over mini-batches using cross-entropy loss and AdamW updates applied only to trainable parameters.
Fig. 2: Total training time for different fine-tuning strategies.
Fig. 3: Classification accuracy on training, validation, and test sets across fine-tuning strategies.
Fig. 4: Test-set performance across fine-tuning strategies. The figure compares F1 score and AUROC for linear head–only training, selective fine-tuning of final transformer block, and full model fine-tuning, illustrating the trade-off between predictive performance and the number of trainable parameters.
Label CategoryNumber of ReportsPercentage
Positive or Uncertain (1)16,50133.0%
Negative (0)33,49967.0%
Total50,000100%
TABLE I: Label Distribution for Binary Classification Task
Fine-Tuning StrategyTrainable ParametersTime / Epoch (min)Val. Acc.Test Acc.F1 ScoreAUROC
Linear head only1.5k12.7968.4267.530.610.45
Selective fine-tuning (proposed)7.08M14.9691.4091.000.850.96
Full fine-tuning124M35.2195.9496.230.990.94
TABLE II: Comparison of model performance and training efficiency across fine-tuning strategies. Selective fine-tuning achieves strong generalization performance while substantially reducing the number of trainable parameters and training time per epoch compared to full fine-tuning.
More in this topic

Selective Fine-Tuning

Selective fine-tuning adapts chosen model parameters, layers, neurons, tokens, or training examples instead of updating everything.

See all 43 papers

Explore similar work

CardsList
Mar 23, 2026

Parameter-Efficient Fine-Tuning for Medical Text Summarization: A Comparative Study of Lora, Prompt Tuning, and Full Fine-Tuning

Fine-tuning large language models for domain-specific tasks such as medical text summarization demands substantial computational resources. Parameter-efficient fine-tuning (PEFT) methods offer promising alternatives by updating only a small fraction of parameters. This paper compares three adaptation approaches-Low-Rank Adaptation (LoRA), Prompt Tuning, and Full Fine-Tuning-across the Flan-T5 model family on the PubMed medical summarization dataset. Through experiments with multiple random seeds, we demonstrate that LoRA consistently outperforms full fine-tuning, achieving 43.52 +/- 0.18 ROUGE-1 on Flan-T5-Large with only 0.6% trainable parameters compared to 40.67 +/- 0.21 for full fine-tuning. Sensitivity analyses examine the impact of LoRA rank and prompt token count. Our findings suggest the low-rank constraint provides beneficial regularization, challenging assumptions about the necessity of full parameter updates. Code is available at https://github.com/eracoding/llm-medical-summarization

Ulugbek Shernazarov, Rostislav Svitsov, Bin Shi

Telecom SudParis, Institut Polytechnique de Paris, France

Parameter-Efficient Fine-TuningText SummarizationLow-Rank AdaptationPrompt Tuning
AskPDF
Sep 30, 2026

Comparison of techniques for fine-tuning open-weight models for entity extraction from radiology reports

Converting free-text radiology reports into structured labels supports cohort building, quality assurance, and monitoring of clinical imaging models, but the strongest label extractors are hosted proprietary models whose use raises privacy, cost, and reproducibility concerns. We asked whether a fine-tuned open-weight model (Gemma-3-12B) can match GPT-4o at multi-label intracranial hemorrhage (ICH) acuity extraction from non-contrast head-CT reports, and which ingredients matter. Using a 2x2 design, we crossed two adaptation strategies (a discriminative classification head, CH; generative instruction fine-tuning, IFT) with two training-data sources (distillation of real GPT-4o-labeled reports; synthetic reports generated by GPT-4o from real exemplars), across five training sizes, benchmarked on 100 expert-adjudicated reports against GPT-4o and the un-tuned open-weight base. The distilled instruction-tuned model (DIFT) matched GPT-4o (macro-F1 0.845 vs 0.850; p = 1.000) and exceeded the base model by 0.178. The decisive factor was the training-data source, not the fine-tuning method: both synthetic-data models failed to exceed the un-tuned open-weight base at any training size and underperformed the distilled models across all acuity classes. Fine-tuning and inference fit within the memory envelope of a single 24 GB consumer GPU. For narrow, high-value clinical label-extraction tasks, distilling real reports, rather than generating synthetic ones, is what closes the gap to a hosted model, enabling a private, low-cost, version-stable on-premises alternative.

Aawez Mansuri, Kush Mehta, Mohammadreza Chavoshi +10

Department of Radiology and Imaging Sciences, Emory University School of Medicine, Atlanta, GA, USA · Department of Computer Science, Emory University, Atlanta, GA, USA

Supervised Fine-TuningSynthetic Data GenerationClinical Information ExtractionLLM Information ExtractionKnowledge DistillationMedical LLMs
AskPDF
Apr 8, 2026

A Parameter-Efficient Transfer Learning Approach through Multitask Prompt Distillation and Decomposition for Clinical NLP

Existing prompt-based fine-tuning methods typically learn task-specific prompts independently, imposing significant computing and storage overhead at scale when deploying multiple clinical natural language processing (NLP) systems. We present a multitask prompt distillation and decomposition framework that learns a single shared metaprompt from 21 diverse clinical source tasks and adapts it to unseen target tasks with fewer than 0.05% trainable parameters. Evaluated across five clinical NLP task types (named entity recognition, relation extraction, question answering, natural language inference, and summarization) on 10 held-out target datasets using three backbone models (LLaMA 3.1 8B, Meditron3 8B, gpt-oss 20B), our framework consistently outperforms LoRA by 1.51.7% despite using orders of magnitude fewer parameters, and exceeds single-task prompt tuning by 6.16.6%. The gpt-oss 20B model achieves the highest overall performance, particularly on clinical reasoning tasks. The strong zero- and few-shot performance demonstrates better transferability of the shared prompt representation.

Cheng Peng, Mengxian Lyu, Ziyi Chen +1

Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA · Cancer Informatics Shared Resource, University of Florida Health Cancer Center, Gainesville, Florida, USA

Parameter-Efficient Fine-TuningTransfer LearningClinical NLPMulti-Task LearningKnowledge Distillation
AskPDF