A Parameter-Efficient Transfer Learning Approach through Multitask Prompt Distillation and Decomposition for Clinical NLP

Cheng Peng; Mengxian Lyu; Ziyi Chen; Yonghui Wu

cs.CLApr 8, 2026

A Parameter-Efficient Transfer Learning Approach through Multitask Prompt Distillation and Decomposition for Clinical NLP

Authors: Cheng Peng, Mengxian Lyu, Ziyi Chen, Yonghui Wu

Organizations: Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA · Cancer Informatics Shared Resource, University of Florida Health Cancer Center, Gainesville, Florida, USA

Abstract

Existing prompt-based fine-tuning methods typically learn task-specific prompts independently, imposing significant computing and storage overhead at scale when deploying multiple clinical natural language processing (NLP) systems. We present a multitask prompt distillation and decomposition framework that learns a single shared metaprompt from 21 diverse clinical source tasks and adapts it to unseen target tasks with fewer than 0.05% trainable parameters. Evaluated across five clinical NLP task types (named entity recognition, relation extraction, question answering, natural language inference, and summarization) on 10 held-out target datasets using three backbone models (LLaMA 3.1 8B, Meditron3 8B, gpt-oss 20B), our framework consistently outperforms LoRA by 1.5~~1.7% despite using orders of magnitude fewer parameters, and exceeds single-task prompt tuning by 6.1~~6.6%. The gpt-oss 20B model achieves the highest overall performance, particularly on clinical reasoning tasks. The strong zero- and few-shot performance demonstrates better transferability of the shared prompt representation.

Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning adapts pretrained models by updating small parameter subsets or lightweight modules instead of all weights.

See all 605 papers

Exploring Recurrent-Depth Language Models for Multitask Clinical Text Processing
Oct 20, 2026Xi Yang, Guodong Liu, Cheng Peng +7Recurrent Language Models Clinical NLP
Advanced Intelligence and Research Eli Lilly and Company Indianapolis, IN, USA · Dept. of Health Outcomes and Biomedical Informatics University of Florida Gainesville, FL, USA · Higher Education and Research NVIDIA Santa Clara, CA, USA +2
PEML: Parameter-efficient Multi-Task Learning with Optimized Continuous Prompts
May 13, 2026Anjir Ahmed Chowdhury, Syed Zawad, Xiaolong Ma +2LLM Fine-Tuning Multi-Task Learning
Department of Computer Science University of Houston · IBM Research · Argonne National Laboratory
Small LLMs for Medical NLP: a Systematic Analysis of Few-Shot, Constraint Decoding, Fine-Tuning and Continual Pre-Training in Italian
Feb 19, 2026Pietro Ferrazzi, Mattia Franzin, Alberto Lavelli +1Small Language Models Clinical NLP
Fondazione Bruno Kessler, Povo, Trento, Italy · University of Padova, Padova, Italy

Oct 20, 2026J/K move · Enter open · S save

Exploring Recurrent-Depth Language Models for Multitask Clinical Text Processing

Xi Yang, Guodong Liu, Cheng Peng, Kaleb Smith+6

Advanced Intelligence and Research Eli Lilly and Company Indianapolis, IN, USA · Dept. of Health Outcomes and Biomedical Informatics University of Florida Gainesville, FL, USA · Higher Education and Research NVIDIA Santa Clara, CA, USA+2

Recurrent Language Models Clinical NLP Clinical Language Model Evaluation LLM Fine-Tuning Multi-Task Learning

Open paper AskPDF arXiv

Clinical natural language processing includes diverse tasks such as information extraction, classification, and question answering. Developing one model for all tasks could simplify deployment and improve knowledge transfer. We evaluated whether small recurrent-depth language models can meet this need through joint fine-tuning and compared their performance with standard transformers. We fine-tuned Ouro1.4B-Thinking, Ouro-2.6B-Thinking, HRM-Text-1B, Qwen3-1.7B, and Qwen3-8B on 9 datasets covering information extraction, classification, inference, and question answering, and evaluated them on four additional datasets excluded from training. Across these 13 datasets, Ouro-1.4B-Thinking achieved higher performances than Qwen3-1.7B on 11 datasets; Ouro-2.6BThinking was competitive with Qwen3-8B on most tasks. Ouro models showed the largest gains on inference and question answering, whereas HRM-Text-1B showed mixed results. These findings support further study of recurrent-depth models for clinical tasks through controlled experiments to isolate architectural effects and investigate how recurrent computation influences reasoning, knowledge acquisition, and generalization.

Abstract

Parameter-Efficient Fine-Tuning

Explore similar work

Exploring Recurrent-Depth Language Models for Multitask Clinical Text Processing

PEML: Parameter-efficient Multi-Task Learning with Optimized Continuous Prompts

Small LLMs for Medical NLP: a Systematic Analysis of Few-Shot, Constraint Decoding, Fine-Tuning and Continual Pre-Training in Italian