Exploring Recurrent-Depth Language Models for Multitask Clinical Text Processing

Xi Yang; Guodong Liu; Cheng Peng; Kaleb Smith; Yingying Zhang; Xing He; Yi Guo; Jiang Bian

cs.CLOct 20, 2026

Exploring Recurrent-Depth Language Models for Multitask Clinical Text Processing

Authors: Xi Yang, Guodong Liu, Cheng Peng, Kaleb Smith, Yingying Zhang, Xing He, Yi Guo, Jiang Bian,

Organizations: Advanced Intelligence and Research Eli Lilly and Company Indianapolis, IN, USA · Dept. of Health Outcomes and Biomedical Informatics University of Florida Gainesville, FL, USA · Higher Education and Research NVIDIA Santa Clara, CA, USA · Dept. of Computer Science University of Texas Rio Grande Valley Edinburg, TX, USA ·

Abstract

Clinical natural language processing includes diverse tasks such as information extraction, classification, and question answering. Developing one model for all tasks could simplify deployment and improve knowledge transfer. We evaluated whether small recurrent-depth language models can meet this need through joint fine-tuning and compared their performance with standard transformers. We fine-tuned Ouro1.4B-Thinking, Ouro-2.6B-Thinking, HRM-Text-1B, Qwen3-1.7B, and Qwen3-8B on 9 datasets covering information extraction, classification, inference, and question answering, and evaluated them on four additional datasets excluded from training. Across these 13 datasets, Ouro-1.4B-Thinking achieved higher performances than Qwen3-1.7B on 11 datasets; Ouro-2.6BThinking was competitive with Qwen3-8B on most tasks. Ouro models showed the largest gains on inference and question answering, whereas HRM-Text-1B showed mixed results. These findings support further study of recurrent-depth models for clinical tasks through controlled experiments to isolate architectural effects and investigate how recurrent computation influences reasoning, knowledge acquisition, and generalization.

Recurrent Language Models

See all 31 papers

A Parameter-Efficient Transfer Learning Approach through Multitask Prompt Distillation and Decomposition for Clinical NLP
Apr 8, 2026Cheng Peng, Mengxian Lyu, Ziyi Chen +1Parameter-Efficient Fine-Tuning Transfer Learning
Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA · Cancer Informatics Shared Resource, University of Florida Health Cancer Center, Gainesville, Florida, USA
Retrofitting Recurrent Depth into a Pretrained Language Model: Installation, Extrapolation, Transfer, and Retention at Two Parameter Budgets
Jul 31, 2026Mark ShapiroLLM Fine-Tuning Recurrent Transformers
Closing the Loop: Practical Training Recipes for Looped Language Models
Sep 30, 2026Andrei Marchenko, Viacheslav Bezrukov, Oleg Kashurin +5Efficient Language Model Training LLM Training
RND NLP, DAIMLD, Russian Federation

Apr 8, 2026J/K move · Enter open · S save

A Parameter-Efficient Transfer Learning Approach through Multitask Prompt Distillation and Decomposition for Clinical NLP

Cheng Peng, Mengxian Lyu, Ziyi Chen, Yonghui Wu

Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA · Cancer Informatics Shared Resource, University of Florida Health Cancer Center, Gainesville, Florida, USA

Parameter-Efficient Fine-Tuning Transfer Learning Clinical NLP Multi-Task Learning Knowledge Distillation

Open paper AskPDF arXiv

Existing prompt-based fine-tuning methods typically learn task-specific prompts independently, imposing significant computing and storage overhead at scale when deploying multiple clinical natural language processing (NLP) systems. We present a multitask prompt distillation and decomposition framework that learns a single shared metaprompt from 21 diverse clinical source tasks and adapts it to unseen target tasks with fewer than 0.05% trainable parameters. Evaluated across five clinical NLP task types (named entity recognition, relation extraction, question answering, natural language inference, and summarization) on 10 held-out target datasets using three backbone models (LLaMA 3.1 8B, Meditron3 8B, gpt-oss 20B), our framework consistently outperforms LoRA by 1.5~~1.7% despite using orders of magnitude fewer parameters, and exceeds single-task prompt tuning by 6.1~~6.6%. The gpt-oss 20B model achieves the highest overall performance, particularly on clinical reasoning tasks. The strong zero- and few-shot performance demonstrates better transferability of the shared prompt representation.

Abstract

Recurrent Language Models

Explore similar work

A Parameter-Efficient Transfer Learning Approach through Multitask Prompt Distillation and Decomposition for Clinical NLP

Retrofitting Recurrent Depth into a Pretrained Language Model: Installation, Extrapolation, Transfer, and Retention at Two Parameter Budgets

Closing the Loop: Practical Training Recipes for Looped Language Models