Large Language Models (LLMs) consistently excel in diverse medical Natural Language Processing (NLP) tasks, yet their substantial computational requirements often limit deployment in real-world healthcare settings. In this work, we investigate whether "small" LLMs (around one billion parameters) can effectively perform medical tasks while maintaining competitive accuracy. We evaluate models from three major families-Llama-3, Gemma-3, and Qwen3-across 20 clinical NLP tasks among Named Entity Recognition, Relation Extraction, Case Report Form Filling, Question Answering, and Argument Mining. We systematically compare a range of adaptation strategies, both at inference time (few-shot prompting, constraint decoding) and at training time (supervised fine-tuning, continual pretraining). Fine-tuning emerges as the most effective approach, while the combination of few-shot prompting and constraint decoding offers strong lower-resource alternatives. Our results show that small LLMs can match or even surpass larger baselines, with our best configuration based on Qwen3-1.7B achieving an average score +9.2 points higher than Qwen3-32B. We release a comprehensive collection of all the publicly available Italian medical datasets for NLP tasks, together with our top-performing models. Furthermore, we release an Italian dataset of 126M words from the Emergency Department of an Italian Hospital, and 175M words from various sources that we used for continual pre-training.
Figures & tables
Figure 1: Average performances of 1B LLMs on the 14 medical sub-tasks when different methods are applied at both inference ( left ) and training ( right ) time. Exposing models to Fine-Tuning ( FT ) turns out to be the most effective approach overall, consistently outperforming the baseline (Qwen3-32B with 4-shot). Continual Pre-Training ( CPT ) has a positive impact with respect to simple FT only in one case (gemma-3-1b-it). 4-shot is consistently better than Constraint Decoding ( CD ), and the combination of the two shows to be beneficial.
source
words
source
open
scientific
Med Common Crawl
65.7M
Med-mT5
x
Drug Instructions
36.8M
Med-mT5
x
Medical Wikipedia
12.8M
Med-mT5
x
E3C corpus
11.6M
Med-mT5
x
WebHoseAZ
7.2M
Med-mT5
x
Table 1: Continual Pre-Training data size by source. The scientific dataset is composed by several sources, resulting in 278M words. The clinical dataset is composed by documents coming from the emergency department of an Italian hospital, comprising 126M words. Around half of the scientific data is sourced from Med-mT5 ( García-Ferrero et al., 2024 ) , while the remaining is from our new data sources. All datasets are made open-source ( open ), with the exception of Pensiero Scientifico and Zadig , due to their restrictive licence.
Tasks definition
Examples
task
dataset
n
train
val
test
ner
cardioccc
1
250
100
150
ner
pharmaer
4
1316
404
64
ner
e3c
2
451
67
628
re
e3c
1
451
67
628
arg
casimed
1
434
63
125
Table 2: Datasets selected for Italian medical NLP tasks definition: named entity recognition ( ner ); relation extraction ( re ); argument mining ( arg ); case report forms filling ( crf ); multiple-choice question answering ( qa ). The number of sub-tasks defined per dataset ( n ) is reported. The table reports datasets used at both training and testing time ( in-distribution ), together with the ones utilized solely for testing purposes ( out-of-distribution ) in the last six rows.
Model
Method
NER
CRF
RE
QA
ARG
AVG
δ baseline
p-val
medgemma-27b
+ 4-shot
52.4
62.2
8.9
83.2
62.6
53.8
Qwen3-32B
+ 4-shot
49.7
63.3
11.5
82.8
66.3
54.7
0-shot
0.0
0.0
0.0
0.0
0.0
0.0
-54.7
+ CD
4.8
0.3
0.0
16.8
0.0
4.4
-50.3
+ 4-shot
3.9
13.2
3.1
23.2
1.9
9.1
-45.6
+ CD, 4-shot
12.6
13.3
4.1
23.2
12.4
13.1
-41.6
Table 3: Impact of different methods on the five tasks for the three selected model families. For each model family, the best and second-best performances are in bold and underlined respectively. The first two rows report the performances of the baseline models Qwen3-32B and medgemma-27b-text-it. The " δ baseline" column reports the delta from the best performing baseline (Qwen3-32B with 4-shot). The last column ( p-val) reports the significance for the hypothesis of the adaptation method performances being higher than the baseline. One * refers to an observed significance higher than 0.7 , ** higher than 0.8 , and *** higher than 0.95 . The test is performed by pairing results at a sub-task level.
Figure 2: Impact on inference time of using 4-shot and Constraint Decoding ( CD ) settings. While 4-shot significantly increases the time required to run the inference, CD does not. The average is calculated among 5 models and 14 subtasks, using the vLLM and outlines libraries for model serving.
model
method
AVG
delta
Qwen3-32B (baseline)
4-shot
61.7
Llama-3.2-1B-Instruct
4-shot + CD
24.1
Llama-3.2-1B-Instruct
FT
35.9
+11.9
Qwen3-1.7B
4-shot + CD
33.2
Qwen3-1.7B
FT
39.3
+6.1
gemma-3-1b-it
4-shot + CD
27.6
Table 4: Average performances of the adapted models and their counterparts for each of the families on Out-Of-Distribution datasets. The delta column represents the improvement in performance due to FT/CPT.
Figure 3: Visualization of sequence packing and the corresponding 2D attention mask that prevents cross-sample attention.
4-shot vs 0-shot
CD vs w/o CD
AVG
w/o CD
CD
0-shot
4-shot
all
gemma-3-1b-pt
9.1
8.7
4.4
4.0
6.6
gemma-3-1b-it
4.1
8.2
1.7
5.8
5.0
Llama-3.2-1B
14.8
12.3
5.4
2.9
8.9
Llama-3.2-1B-Instruct
0.0
-1.9
2.2
0.3
0.2
Qwen3-1.7B-Base
25.0
11.4
14.1
0.5
12.8
Table 5: Analysis of the impact of different methods to enhance performances at inference time. The first two columns ( 4-shot vs 0-shot ) report the increase of performances moving from 0- to 4-shot with and without Constraint Decoding, while the other two ( CD vs w/o CD ) the increase of performances moving from not using Constraint Decoding to using it in both 0- and 4- shot.
Model
0-shot
0-shot + CD
4-shot
4-shot + CD
gemma-3-1b-pt
385
360
488
403
gemma-3-1b-it
194
202
227
234
Llama-3.2-1B
113
131
135
151
Llama-3.2-1B-Instruct
91
103
136
152
Qwen3-1.7B
64
71
173
180
Table 6: Inference time (in seconds) across different configurations for five selected models. Reported time refers to the execution of inference over the 14 sub-tasks on a single H100, serving them through the vLLM library with outlines for constraint decoding. Overall, inference results in around one hour. A notable difference can be observed among models, some being much faster than others. This is primarily due to the different average output length, as well as the differences in structures.
Named Entity Recognition
Model
Method
cardioccc
e3c
pharmaer
medication
bodypart
clinical
anat. part
disease
drugs
symptoms
gemma-3-1b
0-shot
0
0
0
0
0
0
0
base
+ CD
12.98
0.95
9.99
0
0
9.92
0
+ 4-shot
5.78
0
0
0
2.67
15.65
3.51
+ CD, 4-shot
26.01
0
4.42
0
10.2
32.86
14.93
Table 7: Results (F1 score) per subtask in Named Entity Recognition (NER)
Table 8: Results (F1 score for RE, CRF, ARG; accuracy for QA) per subtask for Queston Answering (QA), Relation Extraction (RE), Case Report Form filling (CRF) and Argument Mining (ARG).
Out Of Distribution
Model
Method
QA
NER
at 2023
medmcqa
medqa
e3c clinical
distemist
psynit
gemma-3-1b
0-shot
28.0
27.5
28.1
25.5
21.5
17.9
instruct
+ CD
26.8
30.2
29.9
22.9
21.1
22.0
+ 4-shot
27.0
27.2
30.8
27.0
23.1
25.1
+CD, 4-shot
28.4
28.5
31.5
27.4
25.7
23.8
Table 9: Results (F1 score for NER; accuracy for QA) per subtask for Question Answering (QA) and Named Entity Recognition (NER).
Model
Method
AVG F1 ( δ baseline)
sample-weight
per sub-task
per task
Qwen3-32B
+ 4-shot
52.0
56.8
54.7
0-shot
0.0 -52.0
0.0 -56.8
0.0 -54.7
+ CD
2.8 -49.2
4.9 -51.9
4.4 -50.4
+ 4-shot
8.4 -43.5
8.5 -48.3
9.1 -45.7
+ CD, 4-shot
10.4 -41.6
13.7 -43.2
13.1 -41.6
Table 10: Average performances of models and adaptation techniques by aggregation strategy. The five tasks are composed by 14 subtasks, unevenly distridubets. Each subtask is the combination of a task and a dataset. Each dataset has a different number of testing examples. We report the F1 score averaged by number of examples in each task ( sample-weight ), by number of sub-tasks per task, and by tasks. For each model family, the best and second-best performances are in bold and underlined respectively. It can be noticed that there is not significant difference among aggregation method.
Small language models are compact neural language systems designed to deliver useful text and reasoning capabilities with limited compute, memory, and energy.
Open medical language models have converged on a single scale: every widely used system runs at 7B parameters or more, leaving the sub-billion regime uncharacterized. We present MedLLM, an open 0.1B-parameter medical language model trained through a fully open three-phase pipeline: general pretraining with curriculum sequence-length scheduling, domain fine-tuning on MedFineWeb, a reference-guided medical corpus we release that is selected from general web data by embedding similarity to medical question-answering (QA) data, and preference-aligned fine-tuning combining SFT with direct preference optimization (DPO). Across medical benchmarks, MedLLM shows a pattern visible only at sub-billion scale: medical competence does not degrade uniformly under compression but splits by task type. On context-grounded QA it comes within 2.9pp of a medically adapted 7B model and surpasses the instruction-tuned and general-purpose 7B baselines; on knowledge-recall QA it stays near the task floor on clinical-vignette MedQA yet significantly exceeds every 7B and sub-7B baseline on MedMCQA, indicating that where recall fails the constraint is model capacity rather than adaptation. This dissociation is masked at 7B, where both capabilities are present, and surfaces only when capacity is scarce.
Maxx Richard Rahman, Asim Ahmed, Mihan Mohagheghzadeh +1
German Research Center for Artificial Intelligence (DFKI), Germany · Saarland University, Germany
Recently, Large Language Models (LLMs) have gained significant traction in medical domain, especially in developing a QA systems to Medical QA systems for enhancing access to healthcare in low-resourced settings. This paper compares five LLMs deployed between April 2024 and August 2025 for medical QA, using the iCliniq dataset, containing 38,000 medical questions and answers of diverse specialties. Our models include Llama-3-8B-Instruct, Llama 3.2 3B, Llama 3.3 70B Instruct, Llama-4-Maverick-17B-128E-Instruct, and GPT-5-mini. We are using a zero-shot evaluation methodology and using BLEU and ROUGE metrics to evaluate performance without specialized fine-tuning. Our results show that larger models like Llama 3.3 70B Instruct outperform smaller models, consistent with observed scaling benefits in clinical tasks. It is notable that, Llama-4-Maverick-17B exhibited more competitive results, thus highlighting evasion efficiency trade-offs relevant for practical deployment. These findings align with advancements in LLM capabilities toward professional-level medical reasoning and reflect the increasing feasibility of LLM-supported QA systems in the real clinical environments. This benchmark aims to serve as a standardized setting for future study to minimize model size, computational resources and to maximize clinical utility in medical NLP applications.
Shefayat E Shams Adib, Ahmed Alfey Sani, Ekramul Alam Esham +2
Department of Computer Science and Engineering, Islamic University of Technology, Gazipur, Bangladesh
This paper narrows the performance gap between small, specialized models and significantly larger general-purpose models through domain adaptation via continual pre-training and merging. We address the scarcity of specialized non-English data by constructing a high-quality German medical corpus (FineMed-de) from FineWeb2. This corpus is used to continually pre-train and merge three well-known LLMs (ranging from 7B to 24B parameters), creating the DeFineMed model family. A comprehensive evaluation confirms that specialization dramatically enhances 7B model performance on German medical benchmarks. Furthermore, the pairwise win-rate analysis of the Qwen2.5-based models demonstrates an approximately 3.5-fold increase in the win-rate against the much larger Mistral-Small-24B-Instruct through domain adaptation. This evidence positions specialized 7B models as a competitive, resource-efficient solution for complex medical instruction-following tasks. While model merging successfully restores instruction-following abilities, a subsequent failure mode analysis reveals inherent trade-offs, including the introduction of language mixing and increased verbosity, highlighting the need for more targeted fine-tuning in future work. This research provides a robust, compliant methodology for developing specialized LLMs, serving as the foundation for practical use in German-speaking healthcare contexts.