Small LLMs for Medical NLP: a Systematic Analysis of Few-Shot, Constraint Decoding, Fine-Tuning and Continual Pre-Training in Italian

Pietro Ferrazzi; Mattia Franzin; Alberto Lavelli; Bernardo Magnini

cs.CLFeb 19, 2026

Small LLMs for Medical NLP: a Systematic Analysis of Few-Shot, Constraint Decoding, Fine-Tuning and Continual Pre-Training in Italian

Authors: Pietro Ferrazzi, Mattia Franzin, Alberto Lavelli, Bernardo Magnini

Organizations: Fondazione Bruno Kessler, Povo, Trento, Italy · University of Padova, Padova, Italy

Abstract

Large Language Models (LLMs) consistently excel in diverse medical Natural Language Processing (NLP) tasks, yet their substantial computational requirements often limit deployment in real-world healthcare settings. In this work, we investigate whether "small" LLMs (around one billion parameters) can effectively perform medical tasks while maintaining competitive accuracy. We evaluate models from three major families-Llama-3, Gemma-3, and Qwen3-across 20 clinical NLP tasks among Named Entity Recognition, Relation Extraction, Case Report Form Filling, Question Answering, and Argument Mining. We systematically compare a range of adaptation strategies, both at inference time (few-shot prompting, constraint decoding) and at training time (supervised fine-tuning, continual pretraining). Fine-tuning emerges as the most effective approach, while the combination of few-shot prompting and constraint decoding offers strong lower-resource alternatives. Our results show that small LLMs can match or even surpass larger baselines, with our best configuration based on Qwen3-1.7B achieving an average score +9.2 points higher than Qwen3-32B. We release a comprehensive collection of all the publicly available Italian medical datasets for NLP tasks, together with our top-performing models. Furthermore, we release an Italian dataset of 126M words from the Emergency Department of an Italian Hospital, and 175M words from various sources that we used for continual pre-training.

Figures & tables

Panel of Figure 1 — Figure 1: Average performances of 1B LLMs on the 14 medical sub-tasks when different methods are applied at both inference ( left ) and training ( right ) time. Exposing models to Fine-Tuning ( FT ) turns out to be the most effective approach overall, consistently outperforming the baseline (Qwen3-32B with 4-shot). Continual Pre-Training ( CPT ) has a positive impact with respect to simple FT only in one case (gemma-3-1b-it). 4-shot is consistently better than Constraint Decoding ( CD ), and the combination of the two shows to be beneficial.

source	words	source	open
scientific
Med Common Crawl	65.7M	Med-mT5	x
Drug Instructions	36.8M	Med-mT5	x
Medical Wikipedia	12.8M	Med-mT5	x
E3C corpus	11.6M	Med-mT5	x
WebHoseAZ	7.2M	Med-mT5	x

Table 1: Continual Pre-Training data size by source. The scientific dataset is composed by several sources, resulting in 278M words. The clinical dataset is composed by documents coming from the emergency department of an Italian hospital, comprising 126M words. Around half of the scientific data is sourced from Med-mT5 ( García-Ferrero et al., 2024 ) , while the remaining is from our new data sources. All datasets are made open-source ( open ), with the exception of Pensiero Scientifico and Zadig , due to their restrictive licence.

Tasks definition			Examples
task	dataset	n	train	val	test
ner	cardioccc	1	250	100	150
ner	pharmaer	4	1316	404	64
ner	e3c	2	451	67	628
re	e3c	1	451	67	628
arg	casimed	1	434	63	125

Table 2: Datasets selected for Italian medical NLP tasks definition: named entity recognition ( ner ); relation extraction ( re ); argument mining ( arg ); case report forms filling ( crf ); multiple-choice question answering ( qa ). The number of sub-tasks defined per dataset ( n ) is reported. The table reports datasets used at both training and testing time ( in-distribution ), together with the ones utilized solely for testing purposes ( out-of-distribution ) in the last six rows.

Model	Method	NER	CRF	RE	QA	ARG	AVG	$\delta$ baseline	p-val
medgemma-27b	+ 4-shot	52.4	62.2	8.9	83.2	62.6	53.8
Qwen3-32B	+ 4-shot	49.7	63.3	11.5	82.8	66.3	54.7
	0-shot	0.0	0.0	0.0	0.0	0.0	0.0	-54.7
	+ CD	4.8	0.3	0.0	16.8	0.0	4.4	-50.3
	+ 4-shot	3.9	13.2	3.1	23.2	1.9	9.1	-45.6
	+ CD, 4-shot	12.6	13.3	4.1	23.2	12.4	13.1	-41.6

Table 3: Impact of different methods on the five tasks for the three selected model families. For each model family, the best and second-best performances are in bold and underlined respectively. The first two rows report the performances of the baseline models Qwen3-32B and medgemma-27b-text-it. The "

\delta

baseline" column reports the delta from the best performing baseline (Qwen3-32B with 4-shot). The last column ( p-val) reports the significance for the hypothesis of the adaptation method performances being higher than the baseline. One * refers to an observed significance higher than

0.7

, ** higher than

0.8

, and *** higher than

0.95

. The test is performed by pairing results at a sub-task level.

Figure 2: Impact on inference time of using 4-shot and Constraint Decoding ( CD ) settings. While 4-shot significantly increases the time required to run the inference, CD does not. The average is calculated among 5 models and 14 subtasks, using the vLLM and outlines libraries for model serving.

model	method	AVG	delta
Qwen3-32B (baseline)	4-shot	61.7
Llama-3.2-1B-Instruct	4-shot + CD	24.1
Llama-3.2-1B-Instruct	FT	35.9	+11.9
Qwen3-1.7B	4-shot + CD	33.2
Qwen3-1.7B	FT	39.3	+6.1
gemma-3-1b-it	4-shot + CD	27.6

Table 4: Average performances of the adapted models and their counterparts for each of the families on Out-Of-Distribution datasets. The delta column represents the improvement in performance due to FT/CPT.

Figure 3: Visualization of sequence packing and the corresponding 2D attention mask that prevents cross-sample attention.

	4-shot vs 0-shot		CD vs w/o CD		AVG
	w/o CD	CD	0-shot	4-shot	all
gemma-3-1b-pt	9.1	8.7	4.4	4.0	6.6
gemma-3-1b-it	4.1	8.2	1.7	5.8	5.0
Llama-3.2-1B	14.8	12.3	5.4	2.9	8.9
Llama-3.2-1B-Instruct	0.0	-1.9	2.2	0.3	0.2
Qwen3-1.7B-Base	25.0	11.4	14.1	0.5	12.8

Table 5: Analysis of the impact of different methods to enhance performances at inference time. The first two columns ( 4-shot vs 0-shot ) report the increase of performances moving from 0- to 4-shot with and without Constraint Decoding, while the other two ( CD vs w/o CD ) the increase of performances moving from not using Constraint Decoding to using it in both 0- and 4- shot.

Model	0-shot	0-shot + CD	4-shot	4-shot + CD
gemma-3-1b-pt	385	360	488	403
gemma-3-1b-it	194	202	227	234
Llama-3.2-1B	113	131	135	151
Llama-3.2-1B-Instruct	91	103	136	152
Qwen3-1.7B	64	71	173	180

Table 6: Inference time (in seconds) across different configurations for five selected models. Reported time refers to the execution of inference over the 14 sub-tasks on a single H100, serving them through the vLLM library with outlines for constraint decoding. Overall, inference results in around one hour. A notable difference can be observed among models, some being much faster than others. This is primarily due to the different average output length, as well as the differences in structures.

		Named Entity Recognition
Model	Method	cardioccc	e3c		pharmaer
		medication	bodypart	clinical	anat. part	disease	drugs	symptoms
gemma-3-1b	0-shot	0	0	0	0	0	0	0
base	+ CD	12.98	0.95	9.99	0	0	9.92	0
	+ 4-shot	5.78	0	0	0	2.67	15.65	3.51
	+ CD, 4-shot	26.01	0	4.42	0	10.2	32.86	14.93

Table 7: Results (F1 score) per subtask in Named Entity Recognition (NER)

		Question Answering, Relation Extraction, CRF, Argument Mining
Model	Method	QA		RE	CRF			ARG
		basic	expert ctx	pertainsto	diagnosis	history	rml	arglong
gemma-3-1b	0-shot	0	0	0	0	0	0	0
base	+ CD	16	17.6	0	0	0.44	0.56	0
	+ 4-shot	25.6	20.8	3.12	14.09	18.21	7.43	1.91
	+CD, 4-shot	25.6	20.8	4.11	14.17	18.26	7.43	12.41

Table 8: Results (F1 score for RE, CRF, ARG; accuracy for QA) per subtask for Queston Answering (QA), Relation Extraction (RE), Case Report Form filling (CRF) and Argument Mining (ARG).

		Out Of Distribution
Model	Method	QA			NER
		at 2023	medmcqa	medqa	e3c clinical	distemist	psynit
gemma-3-1b	0-shot	28.0	27.5	28.1	25.5	21.5	17.9
instruct	+ CD	26.8	30.2	29.9	22.9	21.1	22.0
	+ 4-shot	27.0	27.2	30.8	27.0	23.1	25.1
	+CD, 4-shot	28.4	28.5	31.5	27.4	25.7	23.8

Table 9: Results (F1 score for NER; accuracy for QA) per subtask for Question Answering (QA) and Named Entity Recognition (NER).

Model	Method	AVG F1 ( $\delta$ baseline)
		sample-weight	per sub-task	per task
Qwen3-32B	+ 4-shot	52.0	56.8	54.7
	0-shot	0.0 -52.0	0.0 -56.8	0.0 -54.7
	+ CD	2.8 -49.2	4.9 -51.9	4.4 -50.4
	+ 4-shot	8.4 -43.5	8.5 -48.3	9.1 -45.7
	+ CD, 4-shot	10.4 -41.6	13.7 -43.2	13.1 -41.6

Table 10: Average performances of models and adaptation techniques by aggregation strategy. The five tasks are composed by

14

subtasks, unevenly distridubets. Each subtask is the combination of a task and a dataset. Each dataset has a different number of testing examples. We report the F1 score averaged by number of examples in each task ( sample-weight ), by number of sub-tasks per task, and by tasks. For each model family, the best and second-best performances are in bold and underlined respectively. It can be noticed that there is not significant difference among aggregation method.

Small Language Models

Small language models are compact neural language systems designed to deliver useful text and reasoning capabilities with limited compute, memory, and energy.

See all 276 papers

MedLLM: An Open Medical Language Model at the Sub-Billion Scale
Jul 29, 2026Maxx Richard Rahman, Asim Ahmed, Mihan Mohagheghzadeh +1LLM Fine-Tuning Small Language Models
German Research Center for Artificial Intelligence (DFKI), Germany · Saarland University, Germany
Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation
Feb 16, 2026Shefayat E Shams Adib, Ahmed Alfey Sani, Ekramul Alam Esham +2Medical QA Medical LLMs
Department of Computer Science and Engineering, Islamic University of Technology, Gazipur, Bangladesh
Can Continual Pre-training Bridge the Performance Gap between General-purpose and Specialized Language Models in the Medical Domain?
Apr 21, 2026Niclas Doll, Jasper Schulze Buschhoff, Shalaka Satheesh +3Domain-Adaptive Pretraining Domain Adaptation for LLMs
Fraunhofer IAIS · Lamarr Institute

Jul 29, 2026J/K move · Enter open · S save

MedLLM: An Open Medical Language Model at the Sub-Billion Scale

Maxx Richard Rahman, Asim Ahmed, Mihan Mohagheghzadeh, Wolfgang Maass

German Research Center for Artificial Intelligence (DFKI), Germany · Saarland University, Germany

LLM Fine-Tuning Small Language Models Medical QA Medical LLMs

Open paper AskPDF arXiv

Open medical language models have converged on a single scale: every widely used system runs at 7B parameters or more, leaving the sub-billion regime uncharacterized. We present MedLLM, an open 0.1B-parameter medical language model trained through a fully open three-phase pipeline: general pretraining with curriculum sequence-length scheduling, domain fine-tuning on MedFineWeb, a reference-guided medical corpus we release that is selected from general web data by embedding similarity to medical question-answering (QA) data, and preference-aligned fine-tuning combining SFT with direct preference optimization (DPO). Across medical benchmarks, MedLLM shows a pattern visible only at sub-billion scale: medical competence does not degrade uniformly under compression but splits by task type. On context-grounded QA it comes within

2.9

pp of a medically adapted 7B model and surpasses the instruction-tuned and general-purpose 7B baselines; on knowledge-recall QA it stays near the task floor on clinical-vignette MedQA yet significantly exceeds every 7B and sub-7B baseline on MedMCQA, indicating that where recall fails the constraint is model capacity rather than adaptation. This dissociation is masked at 7B, where both capabilities are present, and surfaces only when capacity is scarce.

Abstract

Small Language Models

Explore similar work

MedLLM: An Open Medical Language Model at the Sub-Billion Scale

Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation

Can Continual Pre-training Bridge the Performance Gap between General-purpose and Specialized Language Models in the Medical Domain?