Large Language Models for Biomedical Article Classification

Jakub Proboszcz; Paweł Cichosz

cs.CLMar 12, 2026

Large Language Models for Biomedical Article Classification

Authors: Jakub Proboszcz, Paweł Cichosz

Organizations: Institute of Computer Science, Warsaw University of Technology, Nowowiejska 15/19, 00-665 Warsaw, Poland

Abstract

This work presents a systematic and in-depth investigation of the utility of large language models as text classifiers for biomedical article classification. The study uses several small and mid-size open source models, as well as selected closed source ones, and is more comprehensive than most prior work with respect to the scope of evaluated configurations: different types of prompts, output processing methods for generating both class and class probability predictions, as well as few-shot example counts and selection methods. The performance of the most successful configurations is compared to that of conventional classification algorithms. The obtained average PR AUC over 15 challenging datasets above 0.4 for zero-shot prompting and nearly 0.5 for few-shot prompting comes close to that of the naïve Bayes classifier (0.5), the random forest algorithm (0.5 with default settings or 0.55 with hyperparameter tuning) and fine-tuned transformer models (0.5). These results confirm the utility of large language models as text classifiers for non-trivial domains and provide practical recommendations of the most promising setups, including in particular using output token probabilities for class probability prediction.

Figures & tables

Ref.	Data	Models	Prompts	Example Selection	Evaluation
[ 52 ]	sentiment phrases, movie reviews, news articles	GPT-3 DaVinci	zero-shot, few-shot; vanilla, CoT, CARP	random, SimCSE [ 22 ] and fine-tuned SimCSE embedding	accuracy; train-test split
[ 59 ]	COVID-19 tweets, economic and e-commerce texts, SMS spam detection	Llama-2-70B-Instruct, GPT-3.5 Turbo, GPT-4	zero-shot; vanilla	-	accuracy, F1
[ 11 ]	tweets and Facebook comments on US elections	FLAN-T5 XXL, Mistral-7B, Llama-3-8B, Llama-3-70B, GPT-3 Ada, GPT-3 DaVinci, GPT-4o (all except GPT-4o fine-tuned)	zero-shot and one/few-shot (one-shot for threads); varying complexity	random, with class balancing	F1; bootstrapping
[ 58 ]	COVID-19 tweets, economic and e-commerce texts, SMS spam detection	GPT-3.5 Turbo, GPT-4, Gemini-pro, Llama-3-8B Instruct (fine-tuned), Qwen-Chat 7B (fine-tuned), Qwen-Chat 14B, Vicuna v1.5 7B and 13B	zero-shot, few-shot; vanilla	manual	accuracy, F1, uncertainty-error rate (custom metric of LLM’s refusals to answer); train-test split
[ 21 ]	tweets, news articles, movie reviews, Wikipedia comments, medical abstracts, legislation documents, safeguarding reports	Llama-1-7B, Llama-2-7B, Flan-T5 780M, GPT-3.5 Turbo	zero-shot, few-shot with one example per class; varying scope of task/domain description	random, with class balancing	F1; train-test split
[ 26 ]	healthcare-related tweets	GPT-3.5, GPT-4	zero-shot; vanilla	-	precision, recall, F1; k-CV

Table 1: Related work.

Topic	# of samples	Positive class %	Avg. # of words in sample
ACEInhibitors	$2236$	$7.6$	$243.9\pm 84.5$
ADHD	$803$	$10.3$	$222.9\pm 78.6$
Antihistamines	$287$	$31.4$	$224.1\pm 79.3$
AtypicalAntipsychotics	$1031$	$32.3$	$211.5\pm 81.0$
BetaBlockers	$1876$	$14.4$	$240.4\pm 84.7$
CalciumChannelBlockers	$1109$	$23.2$	$244.7\pm 80.6$

Table 2: Statistics of DCR datasets

Prompt	Complexity	System message	User message
A	simple	question about task	only article abstract
B	simple	expected model role	question about task article abstract
C	medium	expected model role information about task	only article abstract
D	medium	expected model role information about task	question about task article abstract
E	complex	expected model role information about task	only article abstract
F	complex	expected model role information about task	question about task article abstract

Table 3: Prompts used in the Prompts experiment

Prompt	A	B	C	D	E	F
ACEInhibitors	$0.080$	$\mathbf{0.106}$	$0.080$	$0.094$	$0.077$	$0.091$
ADHD	$0.402$	$0.441$	$0.436$	$0.476$	$0.490$	$\mathbf{0.523}$
Antihistamines	$0.163$	$0.214$	$0.175$	$0.183$	$0.187$	$\mathbf{0.254}$
AtypicalAntipsychotics	$0.093$	$0.143$	$0.086$	$0.122$	$0.116$	$\mathbf{0.147}$
BetaBlockers	$0.093$	$0.129$	$0.110$	$0.126$	$0.113$	$\mathbf{0.130}$
CalciumChannelBlockers	$0.094$	$0.119$	$0.100$	$0.113$	$0.104$	$\mathbf{0.126}$

Table 4: Macro-averaged MCC scores for the Prompts experiment.

Output	Word	JSON	Score	Token-Word	Token-JSON
ACEInhibitors	$0.088$	$0.088$	$0.107$	$\mathbf{0.173}$	$0.153$
ADHD	$0.352$	$0.371$	$0.386$	$\mathbf{0.579}$	$0.531$
Antihistamines	$0.359$	$0.367$	$0.386$	$\mathbf{0.496}$	$0.477$
AtypicalAntipsychotics	$0.349$	$0.361$	$0.389$	$\mathbf{0.499}$	$0.471$
BetaBlockers	$0.171$	$0.171$	$0.195$	$0.253$	$\mathbf{0.255}$
CalciumChannelBlockers	$0.256$	$0.256$	$0.274$	$0.287$	$\mathbf{0.291}$

Table 5: Macro-averaged AUPRC scores for the Output experiment.

Examples	0	1	2	3	4
ACEInhibitors	$0.172$	$0.210$	$0.241$	$\mathbf{0.264}$	$0.232$
ADHD	$0.551$	$0.585$	$0.580$	$0.594$	$\mathbf{0.601}$
Antihistamines	$0.459$	$0.487$	$\mathbf{0.503}$	$0.490$	$0.463$
AtypicalAntipsychotics	$0.520$	$0.602$	$0.622$	$\mathbf{0.633}$	$0.623$
BetaBlockers	$0.246$	$0.282$	$0.311$	$0.320$	$\mathbf{0.330}$
CalciumChannelBlockers	$0.297$	$0.425$	$0.478$	$0.491$	$\mathbf{0.510}$

Table 6: Macro-averaged AUPRC scores for the Few-Shot-Count experiment.

Method	random	minilm	distroberta	mpnet	mpnet-bal
ACEInhibitors	$0.176$	$0.265$	$\mathbf{0.305}$	$0.267$	$0.155$
ADHD	$0.584$	$0.595$	$0.647$	$\mathbf{0.664}$	$0.536$
Antihistamines	$0.473$	$0.490$	$\mathbf{0.517}$	$0.497$	$0.434$
AtypicalAntipsychotics	$0.500$	$\mathbf{0.633}$	$0.616$	$0.625$	$0.542$
BetaBlockers	$0.247$	$0.319$	$0.327$	$\mathbf{0.344}$	$0.275$
CalciumChannelBlockers	$0.295$	$\mathbf{0.492}$	$0.469$	$0.479$	$0.350$

Table 7: Macro-averaged AUPRC scores for the Few-Shot-Selection experiment.

Method	random	minilm	distroberta	mpnet	mpnet-bal
ACEInhibitors	$95.7$	$81.3$	$81.3$	$83.5$	$76.8$
ADHD	$95.4$	$77.3$	$82.1$	$81.9$	$76.6$
Antihistamines	$93.4$	$80.1$	$80.8$	$83.3$	$84.7$
AtypicalAntipsychotics	$95.2$	$79.6$	$79.4$	$81.2$	$80.6$
BetaBlockers	$94.9$	$80.7$	$80.4$	$82.7$	$79.3$
CalciumChannelBlockers	$94.5$	$81.1$	$82.1$	$82.0$	$80.1$

Table 8: Percent of training set selected as examples during the Few-Shot-Selection experiment.

Variant	base-zs	base-fs	chu	chu-fs	cot1	cot2	tot
ACEInhibitors	$0.172$	$\mathbf{0.305}$	$0.099$	$0.093$	$0.103$	$0.092$	$0.096$
ADHD	$0.551$	$\mathbf{0.647}$	$0.398$	$0.353$	$0.426$	$0.377$	$0.259$
Antihistamines	$0.459$	$\mathbf{0.517}$	$0.382$	$0.376$	$0.385$	$0.363$	$0.353$
AtypicalAntipsychotics	$0.520$	$\mathbf{0.616}$	$0.370$	$0.367$	$0.373$	$0.375$	$0.373$
BetaBlockers	$0.246$	$\mathbf{0.327}$	$0.193$	$0.184$	$0.182$	$0.175$	$0.174$
CalciumChannelBlockers	$0.297$	$\mathbf{0.469}$	$0.258$	$0.256$	$0.258$	$0.260$	$0.258$

Table 9: Macro-averaged AUPRC scores for the Advanced-Techniques experiment.

Variant	base-zs	base-fs	chu	chu-fs	cot1	cot2	tot
ACEInhibitors	$0.099$	$0.104$	$0.106$	$0.097$	$\mathbf{0.136}$	$0.102$	$0.107$
ADHD	$0.470$	$0.472$	$0.481$	$0.448$	$\mathbf{0.540}$	$0.520$	$0.349$
Antihistamines	$0.214$	$0.190$	$\mathbf{0.282}$	$0.235$	$0.259$	$0.207$	$0.141$
AtypicalAntipsychotics	$0.153$	$0.144$	$0.181$	$0.174$	$0.157$	$\mathbf{0.195}$	$0.163$
BetaBlockers	$0.129$	$0.130$	$0.145$	$0.130$	$\mathbf{0.146}$	$0.132$	$0.118$
CalciumChannelBlockers	$\mathbf{0.134}$	$0.121$	$0.117$	$0.124$	$0.095$	$0.121$	$0.103$

Table 10: Macro-averaged MCC scores for the Advanced-Techniques experiment.

Alg	gemm	ll-70b	ll-8b	mistr	gptmi	gptna	nb	rf-d	rf-t	debe	scid
Ace	$0.323$	$0.304$	$0.341$	$0.252$	$0.113$	$0.205$	$0.314$	$0.310$	$0.387$	$\mathbf{0.429}$	$0.402$
Adh	$\mathbf{0.693}$	$0.676$	$0.684$	$0.534$	$0.572$	$0.594$	$0.519$	$0.681$	$0.643$	$0.549$	$0.671$
Ant	$0.485$	$0.504$	$0.572$	$0.506$	$0.402$	$0.564$	$0.548$	$0.547$	$\mathbf{0.640}$	$0.529$	$0.408$
Aty	$0.620$	$0.615$	$0.624$	$0.604$	$0.497$	$0.575$	$0.667$	$0.638$	$0.674$	$0.593$	$\mathbf{0.678}$
Bet	$0.292$	$0.337$	$0.349$	$0.329$	$0.222$	$0.322$	$0.400$	$0.349$	$0.416$	$0.384$	$\mathbf{0.454}$
Cal	$0.441$	$0.486$	$0.491$	$0.458$	$0.314$	$0.374$	$0.569$	$0.663$	$\mathbf{0.686}$	$0.510$	$0.630$

Table 11: Macro-averaged AUPRC scores for the Baseline experiment.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Name	Role	Message content
A	system	Is this an abstract of a high-quality article on {topic}? Answer yes or no.
A	user	{text}
B	system	You are a helpful assistant.
B	user	Is this an abstract of a high-quality article on {topic}? Answer yes or no. ”{text}”
C	system	You are an expert on medical articles classification. You are going to help classify article abstracts. The user will pass in a text and you should answer yes if it is an abstract of a high-quality article on the topic of {topic} and no otherwise.
C	user	{text}

Appendix

Table 12: Prompts used in Prompts experiment.

Name	Role	Message content
B (Score)	system	You are a helpful assistant.
B (Score)	user	Is this an abstract of a high-quality article on {topic}? Rank the article on a scale from 0 to 5, where 5 means that it is a high-quality article on {topic}. ”{text}”
F (Score)	system	You are an AI assistant and you are very good at text classification. You are going to help classify medical article abstracts. You will be given an abstract of an article considered for inclusion in a review of scientific articles on the topic of {topic}.
F (Score)	user	Should the article the following abstract describes be included in the review on {topic}? ”{text}” Rank the article on a scale from 0 to 5, where 0 means ”should not be included” and 5 means ”should definitely be included”.
B (JSON)	system	You are a helpful assistant.
B (JSON)	user	Is this an abstract of a high-quality article on {topic}? Answer in JSON format: {”high_quality”: false} or {”high_quality”: true}. ”{text}”

Appendix

Table 13: Prompts used in Output Experiment.

Name	Role	Message content
B (few-shot)	system	You are a helpful assistant.
	user	Is this an abstract of a high-quality article on {topic}? Answer yes or no. ”{example_text}”
	assistant	{example_answer}
	user	Is this an abstract of a high-quality article on {topic}? Answer yes or no. ”{text}”

Appendix

Table 14: Prompt used in Few-Shot-Count and Few-Shot-Selection experiments.

Name	Role	Message content
CoT (1 call)	system	You are an AI assistant and you are very good at text classification. You are going to help classify medical article abstracts. You will be given an abstract of an article considered for inclusion in a review of scientific articles on the topic of {topic}.
	user	Should the article the following abstract describes be included in the review on {topic}? ”{text}” Answer in JSON format: {”included”: false} or {”included”: true}.
	assistant	Let’s think step by step.
CoT (2 calls) Model $1$	system	You are an AI assistant and you are very good at text classification. You are going to help classify medical article abstracts. You will be given an abstract of an article considered for inclusion in a review of scientific articles on the topic of {topic}.
	user	From this abstract, extract phrases that could point towards the article containing high quality evidence or a lack thereof. ”{text}” Give your answer as a JSON list.
	assistant	[”

Appendix

Table 15: Prompts used in Advanced-Techniques experiment.

Output	Word	JSON	Score	Token-Word	Token-JSON
ACEInhibitors	$0.099$	$0.091$	$0.079$	$\mathbf{0.099}$	$\mathbf{0.099}$
ADHD	$0.482$	$0.504$	$0.419$	$0.510$	$\mathbf{0.516}$
Antihistamines	$0.234$	$0.228$	$0.168$	$\mathbf{0.240}$	$0.221$
AtypicalAntipsychotics	$0.145$	$\mathbf{0.176}$	$0.097$	$0.151$	$0.168$
BetaBlockers	$0.130$	$0.122$	$0.106$	$\mathbf{0.136}$	$0.135$
CalciumChannelBlockers	$0.122$	$0.114$	$0.110$	$\mathbf{0.126}$	$0.106$

Appendix

Table 16: Macro-averaged MCC scores for the Output experiment.

Examples	0	1	2	3	4
ACEInhibitors	$0.099$	$\mathbf{0.116}$	$0.113$	$0.091$	$0.088$
ADHD	$0.470$	$\mathbf{0.482}$	$0.480$	$0.468$	$0.466$
Antihistamines	$\mathbf{0.214}$	$0.202$	$0.184$	$0.181$	$0.171$
AtypicalAntipsychotics	$0.153$	$\mathbf{0.182}$	$0.153$	$0.142$	$0.137$
BetaBlockers	$0.129$	$\mathbf{0.153}$	$0.126$	$0.125$	$0.123$
CalciumChannelBlockers	$0.134$	$\mathbf{0.174}$	$0.137$	$0.116$	$0.124$

Appendix

Table 17: Macro-averaged MCC scores the Few-Shot-Count experiment.

Method	random	minilm	distroberta	mpnet	mpnet-bal
ACEInhibitors	$0.102$	$0.091$	$\mathbf{0.104}$	$0.098$	$0.077$
ADHD	$0.482$	$0.468$	$0.472$	$\mathbf{0.487}$	$0.372$
Antihistamines	$\mathbf{0.197}$	$0.180$	$0.190$	$0.191$	$0.157$
AtypicalAntipsychotics	$\mathbf{0.156}$	$0.142$	$0.144$	$0.138$	$0.118$
BetaBlockers	$\mathbf{0.130}$	$0.125$	$\mathbf{0.130}$	$0.125$	$0.120$
CalciumChannelBlockers	$0.112$	$0.115$	$0.121$	$\mathbf{0.128}$	$0.126$

Appendix

Table 18: Macro-averaged MCC scores for the Few-Shot-Selection experiment.

Alg	gemm	ll-70b	ll-8b	mistr	gptmi	gptna	nb	rf-d	rf-t	debe	scid
Ace	$0.164$	$0.134$	$0.121$	$-0.003$	$0.135$	$0.190$	$0.327$	$0.219$	$0.243$	$\mathbf{0.368}$	$0.360$
Adh	$0.634$	$\mathbf{0.646}$	$0.563$	$0.044$	$0.570$	$0.587$	$0.558$	$0.593$	$0.562$	$0.534$	$0.599$
Ant	$0.261$	$0.269$	$0.193$	$0.036$	$0.223$	$0.277$	$0.356$	$0.268$	$\mathbf{0.417}$	$0.192$	$0.038$
Aty	$0.219$	$0.176$	$0.135$	$0.047$	$0.228$	$0.312$	$0.421$	$0.385$	$0.270$	$0.378$	$\mathbf{0.473}$
Bet	$0.146$	$0.164$	$0.162$	$0.046$	$0.137$	$0.207$	$0.304$	$0.232$	$0.205$	$0.237$	$\mathbf{0.350}$
Cal	$0.140$	$0.146$	$0.168$	$0.028$	$0.163$	$0.187$	$0.478$	$0.507$	$0.509$	$0.318$	$\mathbf{0.518}$

Appendix

Table 19: Macro-averaged MCC scores for the Baseline experiment.

Prompt	Model	Accuracy	Recall	Precision	F1 Score	MCC
A	Gemma 2 9B	$0.412$	$0.933$	$0.231$	$0.356$	$0.175$
	Llama 3.1 70B	$0.444$	$0.921$	$0.245$	$0.369$	$0.196$
	Llama 3.1 8B	$0.347$	$0.950$	$0.213$	$0.334$	$0.124$
	Mistral 7B	$0.204$	$\mathbf{0.996}$	$0.189$	$0.303$	$0.030$
	Average	$0.352$	$0.950$	$0.219$	$0.341$	$0.131$
B	Gemma 2 9B	$0.440$	$0.916$	$0.242$	$0.367$	$0.193$

Appendix

Table 20: Performance scores for the Prompts experiment macro-averaged over all datasets.

Prompt	Output	Model	Acc	Rec	Prec	F1	MCC	AUPRC
B	Word	Gemma 9B	$0.440$	$0.916$	$0.242$	$0.367$	$0.193$	$0.236$
		Llama 70B	$0.520$	$0.869$	$0.266$	$0.389$	$0.230$	$0.258$
		Llama 8B	$0.397$	$0.951$	$0.228$	$0.353$	$0.172$	$0.226$
		Mistral 7B	$0.225$	$0.993$	$0.191$	$0.307$	$0.057$	$0.191$
		Average	$0.396$	$0.932$	$0.232$	$0.354$	$0.163$	$0.228$
B	JSON	Gemma 9B	$0.439$	$0.900$	$0.235$	$0.358$	$0.181$	$0.230$

Appendix

Table 21: Performance scores for the Output experiment macro-averaged over all datasets.

Examples	Model	Acc	Rec	Prec	F1	MCC	AUPRC
0	Gemma 2 9B	$0.487$	$0.876$	$0.255$	$0.378$	$0.209$	$0.403$
	Llama 3.1 70B	$0.479$	$0.904$	$0.255$	$0.380$	$0.217$	$0.376$
	Llama 3.1 8B	$0.449$	$0.918$	$0.243$	$0.366$	$0.194$	$0.366$
	Mistral 7B	$0.236$	$0.991$	$0.193$	$0.308$	$0.066$	$0.308$
	Average	$0.413$	$0.922$	$0.236$	$0.358$	$0.171$	$0.363$
1	Gemma 2 9B	$0.545$	$0.840$	$0.271$	$0.392$	$0.232$	$0.438$

Appendix

Table 22: Performance scores for the Few-Shot-Count experiment macro-averaged over all datasets.

Method	Model	Acc	Rec	Prec	F1	MCC	AUPRC
random	Gemma 2 9B	$\mathbf{0.532}$	$0.858$	$\mathbf{0.270}$	$\mathbf{0.392}$	$\mathbf{0.233}$	$0.377$
	Llama 3.1 70B	$0.439$	$0.934$	$0.246$	$0.372$	$0.202$	$0.359$
	Llama 3.1 8B	$0.462$	$0.876$	$0.242$	$0.364$	$0.184$	$0.351$
	Mistral 7B	$0.214$	$0.996$	$0.191$	$0.307$	$0.058$	$0.340$
	Average	$\mathbf{0.412}$	$0.916$	$\mathbf{0.238}$	$0.359$	$\mathbf{0.169}$	$0.357$
minilm	Gemma 2 9B	$0.515$	$0.872$	$0.261$	$0.384$	$0.223$	$0.455$

Appendix

Table 23: Performance scores for the Few-Shot-Selection experiment macro-averaged over all datasets.

Variant	Model	Acc	Rec	Prec	F1	MCC	AUPRC
base-zs	Gemma 2 9B	$0.487$	$0.876$	$0.255$	$0.378$	$0.209$	$0.403$
	Llama 3.1 70B	$0.479$	$0.904$	$0.255$	$0.380$	$0.217$	$0.376$
	Llama 3.1 8B	$0.449$	$0.918$	$0.243$	$0.366$	$0.194$	$0.366$
	Mistral 7B	$0.236$	$0.991$	$0.193$	$0.308$	$0.066$	$0.308$
	Average	$0.413$	$0.922$	$0.236$	$0.358$	$0.171$	$0.363$
base-fs	Gemma 2 9B	$0.523$	$0.864$	$0.264$	$0.387$	$0.226$	$0.469$

Appendix

Table 24: Performance scores for the Advanced-Techniques experiment macro-averaged over all datasets.

Algorithm	Accuracy	Recall	Precision	F1 Score	MCC	AUPRC
gemm	$0.523$	$0.864$	$0.264$	$0.387$	$0.226$	$0.469$
ll-70b	$0.455$	$0.928$	$0.250$	$0.375$	$0.208$	$0.485$
ll-8b	$0.437$	$0.927$	$0.239$	$0.365$	$0.188$	$0.495$
mistr	$0.197$	$\mathbf{0.998}$	$0.188$	$0.303$	$0.028$	$0.456$
gptmi	$0.546$	$0.811$	$0.272$	$0.385$	$0.221$	$0.335$
gptna	$0.711$	$0.664$	$0.338$	$0.428$	$0.291$	$0.439$

Appendix

Table 25: Performance scores for the Baseline experiment macro-averaged over all datasets.

Explore similar work

Med-V1: Small Language Models for Zero-shot and Scalable Biomedical Evidence Attribution
Mar 5, 2026Qiao Jin, Yin Fang, Lauren He +12LLM Answer Verification Healthcare
Division of Intramural Research, National Library of Medicine, National Institutes of Health · Department of Computer Science, University of Virginia · Center for Cancer Research, National Cancer Institute, National Institutes of Health +3
Small LLMs for Biomedical Claim Verification: Cost-Effective Fine-Tuning, Structural Dataset Shortcuts, and Cross-Domain Generalization
Jun 11, 2026Gaurav KumarClaim Verification LLM Fine-Tuning
Moveworks AI · University of California San Diego
Few-Shot Biomedical Relation Extraction with Large Language Models: A Viable Alternative to Supervised Learning?
Jun 13, 2026Jakob Mraz, Tomaž Curk, Blaž ZupanRelation Extraction Few-Shot Learning
University of Ljubljana, Ljubljana, Slovenia · Baylor College of Medicine, Houston, Texas, USA

Mar 5, 2026J/K move · Enter open · S save

Med-V1: Small Language Models for Zero-shot and Scalable Biomedical Evidence Attribution

Qiao Jin, Yin Fang, Lauren He, Yifan Yang+11

Division of Intramural Research, National Library of Medicine, National Institutes of Health · Department of Computer Science, University of Virginia · Center for Cancer Research, National Cancer Institute, National Institutes of Health+3

LLM Answer Verification Healthcare Small Language Models LLM Hallucination Detection Medical LLMs Scientific Claim Verification

Open paper AskPDF arXiv

Assessing whether an article supports an assertion is essential for hallucination detection and claim verification. While large language models (LLMs) have the potential to automate this task, achieving strong performance requires frontier models such as GPT-5 that are prohibitively expensive to deploy at scale. To efficiently perform biomedical evidence attribution, we present Med-V1, a family of small language models with only three billion parameters. Trained on high-quality synthetic data newly developed in this study, Med-V1 substantially outperforms (+27.0% to +71.3%) its base models on five biomedical benchmarks unified into a verification format. Despite its smaller size, Med-V1 performs comparably to frontier LLMs such as GPT-5, along with high-quality explanations for its predictions. We use Med-V1 to conduct a first-of-its-kind use case study that quantifies hallucinations in LLM-generated answers under different citation instructions. Results show that the format instruction strongly affects citation validity and hallucination, with GPT-5 generating more claims but exhibiting hallucination rates similar to GPT-4o. Additionally, we present a second use case showing that Med-V1 can automatically identify high-stakes evidence misattributions in clinical practice guidelines, revealing potentially negative public health impacts that are otherwise challenging to identify at scale. Overall, Med-V1 provides an efficient and accurate lightweight alternative to frontier LLMs for practical, real-world biomedical evidence attribution. Med-V1 is available at https://github.com/NLM-DIR/Med-V1.

Abstract

Explore similar work

Med-V1: Small Language Models for Zero-shot and Scalable Biomedical Evidence Attribution

Small LLMs for Biomedical Claim Verification: Cost-Effective Fine-Tuning, Structural Dataset Shortcuts, and Cross-Domain Generalization

Few-Shot Biomedical Relation Extraction with Large Language Models: A Viable Alternative to Supervised Learning?