A dataset of medication images with instance segmentation masks for preventing adverse drug events

W. I. Chu; S. Hirani; G. Tarroni; L. Li

cs.CVMar 11, 2026

A dataset of medication images with instance segmentation masks for preventing adverse drug events

Authors: W. I. Chu, S. Hirani, G. Tarroni, L. Li

Organizations: City St George’s, University of London, School of Science & Technology, London, EC1V 0HB, United Kingdom · City St George’s, University of London, School of Health & Medical Sciences, London, EC1V 0HB, United Kingdom · Imperial College London, Dept of Computer Science, London, SW7 2AZ, United Kingdom

Abstract

Medication errors and adverse drug events (ADEs) pose significant risks to patient safety, often arising from difficulties in reliably identifying pharmaceuticals in real-world settings. AI-based pill recognition models offer a promising solution, but the lack of comprehensive datasets hinders their development. Existing pill image datasets rarely capture real-world complexities such as overlapping pills, varied lighting, and occlusions. MEDISEG addresses this gap by providing instance segmentation annotations for 32 distinct pill types across 8262 images, encompassing diverse conditions from individual pill images to cluttered dosette boxes. We trained YOLOv8 and YOLOv9 on MEDISEG to demonstrate their usability, achieving mean average precision at IoU 0.5 of 99.5 percent on the 3-Pills subset and 80.1 percent on the 32-Pills subset. We further evaluate MEDISEG under a few-shot detection protocol, demonstrating that base training on MEDISEG significantly improves recognition of unseen pill classes in occluded multi-pill scenarios compared to existing datasets. These results highlight the dataset's ability not only to support robust supervised training but also to promote transferable representations under limited supervision, making it a valuable resource for developing and benchmarking AI-driven systems for medication safety.

Figures & tables

Figure 1: (A) Trends in AEMT events from 1980 to 2014 and (B) the distribution of these events across age groups [ 10 ] .

Figure 2: Examples of images taken from (A) the dataset by Lee et al. [ 4 ] , (B) the NIH Pillbox dataset [ 15 ] , (C) the CURE dataset [ 5 ] , (D) the dataset by Wong et al. [ 14 ] , and (E) the dataset by Tan et al. [ 11 ] .

Figure 3: Pills organised in a standard four-by-seven dosette box (left) and after cropping into individual images (right).

Figure 4: Top-down and side profile views of Pill A (left), Pill B (centre), and Pill C (right).

Figure 5: Examples of image variability in the MEDISEG dataset, ranging from single-pill frames to compositions containing up to six pills.

Figure 6: Visualisation examples for the MEDISEG (3-Pills) dataset with instance masks, bounding boxes, and labels.

Figure 7: Visualisation examples for the MEDISEG (32-Pills) dataset with instance masks, bounding boxes, and labels.

Figure 8: Example images of the 32 unique pill types included in the MEDISEG (32-Pills) dataset.

Figure 9: Spatial distribution of bounding boxes across the MEDISEG (32-Pills) dataset.

Figure 10: Instance distribution of images in the MEDISEG dataset.

Figure 11: Fitness score progression over 70 iterations of hyperparameter tuning using GA

Figure 12: Comparison of YOLOv8 and YOLOv9 training metrics on the MEDISEG (3-Pills) dataset.

Figure 13: Confusion matrices for YOLOv8 (left) and YOLOv9 (right) trained on the MEDISEG (3-Pills) dataset, evaluated on the test set.

Figure 14: Common misclassifications: (A) Pill B and (B) Pill C. Left: Sample image; Middle: Top-down view; Right: Side profile.

Figure 16: Comparison of YOLOv8 and YOLOv9 training metrics on the MEDISEG (32-Pills) dataset.

Figure 17: Confusion matrices for YOLOv8 (left) and YOLOv9 (right) trained on the MEDISEG (32-Pills) dataset, evaluated on the test set.

Figure 18: Examples of model prediction on 32-Pills test images.

Figure 19: Detection results under 1-shot, 5-shot, and 10-shot fine-tuning for models initialised from MEDISEG and CURE.

	Lee et al. [ 4 ]	NIH Pillbox	NIH Pillbox	CURE [ 5 ]	Wong et al. [ 14 ]	Tan et al. [ 11 ]
		(Reference) [ 15 ]	(Consumer) [ 15 ]
Total Images	15,031	4,392	133,774	8,973	5,284	5,131
Labels	None	None	None	Partial Instance	None	Bounding Box
				Segmentation
Publicly Available	No	Yes	No	Yes	No	No
Synthetic images	No	No	No	Yes	No	No

Table 1: Comparison between the different pill recognition datasets.

			MEDISEG	MEDISEG
	NIH Pillbox [ 15 ]	CURE [ 5 ]	(3-Pills) (Ours)	(32-Pills) (Ours)
Number of Pill Images	4392	8973	2333	8262
Pill Categories	9100	196	3	32
Instances per Image	1	1	1-6	1-11
Instances per Category	1-2	18-60	1181-1900	61-1900
Segmentation Labels	None	Partially labelled	All	All

Table 2: Comparison of the publicly available datasets and MEDISEG datasets.

	fg_cls_accuracy	false_negative	loss_cls	loss_rpn_cls	total_loss
CURE 1-shot	0.131	0.816	0.351	0.863	1.326
CURE 5-shot	0.372	0.465	0.421	0.224	0.844
CURE 10-shot	0.558	0.342	0.320	0.133	0.674
MEDISEG 1-shot	0.406	0.513	0.383	0.312	0.963
MEDISEG 5-shot	0.625	0.246	0.279	0.182	0.680
MEDISEG 10-shot	0.740	0.210	0.191	0.059	0.445

Table 3: Few-shot detection results on the overlap-only test set for models base-trained on CURE and MEDISEG under 1-, 5-, and 10-shot settings.

epoch	train	train	val	val	precision	recall	F1-	mAP	mAP
	box loss	cls loss	box loss	cls loss			Score	50	50-95
1	0.4425	2.3809	0.38846	1.2704	0.98261	0.9841	0.98335	0.99245	0.90272
2	0.37488	0.89534	0.3755	0.83065	0.99558	0.99392	0.99475	0.99338	0.90903
3	0.36016	0.78955	0.37174	0.72901	0.99494	0.99544	0.99519	0.9933	0.91417
4	0.33755	0.68899	0.33344	0.66842	0.99431	0.99376	0.99403	0.99405	0.92862
5	0.31921	0.62769	0.35102	0.60851	0.99713	0.99555	0.99634	0.99438	0.92478

Table 4: YOLOv8 training results on MEDISEG (3-Pills)

epoch	train	train	val	val	precision	recall	F1-	mAP	mAP
	box loss	cls loss	box loss	cls loss			Score	50	50-95
1	0.41587	1.4857	0.51442	1.1569	0.35119	0.88681	0.50313	0.38442	0.34005
2	0.40709	0.67327	0.51442	1.1569	0.35119	0.88681	0.50313	0.38442	0.34005
3	0.4067	0.67247	4.007	1023.8	0.00065	0.10815	0.00129	0.00063	0.00019
4	0.38313	0.58268	0.50564	1.1569	0.94721	0.88833	0.91683	0.93286	0.81646
5	0.36867	0.50864	0.35888	0.56471	0.9764	0.98175	0.97907	0.99162	0.92605

Table 5: YOLOv9 training results on MEDISEG (3-Pills)

epoch	train	train	val	val	precision	recall	F1-	mAP	mAP
	box loss	cls loss	box loss	cls loss			Score	50	50-95
1	0.73515	5.5465	0.79503	3.4362	0.43798	0.24101	0.31093	0.18367	0.14918
2	0.604	2.6415	0.68777	2.5692	0.5086	0.34743	0.41284	0.31229	0.25544
3	0.53917	2.0917	0.67365	2.1735	0.55653	0.41183	0.47337	0.41752	0.33367
4	0.51319	1.7586	0.63762	1.9514	0.59987	0.47221	0.52844	0.50077	0.40223
5	0.49068	1.4985	0.62279	1.8043	0.60842	0.49662	0.54686	0.54505	0.43708

Table 6: YOLOv8 training results on MEDISEG (32-Pills)

epoch	train	train	val	val	precision	recall	F1-	mAP	mAP
	box loss	cls loss	box loss	cls loss			Score	50	50-95
1	0.50243	2.9938	0.62794	2.4668	0.56949	0.47	0.51498	0.4475	0.36437
2	0.49417	1.6132	0.61723	2.0512	0.19794	0.44166	0.27337	0.18848	0.14914
3	0.48833	1.3587	0.59486	1.5773	0.51456	0.62207	0.56323	0.60872	0.48423
4	0.46465	1.1243	0.5547	1.4804	0.71965	0.55804	0.62862	0.62773	0.51653
5	0.44736	0.97132	0.54143	1.3826	0.70975	0.64348	0.67499	0.68883	0.56494

Table 7: YOLOv9 training results on MEDISEG (32-Pills)

Object Detection

Object detection locates and classifies objects in images or video, balancing localization accuracy, robustness, and inference speed.

See all 624 papers

Evaluating Few-Shot Pill Recognition Under Visual Domain Shift
Mar 11, 2026W. I. Chu, G. Tarroni, L. LiFew-Shot Object Detection Object Detection
Department of Engineering, City St George’s, University of London, London, United Kingdom · Department of Computer Science, City St George’s, University of London, London, United Kingdom
FDDet: Achieving Data-Efficient Food Defect Detection Under Real-World Scenarios
May 23, 2026Ruihao Xu, Yong Liu, Yansong TangObject Detection Semi-Supervised Learning
Tsinghua Shenzhen International Graduate School, Tsinghua University
EgoMAGIC- An Egocentric Video Field Medicine Dataset for Training Perception Algorithms
Apr 23, 2026Brian VanVoorst, Nicholas Walczak, Christopher Gilleo +9Medical Image Object Detection Egocentric Video Understanding
RTX BBN Technologies · New York University · Northeastern University

Mar 11, 2026J/K move · Enter open · S save

Evaluating Few-Shot Pill Recognition Under Visual Domain Shift

W. I. Chu, G. Tarroni, L. Li

Department of Engineering, City St George’s, University of London, London, United Kingdom · Department of Computer Science, City St George’s, University of London, London, United Kingdom

Few-Shot Object Detection Object Detection

Open paper AskPDF arXiv

Adverse drug events are a significant source of preventable harm, which has led to the development of automated pill recognition systems to enhance medication safety. Real-world deployment of these systems is hindered by visually complex conditions, including cluttered scenes, overlapping pills, reflections, and diverse acquisition environments. This study investigates few-shot pill recognition from a deployment-oriented perspective, prioritizing generalization under realistic cross-dataset domain shifts over architectural innovation. A two-stage object detection framework is employed, involving base training followed by few-shot fine-tuning. Models are adapted to novel pill classes using one, five, or ten labeled examples per class and are evaluated on a separate deployment dataset featuring multi-object, cluttered scenes. The evaluation focuses on classification-centric and error-based metrics to address heterogeneous annotation strategies. Findings indicate that semantic pill recognition adapts rapidly with few-shot supervision, with classification performance reaching saturation even with a single labeled example. However, stress testing under overlapping and occluded conditions demonstrates a marked decline in localization and recall, despite robust semantic classification. Models trained on visually realistic, multi-pill data consistently exhibit greater robustness in low-shot scenarios, underscoring the importance of training data realism and the diagnostic utility of few-shot fine-tuning for deployment readiness.

Abstract

Object Detection

Explore similar work

Evaluating Few-Shot Pill Recognition Under Visual Domain Shift

FDDet: Achieving Data-Efficient Food Defect Detection Under Real-World Scenarios

EgoMAGIC- An Egocentric Video Field Medicine Dataset for Training Perception Algorithms