Example-based Robust Abnormality Detection with Minimal Annotations using Exemplar Med-DETR

Sheethal Bhat; Bogdan Georgescu; Awais Mansoor; Mathias Zinnen; Pranjal Sahu; Florin C. Ghesu; Sasa Grbic; Andreas Maier

cs.CVAug 25, 2026

Example-based Robust Abnormality Detection with Minimal Annotations using Exemplar Med-DETR

Authors: Sheethal Bhat, Bogdan Georgescu, Awais Mansoor, Mathias Zinnen, Pranjal Sahu, Florin C. Ghesu, Sasa Grbic, Andreas Maier

Abstract

Reducing annotation requirements remains a key challenge in developing robust medical object detectors. To address this, Vision-Language (VL) object detection methods leverage grounding text information to enable powerful zero-shot and few-shot object detectors in the natural image domain [1, 2, 3, 4]. However, transferring these methods to the medical domain is challenging due to the absence of comparable quality and quantity of the grounding data. Regardless, significant contextual and non-imaging information exists in medical images that remains underutilized. Few-shot learning (FSL) techniques partially address this limitation but struggle to general ize to unseen medical findings and require extensive retraining when new findings are introduced [5, 6]. To overcome these challenges, we extend our prior EM-DETR framework [7] and introduce a scalable FS detection approach designed for efficient abnormality detection in Chest X-Ray (CXR) images under minimal supervision. The proposed architecture incorporates exemplar-based feature generation and domain-aware contrastive optimization, enabling effective adaptation to novel disease findings without exhaustive retraining. Our method achieves near state-of-the-art (SOTA) detection performance using less than 10% of the annotated data, demonstrating its potential for practical, annotation-efficient clinical deployment across both proprietary and public CXR datasets.

Figures & tables

Figure 2: Example image from the anatomy dataset, where we only use the frontal views and discard the lateral views.

Finding	Images (Positive samples)
Finding	Training	Test
Pleural effusion	18,681(4037)	331(74)
Pneumothorax	18,681(1868)	412(146)
Lesion	18,681(5654)	412(109)

Table 1: Number of images per finding in the training and test sets. The number in the brackets indicate the number of positive samples for each finding.

Figure 3: Overview of EM-DETR pretraining. Visual features $\mathbf{X}_{k}$ & positional embeddings $\mathbf{P}_{k}$ are extracted from the frozen image backbone based on the $k^{th}$ class location (red). $\mathbf{F}$ uses learnable class-specific embeddings $\mathbf{ct}_{k}$ & $\mathbf{ce}_{k}$ to calculate $\mathbf{q}_{k}$ . A moving average of $\mathbf{q}_{k}$ results in the prototype embedding $\mathbf{e}_{k}$ . Interleaving the text embedding $\mathbf{t}_{k}$ with $\mathbf{e}_{k}$ enables text-and feature-based detection for each base class. $\mathbf{e}_{k}$ is used in additional $\mathcal{L}_{contrast}$ & $\mathcal{L}_{feat}$ losses to improve the decoder search operations.

Figure 4: Representation of the per-class interleaving of the text and feature embeddings. The tokens generated from the encoder undergo self-attention at a sub-sentence level through additional attention masks [ 1 ] .

Figure 5: Schematic of iterative training strategy, where $*$ denotes generated annotations.

Figure 6: Overview of the finetuning and inference process. The memory bank accumulated during Stage I is either discarded or the novel class embeddings are added to create a new memory bank II. The inference path is shown in dotted arrow line.

Panel of Figure 7 — Figure 7: 2D t-SNE analysis of pneumothorax (left) and pleural effusion (right) prototypes contrasted with their background samples.

Figure 8: Diagram to show how background areas are selected based on the finding of interest.

Category	Baseline G.DINO [ 1 ]	EM-DETR [ 7 ]
Clavicle	0.94	0.95
Aorta	1.0	0.99
Heart	0.23	0.92
Lungs	1.0	1.0
Upper ribs	1.0	1.0
Middle ribs	1.0	0.97

Table 2: Few shot detection of the heart class with only 5 training images for heart and 80 training images for other classes. The results are depicted in average

mAP_{50}

. The inclusion of the Exemplar Generation module

\mathcal{G}

results in a decrease in detection performance for classes that overlap with the heart region due to overlapping region boundaries.

	Method	Data %	Sensitivity		AUC	$mAP_{50}$
			0.25 FP	0.5 FP
Ptx	Baseline G.DINO [ 1 ]	7%	$0.518\pm\gamma$	$0.634\pm\gamma$	$0.837\pm\epsilon$	$0.258\pm 0.05$
	RPS [ 18 ]	7%	0.518	0.642	0.877	-
	EM-DETR [ 7 ]	7%	$0.885\pm\epsilon$	$0.932\pm\epsilon$	$0.957\pm 0$	$0.427\pm 0.02$
	Baseline G.DINO [ 1 ]	100%	$0.843\pm\gamma$	$0.925\pm\gamma$	$0.960\pm\epsilon$	$0.524\pm 0.03$
	RPS [ 18 ]	100%	0.970	0.970	0.983	-

Table 3: Comparison of Sensitivity and AUC Scores for Different Findings at Various False Positive Rates and Dataset Sizes. All experiments conducted with a Swin-B image encoder backbone. In the case of Lesion

\dagger

we evaluate the highest scale of the multi-scale feature from the Swin-B backbone. For readability we denote the standard deviation as

\epsilon\leq 0.01

and

\gamma\leq 0.1

Panel of Figure 9 — Figure 9: Example FROC curves from EM-DETR for a pair of findings, displaying the average and standard deviation when finetuned on less than 10% of the training data.

Method (Dataset size)	Sensitivity		AUC
	0.25FP	0.5FP
EM-DETR (7%) [ 7 ]	$0.722\pm 0.02$ *	$0.833\pm 0.01$	$0.909\pm 0.01$ *
Grounding DINO (7%) [ 1 ]	$0.556\pm 0.06$	$0.777\pm 0.13$	$0.853\pm 0.02$
RPS (7%) [ 18 ]	0.278	0.333	0.682
Deformable DETR (7%) [ 23 ]	0.271	0.542	0.524
Deformable DETR (100%) [ 23 ]	0.944	0.944	0.995

Table 4: Comparison of pneumothorax performance with various SOTA models.

\dagger

indicates within-distribution tests where the model was trained and evaluated on VinDR-CXR dataset. * indicates statistical significance with p

<

0.05 when compared to the baseline. Standard deviation provided where available.

Configuration	Sensitivity		AUC	$mAP_{50}$
	0.25FP	0.5FP
Baseline	$0.518\pm 0.01$	$0.634\pm 0.01$	$0.837\pm 0.01$	$0.258\pm 0.11$
Baseline+bkgnd( $B$ )	$0.611\pm 0.01$	$0.756\pm 0.01$	$0.902\pm 0.01$	$0.331\pm 0.01$
Baseline + B+ $\mathcal{G}$	$0.728\pm 0.02$	$0.806\pm 0.02$	$0.939\pm 0.00$	$0.379\pm 0.03$
Baseline + $B$ + $\mathcal{G}$ + frozen $F$	$0.766\pm 0.01$	$0.878\pm 0.02$	$0.937\pm 0.00$	$0.346\pm 0.02$
EM-DETR	$0.885\pm 0.01$	$0.932\pm 0.01$	$0.957\pm 0.00$	$0.427\pm 0.02$

Table 5: Impact of each module on classification and detection performance for pneumothorax. Experiments are conducted using the Swin-Base backbone within the MMDetection framework, with default data augmentations. The first row shows the baseline performance. The second row adds

B

, indicating inclusion of background annotations in the minimal-shot training file. The third row adds

\mathcal{G}

, denoting the exemplar feature extraction module. In the fourth row, frozen

F

indicates that the attention pooling module is frozen after convergence to further refine the decoder. The last row is the full EM-DETR pipeline finetuned after pretraining on base classes.

Training data	Sensitivity		AUC	$mAP_{50}$
	0.25FP	0.5FP
50 ptx + bkgnd	$0.679\pm 0.01$	$0.773\pm 0.01$	0.914	$0.262\pm 0.24$
50 ptx + FP as bkgnd	$0.753\pm 0.01$	$0.856\pm 0.01$	0.920	$0.315\pm 0.02$
100 ptx + bkgnd	$0.823\pm 0.01$	$0.893\pm 0.01$	0.951	$0.340\pm 0.01$
200 ptx + bkgnd	$0.885\pm 0.01$	$0.932\pm 0.01$	0.957	$0.427\pm 0.02$
400 ptx + bkgnd	$0.897\pm 0.01$	$0.934\pm 0.00$	0.964	$0.506\pm 0.02$

Table 6: Comparison of pneumothorax performance with different number of positive annotations. The negative annotations are kept equal to the positive sample set in each experiment. The second row shows the impact of further retraining the model with False Positives as background annotations in a second stage of finetuning.

Figure 10: Top row illustrates examples of top 2 predictions for pleural effusion while bottom row illustrates examples of top 2 predictions for pneumothorax. The groundtruth is in yellow and predictions in green. We observe detected regions are more tightly localized than ground truth, which may span multiple nearby findings of same class, leading to an artificial reduction in mAP.

Figure 11: Example images of incorrect detections of pleural effusion on the test dataset. Yellow boxes mark the groundtruth annotation, while blue indicates the predicted pleural effusion with the probability score. The background regions are marked in orange and reflect the training distribution of background annotations.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 12: 2D TSNE analysis of feature embeddings per class when input to a torchxray vision backbone

Panel of Figure 13 — Figure 13: mAP@50 performance for the Heart class across different few-shot settings.

Object Detection

Object detection locates and classifies objects in images or video, balancing localization accuracy, robustness, and inference speed.

See all 624 papers

Dual Distillation for Few-Shot Anomaly Detection
Mar 2, 2026Le Dong, Qinzhong Tan, Chunlei Li +5Few-Shot Anomaly Detection Anomaly Detection
Xidian University, China · MedAI Technology (Wuxi) Co. Ltd., China · Technical University of Munich, Germany
Beyond Static Anchors: Bounded Prototype Conditioning for Language-Free Medical Anomaly Detection
Aug 1, 2026Yibo Wan, Jinyu Cai, See-kiong NgFew-Shot Anomaly Detection Medical Image Anomaly Detection
School of Computing, National University of Singapore, Singapore · Institute of Data Science, National University of Singapore, Singapore
Bridging Vision Foundation Model Priors with CLIP for Spatial-aware Few-shot Anomaly Detection in Medical Images
Sep 11, 2026Juzheng Miao, Yuchen Yuan, Cheng Chen +1Few-Shot Anomaly Detection Medical Image Anomaly Detection
Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China · Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong, China · School of Biomedical Engineering, The University of Hong Kong, Hong Kong, China +1

Mar 2, 2026J/K move · Enter open · S save

Dual Distillation for Few-Shot Anomaly Detection

Le Dong, Qinzhong Tan, Chunlei Li, Jingliang Hu+4

Xidian University, China · MedAI Technology (Wuxi) Co. Ltd., China · Technical University of Munich, Germany

Few-Shot Anomaly Detection Anomaly Detection Medical Image Anomaly Detection Knowledge Distillation

Open paper AskPDF arXiv

Anomaly detection is a critical task in computer vision with profound implications for medical imaging, where identifying pathologies early can directly impact patient outcomes. While recent unsupervised anomaly detection approaches show promise, they require substantial normal training data and struggle to generalize across anatomical contexts. We introduce D

^2

4FAD, a novel dual distillation framework for few-shot anomaly detection that identifies anomalies in previously unseen tasks using only a small number of normal reference images. Our approach leverages a pre-trained encoder as a teacher network to extract multi-scale features from both support and query images, while a student decoder learns to distill knowledge from the teacher on query images and self-distill on support images. We further propose a learn-to-weight mechanism that dynamically assesses the reference value of each support image conditioned on the query, optimizing anomaly detection performance. To evaluate our method, we curate a comprehensive benchmark dataset comprising 13,084 images across four organs, four imaging modalities, and five disease categories. Extensive experiments demonstrate that D

^2

4FAD significantly outperforms existing approaches, establishing a new state-of-the-art in few-shot medical anomaly detection. Code is available at https://github.com/ttttqz/D24FAD.

Abstract

Object Detection

Explore similar work

Dual Distillation for Few-Shot Anomaly Detection

Beyond Static Anchors: Bounded Prototype Conditioning for Language-Free Medical Anomaly Detection

Bridging Vision Foundation Model Priors with CLIP for Spatial-aware Few-shot Anomaly Detection in Medical Images