MedSAD-CLIP: Supervised CLIP with Token-Patch Cross-Attention for Medical Anomaly Detection and Segmentation

Thuy Truong Tran; Minh Kha Do; Phuc Nguyen Duy; Min Hun Lee

cs.CVMar 18, 2026

MedSAD-CLIP: Supervised CLIP with Token-Patch Cross-Attention for Medical Anomaly Detection and Segmentation

Authors: Thuy Truong Tran, Minh Kha Do, Phuc Nguyen Duy, Min Hun Lee

Organizations: Singapore Management University · La Trobe University · Vietnam National University, Hanoi

Abstract

Medical anomaly detection (MAD) and segmentation play a critical role in assisting clinical diagnosis by identifying abnormal regions in medical images and localizing pathological regions. Recent CLIP-based studies are promising for anomaly detection in zero-/few-shot settings, and typically rely on global representations and weak supervision, often producing coarse localization and limited segmentation quality. In this work, we study supervised adaptation of CLIP for MAD under a realistic clinical setting where a limited yet meaningful amount of labeled abnormal data is available. Our model MedSAD-CLIP leverages fine-grained text-visual cues via the Token-Patch Cross-Attention(TPCA) to improve lesion localization while preserving the generalization capability of CLIP representations. Lightweight image adapters and learnable prompt tokens efficiently adapt the pretrained CLIP encoder to the medical domain while preserving its rich semantic alignment. Furthermore, a Margin-based image-text Contrastive Loss is designed to enhance global feature discrimination between normal and abnormal representations. Extensive experiments on four diverse benchmarks-Brain, Retina, Lung, and Breast datasets-demonstrate the effectiveness of our approach, achieving superior performance in both pixel-level segmentation and image-level classification over state-of-the-art methods. Our results highlight the potential of supervised CLIP adaptation as a unified and scalable paradigm for medical anomaly understanding. Code will be made available at https://github.com/thuy4tbn99/MedSAD-CLIP

Figures & tables

Figure 1: (a) Qualitative result comparisons of few-shot MVFA and our supervised MedSAD-CLIP on a Breast image. (b) Radar comparison of MedSAD-CLIP and zero-/few-shot CLIP-based SOTAs on Brain and Breast datasets.

Figure 2: Overall architecture of our proposed MedSAD-CLIP. It consists of two branches: the classification branch (Det) and the segmentation branch (Seg). The model extends CLIP [ 30 ] with adapter modules and learnable prompts to adapt to medical image analysis.

Dataset	Anatomical region	Image type	#Train	#Test
Brain	Brain Tumor	MRI	3715	83
Retina	Retinal Edema	OCT	1805	115
Lung	Lung Covid	CT	894	383
Breast	Breast Cancer	Ultrasound	486	161

Table 1: Summary of datasets used.

Setting	Method	Brain		Retina		Lung		Breast
Setting	Method	Dice	Acc	Dice	Acc	Dice	Acc	Dice	Acc
Zero-shot	Adaclip [ 5 ]	43.52	56.63	38.62	61.74	9.09	56.25	36.19	23.60
	Aaclip [ 27 ]	22.68	55.42	47.98	73.91	21.73	66.41	41.54	50.00
	Anomalyclip [ 49 ]	43.25	53.01	37.25	60.86	7.47	55.46	13.52	26.92
Few-shot	MVFA [ 17 ]	48.34	54.87	66.93	79.82	68.54	76.56	52.09	70.22
	MadCLIP [ 36 ]	38.55	81.93	85.22	83.48	63.84	92.19	44.71	52.81

Table 2: Comparison results of MedSAD-CLIP and baselines under Zero-shot, Few-shot, and Supervised settings on four medical anomaly detection datasets. Dice (%) and Accuracy (%) are reported. The best results are in bold , and the second best are underlined .

Model	Brain		Retina		Lung		Breast
	Dice	Acc	Dice	Acc	Dice	Acc	Dice	Acc
CLIP-adapt [ 17 ]	64.71	92.68	91.18	97.12	73.19	94.53	76.57	88.76
CLIP-adapt + learnable prompt	58.62	85.37	91.79	98.25	81.43	97.66	76.40	91.01
CLIP-adapt + learnable prompt + TPCA	84.24	91.46	91.94	96.49	86.10	98.44	84.00	90.70
MedSAD-CLIP (learnable prompt +TPCA + MC-Loss)	89.47	96.34	93.18	97.37	87.16	99.22	84.96	91.01

Table 3: Ablation study results on four datasets. We report Dice and Accuracy (%) for both classification and segmentation tasks.

Figure 3: Qualitative comparison between our model MedSAD-CLIP and MVFA. Our MedSAD-CLIP produces sharper and more accurate segmentations.

	Brain			Retina			Lung			Breast
Method	Dice	Acc	pAUC	Dice	Acc	pAUC	Dice	Acc	pAUC	Dice	Acc	pAUC
Adaclip (zero) [ 5 ]	43.52	56.63	90.89	38.62	61.74	92.92	9.09	56.25	60.76	36.19	23.60	86.53
Adaclip (full) [ 5 ]	46.99	48.19	98.66	89.47	96.52	99.68	84.93	92.19	99.63	79.87	87.64	93.18
MVFA (zero) [ 17 ]	48.34	54.87	95.06	66.93	79.82	97.10	68.54	76.56	97.19	52.09	70.22	89.51
MVFA (full) [ 17 ]	64.71	92.68	99.18	91.18	97.12	99.53	73.19	94.53	98.07	76.57	88.76	90.79
MedSAD-CLIP	89.47	96.34	99.54	93.18	97.37	98.97	87.16	99.22	98.47	84.96	91.01	87.48

Table 4: Comparison of Dice, Accuracy, and pAUC across four datasets.

Figure 4: Qualitative results comparing MVFA (few/full) and our method on brain anomaly segmentation, showing both binary masks and score maps.

#Tokens	Brain		Retina		Lung		Breast
#Tokens	Dice	Acc	Dice	Acc	Dice	Acc	Dice	Acc
10	89.47	96.34	93.18	97.37	87.16	99.22	84.96	91.01
20	86.37	93.90	89.68	97.36	86.71	97.65	75.28	89.88
35	85.42	89.02	90.73	98.24	86.44	99.21	76.41	89.89

Table 5: Effect of the number of learnable tokens on Dice (%) and Accuracy (%) across four datasets.

Margin	Brain		Retina		Lung		Breast
Margin	Dice	Acc	Dice	Acc	Dice	Acc	Dice	Acc
0.2	84.87	89.02	92.18	97.36	84.64	98.43	76.40	94.38
0.4	89.47	96.34	93.18	97.37	87.16	99.22	84.96	91.01
0.6	84.66	93.90	89.69	96.49	86.21	98.45	76.38	90.44
0.8	81.17	82.92	62.92	96.49	85.53	98.43	75.28	92.69

Table 6: Effect of different margins in MC-Loss on Dice (%) and Accuracy (%) across four datasets.

	Aaclip [ 27 ]				Anomalyclip [ 49 ]
Threshold	Brain	Retina	Lung	Breast	Brain	Retina	Lung	Breast
0.3	14.04	38.03	9.54	10.74	46.99	39.13	44.53	76.40
0.4	18.59	46.94	12.06	20.81	46.99	39.13	44.53	76.40
0.5	22.68	47.98	21.73	41.54	46.99	39.13	44.53	76.40
0.6	35.48	43.94	27.32	56.92	46.99	39.13	44.53	76.40
0.7	41.59	41.88	34.79	67.21	46.99	39.13	44.53	76.40

Table 7: Comparison of threshold-based Dice (%) across baseline models on four datasets.

Medical Image Anomaly Detection

Medical image anomaly detection finds and localizes unusual anatomy or pathology in scans, often when labeled disease examples are scarce.

See all 87 papers

Bridging Vision Foundation Model Priors with CLIP for Spatial-aware Few-shot Anomaly Detection in Medical Images
Sep 11, 2026Juzheng Miao, Yuchen Yuan, Cheng Chen +1Few-Shot Anomaly Detection Medical Image Anomaly Detection
Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China · Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong, China · School of Biomedical Engineering, The University of Hong Kong, Hong Kong, China +1
PSMP-CLIP: Patch-Prompt SAM and Multi-Semantic Prompting for CLIP-Based Zero-Shot Anomaly Detection
Sep 15, 2026Xuezhi Xiang, Guanghao Wu, Heqi Xiang +4Promptable Image Segmentation Anomaly Localization
Information and Communication Engineering, Harbin Engineering University, Harbin, China · Key Laboratory of Advanced Marine Communication and Information Technology, Harbin, China · Department of Computer Science, University of Toronto, Toronto, ON M5S 2E4, Canada +1
MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation
Feb 23, 2026Taha Koleilat, Hojat Asgariandehkordi, Omid Nejati Manzari +3Medical Image Segmentation Text-Guided Image Segmentation
Concordia University, Montreal, Canada

Sep 11, 2026J/K move · Enter open · S save

Bridging Vision Foundation Model Priors with CLIP for Spatial-aware Few-shot Anomaly Detection in Medical Images

Juzheng Miao, Yuchen Yuan, Cheng Chen, Pheng-Ann Heng

Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China · Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong, China · School of Biomedical Engineering, The University of Hong Kong, Hong Kong, China+1

Few-Shot Anomaly Detection Medical Image Anomaly Detection Vision Foundation Model Adaptation Medical VLMs Medical Image Localization

Open paper AskPDF arXiv

Vision-Language Models such as CLIP enable effective few-shot medical anomaly detection (AD) via strong image-text semantic alignment. However, their globally contrastive pretraining lacks explicit spatial supervision, limiting precise lesion localization. In contrast, Vision Foundation Models (VFMs) such as DINO learn spatially coherent patch representations via self-distillation and local-to-global consistency, better capturing fine-grained anatomical structures. Leveraging this complementarity, we propose Spatial-FAD, a spatial-aware few-shot medical AD framework that improves lesion localization by combining VFM spatial priors with CLIP semantics. Specifically, we introduce a VFM-enhanced adapter that injects a structural affinity prior derived from DINO into CLIP features. This structure-guided refinement encourages visual embeddings to better adhere to lesion boundaries while maintaining semantic alignment. To address the loss of spatial detail from patchification and the limited input resolution of CLIP, we adopt a sliding-window aggregation strategy. This generates high-resolution, spatially dense embeddings to further enhance localization granularity. Moreover, we introduce a prototype-enhanced support memory scheme to efficiently exploit the few-shot support set. This module stores compact prototypes for normal and abnormal patterns, reducing memory costs while boosting performance by fusing patch-to-prototype and image-text similarities. Extensive experiments on three benchmark datasets, including Liver CT, Retinal OCT, and Brain MRI, demonstrate that Spatial-FAD significantly outperforms state-of-the-art methods, especially in lesion segmentation. Notably, in the 4-shot scenario, our method achieves an average improvement of over 11.4% in Dice score and 1.8% in AUC. Code is available at: https://github.com/JuzhengMiao/Spatial-FAD.

Abstract

Medical Image Anomaly Detection

Explore similar work

Bridging Vision Foundation Model Priors with CLIP for Spatial-aware Few-shot Anomaly Detection in Medical Images

PSMP-CLIP: Patch-Prompt SAM and Multi-Semantic Prompting for CLIP-Based Zero-Shot Anomaly Detection

MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation