MTL-MAD: Multi-Task Learners are Effective Medical Anomaly Detectors

Anomaly detection is essential for medical image analysis, where pathological regions often appear as rare deviations from normal anatomical structures. While training-based methods have achieved promising performance, they require task-specific optimization and extensive normal data, which limits scalability across modalities and institutions. Training-free approaches offer greater flexibility by leveraging pretrained visual representations, yet existing methods typically rely on simple nearest-neighbor retrieval and naive aggregation strategies, which may fail to capture hierarchical semantics and ignore the reliability of multiple anomaly responses. In this work, we propose HiMatch-AD, a DINOv3-driven hierarchical matching framework for training-free medical anomaly detection. Our method first retrieves semantically relevant normal references via dual-branch matching that jointly considers global CLS-token similarity and patch-level representations. Hierarchical anomaly maps are then generated across multiple transformer stages by comparing clustered normal features with query representations. To robustly aggregate anomaly responses, we introduce a unified uncertainty-based fusion mechanism that adaptively weights maps according to their reliability. The entire framework operates without any task-specific training. Extensive experiments on the BMAD benchmark, including brain MRI, liver CT, and retinal OCT datasets, demonstrate that HiMatch-AD consistently outperforms both training-based and DINO-based state-of-the-art methods, which highlights the effectiveness of multi-level matching and uncertainty-aware fusion for scalable medical anomaly detection.

Anomaly detection in medical images is challenging due to limited annotations and the domain gap. Existing reconstruction-based methods often rely on frozen pre-trained encoders, restricting adaptation to domain-specific patterns and degrading localization accuracy. Meanwhile, prototype-based learning offers interpretable representations but commonly suffers from prototype collapse, where a few prototypes dominate training and reduce diversity. To address these issues, we propose DNP-ConFormer, a unified framework that integrates a trainable encoder with prototype-guided reconstruction and a Diversity-Aware Alignment Loss. A momentum encoder enables stable domain-adaptive representation learning, while a lightweight Prototype Extractor discovers informative normal prototypes and injects them into the decoder via attention to guide reconstruction. The proposed alignment objective further encourages balanced feature-to-prototype assignments, effectively mitigating prototype collapse. Extensive experiments on multiple medical imaging benchmarks demonstrate improved representation quality and anomaly localization compared with prior methods. Visualization and prototype assignment analyses further validate the effectiveness and interpretability of our approach. The code is available at https://github.com/liluhu0/DNP-ConFormer.

Deploying AI-based anomaly detection across diverse clinical imaging settings remains challenging because most existing methods rely on modality-specific architectures, anatomical priors, or extensive retraining, limiting their use as general-purpose screening tools. One-class classification (OCC) offers a label-efficient alternative by training exclusively on normal data, but conventional two-stage pipelines fit a density estimator directly on raw pretrained embeddings, leaving substantial discriminative structure in the latent space unexploited. We introduce a training-free, modality-agnostic framework that inserts an explicit manifold-refinement stage between feature extraction and anomaly scoring. Empirical density weights, estimated via a UMAP-derived neighborhood graph, guide an iterative shift of embeddings toward locally dense regions, compacting normal samples, leaving anomalies relatively isolated prior to Gaussian density estimation and Mahalanobis-based scoring. This refinement introduces no additional trainable parameters and no architectural modification, allowing it to be layered onto any pretrained encoder. Evaluated on the MedIAnomaly benchmark across seven datasets spanning five imaging modalities (X-ray, MRI, fundus, dermatoscopy, histopathology), the framework achieves the best AUC on four datasets and the best Average Precision on five datasets among methods evaluated in the benchmark, outperforming specialized reconstruction and diffusion-based methods with a single fixed hyperparameter configuration across all modalities. These results demonstrate that meaningful gains can be achieved through post-hoc geometric refinement of existing representations rather than bespoke encoders, offering a practical and scalable AI screening framework for real-world, multi-modality clinical workflows where retraining and abnormal-case annotation are costly or infeasible.