Unsupervised Causal Prototypical Networks for De-biased Interpretable Dermoscopy Diagnosis

Junhao Jia; Yueyi Wu; Huangwei Chen; Haodong Jing; Haishuai Wang; Jiajun Bu; Lei Wu

eess.IVFeb 27, 2026

Unsupervised Causal Prototypical Networks for De-biased Interpretable Dermoscopy Diagnosis

Authors: Junhao Jia, Yueyi Wu, Huangwei Chen, Haodong Jing, Haishuai Wang, Jiajun Bu, Lei Wu

Organizations: Zhejiang Key Laboratory of Accessible Perception and Intelligent Systems, College of Computer Science and Technology, Zhejiang University · Hangzhou Dianzi University · Xi’an Jiaotong University

Abstract

Despite the success of deep learning in dermoscopy image analysis, its inherent black-box nature hinders clinical trust, motivating the use of prototypical networks for case-based visual transparency. However, inevitable selection bias in clinical data often drives these models toward shortcut learning, where environmental confounders are erroneously encoded as predictive prototypes, generating spurious visual evidence that misleads medical decision-making. To mitigate these confounding effects, we propose CausalProto, an Unsupervised Causal Prototypical Network that fundamentally purifies the visual evidence chain. Framed within a Structural Causal Model, we employ an Information Bottleneck-constrained encoder to enforce strict unsupervised orthogonal disentanglement between pathological features and environmental confounders. By mapping these decoupled representations into independent prototypical spaces, we leverage the learned spurious dictionary to perform backdoor adjustment via do-calculus, transforming complex causal interventions into efficient expectation pooling to marginalize environmental noise. Extensive experiments on multiple dermoscopy datasets demonstrate that CausalProto achieves superior diagnostic performance and consistently outperforms standard black box models, while simultaneously providing transparent and high purity visual interpretability without suffering from the traditional accuracy compromise.

Figures & tables

Figure 1: The underlying Structural Causal Model and performance overview.

Figure 2: The framework of CausalProto. (a) Training Phase: A dual-branch network disentangles causal ( $Z_{C}$ ) and spurious ( $Z_{S}$ ) features via mutual information minimization. Distinct causal ( $P_{C}$ ) and spurious ( $P_{S}$ ) prototype spaces are optimized for interpretable representation learning. (b) Inference Phase: Causal backdoor adjustment is performed by marginalizing over the learned spurious prototype dictionary ( $P_{S}$ ) via NWGM pooling to achieve de-biased interventional prediction $P(Y|do(X))$ .

Black-box ERM Models
Method	HAM10000			ISIC 2019			PAD-UFES-20
Method	Acc	BAcc	F1	Acc	BAcc	F1	Acc	BAcc	F1
ResNet-50	82.0 $\pm$ 1.2	73.5 $\pm$ 1.6	71.0 $\pm$ 1.9	78.0 $\pm$ 1.4	68.0 $\pm$ 1.7	65.0 $\pm$ 2.0	72.0 $\pm$ 2.1	60.0 $\pm$ 2.7	57.0 $\pm$ 2.9
EfficientNet	83.5 $\pm$ 1.1	75.0 $\pm$ 1.5	72.5 $\pm$ 1.8	79.5 $\pm$ 1.3	70.0 $\pm$ 1.6	67.0 $\pm$ 1.9	73.0 $\pm$ 2.0	61.5 $\pm$ 2.6	58.5 $\pm$ 2.7
Swin-T	83.0 $\pm$ 1.1	74.2 $\pm$ 1.6	72.0 $\pm$ 1.8	80.0 $\pm$ 1.2	70.8 $\pm$ 1.6	67.8 $\pm$ 1.8	72.5 $\pm$ 2.3	60.8 $\pm$ 2.9	57.8 $\pm$ 3.1
Prototype-baesd Interpretable Models

Table 1: Quantitative comparison of diagnostic performance and key ablations.

Baselines
Method	HAM10000			ISIC 2019			PAD-UFES-20
Method	NMI $\downarrow$	Purity $\uparrow$	Div $\uparrow$	NMI $\downarrow$	Purity $\uparrow$	Div $\uparrow$	NMI $\downarrow$	Purity $\uparrow$	Div $\uparrow$
ProtoPNet	-	$0.52\pm 0.03$	-	-	$0.58\pm 0.02$	-	-	$0.49\pm 0.04$	-
PIP-Net	-	$0.58\pm 0.02$	-	-	$0.62\pm 0.02$	-	-	$0.54\pm 0.03$	-
Key ablations
w/o MI	0.35	$0.55\pm 0.03$	2.00	0.31	$0.60\pm 0.03$	2.15	0.38	$0.52\pm 0.04$	2.25

Table 2: Prototype purity and disentanglement quality. NMI is the estimated

I(Z_{C};Z_{S})

upper bound (

\downarrow

). Purity measures the class-consistency of causal prototypes (

\uparrow

). Div is the spurious assignment entropy indicating confounder coverage (

\uparrow

Figure 3: Qualitative visualization of CausalProto’s de-confounded reasoning.

Explore similar work

Mar 20, 2026

DCG-Net: Dual Cross-Attention with Concept-Value Graph Reasoning for Interpretable Medical Diagnosis

Deep learning models have achieved strong performance in medical image analysis, but their internal decision processes remain difficult to interpret. Concept Bottleneck Models (CBMs) partially address this limitation by structuring predictions through human-interpretable clinical concepts. However, existing CBMs typically overlook the contextual dependencies among concepts. To address these issues, we propose an end-to-end interpretable framework DCG-Net that integrates multimodal alignment with structured concept reasoning. DCG-Net introduces a Dual Cross-Attention module that replaces cosine similarity matching with bidirectional attention between visual tokens and canonicalized textual concept-value prototypes, enabling spatially localized evidence attribution. To capture the relational structure inherent to clinical concepts, we develop a Parametric Concept Graph initialized with Positive Pointwise Mutual Information priors and refined through sparsity-controlled message passing. This formulation models inter-concept dependencies in a manner consistent with clinical domain knowledge. Experiments on white blood cell morphology and skin lesion diagnosis demonstrate that DCG-Net achieves state-of-the-art classification performance while producing clinically interpretable diagnostic explanations.

Getamesay Dagnaw, Xuefei Yin, Muhammad Hassan Maqsood +2

School of Information and Communication Technology, Griffith University, QLD, Australia

Explainable Medical Image Analysis Medical Image Classification Concept Bottleneck Models Multimodal Model Interpretability Graph Reasoning

Ask PDF

Aug 4, 2026

Open-Linguistic Concept Unified Learning for Cross-Site Interpretable Dermatology Image Diagnosis

Human-interpretable computer-aided diagnosis is crucial for clinical decision making. Concept-based models excel by providing transparent reasoning and enabling post-hoc, clinician-in-the-loop interventions. However, their rigid dataset-specific adaptation inherently restricts cross-site generalization. Applying them across diverse modalities, such as dermoscopic and clinical photographs, is challenging due to heterogeneous concept taxonomies varying in availability, granularity, and semantics across cohorts. Consequently, adapting Foundation Vision-Language Models (FVLMs) demands costly label engineering and repeated post-training. Existing intervention mechanisms remain rigidly tied to predefined concepts, lacking adaptability and hindering scalable dermatology CAD deployment. To address these bottlenecks, we propose UniCon, an open-linguistic unified concept learning framework for multimodal interpretable vision-language diagnosis. UniCon resolves these challenges through three contributions: (1) A shared semantic representation space via a unified concept prototype codebook, seamlessly coordinating heterogeneous concept systems across modalities without dataset-specific retraining. (2) Open-linguistic based multi-faceted semantic specifications to overcome sparse textual label limitations, improving boundary sensitivity in uncertain clinical contexts. (3) A robust, cross-site adjustable intervention interface powered by reliability-gated bottleneck aggregation, enabling consistent reasoning and transferable clinician corrections. Extensive experiments demonstrate that beyond securing top-tier diagnostic accuracy, UniCon successfully bridges disparate clinical taxonomies, unlocking unprecedented cross-site intervention capabilities. Code is available at https://github.com/wuchengyu123/UniCon.

Chengyu Wu, Junpeng Tan, Wanxiang Luo +3

Zhejiang University Hangzhou, China · Westlake University Hangzhou, China · Harbin Institute of Technology Shenzhen, China +1

Concept Bottleneck Models Medical Diagnosis Domain Generalization Medical VLMs Explainable Medical Image Analysis

Ask PDF

Aug 11, 2026

Uncertainty-Aware and Explainable Ensemble Deep Learning Framework for Multi-Class Skin Lesion Classification

Skin cancer diagnosis from dermoscopic images remains challenging due to high intra-class variability, inter-class similarity, class imbalance, and the limited interpretability of deep learning models. This paper proposes an uncertainty-aware and explainable deep learning framework for multi-class skin lesion classification. The framework combines a vision transformer model (MaxViT-Tiny) with CNN-based models (ConvNeXt-Tiny and EfficientNetV2-B0) through deep ensemble learning. Monte Carlo (MC) Dropout estimates predictive uncertainty and identifies unreliable predictions, while Grad-CAM++, an explainable AI (XAI) technique, provides visual explanations by highlighting lesion regions that influence model decisions. Evaluated on the HAM10000 dataset, the framework achieves 96% accuracy and 99% ROC-AUC under uncertainty-aware filtering (entropy < 1.0, confidence >= 0.7), with macro-average precision, recall, and F1-score of 94%, 95%, and 95%, respectively, and 96% weighted-average scores across all three metrics. The results demonstrate accurate, interpretable, and uncertainty-aware skin lesion classification for trustworthy computer-aided diagnosis.

Rofiqul Islam, Lilatul Ferdouse

Department of Computer Science and Physics, Wilfrid Laurier University, Waterloo, ON, Canada

Skin Lesion Classification Deep Ensembles Medical Image Classification Explainable Medical Image Analysis

Ask PDF