Unsupervised Causal Prototypical Networks for De-biased Interpretable Dermoscopy Diagnosis

Junhao Jia; Yueyi Wu; Huangwei Chen; Haodong Jing; Haishuai Wang; Jiajun Bu; Lei Wu

eess.IVFeb 27, 2026

Unsupervised Causal Prototypical Networks for De-biased Interpretable Dermoscopy Diagnosis

Authors: Junhao Jia, Yueyi Wu, Huangwei Chen, Haodong Jing, Haishuai Wang, Jiajun Bu, Lei Wu

Organizations: Zhejiang Key Laboratory of Accessible Perception and Intelligent Systems, College of Computer Science and Technology, Zhejiang University · Hangzhou Dianzi University · Xi’an Jiaotong University

Abstract

Despite the success of deep learning in dermoscopy image analysis, its inherent black-box nature hinders clinical trust, motivating the use of prototypical networks for case-based visual transparency. However, inevitable selection bias in clinical data often drives these models toward shortcut learning, where environmental confounders are erroneously encoded as predictive prototypes, generating spurious visual evidence that misleads medical decision-making. To mitigate these confounding effects, we propose CausalProto, an Unsupervised Causal Prototypical Network that fundamentally purifies the visual evidence chain. Framed within a Structural Causal Model, we employ an Information Bottleneck-constrained encoder to enforce strict unsupervised orthogonal disentanglement between pathological features and environmental confounders. By mapping these decoupled representations into independent prototypical spaces, we leverage the learned spurious dictionary to perform backdoor adjustment via do-calculus, transforming complex causal interventions into efficient expectation pooling to marginalize environmental noise. Extensive experiments on multiple dermoscopy datasets demonstrate that CausalProto achieves superior diagnostic performance and consistently outperforms standard black box models, while simultaneously providing transparent and high purity visual interpretability without suffering from the traditional accuracy compromise.

Figures & tables

Figure 1: The underlying Structural Causal Model and performance overview.

Figure 2: The framework of CausalProto. (a) Training Phase: A dual-branch network disentangles causal ( $Z_{C}$ ) and spurious ( $Z_{S}$ ) features via mutual information minimization. Distinct causal ( $P_{C}$ ) and spurious ( $P_{S}$ ) prototype spaces are optimized for interpretable representation learning. (b) Inference Phase: Causal backdoor adjustment is performed by marginalizing over the learned spurious prototype dictionary ( $P_{S}$ ) via NWGM pooling to achieve de-biased interventional prediction $P(Y|do(X))$ .

Black-box ERM Models
Method	HAM10000			ISIC 2019			PAD-UFES-20
Method	Acc	BAcc	F1	Acc	BAcc	F1	Acc	BAcc	F1
ResNet-50	82.0 $\pm$ 1.2	73.5 $\pm$ 1.6	71.0 $\pm$ 1.9	78.0 $\pm$ 1.4	68.0 $\pm$ 1.7	65.0 $\pm$ 2.0	72.0 $\pm$ 2.1	60.0 $\pm$ 2.7	57.0 $\pm$ 2.9
EfficientNet	83.5 $\pm$ 1.1	75.0 $\pm$ 1.5	72.5 $\pm$ 1.8	79.5 $\pm$ 1.3	70.0 $\pm$ 1.6	67.0 $\pm$ 1.9	73.0 $\pm$ 2.0	61.5 $\pm$ 2.6	58.5 $\pm$ 2.7
Swin-T	83.0 $\pm$ 1.1	74.2 $\pm$ 1.6	72.0 $\pm$ 1.8	80.0 $\pm$ 1.2	70.8 $\pm$ 1.6	67.8 $\pm$ 1.8	72.5 $\pm$ 2.3	60.8 $\pm$ 2.9	57.8 $\pm$ 3.1
Prototype-baesd Interpretable Models

Table 1: Quantitative comparison of diagnostic performance and key ablations.

Baselines
Method	HAM10000			ISIC 2019			PAD-UFES-20
Method	NMI $\downarrow$	Purity $\uparrow$	Div $\uparrow$	NMI $\downarrow$	Purity $\uparrow$	Div $\uparrow$	NMI $\downarrow$	Purity $\uparrow$	Div $\uparrow$
ProtoPNet	-	$0.52\pm 0.03$	-	-	$0.58\pm 0.02$	-	-	$0.49\pm 0.04$	-
PIP-Net	-	$0.58\pm 0.02$	-	-	$0.62\pm 0.02$	-	-	$0.54\pm 0.03$	-
Key ablations
w/o MI	0.35	$0.55\pm 0.03$	2.00	0.31	$0.60\pm 0.03$	2.15	0.38	$0.52\pm 0.04$	2.25

Table 2: Prototype purity and disentanglement quality. NMI is the estimated

I(Z_{C};Z_{S})

upper bound (

\downarrow

). Purity measures the class-consistency of causal prototypes (

\uparrow

). Div is the spurious assignment entropy indicating confounder coverage (

\uparrow

Figure 3: Qualitative visualization of CausalProto’s de-confounded reasoning.

Explore similar work

DCG-Net: Dual Cross-Attention with Concept-Value Graph Reasoning for Interpretable Medical Diagnosis
Mar 20, 2026Getamesay Dagnaw, Xuefei Yin, Muhammad Hassan Maqsood +2Explainable Medical Image Analysis Medical Image Classification
School of Information and Communication Technology, Griffith University, QLD, Australia
Open-Linguistic Concept Unified Learning for Cross-Site Interpretable Dermatology Image Diagnosis
Aug 4, 2026Chengyu Wu, Junpeng Tan, Wanxiang Luo +3Concept Bottleneck Models Medical Diagnosis
Zhejiang University Hangzhou, China · Westlake University Hangzhou, China · Harbin Institute of Technology Shenzhen, China +1
Uncertainty-Aware and Explainable Ensemble Deep Learning Framework for Multi-Class Skin Lesion Classification
Aug 11, 2026Rofiqul Islam, Lilatul FerdouseSkin Lesion Classification Deep Ensembles
Department of Computer Science and Physics, Wilfrid Laurier University, Waterloo, ON, Canada

Mar 20, 2026J/K move · Enter open · S save

DCG-Net: Dual Cross-Attention with Concept-Value Graph Reasoning for Interpretable Medical Diagnosis

Getamesay Dagnaw, Xuefei Yin, Muhammad Hassan Maqsood, Yanming Zhu+1

School of Information and Communication Technology, Griffith University, QLD, Australia

Explainable Medical Image Analysis Medical Image Classification Concept Bottleneck Models Multimodal Model Interpretability Graph Reasoning

Open paper AskPDF arXiv

Deep learning models have achieved strong performance in medical image analysis, but their internal decision processes remain difficult to interpret. Concept Bottleneck Models (CBMs) partially address this limitation by structuring predictions through human-interpretable clinical concepts. However, existing CBMs typically overlook the contextual dependencies among concepts. To address these issues, we propose an end-to-end interpretable framework DCG-Net that integrates multimodal alignment with structured concept reasoning. DCG-Net introduces a Dual Cross-Attention module that replaces cosine similarity matching with bidirectional attention between visual tokens and canonicalized textual concept-value prototypes, enabling spatially localized evidence attribution. To capture the relational structure inherent to clinical concepts, we develop a Parametric Concept Graph initialized with Positive Pointwise Mutual Information priors and refined through sparsity-controlled message passing. This formulation models inter-concept dependencies in a manner consistent with clinical domain knowledge. Experiments on white blood cell morphology and skin lesion diagnosis demonstrate that DCG-Net achieves state-of-the-art classification performance while producing clinically interpretable diagnostic explanations.

Abstract

Explore similar work

DCG-Net: Dual Cross-Attention with Concept-Value Graph Reasoning for Interpretable Medical Diagnosis

Open-Linguistic Concept Unified Learning for Cross-Site Interpretable Dermatology Image Diagnosis

Uncertainty-Aware and Explainable Ensemble Deep Learning Framework for Multi-Class Skin Lesion Classification