Deep learning models have achieved strong performance in medical image analysis, but their internal decision processes remain difficult to interpret. Concept Bottleneck Models (CBMs) partially address this limitation by structuring predictions through human-interpretable clinical concepts. However, existing CBMs typically overlook the contextual dependencies among concepts. To address these issues, we propose an end-to-end interpretable framework DCG-Net that integrates multimodal alignment with structured concept reasoning. DCG-Net introduces a Dual Cross-Attention module that replaces cosine similarity matching with bidirectional attention between visual tokens and canonicalized textual concept-value prototypes, enabling spatially localized evidence attribution. To capture the relational structure inherent to clinical concepts, we develop a Parametric Concept Graph initialized with Positive Pointwise Mutual Information priors and refined through sparsity-controlled message passing. This formulation models inter-concept dependencies in a manner consistent with clinical domain knowledge. Experiments on white blood cell morphology and skin lesion diagnosis demonstrate that DCG-Net achieves state-of-the-art classification performance while producing clinically interpretable diagnostic explanations.
Figures & tables
Fig. 1: Overview of DCG-Net. The CDE generates canonical concept-value prototypes ( TM ). A vision transformer encodes images into patch-level visual tokens ( V ). The DCA module aligns visual and textual representations via bidirectional attention at the concept-value level, generating local evidence and global relevance ( αM ), which are gated to produce the initial node features H(0) . The PCG models concept-value dependencies using PPMI-initialized edges refined through learnable message passing, resulting in the refined state H(L) that forms the interpretable bottleneck for diagnosis.
Dataset
Method
Diagnosis Metrics
Concept Metrics
Acc (%)
F1 (%)
Acc (%)
F1 (%)
Fitzpatrick17k
CBM [ 15 ] (ICML 2020)
78.37±0.70
56.79±0.45
78.42±0.50
54.46±0.32
CEM [ 6 ] (NeurIPS 2022)
77.07±1.44
58.02±0.61
78.96±0.50
54.95±0.83
ProbCBM [ 13 ] (ICML 2023)
72.13±1.00
54.03±0.15
88.49±0.97
57.01±1.52
evi-CEM [ 8 ] (MICCA 2024)
76.50±2.52
58.55±2.27
90.33±0.37
64.99±0.73
Mica [ 4 ] (AAAI 2024)
75.63±1.07
75.43±1.24
91.7±−−
63.8±−−
TABLE I: Performance comparison on PBC and SkinCon classification tasks. Bold indicates the best results, and underlined text denotes the second-best. Some compared results are directly reported from the cited paper [ 9 ] . For DCG-Net, concept metrics are computed on the pre-graph concept-value logits, which are directly supervised during training.
Fig. 2: Qualitative explanation of DCG-Net. (A) Input image with predicted diagnosis, concept-value probabilities, and ground-truth labels. (B) Top concept contributions ( α×ppresent ), showing which clinical findings most strongly support the model’s diagnostic decision. (C) PCG-based relational reasoning, showing each top concept and its associated concept-value nodes, reflecting learned dependencies among concept-value pairs.
Fitzpatrick17k
Performance Metrics
Diagnosis (Acc/F1)
Concept (Acc/F1)
Full (ours)
83.44 / 72.13
92.18 / 71.05
w/o CDE
82.51 / 71.36
91.95 / 68.87
No PCG
81.66 / 71.63
92.15 / 70.40
TABLE II: Ablation for methodological justification.
Concept Bottleneck Models (CBMs) are a prominent framework for interpretable AI that map learned visual features to a set of meaningful concepts for task-specific downstream predictions. Their sequential structure enhances transparency by connecting model predictions to the underlying concepts that support them. In medical imaging, where transparency is essential, CBMs offer an appealing foundation for explainable model design. However, discrete concept representations often overlook broader clinical context such as diagnostic guidelines and expert heuristics, reducing reliability in complex cases. We propose MedCBR, a concept-based reasoning framework that integrates clinical guidelines with vision-language and reasoning models. Labeled clinical descriptors are transformed into guideline-conformant text, and a concept-based model is trained with a multitask objective combining multimodal contrastive alignment, concept supervision, and diagnostic classification to jointly ground image features, concepts, and pathology. A reasoning model then converts these predictions into structured clinical narratives that explain the diagnosis, emulating expert reasoning based on established guidelines. MedCBR achieves superior diagnostic and concept-level performance, with AUROCs of 94.2% on ultrasound and 84.0% on mammography. Further experiments on non-medical datasets achieve 86.1% accuracy. Our framework enhances interpretability and forms an end-to-end bridge from medical image analysis to decision-making.
Mohamed Harmanani, Bining Long, Zhuoxin Guo +6
Queen’s University · Vector Institute · McMaster University +1
Medical imaging models often operate as black boxes, limiting interpretability and systematic debugging. We introduce an easy-to-use, plug-and-play framework for concept-based interpretation and model refinement. By aligning a single-modality encoder to BioMedCLIP, we construct a Concept Bottleneck Model (CBM) that enables concept-level interventions. These interventions allow us to isolate causal versus spuriously correlated concepts, validate insights with domain experts, and generate counterfactual samples for targeted fine-tuning. We evaluate our framework on a Mayo Clinic ultrasound dataset and the CheXpert 5x200 chest X-ray dataset. Results demonstrate that concept intervention enables reliable model diagnosis while maintaining, and occasionally improving predictive performance via guided fine-tuning. Our findings highlight the practical value of this framework for controlled, interpretable refinement of clinical deep learning models.
Samrajya Thapa, Daniel J. Quest, Timothy L. Kline +3
Iowa State University, Ames IA, USA · Mayo Clinic, Rochester MN, USA
Concept Bottleneck Models (CBMs) provide an intrinsically interpretable alternative to post-hoc explanations. However, existing CBMs often rely on predefined concept vocabularies or supervised annotations, lack explicit concept grounding, and summarize each concept with a single image-level score -- discarding spatial recurrence and inter-concept dependencies. We propose a Graph-based Concept Bottleneck Model (G-CBM), an intrinsically interpretable framework that performs unsupervised concept discovery via Non-negative Matrix Factorization (NMF) and represents the discovered concepts as nodes in a per-image concept-graph representation. G-CBM matches region-level features to these concept nodes -- providing concept grounding and capturing concept recurrence across the image -- and applies a tunable concept filtering thresholdτ to suppress weak region-level features. A Graph Attention Network (GAT) then performs concept-level reasoning by modeling nonlinear dependencies across nodes. Across ImageNet, HAM10000, PH2, and Derm7pt, G-CBM achieves an average relative AUC improvement of 3.7% over a ResNet-50 baseline. Concept filtering frequently improves predictive performance while inducing selective concept use, achieving peak AUC of 0.96 on PH2 with only 2 of 10 concepts and 0.92 on HAM10000 with 3.8 of 9 concepts. On dermoscopy benchmarks, G-CBM is competitive with supervised approaches requiring external annotations. Deletion/insertion analyses with random ablation controls show that the learned concept ranking faithfully reflects model predictions.
Md Mohasin Hossain, Anar Amirli, Robert Leist +2
German Research Center for Artificial Intelligence · Saarland University, Saarbrücken, Germany · German Research Center for Artificial Intelligence (DFKI), Saarbrücken, Germany +2