Vision-Language Models Encode Clinical Guidelines for Concept-Based Medical Reasoning
Authors: Mohamed Harmanani, Bining Long, Zhuoxin Guo, Paul F. R. Wilson, Amirhossein Sabour, Minh Nguyen Nhat To, Gabor Fichtinger, Purang Abolmaesumi, +1 more
Organizations: Queen’s University · Vector Institute · McMaster University · University of British Columbia
Concept Bottleneck Models (CBMs) are a prominent framework for interpretable AI that map learned visual features to a set of meaningful concepts for task-specific downstream predictions. Their sequential structure enhances transparency by connecting model predictions to the underlying concepts that support them. In medical imaging, where transparency is essential, CBMs offer an appealing foundation for explainable model design. However, discrete concept representations often overlook broader clinical context such as diagnostic guidelines and expert heuristics, reducing reliability in complex cases. We propose MedCBR, a concept-based reasoning framework that integrates clinical guidelines with vision-language and reasoning models. Labeled clinical descriptors are transformed into guideline-conformant text, and a concept-based model is trained with a multitask objective combining multimodal contrastive alignment, concept supervision, and diagnostic classification to jointly ground image features, concepts, and pathology. A reasoning model then converts these predictions into structured clinical narratives that explain the diagnosis, emulating expert reasoning based on established guidelines. MedCBR achieves superior diagnostic and concept-level performance, with AUROCs of 94.2% on ultrasound and 84.0% on mammography. Further experiments on non-medical datasets achieve 86.1% accuracy. Our framework enhances interpretability and forms an end-to-end bridge from medical image analysis to decision-making.
Figures & tables
Figure 1: MedCBR: We frame interpretable medical image analysis as reasoning over diverse sources of evidence, including model predictions and clinical guidelines.
Figure 2: Overview of the MedCBR framework. MedCBR integrates clinical knowledge, vision–language alignment, and reasoning for interpretable cancer detection. (a) A large vision–language model (LVLM) generates guideline-conformant reports from input images and coarse concept annotations. (b) A concept-based CLIP model aligns visual and textual embeddings while jointly predicting concepts and diagnostic labels. (c) A frozen large reasoning model (LRM) synthesizes guideline information, predicted concepts, and model outputs to produce structured clinical explanations.
Method
BUS-BRA
CBIS-DDSM
CUB-200-2011
AUROC
Bal. Accuracy
AUROC
Bal. Accuracy
Accuracy
CLIP RN50
87.4 ± 1.4
80.9 ± 1.7
73.3 ± 1.7
67.7 ± 1.6
60.1 ± 1.1
CLIP ViT-B/32
90.1 ± 1.5
83.1 ± 1.9
79.8 ± 0.9
72.3 ± 0.9
69.0 ± 0.4
CLIP ViT-L/14
93.5 ± 1.2
88.1 ± 1.7
82.4 ± 2.5
75.5 ± 2.4
85.7 ± 0.2
SigLIP
90.8 ± 0.8
85.0 ± 1.5
82.3 ± 1.4
74.6 ± 1.3
78.6 ± 0.6
BiomedCLIP
89.0 ± 0.5
82.1 ± 0.9
77.9 ± 0.5
71.1 ± 0.7
–
Table 1: Quantitative classification results on Ultrasound and Mammography (5-fold cross-validation), with external validation on CUB-200 (holdout set). All results are averaged across multiple runs, with standard deviation shown. ∗ indicates reported accuracy.
Variant
BUS-BRA
CBIS-DDSM
CUB-200
CLIP ViT
93.5
82.4
85.7
CLIP+CBL
91.8
81.8
67.0
+Guideline
92.0 ↑0.2
83.1 ↑1.3
72.9 ↑5.0
CLIP+MTL
93.6
83.2
82.3
+Guideline
94.2 ↑0.8
84.0 ↑0.8
86.1 ↑3.8
Table 2: Ablation on model components. Performance improves as concept and text supervision are successively introduced.
Figure 3: Quantitative results on concept detection for ultrasound and mammography. For each dataset, we focus on a subset of concepts most relevant to cancer detection.
Figure 4: Case study of MedCBR’s reasoning on medical images. Clinical concepts highlighted in green support a benign conclusion, those in red support cancer, and those in yellow are neutral. Key observations are emphasized in green (benign) and red (cancer).
Method
I,G,c^
Sens. (%)
Spec. (%)
F1 (%)
CIntS (%)
CIgS (%)
BAS. (%)
n=379 / n=721
n=20
Radiologist
–
93.9 / 90.8
75.8 / 52.3
75.1 / 70.9
–
–
–
Llama 3.2-11B
✓,✓,×
33.0 / 0.00
70.0 / 58.0
32.6 / 0.00
17.5 / 0.00
15.0 / 0.00
0.00 / 0.00
Qwen2.5VL-7B
✓,✓,×
30.0 / 42.0
0.00 / 53.0
16.6 / 40.7
100 / 100
100 / 95.0
98.0 / 90.0
MedGemma-4B
✓,✓,×
35.0 / 45.0
85.0 / 100
41.2 / 62.1
100 / 100
100 / 100
99.0 / 100
CLIP+Qwen3-8B
✓,×,✓
82.5 / 61.5
92.5 / 87.5
82.5 / 68.9
98.4 / 93.9
86.3 / 80.0
73.0 / 65.0
Table 3: Evaluation of clinical utility against general-purpose and medical VLMs on BUS-BRA (left) / CBIS-DDSM (right). ( I : image, G : guideline, c^ : vision-predicted concepts.)
Figure 5: Validation of MedCBR’s reasoning on CUB-200. Observations matching the guideline are highlighted in blue , conflicting ones are highlighted in orange .
Concept Category
Concepts
Posterior features
(1) shadowing; (2) enhancement
Other findings
(3) halo; (4) skin thickening;
(5) calcifications
Margins
(6) circumscribed; (7) indistinct;
(8) angular; (9) microlobulated;
(10) spiculated
Table 4: Clinically annotated BI-RADS concepts used for concept-based modeling.
Concept Category
Concepts
Mass Shape
(1) regular (round/oval); (2) irregular;
(3) lobulated
Mass Margins
(4) circumscribed; (5) ill-defined;
(6) spiculated; (7) obscured;
(8) microlobulated
Calcification
(9) pleomorphic; (10) amorphous;
Table 5: Ground-truth concept labels for CBIS-DDSM.
Metric
Description and Scoring Criteria
Concept Interpretation Score (CIntS)
Definition: Measures whether each predicted concept is interpreted correctly according to BI-RADS semantics. Score: CIntS=# predicted concepts# correctly interpreted concepts
Concept Integration Score (CIgS)
Definition: Evaluates whether multiple concepts are integrated coherently and consistently with BI-RADS guidelines. Scoring Levels: 1.0 Fully Correct: All concept combinations follow BI-RADS guidelines; no contradictions. 0.75 Mostly Correct: Most interpretations are reasonable, but some combinations deviate from BI-RADS (e.g., many malignant cues and a few misinterpreted concept combinations, yet the overall impression is still malignant). 0.25 Partially Correct: Only a small portion is reasonable; several contradictions or guideline violations (e.g., several malignant cues and a few benign concept combinations, with a benign overall impression). 0.0 Incorrect: Concept combinations or guideline references are entirely inconsistent or unreasonable.
BI-RADS Assignment Score (BAS)
Definition: Measures whether the final BI-RADS category aligns with guideline-based decision criteria for the given case. Scoring Levels: 1.0 Correct Assignment: Correct BI-RADS category chosen based on BI-RADS decision logic. 0.8 Near-Correct: Unreasonable category, but correct malignant/benign implication (e.g., 4C vs. 5). 0.0 Incorrect: Category contradicts BI-RADS criteria and results in an incorrect implication.
Table 6: Rubric used by the radiologist for evaluating the clinical validity of model reasoning. CIntS measures accuracy of individual concept interpretation, CIgS measures the coherence of multi-concept integration, and BAS evaluates correctness of the final BI-RADS assignment.
Figure 10: Additional examples of model reasoning on four challenging cases from BUS-BRA (top) and CBIS-DDSM (bottom). Malignant and benign-leaning clinical concepts are highlighted in red and green , and their corresponding reasoning steps are emphasized. Neutral concepts are highlighted in yellow .
Figure 11: Additional examples of model reasoning on 4 complex cases from CUB-200. Observations highlighted in orange contradict the guideline, whereas ones in blue are consistent. Key reasoning steps are emphasized in blue when they support the final conclusion and red when they do not.