Vocabulary Aligned Explanations for Computer Vision Models
Abstract
Optimizing vision models for accuracy does not force the model to learn and utilize human-understandable and domain-specific concepts and rules. We propose explaining image classifications through domain-specific concepts, visual evidence, and expert knowledge expressed as logical rules. We introduce FLUENT (Framework for Localized, Uncertainty-Aware Explanations in Normative Terminology), a post-hoc framework that ranks expert-authored rules combining jointly required, alternative, and absent concepts. To account for prediction errors, FLUENT combines predicted labels and concept states with empirically estimated likelihoods and concept priors. It scores each rule by calculating support over satisfying assignments to the concepts mentioned in that rule, then normalizes these scores to express relative confidence among candidate explanations. The selected rule is illustrated through occlusion-based highlights for positive concept mentions in the query image and localized examples in retrieved reference images for negative mentions. These illustrations are intended to show end users unfamiliar with the domain what the concepts named in an explanation look like. FLUENT combines domain knowledge and model accuracy, and delivers visual concept highlights and explicit rule confidence to explain an image in the vocabulary of an application domain.
Figures & tables
| Backbone | Label | Concept | Rule | RuleLabel | Avg |
|---|---|---|---|---|---|
| Acc | Acc | Acc | Acc | ||
| Synthetic | |||||
| RN50 (TV) | 39.0 | 79.9 | 69.7 | 85.3 | 0.634 |
| CNXv2-L | 99.3 | 93.3 | 93.7 | 99.7 | 0.925 |
| ViT-B/16 (DINO) | 99.7 | 92.8 | 96.7 | 99.7 | 0.935 |
| Swin-B (IN-22k) | 99.0 | 95.2 | 94.7 | 99.0 | 0.949 |
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
| Class | Expert Rule |
|---|---|
| Kallow | |
| Kallow | |
| Vumble | |
| Split | Img./Class | Total | Used by |
|---|---|---|---|
| Label Train | 5 000 | 15 000 | {\color[rgb]{0,0,0}\phi_{\mathrm{cls}}} training |
| Label Test | 1 000 | 3 000 | {\color[rgb]{0,0,0}\phi_{\mathrm{cls}}} evaluation |
| Explanation Test | 100 | 300 | Fluent benchmark |
| Img./Concept | Total | ||
| Concept Train | 500 pos./neg. pairs | 16 000 | {\color[rgb]{0,0,0}\phi_{c_{i}}} training |
| Concept Test | 100 pos./neg. pairs | 3 200 | {\color[rgb]{0,0,0}\phi_{c_{i}}} evaluation |
| Class | Expert Rule |
|---|---|
| Bumble Bee | |
| Green Metallic | |
| Sweat Bee | |
| Hairy Leg Bee |
| Split | Img./Class | Total | Used by |
|---|---|---|---|
| Label Train | 24 271–24 301 | 72 873 | {\color[rgb]{0,0,0}\phi_{\mathrm{cls}}} training |
| Label Test | 2 698–2 729 | 8 125 | {\color[rgb]{0,0,0}\phi_{\mathrm{cls}}} evaluation |
| Explanation Test | 200 | 600 | Fluent benchmark |
| Img./Concept | Total | ||
| Concept Train | 929–1 160 | 4 119 | {\color[rgb]{0,0,0}\phi_{c_{i}}} training |
| Concept Test | 118–184 | 577 | {\color[rgb]{0,0,0}\phi_{c_{i}}} evaluation |
| Paper label | Model identifier | Probe layer |
|---|---|---|
| CNXv2-L | convnextv2_large.fcmae_ft_in22k_in1k | head.global_pool |
| RN152 (A1h) | resnet152.a1h_in1k | global_pool |
| ViT-B/16 (DINO) | vit_base_patch16_224.dino | fc_norm |
| SwinV2-S | swinv2_cr_small_ns_224.sw_in1k | head.global_pool |
| ViT-B/16 (SigLIP 2) | vit_base_patch16_siglip_gap_224.v2_webli | fc_norm |
| BEiT-L/16 | beit_large_patch16_224.in22k_ft_in22k_in1k | fc_norm |
| Backbone | LabelAcc | ConceptAcc | RuleAcc | RuleLabelAcc | Avg Conf |
|---|---|---|---|---|---|
| CNXv2-L | 99.3% | 93.3% | 93.7% | 99.7% | 0.925 |
| CNXv2-L (FCMAE only) | 94.7% | 56.9% | 54.0% | 94.0% | 0.687 |
| RN152 | 40.3% | 73.6% | 47.7% | 67.3% | 0.540 |
| RN50 (TV) | 39.0% | 79.9% | 69.7% | 85.3% | 0.634 |
| RN50 (A1h) | 51.0% | 87.0% | 67.7% | 77.7% | 0.698 |
| ViT-B/16 (DINO) | 99.7% | 92.8% | 96.7% | 99.7% | 0.935 |
| Backbone | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| CNXv2-L | 85.7 | 89.0 | 89.0 | 87.0 | 89.7 | 89.0 | 90.0 | 92.3 | 93.7 |
| CNXv2-L (FCMAE only) | 55.7 | 46.3 | 51.7 | 46.0 | 48.7 | 47.0 | 49.7 | 53.0 | 54.0 |
| RN152 | 31.7 | 33.0 | 35.0 | 41.7 | 45.3 | 46.0 | 41.7 | 43.3 | 47.7 |
| RN50 (TV) | 45.7 | 59.0 | 62.3 | 42.3 | 52.3 | 73.0 | 35.3 | 41.7 | 69.7 |
| RN50 (A1h) | 53.7 | 57.7 | 63.7 | 55.3 | 61.7 | 69.7 | 53.7 | 59.0 | 67.7 |
| Backbone | LabelAcc | ConceptAcc | RuleAcc | RuleLabelAcc | Avg Conf | Switch% | |
|---|---|---|---|---|---|---|---|
| CNXv2-L | 100 | 99.3% | 89.8% | 85.7% | 98.7% | 0.903 | 0.7 |
| 200 | 99.3% | 91.7% | 89.0% | 99.7% | 0.895 | 0.3 | |
| 500 | 99.3% | 93.9% | 89.0% | 99.3% | 0.916 | 0.0 | |
| CNXv2-L (FCMAE only) | 100 | 95.7% | 55.2% | 55.7% | 95.3% | 0.664 | 1.0 |
| 200 | 95.7% | 58.5% | 46.3% | 94.3% | 0.681 | 1.3 | |
| 500 | 95.7% | 57.8% | 51.7% | 95.7% | 0.668 | 0.3 |
| Backbone | LabelAcc | ConceptAcc | RuleAcc | RuleLabelAcc | Avg Conf | Switch% | |
|---|---|---|---|---|---|---|---|
| CNXv2-L | 100 | 99.3% | 89.8% | 85.7% | 98.7% | 0.902 | 0.7 |
| 200 | 99.3% | 91.7% | 89.0% | 99.7% | 0.895 | 0.3 | |
| 500 | 99.3% | 93.9% | 89.0% | 99.3% | 0.912 | 0.0 | |
| CNXv2-L (FCMAE only) | 100 | 95.7% | 55.2% | 56.3% | 95.3% | 0.672 | 1.0 |
| 200 | 95.7% | 58.5% | 48.0% | 94.3% | 0.686 | 1.3 | |
| 500 | 95.7% | 57.8% | 52.7% | 95.7% | 0.672 | 0.3 |
| Backbone | LabelAcc | ConceptAcc | RuleAcc | RuleLabelAcc | Avg Conf | Switch% | |
|---|---|---|---|---|---|---|---|
| CNXv2-L | 100 | 99.7% | 90.3% | 87.0% | 99.7% | 0.901 | 0.0 |
| 200 | 99.7% | 91.1% | 89.7% | 99.7% | 0.902 | 0.0 | |
| 500 | 99.7% | 94.1% | 89.0% | 99.3% | 0.917 | 0.3 | |
| CNXv2-L (FCMAE only) | 100 | 95.0% | 55.1% | 46.0% | 93.3% | 0.613 | 1.7 |
| 200 | 95.0% | 57.1% | 48.7% | 95.0% | 0.723 | 0.0 | |
| 500 | 95.0% | 57.3% | 47.0% | 94.3% | 0.683 | 0.7 |
| Backbone | LabelAcc | ConceptAcc | RuleAcc | RuleLabelAcc | Avg Conf | Switch% | |
|---|---|---|---|---|---|---|---|
| CNXv2-L | 100 | 99.7% | 90.3% | 87.0% | 99.7% | 0.898 | 0.0 |
| 200 | 99.7% | 91.1% | 90.0% | 99.7% | 0.899 | 0.0 | |
| 500 | 99.7% | 94.1% | 89.0% | 99.3% | 0.915 | 0.3 | |
| CNXv2-L (FCMAE only) | 100 | 95.0% | 55.1% | 45.7% | 93.3% | 0.618 | 1.7 |
| 200 | 95.0% | 57.1% | 49.0% | 95.0% | 0.731 | 0.0 | |
| 500 | 95.0% | 57.3% | 46.3% | 94.3% | 0.690 | 0.7 |
| Backbone | LabelAcc | ConceptAcc | RuleAcc | RuleLabelAcc | Avg Conf | Switch% | |
|---|---|---|---|---|---|---|---|
| CNXv2-L | 100 | 99.3% | 90.0% | 90.0% | 99.3% | 0.908 | 0.0 |
| 200 | 99.3% | 91.6% | 92.3% | 99.7% | 0.904 | 0.3 | |
| 500 | 99.3% | 93.3% | 93.7% | 99.7% | 0.925 | 0.3 | |
| CNXv2-L (FCMAE only) | 100 | 94.7% | 54.9% | 49.7% | 94.0% | 0.666 | 1.3 |
| 200 | 94.7% | 56.8% | 53.0% | 94.0% | 0.752 | 0.7 | |
| 500 | 94.7% | 56.9% | 54.0% | 94.0% | 0.687 | 0.7 |
| Backbone | LabelAcc | ConceptAcc | RuleAcc | RuleLabelAcc | Avg Conf | Switch% | |
|---|---|---|---|---|---|---|---|
| CNXv2-L | 100 | 99.3% | 90.0% | 90.0% | 99.3% | 0.907 | 0.0 |
| 200 | 99.3% | 91.6% | 92.3% | 99.7% | 0.902 | 0.3 | |
| 500 | 99.3% | 93.3% | 93.7% | 99.7% | 0.922 | 0.3 | |
| CNXv2-L (FCMAE only) | 100 | 94.7% | 54.9% | 48.3% | 94.0% | 0.657 | 1.3 |
| 200 | 94.7% | 56.8% | 53.0% | 94.0% | 0.754 | 0.7 | |
| 500 | 94.7% | 56.9% | 53.0% | 94.0% | 0.684 | 0.7 |
| RuleAcc, | Maximum absolute change | ||||
|---|---|---|---|---|---|
| Backbone | RuleAcc (pp) | RuleLabelAcc (pp) | Avg Conf | ||
| CNXv2-L | 93.7% | 93.7% | 0.3 | 0.0 | 0.003 |
| CNXv2-L (FCMAE only) | 54.0% | 53.0% | 1.7 | 0.0 | 0.009 |
| RN152 | 47.7% | 47.3% | 1.3 | 2.0 | 0.004 |
| RN50 (TV) | 69.7% | 62.7% | 7.0 | 7.7 | 0.005 |
| RN50 (A1h) | 67.7% | 68.0% | 0.7 | 1.3 | 0.007 |
| Backbone | Alorn | Floom | Frib | Guffit | Hebda | Heffix | Krad | Opple | |
|---|---|---|---|---|---|---|---|---|---|
| CNXv2-L | 100 | 86.0 | 91.5 | 76.0 | 94.0 | 98.5 | 95.5 | 89.5 | 88.5 |
| 200 | 90.0 | 93.0 | 81.0 | 92.0 | 98.0 | 94.0 | 92.0 | 90.5 | |
| 500 | 93.5 | 95.5 | 85.0 | 94.0 | 99.0 | 96.5 | 92.5 | 89.5 | |
| CNXv2-L (FCMAE only) | 100 | 58.5 | 56.0 | 50.0 | 62.0 | 61.5 | 50.5 | 58.0 | 55.0 |
| 200 | 72.0 | 54.0 | 57.0 | 72.0 | 62.0 | 51.5 | 64.0 | 52.0 | |
| 500 | 61.5 | 48.5 | 51.5 | 80.5 | 65.5 | 62.0 | 52.0 | 55.0 |
| Backbone | Smeek | Sorbax | Voop | Vornex | Wexil | Yintar | Zandool | Zorf | |
|---|---|---|---|---|---|---|---|---|---|
| CNXv2-L | 100 | 83.5 | 83.0 | 94.5 | 97.0 | 90.0 | 92.0 | 95.0 | 82.5 |
| 200 | 88.0 | 86.0 | 95.0 | 91.5 | 94.0 | 96.5 | 96.5 | 88.5 | |
| 500 | 87.5 | 88.0 | 98.0 | 97.5 | 98.5 | 98.0 | 98.0 | 92.0 | |
| CNXv2-L (FCMAE only) | 100 | 61.5 | 57.5 | 51.0 | 52.0 | 53.5 | 52.5 | 54.0 | 49.0 |
| 200 | 64.5 | 57.5 | 48.5 | 54.0 | 64.0 | 50.5 | 62.0 | 50.5 | |
| 500 | 75.5 | 55.0 | 54.0 | 51.5 | 51.0 | 53.0 | 55.0 | 53.5 |
| Backbone | Alorn | Floom | Frib | Guffit | Hebda | Heffix | Krad | Opple | |
|---|---|---|---|---|---|---|---|---|---|
| CNXv2-L | 100 | 85.0 | 90.5 | 79.5 | 95.5 | 98.5 | 96.0 | 89.5 | 85.5 |
| 200 | 88.0 | 85.0 | 79.5 | 92.0 | 99.0 | 96.5 | 89.5 | 89.5 | |
| 500 | 90.5 | 94.5 | 85.0 | 95.0 | 98.5 | 95.5 | 95.5 | 93.0 | |
| CNXv2-L (FCMAE only) | 100 | 58.5 | 52.0 | 52.0 | 61.0 | 59.5 | 52.5 | 53.5 | 58.0 |
| 200 | 68.0 | 52.0 | 56.0 | 72.5 | 63.0 | 54.0 | 59.0 | 52.0 | |
| 500 | 58.5 | 49.0 | 53.0 | 73.0 | 63.5 | 62.0 | 52.0 | 55.0 |
| Backbone | Smeek | Sorbax | Voop | Vornex | Wexil | Yintar | Zandool | Zorf | |
|---|---|---|---|---|---|---|---|---|---|
| CNXv2-L | 100 | 85.0 | 83.5 | 94.5 | 97.5 | 91.5 | 92.5 | 96.0 | 84.0 |
| 200 | 87.5 | 82.5 | 94.5 | 94.0 | 94.5 | 97.5 | 95.5 | 92.0 | |
| 500 | 89.5 | 90.0 | 99.0 | 95.5 | 97.5 | 96.5 | 98.0 | 92.5 | |
| CNXv2-L (FCMAE only) | 100 | 67.0 | 57.0 | 51.5 | 51.0 | 54.0 | 51.0 | 53.5 | 49.0 |
| 200 | 59.0 | 56.5 | 48.5 | 57.5 | 57.0 | 51.5 | 52.5 | 55.0 | |
| 500 | 66.0 | 58.0 | 55.0 | 50.5 | 55.0 | 52.0 | 61.5 | 53.5 |
| Backbone | Alorn | Floom | Frib | Guffit | Hebda | Heffix | Krad | Opple | |
|---|---|---|---|---|---|---|---|---|---|
| CNXv2-L | 100 | 84.5 | 89.0 | 81.5 | 95.5 | 95.5 | 96.5 | 90.0 | 86.5 |
| 200 | 91.5 | 92.0 | 82.0 | 92.5 | 99.0 | 93.0 | 91.0 | 93.0 | |
| 500 | 93.5 | 89.0 | 83.0 | 95.5 | 98.5 | 94.5 | 95.5 | 88.0 | |
| CNXv2-L (FCMAE only) | 100 | 57.5 | 54.0 | 51.0 | 63.0 | 57.0 | 50.0 | 52.0 | 59.0 |
| 200 | 64.5 | 58.0 | 55.5 | 72.5 | 61.0 | 50.5 | 63.5 | 51.5 | |
| 500 | 62.5 | 50.5 | 53.5 | 77.5 | 61.0 | 62.0 | 48.0 | 56.0 |
| Backbone | Smeek | Sorbax | Voop | Vornex | Wexil | Yintar | Zandool | Zorf | |
|---|---|---|---|---|---|---|---|---|---|
| CNXv2-L | 100 | 84.0 | 85.0 | 94.5 | 96.5 | 93.0 | 91.0 | 93.5 | 83.5 |
| 200 | 87.0 | 85.0 | 93.5 | 91.5 | 91.0 | 97.0 | 94.5 | 91.5 | |
| 500 | 89.5 | 85.0 | 97.5 | 96.5 | 98.0 | 97.0 | 97.5 | 94.5 | |
| CNXv2-L (FCMAE only) | 100 | 63.0 | 57.0 | 54.0 | 50.5 | 54.5 | 51.5 | 53.5 | 50.5 |
| 200 | 58.5 | 56.0 | 49.0 | 56.0 | 60.0 | 51.0 | 53.0 | 48.0 | |
| 500 | 63.5 | 56.0 | 56.0 | 52.0 | 51.0 | 53.0 | 56.0 | 52.0 |
| Backbone | Validation | Test | Validation | Test | Validation | Test |
|---|---|---|---|---|---|---|
| CNXv2-L | 100% | 99.0% | 100% | 99.4% | 100% | 99.5% |
| CNXv2-L (FCMAE only) | 89% | 93.0% | 95% | 95.4% | 97% | 96.6% |
| RN152 | 49% | 43.9% | 42% | 43.5% | 40% | 41.5% |
| RN50 (TV) | 43% | 37.7% | 33% | 34.0% | 39% | 37.0% |
| RN50 (A1h) | 60% | 49.3% | 52% | 49.6% | 53% | 49.0% |
| Backbone | LabelAcc | ConceptAcc | RuleAcc | RuleLabelAcc | Avg Conf |
|---|---|---|---|---|---|
| CNXv2-L | 99.7% | 96.4% | 85.8% | 85.8% | 0.967 |
| CNXv2-L (FCMAE only) | 63.3% | 68.3% | 43.8% | 43.8% | 0.633 |
| RN152 | 84.7% | 87.0% | 62.8% | 62.8% | 0.803 |
| RN50 (TV) | 91.2% | 85.5% | 87.8% | 87.8% | 0.770 |
| RN50 (A1h) | 87.0% | 87.0% | 66.7% | 66.7% | 0.792 |
| ViT-B/16 (DINO) | 97.7% | 96.2% | 84.3% | 84.3% | 0.927 |
| Backbone | LabelAcc | ConceptAcc | RuleAcc | RuleLabelAcc | Avg Conf |
|---|---|---|---|---|---|
| CNXv2-L | 99.7% | 96.4% | 85.8% | 85.8% | 0.967 |
| CNXv2-L (FCMAE only) | 63.3% | 68.3% | 43.8% | 43.8% | 0.633 |
| RN152 | 84.7% | 87.0% | 62.8% | 62.8% | 0.803 |
| RN50 (TV) | 91.2% | 85.5% | 87.8% | 87.8% | 0.770 |
| RN50 (A1h) | 87.0% | 87.0% | 66.7% | 66.7% | 0.792 |
| ViT-B/16 (DINO) | 97.7% | 96.2% | 84.3% | 84.3% | 0.927 |
| Backbone | Correct labels | Correct rules | Correct rule labels | Switches | Switch% |
|---|---|---|---|---|---|
| CNXv2-L | 598 | 515 | 515 | 83 | 13.8 |
| CNXv2-L (FCMAE only) | 380 | 263 | 263 | 219 | 36.5 |
| RN152 | 508 | 377 | 377 | 163 | 27.2 |
| RN50 (TV) | 547 | 527 | 527 | 43 | 7.2 |
| RN50 (A1h) | 522 | 400 | 400 | 151 | 25.2 |
| ViT-B/16 (DINO) | 586 | 506 | 506 | 80 | 13.3 |
| Concept accuracy | Label-head accuracy | ||||||
|---|---|---|---|---|---|---|---|
| Backbone | Fuzzy | Dense | Pollen | Green | Mean | Validation | Test |
| CNXv2-L | 97.8 | 96.2 | 93.2 | 98.6 | 96.4 | 95 | 95.4 |
| CNXv2-L (FCMAE only) | 80.6 | 73.4 | 69.5 | 49.6 | 68.3 | 59 | 60.1 |
| RN152 | 88.8 | 84.8 | 81.4 | 92.9 | 87.0 | 75 | 73.1 |
| RN50 (TV) | 87.3 | 78.3 | 84.7 | 91.5 | 85.5 | 80 | 79.5 |
| RN50 (A1h) | 93.3 | 80.4 | 81.4 | 92.9 | 87.0 | 79 | 77.7 |
Explainable Image Classification
Explainable image classification develops ways to show which pixels, regions, concepts, or model features support an image classifier’s predictions.
See all 106 papers