Organizations: Department of Computer Science, University of Manchester, United Kingdom. · School of Electrical Engineering and Computer Science, the University of Queensland, Australia. · Department of Computer Science, University of California at Merced, United State.
Concept Bottleneck Models (CBMs) introduce interpretability to black-box deep learning models by predicting labels through human-understandable concepts. However, unlike humans, who identify objects at different levels of abstraction using both general and specific features, existing CBMs operate at a single semantic level in both concept and label space. We propose HIL-CBM, a Hierarchical Interpretable Label-Free Concept Bottleneck Model that extends CBMs into a hierarchical framework to enhance interpretability by more closely mirroring the human cognitive process. HIL-CBM enables classification and explanation across multiple semantic levels without requiring relational concept annotations. HIL-CBM aligns the abstraction level of concept-based explanations with that of model predictions, progressing from abstract to concrete. This is achieved by (i) introducing a gradient-based visual consistency loss that encourages abstraction layers to focus on similar spatial regions, and (ii) training dual classification heads, each operating on feature concepts at different abstraction levels. Experiments on benchmark datasets demonstrate that HIL-CBM outperforms state-of-the-art sparse CBMs in classification accuracy. Human evaluations further show that HIL-CBM provides more interpretable and accurate explanations, while maintaining a hierarchical and label-free approach to feature concepts.
Figures & tables
Fig. 1: Our proposed HIL-CBM extends the CBM architecture into a hierarchical framework. The upper concept and label spaces capture general semantic features and broader class categories, while the lower spaces focus on more specific features and classes.
Fig. 2: Overview of our proposed model, HIL-CBM. A pre-trained backbone processes input images to produce image embeddings. Two concept layers, each focusing on a different level of abstraction, project the feature maps into interpretable hierarchical concept spaces. These layers are trained using CLIP-Dissect similarity loss and visual consistency loss. Finally, two hierarchical classifiers predict classes at different levels of abstraction, using cross-entropy loss and semantic consistency loss based on the learned hierarchical concept features.
Model
Backbone
Sparse
Lower Level ∥ Higher Level
CIFAR100
CUB-200
Places365
ImageNet
Standard
ResNet
Yes
58.34%
75.96%
38.46%
74.35%
Standard Hierarchical
ResNet
Yes
58.60% ∥ 72.39%
75.92% ∥ 83.78%
40.42% ∥ 62.66%
74.36% ∥ 82.46%
P-CBM
ResNet
Yes
43.20%
59.60%
–
–
LF-CBM
ResNet
Yes
65.13%
74.31%
43.68%
71.95%
HybridCBM
ResNet
Yes
68.38%
–
–
–
TABLE I: The performance of our proposed model, HIL-CBM, compare with state-of-the-art Concept Bottleneck Models.
With
CIFAR-100
CUB
Place365
ImageNet
Neither
Higher
73.45
81.35
57.45
75.55
Lower
68.83
73.90
45.41
72.03
Visual
Higher
73.65
82.47
57.32
77.62
Lower
69.23
74.80
44.09
72.74
Semantic
Higher
73.31
82.75
57.37
78.43
Lower
69.12
74.75
46.35
74.90
TABLE II: Higher and lower-level classifiers’ accuracy under four training settings.
CUB-200
ImageNet
Visual/Semantic
Semantic 0.1
Semantic 0.3
Semantic 0.5
Semantic 0.1
Semantic 0.3
Semantic 0.5
Visual 0.1
75.27%
74.92%
74.82%
75.36%
75.26%
75.17%
Visual 0.3
75.01%
75.03%
75.08%
75.39%
75.37%
75.30%
Visual 0.5
74.78%
74.91%
75.01%
75.48%
75.40%
75.32%
Visual 0.7
75.35%
74.92%
75.03%
75.54%
75.35%
75.29%
TABLE III: Results under different visual and semantic weights on CUB-200 and ImageNet datasets.
Datasets
CUB-200
ImageNet
IoU Based
74.77% ∥ 82.95%
73.52% ∥ 80.93%
MSE Based
75.53% ∥ 83.08%
75.54% ∥ 81.50%
TABLE IV: Results of HIL-CBM with two different visual consistency loss settings
Metric
Semantic
CUB-200
ImageNet
Model consistency
Without
87.51%
80.75%
With
89.64%
84.40%
Ground-truth consistency
Without
83.96%
86.17%
With
84.32%
87.77%
TABLE V: Effect of semantic consistency loss on hierarchical consistency.
Criterion
Score
Model
A
B
Mean ± std
LF-CBM
2.85 ± 1.28
2.84 ± 1.2
HIL-CBM
3.39 ± 1.22
3.38 ± 1.20
Response Distribution (%)
1. None at All
LF-CBM
19.09
17.09
HIL-CBM
9.66
8.23
TABLE VI: Comparison of Likert-scale responses between LF-CBM and HIL-CBM for criteria A and B.
Fig. 3: Visualization of two-level predictions from HIL-CBM and the corresponding explanations at each level. The model predicts both levels of class labels, accompanied by concept-based explanations aligned with each level of abstraction. This hierarchical interpretability enables users to understand the model decisions from general to specific.
Fig. 4: Examples of model debugging. (a) An example where the higher-level prediction is correct, but the lower-level prediction is wrong. Debugging is performed by editing weights using domain knowledge. (b) An example where both predictions are incorrect. Debugging proceeds hierarchically: correcting the higher-level prediction enables guided refinement of the lower-level prediction via class masking.