Organizations: Department of Computer Science, University of Manchester, United Kingdom. · School of Electrical Engineering and Computer Science, the University of Queensland, Australia. · Department of Computer Science, University of California at Merced, United State.
Concept Bottleneck Models (CBMs) introduce interpretability to black-box deep learning models by predicting labels through human-understandable concepts. However, unlike humans, who identify objects at different levels of abstraction using both general and specific features, existing CBMs operate at a single semantic level in both concept and label space. We propose HIL-CBM, a Hierarchical Interpretable Label-Free Concept Bottleneck Model that extends CBMs into a hierarchical framework to enhance interpretability by more closely mirroring the human cognitive process. HIL-CBM enables classification and explanation across multiple semantic levels without requiring relational concept annotations. HIL-CBM aligns the abstraction level of concept-based explanations with that of model predictions, progressing from abstract to concrete. This is achieved by (i) introducing a gradient-based visual consistency loss that encourages abstraction layers to focus on similar spatial regions, and (ii) training dual classification heads, each operating on feature concepts at different abstraction levels. Experiments on benchmark datasets demonstrate that HIL-CBM outperforms state-of-the-art sparse CBMs in classification accuracy. Human evaluations further show that HIL-CBM provides more interpretable and accurate explanations, while maintaining a hierarchical and label-free approach to feature concepts.
Figures & tables
Fig. 1: Our proposed HIL-CBM extends the CBM architecture into a hierarchical framework. The upper concept and label spaces capture general semantic features and broader class categories, while the lower spaces focus on more specific features and classes.
Fig. 2: Overview of our proposed model, HIL-CBM. A pre-trained backbone processes input images to produce image embeddings. Two concept layers, each focusing on a different level of abstraction, project the feature maps into interpretable hierarchical concept spaces. These layers are trained using CLIP-Dissect similarity loss and visual consistency loss. Finally, two hierarchical classifiers predict classes at different levels of abstraction, using cross-entropy loss and semantic consistency loss based on the learned hierarchical concept features.
Model
Backbone
Sparse
Lower Level ∥ Higher Level
CIFAR100
CUB-200
Places365
ImageNet
Standard
ResNet
Yes
58.34%
75.96%
38.46%
74.35%
Standard Hierarchical
ResNet
Yes
58.60% ∥ 72.39%
75.92% ∥ 83.78%
40.42% ∥ 62.66%
74.36% ∥ 82.46%
P-CBM
ResNet
Yes
43.20%
59.60%
–
–
LF-CBM
ResNet
Yes
65.13%
74.31%
43.68%
71.95%
HybridCBM
ResNet
Yes
68.38%
–
–
–
TABLE I: The performance of our proposed model, HIL-CBM, compare with state-of-the-art Concept Bottleneck Models.
With
CIFAR-100
CUB
Place365
ImageNet
Neither
Higher
73.45
81.35
57.45
75.55
Lower
68.83
73.90
45.41
72.03
Visual
Higher
73.65
82.47
57.32
77.62
Lower
69.23
74.80
44.09
72.74
Semantic
Higher
73.31
82.75
57.37
78.43
Lower
69.12
74.75
46.35
74.90
TABLE II: Higher and lower-level classifiers’ accuracy under four training settings.
CUB-200
ImageNet
Visual/Semantic
Semantic 0.1
Semantic 0.3
Semantic 0.5
Semantic 0.1
Semantic 0.3
Semantic 0.5
Visual 0.1
75.27%
74.92%
74.82%
75.36%
75.26%
75.17%
Visual 0.3
75.01%
75.03%
75.08%
75.39%
75.37%
75.30%
Visual 0.5
74.78%
74.91%
75.01%
75.48%
75.40%
75.32%
Visual 0.7
75.35%
74.92%
75.03%
75.54%
75.35%
75.29%
TABLE III: Results under different visual and semantic weights on CUB-200 and ImageNet datasets.
Datasets
CUB-200
ImageNet
IoU Based
74.77% ∥ 82.95%
73.52% ∥ 80.93%
MSE Based
75.53% ∥ 83.08%
75.54% ∥ 81.50%
TABLE IV: Results of HIL-CBM with two different visual consistency loss settings
Metric
Semantic
CUB-200
ImageNet
Model consistency
Without
87.51%
80.75%
With
89.64%
84.40%
Ground-truth consistency
Without
83.96%
86.17%
With
84.32%
87.77%
TABLE V: Effect of semantic consistency loss on hierarchical consistency.
Criterion
Score
Model
A
B
Mean ± std
LF-CBM
2.85 ± 1.28
2.84 ± 1.2
HIL-CBM
3.39 ± 1.22
3.38 ± 1.20
Response Distribution (%)
1. None at All
LF-CBM
19.09
17.09
HIL-CBM
9.66
8.23
TABLE VI: Comparison of Likert-scale responses between LF-CBM and HIL-CBM for criteria A and B.
Fig. 3: Visualization of two-level predictions from HIL-CBM and the corresponding explanations at each level. The model predicts both levels of class labels, accompanied by concept-based explanations aligned with each level of abstraction. This hierarchical interpretability enables users to understand the model decisions from general to specific.
Fig. 4: Examples of model debugging. (a) An example where the higher-level prediction is correct, but the lower-level prediction is wrong. Debugging is performed by editing weights using domain knowledge. (b) An example where both predictions are incorrect. Debugging proceeds hierarchically: correcting the higher-level prediction enables guided refinement of the lower-level prediction via class masking.
Concept Bottleneck Models (CBMs) enhance interpretability by projecting learned features into a human-understandable concept space. Recent approaches leverage vision-language models to generate concept embeddings, reducing the need for manual concept annotations. However, these models suffer from a critical limitation: as the number of concepts approaches the embedding dimension, information leakage increases, enabling the model to exploit spurious or semantically irrelevant correlations and undermining interpretability. In this work, we propose Concept Flow Models (CFMs), which replace the flat bottleneck with a hierarchical, concept-driven decision tree. Each internal node in the hierarchy focuses on a localized subset of discriminative concepts, progressively narrowing the prediction scope. Our framework constructs decision hierarchies from visual embeddings, distributes semantic concepts at each hierarchy level, and trains differentiable concept weights through probabilistic tree traversal. Extensive experiments on diverse benchmarks demonstrate that CFMs match the predictive performance of flat CBMs, while substantially mitigating information leakage by reducing effective concept usage. Furthermore, CFMs yield stepwise decision flows that enable transparent and auditable model reasoning with hierarchical class structures.
Ya Wang, Adrian Paschke
Fraunhofer Institute for Open Communication Systems · Freie Universität Berlin
Concept Bottleneck Models (CBMs) have become a popular approach to enable interpretability in neural networks by constraining classifier inputs to a set of human-understandable concepts. While effective, current models embed concepts in flat Euclidean space, treating them as independent, orthogonal dimensions. Concepts, however, are highly structured and organized in semantic hierarchies. To resolve this mismatch, we propose Hyperbolic Concept Bottleneck Models (HypCBM), a post-hoc framework that grounds the bottleneck in this structure by reformulating concept activation as asymmetric geometric containment in hyperbolic space. Rather than treating entailment cones as a pre-training penalty, we show they encode a natural test-time activation signal: the margin of inclusion within a concept's entailment cone yields sparse, hierarchy-aware activations without any additional supervision or learned modules. We further introduce an adaptive scaling law for hierarchically faithful interventions, propagating user corrections coherently through the concept tree. Empirically, HypCBM rivals post-hoc Euclidean models trained on 20× more data in sparse regimes required for human interpretability, with stronger hierarchical consistency and improved robustness to input corruptions.
Daniel Uyterlinde, Swasti Shreya Mishra, Pascal Mettes
Informatics Institute, University of Amsterdam, The Netherlands
Concept Bottleneck Models (CBMs) provide interpretability by routing predictions through a layer of human-understandable concepts. However, defining an optimal concept set for a specific dataset remains an open challenge. Existing approaches rely on expensive expert annotations or LLM-generated lists based solely on class names. Even "open-vocabulary" variants typically depend on static concept sets, which restrict discovery and introduce label bias. Furthermore, traditional CBMs often suffer from information leakage, where unmodeled visual features bypass the bottleneck and compromise the integrity of the explanations. To overcome these limitations, we propose Caption Bottleneck Models (CaBM), a framework that circumvents the need for predefined concept sets by replacing rigid concept layers with free-form natural language. By representing images via LMM-generated captions and training a classifier strictly on this text, CaBM ensures a leakage-free architecture by construction. Additionally, by analyzing the text classifier post-training, CaBM autonomously discovers high-quality, dataset-specific concepts. Our results across fine- and coarse-grained benchmarks demonstrate that CaBM achieves competitive accuracy while preserving interpretability without the constraints of external dictionaries or manual labeling.
Seref Baris Cagliyan, Umut Ozdemir, Merve Tapli +1
Dept. of Computer Eng., Middle East Technical University (METU) · Robotics & AI Center (ROMER), METU