Concept Bottleneck Models (CBMs) are interpretable models that route predictions through a layer of human-interpretable concepts. While widely studied in vision and, more recently, in NLP, CBMs remain largely unexplored in multimodal settings. For their explanations to be faithful, CBMs must satisfy two conditions: concepts must be properly detected, and concept representations must encode only their intended semantics, without smuggling extraneous task-relevant or inter-concept information into final predictions, a phenomenon known as leakage. Existing approaches treat concept detection and leakage mitigation as separate problems, and typically improve one at the expense of predictive accuracy. In this work, we introduce f-CBM, a faithful multimodal CBM framework built on a vision-language backbone that jointly targets both aspects through two complementary strategies: a differentiable leakage loss to mitigate leakage, and a Kolmogorov-Arnold Network prediction head that provides sufficient expressiveness to improve concept detection. Experiments demonstrate that f-CBM achieves the best trade-off between task accuracy, concept detection, and leakage reduction, while applying seamlessly to both image and text or text-only datasets, making it versatile across modalities.
Figures & tables
Figure 1: Pareto frontier: concept detection accuracy versus aggregate leakage. The x-axis represents the average of task-related and inter-concept leakage as introduced in Section 2 , and the y-axis represents RMSE concept detection performance.
Figure 2: Leakage analysis in multimodal CBMs
Figure 3: Overview of f-CBM , illustrated on an instance from the N24News dataset belonging to the Sport category.
Model backbone (size)
CLIP-base (110M)
CLIP-large (395M)
Dataset
Metric
Black -box
Indep. -CBM
Label -free
CT-CBM
f-CBM (ours)
Black -box
Indep. -CBM
Label -free
CT-CBM
f-CBM (ours)
AG News
%ACC ↑
90.6
86.6
91.0
91.0
9 0.6
91.5
86.2
91.7
9 1.2
90.7
c-RMSE ↓
–
0.043
1.264
0.101
0 .056
–
0.042
1.426
0.146
0 .068
CTL ↓
–
0 .028
0.212
0.244
0.005
–
0.922
0 .207
0.262
0.004
ICL ↓
–
0.005
0.050
0.059
0 .006
–
0 .004
0.047
0.079
0.002
DBpedia
%ACC ↑
100.0
97.3
99.2
99.3
9 9.2
99.3
98.1
99.3
9 9.0
98.9
Table 1: f-CBM and competitors evaluation on four test sets and two backbone sizes. Best results per row are in bold and second best are underlined . Black-box is excluded from ranking (marked –) as it produces no concept-level outputs.
N24 News
DBpedia
Baseline
Δ Task
Δ Concept
Δ Task
Δ Task
Δ Concept
Δ Task
Accuracy ↑
RMSE ↓
Leakage ↓
Accuracy ↑
RMSE ↓
Leakage ↓
CT-CBM
−0.3%
−50.4%
−69.9%
+8.7%
−38.3%
−96.6%
Indep.-CBM
−2.0%
+15.7%
−73.8%
−2.7%
−10.5%
−46.1%
Table 2: Impact of f-CBM enhancement of several CBM baselines with CLIP-base as a backbone.
Figure 4: Ablation study of f-CBM : effect of the KAN layer and the leakage loss on task accuracy, concept RMSE, and leakage (N24 dataset, CLIP-base backbone).
Figure 5: Effect of the leakage loss on concept activation distributions. Early in training (left), activations separate by predicted class, revealing concept-task leakage. Later (right), concept detection improved while the leakage loss reduces class information encoded in the activations, mitigating leakage.
Concept Bottleneck Models (CBMs) enhance the interpretability of deep learning networks by aligning the features extracted from images with natural concepts. However, existing CBMs are constrained in their ability to generalize beyond a fixed set of predefined classes and the risk of non-concept information leakage, where predictive signals outside the intended concepts are inadvertently exploited. In this paper, we propose Multimodal Concept Bottleneck Model (MM-CBM) to address these issues and extend CBMs into CLIP. MM-CBM utilizes dual Concept Bottleneck Layers (CBLs) to align both the image and text embeddings into interpretable features. This allows us to perform new vision tasks like zero-shot classification or image retrieval in an interpretable way. Compared to existing methods, MM-CBM achieves up to 51.26% accuracy improvement on average across four standard benchmarks. Our method maintains high accuracy, staying within ~5% of black-box performance while offering greater interpretability.
Concept Bottleneck Models (CBMs) aim for ante-hoc interpretability by learning a bottleneck layer that predicts interpretable concepts before the decision. State-of-the-art approaches typically select which concepts to learn via human specification, open knowledge graphs, prompting an LLM, or using general CLIP concepts. However, concepts defined a-priori may not have sufficient predictive power for the task or even be learnable from the available data. As a result, these CBMs often significantly trail their black-box counterpart when controlling for information leakage. To address this, we introduce a novel CBM pipeline named Mechanistic CBM (M-CBM), which builds the bottleneck directly from a black-box model's own learned concepts. These concepts are extracted via Sparse Autoencoders (SAEs) and subsequently named and annotated on a selected subset of images using a Multimodal LLM. For fair comparison and leakage control, we also introduce the Number of Contributing Concepts (NCC), a decision-level sparsity metric that extends the recently proposed NEC metric. Across diverse datasets, we show that M-CBMs consistently surpass prior CBMs at matched sparsity, while improving concept predictions and providing concise explanations. Our code is available at https://github.com/Antonio-Dee/M-CBM.
Antonio De Santis, Schrasing Tong, Marco Brambilla +1
Concept Bottleneck Models (CBMs) ground predictions in human-understandable concepts but face fundamental limitations: the absence of a metric to pre-evaluate concept relevance, the "linearity problem" causing recent CBMs to bypass the concept bottleneck entirely, an accuracy gap compared to opaque models, and finally the lack of systematic study on the impact of different visual backbones and VLMs. We introduce CBM-Suite, a methodological framework to systematically addresses these challenges. First, we propose an entropy-based metric to quantify the intrinsic suitability of a concept set for a given dataset. Second, we resolve the linearity problem by inserting a non-linear layer between concept activations and the classifier, which ensures that model accuracy faithfully reflects concept relevance. Third, we narrow the accuracy gap by leveraging a distillation loss guided by a linear teacher probe. Finally, we provide comprehensive analyses on how different vision encoders, vision-language models, and concept sets interact to influence accuracy and interpretability in CBMs. Extensive evaluations show that CBM-Suite yields more accurate models and provides insights for improving concept-based interpretability.
Merve Tapli, Quentin Bouniot, Wolfgang Stammer +2
Middle East Technical University (METU) · Helmholtz Munich · Munich Center for Machine Learning +5