Learning Concept Bottleneck Models from Mechanistic Explanations

Antonio De Santis; Schrasing Tong; Marco Brambilla; Lalana Kagal

cs.LGMar 7, 2026

Learning Concept Bottleneck Models from Mechanistic Explanations

Authors: Antonio De Santis, Schrasing Tong, Marco Brambilla, Lalana Kagal

Organizations: Politecnico di Milano · MIT CSAIL

Abstract

Concept Bottleneck Models (CBMs) aim for ante-hoc interpretability by learning a bottleneck layer that predicts interpretable concepts before the decision. State-of-the-art approaches typically select which concepts to learn via human specification, open knowledge graphs, prompting an LLM, or using general CLIP concepts. However, concepts defined a-priori may not have sufficient predictive power for the task or even be learnable from the available data. As a result, these CBMs often significantly trail their black-box counterpart when controlling for information leakage. To address this, we introduce a novel CBM pipeline named Mechanistic CBM (M-CBM), which builds the bottleneck directly from a black-box model's own learned concepts. These concepts are extracted via Sparse Autoencoders (SAEs) and subsequently named and annotated on a selected subset of images using a Multimodal LLM. For fair comparison and leakage control, we also introduce the Number of Contributing Concepts (NCC), a decision-level sparsity metric that extends the recently proposed NEC metric. Across diverse datasets, we show that M-CBMs consistently surpass prior CBMs at matched sparsity, while improving concept predictions and providing concise explanations. Our code is available at https://github.com/Antonio-Dee/M-CBM.

Figures & tables

Figure 1: Overview of the M-CBM pipeline. (1) Given a trained black-box backbone, we extract its features and learn sparse, disentangled concept directions using a Sparse Autoencoder (SAE). (2) A Multimodal LLM is prompted with examples of highly activating and non-activating images to assign concept names to each SAE neuron. (3) The MLLM then annotates a subset of the dataset containing an equal split of active and non-active examples, indicating the presence or absence of each concept in selected images. (4) Using these concept annotations, we train a Concept Bottleneck Layer (CBL) and a sparse linear classifier to predict target classes from the learned concepts.

Panel of Figure 2 — Figure 2: Accuracy vs NCC ( $\tau=0.95)$ on CUB. (a) With class-conditioned annotation, VLG-CBM reaches near black-box accuracy with NCC=1.5 (i.e., using only 1 to 2 concepts per prediction). The same happens using random concept names, showing evidence of leakage. (b) Making annotation class-agnostic (VLG-CBM ${}_{\text{CA}}$ ) restores the accuracy–interpretability trade-off, with real concepts slightly beating random words at low NCC. (c) M-CBM outperforms both VLG-CBM ${}_{\text{CA}}$ and the random baselines at low NCC, while leakage is significant for both methods as NCC increases.

Dataset	CUB		ISIC2018				ImageNet
Metrics	Accuracy		Accuracy		Balanced Accuracy		Accuracy
Black-box	76.67%		79.37%		75.37%		76.15%
Sparsity	NCC=5	NCC=avg	NCC=5	NCC=avg	NCC=5	NCC=avg	NCC=5	NCC=avg
LF-CBM	58.08%	71.09%	61.44%	67.55%	64.29%	67.30%	62.20%	69.08%
DN-CBM ${}_{\text{RN}}$	38.21%	48.98%	35.38%	54.61%	39.85%	52.85%	46.71%	57.24%
DN-CBM ${}_{\text{ViT}}$	48.12%	66.19%	43.92%	56.08%	42.47%	53.56%	60.23%	69.98%

Table 1: Accuracy comparison at NCC=5 and NCC=avg with best model in bold. The results for M-CBM are averaged over 3 seeds with same annotations. N/A denotes computationally unfeasible.

Dataset	CUB		ISIC2018		ImageNet
Metrics	ROC-AUC		ROC-AUC		ROC-AUC
Metrics	Macro	Worst-10%	Macro	Worst-10%	Macro	Worst-10%
VLG-CBM ${}_{\text{CA}}$	62.03%	45.60%	73.37%	52.92%	N/A	N/A
M-CBM (Ours)	90.04%	79.05%	80.57%	66.98%	88.90%	78.36%

Table 2: ROC-AUC evaluation of concept predictions on test set. Each method is evaluated on its own concepts. We report the macro-average across concepts and the average of the worst 10%.

Panel of Figure 3 — Figure 3: Sankey plots of concept–class weights of our M-CBM at NCC=5. Concepts on the left and classes on the right. Concepts with negative weights are labeled as “NOT concept” .

Panel of Figure 4 — Figure 4: Per-image explanations of M-CBM at NCC=5 for a correct prediction in CUB (a) and a misclassification in ISIC 2018 (b). Concepts with negative logit are labeled as “NOT concept” .

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	CUB	ISIC2018	ImageNet
Backbone layer dimension	512	2048	2048
Expansion factor	$1\times$	$0.25\times$	$4\times$
Optimizer	Adam	Adam	Adam
Learning rate	$1\times 10^{-4}$	$1\times 10^{-4}$	$1\times 10^{-3}$
L1 coefficient ( $\lambda_{\mathrm{SAE}}$ )	$2\times 10^{-3}$	$5\times 10^{-4}$	$1\times 10^{-3}$
Epochs	1000	1000	1000

Appendix

Table 3: Training hyperparameters for the SAEs used in the paper.

Metric	CUB	ISIC2018	ImageNet
L2 reconstruction loss	0.0231	0.0066	0.0462
Average $\ell_{0}$	7.66	17.14	39.23
Recovered loss (CE)	89.49%	99.58%	97.74%
Recovered accuracy	98.39%	acc: 96.08%, bal. acc: 96.84%	96.68%

Appendix

Table 4: Evaluation metrics for the SAEs used in the concept extraction phase (pre-pruning). These are computed on the validation set, except for

\ell_{0}

, which is computed on the training set.

Panel of Figure 5 — Figure 5: Feature density histogram for CUB, ISIC2018 and ImageNet. Purple indicates dead neurons (never active in the training set). Yellow indicates neurons that were pruned due to low density and little to no impact on recovered loss. Green indicates neurons that are kept as concepts for the subsequent steps.

Panel of Figure 6 — Figure 6: Effect of pruning by activation-count threshold for CUB, ISIC2018, and ImageNet. Higher thresholds typically reduce recovered performance, but when discarding low-density neurons, the reduction tends to be negligible. The point highlighted by a red star indicates the cutoff used in our experiments.

Figure 7: Examples of VLG-CBM explanations where replacing all concept names with random ones still yields correct predictions using just 1–2 concepts, illustrating how class-conditioned annotation can leak class-specific information unrelated to concept semantics.

Method	Dataset	Backbone	Backbone (M)	CBL $K$	CBM (M)	Total (M)
LF-CBM	CUB	ResNet-18	11.69	208	0.15	11.84
LF-CBM	ISIC2018	ResNet-50	25.56	35	0.07	25.63
LF-CBM	ImageNet	ResNet-50	25.56	4523	13.79	39.35
DN-CBM ${}_{\text{RN}}$	CUB	CLIP RN50	38.30	8192	10.04	48.34
DN-CBM ${}_{\text{RN}}$	ISIC2018	CLIP RN50	38.30	8192	8.45	46.75
DN-CBM ${}_{\text{RN}}$	ImageNet	CLIP RN50	38.30	8192	16.59	54.89

Appendix

Table 5: Inference-time parameter counts (in millions). Backbone counts include only the pretrained feature extractor. CBM counts include the CBL and the final classifier.

Panel of Figure 8 — Figure 8: Top-5 activating images for CUB concepts

Panel of Figure 9 — Figure 9: Top-5 activating images for ISIC2018 concepts

Panel of Figure 10 — Figure 10: Top-5 activating images for ImageNet concepts

Panel of Figure 11 — Figure 11: Examples of local explanations from CUB test set.

Panel of Figure 12 — Figure 12: Examples of local explanations from ISIC2018 test set.

Panel of Figure 13 — Figure 13: Examples of local explanations from ImageNet test set.

Metric	Value
FPR (poisoned)	0.232
FPR (normal)	0.198
Two-proportion $z$ -test ( $p$ -value)	0.157
Cohen’s $h$ effect size	0.084

Appendix

Table 6: Results of MLLM-based concept annotation under poisoning of the reference set on the CUB dataset using human-annotated concepts as ground truth. We report the false positive rates (FPR) for poisoned vs normal negative examples, the two-proportion

z

-test, and the effect size (Cohen’s

h

Dataset	CUB		ISIC2018				ImageNet
Metrics	Accuracy		Accuracy		Balanced Accuracy		Accuracy
Sparsity	NEC=5	NEC=avg	NEC=5	NEC=avg	NEC=5	NEC=avg	NEC=5	NEC=avg
M-CBM	72.54%	73.70%	69.38%	73.47%	68.97%	70.45%	71.23%	73.00%

Appendix

Table 7: Accuracy results at NEC=5 and NEC=avg.

Panel of Figure 14 — Figure 14: Top-5 activating images for CBL neurons that have all zero weights at NEC=5, but not at NCC=5 for all datasets.

Panel of Figure 15 — Figure 15: Top-5 activating images for SAE features in CUB

Panel of Figure 16 — Figure 16: Top-5 activating images for SAE features in ImageNet

Panel of Figure 17 — Figure 17: Top-5 activating images for SAE features in ISIC2018

Dataset	CUB		ISIC2018				ImageNet
Metrics	Accuracy		Accuracy		Balanced Accuracy		Accuracy
Sparsity	NCC=5	NCC=avg	NCC=5	NCC=avg	NCC=5	NCC=avg	NCC=5	NCC=avg
M-CBM	71.40%	74.02%	67.33%	74.25%	69.26%	72.20%	56.76%	68.76%

Appendix

Table 8: Accuracy results at NCC=5 and NCC=avg of M-CBM using an open-source MLLM ( InternVL3.5-241B-A28B ) for naming and annotation and e5-large-v2 for concept merging.

Dataset	CUB		ISIC2018		ImageNet
Metrics	ROC-AUC		ROC-AUC		ROC-AUC
Metrics	Macro	Worst-10%	Macro	Worst-10%	Macro	Worst-10%
M-CBM	74.49%	58.14%	74.36%	56.53%	70.97%	59.28%

Appendix

Table 9: Evaluation of concept predictions on the test set for M-CBM using an open-source MLLM ( InternVL3.5-241B-A28B ) for naming and annotation and e5-large-v2 for concept merging.

Concept Bottleneck Models

Concept Bottleneck Models route predictions through human-understandable concepts that people can inspect, correct, or intervene on.

See all 82 papers

When Are Concept Bottleneck Model Explanations Faithful and Compact?
Oct 5, 2026Stefano Teso, Emanuele Marconato, Steve Azzolin +1Explainable Artificial Intelligence Concept Bottleneck Models
CIMeC & DISI University of Trento · Department of Mathematical Sciences University of Copenhagen · DISI University of Trento +1
Caption Bottleneck Models
Jul 1, 2026Seref Baris Cagliyan, Umut Ozdemir, Merve Tapli +1Vision-Language Models Concept Bottleneck Models
Dept. of Computer Eng., Middle East Technical University (METU) · Robotics & AI Center (ROMER), METU
Towards Reasonable Concept Bottleneck Models
Jun 5, 2025Nektarios Kalampalikis, Kavya Gupta, Georgi Vitanov +1Concept Bottleneck Models Interpretable ML
Department of Computer Science Saarland University, Germany

Oct 5, 2026J/K move · Enter open · S save

When Are Concept Bottleneck Model Explanations Faithful and Compact?

Stefano Teso, Emanuele Marconato, Steve Azzolin, Antonio Vergari

CIMeC & DISI University of Trento · Department of Mathematical Sciences University of Copenhagen · DISI University of Trento+1

Explainable Artificial Intelligence Concept Bottleneck Models

Open paper AskPDF arXiv

Concept bottleneck models (CBMs) are neural classifiers that allow to explain their decisions via high-level concepts, potentially enabling understanding, steering and debugging. However, their explanations are often derived heuristically. Building on formal explainability, we argue they should also be faithful, i.e., not misreport which concepts actually matter. We show that, for widespread CBM architectures, including recent VLM-based variants, faithful explanations must include all concepts in the bottleneck, compromising interpretability when this is large. This result applies to both heuristic and faithful-by-construction formal explanations. To encourage the existence of compact faithful explanations, we suggest i) modeling concepts probabilistically as binary or categorical random variables (rather than logits), and ii) employing per-concept training-time sparsification via group lasso (rather than regular elastic net). We also extend algorithms from formal explainability to CBMs, and show they outperform natural heuristics in terms of guarantees and explanation size. Overall, our work warns against naive interpretability claims and provides formal conditions and practical strategies for ensuring CBMs are as interpretable as advertised.

Abstract

Concept Bottleneck Models

Explore similar work

When Are Concept Bottleneck Model Explanations Faithful and Compact?

Caption Bottleneck Models

Towards Reasonable Concept Bottleneck Models