Concepts' Information Bottleneck Models

Karim Galliamov; Syed M Ahsan Kazmi; Adil Khan; Adín Ramírez Rivera

cs.LGFeb 16, 2026

Concepts' Information Bottleneck Models

Authors: Karim Galliamov, Syed M Ahsan Kazmi, Adil Khan, Adín Ramírez Rivera

Organizations: University of Amsterdam · University of West of England · University of Hull · University of Oslo

Abstract

Concept Bottleneck Models (CBMs) aim to deliver interpretable predictions by routing decisions through a human-understandable concept layer, yet they often suffer reduced accuracy and concept leakage that undermines faithfulness. We introduce an explicit Information Bottleneck regularizer on the concept layer that penalizes $I(X;C)$ while preserving task-relevant information in $I(C;Y)$ , encouraging minimal-sufficient concept representations. We derive two practical variants (a variational objective and an entropy-based surrogate) and integrate them into standard CBM training without architectural changes or additional supervision. Evaluated across six CBM families and three benchmarks, the IB-regularized models consistently outperform their vanilla counterparts. Information-plane analyses further corroborate the intended behavior. These results indicate that enforcing a minimal-sufficient concept bottleneck improves both predictive performance and the reliability of concept-level interventions. The proposed regularizer offers a theoretic-grounded, architecture-agnostic path to more faithful and intervenable CBMs, resolving prior evaluation inconsistencies by aligning training protocols and demonstrating robust gains across model families and datasets.

Figures & tables

Method	Concept	Class
CUB
Black-box	–	0.919 $\pm$ 0.002
CBM (HJ)	0.956 $\pm$ 0.001	0.650 $\pm$ 0.002
CBM (HJ) + $\text{IB}_{B}$	0.951 $\pm$ 0.005	0.654 $\pm$ 0.000
CBM (HJ) + $\text{IB}_{E}$	0.951 $\pm$ 0.000	0.655 $\pm$ 0.000
CBM (HI)	0.956 $\pm$ 0.001	0.644 $\pm$ 0.001

Table 1: We report accuracy results (over 5 runs) of our proposed regularizers,

\text{IB}_{B}

and

\text{IB}_{E}

, applied to different CBMs. Black-box is a gold standard for class prediction that offers no explainability over the concepts.

Model	OIS	NIS	OIS	NIS	OIS	NIS
	Complete CS		Selective Drop-out CS		Random Drop-out CS
CBM (HJ)	$4.31\pm 0.57$	$65.03\pm 1.84$	$15.71$	$77.02$	$13.10\pm 1.20$	$73.09\pm 2.12$
CBM (HJ) + $\text{IB}_{B}$	$2.59\pm 0.65$	$60.02\pm 1.28$	$12.67$	$70.86$	$10.32\pm 0.57$	$70.38\pm 2.12$
CBM (HJ) + $\text{IB}_{E}$	$2.56\pm 0.47$	$59.52\pm 1.91$	$12.47$	$71.09$	$10.36\pm 0.80$	$69.95\pm 1.35$
CBM (HI)	$3.77\pm 0.58$	$65.15\pm 2.03$	$14.27$	$74.16$	$12.31\pm 1.04$	$71.78\pm 1.86$
CBM (HI) + $\text{IB}_{B}$	$3.58\pm 0.76$	$64.95\pm 1.59$	$14.01$	$73.16$	$12.63\pm 1.31$	$70.89\pm 2.04$

Table 2: Concept leakage evaluation (lower is better) on CUB.

CUB	AUC			NAUC
Corrupt	CBM	+ $\text{IB}_{B}$	+ $\text{IB}_{E}$	CBM	+ $\text{IB}_{B}$	+ $\text{IB}_{E}$
0	54.374	65.5886	64.6338	0.001260	0.0691	0.0035
4	53.135	64.4933	63.4168	0.001198	0.0967	0.0269
8	51.291	53.1240	60.1120	0.001166	0.0758	0.0608
16	50.694	60.2241	59.4141	0.001068	0.0399	0.0451
32	46.101	52.8855	51.1966	0.000863	0.0045	0.0432

Table 3: Change in interventions performance with concept set corruption for CBM (SJ) and its regularized versions with our proposed methods. We show the disaggregated plots in Fig. F.2 .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure C.1: Calculation of the empirical entropy of the concepts,

H(C)

, during training for CBM (SJ) on CUB.

Method	Backbone	Concept	Class
CBM	Inception	0.956	0.708
CBM + $\text{IB}_{B}$	Inception	0.957	0.726
CBM + $\text{IB}_{E}$	Inception	0.957	0.727
CBM	Resnet18	0.951	0.683
CBM + $\text{IB}_{B}$	Resnet18	0.954	0.701
CBM + $\text{IB}_{E}$	Resnet18	0.954	0.703

Appendix

Table D.1: Effect of the backbone on the CBM (SJ) and our regularizers.

Method	Epochs	Concept	Class
CBM	100	0.956	0.708
CBM + $\text{IB}_{B}$	100	0.957	0.726
CBM + $\text{IB}_{E}$	100	0.957	0.727
CBM	1000	0.957	0.795
CBM + $\text{IB}_{B}$	1000	0.958	0.806
CBM + $\text{IB}_{E}$	1000	0.958	0.810

Appendix

Table D.2: Comparison of concept and class prediction accuracies when changing the number of epochs in CUB.

Method	Concept	Class
$\text{IB}_{B}$ (vanilla)	0.934	0.608
$\text{IB}_{B}$ ( $\text{clip\_norm}=1.0$ )	0.948	0.663
$\text{IB}_{B}$ ( $\text{clip\_norm}=0.1$ )	0.948	0.650
$\text{IB}_{B}$ (stop grad. from $H(C)$ into $p(z\>\lvert\>x)$ )	0.958	0.726
$\text{IB}_{E}$	0.958	0.727

Appendix

Table D.3: Accuracies of CBM (SJ) with our proposed regularizers,

\text{IB}_{B}

and

\text{IB}_{E}

, on CUB dataset (avg. 3 runs).

$\beta$	Method	Concept	Class	Concept	Class	Concept	Class
		CUB		AwA2		aPY
0.10	$\text{IB}_{B}$	0.957	0.726	0.978	0.879	0.967	0.851
	$\text{IB}_{E}$	0.958	0.728	0.979	0.881	0.967	0.854
0.20	$\text{IB}_{B}$	0.957	0.726	0.980	0.880	0.967	0.853
	$\text{IB}_{E}$	0.958	0.727	0.978	0.880	0.967	0.852
0.25	$\text{IB}_{B}$	0.958	0.725	0.980	0.886	0.967	0.850

Appendix

Table D.4: Evaluation of CBM (SJ) with the proposed regularizers on three datasets with two different values of

\beta

Figure D.1: Losses on the validation set of CUB for CBM (SJ) and its variants regularized with our proposed methods.

Method	Concept	Class
Black-box	–	0.919 $\pm$ 0.002
CBM (HJ)	0.956 $\pm$ 0.001	0.650 $\pm$ 0.002
CBM (HJ) + $\text{IB}_{B}$	0.946 $\pm$ 0.000	0.653 $\pm$ 0.004
CBM (HJ) + $\text{IB}_{E}$	0.960 $\pm$ 0.000	0.647 $\pm$ 0.000
CBM (HI)	0.956 $\pm$ 0.001	0.644 $\pm$ 0.001
CBM (HI) + $\text{IB}_{B}$	0.955 $\pm$ 0.000	0.683 $\pm$ 0.000

Appendix

Table E.1: Extended results for Table 1 on CUB. Accuracy results include mean and std. over 5 runs. We report results of our proposed regularizer methods,

\text{IB}_{B}

and

\text{IB}_{E}

, applied to different CBMs. Black-box is a gold standard for class prediction that offers no explainability over the concepts.

Panel of Figure 13 — Figure F.1: Expanded results from Fig. 1 . Change in target prediction accuracy after intervening on concept groups following the random strategy as described in Section 4.3 . (TTI stands for Test-Time Interventions and NR for non-regularized.)

Figure F.2: Change in target prediction accuracy for different number of corrupted concepts. These are the expanded results of Table 3 . (TTI stands for Test-Time Interventions.)

Model	OIS	NIS	OIS	NIS	OIS	NIS
	Complete CS		Selective Drop-out CS		Random Drop-out CS
CBM (HJ)	$2.61\pm 0.27$	$57.98\pm 1.15$	$10.09$	$67.58$	$9.49\pm 1.36$	$70.74\pm 1.89$
CBM (HJ) + $\text{IB}_{B}$	$2.29\pm 0.23$	$56.89\pm 1.91$	$9.71$	$64.37$	$9.28\pm 0.88$	$69.79\pm 1.37$
CBM (HI)	$2.01\pm 0.43$	$63.03\pm 1.97$	$9.27$	$64.18$	$8.71\pm 1.03$	$69.14\pm 0.34$
CBM (HI) + $\text{IB}_{B}$	$1.78\pm 0.34$	$60.05\pm 0.91$	$8.40$	$67.23$	$8.31\pm 1.94$	$67.25\pm 1.52$
CBM (SJ)	$3.01\pm 0.29$	$64.44\pm 1.92$	$11.24$	$70.41$	$10.69\pm 1.00$	$70.68\pm 0.57$

Appendix

Table G.1: Concept leakage evaluation (lower is better) on AwA2.

Model	OIS	NIS	OIS	NIS	OIS	NIS
	Complete CS		Selective Drop-out CS		Random Drop-out CS
CBM (HJ)	$2.66\pm 0.19$	$57.97\pm 1.20$	$10.17$	$67.54$	$9.44\pm 1.45$	$70.78\pm 1.84$
CBM (HJ) + $\text{IB}_{B}$	$2.35\pm 0.16$	$56.91\pm 1.91$	$9.78$	$64.38$	$9.35\pm 0.81$	$69.70\pm 1.34$
CBM (HI)	$1.94\pm 0.41$	$62.97\pm 1.91$	$9.19$	$64.26$	$8.78\pm 0.95$	$69.06\pm 0.24$
CBM (HI) + $\text{IB}_{B}$	$1.80\pm 0.43$	$60.03\pm 0.86$	$8.31$	$67.18$	$8.25\pm 2.02$	$67.30\pm 1.59$
CBM (SJ)	$2.97\pm 0.37$	$64.39\pm 1.96$	$11.23$	$70.40$	$10.72\pm 1.01$	$70.73\pm 0.58$

Appendix

Table G.2: Concept leakage evaluation (lower is better) on aPY.

Panel of Figure 17 — Figure H.1: Information plane dynamics (in nats) for (a,b) IntCEM, (c,d) AR-CBM, (e, f) CBM (SJ) and our proposed methods, $\text{IB}_{B}$ and $\text{IB}_{E}$ . Warmer colors denote later steps in training. We show the information plane of (a, c, e) the variables $X$ , $C$ , and $Y$ ; and (b, d, f) the variables $X$ , $Z$ , and $C$ .

Concept Bottleneck Models

Concept Bottleneck Models route predictions through human-understandable concepts that people can inspect, correct, or intervene on.

See all 82 papers

Towards Reasonable Concept Bottleneck Models
Jun 5, 2025Nektarios Kalampalikis, Kavya Gupta, Georgi Vitanov +1Concept Bottleneck Models Interpretable ML
Department of Computer Science Saarland University, Germany
Learning Concept Bottleneck Models from Mechanistic Explanations
Mar 7, 2026Antonio De Santis, Schrasing Tong, Marco Brambilla +1Concept Bottleneck Models Explainability Evaluation
Politecnico di Milano · MIT CSAIL
Rethinking Concept Bottleneck Models: From Pitfalls to Solutions
Mar 5, 2026Merve Tapli, Quentin Bouniot, Wolfgang Stammer +2Concept Bottleneck Models
Middle East Technical University (METU) · Helmholtz Munich · Munich Center for Machine Learning +5

Jun 5, 2025J/K move · Enter open · S save

Towards Reasonable Concept Bottleneck Models

Nektarios Kalampalikis, Kavya Gupta, Georgi Vitanov, Isabel Valera

Department of Computer Science Saarland University, Germany

Concept Bottleneck Models Interpretable ML

Open paper AskPDF arXiv

We propose a novel, flexible, and efficient framework for designing Concept Bottleneck Models (CBMs) that enables practitioners to explicitly encode and extend their prior knowledge and beliefs about the concept-concept (

C-C

) and concept-task (

C \to Y

) relationships within the model's reasoning when making predictions. The resulting

\textbf{C}

oncept

\textbf{REA}

soning

\textbf{M}

odels (CREAMs) architecturally encode arbitrary types of

C-C

relationships such as mutual exclusivity, hierarchical associations, and/or correlations, as well as potentially sparse

C \to Y

relationships. Moreover, CREAM can optionally incorporate a regularized side-channel to complement the potentially {incomplete concept sets}, achieving competitive task performance while encouraging predictions to be concept-grounded. To evaluate CBMs in such settings, we introduce a

C \to Y

agnostic metric that quantifies interpretability when predictions partially rely on the side-channel. In our experiments, we show that, without additional computational overhead, CREAM models support efficient interventions, can avoid concept leakage, and achieve black-box-level performance under missing concepts. We further analyze how an optional side-channel affects interpretability and intervenability. Importantly, the side-channel enables CBMs to remain effective even in scenarios where only a limited number of concepts are available.

Abstract

Concept Bottleneck Models

Explore similar work

Towards Reasonable Concept Bottleneck Models

Learning Concept Bottleneck Models from Mechanistic Explanations

Rethinking Concept Bottleneck Models: From Pitfalls to Solutions