Interpretable but Fragile? Robustness of Concept Bottlenecks under Geometric-Semantic Perturbations

Hanwei Zhang; Tianma Hu; Gaojie Jin; Xu Cheng; Ronghui Mu

cs.LGSep 29, 2026

Interpretable but Fragile? Robustness of Concept Bottlenecks under Geometric-Semantic Perturbations

Authors: Hanwei Zhang, Tianma Hu, Gaojie Jin, Xu Cheng, Ronghui Mu

Organizations: Department of Computer Science Saarland University Saarbrücken, Germany · School of Computer Science and Engineering Tianjin University of Technology Tianjin, China · Department of Artificial Intelligence & Institute of Artificial Intelligence and Brain Sciences University of Macau Macau SAR, China · Department of Computer Science University of Exeter Exeter,UK

Abstract

Concept Bottleneck Models (CBMs) are designed to provide interpretable intermediate representations, yet how such bottlenecks affect robustness remains unclear, with existing studies reporting mixed and sometimes contradictory findings. We argue that these discrepancies arise from conflating different robustness notions and perturbation regimes, rather than from fundamental disagreements about CBMs themselves. To disentangle these factors, we introduce a generator-based evaluation framework that enables controlled comparisons between standard classifiers and CBMs under two distinct perturbation types: continuous geometric perturbations in latent space and discrete semantic interventions in concept space. Within this framework, we evaluate robustness both empirically, via prediction and concept-level sensitivity metrics, and certifiably, using randomized smoothing in latent and concept spaces. Across experiments, we reconcile previously conflicting findings by clarifying when, and in what sense, concept bottlenecks do or do not improve robustness. By further analyzing robustness under varying task conditions, including class semantic similarity and concept vocabulary size, we show that interpretability does not inherently confer robustness. Instead, concept bottlenecks shift where and how sensitivity manifests, revealing a nuanced interpretability robustness trade off that depends critically on the perturbation regime and task structure. Together, our results show that interpretability and robustness are distinct objectives: interpretable intermediate representations do not uniformly improve robustness, but instead redistribute sensitivity across perturbation spaces and model families.

Figures & tables

Panel of Figure 1 — Figure 1 : Evaluation framework and representative results. (a) Evaluation robustness across geometric vs. semantic perturbations and empirical vs. certified criteria. (b) The heatmap reports representative $f_{cbm}$ – $f_{std}$ robustness gaps using $\epsilon=0.3$ , $\tau=5$ , $\sigma=0.10$ , and $\rho=0.10$ for the four columns, respectively. All gaps are oriented so that positive values indicate greater CBM robustness. Detailed computations in Appendix C.1 (c) Certified concept-radius examples at $\rho=0.10$ show opposite outcomes across task settings.

Panel of Figure 2 — Figure 2 : Empirical robustness under geometric and semantic perturbations. Orange (left axis): CUB-200-2011; blue (right axis): RIVAL-10. Dataset-specific y-axis scales are used for clarity.

Panel of Figure 3 — Figure 3 : Prediction sensitivity in concept - level measures on CUB - 200 - 2011.

Panel of Figure 4 — Figure 4 : Certified robustness under latent - space and concept - space randomized smoothing.

Panel of Figure 5 — Figure 5 : Adversarial robustness under APGD - CE ( Croce and Hein, 2020 ) attacks on the CUB generated dataset.

Panel of Figure 6 — Figure 6 : Effect of class structure and semantic similarity on task - conditional robustness under geometric perturbations. (a,b) PFR v.s. $\epsilon$ for dissimilar and similar class settings. (c) Difference in PFR (denoted as $\Delta$ PFR) between $f_{cbm}$ and $f_{std}$ when $\epsilon=0.8$ . Lower $\Delta$ PFR favors $f_{cbm}$ .

Panel of Figure 7 — Figure 7 : Impact of concept vocabulary size on robustness. Concept-vocabulary ablation on RIVAL-10. (a,b) Empirical PFR under geometric latent perturbations and semantic concept flips. (c,d) Certified robustness under Gaussian latent smoothing and Bernoulli concept smoothing.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8 : Generator-based evaluation framework for geometric vs. semantic perturbations. (i) geometric perturbations by adding latent-space noise within an $\ell_{2}$ budget $\epsilon$ , and (ii) semantic perturbations by flipping up to $\tau$ binary concepts in concept space before image generation.

Perturbation		CUB-200-2011				RIVAL-10
Type	Budget	$\mathrm{CFR}$	PFR (%) $\downarrow$		$\mathrm{DFS}\downarrow$	$\mathrm{CFR}$	PFR (%) $\downarrow$		$\mathrm{DFS}\downarrow$
			$f_{\mathrm{std}}$	$f_{\mathrm{cbm}}$			$f_{\mathrm{std}}$	$f_{\mathrm{cbm}}$
Geo.	$\epsilon=0.05$	0.40 $\pm$ 0.01	27.1 $\pm$ 2.1	31.4 $\pm$ 1.5	1.28 $\pm$ 0.03	0.32 $\pm$ 0.14	0.00	0.00	0.48 $\pm$ 0.01
	$\epsilon=0.1$	0.79 $\pm$ 0.02	42.2 $\pm$ 2.5	46.6 $\pm$ 1.4	1.59 $\pm$ 0.01	0.67 $\pm$ 0.22	0.00	0.00	0.75 $\pm$ 0.01
	$\epsilon=0.3$	2.14 $\pm$ 0.04	74.6 $\pm$ 1.6	76.6 $\pm$ 0.8	3.26 $\pm$ 0.05	2.11 $\pm$ 0.37	0.00	0.00	1.28 $\pm$ 0.03
	$\epsilon=0.5$	3.15 $\pm$ 0.04	86.3 $\pm$ 1.0	87.8 $\pm$ 1.3	4.08 $\pm$ 0.04	4.09 $\pm$ 0.47	0.00	0.00	1.62 $\pm$ 0.02

Appendix

Table 1: Robustness comparison under geometric (latent

\ell_{2}

noise) and semantic (concept flip) perturbations.

Type	Budget	$\overline{\Delta}\pm\sigma_{\Delta}$ (pp)	$p$	$p_{\mathrm{Holm}}$
Geo.	$\epsilon=0.05$	$-4.25\pm 2.43$	0.017	0.098
Geo.	$\epsilon=0.10$	$-4.39\pm 2.26$	0.012	0.086
Geo.	$\epsilon=0.30$	$-2.00\pm 1.21$	0.021	0.098
Geo.	$\epsilon=0.50$	$-1.12\pm 0.63$	0.016	0.098
Geo.	$\epsilon=0.80$	$-0.51\pm 0.31$	0.022	0.098
Sem.	$\tau=1$	$-2.33\pm 1.49$	0.025	0.098

Appendix

Table 2: Paired significance tests for PFR differences between

f_{\text{std}}

and

f_{\text{cbm}}

across

K=5

subset repeats (each repeat uses

n=200

images).

\Delta

is measured in percentage points (pp). We report raw two-sided paired

t

-test

p

-values and Holm–Bonferroni corrected

p_{\mathrm{Holm}}

across all 9 budgets in the table.

Panel of Figure 9 — Figure 9 : mean PFR vs mean DFS (left) and mean DFS vs mean CFR (right). Red: Geometric , Blue: Semantic . Solid: $f_{std}$ ; Dashed: $f_{cbm}$ .

Type	Budget	Model	CUB-200-2011		RIVAL-10 (10-class)
			SmAcc (%) $\uparrow$	$R/\Delta\uparrow$	SmAcc (%) $\uparrow$	$R/\Delta\uparrow$
Latent RM	$\sigma=0.01$	$f_{\mathrm{std}}$	96.20 $\pm$ 0.84	0.0228 $\pm$ 0.0005	100.00 $\pm$ 0.00	0.0246 $\pm$ 0.0000
	$\sigma=0.01$	$f_{\mathrm{cbm}}$	97.40 $\pm$ 0.55	0.0215 $\pm$ 0.0004	100.00 $\pm$ 0.00	0.0246 $\pm$ 0.0000
	$\sigma=0.05$	$f_{\mathrm{std}}$	88.40 $\pm$ 1.39	0.0747 $\pm$ 0.0031	100.00 $\pm$ 0.00	0.1232 $\pm$ 0.0000
	$\sigma=0.05$	$f_{\mathrm{cbm}}$	84.90 $\pm$ 1.67	0.0672 $\pm$ 0.0026	100.00 $\pm$ 0.00	0.1232 $\pm$ 0.0000
	$\sigma=0.10$	$f_{\mathrm{std}}$	81.70 $\pm$ 2.39	0.1034 $\pm$ 0.0030	100.00 $\pm$ 0.00	0.2463 $\pm$ 0.0000

Appendix

Table 3: Unified randomized smoothing results in latent and concept space. Latent RM uses Gaussian noise

\delta\sim\mathcal{N}(0,\sigma^{2}I)

and reports certified

\ell_{2}

radius

R

. Concept RM uses Bernoulli concept flips with rate

\rho

and reports certified

\ell_{0}

radius

\Delta

. SmAcc denotes smoothed accuracy. Values are mean

\pm

std over five random subsets. Bold indicates the better value between

f_{\mathrm{std}}

and

f_{\mathrm{cbm}}

for CUB-200-2011 only; we omit highlighting on RIVAL-10 due to saturation.

$\sigma$	$\Delta$ SmAcc (pp)	$p$	$\Delta R$	$p$
0.01	$-1.90\pm 1.56$	0.053 (Holm: 0.210)	$0.0010\pm 0.0004$	0.004 (Holm: 0.028)
0.05	$0.80\pm 2.59$	0.528 (Holm: 0.917)	$0.0061\pm 0.0012$	$2.93\times 10^{-4}$ (Holm: 0.002)
0.10	$3.60\pm 3.45$	0.080 (Holm: 0.240)	$0.0078\pm 0.0030$	0.004 (Holm: 0.028)
0.30	$1.30\pm 3.55$	0.458 (Holm: 0.917)	$0.0089\pm 0.0064$	0.036 (Holm: 0.182)

Appendix

Table 4: Significance analysis over

K=5

random 200-sample subsets (mean

\pm

std across subsets). Differences are reported as

\Delta=f_{std}-f_{cbm}

(pp = percentage points). We report paired two-sided p-values across the five subset-level measurements; Holm denotes Holm–Bonferroni correction over all tests.

$m$	$\Delta$ SmAcc (pp)	$p$ (Holm)	$\Delta\tau$	$p$ (Holm)
1	-1.50 $\pm$ 2.09	0.1841 (0.5334)	0.174 $\pm$ 0.051	0.0016 (0.0094)
2	1.80 $\pm$ 2.46	0.1778 (0.5334)	0.020 $\pm$ 0.069	0.5501 (0.5501)
4	-2.90 $\pm$ 2.61	0.0677 (0.3386)	0.019 $\pm$ 0.020	0.0950 (0.3798)

Appendix

Table 5: Significance analysis for concept-space (

\ell_{0}

) randomized smoothing over

K=5

random 200-sample subsets. Differences are

\Delta=\text{fstd}-\text{fcbm}

(pp = percentage points), reported as mean

\pm

std across subsets. We report paired two-sided p-values across the five subset-level measurements; Holm denotes Holm–Bonferroni correction over all

6

tests (3 flip budgets

\times

2 metrics).

Panel of Figure 10 — Figure 10 : Distance heatmap among 20 classes under similar and dissimilar settings. Class distances are computed as Euclidean distances between class - level centroid features extracted using a ResNet - 50 backbone.

Perturbation		10+ 5 dissimilar classes				10+ 5 similar classes				10+ 10 dissimilar classes				10+ 10 similar classes
Type	Budget	CFR	PFR (%) $\downarrow$		DFS $\downarrow$	CFR	PFR (%) $\downarrow$		DFS $\downarrow$	CFR	PFR (%) $\downarrow$		DFS $\downarrow$	CFR	PFR (%) $\downarrow$		DFS $\downarrow$
			$f_{\mathrm{std}}$	$f_{\mathrm{cbm}}$			$f_{\mathrm{std}}$	$f_{\mathrm{cbm}}$			$f_{\mathrm{std}}$	$f_{\mathrm{cbm}}$			$f_{\mathrm{std}}$	$f_{\mathrm{cbm}}$
Geo.	$\varepsilon=0.05$	0.81	$0.29\pm 0.16$	$0.06\pm 0.04$	$0.37\pm 0.01$	0.56	$3.72\pm 0.64$	$2.06\pm 0.47$	$0.60\pm 0.01$	0.44	$0.06\pm 0.04$	$0.08\pm 0.05$	$0.48\pm 0.01$	0.14	$1.86\pm 0.35$	$1.85\pm 0.43$	$0.45\pm 0.01$
	$\varepsilon=0.1$	1.77	$0.46\pm 0.19$	$0.15\pm 0.08$	$0.60\pm 0.02$	1.11	$5.96\pm 0.86$	$3.61\pm 0.65$	$0.92\pm 0.02$	0.93	$0.17\pm 0.08$	$0.25\pm 0.11$	$0.75\pm 0.02$	0.18	$3.81\pm 0.57$	$3.40\pm 0.65$	$0.72\pm 0.02$
	$\varepsilon=0.3$	5.77	$0.80\pm 0.24$	$0.33\pm 0.11$	$1.10\pm 0.03$	3.85	$10.06\pm 1.20$	$6.94\pm 0.98$	$1.60\pm 0.03$	3.38	$0.47\pm 0.14$	$0.61\pm 0.17$	$1.32\pm 0.03$	1.02	$8.01\pm 0.98$	$6.10\pm 0.93$	$1.29\pm 0.03$
	$\varepsilon=0.5$	9.83	$1.03\pm 0.29$	$0.53\pm 0.16$	$1.39\pm 0.04$	7.17	$11.71\pm 1.33$	$8.34\pm 1.11$	$1.95\pm 0.04$	6.89	$0.59\pm 0.16$	$0.81\pm 0.18$	$1.67\pm 0.03$	2.42	$9.91\pm 1.12$	$7.37\pm 1.03$	$1.62\pm 0.03$

Appendix

Table 6: Robustness on RIVAL-10 ( 18 concepts, varying class count ). Backbone: ResNet50. Geo. : latent

\ell_{2}

perturbation (values mean

\pm

SE). Sem. : concept number flip,

\tau

concepts flipped (global average). CFR (%): concept flip rate /

\tau

; PFR (%): prediction flip rate; DFS: mean PCBM concept distance.

Type	Budget	Model	10 + 5 dissimilar classes		0 + 5 similar classes		10 + 10 dissimilar classes		10 + 10 similar classes
			SmAcc (%) $\uparrow$	$R/\Delta_{\tau}\uparrow$	SmAcc (%) $\uparrow$	$R/\Delta_{\tau}\uparrow$	SmAcc (%) $\uparrow$	$R/\Delta_{\tau}\uparrow$	SmAcc (%) $\uparrow$	$R/\Delta_{\tau}\uparrow$
	$\sigma=0.01$	$f_{\mathrm{std}}$	100.00 $\pm$ 0.00	0.0699 $\pm$ 0.0001	99.50 $\pm$ 0.00	0.0659 $\pm$ 0.0007	100.00 $\pm$ 0.00	0.0699 $\pm$ 0.0002	99.70 $\pm$ 0.24	0.0660 $\pm$ 0.0006
		$f_{\mathrm{cbm}}$	100.00 $\pm$ 0.00	0.0703 $\pm$ 0.0000	100.00 $\pm$ 0.00	0.0681 $\pm$ 0.0006	100.00 $\pm$ 0.00	0.0700 $\pm$ 0.0002	99.70 $\pm$ 0.24	0.0677 $\pm$ 0.0009
	$\sigma=0.05$	$f_{\mathrm{std}}$	99.70 $\pm$ 0.24	0.3446 $\pm$ 0.0019	95.70 $\pm$ 1.50	0.3112 $\pm$ 0.0065	100.00 $\pm$ 0.00	0.3401 $\pm$ 0.0017	98.60 $\pm$ 0.49	0.2973 $\pm$ 0.0087
Latent RM		$f_{\mathrm{cbm}}$	100.00 $\pm$ 0.00	0.3475 $\pm$ 0.0006	98.30 $\pm$ 0.93	0.3179 $\pm$ 0.0037	100.00 $\pm$ 0.00	0.3401 $\pm$ 0.0012	97.50 $\pm$ 0.84	0.3218 $\pm$ 0.0035
	$\sigma=0.10$	$f_{\mathrm{std}}$	100.00 $\pm$ 0.00	0.6808 $\pm$ 0.0056	92.20 $\pm$ 1.12	0.6368 $\pm$ 0.0140	100.00 $\pm$ 0.00	0.6686 $\pm$ 0.0046	95.30 $\pm$ 0.68	0.5909 $\pm$ 0.0146

Appendix

Table 7: Randomized smoothing ablation on RIVAL-10 variants after removing the saturated 10-class setting. Latent RM uses Gaussian latent noise

\eta\sim\mathcal{N}(0,\sigma^{2}I)

and reports certified

\ell_{2}

radius

R

. Concept RM uses Bernoulli concept flips with probability

\rho

and reports certified

\ell_{0}

concept-flip radius

\Delta_{\tau}

. SmAcc denotes smoothed accuracy. All variants use 18 concepts.

Perturbation		18 concepts				27 concepts				36 concepts
Type	Budget	CFR	PFR (%) $\downarrow$		DFS $\downarrow$	CFR	PFR (%) $\downarrow$		DFS $\downarrow$	CFR	PFR (%) $\downarrow$		DFS $\downarrow$
			$f_{\mathrm{std}}$	$f_{\mathrm{cbm}}$			$f_{\mathrm{std}}$	$f_{\mathrm{cbm}}$			$f_{\mathrm{std}}$	$f_{\mathrm{cbm}}$
Geo.	$\varepsilon=0.05$	0.32	0.00	0.00	$0.48\pm 0.01$	0.15	0.00	0.00	$0.71\pm 0.02$	1.0	0.00	0.00	$0.77\pm 0.03$
	$\varepsilon=0.10$	0.67	0.00	0.00	$0.74\pm 0.01$	0.56	0.00	$0.01\pm 0.01$	$1.10\pm 0.02$	1.67	0.00	0.00	$1.21\pm 0.04$
	$\varepsilon=0.30$	2.11	0.00	0.00	$1.29\pm 0.02$	2.58	0.00	$0.01\pm 0.01$	$1.91\pm 0.04$	4.09	0.00	0.00	$2.20\pm 0.06$
	$\varepsilon=0.50$	4.09	0.00	0.00	$1.61\pm 0.03$	5.16	$0.01\pm 0.00$	$0.01\pm 0.01$	$2.36\pm 0.04$	6.94	$0.01\pm 0.00$	0.00	$2.86\pm 0.08$

Appendix

Table 8: Empirical robustness under geometric (latent

\ell_{2}

) and semantic (concept flip) perturbations on RIVAL-10 with varying concept vocabulary size. CFR: concept flip rate; PFR (%): prediction flip rate; DFS: PCBM concept distance.

Type	Budget	Model	27 concepts		36 concepts
			SmAcc (%) $\uparrow$	$R\uparrow$	SmAcc (%) $\uparrow$	$R\uparrow$
Latent	$\sigma=0.01$	$f_{\mathrm{std}}$	100.00 $\pm$ 0.00	0.0703 $\pm$ 0.0000	100.00 $\pm$ 0.00	0.0703 $\pm$ 0.0000
		$f_{\mathrm{cbm}}$	100.00 $\pm$ 0.00	0.0703 $\pm$ 0.0000	100.00 $\pm$ 0.00	0.0703 $\pm$ 0.0000
	$\sigma=0.05$	$f_{\mathrm{std}}$	100.00 $\pm$ 0.00	0.3517 $\pm$ 0.0000	100.00 $\pm$ 0.00	0.2876 $\pm$ 0.0786
		$f_{\mathrm{cbm}}$	100.00 $\pm$ 0.00	0.2607 $\pm$ 0.0747	100.00 $\pm$ 0.00	0.1996 $\pm$ 0.0107
	$\sigma=0.10$	$f_{\mathrm{std}}$	100.00 $\pm$ 0.00	0.4756 $\pm$ 0.1141	100.00 $\pm$ 0.00	0.4052 $\pm$ 0.0261

Appendix

Table 9: RIVAL-10 10-class results with 27 vs 36 concepts (ResNet50), recomputed with image-level 5

\times

200 subset aggregation (mean

\pm

std). Better value is bolded only when strictly better (ties are not bolded).

		10 class;K=18			10 + 10 similar classes; K=18			10 + 10 dissimilar classes; K=18			10 + 10 dissimilar classes; K=27
Type	Budget	PFR ${}_{f_{\text{dn}}}$	PFR ${}_{f_{\text{cbm}}}$	DFS	PFR ${}_{f_{\text{dn}}}$	PFR ${}_{f_{\text{cbm}}}$	DFS	PFR ${}_{f_{\text{dn}}}$	PFR ${}_{f_{\text{cbm}}}$	DFS	PFR ${}_{f_{\text{dn}}}$	PFR ${}_{f_{\text{cbm}}}$	DFS
Geo. ( $\ell_{2}$ latent)
	$\varepsilon=0.05$	$0.00\pm 0.00$	$0.00$	$0.391\pm 0.007$	$1.72\pm 0.41$	$4.21\pm 0.61$	$0.537\pm 0.004$	$0.03\pm 0.04$	$0.07\pm 0.05$	$0.491\pm 0.012$	$0.22\pm 0.09$	$0.30\pm 0.18$	$0.710\pm 0.013$
	$\varepsilon=0.1$	$0.00$	$0.00\pm 0.00$	$0.663\pm 0.012$	$3.39\pm 0.52$	$6.02\pm 0.75$	$0.895\pm 0.023$	$0.10\pm 0.05$	$0.25\pm 0.13$	$0.830\pm 0.024$	$0.49\pm 0.17$	$0.36\pm 0.22$	$1.192\pm 0.035$
	$\varepsilon=0.3$	$0.09\pm 0.03$	$0.00\pm 0.00$	$1.343\pm 0.010$	$6.68\pm 1.04$	$9.95\pm 1.81$	$1.775\pm 0.044$	$0.12\pm 0.07$	$0.29\pm 0.15$	$1.547\pm 0.037$	$0.66\pm 0.17$	$0.30\pm 0.14$	$2.230\pm 0.061$
	$\varepsilon=0.5$	$0.10\pm 0.06$	$0.01\pm 0.00$	$1.761\pm 0.021$	$9.07\pm 1.25$	$12.24\pm 1.70$	$2.341\pm 0.064$	$0.18\pm 0.08$	$0.43\pm 0.15$	$2.054\pm 0.049$	$1.04\pm 0.15$	$0.43\pm 0.08$	$2.879\pm 0.085$

Appendix

Table 10: DenseNet-161 direct robustness on RIVAL-10 variants. PFR (%): prediction flip rate for

f_{\text{dn}}

(DenseNet-161) and

f_{\text{cbm}}

; DFS: average PCBM concept distance. Mean

\pm

std over 5 random subsets of 200 images.

		10 class; K=18		10 + 10 similar classes; K=18		10 + 10 dissimilar classes; K=18		10 + 10 dissimilar classes; K=27
$\sigma$	Model	SmAcc (%) $\uparrow$	$R\,\uparrow$	SmAcc (%) $\uparrow$	$R\,\uparrow$	SmAcc (%) $\uparrow$	$R\,\uparrow$	SmAcc (%) $\uparrow$	$R\,\uparrow$
0.01	$f_{\mathrm{dn}}$	$100.00$	$0.0235$	$99.60\pm 0.22$	$0.0231$	$100.00$	$0.0235$	$100.00$	$0.0234$
0.01	$f_{\mathrm{cbm}}$	$100.00$	$0.0235$	$99.60\pm 0.42$	$0.0227$	$100.00$	$0.0234$	$100.00$	$0.0235$
0.05	$f_{\mathrm{dn}}$	$100.00$	$0.1171$	$98.10\pm 0.42$	$0.1118\pm 0.0008$	$100.00$	$0.1170$	$99.70\pm 0.27$	$0.1149\pm 0.0005$
0.05	$f_{\mathrm{cbm}}$	$100.00$	$0.1173$	$95.60\pm 0.82$	$0.1112\pm 0.0009$	$100.00$	$0.1164$	$100.00$	$0.1170$
0.10	$f_{\mathrm{dn}}$	$100.00$	$0.2335\pm 0.0006$	$96.80\pm 0.76$	$0.2196\pm 0.0034$	$100.00$	$0.2330\pm 0.0008$	$99.60\pm 0.22$	$0.2271\pm 0.0015$

Appendix

Table 11: DenseNet-161 latent Gaussian randomized smoothing.

\tilde{z}=z+\eta

\eta\sim\mathcal{N}(0,\sigma^{2}I)

. SmAcc: smoothed accuracy;

R

: average certified radius over correctly classified and certified samples. Mean

\pm

std over five subsets of 200 images.

Attack- $\epsilon$	Attack Space	Model	Clean Acc.	Robust Acc.
$0.03$	latent $z$	$f_{std}$	100%	8%
		$\hat{f}_{std}$ (smoothed)	100%	16.5 %
		$f_{cbm}$	100%	23.5%
		$\hat{f}_{cbm}$ (smoothed)	100%	33.5 %
	concept $c$	$f_{std}$	100 %	44%
		$\hat{f}_{std}$ (smoothed)	88.5%	58.5 %

Appendix

Table 12: APGD-CE attack results on the CUB generated dataset based on backbone ResNet-50.

Perturbation	Strength	ResNet PFR (%)	PixPNet PFR (%)
Gaussian latent	$\sigma=0.05$	23.99	28.71
Gaussian latent	$\sigma=0.10$	39.21	44.03
Bernoulli semantic	$\rho=0.10$	33.97	36.58
Bernoulli semantic	$\rho=0.20$	54.31	57.40

Appendix

Table 13 : Prediction flip rate (PFR) for ResNet and PixPNet under the same generator-defined perturbations. The semantic perturbations are defined in the generator concept space; they are not prototype-space interventions.

	ResNet		PCBM		PixPNet
$\sigma$	SmAcc	ACR	SmAcc	ACR	SmAcc	ACR
0.01	96.20	0.0228	97.40	0.0215	95.00	0.0199
0.05	88.40	0.0747	84.90	0.0672	88.90	0.0553
0.10	81.70	0.1034	77.20	0.0918	82.90	0.0682
0.30	62.40	0.1263	57.10	0.1183	56.20	0.0839

Appendix

Table 14 : Gaussian randomized smoothing across standard, concept-based, and prototype-based classifiers. SmAcc denotes smoothed accuracy (%), with abstentions counted as incorrect. ACR denotes the average certified latent-space radius over correctly classified, non-abstaining samples.

$\rho$	ResNet SmAcc (%)	PixPNet SmAcc (%)
0.05	100.00	99.75
0.10	75.25	71.25
0.20	28.75	23.75
0.30	12.50	9.00

Appendix

Table 15 : Smoothed accuracy under Bernoulli generator-concept interventions on CUB-200-2011. The same generator-defined perturbation protocol is applied to ResNet and PixPNet.

	ACR-H		CA@1 (%)
$\rho$	ResNet	PixPNet	ResNet	PixPNet
0.05	0.0125	0.0150	1.25	1.50
0.10	0.0133	0.0211	1.00	1.50
0.20	0.0609	0.0526	1.50	1.25
0.30	0.2200	0.2222	1.75	1.50

Appendix

Table 16 : Certified robustness under Bernoulli generator-concept interventions on CUB-200-2011. ACR-H denotes the average certified Hamming radius over correctly classified, non-abstaining samples. CA@1 denotes the percentage of all evaluated images certified against at least one additional generator-concept flip.

Explore similar work

May 25, 2026cs.LG

When Interpretability Becomes a Liability: Adversarial Attacks on CBM Concept Layers

Concept Bottleneck Models (CBMs) have emerged as a cornerstone approach for interpretable machine learning, providing human-understandable intermediate representations through explicit concept activations. However, this interpretability fundamentally introduces a critical, previously unexplored attack surface: the concept bottleneck layer itself. We present a comprehensive, systematic study of concept-level adversarial vulnerabilities in CBMs, revealing that targeted, minimal perturbations operating on input pixels can induce catastrophic misclassification by manipulating semantic representations. We develop a rigorous theoretical framework to quantify concept-space robustness, establishing novel metrics that expose the vulnerability landscape of these architectures. Our extensive analysis on the CUB-200-2011 dataset demonstrates that standard CBMs exhibit severe susceptibility to concept-level manipulation. To address this critical weakness, we introduce SPECTRA (Semantic Perturbation-based Concept Training for Robustness against Attacks), a principled stability regularization defense. SPECTRA effectively hardens the semantic representation space, increasing the minimal perturbation norm required for a successful attack from 0.46 to over 4,200, rendering targeted concept manipulation computationally prohibitive. Furthermore, SPECTRA preserves baseline classification accuracy to within 2.2%. By establishing concept-level attacks as a fundamentally distinct threat model, this work opens a new research frontier at the intersection of interpretable machine learning and adversarial robustness.

Aditya Sridhar

Independent Researcher · Atlanta, GA, USA

Concept Bottleneck Models Interpretability

Ask PDF

May 7, 2026cs.LG

Hyperbolic Concept Bottleneck Models

Concept Bottleneck Models (CBMs) have become a popular approach to enable interpretability in neural networks by constraining classifier inputs to a set of human-understandable concepts. While effective, current models embed concepts in flat Euclidean space, treating them as independent, orthogonal dimensions. Concepts, however, are highly structured and organized in semantic hierarchies. To resolve this mismatch, we propose Hyperbolic Concept Bottleneck Models (HypCBM), a post-hoc framework that grounds the bottleneck in this structure by reformulating concept activation as asymmetric geometric containment in hyperbolic space. Rather than treating entailment cones as a pre-training penalty, we show they encode a natural test-time activation signal: the margin of inclusion within a concept's entailment cone yields sparse, hierarchy-aware activations without any additional supervision or learned modules. We further introduce an adaptive scaling law for hierarchically faithful interventions, propagating user corrections coherently through the concept tree. Empirically, HypCBM rivals post-hoc Euclidean models trained on 20

\times

more data in sparse regimes required for human interpretability, with stronger hierarchical consistency and improved robustness to input corruptions.

Daniel Uyterlinde, Swasti Shreya Mishra, Pascal Mettes

Informatics Institute, University of Amsterdam, The Netherlands

Dermoscopic Concept Bottleneck Models Concept Bottleneck Models Hyperbolic Learning Interpretability Bottlenecks Entailment

Ask PDF

Aug 7, 2026cs.AI

ReCBM: Uncertainty-Gated Relational Reasoning for Concept Bottleneck Models

Concept Bottleneck Models (CBMs) provide an interpretable framework by grounding predictions in human-understandable concepts, enabling semantic inspection and test-time intervention. Recent variants have improved CBMs through richer concept representations, uncertainty estimation, and dependency modeling. However, robust reasoning under unreliable concept states remains underexplored. Without such reasoning, misleading semantic evidence can propagate through the bottleneck, compromising both explanations and downstream predictions. To address this issue, we propose ReCBM, an uncertainty-gated relational reasoning framework for CBMs. ReCBM introduces semantically defined concept relations into the bottleneck and uses uncertainty to guide their refinement. By modeling co-occurrence, implication, and exclusion, ReCBM specifies how evidence is exchanged across concepts, while uncertainty modulates the contribution of each concept during this process. Experiments across diverse datasets showed that ReCBM improved concept and task recovery under missing and flipped concepts, supported uncertainty-aware intervention, and extracted compact task-relevant concept subsets without degrading downstream performance.

An Sui, Yuzhu Li, Fuping Wu +1

School of Data Science, Fudan University, Shanghai, China · Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China. · National Heart and Lung Institute, Imperial College London, London, United Kingdom.

Concept Bottleneck Models Uncertainty-Aware Classification Large Language Model Uncertainty Large Reasoning Models

Ask PDF