Concept Bottleneck Models (CBMs) are designed to make visual classification interpretable by expressing predictions through human-understandable concepts. Although interpretability is the central motivation for CBMs, they are still largely evaluated as predictive models by downstream classification accuracy, supplemented by isolated qualitative examples. This highlights a pressing need for quantitative measures, a challenge complicated by the infeasibility of ground-truth concept annotation at scale and the open nature of concept lists due to a lack of consensus. To fill this gap, we develop a multimodal large language model (MLLM) council that, given an image and its CBM explanation, produces an explanation quality score. To ground and validate the council, we first conduct a human study to establish a ground-truth reference for CBM explanation quality: for an image, annotators compare explanations from two of LF-CBM, VLG-CBM, and CBM-Suite and choose the more useful one, or mark them as equally good or equally bad, yielding 2700 judgments over 900 image-comparison items on CUB-200, ImageNet-100, and Places365. Against this human reference, our five-model council, consisting of open-weight MLLMs, recovers over 70% of strict human preference rankings, rising to 83% on items where human annotators unanimously agree. Building on this validated council, we introduce CBX-Bench, a public benchmark and leaderboard: authors of new CBMs can submit their model's explanations, and CBX-Bench scores them with the council and maintains dataset-level rankings of explanation quality. CBX-Bench thus provides a human-aligned, scalable evaluation of CBM explanations beyond accuracy and isolated qualitative examples. The benchmark is available at https://github.com/meric-karadag/cbx-bench.
Figures & tables
Figure 1: CBX-Bench motivation and contribution. Existing CBM evaluations lack a standardized, human-aligned metric for explanation quality. CBX-Bench addresses this gap by validating MLLM judges with human preferences and using the resulting council to produce normalized, model-agnostic CBM explanation rankings.
Figure 2: CBX-Bench methodology. The human study collects pairwise preferences over CBM explanations for the same image, collapses equality labels into a tie for agreement analysis, and uses human consensus to validate score-based MLLM judges. The selected score-sum council then scores each explanation independently and normalizes the aggregate score to define CBX-Bench rankings.
Dataset
Images
CBM pairs
Items
Human votes
CUB-200
100
3
300
900
ImageNet-100
100
3
300
900
Places365
100
3
300
900
Total
300
3
900
2700
Table 1: Human study design. Each dataset contributes 100 images and three pairwise CBM comparisons per image.
Consensus strength
Items
Outcome breakdown
No consensus
109 (12.1%)
–
Majority consensus (2/3)
508 (56.4%)
380 CBM preferences & 128 ties
Unanimous consensus (3/3)
283 (31.4%)
239 CBM preferences & 44 ties
Total
900 (100.0%)
619 CBM preferences & 172 ties
Table 2: Human consensus strength across 900 pairwise CBM explanation comparisons in our human-study.
Scale
Predictor
CUB200 Align.
IN100 Align.
Places365 Align.
Overall Align.
1–5
InternVL3.5-8B
37.0
37.5
48.4
40.7
Qwen3.5-4B
56.5
51.0
52.7
53.5
Qwen3.5-9B
49.1
49.0
60.2
52.5
Gemma4-E2B
38.0
41.7
38.7
39.4
Gemma4-E4B
48.1
53.1
46.2
49.2
Vote-majority council
43.5
49.0
53.8
48.5
Table 3: Instance-wise ranking recovery on CBM-only human consensus items. All human pairwise relations are strict preferences. Values are percentages; bold marks the best result within each scale and column.
Scale
Predictor
CUB200 Align.
IN100 Align.
Places365 Align.
Overall Align.
1–5
InternVL3.5-8B
41.4
44.6
50.5
45.6
Qwen3.5-4B
56.1
45.1
54.3
51.9
Qwen3.5-9B
48.5
49.2
57.6
51.9
Gemma4-E2B
43.9
42.1
42.4
42.8
Gemma4-E4B
51.5
50.8
50.0
50.7
Vote-majority council
51.5
50.8
59.5
54.1
Table 4: Instance-wise ranking recovery on human consensus items including ties. Values are percentages of recovered pairwise relations per image; bold marks the best result within each scale and column.
Unanimous Human Preference
CUB200
ImageNet100
Places365
Overall
a CBM
66.7
88.9
88.9
83.3
a CBM or tie
66.7
76.2
80.0
75.6
Table 5: Instance-wise ranking recovery for the 1–9 score-sum council on images where human annotators are unanimous (3/3 agree) on the relevant pairwise comparisons.
Table 6: CBX-Bench explanation-quality scores. Scores are computed with the 1–9 score-sum MLLM council and normalized from the five-judge sum range [5,45] to [0,1] . Bold marks the highest CBX-Bench score for each dataset. Test accuracy is the corresponding full-test-set classification accuracy
Judge
Disagreement (%)
Qwen3.5-9B
17.2
Qwen3.5-4B
17.7
Gemma4-E4B
24.8
InternVL3.5-8B
29.8
Gemma4-E2B
33.2
Table 7: Position-bias instability under direct pairwise MLLM prompting. Disagreement is the percentage of comparisons for which the preferred CBM label changes after swapping the positions of the two explanations in the prompt.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 3: User-study annotation interface for pairwise CBM explanation evaluation. Annotators see the input image, the ground-truth class, each CBM’s predicted class, and the top contributing concepts for each explanation. They then choose the more useful explanation, or mark the two explanations as equally good or equally bad.
Concept Bottleneck Models (CBMs) aim for ante-hoc interpretability by learning a bottleneck layer that predicts interpretable concepts before the decision. State-of-the-art approaches typically select which concepts to learn via human specification, open knowledge graphs, prompting an LLM, or using general CLIP concepts. However, concepts defined a-priori may not have sufficient predictive power for the task or even be learnable from the available data. As a result, these CBMs often significantly trail their black-box counterpart when controlling for information leakage. To address this, we introduce a novel CBM pipeline named Mechanistic CBM (M-CBM), which builds the bottleneck directly from a black-box model's own learned concepts. These concepts are extracted via Sparse Autoencoders (SAEs) and subsequently named and annotated on a selected subset of images using a Multimodal LLM. For fair comparison and leakage control, we also introduce the Number of Contributing Concepts (NCC), a decision-level sparsity metric that extends the recently proposed NEC metric. Across diverse datasets, we show that M-CBMs consistently surpass prior CBMs at matched sparsity, while improving concept predictions and providing concise explanations. Our code is available at https://github.com/Antonio-Dee/M-CBM.
Antonio De Santis, Schrasing Tong, Marco Brambilla +1
Concept bottleneck models (CBMs) are neural classifiers that allow to explain their decisions via high-level concepts, potentially enabling understanding, steering and debugging. However, their explanations are often derived heuristically. Building on formal explainability, we argue they should also be faithful, i.e., not misreport which concepts actually matter. We show that, for widespread CBM architectures, including recent VLM-based variants, faithful explanations must include all concepts in the bottleneck, compromising interpretability when this is large. This result applies to both heuristic and faithful-by-construction formal explanations. To encourage the existence of compact faithful explanations, we suggest i) modeling concepts probabilistically as binary or categorical random variables (rather than logits), and ii) employing per-concept training-time sparsification via group lasso (rather than regular elastic net). We also extend algorithms from formal explainability to CBMs, and show they outperform natural heuristics in terms of guarantees and explanation size. Overall, our work warns against naive interpretability claims and provides formal conditions and practical strategies for ensuring CBMs are as interpretable as advertised.
Stefano Teso, Emanuele Marconato, Steve Azzolin +1
CIMeC & DISI University of Trento · Department of Mathematical Sciences University of Copenhagen · DISI University of Trento +1
Concept Bottleneck Models (CBMs) provide an intrinsically interpretable alternative to post-hoc explanations. However, existing CBMs often rely on predefined concept vocabularies or supervised annotations, lack explicit concept grounding, and summarize each concept with a single image-level score -- discarding spatial recurrence and inter-concept dependencies. We propose a Graph-based Concept Bottleneck Model (G-CBM), an intrinsically interpretable framework that performs unsupervised concept discovery via Non-negative Matrix Factorization (NMF) and represents the discovered concepts as nodes in a per-image concept-graph representation. G-CBM matches region-level features to these concept nodes -- providing concept grounding and capturing concept recurrence across the image -- and applies a tunable concept filtering thresholdτ to suppress weak region-level features. A Graph Attention Network (GAT) then performs concept-level reasoning by modeling nonlinear dependencies across nodes. Across ImageNet, HAM10000, PH2, and Derm7pt, G-CBM achieves an average relative AUC improvement of 3.7% over a ResNet-50 baseline. Concept filtering frequently improves predictive performance while inducing selective concept use, achieving peak AUC of 0.96 on PH2 with only 2 of 10 concepts and 0.92 on HAM10000 with 3.8 of 9 concepts. On dermoscopy benchmarks, G-CBM is competitive with supervised approaches requiring external annotations. Deletion/insertion analyses with random ablation controls show that the learned concept ranking faithfully reflects model predictions.
Md Mohasin Hossain, Anar Amirli, Robert Leist +2
German Research Center for Artificial Intelligence · Saarland University, Saarbrücken, Germany · BEGO GmbH & Co. KG, Bremen, Germany +1