Concept Bottleneck Models (CBMs) are designed to make visual classification interpretable by expressing predictions through human-understandable concepts. Although interpretability is the central motivation for CBMs, they are still largely evaluated as predictive models by downstream classification accuracy, supplemented by isolated qualitative examples. This highlights a pressing need for quantitative measures, a challenge complicated by the infeasibility of ground-truth concept annotation at scale and the open nature of concept lists due to a lack of consensus. To fill this gap, we develop a multimodal large language model (MLLM) council that, given an image and its CBM explanation, produces an explanation quality score. To ground and validate the council, we first conduct a human study to establish a ground-truth reference for CBM explanation quality: for an image, annotators compare explanations from two of LF-CBM, VLG-CBM, and CBM-Suite and choose the more useful one, or mark them as equally good or equally bad, yielding 2700 judgments over 900 image-comparison items on CUB-200, ImageNet-100, and Places365. Against this human reference, our five-model council, consisting of open-weight MLLMs, recovers over 70% of strict human preference rankings, rising to 83% on items where human annotators unanimously agree. Building on this validated council, we introduce CBX-Bench, a public benchmark and leaderboard: authors of new CBMs can submit their model's explanations, and CBX-Bench scores them with the council and maintains dataset-level rankings of explanation quality. CBX-Bench thus provides a human-aligned, scalable evaluation of CBM explanations beyond accuracy and isolated qualitative examples. The benchmark is available at https://github.com/meric-karadag/cbx-bench.
Figures & tables
Figure 1: CBX-Bench motivation and contribution. Existing CBM evaluations lack a standardized, human-aligned metric for explanation quality. CBX-Bench addresses this gap by validating MLLM judges with human preferences and using the resulting council to produce normalized, model-agnostic CBM explanation rankings.
Figure 2: CBX-Bench methodology. The human study collects pairwise preferences over CBM explanations for the same image, collapses equality labels into a tie for agreement analysis, and uses human consensus to validate score-based MLLM judges. The selected score-sum council then scores each explanation independently and normalizes the aggregate score to define CBX-Bench rankings.
Dataset
Images
CBM pairs
Items
Human votes
CUB-200
100
3
300
900
ImageNet-100
100
3
300
900
Places365
100
3
300
900
Total
300
3
900
2700
Table 1: Human study design. Each dataset contributes 100 images and three pairwise CBM comparisons per image.
Consensus strength
Items
Outcome breakdown
No consensus
109 (12.1%)
–
Majority consensus (2/3)
508 (56.4%)
380 CBM preferences & 128 ties
Unanimous consensus (3/3)
283 (31.4%)
239 CBM preferences & 44 ties
Total
900 (100.0%)
619 CBM preferences & 172 ties
Table 2: Human consensus strength across 900 pairwise CBM explanation comparisons in our human-study.
Scale
Predictor
CUB200 Align.
IN100 Align.
Places365 Align.
Overall Align.
1–5
InternVL3.5-8B
37.0
37.5
48.4
40.7
Qwen3.5-4B
56.5
51.0
52.7
53.5
Qwen3.5-9B
49.1
49.0
60.2
52.5
Gemma4-E2B
38.0
41.7
38.7
39.4
Gemma4-E4B
48.1
53.1
46.2
49.2
Vote-majority council
43.5
49.0
53.8
48.5
Table 3: Instance-wise ranking recovery on CBM-only human consensus items. All human pairwise relations are strict preferences. Values are percentages; bold marks the best result within each scale and column.
Scale
Predictor
CUB200 Align.
IN100 Align.
Places365 Align.
Overall Align.
1–5
InternVL3.5-8B
41.4
44.6
50.5
45.6
Qwen3.5-4B
56.1
45.1
54.3
51.9
Qwen3.5-9B
48.5
49.2
57.6
51.9
Gemma4-E2B
43.9
42.1
42.4
42.8
Gemma4-E4B
51.5
50.8
50.0
50.7
Vote-majority council
51.5
50.8
59.5
54.1
Table 4: Instance-wise ranking recovery on human consensus items including ties. Values are percentages of recovered pairwise relations per image; bold marks the best result within each scale and column.
Unanimous Human Preference
CUB200
ImageNet100
Places365
Overall
a CBM
66.7
88.9
88.9
83.3
a CBM or tie
66.7
76.2
80.0
75.6
Table 5: Instance-wise ranking recovery for the 1–9 score-sum council on images where human annotators are unanimous (3/3 agree) on the relevant pairwise comparisons.
Table 6: CBX-Bench explanation-quality scores. Scores are computed with the 1–9 score-sum MLLM council and normalized from the five-judge sum range [5,45] to [0,1] . Bold marks the highest CBX-Bench score for each dataset. Test accuracy is the corresponding full-test-set classification accuracy
Judge
Disagreement (%)
Qwen3.5-9B
17.2
Qwen3.5-4B
17.7
Gemma4-E4B
24.8
InternVL3.5-8B
29.8
Gemma4-E2B
33.2
Table 7: Position-bias instability under direct pairwise MLLM prompting. Disagreement is the percentage of comparisons for which the preferred CBM label changes after swapping the positions of the two explanations in the prompt.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 3: User-study annotation interface for pairwise CBM explanation evaluation. Annotators see the input image, the ground-truth class, each CBM’s predicted class, and the top contributing concepts for each explanation. They then choose the more useful explanation, or mark the two explanations as equally good or equally bad.