Unsupervised Concept Extraction aims to extract concepts from a single image; however, existing methods suffer from the inability to extract composable intrinsic concepts. To address this, this paper introduces a new task called Compositional and Interpretable Intrinsic Concept Extraction (CI-ICE). The CI-ICE task aims to leverage diffusion-based text-to-image models to extract composable object-level and attribute-level concepts from a single image, such that the original concept can be reconstructed through the combination of these concepts. To achieve this goal, we propose a method called HyperExpress, which addresses the CI-ICE task through two core aspects. Specifically, first, we propose a concept learning approach that leverages the inherent hierarchical modeling capability of hyperbolic space to achieve accurate concept disentanglement while preserving the hierarchical structure and relational dependencies among concepts; second, we introduce a concept-wise optimization method that maps the concept embedding space to maintain complex inter-concept relationships while ensuring concept composability. Our method demonstrates outstanding performance in extracting compositionally interpretable intrinsic concepts from a single image.
Figures & tables
Figure 1: The difference between unsupervised concept extraction (UCE) methods [ 41 , 5 , 8 ] and the composable and interpretable intrinsic concept extraction method . Object-level concept extraction method [ 8 , 14 ] can only extract object-level concepts and is unable to extract attribute-level concepts such as color and material. Although the intrinsic concept extraction method [ 5 ] can extract both object-level concepts and attribute-level concepts, the concepts it extracts are not sufficiently close to the original concepts, and it fails to consider the compositionality of the embedding space; thus, it has poor interpretability. In contrast, HyperExpress considers the relationships between concepts when learning them. As a result, the extracted concepts are more aligned with the objects in the image. Additionally, it imposes compositional constraints on the concept embedding space, thereby enabling concept extraction and combination capabilities that are understandable to humans.
Figure 2: The Proposed Method and Its Components . (a) The overall structure of HyperExpress : It addresses the CI-ICE task from two aspects: concept learning and concept-wise optimization. (b) Concept learning : It leverages triplet loss Ltriplet and hyperbolic entailment loss Lentail to learn the hierarchical structure and associative relationships between object-level concepts and attribute-level concepts. (c) Concept-wise Optimization : It uses Horosphere Projector (HP) to constrain the concept space, thereby ensuring the compositionality of concepts.
Figure 3: Explanation of the proposed HEL module . Unlike the HCL module, the hyperbolic entailment loss is computed within the Lorentz model. If the object-level concept ( vkobj ) and attribute-level concept ( vkcolor and vkmaterial ) satisfy the condition in Eq. 9 , the entailment loss will be 0 ; otherwise, the corresponding entailment loss will be calculated.
Figure 4: Horosphere Projection Module and an illustration of concept compositionality . If no constraints are applied to the concept embedding space, it may lead to incorrect composition of concepts. For instance, using vkatt+vkobj alone cannot reconstruct the primary concept. To address this, we reproject the embedding space to ensure composability between concepts.
Figure 5: Comparison of Qualitative Results Between HyperExpress and ICE [ 5 ] . This comparison includes two processes: one is extracting object-level concepts and further attribute-level concepts from a single image, and the other is performing compositional reconstruction using the extracted concepts.
Method
SIMI (%)
SIMC (%)
ACC1 (%)
ACC3 (%)
Break-A-scene [ 2 ]
0.627
0.773
0.174
0.282
ConceptExpress [ 14 ]
0.689
0.784
0.263
0.385
AutoConcept [ 8 ]
0.690
0.770
0.350
0.520
ICE [ 5 ]
0.738
0.822
0.325
0.518
HyperExpress (Ours)
0.699
0.786
0.504
0.736
Table 1: Performance comparison on UCEBench [ 14 ] .
Method
SIMobjectT−T
SIMmaterialT−T
SIMcolorT−T
SIMobjectT−V
SIMmaterialT−V
SIMcolorV−T
ICE [ 5 ]
0.249
0.101
0.093
0.264
0.208
0.215
HyperExpress (Ours)
0.280
0.115
0.098
0.305
0.211
0.222
Table 2: Performance comparison on ICBench [ 5 ] .
HCL
HEL
HP
SIM I
SIM C
ACC 1
ACC 3
Module
Module
Module
(%)
(%)
(%)
(%)
✔
✗
✗
0.625
0.769
0.326
0.509
✔
✔
✗
0.688
0.771
0.330
0.518
✔
✗
✔
0.621
0.765
0.348
0.522
✔
✔
✔
0.699
0.786
0.504
0.736
Table 3: Ablation study on HyperExpress model components on the D1 dataset [ 5 ] .
Recently impressive performance has been achieved in Concept Bottleneck Models (CBM) by utilizing the image-text alignment learned by a large pre-trained vision-language model (i.e. CLIP). However, there exist two key limitations in concept modeling. Existing methods often suffer from pre-training biases, manifested as granularity misalignment or reliance on structural priors. Moreover, fine-tuning with Binary Cross-Entropy (BCE) loss treats each concept independently, which ignores mutual exclusivity among concepts, leading to suboptimal alignment. To address these limitations, we propose Concept-wise Attention for Fine-grained Concept Bottleneck Models (CoAt-CBM), a novel framework that achieves adaptive fine-grained image-concept alignment and high interpretability. Specifically, CoAt-CBM employs learnable concept-wise visual queries to adaptively obtain fine-grained concept-wise visual embeddings, which are then used to produce a concept score vector. Then, a novel concept contrastive optimization guides the model to handle the relative importance of the concept scores, enabling concept predictions to faithfully reflect the image content and improved alignment. Extensive experiments demonstrate that CoAt-CBM consistently outperforms state-of-the-art methods. The codes will be available upon acceptance.
Minghong Zhong, Guoshuai Zou, Kanghao Chen +2
Sun Yat-sen University · Peng Cheng Laboratory · Key Laboratory of Machine Intelligence and Advanced Computing, MOE +1
Modern deep neural networks remain challenging to interpret due to the opacity of their latent representations, impeding model understanding, debugging, and debiasing. Concept Embedding Models (CEMs) address this by mapping inputs to human-interpretable concept representations from which tasks can be predicted. Yet, CEMs fail to represent inter-concept relationships and require concept annotations at different granularities during training, limiting their applicability. In this paper, we introduce Hierarchical Concept Embedding Models (HiCEMs), a new family of CEMs that explicitly model concept relationships through hierarchical structures. To enable HiCEMs in real-world settings, we propose Concept Splitting, a method for automatically discovering finer-grained sub-concepts from a pretrained CEM's embedding space without requiring additional annotations. This allows HiCEMs to generate fine-grained explanations from limited concept labels, reducing annotation burdens. Our evaluation across multiple datasets, including a user study and experiments on PseudoKitchens, a newly proposed concept-based dataset of 3D kitchen renders, demonstrates that (1) Concept Splitting discovers human-interpretable sub-concepts absent during training that can be used to train highly accurate HiCEMs, and (2) HiCEMs enable powerful test-time concept interventions at different granularities, leading to improved task accuracy.
Oscar Hill, Mateo Espinosa Zarlenga, Mateja Jamnik
Concept Bottleneck Models (CBMs) provide an intrinsically interpretable alternative to post-hoc explanations. However, existing CBMs often rely on predefined concept vocabularies or supervised annotations, lack explicit concept grounding, and summarize each concept with a single image-level score -- discarding spatial recurrence and inter-concept dependencies. We propose a Graph-based Concept Bottleneck Model (G-CBM), an intrinsically interpretable framework that performs unsupervised concept discovery via Non-negative Matrix Factorization (NMF) and represents the discovered concepts as nodes in a per-image concept-graph representation. G-CBM matches region-level features to these concept nodes -- providing concept grounding and capturing concept recurrence across the image -- and applies a tunable concept filtering thresholdτ to suppress weak region-level features. A Graph Attention Network (GAT) then performs concept-level reasoning by modeling nonlinear dependencies across nodes. Across ImageNet, HAM10000, PH2, and Derm7pt, G-CBM achieves an average relative AUC improvement of 3.7% over a ResNet-50 baseline. Concept filtering frequently improves predictive performance while inducing selective concept use, achieving peak AUC of 0.96 on PH2 with only 2 of 10 concepts and 0.92 on HAM10000 with 3.8 of 9 concepts. On dermoscopy benchmarks, G-CBM is competitive with supervised approaches requiring external annotations. Deletion/insertion analyses with random ablation controls show that the learned concept ranking faithfully reflects model predictions.
Md Mohasin Hossain, Anar Amirli, Robert Leist +2
German Research Center for Artificial Intelligence · Saarland University, Saarbrücken, Germany · BEGO GmbH & Co. KG, Bremen, Germany +1