Intrinsic Concept Extraction Based on Compositional Interpretability

Hanyu Shi; Hong Tao; Guoheng Huang; Jianbin Jiang; Xuhang Chen; Chi-Man Pun; Shanhu Wang; Pan Pan

cs.CVMar 12, 2026

Intrinsic Concept Extraction Based on Compositional Interpretability

Authors: Hanyu Shi, Hong Tao, Guoheng Huang, Jianbin Jiang, Xuhang Chen, Chi-Man Pun, Shanhu Wang, Pan Pan

Organizations: Guangdong University of Technology · VIPSHOP · Huizhou University · University of Macau

Abstract

Unsupervised Concept Extraction aims to extract concepts from a single image; however, existing methods suffer from the inability to extract composable intrinsic concepts. To address this, this paper introduces a new task called Compositional and Interpretable Intrinsic Concept Extraction (CI-ICE). The CI-ICE task aims to leverage diffusion-based text-to-image models to extract composable object-level and attribute-level concepts from a single image, such that the original concept can be reconstructed through the combination of these concepts. To achieve this goal, we propose a method called HyperExpress, which addresses the CI-ICE task through two core aspects. Specifically, first, we propose a concept learning approach that leverages the inherent hierarchical modeling capability of hyperbolic space to achieve accurate concept disentanglement while preserving the hierarchical structure and relational dependencies among concepts; second, we introduce a concept-wise optimization method that maps the concept embedding space to maintain complex inter-concept relationships while ensuring concept composability. Our method demonstrates outstanding performance in extracting compositionally interpretable intrinsic concepts from a single image.

Figures & tables

Figure 1: The difference between unsupervised concept extraction (UCE) methods [ 41 , 5 , 8 ] and the composable and interpretable intrinsic concept extraction method . Object-level concept extraction method [ 8 , 14 ] can only extract object-level concepts and is unable to extract attribute-level concepts such as color and material. Although the intrinsic concept extraction method [ 5 ] can extract both object-level concepts and attribute-level concepts, the concepts it extracts are not sufficiently close to the original concepts, and it fails to consider the compositionality of the embedding space; thus, it has poor interpretability. In contrast, HyperExpress considers the relationships between concepts when learning them. As a result, the extracted concepts are more aligned with the objects in the image. Additionally, it imposes compositional constraints on the concept embedding space, thereby enabling concept extraction and combination capabilities that are understandable to humans.

Figure 2: The Proposed Method and Its Components . (a) The overall structure of HyperExpress : It addresses the CI-ICE task from two aspects: concept learning and concept-wise optimization. (b) Concept learning : It leverages triplet loss $\mathcal{L}_{triplet}$ and hyperbolic entailment loss $\mathcal{L}_{entail}$ to learn the hierarchical structure and associative relationships between object-level concepts and attribute-level concepts. (c) Concept-wise Optimization : It uses Horosphere Projector (HP) to constrain the concept space, thereby ensuring the compositionality of concepts.

Figure 3: Explanation of the proposed HEL module . Unlike the HCL module, the hyperbolic entailment loss is computed within the Lorentz model. If the object-level concept ( $v_{k}^{obj}$ ) and attribute-level concept ( $v_{k}^{color}$ and $v_{k}^{material}$ ) satisfy the condition in Eq. 9 , the entailment loss will be $0$ ; otherwise, the corresponding entailment loss will be calculated.

Figure 4: Horosphere Projection Module and an illustration of concept compositionality . If no constraints are applied to the concept embedding space, it may lead to incorrect composition of concepts. For instance, using

v_{k}^{att}+v_{k}^{obj}

alone cannot reconstruct the primary concept. To address this, we reproject the embedding space to ensure composability between concepts.

Figure 5: Comparison of Qualitative Results Between HyperExpress and ICE [ 5 ] . This comparison includes two processes: one is extracting object-level concepts and further attribute-level concepts from a single image, and the other is performing compositional reconstruction using the extracted concepts.

Method	$\text{SIM}^{I}$ (%)	$\text{SIM}^{C}$ (%)	$\text{ACC}^{1}$ (%)	$\text{ACC}^{3}$ (%)
Break-A-scene [ 2 ]	$0.627$	$0.773$	$0.174$	$0.282$
ConceptExpress [ 14 ]	$0.689$	$0.784$	$0.263$	$0.385$
AutoConcept [ 8 ]	$0.690$	$0.770$	$\underline{0.350}$	$\underline{0.520}$
ICE [ 5 ]	0.738	0.822	$0.325$	$0.518$
HyperExpress (Ours)	$\underline{0.699}$	$\underline{0.786}$	0.504	0.736

Table 1: Performance comparison on UCEBench [ 14 ] .

Method	$\text{SIM}^{T-T}_{object}$	$\text{SIM}^{T-T}_{material}$	$\text{SIM}^{T-T}_{color}$	$\text{SIM}^{T-V}_{object}$	$\text{SIM}^{T-V}_{material}$	$\text{SIM}^{V-T}_{color}$
ICE [ 5 ]	$0.249$	$0.101$	$0.093$	$0.264$	$0.208$	$0.215$
HyperExpress (Ours)	0.280	0.115	0.098	0.305	0.211	0.222

Table 2: Performance comparison on ICBench [ 5 ] .

HCL	HEL	HP	SIM I	SIM C	ACC 1	ACC 3
Module	Module	Module	(%)	(%)	(%)	(%)
✔	✗	✗	$0.625$	$0.769$	$0.326$	$0.509$
✔	✔	✗	$0.688$	$0.771$	$0.330$	$0.518$
✔	✗	✔	$0.621$	$0.765$	$0.348$	$0.522$
✔	✔	✔	0.699	0.786	0.504	0.736

Table 3: Ablation study on HyperExpress model components on the

D1

dataset [ 5 ] .

Compositional Representation Learning

Compositional representation learning studies how models encode reusable parts and relations to understand combinations they have not seen.

See all 83 papers

Concept-wise Attention for Fine-grained Concept Bottleneck Models
Apr 17, 2026Minghong Zhong, Guoshuai Zou, Kanghao Chen +2Concept Bottleneck Models Vision-Language Alignment
Sun Yat-sen University · Peng Cheng Laboratory · Key Laboratory of Machine Intelligence and Advanced Computing, MOE +1
Hierarchical Concept-based Interpretable Models
Feb 27, 2026Oscar Hill, Mateo Espinosa Zarlenga, Mateja JamnikInterpretable ML Concept-Based Explanations
University of Cambridge · University of Oxford
Beyond Heatmaps: Unsupervised Concept-Graph Reasoning for Interpretable Visual Explanation
Jul 1, 2026Md Mohasin Hossain, Anar Amirli, Robert Leist +2Concept Bottleneck Models Concept-Based Explanations
German Research Center for Artificial Intelligence · Saarland University, Saarbrücken, Germany · BEGO GmbH & Co. KG, Bremen, Germany +1

Apr 17, 2026J/K move · Enter open · S save

Concept-wise Attention for Fine-grained Concept Bottleneck Models

Minghong Zhong, Guoshuai Zou, Kanghao Chen, Dexia Chen+1

Sun Yat-sen University · Peng Cheng Laboratory · Key Laboratory of Machine Intelligence and Advanced Computing, MOE+1

Concept Bottleneck Models Vision-Language Alignment

Open paper AskPDF arXiv

Recently impressive performance has been achieved in Concept Bottleneck Models (CBM) by utilizing the image-text alignment learned by a large pre-trained vision-language model (i.e. CLIP). However, there exist two key limitations in concept modeling. Existing methods often suffer from pre-training biases, manifested as granularity misalignment or reliance on structural priors. Moreover, fine-tuning with Binary Cross-Entropy (BCE) loss treats each concept independently, which ignores mutual exclusivity among concepts, leading to suboptimal alignment. To address these limitations, we propose Concept-wise Attention for Fine-grained Concept Bottleneck Models (CoAt-CBM), a novel framework that achieves adaptive fine-grained image-concept alignment and high interpretability. Specifically, CoAt-CBM employs learnable concept-wise visual queries to adaptively obtain fine-grained concept-wise visual embeddings, which are then used to produce a concept score vector. Then, a novel concept contrastive optimization guides the model to handle the relative importance of the concept scores, enabling concept predictions to faithfully reflect the image content and improved alignment. Extensive experiments demonstrate that CoAt-CBM consistently outperforms state-of-the-art methods. The codes will be available upon acceptance.

Abstract

Compositional Representation Learning

Explore similar work

Concept-wise Attention for Fine-grained Concept Bottleneck Models

Hierarchical Concept-based Interpretable Models

Beyond Heatmaps: Unsupervised Concept-Graph Reasoning for Interpretable Visual Explanation