Disentangling Latent Embeddings with Sparse Linear Concept Cones (SLiCC)

Zhi Li; Hau Phan; Matthew Emigh; Austin J. Brockmeier

cs.CVAug 27, 2025

Disentangling Latent Embeddings with Sparse Linear Concept Cones (SLiCC)

Authors: Zhi Li, Hau Phan, Matthew Emigh, Austin J. Brockmeier

Organizations: Department of Electrical & Computer Engineering, University of Delaware, USA · Naval Surface Warfare Center, Panama City Division, Panama City, Florida, USA · Department of Computer & Information Sciences, University of Delaware, USA

Abstract

Pretrained vision models encode semantic information in latent embedding spaces that are useful for downstream tasks. We hypothesize that this information can be disentangled to separate the content of complex scenes by decomposing each embedding into multiple concept-specific components. We propose Sparse Linear Concept Cones (SLiCC), a supervised dictionary learning approach that estimates a linear synthesis model consisting of sparse, non-negative combinations of groups of dictionary atoms, whose group-wise activity is guided by multi-label information. Each concept-specific component is a non-negative combination of atoms associated with a label and lies in a corresponding concept cone. We introduce an alternating optimization algorithm and an autoencoder formulation of SLiCC. We apply SLiCC to three types of CLIP and DINOv2 embeddings across five datasets. For CLIP, we additionally exploit image-text alignment to enable a self-supervised SLiCC. We also introduce a fully unsupervised autoencoder that enforces group sparsity based on the norm of neuron activations. Quantitative results show that supervised SLiCC improves concept-filtered image retrieval over unfiltered embeddings, including retrieval of finer-grained information not provided during training. Beyond retrieval of individual concepts, the disentangled components enable concept deletion and recombination across images. We further explore conditional image generation using Image-to-Prompt, with qualitative results illustrating retrieval and generation from manipulated concept representations.

Figures & tables

Figure 1: (Left) Illustration of the disentanglement of a three-dimensional latent space when

S=3

. The gray surface denotes the unit sphere. The concept cones are the green surfaces, defined by 4, 2, and 4 atoms, respectively. The blue dot denotes the original image embedding

\mathbf{x}

. The three black dots denote three components

\mathbf{v}_{1},\mathbf{v}_{2},\mathbf{v}_{3}

. (Right) t-SNE visualization of disentangled concept cones by S-SLiCC-AE on ViT-L/14 MS COCO embeddings.

Panel of Figure 2 — Figure 2: t-SNE visualizations of embeddings. The gray samples in all three figures are original embeddings with no active “animal” concept. The colored samples are either original embeddings with active “animal” concept (a), or semantic components disentangled with different $d_{0}$ (b)(c).

Panel of Figure 3 — Figure 3: (Left) Top image retrieved by S-SLiCC-AE and UF using DINO embeddings. For the first query, the top two UF results omit the bicycle present in the query’s background. For the second query, UF overlooks the child at the edge of the frame. In the two cases, S-SLiCC-AE retrieves relevant images when filtered by the corresponding concepts. (Right) Qualitative results for concept deletion on ViT-L/14 MS COCO embeddings. Concepts are deleted at the general-label level, with the corresponding finer-grained sub-labels indicated in the figure.

CLIP-RN50	Double MNIST	MIRFlickr25K		MS COCO		Pascal VOC 2012		CIFAR100
CLIP-RN50	Gen.	Gen.	Sub.	Gen.	Sub.	Gen.	Sub.	Gen.	Sub.
UF	0.751	0.723	0.693	0.782	0.663	0.847	0.780	0.507	0.342
SpLiCE	0.391	0.646	0.561	0.707	0.584	0.792	0.758	0.389	0.222
F-SpLiCE	0.302	0.653	0.594	0.628	0.496	0.840	0.797	0.487	0.256
S-SVD	0.948	0.909	0.633	0.908	0.674	0.922	0.300	0.717	0.252
S-SLiCC	0.979	0.932	0.801	0.922	0.717	0.961	0.842	0.806	0.494

Table 1: Image retrieval quantitative results (mAP@20) for RN50, ViT-B/32, ViT-L/14, and DINO embeddings. Gen. and Sub. denote general-label and finer-grained sub-label retrieval, respectively. Autoencoder results are reported as mean

\pm

standard deviation across five runs. Within each type of retrieval for each dataset, the best performance is marked in bold and second best is underlined.

Panel of Figure 4 — Figure 4: (Left) Qualitative results for S-SLiCC-AE on ViT-L/14 MS COCO embeddings. The first column are queries. The second column are top images retrieved by UF. The third and the fourth columns are top images retrieved by S-SLiCC-AE. The last two columns are Image-to-Prompt generations. (Right) Qualitative results for unsupervised SLiCC on ViT-L/14 MS COCO embeddings. Top-3 word tokens whose embeddings have the highest cosine similarity with the concept component are presented.

Panel of Figure 5 — Figure 5: Qualitative results for concept combination on ViT-L/14 MS COCO embeddings. (Top) Disentanglement is performed by S-SLiCC-AE. Concepts are combined at the general-label level, with the corresponding finer-grained sub-labels indicated in the figure. (Bottom) Disentanglement is performed by unsupervised SLiCC-AE. Concepts are identified based on the word tokens presented in Figure 4 (Right).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Concept Cone	S-SLiCC-AE word caption	SS-SLiCC-AE word caption
person	granddaughter, girl, participants, woman, grandson	granddaughter, girl, grandma, woman, herself
vehicle	bike, bikes, boat, boats, motorcycle	truck, buses, automotive, vehicle, motorcycle
outdoor	benches, bench, nyc, brooklyn, lights	street, beach, lake, nature, nyc
animal	dogs, horses, horse, pups, cows	bear, dogs, pups, bears, horse
accessory	rainy, rain, rainfall, umbrellas, rains	luggage, umbrellas, suitcase, umbrella, rainy
sports	skate, baseball, tennis, skateboard, surf	baseball, tennis, surfers, skate, surf

Appendix

Table 2: Cone captions of MS COCO on CLIP ViT-L/14 embeddings. “Concept Cone” column shows the ground truth concept label. Words are ordered by the ascending reconstruction error.

Panel of Figure 6 — Figure 6: Relationship between the atoms across the concept dictionaries and their coefficients in supervised SLiCC. Dimensionality chosen by retrieval performance on validation set (see Section 4.2 ). (Left) the co-occurrence matrix of active atoms for images in the validation set. Each row is normalized separately so that the diagonal entries are 1. Within each group the co-occurrence rate is generally higher than between groups. (Right) The cosine similarity matrix of the atoms in the same dictionary. The within-group similarity ( $-0.056\pm 0.062$ ) is slightly lower than cross-group similarity ( $0.000\pm 0.134$ ).

Figure 7: The DPS sampling results from the cone with DINOv2 ViT-B/14 embeddings (MS COCO). Each columns contains five samples of one concept.

Concept pair		RN50			ViT-B/32			ViT-L/14			DINO
Concept 1	Concept 2	Weighted	B1	B2	Weighted	B1	B2	Weighted	B1	B2	Weighted	B1	B2
0	1	0.459	0.277	0.474	0.488	0.289	0.503	0.515	0.252	0.540	0.516	0.269	0.551
0	4	0.424	0.193	0.380	0.460	0.189	0.408	0.496	0.214	0.464	0.432	0.203	0.452
0	5	0.696	0.622	0.760	0.733	0.656	0.780	0.787	0.673	0.846	0.628	0.692	0.753
0	6	0.370	0.147	0.334	0.347	0.132	0.322	0.417	0.107	0.412	0.430	0.118	0.385
0	8	0.268	0.165	0.238	0.301	0.132	0.260	0.338	0.171	0.289	0.400	0.193	0.379

Appendix

Table 3: mAP@20 for concept combination retrieval on MS COCO using four type of embeddings. Weighted : optimal weighted combination. B1 : unfiltered embedding combination. B2 : average concept combination. Best scores within each concept pair and embedding model are shown in bold.

Disentangled Representation Learning

Disentangled representation learning separates distinct factors of variation so models can interpret, control, and generalize their predictions.

See all 234 papers

May 21, 2026

Conceptualizing Embeddings: Sparse Disentanglement for Vision-Language Models

Vision-language models learn powerful multimodal embeddings, yet their internal semantics remain opaque. While sparse autoencoders (SAEs) can extract interpretable features, they rely on expanding the representation dimension, which compromises the original geometry and introduces redundancy. We introduce CEDAR (Conceptual Embedding Disentanglement via Adaptive Rotation), a post-hoc method that reveals the compositional structure of pretrained embeddings without increasing dimensionality. By learning an invertible transformation with a top-

k

sparsity bottleneck, CEDAR concentrates semantic information into axis-aligned disentangled coordinates. In CLIP-like architecture, individual coordinates can be interpreted with textual concepts, while for generative models such as BLIP, they can be decoded into natural language descriptions. Experiments demonstrate that CEDAR achieves a competitive reconstruction-sparsity trade-off while producing explanations that are more interpretable and better aligned with human perception. Our results suggest that the apparent entanglement in vision-language representations can be resolved through a suitable change of basis, eliminating the need for overcomplete expansions.

Piotr Kubaty, Patryk Marszałek, Łukasz Struski +3

Faculty of Mathematics and Computer Science, Jagiellonian University, Kraków, Poland · Doctoral School of Exact and Natural Sciences, Jagiellonian University, Kraków, Poland

Disentangled Representation Learning Multimodal Disentangled Representation Learning VLM Interpretability Sparse Autoencoders Multimodal Embedding

Ask PDF

Jun 12, 2026

S $^2$ COPE: Self-Supervised Concept Discovery via Preference Learning

Current representation learning paradigms force a fundamental compromise: self-supervised methods scale to massive datasets but yield opaque features, whereas interpretable models remain bottlenecked by the need for dense human annotation. We introduce Self-Supervised Concept discOvery via Preference lEarning (\model), a label-free framework that resolves this dilemma. Instead of treating Vision-Large-Language Models (VLLMs) as static feature extractors, \model leverages them as active participants in a self-supervised preference optimization loop. By autonomously hypothesizing, validating, and reinforcing candidate visual attributes directly from raw imagery, our framework discovers novel, structured concepts without a single label. Extensive experiments across natural, medical, and physics domains demonstrate that \model successfully extracts domain-specific concepts where standard VLLMs often fail to generate. By amortizing concept discovery directly into the VLLM backbone through our self-supervised preference objective -- rather than relying on static generation and disjoint filtering -- we achieve up to a 24-point absolute improvement in downstream top-1 classification accuracy on unseen data. Our work suggest that interpretability can emerge through a model's autonomous interaction with incidental visual structures, without any human supervision.

Shilong Xiang, Zirui Zhang, Chengzhi Mao

Rutgers University

Vision-Language Models Self-Supervised Learning VLM Interpretability Preference Learning Unsupervised Visual Concept Discovery

Ask PDF

Aug 25, 2026

What Does Prompt Learning Change? -A Natural-Language Concept Analysis of Vision-Language Models

Prompt learning adapts vision-language models such as CLIP by optimizing continuous prompt vectors, but the learned prompts are difficult to interpret in natural language. We present PromptSpLiCE, a post-hoc method that expresses each class-conditioned text embedding as a sparse combination of concepts from a fixed natural-language dictionary. Using the same dictionary before and after prompt learning allows us to compare changes in their concept profiles. We evaluate PromptSpLiCE on CoOp, a representative prompt-learning method, across 11 image-classification datasets. The concept profiles change substantially: on average, only 1.6 of the initial top-10 concepts remain in the top 10 after learning. Across datasets, profile change is positively associated with accuracy gain. We also derive a local gradient expression that provides geometric intuition for why image-aligned concept directions distinct from the current prompt can have greater loss sensitivity.

Ryo Kamiya, Hiroshi Kera, Kazuhiko Kawamoto

Chiba University, Chiba, Japan

Prompt Learning Prompt Learning for VLMs VLM Adaptation VLM Interpretability Concept-Based Explanations

Ask PDF

Abstract

Disentangled Representation Learning

Explore similar work

Conceptualizing Embeddings: Sparse Disentanglement for Vision-Language Models

S2^22COPE: Self-Supervised Concept Discovery via Preference Learning

What Does Prompt Learning Change? -A Natural-Language Concept Analysis of Vision-Language Models

S $^2$ COPE: Self-Supervised Concept Discovery via Preference Learning