Disentangling Latent Embeddings with Sparse Linear Concept Cones (SLiCC)

Zhi Li; Hau Phan; Matthew Emigh; Austin J. Brockmeier

cs.CVAug 27, 2025

Disentangling Latent Embeddings with Sparse Linear Concept Cones (SLiCC)

Authors: Zhi Li, Hau Phan, Matthew Emigh, Austin J. Brockmeier

Organizations: Department of Electrical & Computer Engineering, University of Delaware, USA · Naval Surface Warfare Center, Panama City Division, Panama City, Florida, USA · Department of Computer & Information Sciences, University of Delaware, USA

Abstract

Pretrained vision models encode semantic information in latent embedding spaces that are useful for downstream tasks. We hypothesize that this information can be disentangled to separate the content of complex scenes by decomposing each embedding into multiple concept-specific components. We propose Sparse Linear Concept Cones (SLiCC), a supervised dictionary learning approach that estimates a linear synthesis model consisting of sparse, non-negative combinations of groups of dictionary atoms, whose group-wise activity is guided by multi-label information. Each concept-specific component is a non-negative combination of atoms associated with a label and lies in a corresponding concept cone. We introduce an alternating optimization algorithm and an autoencoder formulation of SLiCC. We apply SLiCC to three types of CLIP and DINOv2 embeddings across five datasets. For CLIP, we additionally exploit image-text alignment to enable a self-supervised SLiCC. We also introduce a fully unsupervised autoencoder that enforces group sparsity based on the norm of neuron activations. Quantitative results show that supervised SLiCC improves concept-filtered image retrieval over unfiltered embeddings, including retrieval of finer-grained information not provided during training. Beyond retrieval of individual concepts, the disentangled components enable concept deletion and recombination across images. We further explore conditional image generation using Image-to-Prompt, with qualitative results illustrating retrieval and generation from manipulated concept representations.

Figures & tables

Figure 1: (Left) Illustration of the disentanglement of a three-dimensional latent space when

S=3

. The gray surface denotes the unit sphere. The concept cones are the green surfaces, defined by 4, 2, and 4 atoms, respectively. The blue dot denotes the original image embedding

\mathbf{x}

. The three black dots denote three components

\mathbf{v}_{1},\mathbf{v}_{2},\mathbf{v}_{3}

. (Right) t-SNE visualization of disentangled concept cones by S-SLiCC-AE on ViT-L/14 MS COCO embeddings.

Panel of Figure 2 — Figure 2: t-SNE visualizations of embeddings. The gray samples in all three figures are original embeddings with no active “animal” concept. The colored samples are either original embeddings with active “animal” concept (a), or semantic components disentangled with different $d_{0}$ (b)(c).

Panel of Figure 3 — Figure 3: (Left) Top image retrieved by S-SLiCC-AE and UF using DINO embeddings. For the first query, the top two UF results omit the bicycle present in the query’s background. For the second query, UF overlooks the child at the edge of the frame. In the two cases, S-SLiCC-AE retrieves relevant images when filtered by the corresponding concepts. (Right) Qualitative results for concept deletion on ViT-L/14 MS COCO embeddings. Concepts are deleted at the general-label level, with the corresponding finer-grained sub-labels indicated in the figure.

CLIP-RN50	Double MNIST	MIRFlickr25K		MS COCO		Pascal VOC 2012		CIFAR100
CLIP-RN50	Gen.	Gen.	Sub.	Gen.	Sub.	Gen.	Sub.	Gen.	Sub.
UF	0.751	0.723	0.693	0.782	0.663	0.847	0.780	0.507	0.342
SpLiCE	0.391	0.646	0.561	0.707	0.584	0.792	0.758	0.389	0.222
F-SpLiCE	0.302	0.653	0.594	0.628	0.496	0.840	0.797	0.487	0.256
S-SVD	0.948	0.909	0.633	0.908	0.674	0.922	0.300	0.717	0.252
S-SLiCC	0.979	0.932	0.801	0.922	0.717	0.961	0.842	0.806	0.494

Table 1: Image retrieval quantitative results (mAP@20) for RN50, ViT-B/32, ViT-L/14, and DINO embeddings. Gen. and Sub. denote general-label and finer-grained sub-label retrieval, respectively. Autoencoder results are reported as mean

\pm

standard deviation across five runs. Within each type of retrieval for each dataset, the best performance is marked in bold and second best is underlined.

Panel of Figure 4 — Figure 4: (Left) Qualitative results for S-SLiCC-AE on ViT-L/14 MS COCO embeddings. The first column are queries. The second column are top images retrieved by UF. The third and the fourth columns are top images retrieved by S-SLiCC-AE. The last two columns are Image-to-Prompt generations. (Right) Qualitative results for unsupervised SLiCC on ViT-L/14 MS COCO embeddings. Top-3 word tokens whose embeddings have the highest cosine similarity with the concept component are presented.

Panel of Figure 5 — Figure 5: Qualitative results for concept combination on ViT-L/14 MS COCO embeddings. (Top) Disentanglement is performed by S-SLiCC-AE. Concepts are combined at the general-label level, with the corresponding finer-grained sub-labels indicated in the figure. (Bottom) Disentanglement is performed by unsupervised SLiCC-AE. Concepts are identified based on the word tokens presented in Figure 4 (Right).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Concept Cone	S-SLiCC-AE word caption	SS-SLiCC-AE word caption
person	granddaughter, girl, participants, woman, grandson	granddaughter, girl, grandma, woman, herself
vehicle	bike, bikes, boat, boats, motorcycle	truck, buses, automotive, vehicle, motorcycle
outdoor	benches, bench, nyc, brooklyn, lights	street, beach, lake, nature, nyc
animal	dogs, horses, horse, pups, cows	bear, dogs, pups, bears, horse
accessory	rainy, rain, rainfall, umbrellas, rains	luggage, umbrellas, suitcase, umbrella, rainy
sports	skate, baseball, tennis, skateboard, surf	baseball, tennis, surfers, skate, surf

Appendix

Table 2: Cone captions of MS COCO on CLIP ViT-L/14 embeddings. “Concept Cone” column shows the ground truth concept label. Words are ordered by the ascending reconstruction error.

Panel of Figure 6 — Figure 6: Relationship between the atoms across the concept dictionaries and their coefficients in supervised SLiCC. Dimensionality chosen by retrieval performance on validation set (see Section 4.2 ). (Left) the co-occurrence matrix of active atoms for images in the validation set. Each row is normalized separately so that the diagonal entries are 1. Within each group the co-occurrence rate is generally higher than between groups. (Right) The cosine similarity matrix of the atoms in the same dictionary. The within-group similarity ( $-0.056\pm 0.062$ ) is slightly lower than cross-group similarity ( $0.000\pm 0.134$ ).

Figure 7: The DPS sampling results from the cone with DINOv2 ViT-B/14 embeddings (MS COCO). Each columns contains five samples of one concept.

Concept pair		RN50			ViT-B/32			ViT-L/14			DINO
Concept 1	Concept 2	Weighted	B1	B2	Weighted	B1	B2	Weighted	B1	B2	Weighted	B1	B2
0	1	0.459	0.277	0.474	0.488	0.289	0.503	0.515	0.252	0.540	0.516	0.269	0.551
0	4	0.424	0.193	0.380	0.460	0.189	0.408	0.496	0.214	0.464	0.432	0.203	0.452
0	5	0.696	0.622	0.760	0.733	0.656	0.780	0.787	0.673	0.846	0.628	0.692	0.753
0	6	0.370	0.147	0.334	0.347	0.132	0.322	0.417	0.107	0.412	0.430	0.118	0.385
0	8	0.268	0.165	0.238	0.301	0.132	0.260	0.338	0.171	0.289	0.400	0.193	0.379

Appendix

Table 3: mAP@20 for concept combination retrieval on MS COCO using four type of embeddings. Weighted : optimal weighted combination. B1 : unfiltered embedding combination. B2 : average concept combination. Best scores within each concept pair and embedding model are shown in bold.

Disentangled Representation Learning

Disentangled representation learning separates distinct factors of variation so models can interpret, control, and generalize their predictions.

See all 234 papers

Conceptualizing Embeddings: Sparse Disentanglement for Vision-Language Models
May 21, 2026Piotr Kubaty, Patryk Marszałek, Łukasz Struski +3Disentangled Representation Learning Multimodal Disentangled Representation Learning
Faculty of Mathematics and Computer Science, Jagiellonian University, Kraków, Poland · Doctoral School of Exact and Natural Sciences, Jagiellonian University, Kraków, Poland
S $^2$ COPE: Self-Supervised Concept Discovery via Preference Learning
Jun 12, 2026Shilong Xiang, Zirui Zhang, Chengzhi MaoVision-Language Models Self-Supervised Learning
Rutgers University
What Does Prompt Learning Change? -A Natural-Language Concept Analysis of Vision-Language Models
Aug 25, 2026Ryo Kamiya, Hiroshi Kera, Kazuhiko KawamotoPrompt Learning Prompt Learning for VLMs
Chiba University, Chiba, Japan

May 21, 2026J/K move · Enter open · S save

Conceptualizing Embeddings: Sparse Disentanglement for Vision-Language Models

Piotr Kubaty, Patryk Marszałek, Łukasz Struski, Adam Wróbel+2

Faculty of Mathematics and Computer Science, Jagiellonian University, Kraków, Poland · Doctoral School of Exact and Natural Sciences, Jagiellonian University, Kraków, Poland

Disentangled Representation Learning Multimodal Disentangled Representation Learning VLM Interpretability Sparse Autoencoders Multimodal Embedding

Open paper AskPDF arXiv

Vision-language models learn powerful multimodal embeddings, yet their internal semantics remain opaque. While sparse autoencoders (SAEs) can extract interpretable features, they rely on expanding the representation dimension, which compromises the original geometry and introduces redundancy. We introduce CEDAR (Conceptual Embedding Disentanglement via Adaptive Rotation), a post-hoc method that reveals the compositional structure of pretrained embeddings without increasing dimensionality. By learning an invertible transformation with a top-

k

sparsity bottleneck, CEDAR concentrates semantic information into axis-aligned disentangled coordinates. In CLIP-like architecture, individual coordinates can be interpreted with textual concepts, while for generative models such as BLIP, they can be decoded into natural language descriptions. Experiments demonstrate that CEDAR achieves a competitive reconstruction-sparsity trade-off while producing explanations that are more interpretable and better aligned with human perception. Our results suggest that the apparent entanglement in vision-language representations can be resolved through a suitable change of basis, eliminating the need for overcomplete expansions.

Abstract

Disentangled Representation Learning

Explore similar work

Conceptualizing Embeddings: Sparse Disentanglement for Vision-Language Models

S2^22COPE: Self-Supervised Concept Discovery via Preference Learning

What Does Prompt Learning Change? -A Natural-Language Concept Analysis of Vision-Language Models

S $^2$ COPE: Self-Supervised Concept Discovery via Preference Learning