SCoCCA: Multi-modal Sparse Concept Decomposition via Canonical Correlation Analysis

Ehud Gordon; Meir Yossef Levi; Guy Gilboa

cs.CVMar 14, 2026

SCoCCA: Multi-modal Sparse Concept Decomposition via Canonical Correlation Analysis

Authors: Ehud Gordon, Meir Yossef Levi, Guy Gilboa

Organizations: Viterbi Faculty of Electrical and Computer Engineering, Technion – Israel Institute of Technology, Haifa, Israel

Abstract

Interpreting the internal reasoning of vision-language models is essential for deploying AI in safety-critical domains. Concept-based explainability provides a human-aligned lens by representing a model's behavior through semantically meaningful components. However, existing methods are largely restricted to images and overlook the cross-modal interactions. Text-image embeddings, such as those produced by CLIP, suffer from a modality gap, where visual and textual features follow distinct distributions, limiting interpretability. Canonical Correlation Analysis (CCA) offers a principled way to align features from different distributions, but has not been leveraged for multi-modal concept-level analysis. We show that the objectives of CCA and InfoNCE are closely related, such that optimizing CCA implicitly optimizes InfoNCE, providing a simple, training-free mechanism to enhance cross-modal alignment without affecting the pre-trained InfoNCE objective. Motivated by this observation, we couple concept-based explainability with CCA, introducing Concept CCA (CoCCA), a framework that aligns cross-modal embeddings while enabling interpretable concept decomposition. We further extend it and propose Sparse Concept CCA (SCoCCA), which enforces sparsity to produce more disentangled and discriminative concepts, facilitating improved activation, ablation, and semantic manipulation. Our approach generalizes concept-based explanations to multi-modal embeddings and achieves state-of-the-art performance in concept discovery, evidenced by reconstruction and manipulation tasks such as concept ablation.

Figures & tables

Figure 1: Concept Swapping. Beyond explainability, concept decomposition enables controllable manipulation. Using SCoCCA, an embedding can be decomposed into interpretable concepts (e.g., cube and cylinder), their magnitudes swapped, and the modified embedding recomposed to synthesize an image reflecting the swapped concepts.

Figure 2: Method Overview. In the Concept Discovery phase, text and image embeddings are aligned via Canonical Correlation Analysis (CCA) to form a shared latent space. The Hungarian algorithm establishes a one-to-one correspondence between each concept vector and its most relevant item in the concept bank. In the Concept Decomposition phase, new embedding is decomposed into concepts by solving a Lasso optimization using the matrix $\mathbf{C}$ and the discovered associations.

Category	Metric	Dual-Modality		Single-Modality				Baseline
Category	Metric	SCoCCA (Ours)	SpLiCE [ 6 ]	TCAV [ 21 ]	Varimax [ 60 ]	NMF	K-Means	CLIP [ 41 ]
Purity and Editing	Ablation prob. drop ( $\uparrow$ )	0.87	0.20	0.93	0.81	0.01	0.47	-
	Target prob gain ( $\uparrow$ )	0.95	0.36	0.10	0.75	0.02	-0.09	-
	Img residual cosine ( $\uparrow$ )	0.76	0.70	0.67	0.67	0.63	0.74	-
	Zero-shot accuracy ( $\uparrow$ )	0.74	0.48	0.62	0.65	0.12	0.53	0.75
	Zero-shot precision@5 ( $\uparrow$ )	0.85	0.61	0.59	0.75	0.29	0.70	0.85

Table 1: Comprehensive Performance Comparison. Results comparing concept decomposition methods on subset of 500 random classes from ImageNet [ 8 ] , grouped into dual-modality and single-modality approaches, and evaluated across a wide range of metrics. Best results are shown in bold , and second-best are underlined . SCoCCA achieves state-of-the-art or on-par performance in Reconstruction, as well as in Purity & Editing. Rightmost column shows CLIP zero-shot performance. Note the SCoCCA obtains CLIP-level performance in accuracy.

Figure 3: Concept Retrieval Generalization. Retrieval of the top four MSCOCO [ 32 ] images with the highest activation for the concepts Microwave and Traffic Light , where the concept bank was calibrated on ImageNet only (images are shown in descending order). While SCoCCA retrieves images with a clear presence of the concept, other methods in some cases return images with little or no evidence of the desired concept.

Figure 4:

\lambda

ablation for SCoCCA on ImageNet-500. Top : Zero-shot accuracy on ImageNet-500 as a function the

\lambda

used when solving the sparse CoCCA coding objective in ( 15 ). Bottom :

\|w\|_{0}

normalized by

k

, as function of

\lambda

. In both plots,

\lambda=0

corresponds to the CoCCA baseline without the sparsity penalty. The curves shows that forcing sparsity yields substantially higher zero-shot accuracy than the non-sparse baseline.

Figure 5: ablation of

k

hyperparameter Zero-Shot Accuracy performance on test set of ImageNet-500, as a function of

k

, the number of concepts computed, for the SCoCCA method. As more concepts allows better separation, this is a monotone increasing function. However the rate of steepness decreases, as additional concepts yield less of a return in classification.

Category	Metric	Dual-Modality		Single-Modality				Baseline
Category	Metric	SCoCCA (Ours)	SpLiCE [ 6 ]	TCAV [ 21 ]	Varimax [ 60 ]	NMF	K-Means	CLIP [ 41 ]
Purity and Editing	Ablation prob. drop ( $\uparrow$ )	0.77	0.12	0.79	0.73	0.00	0.37	-
	Target prob gain ( $\uparrow$ )	0.72	0.11	0.02	0.61	-0.19	-0.39	-
	Img residual cosine ( $\uparrow$ )	0.70	0.63	0.58	0.57	0.49	0.68	-
	Zero-shot accuracy ( $\uparrow$ )	0.60	0.37	0.48	0.52	0.10	0.40	0.63
	Zero-shot precision@5 ( $\uparrow$ )	0.81	0.58	0.71	0.74	0.25	0.68	0.82

Table 2: Comprehensive Performance Comparison. Results comparing concept decomposition methods on subset of 500 random classes from ImageNet [ 8 ] , similar to Table 1 , but for CLIP model B-32 . Best results are shown in bold , and second-best are underlined . We observe similar trends like Table 1 . As expected from a smaller model, for same number of

k=500

, reconstruction metrics improve, whereas Purity metrics decrease.

Canonical Correlation Analysis

See all 10 papers

Conceptualizing Embeddings: Sparse Disentanglement for Vision-Language Models
May 21, 2026Piotr Kubaty, Patryk Marszałek, Łukasz Struski +3Disentangled Representation Learning Multimodal Disentangled Representation Learning
Faculty of Mathematics and Computer Science, Jagiellonian University, Kraków, Poland · Doctoral School of Exact and Natural Sciences, Jagiellonian University, Kraków, Poland
Extraction and Analysis of Multimodal Concepts in Vision Language Models through Sparse Autoencoders
Jun 19, 2026Sergio Lanza, Jae Hee Lee, Stefan WermterVision-Language Models VLM Interpretability
Knowledge Technology, Department of Informatics, University of Hamburg, Hamburg, Germany
Bridging Vision and Language Concepts through Optimal Transport Semantic Flow
Jun 25, 2026Chenyang Zhang, Anqi Dong, Guangming Zhu +4Concept Bottleneck Models VLM Interpretability
School of Computer Science and Technology, Xidian University, Xi’an, China · KTH Royal Institute of Technology, Stockholm, Sweden

May 21, 2026J/K move · Enter open · S save

Conceptualizing Embeddings: Sparse Disentanglement for Vision-Language Models

Piotr Kubaty, Patryk Marszałek, Łukasz Struski, Adam Wróbel+2

Faculty of Mathematics and Computer Science, Jagiellonian University, Kraków, Poland · Doctoral School of Exact and Natural Sciences, Jagiellonian University, Kraków, Poland

Disentangled Representation Learning Multimodal Disentangled Representation Learning VLM Interpretability Sparse Autoencoders Multimodal Embedding

Open paper AskPDF arXiv

Vision-language models learn powerful multimodal embeddings, yet their internal semantics remain opaque. While sparse autoencoders (SAEs) can extract interpretable features, they rely on expanding the representation dimension, which compromises the original geometry and introduces redundancy. We introduce CEDAR (Conceptual Embedding Disentanglement via Adaptive Rotation), a post-hoc method that reveals the compositional structure of pretrained embeddings without increasing dimensionality. By learning an invertible transformation with a top-

k

sparsity bottleneck, CEDAR concentrates semantic information into axis-aligned disentangled coordinates. In CLIP-like architecture, individual coordinates can be interpreted with textual concepts, while for generative models such as BLIP, they can be decoded into natural language descriptions. Experiments demonstrate that CEDAR achieves a competitive reconstruction-sparsity trade-off while producing explanations that are more interpretable and better aligned with human perception. Our results suggest that the apparent entanglement in vision-language representations can be resolved through a suitable change of basis, eliminating the need for overcomplete expansions.

Abstract

Canonical Correlation Analysis

Explore similar work

Conceptualizing Embeddings: Sparse Disentanglement for Vision-Language Models

Extraction and Analysis of Multimodal Concepts in Vision Language Models through Sparse Autoencoders

Bridging Vision and Language Concepts through Optimal Transport Semantic Flow