SCoCCA: Multi-modal Sparse Concept Decomposition via Canonical Correlation Analysis

Ehud Gordon; Meir Yossef Levi; Guy Gilboa

cs.CVMar 14, 2026

SCoCCA: Multi-modal Sparse Concept Decomposition via Canonical Correlation Analysis

Authors: Ehud Gordon, Meir Yossef Levi, Guy Gilboa

Organizations: Viterbi Faculty of Electrical and Computer Engineering, Technion – Israel Institute of Technology, Haifa, Israel

Abstract

Interpreting the internal reasoning of vision-language models is essential for deploying AI in safety-critical domains. Concept-based explainability provides a human-aligned lens by representing a model's behavior through semantically meaningful components. However, existing methods are largely restricted to images and overlook the cross-modal interactions. Text-image embeddings, such as those produced by CLIP, suffer from a modality gap, where visual and textual features follow distinct distributions, limiting interpretability. Canonical Correlation Analysis (CCA) offers a principled way to align features from different distributions, but has not been leveraged for multi-modal concept-level analysis. We show that the objectives of CCA and InfoNCE are closely related, such that optimizing CCA implicitly optimizes InfoNCE, providing a simple, training-free mechanism to enhance cross-modal alignment without affecting the pre-trained InfoNCE objective. Motivated by this observation, we couple concept-based explainability with CCA, introducing Concept CCA (CoCCA), a framework that aligns cross-modal embeddings while enabling interpretable concept decomposition. We further extend it and propose Sparse Concept CCA (SCoCCA), which enforces sparsity to produce more disentangled and discriminative concepts, facilitating improved activation, ablation, and semantic manipulation. Our approach generalizes concept-based explanations to multi-modal embeddings and achieves state-of-the-art performance in concept discovery, evidenced by reconstruction and manipulation tasks such as concept ablation.

Figures & tables

Figure 1: Concept Swapping. Beyond explainability, concept decomposition enables controllable manipulation. Using SCoCCA, an embedding can be decomposed into interpretable concepts (e.g., cube and cylinder), their magnitudes swapped, and the modified embedding recomposed to synthesize an image reflecting the swapped concepts.

Figure 2: Method Overview. In the Concept Discovery phase, text and image embeddings are aligned via Canonical Correlation Analysis (CCA) to form a shared latent space. The Hungarian algorithm establishes a one-to-one correspondence between each concept vector and its most relevant item in the concept bank. In the Concept Decomposition phase, new embedding is decomposed into concepts by solving a Lasso optimization using the matrix $\mathbf{C}$ and the discovered associations.

Category	Metric	Dual-Modality		Single-Modality				Baseline
Category	Metric	SCoCCA (Ours)	SpLiCE [ 6 ]	TCAV [ 21 ]	Varimax [ 60 ]	NMF	K-Means	CLIP [ 41 ]
Purity and Editing	Ablation prob. drop ( $\uparrow$ )	0.87	0.20	0.93	0.81	0.01	0.47	-
	Target prob gain ( $\uparrow$ )	0.95	0.36	0.10	0.75	0.02	-0.09	-
	Img residual cosine ( $\uparrow$ )	0.76	0.70	0.67	0.67	0.63	0.74	-
	Zero-shot accuracy ( $\uparrow$ )	0.74	0.48	0.62	0.65	0.12	0.53	0.75
	Zero-shot precision@5 ( $\uparrow$ )	0.85	0.61	0.59	0.75	0.29	0.70	0.85

Table 1: Comprehensive Performance Comparison. Results comparing concept decomposition methods on subset of 500 random classes from ImageNet [ 8 ] , grouped into dual-modality and single-modality approaches, and evaluated across a wide range of metrics. Best results are shown in bold , and second-best are underlined . SCoCCA achieves state-of-the-art or on-par performance in Reconstruction, as well as in Purity & Editing. Rightmost column shows CLIP zero-shot performance. Note the SCoCCA obtains CLIP-level performance in accuracy.

Figure 3: Concept Retrieval Generalization. Retrieval of the top four MSCOCO [ 32 ] images with the highest activation for the concepts Microwave and Traffic Light , where the concept bank was calibrated on ImageNet only (images are shown in descending order). While SCoCCA retrieves images with a clear presence of the concept, other methods in some cases return images with little or no evidence of the desired concept.

Figure 4:

\lambda

ablation for SCoCCA on ImageNet-500. Top : Zero-shot accuracy on ImageNet-500 as a function the

\lambda

used when solving the sparse CoCCA coding objective in ( 15 ). Bottom :

\|w\|_{0}

normalized by

k

, as function of

\lambda

. In both plots,

\lambda=0

corresponds to the CoCCA baseline without the sparsity penalty. The curves shows that forcing sparsity yields substantially higher zero-shot accuracy than the non-sparse baseline.

Figure 5: ablation of

k

hyperparameter Zero-Shot Accuracy performance on test set of ImageNet-500, as a function of

k

, the number of concepts computed, for the SCoCCA method. As more concepts allows better separation, this is a monotone increasing function. However the rate of steepness decreases, as additional concepts yield less of a return in classification.

Category	Metric	Dual-Modality		Single-Modality				Baseline
Category	Metric	SCoCCA (Ours)	SpLiCE [ 6 ]	TCAV [ 21 ]	Varimax [ 60 ]	NMF	K-Means	CLIP [ 41 ]
Purity and Editing	Ablation prob. drop ( $\uparrow$ )	0.77	0.12	0.79	0.73	0.00	0.37	-
	Target prob gain ( $\uparrow$ )	0.72	0.11	0.02	0.61	-0.19	-0.39	-
	Img residual cosine ( $\uparrow$ )	0.70	0.63	0.58	0.57	0.49	0.68	-
	Zero-shot accuracy ( $\uparrow$ )	0.60	0.37	0.48	0.52	0.10	0.40	0.63
	Zero-shot precision@5 ( $\uparrow$ )	0.81	0.58	0.71	0.74	0.25	0.68	0.82

Table 2: Comprehensive Performance Comparison. Results comparing concept decomposition methods on subset of 500 random classes from ImageNet [ 8 ] , similar to Table 1 , but for CLIP model B-32 . Best results are shown in bold , and second-best are underlined . We observe similar trends like Table 1 . As expected from a smaller model, for same number of

k=500

, reconstruction metrics improve, whereas Purity metrics decrease.

Canonical Correlation Analysis

See all 10 papers

May 21, 2026

Conceptualizing Embeddings: Sparse Disentanglement for Vision-Language Models

Vision-language models learn powerful multimodal embeddings, yet their internal semantics remain opaque. While sparse autoencoders (SAEs) can extract interpretable features, they rely on expanding the representation dimension, which compromises the original geometry and introduces redundancy. We introduce CEDAR (Conceptual Embedding Disentanglement via Adaptive Rotation), a post-hoc method that reveals the compositional structure of pretrained embeddings without increasing dimensionality. By learning an invertible transformation with a top-

k

sparsity bottleneck, CEDAR concentrates semantic information into axis-aligned disentangled coordinates. In CLIP-like architecture, individual coordinates can be interpreted with textual concepts, while for generative models such as BLIP, they can be decoded into natural language descriptions. Experiments demonstrate that CEDAR achieves a competitive reconstruction-sparsity trade-off while producing explanations that are more interpretable and better aligned with human perception. Our results suggest that the apparent entanglement in vision-language representations can be resolved through a suitable change of basis, eliminating the need for overcomplete expansions.

Piotr Kubaty, Patryk Marszałek, Łukasz Struski +3

Faculty of Mathematics and Computer Science, Jagiellonian University, Kraków, Poland · Doctoral School of Exact and Natural Sciences, Jagiellonian University, Kraków, Poland

Disentangled Representation Learning Multimodal Disentangled Representation Learning VLM Interpretability Sparse Autoencoders Multimodal Embedding

Ask PDF

Jun 19, 2026

Extraction and Analysis of Multimodal Concepts in Vision Language Models through Sparse Autoencoders

Vision Language Models (VLMs) have demonstrated impressive performance in tasks requiring joint understanding of images and text, such as image captioning and Visual Question Answering (VQA), but our understanding of their internal processes remains limited. Recently, Sparse Autoencoders (SAEs) have emerged as a promising tool to support the interpretation of concepts encoded in VLMs. However, most SAE-based approaches focus only on textual or visual concepts separately, ignoring multimodal concepts. This limitation hinders a comprehensive understanding of VLMs, since concepts that integrate both modalities can be misclassified. Moreover, previous visual approaches often produce low-quality visual concept descriptions that are vague or incomplete, limiting their usefulness for understanding model reasoning. We propose a framework based on SAEs to extract and analyze visual, textual, and multimodal concepts from VLMs. For each neuron, we propose a candidate human-interpretable concept and compute the alignment between the concept and the dataset samples using cosine similarity scores. Experiments on a VQA dataset (LLaVA-NeXT) demonstrate that our framework improves visual concept quality by up to 45% compared to existing SAE-based methods, while maintaining high textual concept quality and enabling systematic identification of multimodal concepts. This work contributes new insights into the conceptual space of VLMs, providing a structured approach to distinguish between visual, textual, and multimodal concepts. The code is available at https://github.com/PHDLanza/Multidata_SAE

Sergio Lanza, Jae Hee Lee, Stefan Wermter

Knowledge Technology, Department of Informatics, University of Hamburg, Hamburg, Germany

Vision-Language Models VLM Interpretability Sparse Autoencoders Multimodal Learning

Ask PDF

Jun 25, 2026

Bridging Vision and Language Concepts through Optimal Transport Semantic Flow

Concept Bottleneck Models (CBMs) promise transparent reasoning by predicting through human-interpretable concepts, yet their effectiveness fundamentally depends on how well visual and textual representations are aligned or matched. Existing vision-language CBMs often rely on pre-aligned encoders or global cosine similarity, which obscures fine-grained concept localization and fails to reflect true semantic geometry. In this work, we rethink concept alignment as a dynamic cross-modal transport process instead of static projection and propose the Optimal Transport Flow Concept Bottleneck Model (OTF-CBM). It first learns a data-driven semantic cost via Inverse Optimal Transport to measure cross-modal distances, and then performs unbalanced optimal-transport-based flow matching to model semantic transitions between visual patches and textual concepts. With velocity-based concept activation, OTF-CBM captures interpretable geometric relations without ODE integration. Experiments further show that OTF-CBM achieves superior classification accuracy and concept faithfulness, offering a new geometric and dynamical perspective for interpretable cross-modal reasoning.

Chenyang Zhang, Anqi Dong, Guangming Zhu +4

School of Computer Science and Technology, Xidian University, Xi’an, China · KTH Royal Institute of Technology, Stockholm, Sweden

Concept Bottleneck Models VLM Interpretability Vision-Language Alignment Optimal Transport Multimodal Understanding

Ask PDF

Abstract

Canonical Correlation Analysis

Explore similar work

Conceptualizing Embeddings: Sparse Disentanglement for Vision-Language Models

Extraction and Analysis of Multimodal Concepts in Vision Language Models through Sparse Autoencoders

Bridging Vision and Language Concepts through Optimal Transport Semantic Flow