Multimodal Structure Learning: Disentangling Shared and Specific Topology via Cross-Modal Graphical Lasso

Fei Wang; Yutong Zhang; Xiong Wang

cs.CVApr 5, 2026

Multimodal Structure Learning: Disentangling Shared and Specific Topology via Cross-Modal Graphical Lasso

Authors: Fei Wang, Yutong Zhang, Xiong Wang

Organizations: Department of Applied Mathematics and Statistics, Stony Brook University · School of Mathematics, Sichuan University · School of Computer Science and Technology, USTC

Abstract

Learning interpretable multimodal representations inherently relies on uncovering the conditional dependencies between heterogeneous features. However, sparse graph estimation techniques, such as Graphical Lasso (GLasso), to visual-linguistic domains is severely bottlenecked by high-dimensional noise, modality misalignment, and the confounding of shared versus category-specific topologies. In this paper, we propose Cross-Modal Graphical Lasso (CM-GLasso) that overcomes these fundamental limitations. By coupling a novel text-visualization strategy with a unified vision-language encoder, we strictly align multimodal features into a shared latent space. We introduce a cross-attention distillation mechanism that condenses high-dimensional patches into explicit semantic nodes, naturally extracting spatial-aware cross-modal priors. Furthermore, we unify tailored GLasso estimation and Common-Specific Structure Learning (CSSL) into a joint objective optimized via the Alternating Direction Method of Multiplier (ADMM). This formulation guarantees the simultaneous disentanglement of invariant and class-specific precision matrices without multi-step error accumulation. Extensive experiments across eight benchmarks covering both natural and medical domains demonstrate that CM-GLasso establishes a new state-of-the-art in generative classification and dense semantic segmentation tasks.

Figures & tables

Figure 1: The Pipeline of CM-GLasso. (a) Text visualization and unified SigLIP 2 feature extraction. (b) Cross-attention distillation condenses patches into semantic nodes. (c) Attention footprints derive spatial-aware cross-modal priors. (d-e) Nonparanormal transformation strictly ensures Gaussianity for the subsequent joint ADMM optimization, simultaneously disentangling shared ( $\bm{\Theta}_{\text{com}}$ ) and specific ( $\bm{S}^{(c)}$ ) topologies. (f) Learned structures govern generative classification and topology-aware segmentation.

Dataset	Method	F1	ACC
CUB-200-2011	ShuffleNetV2 [ 29 ]	0.8774	0.8763
	DA-VPT [ 31 ]	—	0.9130
	PRO-VPT [ 33 ]	—	0.9170
	VFPT [ 46 ]	—	0.9050
	MT-ASM [ 25 ]	—	0.8800
	CM-GLasso (Ours)	0.8836	0.9283

Table 1: Comparative results on classification tasks.

Dataset	Method	mIoU
ADE20K	OneFormer [ 17 ]	0.5700
	InternImage-H [ 41 ]	0.6290
	OmniVec2 [ 36 ]	0.5850
	CM-GLasso (Ours)	0.6401
Kvasir-SEG	Polyp-PVT [ 6 ]	0.8640
	PolypMixNet [ 19 ]	0.8885

Table 2: Comparative results on semantic segmentation (mIoU).

Encoder	ACC	mIoU
BERT+ViT (Het.)	84.23	53.27
CLIP text+ViT [ 30 ]	88.02	59.89
Render+SigLIP 2	91.97	68.65

Table 3: Ablation: Text Encoding Strategy.

Strategy	ACC	mIoU
PCA ( $768\!\to\!p$ )	70.86	44.18
Linear FC	87.37	62.93
Cross-Attn (Ours)	91.97	68.65

Table 4: Ablation: Patch-to-Node Mapping.

Status	SW Pass	ACC	mIoU
w/o Trans.	$\sim$ 23%	83.58	59.02
w/ Trans.	$\sim$ 88%	91.97	68.65

Table 5: Ablation: Nonparanormal Transform.

Method	CSR	Gen. Gap
Indep. GLasso [ 11 ]	—	8.29%
Two-stage [ 5 ]	0.37	3.04%
Joint ADMM	0.42	1.93%

Table 6: Ablation: Optimization (CSR: Common Ratio).

Matrix Used	ACC	mIoU
Only $\bm{\Theta}_{\text{com}}$	84.82	63.17
Only $\bm{S}^{(c)}$	88.43	65.58
$\bm{\Theta}_{\text{com}}+\bm{S}^{(c)}$	91.97	68.65

Table 7: Ablation: Task Head Precision Matrix.

Direction	$\bar{k}^{*}$	$\bm{k^{*}\!=\!0}$ Ratio
Text $\to$ Image	15.6	13.8%
Image $\to$ Text	6.7	33.1%
Self-Priors	$\approx$ 0.1	¿84.0%

Table 8: Ablation: Prior Selection via eBIC.

Metric	PCA	FC	Ours
Graph Structure Quality
Avg. Edges $\|E\|$ in $\bm{\Theta}^{(c)}$	987	491	238
Spurious Edge Ratio (%) $\downarrow$	68.7	31.2	11.4
$\bar{k}^{*}$ (Prior Utilization)	1.2	5.8	15.6
Interpretability & Architecture
Native $p\times p$ Prior	$\times$	$\times$	$\checkmark$

Table 9: Detailed metrics for Patch-to-Node mapping strategies.

|E|

denotes the average number of edges.

$\rho\downarrow$ \ $\gamma_{s}\rightarrow$	0.01	0.05	0.10	0.20
0.01	90.53/66.94	91.02/67.53	91.18/67.72	90.61/67.03
0.05	91.27/67.68	91.64/68.27	91.97/68.65	91.41/67.91
0.10	91.12/67.56	91.53/68.14	91.82/68.47	91.23/67.63
0.20	90.46/66.81	90.83/67.24	91.04/67.40	90.21/66.59

Table 10: Sensitivity analysis of

\rho

and

\gamma_{s}

. Values are presented as ACC(%) / mIoU(%).

	Input Images	GAM Visualizations
CUB-200
CIFAR-10
CIFAR-100
Caltech-256

Figure 2: GAM visualization of the classification head

\mathcal{H}_{C}

. For each dataset (CUB-200-2011, CIFAR-10, CIFAR-100, Caltech-256), four input images (left block) are paired with their GAM heatmaps (right block). Warm regions (red/yellow) indicate spatially discriminative areas; cool regions (blue) indicate low contribution. CM-GLasso consistently focuses on class-discriminative semantics ( e.g. , bird head/wings, vehicle contours) rather than background noise, validating that cross-modal prior guidance improves classification interpretability.

	Sample 1			Sample 2			Sample 3
	Image	GT	Ours	Image	GT	Ours	Image	GT	Ours
ADE20K
Kvasir
VOC-2012
COCO

Figure 3: Qualitative segmentation results of

\mathcal{H}_{S}

. For each dataset (ADE20K, Kvasir-SEG, PASCAL VOC-2012, MS COCO-2014), three sample triplets are shown: input image, Ground Truth (GT), and CM-GLasso prediction (Ours). Our method produces precise boundaries and correctly captures long-range semantic dependencies ( e.g. , polyp edges, sky–water reflections, building–ground transitions), validating the benefit of joint ADMM optimization with cross-modal prior guidance.

Module	Time (sec)	Complexity
ViT Feat. Extraction	242	$\mathcal{O}(NLd^{2})$
Cross-Attn & Prior	58	$\mathcal{O}(NpN_{p}d_{k}+Cp^{2}N_{p})$
eBIC $k^{*}$ Selection	268	$\mathcal{O}(C\|\mathcal{K}\|p^{3})$
Joint ADMM Opt.	7.2	$\mathcal{O}(TCp^{3})$
$\mathcal{H}_{C}$ / $\mathcal{H}_{S}$ Inference	6.8	$\mathcal{O}(Cp^{2}+pN_{p})$

Table 11: Complexity & Runtime (single NVIDIA A800).

Multimodal Learning

Multimodal learning studies how models combine and reason over different data types, such as images, text, audio, signals, and structured records.

See all 335 papers

Text-Guided Visual Dependency Graph Learning with Cross-Modal Attention Priors
Aug 19, 2026Fei Wang, Yutong Zhang, Yang Ye +3Multimodal Graph Learning Probabilistic Graphical Models
Stony Brook University, USA · Sichuan University, China · Universiti Sains Malaysia, Malaysia +1
Mario: Multimodal Graph Reasoning with Large Language Models
Mar 5, 2026Yuanfu Sun, Kang Li, Pengkang Guo +2LLM Reasoning with Graphs Multimodal Reasoning
New York University Shanghai · New York University · Tsinghua University +1
Topology-Aware Representation Alignment for Semi-Supervised Vision-Language Learning
Apr 29, 2026Junwon You, Mihyun Jang, Sangwoo Mo +1Cross-Modal Learning Cross-Modal Representation Learning
KAIST · POSTECH

Aug 19, 2026J/K move · Enter open · S save

Text-Guided Visual Dependency Graph Learning with Cross-Modal Attention Priors

Fei Wang, Yutong Zhang, Yang Ye, Jinxian Chen+2

Stony Brook University, USA · Sichuan University, China · Universiti Sains Malaysia, Malaysia+1

Multimodal Graph Learning Probabilistic Graphical Models Multimodal Classification Semantic Segmentation Cross-Modal Attention

Open paper AskPDF arXiv

Estimating interpretable conditional-dependence structures from multimodal visual-linguistic features remains largely unexplored. We propose CM-GLasso (Cross-Modal Graphical Lasso), a framework that bridges vision-language representation learning and sparse Gaussian Graphical Models. CM-GLasso introduces three key components: (i) a text visualization strategy that renders class-attribute descriptions as images and processes them through the same SigLIP-2 vision encoder as natural images, yielding prototype-indexed patch-level attention footprints in a shared feature coordinate system; (ii) a cross-attention distillation mechanism that condenses high-dimensional patches into a small set of semantic graph nodes, whose attention-footprint similarities yield cross-modal structural priors for non-uniform L1 penalization; (iii) a joint ADMM formulation that estimates shared and class-specific precision components within a single convex objective, avoiding the need to first estimate and then decompose separate class-wise graphs. The learned sparse graph topologies directly support a parameter-free, precision-based classification rule and a lightweight topology-aware segmentation head. Extensive experiments on eight benchmarks demonstrate that CM-GLasso achieves competitive or superior performance compared with strong feature-based and task-specific baselines. Under the matched controlled protocol, it attains the highest average classification accuracy (91.97%) and the highest segmentation mIoU among the controlled baselines on VOC (74.75%) and ADE20K (64.01%), while also yielding explicit sparse conditional-dependence graphs with common-specific decomposition.

Abstract

Multimodal Learning

Explore similar work

Text-Guided Visual Dependency Graph Learning with Cross-Modal Attention Priors

Mario: Multimodal Graph Reasoning with Large Language Models

Topology-Aware Representation Alignment for Semi-Supervised Vision-Language Learning