The Geometry of Compromise: Unlocking Generative Capabilities via Controllable Modality Alignment

Hongyuan Liu; Qinli Yang; Wen Li; Zhong Zhang; Jiaming Liu; Wei Han; Zhili Qin; Jinxia Guo

cs.CVMar 31, 2026

The Geometry of Compromise: Unlocking Generative Capabilities via Controllable Modality Alignment

Authors: Hongyuan Liu, Qinli Yang, Wen Li, Zhong Zhang, Jiaming Liu, Wei Han, Zhili Qin, Jinxia Guo,

Organizations: University of Electronic Science and Technology of China · University of Bristol

Abstract

Vision-Language Models (VLMs) such as CLIP learn a shared embedding space for images and text, yet their representations remain geometrically separated, a phenomenon known as the modality gap. This gap limits tasks requiring cross-modal interchangeability, such as captioning and joint clustering. Existing post-processing approaches can partially improve cross-modal compatibility; however, we show through geometric analysis that they primarily reduce the global centroid offset while leaving the underlying distributional mismatch intact. We decompose the modality gap into a Centroid Gap and a Distribution Gap, and demonstrate that the Distribution Gap is the true predictor of cross-modal task quality ( $R^2 = 0.986$ ), whereas the commonly used Raw Gap is misleading ( $R^2 = 0.691$ ). Motivated by this observation, we propose TPC-CMA (Three-Phase Curriculum for Cross-Modal Alignment), a fine-tuning framework that explicitly reduces both components. The proposed CMA jointly mitigates centroid offsets and reshapes the distributional structure, while a three-phase curriculum with gradient-aware scheduling progressively introduces alignment during training to enable stable optimization. Experiments demonstrate that our method significantly improves cross-modal alignment. With $α_{\text{target}}{=}0.05$ , the modality gap is reduced by 66.6% with only 4.84% accuracy drop. Under stronger alignment ( $α_{\text{target}}{=}0.5$ ), the gap is reduced by 82.3%, clustering ARI improves from 0.318 to 0.516, and captioning CIDEr increases by 57.1% over the original model. Our code and pre-trained models will be made publicly available upon acceptance.

Figures & tables

Figure 1: The t-SNE visualization of multimodal features (top) and downstream performance of different methods (bottom). (a) Original CLIP: image and text embeddings form separated islands. (b) Mean-Centering Liang et al. (2022) ; Li et al. (2025) ; An et al. (2025) ; Yamashita et al. (2025) : centroids overlap but distributional mismatch persists. (c) Our alignment method (TPC-CMA,

\alpha_{\text{target}}{=}0.5

): true semantic interleaving. Each panel reports the Modality Gap, defined as the residual cosine distance after centroid alignment (see Section 2.2 ); lower values indicate better structural alignment. Bottom : downstream impact on two zero-shot tasks. (d) Image captioning CIDEr improves from 0.210 to 0.330. (e) Joint clustering ARI improves from 0.318 to 0.516.

Figure 2: Overview of TPC-CMA. A : CLIP backbone extracts image and text embeddings. B : CMA Loss combines Negative Reweighting (reduces Centroid Gap) and Intra-modal Geometry Matching (reduces Distribution Gap) via $\alpha$ . C : Three-Phase Curriculum schedules $\alpha$ from 0 to $\alpha_{\text{target}}$ across Anchor, Gradient-aware Ramp-up, and Stabilize stages, with the transition speed dynamically modulated by the observed gradient dynamics between loss terms.

Figure 3: Gap decomposition under Mean-Centering. Despite a 97% Centroid Gap reduction, both Distribution Gap and ROUGE-L remain unchanged, showing that correcting the centroid offset alone does not improve cross-modal compatibility.

Method	Type	Gap	Gap $\downarrow$ %	$\mathcal{G}_{D}$	ImageNet	I2T R@1	T2I R@1	V-Measure	CIDEr	ROUGE-L
Original CLIP Radford et al. (2021)	Pretrained	0.733	-	0.685	62.62	53.42	35.36	0.769	0.210	0.250
Mean-Centering Liang et al. (2022)	Post-proc	0.685	6.5	0.685	60.23	43.74	34.44	0.766	0.227	0.250
AlignCLIP Eslami and de Melo (2025) †	From-scratch	0.353	51.8	0.550	32.79	33.64	22.28	0.749	0.112	0.244
M 2 -Mix Oh et al. (2023)	Fine-tune	0.710	3.1	0.637	54.24	42.98	31.00	0.762	0.197	0.239
CLIP-Refine Yamaguchi et al. (2025)	Fine-tune	0.747	$-$ 1.9	0.647	62.99	53.90	36.31	0.765	0.236	0.265
CS-Aligner Yin et al. (2025)	Fine-tune	0.734	$-$ 0.1	0.617	61.37	49.15	32.89	0.770	0.174	0.249

Table 1: Main comparison across gap reduction, discriminative, generative, and structural tasks.

Figure 4: Gap vs. Accuracy Pareto Frontier. TPC-CMA forms a smooth efficient frontier that envelopes all selected baselines. AlignCLIP and M 2 -Mix fall significantly below the frontier, while CLIP-Refine fails to reduce the gap.

Method	$\mathcal{G}_{D}$	CIDEr	BLEU-4	ROUGE-L
Original CLIP	0.685	0.210	0.029	0.250
Mean-Centering	0.685	0.227	0.040	0.250
AlignCLIP	0.550	0.112	0.015	0.244
CLIP-Refine	0.647	0.236	0.045	0.265
M 2 -Mix	0.637	0.197	0.032	0.239
CS-Aligner	0.617	0.174	0.036	0.249

Table 2: DeCap zero-shot captioning on COCO. Image embeddings are directly used as the decoder prefix without any learned projection layer.

Method	$\mathcal{G}_{D}$	V-Measure	ARI
Original CLIP	0.685	0.769	0.318
Mean-Centering	0.685	0.766	0.306
AlignCLIP	0.550	0.749	0.304
M 2 -Mix	0.637	0.762	0.298
CLIP-Refine	0.647	0.765	0.310
CS-Aligner	0.617	0.770	0.315

Table 3: Joint image-text clustering following Grassucci et al. Grassucci et al. (2026) (ImageNet val, 200 classes

\times

50 imgs, KMeans

k{=}200

Figure 5: Distribution Gap vs. Raw Gap as predictors of DeCap CIDEr score. (a) Distribution Gap is a near-perfect predictor (

R^{2}=0.986

); (b) Raw Gap yields a substantially weaker fit (

R^{2}=0.691

Method	I2T R@1	I2T R@5	T2I R@1	T2I R@5
Original CLIP	53.42	77.18	35.36	61.52
Mean-Centering	43.74	70.70	34.44	60.03
TPC-CMA $\alpha_{\text{target}}{=}0.01$	50.96	76.10	35.96	62.16
TPC-CMA $\alpha_{\text{target}}{=}0.05$	48.56	74.86	35.68	61.87

Table 4: COCO retrieval (5K test split).

Method	ImageNet	CIFAR-100	Food-101	Caltech	Flowers
Orig. CLIP	62.62	69.80	79.50	85.90	67.20
Mean-Centering	60.23	67.60	78.50	83.90	62.00
TPC-CMA $\alpha_{\text{target}}{=}0.01$	60.86	67.80	77.00	87.10	64.00
TPC-CMA $\alpha_{\text{target}}{=}0.05$	57.78	66.20	75.30	87.00	62.30

Table 5: Multi-dataset zero-shot classification.

Variant	NR	Intra	TPC	$\alpha$	Modal. Gap $\downarrow$	ImageNet Top-1	ARI
w/o NR		✓	✓	0.5	0.282	54.66	0.453
w/o Intra	✓		✓	0.5	0.239	53.13	0.348
CMA-Only	✓	✓		0.5	0.133	49.75	0.362
TPC-CMA	✓	✓	✓	0.5	0.130	54.86	0.516
w/o NR		✓	✓	0.05	0.364	55.47	0.372
w/o Intra	✓		✓	0.05	0.379	55.19	0.342

Table 6: Component ablation. w/o Intra applies Negative Reweighting only. CMA-Only applies both CMA mechanisms but with constant

\alpha

from epoch 1 (no three-phase curriculum).

Figure 6: Effective rank dynamics. (a) Single-modal effective rank vs. $\alpha_{\text{target}}$ ; rank follows an inverted-U, peaking near $\alpha_{\text{target}}{=}0.1$ before declining. Baselines are shown as horizontal dotted lines. (b) Modality Fusion Index vs. ARI.

Cross-Modal Alignment

Cross-modal alignment maps representations from different data types into compatible spaces so models can compare, transfer, and combine information.

See all 440 papers

Stabilizing the Modality Gap: Null-Space Projected Continual Learning for Vision-Language Models
Jul 26, 2025Tiantian Peng, Yuyang Liu, Shuo Yang +2Continual Learning for VLMs Modality Gap in VLMs
Peking University, Shenzhen Graduate School · Peng Cheng Laboratory
One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Models
Sep 28, 2026Aditya Sharma, Divya SaxenaCross-Modal Retrieval Modality Gap in VLMs
Indian Institute of Technology Jodhpur
UOT-Gap: A Variational Principle for the Modality Gap in Vision-Language Models via Unbalanced Optimal Transport
Sep 9, 2026Zonglin Yang, Huilan Ma, Xudan Zheng +1Modality Gap in VLMs Vision-Language Retrieval
Guangdong Police College

Jul 26, 2025J/K move · Enter open · S save

Stabilizing the Modality Gap: Null-Space Projected Continual Learning for Vision-Language Models

Tiantian Peng, Yuyang Liu, Shuo Yang, Qiuhe Hong+1

Peking University, Shenzhen Graduate School · Peng Cheng Laboratory

Continual Learning for VLMs Modality Gap in VLMs

Open paper AskPDF arXiv

The modality gap is intrinsic to multimodal alignment, where embeddings from different modalities remain partially separated in a shared representation space. In continual learning, CLIP-based vision-language models (VLMs) not only suffer catastrophic forgetting but also experience modality gap drift that disrupts the shared embedding geometry. Such drift degrades cross-modal alignment and generalization. We propose FPCD, an efficient framework combining Fast Null-space Projection (FNP) and Contrastive Distillation (CD). FNP accumulates compact Gram matrices of previous tasks' activations and projects gradients onto layer-adaptive approximate null spaces to reduce interference with learned representations. By replacing singular value decomposition with symmetric eigendecomposition, FNP accelerates projection-basis construction by a factor of 5-9. CD uses a small reference set to align the bidirectional image-to-text and text-to-image similarity distributions of the fine-tuned model with CLIP, thereby preserving pretrained cross-modal relations. On the 11-task MTIL benchmark, FPCD achieves 87.6% Last, 76.4% Average, and 65.2% Transfer. It also maintains more stable modality gaps and stronger zero-shot retrieval performance on COCO Captions and Flickr30k than competing methods. These results highlight the importance of controlling modality gap drift during continual adaptation of VLMs.

Abstract

Cross-Modal Alignment

Explore similar work

Stabilizing the Modality Gap: Null-Space Projected Continual Learning for Vision-Language Models

One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Models

UOT-Gap: A Variational Principle for the Modality Gap in Vision-Language Models via Unbalanced Optimal Transport