Aligning the True Semantics: Constrained Decoupling and Distribution Sampling for Cross-Modal Alignment

Xiang Ma; Lexin Fang; Litian Xu; Caiming Zhang

cs.LGMar 5, 2026

Aligning the True Semantics: Constrained Decoupling and Distribution Sampling for Cross-Modal Alignment

Authors: Xiang Ma, Lexin Fang, Litian Xu, Caiming Zhang

Organizations: Shandong University, Jinan, China · The University of Exeter, Exeter, United Kingdom

Abstract

Cross-modal alignment is a crucial task in multimodal learning aimed at achieving semantic consistency between vision and language. This requires that image-text pairs exhibit similar semantics. Traditional algorithms pursue embedding consistency to achieve semantic consistency, ignoring the non-semantic information present in the embedding. An intuitive approach is to decouple the embeddings into semantic and modality components, aligning only the semantic component. However, this introduces two main challenges: (1) There is no established standard for distinguishing semantic and modal information. (2) The modality gap can cause semantic alignment deviation or information loss. To align the true semantics, we propose a novel cross-modal alignment algorithm via Constrained Decoupling and Distribution Sampling (CDDS). Specifically, (1) A dual-path UNet is introduced to adaptively decouple the embeddings, applying multiple constraints to ensure effective separation. (2) A distribution sampling method is proposed to bridge the modality gap, ensuring the rationality of the alignment process. Extensive experiments on various benchmarks and model backbones demonstrate the superiority of CDDS, outperforming state-of-the-art methods by 6.6% to 14.2%.

Figures & tables

Figure 1: Comparison of cross-modal alignment and decoupling-based cross-modal alignment.

Figure 3: Related semantics identification.

ViT-224 , 14 $\times$ 14 patches
Methods	Flickr30K							MS-COCO 1K							MS-COCO 5K
	Image-to-Text			Text-to-Image			rSum	Image-to-Text			Text-to-Image			rSum	Image-to-Text			Text-to-Image			rSum
	R@1	R@5	R@10	R@1	R@5	R@10		R@1	R@5	R@10	R@1	R@5	R@10		R@1	R@5	R@10	R@1	R@5	R@10
VSE++	71.8	92.8	96.5	59.4	84.7	90.9	496.1	75.0	94.6	98.0	62.7	89.4	94.9	514.6	52.4	80.3	88.8	40.6	70.4	81.1	413.4
SCAN	69.5	90.9	95.6	56.4	83.1	90.0	485.6	76.0	95.4	98.1	64.5	90.8	95.8	520.6	53.9	81.8	90.0	42.9	72.3	82.5	423.5
SGR	69.7	90.8	95.2	59.1	84.1	89.9	488.7	77.2	95.0	98.0	65.1	90.7	95.8	521.8	54.9	82.8	90.5	42.8	72.2	82.5	425.8

Table 1: Comparisons of performances with state-of-the-art methods. We list the backbones, image resolution, and the number of patches (e.g., ‘ViT-224,

14\times 14

patches’ means the base-version of ViT with

224\times 224

image resolution input, getting

14\times 14

patches for one image, and the base-version of BERT for text words). The bests are in bold .

Methods	Flickr30K				MS-COCO 5K
	I-to-T		T-to-I		I-to-T		T-to-I
	R@1	R@5	R@1	R@5	R@1	R@5	R@1	R@5
VILT	83.5	96.7	64.4	88.7	61.5	86.3	42.7	72.9
SOHO	86.5	98.1	72.5	92.7	66.4	88.2	50.6	78.0
ALBEF	95.9	99.8	85.6	97.5	77.6	94.3	60.7	84.3
BLIP	96.6	99.8	87.2	97.5	80.6	95.2	63.1	85.3

Table 2: Comparisons with Vision-Language Pre-training (VLP) Models. ‘

\#

’ denotes the zero-shot learning.

Methods	Flickr30K				MS-COCO 5K
	I-to-T		T-to-I		I-to-T		T-to-I		$CR$
	R@1	R@5	R@1	R@5	R@1	R@5	R@1	R@5
ViT-224 , 14 $\times$ 14 patches
w/o $Dec.$	71.4	91.8	57.9	82.5	54.8	82.6	43.2	72.2	-4.6%
w/o $Mod.$	73.8	93.2	62.3	86.9	57.6	84.1	44.6	74.1	-0.9%
w/o $Int.$	69.8	91.5	57.9	80.4	52.1	81.2	42.0	70.8	-6.7%

Table 3: Ablation studies of CDDS.

CR

represents the rate of change in the effect of removing the module.

Methods	Flickr30K				MS-COCO 5K
	I-to-T		T-to-I		I-to-T		T-to-I		$CR$
	R@1	R@5	R@1	R@5	R@1	R@5	R@1	R@5
VSE++	71.8	92.8	59.4	84.7	52.4	80.3	40.6	70.4	+0.6%
+ $Sam$	72.1	93.0	59.7	85.0	52.8	80.9	41.5	70.8	+0.6%
SCAN	69.5	90.9	56.4	83.1	53.9	81.8	42.9	72.3	+1.1%
+ $Sam$	69.8	91.2	56.4	83.8	54.9	82.9	44.1	73.9	+1.1%

Table 4: The application effect of distribution sampling method to other models (trained with ‘ ViT-224 ’).

	Batch Time(s)	rSum
	Batch Time(s)	Flickr30K	MS-COCO 5K
Each batch	0.11	510.6	437.8
Random	0.06	502.3	421.6
All	0.02	483.6	411.6

Table 5: The limitation in efficiency. ‘Each batch’ means Eq. 5 is executed at each batch. ‘Random’ is the random selection of features. ‘All’ means appling Eq. 5 to the total dataset before training.

Figure 5: Visualization of textual embeddings and semantic components, showing decoupling can bring textual embeddings with similar semantics (corresponding to the same image) closer. ‘#0’ is the text’s identifier.

Cross-Modal Alignment

Cross-modal alignment maps representations from different data types into compatible spaces so models can compare, transfer, and combine information.

See all 440 papers

The Geometry of Compromise: Unlocking Generative Capabilities via Controllable Modality Alignment
Mar 31, 2026Hongyuan Liu, Qinli Yang, Wen Li +6Cross-Modal Alignment Modality Gap in VLMs
University of Electronic Science and Technology of China · University of Bristol
SEPS: Semantic-enhanced Patch Slimming Framework for fine-grained cross-modal alignment
Nov 3, 2025Xinyu Mao, Junsi Li, Haoji Zhang +2Cross-Modal Alignment Cross-Modal Feature Matching
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China.
Multimodal Representation Alignment for Cross-modal Information Retrieval
Jun 10, 2025Fan Xu, Luis A. LeivaCross-Modal Learning Multimodal IR
aDepartment of Computer Science, University of Luxembourg, 6, avenue de la Fonte, Esch-sur-Alzette, L-4364, Luxembourg

Mar 31, 2026J/K move · Enter open · S save

The Geometry of Compromise: Unlocking Generative Capabilities via Controllable Modality Alignment

Hongyuan Liu, Qinli Yang, Wen Li, Zhong Zhang+5

University of Electronic Science and Technology of China · University of Bristol

Cross-Modal Alignment Modality Gap in VLMs VLM Adaptation

Open paper AskPDF arXiv

Vision-Language Models (VLMs) such as CLIP learn a shared embedding space for images and text, yet their representations remain geometrically separated, a phenomenon known as the modality gap. This gap limits tasks requiring cross-modal interchangeability, such as captioning and joint clustering. Existing post-processing approaches can partially improve cross-modal compatibility; however, we show through geometric analysis that they primarily reduce the global centroid offset while leaving the underlying distributional mismatch intact. We decompose the modality gap into a Centroid Gap and a Distribution Gap, and demonstrate that the Distribution Gap is the true predictor of cross-modal task quality (

R^2 = 0.986

), whereas the commonly used Raw Gap is misleading (

R^2 = 0.691

). Motivated by this observation, we propose TPC-CMA (Three-Phase Curriculum for Cross-Modal Alignment), a fine-tuning framework that explicitly reduces both components. The proposed CMA jointly mitigates centroid offsets and reshapes the distributional structure, while a three-phase curriculum with gradient-aware scheduling progressively introduces alignment during training to enable stable optimization. Experiments demonstrate that our method significantly improves cross-modal alignment. With

α_{\text{target}}{=}0.05

, the modality gap is reduced by 66.6% with only 4.84% accuracy drop. Under stronger alignment (

α_{\text{target}}{=}0.5

), the gap is reduced by 82.3%, clustering ARI improves from 0.318 to 0.516, and captioning CIDEr increases by 57.1% over the original model. Our code and pre-trained models will be made publicly available upon acceptance.

Abstract

Cross-Modal Alignment

Explore similar work

The Geometry of Compromise: Unlocking Generative Capabilities via Controllable Modality Alignment

SEPS: Semantic-enhanced Patch Slimming Framework for fine-grained cross-modal alignment

Multimodal Representation Alignment for Cross-modal Information Retrieval