ViCLIP-OT: The First Foundation Vision-Language Model for Vietnamese Image-Text Retrieval with Optimal Transport

Quoc-Khang Tran; Minh-Thien Nguyen; Nguyen-Khang Pham

cs.CVFeb 26, 2026

ViCLIP-OT: The First Foundation Vision-Language Model for Vietnamese Image-Text Retrieval with Optimal Transport

Authors: Quoc-Khang Tran, Minh-Thien Nguyen, Nguyen-Khang Pham

Organizations: Can Tho University, Can Tho, Vietnam

Abstract

Image-text retrieval has become a fundamental component in intelligent multimedia systems; however, most existing vision-language models are optimized for highresource languages and remain suboptimal for low-resource settings such as Vietnamese. This work introduces ViCLIP-OT, a foundation vision-language model specifically designed for Vietnamese image-text retrieval. The proposed framework integrates CLIP-style contrastive learning with a Similarity-Graph Regularized Optimal Transport (SIGROT) loss to enhance global cross-modal consistency and mitigate modality gap issues. Extensive experiments on three Vietnamese benchmarks (UITOpenViIC, KTVIC, and Crossmodal-3600) demonstrate that ViCLIP-OT consistently outperforms CLIP and SigLIP baselines in both in-domain and zero-shot settings. On UIT-OpenViIC, the model achieves an average Recall@K of 67.34%, improving upon CLIP by 5.75 percentage points. In zero-shot evaluation on Crossmodal-3600, ViCLIPOT surpasses CLIP by 11.72 percentage points. Embedding-space analysis further confirms improved alignment and reduced modality gap. The results indicate that integrating SIGROT provides an effective and scalable strategy for cross-modal retrieval in low-resource languages, offering practical implications for intelligent multimedia retrieval systems in Vietnamese and other underrepresented linguistic contexts.

Figures & tables

Figure 1: ViCLIP-OT architecture overview. The model consists of a DINOv3-based image encoder and a Vietnamese Sentence-BERT text encoder that project images and texts into a shared embedding space. The hybrid training objective combines a CLIP-style contrastive loss with the proposed SIGROT loss, which uses a similarity graph and optimal transport to enforce global cross-modal alignment.

Method/Model	# Params	Text $\rightarrow$ Image			Image $\rightarrow$ Text			Avg.
Method/Model	# Params	R@1	R@5	R@10	R@1	R@5	R@10	Avg.
mSigLIP-base* [ 46 ]	370M	14.34	28.94	36.21	20.49	32.23	37.43	28.27
Jina CLIP v2* [ 19 ]	865M	30.01	52.09	61.70	40.23	65.02	74.41	53.91
Jina Embedding v4* [ 14 ]	4B	23.97	42.22	50.29	41.48	66.77	75.61	50.06
Qwen3-VL-Embedding-2B* [ 21 ]	2B	32.13	54.00	62.93	39.83	66.52	77.01	55.40
CLIP	221M	31.19	59.80	71.23	46.60	75.53	85.19	61.59

Table 1: Image-text retrieval performance on the test set of the UIT-OpenViIC dataset. UOT denotes Unbalanced Optimal Transport. * indicates zero-shot evaluation. Best results are in bold and second-best are underlined . ViCLIP-OT and ViSigLIP-OT outperform their corresponding baselines (CLIP and SigLIP without SIGROT), other loss variants (CLIP + UOT, SigLIP + UOT, SIGROT), and pretrained vision–language models under zero-shot settings, demonstrating the effectiveness of integrating the SIGROT loss.

Panel of Figure 2 — Figure 2: R@K comparison on UIT-OpenViIC for text-to-image (left) and image-to-text (right) retrieval tasks. Incorporating the SIGROT loss consistently improves performance over both CLIP and SigLIP baselines across all R@K metrics.

Method	Text $\rightarrow$ Image			Image $\rightarrow$ Text			Avg.
Method	R@1	R@5	R@10	R@1	R@5	R@10	Avg.
KTVIC-train
CLIP	21.12	46.99	59.22	31.65	59.46	72.49	48.49
SigLIP	23.16	48.78	60.57	35.48	62.22	73.64	50.64
ViCLIP-OT	26.24	52.46	64.14	38.47	64.37	75.48	53.52
ViSigLIP-OT	26.28	52.58	63.49	39.62	66.44	77.78	54.37

Table 2: Zero-shot image–text retrieval results on KTVIC and Crossmodal-3600. KTVIC images are deduplicated against the UIT-OpenViIC training set (see B ). Vietnamese captions are used for Crossmodal-3600. ViCLIP-OT and ViSigLIP-OT consistently outperform their corresponding baselines across both datasets in zero-shot settings, confirming the generalizability of the SIGROT loss.

Figure 3: UMAP visualization of image and text embeddings on the UIT-OpenViIC test set. Each subplot corresponds to a different training objective. Circles represent image embeddings and triangles represent text embeddings, with colors indicating pseudo labels obtained via K-Means clustering ( $k=20$ ). SIGROT-based methods exhibit tighter cross-modal clustering compared to baselines.

Method	UIT-OpenViIC		KTVIC-test		Crossmodal-3600
Method	A $\uparrow$	$\\|\Delta_{\text{gap}}\\|\downarrow$	A $\uparrow$	$\\|\Delta_{\text{gap}}\\|\downarrow$	A $\uparrow$	$\\|\Delta_{\text{gap}}\\|\downarrow$
SIGROT	0.8061	0.1323	0.7670	0.2135	0.6976	0.1625
CLIP	0.5201	0.1952	0.4696	0.2032	0.5329	0.2558
ViCLIP-OT	0.6624	0.1026	0.6212	0.1636	0.6225	0.1273
SigLIP	0.3637	0.5843	0.3182	0.5757	0.3790	0.5789
ViSigLIP-OT	0.3928	0.3177	0.3373	0.3385	0.4142	0.3442

Table 3: Alignment score (A) and Modality gap (

\|\Delta_{\text{gap}}\|

) across different datasets. Higher Alignment scores and lower Modality gaps indicate better cross-modal alignment. SIGROT achieves the highest Alignment scores across all datasets. Integrating SIGROT consistently improves both metrics compared to the corresponding baselines, confirming its role in enhancing cross-modal alignment.

Figure 4: GradCAM visualization comparing the baseline SigLIP and the proposed ViSigLIP-OT on the UIT-OpenViIC test set. Each row shows the original image alongside the GradCAM heatmaps from both models for a given Vietnamese text query. In the first two rows, ViSigLIP-OT focuses more precisely on the query-relevant objects (the girl wearing an Ao dai and the man holding apples in his hands), while SigLIP spreads activations over background regions. In the third row, SigLIP correctly attends to the man standing next to a car, whereas ViSigLIP-OT highlights irrelevant background areas.

Panel of Figure 5 — Figure 5: Effect of (a) the number of last unfrozen groups in the image encoder using ViCLIP-OT, and (b) the hybrid loss weight $\lambda$ where $\lambda=0$ corresponds to SIGROT only. Peak performance occurs at 13 unfrozen groups (69.62%) and $\lambda=0.2$ for ViCLIP-OT, $\lambda=0.1$ for ViSigLIP-OT.

Panel of Figure 6 — Figure 6: Average Recall@K for different similarity graph combination strategies. The cross-modality approach achieves the highest performance for both loss configurations.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Domain	Train		Validation		Test
Dataset	Domain	# Img.	# Cap.	# Img.	# Cap.	# Img.	# Cap.
UIT-OpenViIC [ 2 ]	Open domain	9,088	41,238	2,011	10,002	2,001	10,001
KTVIC [ 28 ]	Daily activities	3,769	18,845	-	-	558	2,790
KTVIC (dedup.)	Daily activities	1,305	6,525	-	-	157	785
XM3600 [ 41 ]	Geographically diverse	-	-	-	-	3,600	7,350

Appendix

Table 4: Summary statistics of the datasets used in this work. KTVIC (dedup.) denotes the deduplicated version after near-duplicate removal against the UIT-OpenViIC training set.

Dataset	Original		Duplicates		Remaining
Dataset	# Img.	# Cap.	# Img.	# Cap.	# Img.	# Cap.
KTVIC-train	3,769	18,845	2,464	12,320	1,305	6,525
KTVIC-test	558	2,790	401	2,005	157	785
Crossmodal-3600	3,600	7,350	0	0	3,600	7,350

Appendix

Table 5: Near-duplicate removal results against the UIT-OpenViIC training set using SSCD with a cosine similarity threshold of 0.8.

# Last Unfrozen Groups	Text $\rightarrow$ Image			Image $\rightarrow$ Text			Avg.
# Last Unfrozen Groups	R@1	R@5	R@10	R@1	R@5	R@10	Avg.
0	28.06	55.94	67.48	44.36	73.45	82.79	58.68
1	28.21	56.12	67.64	43.76	74.14	83.89	58.96
2	35.91	63.63	74.17	52.86	80.11	87.62	65.72
3	39.01	66.04	75.91	55.49	81.20	89.16	67.80
5	40.96	67.96	77.01	57.28	82.35	90.05	69.27

Appendix

Table 6: Effect of partial fine-tuning of the image encoder on retrieval performance using ViCLIP-OT. The row with 0 represents a fully frozen image encoder, while

k

indicates unfreezing the last

k

groups (14 corresponds to all layers).

$\lambda$	Text $\rightarrow$ Image			Image $\rightarrow$ Text			Avg.
$\lambda$	R@1	R@5	R@10	R@1	R@5	R@10	Avg.
0.0	38.48	64.47	74.59	35.90	58.83	68.47	56.79
0.1	41.01	67.67	77.39	56.29	82.05	89.81	69.04
0.2	39.94	67.62	77.24	56.74	82.89	90.75	69.20
0.3	38.38	66.67	76.49	55.10	82.30	89.71	68.11
0.5	37.40	65.88	75.76	53.26	81.15	88.96	67.07

Appendix

Table 7: Effect of the CLIP loss weight

\lambda

on retrieval performance using the proposed hybrid loss (

\lambda=0

corresponds to SIGROT only). The highest scores are highlighted in bold, and the second-highest scores are underlined.

$\lambda$	Text $\rightarrow$ Image			Image $\rightarrow$ Text			Avg.
$\lambda$	R@1	R@5	R@10	R@1	R@5	R@10	Avg.
0.0	38.48	64.47	74.59	35.90	58.83	68.47	56.79
0.1	41.51	68.76	77.91	59.08	85.33	91.99	70.76
0.15	40.91	68.46	77.88	58.53	84.49	91.35	70.27
0.2	40.69	68.27	77.50	58.48	84.24	91.00	70.03
0.3	40.20	67.84	77.10	57.88	83.49	90.20	69.45

Appendix

Table 8: Effect of the SigLIP loss weight

\lambda

on retrieval performance using the proposed hybrid loss (

\lambda=0

corresponds to SIGROT only). The highest scores are highlighted in bold, and the second-highest scores are underlined.

Text tower	Image tower	# Params	Method	Text $\rightarrow$ Image			Image $\rightarrow$ Text			Avg.
Text tower	Image tower	# Params	Method	R@1	R@5	R@10	R@1	R@5	R@10	Avg.
SBERT	ViT-B/16	221M	SigLIP	34.75	63.01	72.96	50.10	79.78	88.04	64.77
SBERT	ViT-B/16	221M	SigLIP-SIGROT	39.19	66.71	76.04	57.21	83.83	90.79	68.96
EG-300M	ViT-B/16	394M	SigLIP	34.02	60.89	71.33	51.60	79.88	87.89	64.27
EG-300M	ViT-B/16	394M	SigLIP-SIGROT	39.63	65.93	75.07	57.61	83.58	90.04	68.64
BGE-M3	ViT-B/16	655M	SigLIP	34.88	62.41	72.70	52.30	81.03	89.14	65.41

Appendix

Table 9: Retrieval results with different model architectures. Results are reported on the UIT-OpenViIC test set. EG-300M denotes EmbeddingGemma-300M. ViT-B/16 and ConvNeXt-base are pretrained with the DINOv3 framework. Integrating SIGROT consistently improves retrieval performance across all model architectures.

Combination strategy	Text $\rightarrow$ Image			Image $\rightarrow$ Text			Avg.
Combination strategy	R@1	R@5	R@10	R@1	R@5	R@10	Avg.
ViCLIP-OT
Arithmetic mean (Eq. 23 )	38.77	66.09	76.21	54.85	81.65	90.45	68.00
Harmonic mean (Eq. 24 )	37.02	64.13	74.80	49.78	80.01	88.21	65.66
Caption only [ 39 ]	33.29	59.24	69.26	42.37	70.01	80.26	59.07
Image only	32.18	60.08	70.94	47.79	76.83	86.28	62.35

Appendix

Table 10: Effect of different similarity graph combination strategies on retrieval performance. The cross-modality strategy achieves the highest average R@K for both ViCLIP-OT and ViSigLIP-OT, while the caption-only strategy yields the lowest performance.

Text tower	Image tower	Method	A $\uparrow$	$\\|\Delta_{\text{gap}}\\|\downarrow$
SBERT	ViT-B/16	SigLIP	0.3637	0.5843
SBERT	ViT-B/16	SigLIP-SIGROT	0.3928	0.3177
EG-300M	ViT-B/16	SigLIP	0.3460	0.5234
EG-300M	ViT-B/16	SigLIP-SIGROT	0.3543	0.4066
BGE-M3	ViT-B/16	SigLIP	0.3866	1.3584
BGE-M3	ViT-B/16	SigLIP-SIGROT	0.4007	0.8405

Appendix

Table 11: Alignment score and modality gap with different model architectures.

A

denotes the alignment score (higher is better), and

\|\Delta_{\text{gap}}\|

denotes the modality gap (lower is better).

Figure 7: Text-to-image retrieval examples on the UIT-OpenViIC test set. For each query, the first row shows results from ViSigLIP-OT and the second row from SigLIP. Retrieved images are sorted by similarity scores.

Figure 8: Image-to-image retrieval examples on the UIT-OpenViIC test set. The first image in each row is the query. For each query, the first row shows results from ViSigLIP-OT and the second row from SigLIP. Retrieved images are sorted by similarity scores.

Figure 9: Additional GradCAM visualizations comparing SigLIP and ViSigLIP-OT. When the relevant objects are easy to identify (rows 1–2: plates of vegetables and lanterns), both models produce similar heatmaps. In more complex scenes (rows 3–4: cars on the street and a woman holding flowers), ViSigLIP-OT attends more precisely to the query-relevant objects, while SigLIP distributes attention to background regions. In the last row, the image does not match the query, and ViSigLIP-OT produces a more spread-out heatmap rather than concentrating on specific regions.

Explore similar work

SEA-CLIP-Tiny: Efficient Multilingual Text-Vision Embedding for Southeast Asian Languages
Sep 25, 2026Puja Ahmad Habibi, Faiz Assabil Firdaus, Ashvanth S +3Vision-Language Models Cross-Modal Retrieval
SEACrowd · University of Indonesia · Cohere Labs Community +4
CLIMP: Contrastive Language-Image Mamba Pretraining
Jan 11, 2026Nimrod Shabtay, Itamar Zimerman, Eli Schwartz +1Cross-Modal Learning VLM Robustness
Tel-Aviv University · IBM Research
UOT-Gap: A Variational Principle for the Modality Gap in Vision-Language Models via Unbalanced Optimal Transport
Sep 9, 2026Zonglin Yang, Huilan Ma, Xudan Zheng +1Modality Gap in VLMs Vision-Language Retrieval
Guangdong Police College

Sep 25, 2026J/K move · Enter open · S save

SEA-CLIP-Tiny: Efficient Multilingual Text-Vision Embedding for Southeast Asian Languages

Puja Ahmad Habibi, Faiz Assabil Firdaus, Ashvanth S, Ekapol Chuangsuwanich+2

SEACrowd · University of Indonesia · Cohere Labs Community+4

Vision-Language Models Cross-Modal Retrieval Efficient VLM Inference

Open paper AskPDF arXiv

Multilingual text-vision embedding models are essential for cross-lingual image-text retrieval, but Southeast Asian languages remain poorly supported due to the region's linguistic diversity and limited data and computing resources. In this paper, we introduce SEA-CLIP-Tiny, a compact multilingual text-vision embedding model for Southeast Asia with fewer than 50M parameters. Our model adapts a CLIP-KD-style framework to Southeast Asian multilingual settings through regional data curation and multilingual teacher guidance. Experiments across seven Southeast Asian languages show that SEA-CLIP-Tiny achieves the strongest average retrieval performance among the evaluated student models, reaching 12.9%, 31.5%, and 42.2% at R@1, R@5, and R@10, respectively. Compared with MobileCLIP2, it improves average R@10 by 12.1 points while using 38.4% fewer parameters and lower measured CPU latency. These results highlight the importance of region-aware training for efficient multilingual text-vision models in Southeast Asia.

Abstract

Explore similar work

SEA-CLIP-Tiny: Efficient Multilingual Text-Vision Embedding for Southeast Asian Languages

CLIMP: Contrastive Language-Image Mamba Pretraining

UOT-Gap: A Variational Principle for the Modality Gap in Vision-Language Models via Unbalanced Optimal Transport