Image-text retrieval has become a fundamental component in intelligent multimedia systems; however, most existing vision-language models are optimized for highresource languages and remain suboptimal for low-resource settings such as Vietnamese. This work introduces ViCLIP-OT, a foundation vision-language model specifically designed for Vietnamese image-text retrieval. The proposed framework integrates CLIP-style contrastive learning with a Similarity-Graph Regularized Optimal Transport (SIGROT) loss to enhance global cross-modal consistency and mitigate modality gap issues. Extensive experiments on three Vietnamese benchmarks (UITOpenViIC, KTVIC, and Crossmodal-3600) demonstrate that ViCLIP-OT consistently outperforms CLIP and SigLIP baselines in both in-domain and zero-shot settings. On UIT-OpenViIC, the model achieves an average Recall@K of 67.34%, improving upon CLIP by 5.75 percentage points. In zero-shot evaluation on Crossmodal-3600, ViCLIPOT surpasses CLIP by 11.72 percentage points. Embedding-space analysis further confirms improved alignment and reduced modality gap. The results indicate that integrating SIGROT provides an effective and scalable strategy for cross-modal retrieval in low-resource languages, offering practical implications for intelligent multimedia retrieval systems in Vietnamese and other underrepresented linguistic contexts.
Figures & tables
Figure 1: ViCLIP-OT architecture overview. The model consists of a DINOv3-based image encoder and a Vietnamese Sentence-BERT text encoder that project images and texts into a shared embedding space. The hybrid training objective combines a CLIP-style contrastive loss with the proposed SIGROT loss, which uses a similarity graph and optimal transport to enforce global cross-modal alignment.
Method/Model
# Params
Text → Image
Image → Text
Avg.
R@1
R@5
R@10
R@1
R@5
R@10
mSigLIP-base* [ 46 ]
370M
14.34
28.94
36.21
20.49
32.23
37.43
28.27
Jina CLIP v2* [ 19 ]
865M
30.01
52.09
61.70
40.23
65.02
74.41
53.91
Jina Embedding v4* [ 14 ]
4B
23.97
42.22
50.29
41.48
66.77
75.61
50.06
Qwen3-VL-Embedding-2B* [ 21 ]
2B
32.13
54.00
62.93
39.83
66.52
77.01
55.40
CLIP
221M
31.19
59.80
71.23
46.60
75.53
85.19
61.59
Table 1: Image-text retrieval performance on the test set of the UIT-OpenViIC dataset. UOT denotes Unbalanced Optimal Transport. * indicates zero-shot evaluation. Best results are in bold and second-best are underlined . ViCLIP-OT and ViSigLIP-OT outperform their corresponding baselines (CLIP and SigLIP without SIGROT), other loss variants (CLIP + UOT, SigLIP + UOT, SIGROT), and pretrained vision–language models under zero-shot settings, demonstrating the effectiveness of integrating the SIGROT loss.
Figure 2: R@K comparison on UIT-OpenViIC for text-to-image (left) and image-to-text (right) retrieval tasks. Incorporating the SIGROT loss consistently improves performance over both CLIP and SigLIP baselines across all R@K metrics.
Method
Text → Image
Image → Text
Avg.
R@1
R@5
R@10
R@1
R@5
R@10
KTVIC-train
CLIP
21.12
46.99
59.22
31.65
59.46
72.49
48.49
SigLIP
23.16
48.78
60.57
35.48
62.22
73.64
50.64
ViCLIP-OT
26.24
52.46
64.14
38.47
64.37
75.48
53.52
ViSigLIP-OT
26.28
52.58
63.49
39.62
66.44
77.78
54.37
Table 2: Zero-shot image–text retrieval results on KTVIC and Crossmodal-3600. KTVIC images are deduplicated against the UIT-OpenViIC training set (see B ). Vietnamese captions are used for Crossmodal-3600. ViCLIP-OT and ViSigLIP-OT consistently outperform their corresponding baselines across both datasets in zero-shot settings, confirming the generalizability of the SIGROT loss.
Figure 3: UMAP visualization of image and text embeddings on the UIT-OpenViIC test set. Each subplot corresponds to a different training objective. Circles represent image embeddings and triangles represent text embeddings, with colors indicating pseudo labels obtained via K-Means clustering ( k=20 ). SIGROT-based methods exhibit tighter cross-modal clustering compared to baselines.
Method
UIT-OpenViIC
KTVIC-test
Crossmodal-3600
A ↑
∥Δgap∥↓
A ↑
∥Δgap∥↓
A ↑
∥Δgap∥↓
SIGROT
0.8061
0.1323
0.7670
0.2135
0.6976
0.1625
CLIP
0.5201
0.1952
0.4696
0.2032
0.5329
0.2558
ViCLIP-OT
0.6624
0.1026
0.6212
0.1636
0.6225
0.1273
SigLIP
0.3637
0.5843
0.3182
0.5757
0.3790
0.5789
ViSigLIP-OT
0.3928
0.3177
0.3373
0.3385
0.4142
0.3442
Table 3: Alignment score (A) and Modality gap ( ∥Δgap∥ ) across different datasets. Higher Alignment scores and lower Modality gaps indicate better cross-modal alignment. SIGROT achieves the highest Alignment scores across all datasets. Integrating SIGROT consistently improves both metrics compared to the corresponding baselines, confirming its role in enhancing cross-modal alignment.
Figure 4: GradCAM visualization comparing the baseline SigLIP and the proposed ViSigLIP-OT on the UIT-OpenViIC test set. Each row shows the original image alongside the GradCAM heatmaps from both models for a given Vietnamese text query. In the first two rows, ViSigLIP-OT focuses more precisely on the query-relevant objects (the girl wearing an Ao dai and the man holding apples in his hands), while SigLIP spreads activations over background regions. In the third row, SigLIP correctly attends to the man standing next to a car, whereas ViSigLIP-OT highlights irrelevant background areas.
Figure 5: Effect of (a) the number of last unfrozen groups in the image encoder using ViCLIP-OT, and (b) the hybrid loss weight λ where λ=0 corresponds to SIGROT only. Peak performance occurs at 13 unfrozen groups (69.62%) and λ=0.2 for ViCLIP-OT, λ=0.1 for ViSigLIP-OT.
Figure 6: Average Recall@K for different similarity graph combination strategies. The cross-modality approach achieves the highest performance for both loss configurations.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Domain
Train
Validation
Test
# Img.
# Cap.
# Img.
# Cap.
# Img.
# Cap.
UIT-OpenViIC [ 2 ]
Open domain
9,088
41,238
2,011
10,002
2,001
10,001
KTVIC [ 28 ]
Daily activities
3,769
18,845
-
-
558
2,790
KTVIC (dedup.)
Daily activities
1,305
6,525
-
-
157
785
XM3600 [ 41 ]
Geographically diverse
-
-
-
-
3,600
7,350
Appendix
Table 4: Summary statistics of the datasets used in this work. KTVIC (dedup.) denotes the deduplicated version after near-duplicate removal against the UIT-OpenViIC training set.
Dataset
Original
Duplicates
Remaining
# Img.
# Cap.
# Img.
# Cap.
# Img.
# Cap.
KTVIC-train
3,769
18,845
2,464
12,320
1,305
6,525
KTVIC-test
558
2,790
401
2,005
157
785
Crossmodal-3600
3,600
7,350
0
0
3,600
7,350
Appendix
Table 5: Near-duplicate removal results against the UIT-OpenViIC training set using SSCD with a cosine similarity threshold of 0.8.
# Last Unfrozen Groups
Text → Image
Image → Text
Avg.
R@1
R@5
R@10
R@1
R@5
R@10
0
28.06
55.94
67.48
44.36
73.45
82.79
58.68
1
28.21
56.12
67.64
43.76
74.14
83.89
58.96
2
35.91
63.63
74.17
52.86
80.11
87.62
65.72
3
39.01
66.04
75.91
55.49
81.20
89.16
67.80
5
40.96
67.96
77.01
57.28
82.35
90.05
69.27
Appendix
Table 6: Effect of partial fine-tuning of the image encoder on retrieval performance using ViCLIP-OT. The row with 0 represents a fully frozen image encoder, while k indicates unfreezing the last k groups (14 corresponds to all layers).
λ
Text → Image
Image → Text
Avg.
R@1
R@5
R@10
R@1
R@5
R@10
0.0
38.48
64.47
74.59
35.90
58.83
68.47
56.79
0.1
41.01
67.67
77.39
56.29
82.05
89.81
69.04
0.2
39.94
67.62
77.24
56.74
82.89
90.75
69.20
0.3
38.38
66.67
76.49
55.10
82.30
89.71
68.11
0.5
37.40
65.88
75.76
53.26
81.15
88.96
67.07
Appendix
Table 7: Effect of the CLIP loss weight λ on retrieval performance using the proposed hybrid loss ( λ=0 corresponds to SIGROT only). The highest scores are highlighted in bold, and the second-highest scores are underlined.
λ
Text → Image
Image → Text
Avg.
R@1
R@5
R@10
R@1
R@5
R@10
0.0
38.48
64.47
74.59
35.90
58.83
68.47
56.79
0.1
41.51
68.76
77.91
59.08
85.33
91.99
70.76
0.15
40.91
68.46
77.88
58.53
84.49
91.35
70.27
0.2
40.69
68.27
77.50
58.48
84.24
91.00
70.03
0.3
40.20
67.84
77.10
57.88
83.49
90.20
69.45
Appendix
Table 8: Effect of the SigLIP loss weight λ on retrieval performance using the proposed hybrid loss ( λ=0 corresponds to SIGROT only). The highest scores are highlighted in bold, and the second-highest scores are underlined.
Text tower
Image tower
# Params
Method
Text → Image
Image → Text
Avg.
R@1
R@5
R@10
R@1
R@5
R@10
SBERT
ViT-B/16
221M
SigLIP
34.75
63.01
72.96
50.10
79.78
88.04
64.77
SigLIP-SIGROT
39.19
66.71
76.04
57.21
83.83
90.79
68.96
EG-300M
ViT-B/16
394M
SigLIP
34.02
60.89
71.33
51.60
79.88
87.89
64.27
SigLIP-SIGROT
39.63
65.93
75.07
57.61
83.58
90.04
68.64
BGE-M3
ViT-B/16
655M
SigLIP
34.88
62.41
72.70
52.30
81.03
89.14
65.41
Appendix
Table 9: Retrieval results with different model architectures. Results are reported on the UIT-OpenViIC test set. EG-300M denotes EmbeddingGemma-300M. ViT-B/16 and ConvNeXt-base are pretrained with the DINOv3 framework. Integrating SIGROT consistently improves retrieval performance across all model architectures.
Combination strategy
Text → Image
Image → Text
Avg.
R@1
R@5
R@10
R@1
R@5
R@10
ViCLIP-OT
Arithmetic mean (Eq. 23 )
38.77
66.09
76.21
54.85
81.65
90.45
68.00
Harmonic mean (Eq. 24 )
37.02
64.13
74.80
49.78
80.01
88.21
65.66
Caption only [ 39 ]
33.29
59.24
69.26
42.37
70.01
80.26
59.07
Image only
32.18
60.08
70.94
47.79
76.83
86.28
62.35
Appendix
Table 10: Effect of different similarity graph combination strategies on retrieval performance. The cross-modality strategy achieves the highest average R@K for both ViCLIP-OT and ViSigLIP-OT, while the caption-only strategy yields the lowest performance.
Text tower
Image tower
Method
A ↑
∥Δgap∥↓
SBERT
ViT-B/16
SigLIP
0.3637
0.5843
SigLIP-SIGROT
0.3928
0.3177
EG-300M
ViT-B/16
SigLIP
0.3460
0.5234
SigLIP-SIGROT
0.3543
0.4066
BGE-M3
ViT-B/16
SigLIP
0.3866
1.3584
SigLIP-SIGROT
0.4007
0.8405
Appendix
Table 11: Alignment score and modality gap with different model architectures. A denotes the alignment score (higher is better), and ∥Δgap∥ denotes the modality gap (lower is better).
Figure 7: Text-to-image retrieval examples on the UIT-OpenViIC test set. For each query, the first row shows results from ViSigLIP-OT and the second row from SigLIP. Retrieved images are sorted by similarity scores.
Figure 8: Image-to-image retrieval examples on the UIT-OpenViIC test set. The first image in each row is the query. For each query, the first row shows results from ViSigLIP-OT and the second row from SigLIP. Retrieved images are sorted by similarity scores.
Figure 9: Additional GradCAM visualizations comparing SigLIP and ViSigLIP-OT. When the relevant objects are easy to identify (rows 1–2: plates of vegetables and lanterns), both models produce similar heatmaps. In more complex scenes (rows 3–4: cars on the street and a woman holding flowers), ViSigLIP-OT attends more precisely to the query-relevant objects, while SigLIP distributes attention to background regions. In the last row, the image does not match the query, and ViSigLIP-OT produces a more spread-out heatmap rather than concentrating on specific regions.
Multilingual text-vision embedding models are essential for cross-lingual image-text retrieval, but Southeast Asian languages remain poorly supported due to the region's linguistic diversity and limited data and computing resources. In this paper, we introduce SEA-CLIP-Tiny, a compact multilingual text-vision embedding model for Southeast Asia with fewer than 50M parameters. Our model adapts a CLIP-KD-style framework to Southeast Asian multilingual settings through regional data curation and multilingual teacher guidance. Experiments across seven Southeast Asian languages show that SEA-CLIP-Tiny achieves the strongest average retrieval performance among the evaluated student models, reaching 12.9%, 31.5%, and 42.2% at R@1, R@5, and R@10, respectively. Compared with MobileCLIP2, it improves average R@10 by 12.1 points while using 38.4% fewer parameters and lower measured CPU latency. These results highlight the importance of region-aware training for efficient multilingual text-vision models in Southeast Asia.
Puja Ahmad Habibi, Faiz Assabil Firdaus, Ashvanth S +3
SEACrowd · University of Indonesia · Cohere Labs Community +4
Contrastive Language-Image Pre-training (CLIP) relies on Vision Transformers whose attention mechanism is susceptible to spurious correlations, and scales quadratically with resolution. To address these limitations, We present CLIMP, the first fully Mamba-based contrastive vision-language model that replaces both the vision and text encoders with Mamba. The new architecture encodes sequential structure in both vision and language, with VMamba capturing visual spatial inductive biases, reducing reliance on spurious correlations and producing an embedding space favorable for cross-modal retrieval and out-of-distribution robustness-surpassing OpenAI's CLIP-ViT-B by 7.5% on ImageNet-O. CLIMP naturally supports variable input resolutions without positional encoding interpolation or specialized training, achieving up to 6.6% higher retrieval accuracy at 16x training resolution while using 5x less memory and 1.8x fewer FLOPs. The autoregressive text encoder further overcomes CLIP's fixed context limitation, enabling dense captioning retrieval. Our findings suggest that Mamba exhibits advantageous properties for vision-language learning, making it a compelling alternative to Transformer-based CLIP.The code and models are publicly available at https://github.com/NimrodShabtay/CLIMP}
Vision-language models such as CLIP embed images and text in a shared space, where modality-specific distributions often remain separated. Existing accounts connect this modality gap to initialization, contrastive dynamics, and information imbalance, while its distributional and pairwise contributions to retrieval remain unresolved. We introduce UOT-Gap, a training-free variational diagnostic that models frozen image and text embeddings with unbalanced entropic optimal transport (UOT). The UOT optimum separates transport, coupling complexity, and marginal mass variation; a complementary pair-aware residual compares observed image-caption pairs with the UOT soft matching. On Flickr8K and COCO-1K with frozen CLIP, OpenCLIP, and SigLIP encoders, caption degradation reduces Flickr8K Recall@1 from 0.559 to 0.003. Across six dataset-model conditions, the pair-aware residual tracks retrieval degradation with mean absolute Spearman 0.973, compared with 0.392 for the mean gap. The association remains stable across five random COCO-1K subsets at 0.954±0.026, with a minimum of 0.943. UOT barycentric updates reduce the transport objective while degrading retrieval, distinguishing geometric objective descent from task improvement. These results establish UOT-Gap as a diagnostic for caption quality, modality alignment, and retrieval robustness.