Organizations: SEACrowd · University of Indonesia · Cohere Labs Community · Technical University of Denmark · Department of Computer Engineering, Faculty of Engineering, Chulalongkorn University · Vidyasirimedhi Institute of Science and Technology · AI Singapore
Multilingual text-vision embedding models are essential for cross-lingual image-text retrieval, but Southeast Asian languages remain poorly supported due to the region's linguistic diversity and limited data and computing resources. In this paper, we introduce SEA-CLIP-Tiny, a compact multilingual text-vision embedding model for Southeast Asia with fewer than 50M parameters. Our model adapts a CLIP-KD-style framework to Southeast Asian multilingual settings through regional data curation and multilingual teacher guidance. Experiments across seven Southeast Asian languages show that SEA-CLIP-Tiny achieves the strongest average retrieval performance among the evaluated student models, reaching 12.9%, 31.5%, and 42.2% at R@1, R@5, and R@10, respectively. Compared with MobileCLIP2, it improves average R@10 by 12.1 points while using 38.4% fewer parameters and lower measured CPU latency. These results highlight the importance of region-aware training for efficient multilingual text-vision models in Southeast Asia.
Figures & tables
Figure 1: Performance-size comparison of compact CLIP models on Southeast Asian image-text retrieval. (a) Language-average R@10 is computed from mean(I2T, T2I) retrieval scores over the available retrieval benchmarks for each of the seven SEA languages. (b) Task-average R@10 is the mean retrieval R@10 over XM3600, Flickr30k-200, and XTD-200.
Figure 2: SEA languages and countries covered in this work, amounting to 569M speakers in the SEA region.
Figure 3: The overview of SEA-CLIP-Tiny . We split the module into two major components: data selection and training framework.
Dataset
Indonesian
Javanese
Malay
Sundanese
Myanmar
Thai
Vietnamese
Total
CG Open-Ended (filtered) [ 19 ]
223,098
39,747
161,397
21,238
—
58,433
199,562
703,475
WIT-HF [ 21 ]
146,744
11,926
59,504
—
9,941
55,594
203,979
487,688
BLOOM [ 14 ] †
2,169
—
102
—
419
3,009
22
21,534
SEA-Mammoth [ 3 ] ‡
54,040
—
54,040
—
54,040
54,040
54,040
540,400
Total
426,051
51,673
275,043
21,238
64,400
171,076
457,603
1,753,097
Table 1: Multilingual training data sources used for SEA-CLIP-Tiny , reported as the number of training examples per target language. The Total column includes all examples used from each dataset, including additional languages not shown in the table. † BLOOM covers 76 languages in total. ‡ SEA-Mammoth covers 10 languages in total; zh, ta, lo, km, and fil are included only in the Total column.
Model
#Params (M)
CPU ms
Thai
Myanmar
Malay
Indonesian
Javanese
Sundanese
Vietnamese
Avg
@1
@5
@10
@1
@5
@10
@1
@5
@10
@1
@5
@10
@1
@5
@10
@1
@5
@10
@1
@5
@10
@1
@5
@10
Teacher
86.19+499.77
90.5
48.7
75.5
84.7
25.3
49.1
60.6
56.1
81.9
89.5
59.9
84.2
91.2
27.4
51.8
63.6
24.5
47.6
58.8
57.1
83.5
90.7
42.7
67.6
77.0
CLIP-KD
5.62+40.49
24.1
0.2
0.6
1.0
0.4
1.3
1.9
1.9
5.2
7.5
2.6
6.9
9.7
2.9
8.6
12.6
4.0
9.9
15.2
0.6
1.6
2.4
1.8
4.9
7.2
TinyCLIP
8.28+15.17
14.9
0.1
0.5
1.0
0.7
1.9
2.5
2.5
6.1
8.9
3.0
8.2
11.4
3.7
9.3
13.4
4.4
11.2
16.5
1.3
3.6
5.2
2.2
5.8
8.4
MobileCLIP2
11.41+63.43
61.6
0.9
1.9
2.7
1.7
3.8
4.8
20.9
40.6
50.8
30.1
53.4
64.1
13.6
30.4
39.8
14.0
29.7
39.5
3.3
7.2
9.3
12.1
23.9
30.1
SEA-CLIP-Tiny
5.62+40.49
24.1
8.6
24.1
33.9
3.6
12.1
18.7
22.6
49.3
63.2
22.5
49.0
62.5
9.8
25.4
35.8
7.1
20.8
29.2
16.4
39.6
52.4
12.9
31.5
42.2
Table 2: Per-language image-text retrieval performance, reported as %. For each language, we report mean(I2T, T2I) R@ k , averaged over the retrieval benchmarks covering that language. CPU ms denotes median single-sample CPU latency for encoding one image and one caption; full computational measurements are reported in Tables 10 and 11 . Bold denotes the best student model in each column.
Model
#Params (M)
CPU ms
Retrieval (R@ k )
Classification (acc@ k )
VQA
XM3600
Flickr30k-200
XTD-200
Avg
Babel-IN
CVQA
@1
@5
@10
@1
@5
@10
@1
@5
@10
@1
@5
@10
@1
@5
@10
acc@1
Teacher
86.19+499.77
90.5
55.7
81.4
88.8
46.0
69.3
77.0
39.2
65.8
77.0
47.0
72.2
81.0
42.3
61.3
67.8
48.6
CLIP-KD
5.62+40.49
24.1
1.1
2.6
3.5
1.5
3.7
5.4
2.2
6.2
9.4
1.6
4.2
6.1
4.2
7.7
10.5
25.9
TinyCLIP
8.28+15.17
14.9
1.6
3.7
4.9
2.1
4.7
6.8
2.4
7.1
10.5
2.0
5.2
7.4
4.4
8.2
10.8
26.8
MobileCLIP2
11.41+63.43
61.6
11.7
20.8
24.6
11.2
22.0
27.5
12.9
25.8
33.1
11.9
22.8
28.4
15.6
23.5
27.5
33.5
Table 3: Per-task SEA performance, reported as %. Retrieval reports mean(I2T, T2I) R@ k on XM3600, Flickr30k-200, and XTD-200, with Avg computed only across these three retrieval benchmarks. Classification reports Babel-IN zero-shot acc@ k , and VQA reports CVQA acc@1 on the LOCAL split. Bold denotes the best student model in each column.
Figure 4: Human preference evaluation comparing SEA-CLIP-Tiny against MobileCLIP2 across four Southeast Asian languages (Indonesian, Javanese, Thai, Sundanese). Each bar shows the number of questions (out of 100) where SEA-CLIP wins (blue), ties (yellow), or MobileCLIP wins (red), per annotator and on average.
Table 4: Controlled ablations of training data and distillation objectives for SEA-CLIP-Tiny , reported as %. All variants use the same student architecture and hyperparameters. Retrieval Avg R@1 is the mean retrieval R@1 over XM3600, Flickr30k-200, and XTD-200, consistent with Table 3 . Note that we use the same multilingual teacher model, ViT-B-16-worldwide@WorldWideCLIP .
Model
#Params (M)
CPU ms
Thai
Myanmar †
Malay †
Indonesian
Javanese †
Sundanese †
Vietnamese
Avg
@1
@5
@10
@1
@5
@10
@1
@5
@10
@1
@5
@10
@1
@5
@10
@1
@5
@10
@1
@5
@10
@1
@5
@10
Teacher
86.19+499.77
90.5
50.7
77.2
86.4
27.2
51.1
62.3
57.4
82.8
90.5
61.3
85.7
91.9
28.1
52.7
63.9
24.6
48.2
59.1
59.2
85.5
91.9
44.1
69.0
78.0
CLIP-KD ∗
5.62+40.49
24.1
0.2
0.7
1.1
0.8
1.7
2.2
2.4
6.2
8.8
3.0
8.2
11.4
3.6
10.0
14.4
4.9
11.4
17.1
1.0
2.6
3.7
2.3
5.8
8.4
TinyCLIP
8.28+15.17
14.9
0.2
0.6
0.8
1.2
2.4
3.3
3.4
7.8
10.8
3.9
10.1
13.6
4.7
11.7
15.9
5.5
13.3
18.6
1.8
4.9
6.7
3.0
7.2
10.0
MobileCLIP2
11.41+63.43
61.6
1.2
2.5
3.6
2.7
5.2
6.1
24.1
46.1
56.8
34.4
60.2
71.2
16.7
35.5
45.7
15.8
33.6
43.3
4.3
9.2
11.9
14.2
27.5
34.1
SEA-CLIP-Tiny
5.62+40.49
24.1
9.1
25.5
35.4
3.8
13.2
19.7
23.7
51.9
66.2
23.6
50.1
63.7
10.7
26.2
36.6
7.5
21.6
30.2
17.4
41.6
54.6
13.7
32.9
43.7
Table 5: Per-language performance (mean across XM3600, Flickr30k-200, XTD-200 I2T R@k), reported as %. Bold = best among student models per column. † Averaged over Flickr30k-200 and XTD-200 only; XM3600 does not cover these languages. ∗ Retrained by on CC12M following CLIPKD’s scripts.
Model
#Params (M)
CPU ms
Retrieval (R@ k )
Classification (acc@ k )
VQA
XM3600
Flickr30k-200
XTD-200
Avg
Babel-IN
CVQA
@1
@5
@10
@1
@5
@10
@1
@5
@10
@1
@5
@10
@1
@5
@10
acc@1
Teacher
86.19+499.77
90.5
58.9
84.4
90.9
47.1
70.6
78.0
40.3
66.8
77.6
48.7
73.9
82.2
42.3
61.3
67.8
48.6
CLIP-KD ∗
5.62+40.49
24.1
1.5
3.4
4.4
1.8
4.4
6.3
2.7
7.4
10.9
2.0
5.1
7.2
4.2
7.7
10.5
25.9
TinyCLIP
8.28+15.17
14.9
2.1
4.7
5.9
2.7
5.8
8.0
3.1
8.9
12.4
2.6
6.5
8.8
4.4
8.2
10.8
26.8
MobileCLIP2
11.41+63.43
61.6
14.2
24.2
28.5
13.2
25.9
31.8
14.8
28.9
36.6
14.0
26.3
32.3
15.6
23.5
27.5
33.5
Table 6: Per-task SEA performance, reported as %. Babel-IN = acc@ k (SEA-lang avg); XM3600, Flickr30k-200, XTD-200 = I2T (image → text) R@ k ; CVQA = acc@1 (LOCAL, SEA-7 subset). Retrieval Avg = mean(Babel-IN, XM3600, Flickr30k-200, XTD-200). Total Avg = equal-weight mean over all 5 tasks. Bold = best student per column.
Model
#Params (M)
CPU ms
Thai
Myanmar †
Malay †
Indonesian
Javanese †
Sundanese †
Vietnamese
Avg
@1
@5
@10
@1
@5
@10
@1
@5
@10
@1
@5
@10
@1
@5
@10
@1
@5
@10
@1
@5
@10
@1
@5
@10
Teacher
86.19+499.77
90.5
46.7
73.8
82.9
23.4
47.0
58.9
54.9
81.0
88.5
58.5
82.6
90.4
26.7
50.9
63.4
24.4
47.1
58.6
55.0
81.4
89.5
41.4
66.3
76.0
CLIP-KD ∗
5.62+40.49
24.1
0.1
0.6
1.0
0.1
0.8
1.7
1.5
4.2
6.2
2.1
5.5
7.9
2.2
7.3
10.9
3.2
8.4
13.4
0.2
0.6
1.1
1.3
3.9
6.0
TinyCLIP
8.28+15.17
14.9
0.0
0.4
1.1
0.3
1.4
1.8
1.5
4.5
7.1
2.2
6.3
9.1
2.6
6.8
10.8
3.3
9.1
14.3
0.9
2.3
3.8
1.5
4.4
6.9
MobileCLIP2
11.41+63.43
61.6
0.6
1.3
1.8
0.8
2.4
3.5
17.7
35.1
44.8
25.8
46.6
56.9
10.4
25.3
33.9
12.2
25.9
35.6
2.3
5.3
6.8
10.0
20.3
26.2
SEA-CLIP-Tiny
5.62+40.49
24.1
8.0
22.6
32.4
3.3
11.0
17.7
21.6
46.7
60.2
21.3
48.0
61.3
8.9
24.6
35.1
6.6
19.9
28.2
15.4
37.6
50.2
12.2
30.0
40.7
Table 7: Per-language performance (mean across Babel-IN = acc@k, XM3600, Flickr30k-200, XTD-200 T2I (text → image) R@ k ), reported as %. Bold = best among student models per column. † Averaged over Flickr30k-200 and XTD-200 only; XM3600 does not cover these languages. ∗ Retrained by on CC12M following CLIPKD’s scripts.
Model
#Params (M)
CPU ms
Retrieval (R@ k )
Classification (acc@ k )
VQA
XM3600
Flickr30k-200
XTD-200
Avg
Babel-IN
CVQA
@1
@5
@10
@1
@5
@10
@1
@5
@10
@1
@5
@10
@1
@5
@10
acc@1
Teacher
86.19+499.77
90.5
52.6
78.5
86.8
45.0
68.0
76.1
38.1
64.8
76.4
45.2
70.5
79.7
42.3
61.3
67.8
48.6
CLIP-KD ∗
5.62+40.49
24.1
0.7
1.8
2.5
1.1
2.9
4.4
1.7
5.1
8.0
1.1
3.3
5.0
4.2
7.7
10.5
25.9
TinyCLIP
8.28+15.17
14.9
1.0
2.8
3.8
1.5
3.6
5.6
1.6
5.2
8.6
1.4
3.9
6.0
4.4
8.2
10.8
26.8
MobileCLIP2
11.41+63.43
61.6
9.2
17.3
20.8
9.2
18.0
23.2
10.9
22.7
29.6
9.8
19.3
24.5
15.6
23.5
27.5
33.5
Table 8: Per-task SEA performance, reported as %. Babel-IN = acc@ k (SEA-lang avg); XM3600, Flickr30k-200, XTD-200 = T2I (text → image) R@ k ; CVQA = acc@1 (LOCAL, SEA-7 subset). Retrieval Avg = mean(Babel-IN, XM3600, Flickr30k-200, XTD-200). Total Avg = equal-weight mean over all 5 tasks. Bold = best student per column.
Figure 5: Human preference evaluation on XTD-200 only. Each language contains 50 samples. Bars show the number of questions where SEA-CLIP-Tiny wins, ties with MobileCLIP2, or MobileCLIP2 wins, reported per annotator and on average.
Figure 6: Human preference evaluation on Flickr30k-200 only. Each language contains 50 samples. Bars show the number of questions where SEA-CLIP-Tiny wins, ties with MobileCLIP2, or MobileCLIP2 wins, reported per annotator and on average.
Figure 7: Example interface for the human preference evaluation. Annotators compare two anonymized captions for the same image and select the better caption, or mark a tie when both captions are equally acceptable or equally incorrect.
Benchmark
Task
Metric
Provenance
SEA language coverage
n /lang
Tha
Mya
Mal
Ind
Jav
Sun
Vie
Babel-ImageNet
Zero-shot classification
acc@ k
BabelNet lexical mapping
✓
✓
✓
✓
✓
✓
✓
98–896
XM3600
Image–text retrieval
R@ k
Native-speaker annotation
✓
–
–
✓
–
–
✓
3,600
Flickr30k-200
Image–text retrieval
R@ k
Machine translation (NLLB)
✓
✓
✓
✓
✓
✓
✓
1,000
XTD-200
Image–text retrieval
R@ k
Machine translation (NLLB)
✓
✓
✓
✓
✓
✓
✓
1,000
CVQA
Visual question answering
acc@1
Native-speaker authored
no per-language split released
–
Table 9: Coverage and provenance of the evaluation benchmarks. XM3600 covers Thai, Indonesian, and Vietnamese; therefore, the per-language retrieval averages for these languages in Table 2 are computed over three benchmarks, while Malay, Javanese, Sundanese, and Myanmar are averaged over Flickr30k-200 and XTD-200. Flickr30k-200 and XTD-200 rely on machine-translated text rather than native annotation, which is an important limitation for evaluating lower-resource languages. Babel-ImageNet also has uneven test-set sizes across languages. Filipino, Khmer, and Lao occur in our training mixture but are not covered by these evaluation sets and are therefore excluded from the main language-level evaluation. n /lang denotes the number of evaluation examples per language.
Table 10: Efficiency comparison on a single NVIDIA A100-SXM4-40GB. GFLOPs are measured per sample for one forward pass using torch.utils.flop_counter (1 MAC = 2 FLOPs) at each model’s native input resolution and context length. Latency is measured at batch size 1 after 20 warm-up iterations and 100 timed runs, while throughput and peak memory are measured at batch size 1,024. Parameters include the final projection heads and token-embedding tables.
Table 11: Single-sample CPU inference latency measured in fp32 at batch size 1 on an AMD EPYC 7713 with OMP_NUM_THREADS=8 . We report the median of 30 runs after 5 warm-up iterations. FLOPs are reported using CUDA profiling in Table 10 , since the CPU fused-attention implementation is not fully captured by our FLOP counter.
Table 12: Parameter breakdown of the text and image towers. We report the token-embedding table separately because published compact CLIP models use different parameter-counting conventions. TinyCLIP excludes the text token-embedding table from its reported text-encoder size; under its published convention, our implementation reproduces the reported parameter counts. For consistency, our efficiency comparisons include the embedding table and final projection heads for all models.
Tokenizer
Vocab.
Thai
Myanmar
Khmer
Lao
VI Latin
SEA-script total
Byte frag.
Student CLIP-BPE
49,408
51
0
0
0
59
51
532
Teacher XLM-V-Base
901,629
14,243
4,214
3,029
2,112
58,773
23,598
–
Table 13: Tokenizer vocabulary coverage of Southeast Asian scripts. For each tokenizer, we count vocabulary entries containing at least one character from the corresponding Unicode script. VI Latin counts entries containing Vietnamese-specific Latin characters with diacritics. SEA-script total is the union of Thai, Myanmar, Khmer, and Lao script entries and therefore excludes Latin-script entries. Byte frag. denotes student vocabulary entries that do not decode to a valid UTF-8 character independently.
Benchmark
Language
n
Chars/text
Mean
Median
P95
Max
Trunc. (%)
XM3600
English
3,600
51.9
13.0
13
20
31
0.0
Indonesian
3,600
93.3
31.4
29
55
95
0.1
Vietnamese
3,600
74.9
68.4
66
116
193
33.3
Thai
3,600
54.0
59.2
59
100
163
26.1
Flickr30k-200
English
1,000
94.6
22.3
20
35
86
0.1
Indonesian
1,000
106.0
37.1
34
61
141
1.5
Table 14: Student CLIP-BPE tokenization statistics by benchmark and language. Token lengths are measured before truncation and include start- and end-of-text tokens. n denotes the number of evaluated text inputs. P95 and Max summarize the pre-truncation token-length distribution, and Trunc. is the percentage of inputs exceeding the context length of 77 tokens.
Language
Student CLIP-BPE
Teacher XLM-V-Base
Tok/cap
Tok/char
Trunc
Frag
Tok/cap
Tok/char
Trunc
unk
English
17.0
0.21
0.0
0.0
17.3
0.25
0.0
0.00
Indonesian
30.9
0.32
0.6
0.0
17.2
0.19
0.0
0.00
Javanese
28.4
0.33
0.6
0.0
18.9
0.24
0.1
0.00
Malay
31.6
0.33
0.9
0.0
17.4
0.20
0.0
0.00
Sundanese
30.1
0.34
0.8
0.0
20.9
0.25
0.1
0.00
Table 15: Caption statistics per language, macro-averaged over the three retrieval benchmarks (XM3600, Flickr30k-200, XTD-200) at a context length of 77 tokens. All Trunc , Frag and unk values are percentages. Trunc is the share of captions exceeding the context length; Frag is the share of emitted student tokens that are UTF-8 byte fragments; unk is the share of emitted teacher tokens that are the unknown token.
Text embeddings are fundamental to many downstream applications, making robustness important for real-world NLP. However, most recent state-of-the-art embedding models are not reproducible because they rely on closed or undisclosed training data, and they remain insufficiently robust for Southeast Asian languages. We present SEA-LION-Embedding, a fully open and reproducible text-embedding pipeline for Southeast Asian languages trained only on publicly available data, and use it to study three core factors of robust embedding design: data composition, training objective, and base encoder initialization. SEA-LION-Embedding achieves state-of-the-art results on SEA-BED while enabling systematic and reproducible analysis of robust text embeddings for the region.
Peerat Limkonchotiwat, Raymond Ng, Sarana Nutanong +1
AI Singapore · School of Information Science and Technology, VISTEC
Image-text retrieval has become a fundamental component in intelligent multimedia systems; however, most existing vision-language models are optimized for highresource languages and remain suboptimal for low-resource settings such as Vietnamese. This work introduces ViCLIP-OT, a foundation vision-language model specifically designed for Vietnamese image-text retrieval. The proposed framework integrates CLIP-style contrastive learning with a Similarity-Graph Regularized Optimal Transport (SIGROT) loss to enhance global cross-modal consistency and mitigate modality gap issues. Extensive experiments on three Vietnamese benchmarks (UITOpenViIC, KTVIC, and Crossmodal-3600) demonstrate that ViCLIP-OT consistently outperforms CLIP and SigLIP baselines in both in-domain and zero-shot settings. On UIT-OpenViIC, the model achieves an average Recall@K of 67.34%, improving upon CLIP by 5.75 percentage points. In zero-shot evaluation on Crossmodal-3600, ViCLIPOT surpasses CLIP by 11.72 percentage points. Embedding-space analysis further confirms improved alignment and reduced modality gap. The results indicate that integrating SIGROT provides an effective and scalable strategy for cross-modal retrieval in low-resource languages, offering practical implications for intelligent multimedia retrieval systems in Vietnamese and other underrepresented linguistic contexts.
We introduce Nemotron-SEA-LION-v4.8, a family of Southeast Asian Languages In One Network (SEA-LION) models built upon NVIDIA Nemotron 3. The family includes 30B-A3B and 120B-A12B models, with both continued-pretrained base checkpoints and post-trained variants. We adapt the models using Southeast Asian, reasoning, code, and multilingual parallel data, followed by post-training with supervised fine-tuning and online on-policy distillation. On SEA-HELM, the 30B-A3B model improves the overall SEA score from 46.06 to 51.57, while the 120B-A12B model improves from 49.30 to 63.44. Across seven Southeast Asian languages, we observe broad capability gains with the 120B-A12B model showing broader and more consistent improvements across tasks.