We introduce V-SONAR, a vision-language embedding space extended from the text-only embedding space SONAR (Omnilingual Embeddings Team et al., 2026), which supports 1500 text languages and 177 speech languages. To construct V-SONAR, we propose a post-hoc alignment pipeline that maps the representations of an existing vision encoder into the SONAR space. We thoroughly evaluate V-SONAR and show that its embeddings achieve competitive performance on text-to-video retrieval. Equipped with the OMNISONAR text decoder, V-SONAR further surpasses state-of-the-art vision-language models on video captioning tasks, including DREAM-1K (BLEU 23.9 vs. 19.6) and PE-VIDEO (BLEU 39.0 vs. 30.0). Leveraging V-SONAR, we first demonstrate that the Large Concept Model (LCM; LCM team et al. 2024) operating in SONAR and trained with English text only, can perform both single- and multi-visual concept understanding in a zero-shot manner. Finally, we introduce V-LCM, which extends the LCM with vision-language instruction tuning. V-LCM encodes vision and language inputs into an unified sequence of latent embeddings via V-SONAR and SONAR, and it is trained with the same latent diffusion objective for next-embedding prediction as in LCM's text-only pre-training. Experiments on a large-scale multilingual and -modal instruction-tuning data mixture highlight the potential of V-LCM: V-LCM matches state-of-the-art vision-language models on tasks covering image/video captioning and question answering, while significantly outperforming them across 61 rich- to low-resource languages out of all 62 tested languages.
Figures & tables
Figure 1: Left: Illustration of v-Sonar . Right: training v-LCM with vision-language instruction tuning.
model
xsim ↓
xsim++ ↓
Sonar 1
1.37
15.27
OmniSONAR
0.65
6.14
Table 1: Similarity search over 200 languages in Flores .
Method
R@1 ↑
R@5 ↑
R@10 ↑
MRR ↑
AC ↑
V. Trace ↑
V. logdet ↑
T. Trace ↑
T. logdet ↑
PE-Vid
SigLIP2-G-OPT
47.55
71.47
79.41
58.47
0.396
0.393
−1.71\times104
0.582
−1.72\times104
PECoreG
63.91
85.98
91.61
73.77
0.476
0.479
−1.35\times104
0.686
−1.35\times104
v-Sonar
73.03
89.75
93.81
80.50
0.519
0.700
−1.17\times104
2.216
−7.98\times103
DREAM
SigLIP2-G-OPT
61.50
83.50
89.10
71.50
0.263
0.401
−1.79\times104
0.662
−1.75\times104
PECoreG
72.10
89.80
93.60
79.90
0.307
0.495
−1.40\times104
0.639
−1.39\times104
v-Sonar
63.30
84.10
89.00
72.46
0.410
0.559
−1.20\times104
2.523
−8.53\times103
Table 2: Zero-shot Retrieval performance on PE-Video , Dream-1k and Vatex . We report the Recall rate at 1/5/10 and MRR scores. We also report the analytical metrics for the embedding space, including 1) trace reflects overall variance, and 2) log determinant (logdet) approximates volume in the space. Best values for each columns are bolded .
Model
Bleu
R-1
R-2
R-L
BS-P
BS-R
BS-F
PE-Video
InternVL2.5-1B
19.4
32.1
9.0
23.4
31.2
27.3
29.3
InternVL2-1B
24.1
35.8
10.7
25.5
30.8
32.1
31.5
Qwen2-VL-2B-Instruct
29.9
41.7
18.8
31.2
34.8
40.0
37.3
Qwen2.5-VL-3B-Instruct
30.0
41.3
16.1
28.9
30.2
38.6
34.4
PLM-1B
21.5
37.6
11.9
26.6
35.8
26.2
31.0
Table 3: Video captioning performance across PE-Video , Dream-1k and VATEX (English and Chinese). Metrics include Bleu , Rouge (R-1, R-2, R-L), and BERTScore (BS-P, BS-R, BS-F).
Figure 2: Comparison for v-Sonar trained with SONAR version 1 and 2 ( OmniSONAR ) embedding space on three captioning datasets.
MSE
Cos. Sim.
Bleu
R-1
R-2
R-L
BS-P
BS-R
BS
Architecture
Linear Proj.
1.45×10−3
0.694
38.0
49.7
21.6
36.7
47.2
40.1
43.7
Full PE
1.54×10−3
0.672
37.1
48.5
21.3
36.5
46.9
38.8
42.9
+ Async. LR
1.43×10−3
0.700
39.7
51.3
23.3
37.7
48.1
42.1
45.1
+ Norm. Init.
1.39×10−3
0.708
39.8
51.8
24.0
38.5
49.4
42.2
45.8
+ Attn. Pooling
1.39×10−3
0.708
39.8
51.9
24.0
38.5
49.7
42.4
46.0
+ Temporal Attn.
1.39×10−3
0.708
39.8
51.9
24.0
38.5
49.7
42.4
46.1
Table 4: Ablation study in model architecture and the three-stage training pipeline. SV: our second stage curriculum with the synthetic video captioning data. IC: our first stage curriculum with image captioning data.
Video Captioning / Summarization
M3IT Image
M3IT Video
PE-Video
Dream-1k
Vatex
VideoXum
COCO
VIQUAE
VisualMRC
ScienceQA
ActivNetQA
MSRVTT-QA
IVQA
R-L
BS
R-L
BS
R-L
BS
R-L
BS
R-L
R-L
R-L
Acc.
R-L
R-L
R-L
InterVL2
1B
25.5
31.5
17.2
19.5
22.4
42.4
15.3
17.7
12.6
24.0
30.6
53.9
40.6
27.6
39.5
4B
15.0
18.4
12.2
14.7
19.2
42.4
15.6
17.4
17.5
20.0
35.3
89.6
27.5
24.5
31.8
8B
18.6
23.4
16.4
19.4
16.2
42.4
29.1
26.1
21.0
21.6
42.9
87.2
29.7
27.1
38.4
InternVL-2.5
1B
23.4
29.3
15.6
18.6
19.2
40.3
17.1
23.2
13.2
10.8
27.3
69.0
16.6
11.7
19.3
Table 5: Main results on vision-language tasks in M3IT and the previous video benchmarks ( PE-Video , Dream-1k , Vatex , VideoXum).
Figure 3: Operating in v-Sonar space, the LCM performs better than only accepting the textual Sonar inputs. We compare LCM -7B-IFT in VideoXum with Rouge -L scores across short, mid, and long categories of video inputs.
VCR
F1
Sim.
LCM
0.385
0.258
vLCM
0.671
0.529
PLM-8B
0.441
0.432
Qwen-2.5-7B
0.275
0.402
Qwen-2-7B
0.502
0.513
InterVL2.5-8B
0.155
0.158
Table 6: VCR results comparing token-level F1 and semantic similarity.
Figure 4: Performance in 62 languages for v-LCM , Qwen2.5-VL-7B and PLM-8B on M3IT testing set for MSRVTT, MSRVTT-QA, ImageNet, VQA-V2, VIST and OKVQA. We report the Rouge -L scores averaged from all datasets. Detailed results for each dataset can be found in the Appendix E .
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Captioning
Retrieval
Bleu
R-L
BS-F1
R@1
MRR
MSE-only
38.9
37.8
44.9
49.0
60.3
MSE + Contrastive
38.6
37.5
44.5
52.4
63.7
Appendix
Table 7: Ablation study on using only MSE loss vs. adding a contrastive loss. Results are reported on the PE-Video benchmark for captioning and retrieval with a single MLP layer as the connector in v-Sonar .
V. Norm
V. Trace
V. Volume
AC (Cosine)
AC (MSE)
MSE
1.22
0.48
-10007.16
0.41
0.32
MSE+Contrastive
1.30
1.74
-8107.94
0.31
0.13
Appendix
Table 8: Additional statistics for v-Sonar representations on PE-Video. We report the norm (V.Norm), trace (V.Trace), Volume (V.Volume), alignment consistency measured in Cosine Similarity (AC Cosine) and MSE (AC MSE) for video embeddings predicted by v-Sonar .
Dataset
#Samples
Duration (s)
Caption Length (sent.)
Caption Length (words)
PLM-Image-Auto-SA1B
7.99M
–
10.7
181.8
PLM-Image-Auto-OpenImages
1.37M
–
7.9
132.1
PLM-Video-Auto-YT1B
2.14M
22.8
2.3
95.5
PE-Video
118K
16.7
4.4
51.4
Appendix
Table 9: Statistics of the datasets used in 3-stage training for alignment. We report the number of samples, average video duration (if applicable), and average caption length in sentences and words.
R@1
R@5
R@10
MRR
PVD-Bench
SONAR1
0.649
0.843
0.895
0.737
OmniSONAR
0.730
0.898
0.938
0.805
DREAM-1K
SONAR1
0.536
0.759
0.838
0.638
OmniSONAR
0.633
0.841
0.890
0.725
VATEX
SONAR1
0.119
0.263
0.350
0.195
OmniSONAR
0.408
0.686
0.789
0.536
Appendix
Table 10: Comparison for v-Sonar trained with SONAR version 1 and 2 embedding space on three text-to-video retrieval datasets.
Figure 5: M3IT evaluation on 61 languages for MSRVTT.
Figure 6: M3IT evaluation on 61 languages for MSRVTT-QA.
Figure 7: M3IT evaluation on 61 languages for OKVQA.
Figure 8: M3IT evaluation on 61 languages for VIST.
Figure 9: M3IT evaluation on 61 languages for VQA-V2.
Figure 10: M3IT evaluation on 61 languages for image captioning.
Figure 11: Visualization with t-SNE for Sonar and v-Sonar embeddings after each stage of curriculum. v-Sonar encodes the video, and Sonar encodes the caption. We randomly sample 200 samples from PE-Video’s testing set for t-SNE, and explicitly plot the lines for connecting the paired video and caption for 50 samples.
Semantic Sim.
Round-trip Retrieval
Cosine
Dist.
R@1
R@5
R@10
MRR
Groundtruth
0.666
0.197
87.00%
95.90%
97.10%
0.9084
Sonar Decoder
0.689
0.175
82.50%
97.00%
98.70%
0.8883
v-LCM
0.562
0.219
82.30%
96.70%
97.90%
0.8867
Appendix
Table 11: Left: semantic similarities and distances between video embedding and the ground-truth captions, generated captions from Sonar Decoder and v-LCM . Right: results for the round-trip retrieval ablation study where we use the ground-truth captions, Sonar Decoder and v-LCM ’s generated captions to retrieve the videos.
Figure 12: vSONAR’s visualization for cross-modal semantic drift.
Figure 13: vLCM’s visualization for cross-modal semantic drift.
Figure 14: Qualitative examples for image captioning task in M3IT benchmark. Each row shows the input image, the given prompt, and the generated model output.
Figure 15: Qualitative examples for image question answering tasks. Each row shows the input image, the given prompt, and the generated model output.
Figure 16: Qualitative examples for video captioning and question answering tasks. The frames are presented from left to right, and top to the bottom.
Figure 17: Qualitative comparison on fine-grained motion understanding in PE-Video. While state-of-the-art VLMs (Qwen-2.5, InternVL-2.5) suffer from hallucinations or captioning erors (highlighted in red ) such as a “paintbrush” or “yellow hand,” and PLM remains vague, SONAR accurately captures the mechanics of the mixer and the blue cylinder attachment (highlighted in green ).
Figure 18: Qualitative comparison in PE-Video. We highlight the errors in red .
Figure 19: Qualitative comparison in DREAM-1K.
Figure 20: Qualitative comparison in DREAM-1K.
Figure 21: Qualitative comparison in Visual-Commonsense-Reasoning (VCR), which requires grounding and spatial reasoning ability connecting with the commonsense.