We introduce V-SONAR, a vision-language embedding space extended from the text-only embedding space SONAR (Omnilingual Embeddings Team et al., 2026), which supports 1500 text languages and 177 speech languages. To construct V-SONAR, we propose a post-hoc alignment pipeline that maps the representations of an existing vision encoder into the SONAR space. We thoroughly evaluate V-SONAR and show that its embeddings achieve competitive performance on text-to-video retrieval. Equipped with the OMNISONAR text decoder, V-SONAR further surpasses state-of-the-art vision-language models on video captioning tasks, including DREAM-1K (BLEU 23.9 vs. 19.6) and PE-VIDEO (BLEU 39.0 vs. 30.0). Leveraging V-SONAR, we first demonstrate that the Large Concept Model (LCM; LCM team et al. 2024) operating in SONAR and trained with English text only, can perform both single- and multi-visual concept understanding in a zero-shot manner. Finally, we introduce V-LCM, which extends the LCM with vision-language instruction tuning. V-LCM encodes vision and language inputs into an unified sequence of latent embeddings via V-SONAR and SONAR, and it is trained with the same latent diffusion objective for next-embedding prediction as in LCM's text-only pre-training. Experiments on a large-scale multilingual and -modal instruction-tuning data mixture highlight the potential of V-LCM: V-LCM matches state-of-the-art vision-language models on tasks covering image/video captioning and question answering, while significantly outperforming them across 61 rich- to low-resource languages out of all 62 tested languages.
Figures & tables
Figure 1: Left: Illustration of v-Sonar . Right: training v-LCM with vision-language instruction tuning.
model
xsim ↓
xsim++ ↓
Sonar 1
1.37
15.27
OmniSONAR
0.65
6.14
Table 1: Similarity search over 200 languages in Flores .
Method
R@1 ↑
R@5 ↑
R@10 ↑
MRR ↑
AC ↑
V. Trace ↑
V. logdet ↑
T. Trace ↑
T. logdet ↑
PE-Vid
SigLIP2-G-OPT
47.55
71.47
79.41
58.47
0.396
0.393
−1.71\times104
0.582
−1.72\times104
PECoreG
63.91
85.98
91.61
73.77
0.476
0.479
−1.35\times104
0.686
−1.35\times104
v-Sonar
73.03
89.75
93.81
80.50
0.519
0.700
−1.17\times104
2.216
−7.98\times103
DREAM
SigLIP2-G-OPT
61.50
83.50
89.10
71.50
0.263
0.401
−1.79\times104
0.662
−1.75\times104
PECoreG
72.10
89.80
93.60
79.90
0.307
0.495
−1.40\times104
0.639
−1.39\times104
v-Sonar
63.30
84.10
89.00
72.46
0.410
0.559
−1.20\times104
2.523
−8.53\times103
Table 2: Zero-shot Retrieval performance on PE-Video , Dream-1k and Vatex . We report the Recall rate at 1/5/10 and MRR scores. We also report the analytical metrics for the embedding space, including 1) trace reflects overall variance, and 2) log determinant (logdet) approximates volume in the space. Best values for each columns are bolded .
Model
Bleu
R-1
R-2
R-L
BS-P
BS-R
BS-F
PE-Video
InternVL2.5-1B
19.4
32.1
9.0
23.4
31.2
27.3
29.3
InternVL2-1B
24.1
35.8
10.7
25.5
30.8
32.1
31.5
Qwen2-VL-2B-Instruct
29.9
41.7
18.8
31.2
34.8
40.0
37.3
Qwen2.5-VL-3B-Instruct
30.0
41.3
16.1
28.9
30.2
38.6
34.4
PLM-1B
21.5
37.6
11.9
26.6
35.8
26.2
31.0
Table 3: Video captioning performance across PE-Video , Dream-1k and VATEX (English and Chinese). Metrics include Bleu , Rouge (R-1, R-2, R-L), and BERTScore (BS-P, BS-R, BS-F).
Figure 2: Comparison for v-Sonar trained with SONAR version 1 and 2 ( OmniSONAR ) embedding space on three captioning datasets.
MSE
Cos. Sim.
Bleu
R-1
R-2
R-L
BS-P
BS-R
BS
Architecture
Linear Proj.
1.45×10−3
0.694
38.0
49.7
21.6
36.7
47.2
40.1
43.7
Full PE
1.54×10−3
0.672
37.1
48.5
21.3
36.5
46.9
38.8
42.9
+ Async. LR
1.43×10−3
0.700
39.7
51.3
23.3
37.7
48.1
42.1
45.1
+ Norm. Init.
1.39×10−3
0.708
39.8
51.8
24.0
38.5
49.4
42.2
45.8
+ Attn. Pooling
1.39×10−3
0.708
39.8
51.9
24.0
38.5
49.7
42.4
46.0
+ Temporal Attn.
1.39×10−3
0.708
39.8
51.9
24.0
38.5
49.7
42.4
46.1
Table 4: Ablation study in model architecture and the three-stage training pipeline. SV: our second stage curriculum with the synthetic video captioning data. IC: our first stage curriculum with image captioning data.
Video Captioning / Summarization
M3IT Image
M3IT Video
PE-Video
Dream-1k
Vatex
VideoXum
COCO
VIQUAE
VisualMRC
ScienceQA
ActivNetQA
MSRVTT-QA
IVQA
R-L
BS
R-L
BS
R-L
BS
R-L
BS
R-L
R-L
R-L
Acc.
R-L
R-L
R-L
InterVL2
1B
25.5
31.5
17.2
19.5
22.4
42.4
15.3
17.7
12.6
24.0
30.6
53.9
40.6
27.6
39.5
4B
15.0
18.4
12.2
14.7
19.2
42.4
15.6
17.4
17.5
20.0
35.3
89.6
27.5
24.5
31.8
8B
18.6
23.4
16.4
19.4
16.2
42.4
29.1
26.1
21.0
21.6
42.9
87.2
29.7
27.1
38.4
InternVL-2.5
1B
23.4
29.3
15.6
18.6
19.2
40.3
17.1
23.2
13.2
10.8
27.3
69.0
16.6
11.7
19.3
Table 5: Main results on vision-language tasks in M3IT and the previous video benchmarks ( PE-Video , Dream-1k , Vatex , VideoXum).
Figure 3: Operating in v-Sonar space, the LCM performs better than only accepting the textual Sonar inputs. We compare LCM -7B-IFT in VideoXum with Rouge -L scores across short, mid, and long categories of video inputs.
VCR
F1
Sim.
LCM
0.385
0.258
vLCM
0.671
0.529
PLM-8B
0.441
0.432
Qwen-2.5-7B
0.275
0.402
Qwen-2-7B
0.502
0.513
InterVL2.5-8B
0.155
0.158
Table 6: VCR results comparing token-level F1 and semantic similarity.
Figure 4: Performance in 62 languages for v-LCM , Qwen2.5-VL-7B and PLM-8B on M3IT testing set for MSRVTT, MSRVTT-QA, ImageNet, VQA-V2, VIST and OKVQA. We report the Rouge -L scores averaged from all datasets. Detailed results for each dataset can be found in the Appendix E .
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Captioning
Retrieval
Bleu
R-L
BS-F1
R@1
MRR
MSE-only
38.9
37.8
44.9
49.0
60.3
MSE + Contrastive
38.6
37.5
44.5
52.4
63.7
Appendix
Table 7: Ablation study on using only MSE loss vs. adding a contrastive loss. Results are reported on the PE-Video benchmark for captioning and retrieval with a single MLP layer as the connector in v-Sonar .
V. Norm
V. Trace
V. Volume
AC (Cosine)
AC (MSE)
MSE
1.22
0.48
-10007.16
0.41
0.32
MSE+Contrastive
1.30
1.74
-8107.94
0.31
0.13
Appendix
Table 8: Additional statistics for v-Sonar representations on PE-Video. We report the norm (V.Norm), trace (V.Trace), Volume (V.Volume), alignment consistency measured in Cosine Similarity (AC Cosine) and MSE (AC MSE) for video embeddings predicted by v-Sonar .
Dataset
#Samples
Duration (s)
Caption Length (sent.)
Caption Length (words)
PLM-Image-Auto-SA1B
7.99M
–
10.7
181.8
PLM-Image-Auto-OpenImages
1.37M
–
7.9
132.1
PLM-Video-Auto-YT1B
2.14M
22.8
2.3
95.5
PE-Video
118K
16.7
4.4
51.4
Appendix
Table 9: Statistics of the datasets used in 3-stage training for alignment. We report the number of samples, average video duration (if applicable), and average caption length in sentences and words.
R@1
R@5
R@10
MRR
PVD-Bench
SONAR1
0.649
0.843
0.895
0.737
OmniSONAR
0.730
0.898
0.938
0.805
DREAM-1K
SONAR1
0.536
0.759
0.838
0.638
OmniSONAR
0.633
0.841
0.890
0.725
VATEX
SONAR1
0.119
0.263
0.350
0.195
OmniSONAR
0.408
0.686
0.789
0.536
Appendix
Table 10: Comparison for v-Sonar trained with SONAR version 1 and 2 embedding space on three text-to-video retrieval datasets.
Figure 5: M3IT evaluation on 61 languages for MSRVTT.
Figure 6: M3IT evaluation on 61 languages for MSRVTT-QA.
Figure 7: M3IT evaluation on 61 languages for OKVQA.
Figure 8: M3IT evaluation on 61 languages for VIST.
Figure 9: M3IT evaluation on 61 languages for VQA-V2.
Figure 10: M3IT evaluation on 61 languages for image captioning.
Figure 11: Visualization with t-SNE for Sonar and v-Sonar embeddings after each stage of curriculum. v-Sonar encodes the video, and Sonar encodes the caption. We randomly sample 200 samples from PE-Video’s testing set for t-SNE, and explicitly plot the lines for connecting the paired video and caption for 50 samples.
Semantic Sim.
Round-trip Retrieval
Cosine
Dist.
R@1
R@5
R@10
MRR
Groundtruth
0.666
0.197
87.00%
95.90%
97.10%
0.9084
Sonar Decoder
0.689
0.175
82.50%
97.00%
98.70%
0.8883
v-LCM
0.562
0.219
82.30%
96.70%
97.90%
0.8867
Appendix
Table 11: Left: semantic similarities and distances between video embedding and the ground-truth captions, generated captions from Sonar Decoder and v-LCM . Right: results for the round-trip retrieval ablation study where we use the ground-truth captions, Sonar Decoder and v-LCM ’s generated captions to retrieve the videos.
Figure 12: vSONAR’s visualization for cross-modal semantic drift.
Figure 13: vLCM’s visualization for cross-modal semantic drift.
Figure 14: Qualitative examples for image captioning task in M3IT benchmark. Each row shows the input image, the given prompt, and the generated model output.
Figure 15: Qualitative examples for image question answering tasks. Each row shows the input image, the given prompt, and the generated model output.
Figure 16: Qualitative examples for video captioning and question answering tasks. The frames are presented from left to right, and top to the bottom.
Figure 17: Qualitative comparison on fine-grained motion understanding in PE-Video. While state-of-the-art VLMs (Qwen-2.5, InternVL-2.5) suffer from hallucinations or captioning erors (highlighted in red ) such as a “paintbrush” or “yellow hand,” and PLM remains vague, SONAR accurately captures the mechanics of the mixer and the blue cylinder attachment (highlighted in green ).
Figure 18: Qualitative comparison in PE-Video. We highlight the errors in red .
Figure 19: Qualitative comparison in DREAM-1K.
Figure 20: Qualitative comparison in DREAM-1K.
Figure 21: Qualitative comparison in Visual-Commonsense-Reasoning (VCR), which requires grounding and spatial reasoning ability connecting with the commonsense.
Embedding-level scaling expands the capacity of language models through lookup-based memory rather than by increasing model depth, width, or dense computation. Extending this idea to vision-language models (VLMs) is challenging because visual features are continuous, spatially structured, and query-dependent, unlike discrete text tokens that naturally support n-gram lookup. We introduce V-LUMEN, a visual lookup memory for VLMs. V-LUMEN discretizes visual features with a frozen codebook, builds spatially aware memory keys through spatial aggregation-based hashing, and modulates them with text-conditioned relevance signals; all addressing is non-parametric, and the retrieved embeddings are fused into the backbone through a gated residual pathway. On five reasoning-intensive benchmarks, V-LUMEN substantially improves a frozen Qwen3-VL-2B backbone and transfers zero-shot to three benchmarks whose images are disjoint from the training data. At a matched parameter budget, moving part of a LoRA or MoE adapter's capacity into V-LUMEN preserves accuracy on two backbones, indicating that external visual memory is an independent capacity axis rather than a stronger adaptation method. Because lookup cost is constant in table size and incurred only once at prefill, this capacity grows in storage rather than in per-token compute. Code is available at https://github.com/miso-choi/V-Lumen.
Miso Choi, Daekeun Kim, Eunji Kim +1
Dept. of Computer Science and Engineering, Korea University · Amazon
Vision-language model (VLM) encoders such as CLIP enable strong retrieval and zero-shot classification in a shared image-text embedding space, yet the semantic organization of this space is rarely inspected. We present a post-hoc framework to explain, verify, and align the semantic hierarchies induced by a VLM over a given set of child classes. First, we extract a binary hierarchy by agglomerative clustering of class centroids and name internal nodes by dictionary-based matching to a concept bank. Second, we quantify plausibility by comparing the extracted tree against human ontologies using efficient tree- and edge-level consistency measures, and we evaluate utility via explainable hierarchical tree-traversal inference with uncertainty-aware early stopping (UAES). Third, we propose an ontology-guided post-hoc alignment method that learns a lightweight embedding-space transformation, using UMAP to generate target neighborhoods from a desired hierarchy. Across 13 pretrained VLMs and 4 image datasets, our method finds systematic modality differences: image encoders are more discriminative, while text encoders induce hierarchies that better match human taxonomies. Overall, the results reveal a persistent trade-off between zero-shot accuracy and ontological plausibility and suggest practical routes to improve semantic alignment in shared embedding spaces.
Gesina Schwalbe, Mert Keser, Moritz Bayerkuhnlein +8
Ulm University, Germany · University of Lübeck, Germany · Technical University of Munich, Germany +4
Recent vision-language models (VLMs) typically rely on a single vision encoder trained with contrastive image-text objectives, such as CLIP-style pretraining. While contrastive encoders are effective for cross-modal alignment and retrieval, self-supervised visual encoders often capture richer dense semantics and exhibit stronger robustness on recognition and understanding tasks. In this work, we investigate how to scale the fusion of these complementary visual representations for vision-language modeling. We propose CoME-VL: Complementary Multi-Encoder Vision-Language, a modular fusion framework that integrates a contrastively trained vision encoder with a self-supervised DINO encoder. Our approach performs representation-level fusion by (i) entropy-guided multi-layer aggregation with orthogonality-constrained projections to reduce redundancy, and (ii) RoPE-enhanced cross-attention to align heterogeneous token grids and produce compact fused visual tokens. The fused tokens can be injected into a decoder-only LLM with minimal changes to standard VLM pipelines. Extensive experiments across diverse vision-language benchmarks demonstrate that CoME-VL consistently outperforms single-encoder baselines. In particular, we observe an average improvement of 4.9% on visual understanding tasks and 5.4% on grounding tasks. Our method achieves state-of-the-art performance on RefCOCO for detection while improving over the baseline by a large margin. Finally, we conduct ablation studies on layer merging, non-redundant feature mixing, and fusion capacity to evaluate how complementary contrastive and self-supervised signals affect VLM performance.
Ankan Deria, Komal Kumar, Xilin He +4
Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, UAE