Omni-Interactive Universal Embedder

Wei-Yao Wang; Kazuya Tateishi; Shuyang Cui; Christian Simon; Takashi Shibuya; Shusuke Takahashi; Yuki Mitsufuji

cs.AIAug 27, 2026

Omni-Interactive Universal Embedder

Authors: Wei-Yao Wang, Kazuya Tateishi, Shuyang Cui, Christian Simon, Takashi Shibuya, Shusuke Takahashi, Yuki Mitsufuji

Organizations: Sony Group Corporation · Sony AI

Abstract

Multimodal representation learning has been shifting from traditional two-tower architectures to large language model (LLM)-based embedders due to their strong instruction-following capabilities. Despite this progress, existing approaches primarily focus on language and image modalities, which also remain the dominant modalities for user-conditioned interactions in current embedders. In this paper, we propose the first Omni-Interactive Universal Embedder (OmniUE), which not only learns a unified embedding space across text, video, and audio by leveraging intermediate-layer representations from dedicated learnable tokens, but also supports omni-interactive querying, enabling users to provide inputs in the form of text, visual regions of interest, and audio spans. Within OmniUE, visual and audio segmenters process diverse user interactions and integrate them with an omni-LLM to produce user-conditioned any-to-any embeddings via context aggregation. To evaluate OmniUE's omni-interactive capabilities, we introduce OmniCHOIR, benchmarking models for omni-interactive compositional audio retrieval based on the given text, video, and audio as well as unimodal or multimodal interaction prompts. OmniUE consistently surpasses state-of-the-art baselines across diverse modalities, with average improvements of 10.4% on textual-interactive video benchmarks (MMEB-v2-video), 1.0% on audio tasks (MAEB), 83.1% on visual-interactive benchmarks (SCaR), and 21.9% on our omni-interactive OmniCHOIR benchmark. We believe that jointly advancing omni-modal representation learning and omni-interactive querying paves the way toward universal embedders. Project page: https://wywywang.github.io/omniue-demo/.

Figures & tables

Figure 1: Applicability of OmniUE, which takes omni-interactive prompts ( $p$ ) and queries ( $q$ ) to retrieve any-modality targets ( $c$ ), including text, image, video, and audio.

Figure 2: OmniUE overview. Compared to existing omni-LLM-based embedders (shown in gray; e.g., [ 58 ] ), OmniUE introduces three key components: 1) audio and visual segmenters that enable emerging interaction capabilities; 2) additional learnable tokens for modeling complex and compositional multimodal inputs; and 3) a context aggregation module that selectively weighs the token-layer importance of contextual tokens to produce the final embedding for contrastive learning.

Figure 3: The data collection pipeline for OmniCHOIR. We use SAM-Audio-Large for sound separation conditioned on text prompts, and Qwen3-Omni-30B as a captioner to synthesize ground-truth and negative background sound categories based on the visual information from the silent video. A dynamic sound mixer then injects background sounds into the target audio track with randomized temporal intervals and volume levels (-15 dB to -5 dB).

	CLS	QA	RET	MRET	Overall
Model $\downarrow$ $\mid$ #Datasets $\rightarrow$	5	5	5	3	18
Text + Image + Video
VLM2Vec-v2 (Qwen2-VL-2B) [ 38 ]	39.3	34.3	28.8	38.5	34.9
UME-R1 (Qwen2-VL-2B) [ 29 ]	44.3	51.2	32.9	39.7	42.2
UniME-V2 (LLaVA-OneVision-7B) [ 22 ]	37.2	50.6	28.9	39.6	39.0
UME-R1 (Qwen2-VL-7B) [ 29 ]	48.6	60.7	38.2	39.3	47.5

Table 1: Results on 18 MMEB tasks, with scores averaged per meta-task. Improvements are computed against the best-performing 2B/3B and 7B baselines. We highlight the best and second-best in each column. All results are from the leaderboard and [ 7 ] , except for LCO-Emb, which we reproduced.

	Clf	M.Clf	PC	Rrnk	Clust	A. Rtrvl	X. Rtrvl	Zero Clf.	Overall
Model $\downarrow$ $\mid$ #Datasets $\rightarrow$	10	2	3	1	3	1	8	2	30
Two-Tower-Based Embedders
microsoft/msclap-2023 [ 16 ]	45.0	5.8	53.6	75.4	15.2	87.3	9.4	12.6	31.1
laion/larger_clap_general [ 57 ]	51.7	2.3	51.9	66.8	6.6	93.2	9.8	14.9	32.2
LLM-Based Embedders
LCO-Emb (Qwen2.5-Omni-3B) [ 58 ]	56.4	41.6	66.7	75.4	1.3	67.7	50.3	62.2	50.7

Table 2: Results on 30 MAEB tasks. Scores are averaged per meta-task. Improvements are computed relative to the best-performing 3B and 7B baselines. We highlight the best and second-best results in each column. Baseline results are from the MAEB leaderboard.

Model $\downarrow$ $\mid$ Datasets $\rightarrow$	RefCOCO+	RefCOCOg	VisualGenome	COCO-Stuff	ADE20K	Overall
Textual-Interactive-Only Models
VLM2Vec-2B [ 27 ]	24.5	29.5	22.3	19.4	24.6	24.1
VLM2Vec-7B [ 27 ]	23.2	29.1	14.7	25.0	22.4	22.9
MMRet-7B [ 72 ]	27.1	21.8	15.2	26.0	22.6	22.5
UniME-7B [ 22 ]	31.4	32.8	19.0	25.3	23.0	26.3
Textual + Visual-Interactive Models

Table 3: Results on 5 SCaR tasks. Improvements are computed relative to the best-performing 2B and 7B baselines. We highlight the best and second-best results in each column. Baseline results are from the original SCaR paper.

Model $\downarrow$ $\mid$ Conditions $\rightarrow$	Text	Span	Mask	Text+Span	Text+Mask	Span+Mask	Text+Span+Mask
Two-Tower-Based Models
laion/larger_clap_general [ 57 ]	16.1	1.7/12.7	-	12.7	-	-	-
Textual-Interactive Omni-Modal Models
ImageBind [ 20 ]	16.8	3.3/13.5	-	18.1	-	-	-
LCO-Emb-3B [ 58 ]	19.6	18.7	-	20.5	-	-	-
LCO-Emb-7B [ 58 ]	22.3	22.4	-	23.8/23.8	-	-	-

Table 4: Results on OmniCHOIR. We highlight the best results in each column. Text refers to the interaction text prompt. For span scenarios, we 1) convert the selected span into text using the same instructions as in OmniUE and 2) directly crop the corresponding audio segment, reported as text/crop.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Configurations	OmniUE-3B	OmniUE-7B
Omni-LLM	Qwen2.5-Omni-3B	Qwen2.5-Omni-7B
Vision Segmenter	facebook/sam3
Audio Segmenter	facebook/sam-audio-base
# of heads in SAC ( $h$ )	2
$\|F^{H}_{vs}\|\times\|F^{W}_{vs}\|$	288 $\times$ 288
$\|F_{as}\|$	maximum 250

Appendix

Table 5: Detailed configurations for OmniUE 3B and 7B models.

Data Source	Modalities	# of Samples
AudioSet [ 19 ]	Video, Audio, Text*	86k
WavCaps [ 37 ]	Audio, Text	404k
MSR-VTT [ 63 ]	Video, Text	180k
PE-Videos [ 5 ]	Video, Text	983k
VALOR [ 33 ]	Video, Audio, Text	2k
VATEX [ 54 ]	Video, Text	5k

Appendix

Table 6: Overview of the training data used to train OmniUE. * denotes synthesized text captions.

	MMEB-v2-video	MAEB	SCaR	OmniCHOIR
OmniUE-3B	44.7	47.8	50.5	22.2
w/o visual stream	42.7	47.6	32.0	20.5
w/o audio stream	44.1	47.4	49.8	18.7
w/o visual and audio streams	42.5	47.5	30.9	16.3
w/o multi-token inputs	43.8	46.6	47.2	20.1
w/o layer aggregation	43.2	45.0	49.8	18.5

Appendix

Table 7: Ablation study on the impact of each proposed component. "w/o visual and audio streams" denotes the removal of visual and audio interactions during both training and testing stages. "w/o visual stream" and "w/o audio stream" denote the removal of visual-only and audio-only interactions during both stages. "w/o multi-token inputs" replaces the

K

learnable tokens with the EOS hidden state, reducing

W

\mathbb{R}^{M}

. "w/o layer aggregation" removes the context aggregation module and applies mean pooling over the

K

tokens from the last layer.

Figure 4: Parameter study on 1) Aggregation from each $l$ layers; 2) # of learnable tokens ( $k$ ); 3) # of attention heads in SA ( $h$ ); 4) LoRA ranks; 5) number of visual-interactive tokens ( $|F^{\prime}_{vs}|$ ); and 6) number of audio-interactive tokens ( $|F^{\prime}_{as}|$ ).

	MMEB-v2-video	MAEB	SCaR	OmniCHOIR
OmniUE-3B	48.4	50.5	51.3	26.4
w/o learnable tokens & context aggregation	43.4	44.7	46.3	23.5
w/o visual & audio streams	46.5	48.2	32.9	19.3

Appendix

Table 8: Ablation study on the omni-interactions and architectural changes.

Choice	MMEB-v2-video	MAEB	SCaR	OmniCHOIR
small	43.4	46.7	49.3	21.5
base	44.7	47.8	50.5	22.2
large	44.2	48.1	50.2	20.8

Appendix

Table 9: Ablation study with OmniUE-3B on 1) audio segmenter size, 2) visual segmenter version, 3) segmenter connector design, and 4) layer aggregation variants. The highlighted row indicates the configurations used in OmniUE-3B and OmniUE-7B.

	Query Mode	Time (per sample)	Memory	Performance
OmniCHOIR
LCO-Emb-7B	Text	1.0s	17.7GB	22.3
WAVE-7B	Text	1.3s	18.2GB	20.0
OmniUE-7B (Ours)	Text	1.3s	20.8GB	25.5
	Span	1.3s	20.8GB	26.3
	Mask	9.4s	20.8GB	26.7

Appendix

Table 10: Analysis on inference time and memory for unimodal and multimodal query modes of OmniUE-7B on OmniCHOIR.

Model $\downarrow$ $\mid$ Conditions $\rightarrow$	Text	Span	Mask	Text+Span	Text+Mask	Span+Mask	Text+Span+Mask
OmniUE-7B	25.5	26.3	26.7	27.4	26.7	27.6	28.4
1) Random masks	-	-	16.8	-	21.3	22.5	22.9
2) Random spans	-	19.6	-	19.8	-	20.5	23.7
3) w/o text prompts	10.7	-	-	16.9	21.0	-	22.3

Appendix

Table 11: Analysis on noisy masks, spans, and text interaction prompts.

	V2A	A2V
WAVE-7B	25.0	25.8
OmniUE-7B	27.2	26.1

Appendix

Table 12: Performance on V2A/A2V on VGGSound and VA2T/T2VA on VALOR-32K.

Multimodal Embedding

Multimodal embedding maps text, images, audio, video, and other data into representations that support comparison and retrieval across modalities.

See all 160 papers

Ovis-Embedding: Pushing the Frontiers of Universal Omni-Modal Embeddings
Sep 21, 2026Ovis-Embedding TeamUnified Multimodal Models Multimodal Embedding
Alibaba Token Hub, Alibaba Group
Efficient and High-Fidelity Omni Modality Retrieval
Mar 2, 2026Chuong Huynh, Manh Luong, Abhinav ShrivastavaMultimodal IR Cross-Modal Retrieval
University of Maryland, College Park, USA · Monash University, Australia
OmniRetriever: Any-to-Any Audio-Video-Text Retrieval via Fusion-as-Teacher Distillation
May 26, 2026Yunze Liu, Chi-Hao Wu, Enmin Zhou +1Cross-Modal Learning Cross-Modal Knowledge Distillation
Memories.ai Research

Sep 21, 2026J/K move · Enter open · S save

Ovis-Embedding: Pushing the Frontiers of Universal Omni-Modal Embeddings

Ovis-Embedding Team

Alibaba Token Hub, Alibaba Group

Unified Multimodal Models Multimodal Embedding Cross-Modal Retrieval Multimodal Contrastive Learning

Open paper AskPDF arXiv

In this report, we introduce Ovis-Embedding, a state-of-the-art omni-modal embedding family built on native integration of text, image, video, and audio. Instead of assembling separate modality towers, Ovis-Embedding uses a shared multimodal backbone to encode different modalities in a common representation space. Specifically, we make three key advances: (1) native omni-modal initialization: we adopt a pretrained Qwen-omni model as the embedding backbone and adapt it through contrastive training with low-rank initialization; (2) data-centric omni-modal training: we construct a broad, high-quality corpus spanning text, images, video, audio, and interleaved multimodal data. To improve data efficiency, we introduce homogeneous-source sampling to form task-consistent batches with informative in-batch negatives; and (3) embedding-specific training and inference optimization: we use focal loss to emphasize hard examples and similarity-based Embedding Distillation to transfer fine-grained similarity structure from complementary experts. At inference time, low-rank feature decomposition enables compact embeddings with flexible dimensionality and minimal performance loss. Empirical evaluations show that the Ovis-Embedding family achieves state-of-the-art performance on MMEB-v3, MMEB-v2, MVEB, MAEB, and RTEB, demonstrating its effectiveness across text, image, video, and audio modalities. These results highlight the potential of unified omni-modal training to overcome modality fragmentation and advance universal embedding models for any-to-any retrieval.

Abstract

Multimodal Embedding

Explore similar work

Ovis-Embedding: Pushing the Frontiers of Universal Omni-Modal Embeddings

Efficient and High-Fidelity Omni Modality Retrieval

OmniRetriever: Any-to-Any Audio-Video-Text Retrieval via Fusion-as-Teacher Distillation