Efficient and High-Fidelity Omni Modality Retrieval

Chuong Huynh; Manh Luong; Abhinav Shrivastava

cs.IRMar 2, 2026

Efficient and High-Fidelity Omni Modality Retrieval

Authors: Chuong Huynh, Manh Luong, Abhinav Shrivastava

Organizations: University of Maryland, College Park, USA · Monash University, Australia

Abstract

Multimodal retrieval is the task of aggregating information from queries across heterogeneous modalities to retrieve desired targets. State-of-the-art multimodal retrieval models can understand complex queries, yet they are typically limited to two modalities: text and vision. This limitation impedes the development of universal retrieval systems capable of comprehending queries that combine more than two modalities. To advance toward this goal, we present OmniRet, the first retrieval model capable of handling complex, composed queries spanning three key modalities: text, vision, and audio. Our OmniRet model addresses two critical challenges for universal retrieval: computational efficiency and representation fidelity. First, feeding massive token sequences from modality-specific encoders to Large Language Models (LLMs) is computationally inefficient. We therefore introduce an attention-based resampling mechanism to generate compact, fixed-size representations from these sequences. Second, compressing rich omni-modal data into a single embedding vector inevitably causes information loss and discards fine-grained details. We propose Attention Sliced Wasserstein Pooling to preserve these fine-grained details, leading to improved omni-modal representations. OmniRet is trained on an aggregation of approximately 6 million query-target pairs spanning 30 datasets. We benchmark our model on 13 retrieval tasks and a MMEBv2 subset. Our model demonstrates significant improvements on composed query, audio and video retrieval tasks, while achieving on-par performance with state-of-the-art models on others. Furthermore, we curate a new Audio-Centric Multimodal Benchmark (ACM). This new benchmark introduces two critical, previously missing tasks-composed audio retrieval and audio-visual retrieval to more comprehensively evaluate a model's omni-modal embedding capacity.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Comparison of retrieval systems. OmniRet is the first to handle composed queries from text, vision, and audio. It introduces a Shared Media Resampler for efficiency and Attention Sliced Wasserstein Pooling to preserve high-fidelity, fine-grained details. (b) Our Audio-Centric MultiModal evaluation suite. We introduce two novel, audio-centric tasks: Visual-Audio Retrieval and Composed Audio-Text Retrieval. These additions fill critical gaps in existing benchmarks, enabling a more comprehensive evaluation of omni-modal retrieval models.

Figure 2: Overall OmniRet architecture. Our universal retrieval model integrates specialized visual and audio encoders with a Large Language Model acting as a cross-modal composer. Media inputs are processed through a Shared Media Resampler, and the final embedding is derived via our Attention Sliced Wasserstein Pooling. The model is optimized using Contrastive, Triplet and Diversity losses.

Figure 3: Left: Our Shared Media Resampler condenses the output of media encoders into a compact set of latent vectors before they are fed to the LLM. Right: Our Attention Sliced Wasserstein Pooling (ASWP) aggregates the final LLM output hidden states into a single, high-fidelity embedding vector. Attention Resampler applies to both output of media encoders and LLM universal encoder.

Figure 4: A composed audio retrieval example from our ACM benchmark where query composes both audio and text modalities.

Evaluation Method	Accuracy
Human	87%
Gemini-2.5 w/ text only	96%

Table 1: Subjective evaluation of our ACM benchmark. For the “Gemini-2.5 w/ text only” method, the audio inputs were replaced with their corresponding generated captions.

Model	LLM Size	Uni-Modality		Textual Binding						Composed Visual Text
		I $\rightarrow$ I	T $\rightarrow$ T	I $\rightarrow$ T	T $\rightarrow$ I	V $\rightarrow$ T	T $\rightarrow$ V	A $\rightarrow$ T	T $\rightarrow$ A	T $\rightarrow$ I,T	I,T $\rightarrow$ T	I,T $\rightarrow$ I	I,T $\rightarrow$ I,T	V,T $\rightarrow$ V
Text-Binding Pretrained Models
CLIP [ 57 ]	-	25.9	40.5	43.1	37.5	14.2	15.9	-	-	-	-	-	-	-
SigLIP [ 78 ]	-	25.9	34.0	56.5	52.1	19.7	22.5	-	-	-	-	-	-	-
PE-Core [ 3 ]	-	32.0	56.8	58.0	53.4	32.3	29.5	-	-	-	-	-	-	-
MMT [ 54 ]	-	-	-	-	-	-	-	49.8	46.8	-	-	-	-	-

Table 2: Performance comparison of OmniRet with state-of-the-art baselines across 13 retrieval tasks on the extended version of M-BEIR benchmark. Our model demonstrates strong, balanced performance on established visual-text tasks while uniquely extending capabilities to the audio modality. Modalities: I (Image), V (Video), A (Audio), T (Text). Bold and green indicate the best performance in each group and overall, respectively. MMEmbed [ 40 ] is included for reference only, as it uses a larger LLM and is not directly comparable.

Model	LLM Size	Image		Video
		CLS	RET	CLS	RET	MRET
Image-Text Embedding Models
UniME [ 17 ]	1.5B	59.0	64.9	-	-	-
LLaVE [ 32 ]	1.5B	62.1	65.2	-	-	-
B3++ [ 65 ]	1.5B	67.0	70.9	-	-	-
MetaEmbed [ 75 ]	2B	68.1	71.9	-	-	-

Table 3: Generalization performance of OmniRet against leading models (

<

7B parameters) on a subset of MMEBv2. Our model achieves outstanding performance on video tasks. Notably, it maintains competitive performance on image retrieval tasks without being fully fine-tuned on their training sets. Bold highlights the best score in each group.

Model	A,T $\rightarrow$ A	A $\rightarrow$ V	V $\rightarrow$ A	A $\rightarrow$ I	I $\rightarrow$ A
QwenOmni [ 76 ] + Gemma [ 68 ]	44.6	3.3	6.3	4.4	5.4
CLAP [ 74 ]	16.1	-	-	-	-
ImageBind [ 16 ]	7.32	35.5	36.3	30.1	29.7
ViT-Lens [ 34 ]	10.1	-	-	24.7	25.9
OmniBind [ 69 ]	15.6	-	-	30.5	29.1
OmniRet	23.0	35.5	34.4	24.5	26.0

Table 4: Performance comparison on our novel ACM benchmark. OmniRet illustrats promising results over baselines on both the composed audio and audio-visual retrieval tasks. Modalities: I (Image), V (Video), A (Audio), T (Text). Bold and underline indicate the best and second-best performance, respectively.

	Avg. Recall	$\Delta$
Our Baseline	50.2	0.0
(a) Embedding Vector
Single $[$ EOS $]$ Vector	43.4	-6.8
Multi (16) Vectors	49.8	-0.4
(b) Number of Projections (L) and References (S)
$L=1536,S=128$	47.5	-2.7

Table 5: Ablation study of our proposed components. We report the impact on Average Recall (across 6 tasks, trained on 1M samples) when each component is removed or modified.

Task	No. Queries	No. Candidates
A,T $\rightarrow$ A	1292	4251
A $\rightarrow$ I	1292	5480
I $\rightarrow$ A	1292	5480
A $\rightarrow$ V	1292	5480
V $\rightarrow$ A	1292	5480

Table 6: The Statistics of the Audio-Centric Modality Benchmark

System	You are a language assistant that helps to generate audio captions.
Prompt	Help me generate a concise audio caption, which does not exceed 30 words. The generated audio should focus on describing acoustic events within the audio and their temporal relationships. Only generate the caption without explanation.
	[AUDIO]

Table 7: Prompt structure to generate audio captions in the Audio-Centric Multimodal Benchmark.

System	You are a language assistant that helps to generate the modification text between two audio captions.
Prompt	Generate the modification text for the following pair of audio captions:
	Caption 1: [CAPTION 1]
	Caption 2: [CAPTION 2]
	<Good example>
	…
	</Good example>

Table 8: Prompt structure to generate modification texts in the Audio-Centric Multimodal Benchmark.

Model	A,T $\rightarrow$ A	A $\rightarrow$ V	V $\rightarrow$ A	A $\rightarrow$ I	I $\rightarrow$ A
CLAP [ 12 ]	18.4	-	-	-	-
ImageBind [ 16 ]	10.5	38.7	39.4	33.9	33.5
OmniRet	28.2	34.2	36.7	26.0	24.0

Table 9: Performance on ACM Clean Subset.

Figure 5: The Qualtric survey for verifying the quality of generated texts in our ACM benchmark.

Task	Dataset	Training		Testing
		No. Queries	No. Candidates	No. Queries	No. Candidates
I $\rightarrow$ I	NIGHTS [ 15 ]	15.9K	15.9K	2.1K	40K
T $\rightarrow$ T	WebQA [ 5 ]	16K	27K	2.5K	544K
	MSMarco [ 2 ]	100K	99K	-	-
	HotpotQA [ 77 ]	84.5K	84.5K	-	-
	NaturalQuestion [ 31 ]	100K	100K	-	-

Table 10: Analytics of dataset use to train and test OmniRet.

	Stage 1	Stage 2
Batch size	2,048	3,072
No. Tasks	6	15
No. Datasets	17	28
No. Dataset/Batch	6	4
No. Intra-Dataset Samples	$\approx$ 341	768
No. GPUs	64	128

Table 11: Training Hyper-parameters in Stage 1 and Stage 2 .

Task	Dataset	VLM2VecV2 [ 50 ]	OmniRet
I-CLS	ImageNet-1K	80.8	57.0
	N24News	72.9	40.6
	HatefulMemes	56.3	49.9
	VOC2007	85.0	68.1
	SUN397	71.0	68.7
	Place365	35.9	38.3

Table 12: MMEBv2 Details with VLM2VecV2 and OmniRet . Bold indicates the best performance in each dataset while underline indices the performance from training on the datasets.

	Avg. Recall	$\Delta$
Batch size = 1024
Our Baseline	49.7	0.0
Multi (16) Vectors	49.8	+0.1
No Resampler	46.7	-3.0
Batch size = 2048
Our Baseline	50.2	0.0

Table 13: Additional Ablation Studies on Proposed Modules. We report the impact on Average Recall (across 6 tasks, trained on 1M samples) when each component is removed or modified.

Model	Speed (samples/s)	Mem (GB)
CLIP	114.35	1.62
SigLIP	53.43	3.36
PE-Core	79.05	2.54
CLIP SF	116.04	1.62
VLM2VecV2	10.21	4.39
OmniRet	35.69	6.01

Table 14: Speed and Memory Usage of Retrieval Models on MSCOCO T2I. The performance is measured on RTXA6000 with SDPA attention.

Task	Dataset	Text-Binding Pretrained Models					Multi-Task Finetuning Models
		CLIP	SigLIP	PE-Core	MMT	CLAP	CLIP SF	MMEmbed	VLM2VecV2	OmniRet	OmniRet
		[ 57 ]	[ 78 ]	[ 3 ]	[ 54 ]	[ 74 ]	[ 71 ]	[ 40 ]	[ 50 ]	(stage 1)	(stage 2)
Uni-Modality Tasks
I $\rightarrow$ I	NIGHTS	25.9	25.9	32.0	-	-	28.4	32.1	30.0	27.3	25.0
T $\rightarrow$ T	WebQA	40.5	34.0	56.8	-	-	83.7	96.7	81.1	91.7	87.8
Text-Binding Tasks

Table 15: Details of Table 2 with OmniRet’s Stage 1 included . Modalities: I (Image), V (Video), A (Audio), T (Text). Bold and green indicate the best performance in each group and overall, respectively. MMEmbed [ 40 ] is included for reference only, as it uses a larger LLM and is not directly comparable.

Multimodal IR

Multimodal information retrieval finds relevant images, videos, documents, audio, or other media using queries that may combine different modalities.

See all 128 papers

Sep 21, 2026

Ovis-Embedding: Pushing the Frontiers of Universal Omni-Modal Embeddings

In this report, we introduce Ovis-Embedding, a state-of-the-art omni-modal embedding family built on native integration of text, image, video, and audio. Instead of assembling separate modality towers, Ovis-Embedding uses a shared multimodal backbone to encode different modalities in a common representation space. Specifically, we make three key advances: (1) native omni-modal initialization: we adopt a pretrained Qwen-omni model as the embedding backbone and adapt it through contrastive training with low-rank initialization; (2) data-centric omni-modal training: we construct a broad, high-quality corpus spanning text, images, video, audio, and interleaved multimodal data. To improve data efficiency, we introduce homogeneous-source sampling to form task-consistent batches with informative in-batch negatives; and (3) embedding-specific training and inference optimization: we use focal loss to emphasize hard examples and similarity-based Embedding Distillation to transfer fine-grained similarity structure from complementary experts. At inference time, low-rank feature decomposition enables compact embeddings with flexible dimensionality and minimal performance loss. Empirical evaluations show that the Ovis-Embedding family achieves state-of-the-art performance on MMEB-v3, MMEB-v2, MVEB, MAEB, and RTEB, demonstrating its effectiveness across text, image, video, and audio modalities. These results highlight the potential of unified omni-modal training to overcome modality fragmentation and advance universal embedding models for any-to-any retrieval.

Ovis-Embedding Team

Alibaba Token Hub, Alibaba Group

Unified Multimodal Models Multimodal Embedding Cross-Modal Retrieval Multimodal Contrastive Learning

Ask PDF

Jun 8, 2026

Conan-embedding-v3: Fusing Modality-Specific Models for Omni-Modal Embedding

Omni-modal retrieval promises a single embedding space for text, image, video, document, and audio inputs, but building such a unified retriever is difficult since these modalities differ in data distribution, architecture, and optimization dynamics. In this work, we present Conan-embedding-v3, a decouple--fuse--recover framework for omni-modal retrieval. Conan-embedding-v3 first trains modality specialists independently and fuses their task vectors into a single dense backbone, a strategy we call Decoupled Specialist Fusion. We show that this fusion composes visual, video, and document retrieval capabilities, but also exposes a failure mode for projector-based modalities: when audio is attached through an external encoder and projector, fusing the backbone leaves the projector calibrated to the audio-specialist backbone, causing a large audio retrieval regression despite copying all audio-specific modules unchanged. We call this failure Projector Drift. To repair it, Conan-embedding-v3 applies Projector Recovery (i.e., full-parameter fine-tuning of the projector while keeping the backbone frozen) followed by balanced multi-modal rehearsal. The resulting model supports these retrieval pathways in one backbone, achieving 74.9 scores on MMEB while obtaining 55.61 on the 30-task MAEB audio suite.

Shiyu Li, Zhiyuan Hu, Yifan Wang +3

Tencent

Multimodal Embedding Cross-Modal Retrieval Task Vector Merging Model Merging

Ask PDF

May 26, 2026

OmniRetriever: Any-to-Any Audio-Video-Text Retrieval via Fusion-as-Teacher Distillation

Unified multimodal embedding spaces have become the standard interface for cross-modal retrieval and multimodal RAG, and recent audio-video-text (AVT) encoders extend this setting to three modalities. Such encoders can produce a joint (T,V,A) embedding whenever all three modalities are available, but standard pairwise InfoNCE objectives leave this signal unused during training. We close this gap with fusion-as-teacher distillation, which treats a stop-gradient copy of the fused embedding as a teacher signal for the single-modal embeddings, paired with a Tuple-InfoNCE term that supervises the fused embedding directly. We instantiate this objective as OmniRetriever-7B. Across six zero-shot retrieval benchmarks, OmniRetriever-7B surpasses the closed-source Gemini Embedding 2 by 13.3-18.0 R@1 on Clotho and SoundDescs, and reaches the contemporary zero-shot specialist band of open video-text encoders on MSR-VTT and MSVD. To stress-test joint representations, we further release OmniRetriever-Bench, a 12-direction AVT retrieval benchmark totaling 3782 triples; on it OmniRetriever-7B attains AVG-all 34.84, improving over Gemini Embedding 2 by 1.72 and over the best prior open-source AVT method by 8.03.

Yunze Liu, Chi-Hao Wu, Enmin Zhou +1

Memories.ai Research

Cross-Modal Learning Cross-Modal Knowledge Distillation Multimodal Embedding Cross-Modal Retrieval Audio-Text Retrieval Multimodal Foundation Models+3

Ask PDF

Abstract

Multimodal IR

Explore similar work

Ovis-Embedding: Pushing the Frontiers of Universal Omni-Modal Embeddings

Conan-embedding-v3: Fusing Modality-Specific Models for Omni-Modal Embedding

OmniRetriever: Any-to-Any Audio-Video-Text Retrieval via Fusion-as-Teacher Distillation