SAVE: Speech-Aware Video Representation Learning for Video-Text Retrieval

Ruixiang Zhao; Zhihao Xu; Bangxiang Lan; Zijie Xin; Jingyu Liu; Xirong Li

cs.CVMar 9, 2026

SAVE: Speech-Aware Video Representation Learning for Video-Text Retrieval

Authors: Ruixiang Zhao, Zhihao Xu, Bangxiang Lan, Zijie Xin, Jingyu Liu, Xirong Li

Organizations: Renmin University of China

Abstract

For video-text retrieval, the use of CLIP has been a de facto choice. Since CLIP provides only image and text encoders, this consensus has led to a biased paradigm that entirely ignores the sound track of videos. While several attempts have been made to reintroduce audio -- typically by incorporating an audio encoder and fusing its output with visual features -- these methods face two challenges: ineffective representation of speech content and suboptimal vision-audio fusion. To address these issues jointly, we propose SAVE, a Speech Aware Video rEpresentation learning method. SAVE improves upon AVIGATE, a SOTA audiovisual method, with a dedicated speech branch for more effective speech embedding. Furthermore, we introduce soft-ALBEF for early vision-audio alignment that facilitates fusion. Extensive experiments on five benchmarks show that SAVE compares favorably against the SOTA, outperforming AVIGATE by +4.1% on MSRVTT-9k, +1.9% on MSRVTT-7k, +2.5% on VATEX, +9.8% on Charades, and +2.1% on LSMDC, in light of the SumR metric.

Figures & tables

Panel of Figure 1 — Figure 1: An overview of this paper . (a) Problem : Current audio encoders ( ResNet-18 [ 13 ] and AST [ 9 ] ), trained on datasets of environmental sounds, are not well suited for speech embedding. (b) Solution : We improve the state-of-the-art in audio-enhanced video-text retrieval by introducing a dedicated speech branch for speech-aware video embedding. (c) Results : Our method consistently outperforms SOTA audio-enhanced models (TEFAL [ 17 ] and AVIGATE [ 18 ] ) across five benchmarks.

Method	Audio	Speech	Video embedding network	Early vision-audio alignment	MV-9k SumR
EclipSE [ 27 ]	ResNet-18	✗	Bi-branch	✗	197.8
TEFAL [ 17 ]	AST	✗	Bi-branch	✗	209.2
AVIGATE [ 18 ]	AST	✗	Bi-branch	✗	207.7
SAVE (ours)	AST	ASR + CLIP	Tri-branch	✓	216.2

Table 1: SOTA audiovisual methods for video-text retrieval .

Figure 2: Hard vs . soft labels for early vision-audio alignment . For the videos in the first two rows, their associated sound tracks are not semantically relevant w.r.t. the video content. Enforcing vision-audio alignment for these videos is adverse. By contrast, the soft labels, estimated by ImageBind [ 8 ] , provide finer supervision for better vision-audio alignment.

Figure 3: Proposed s peech- a ware v ideo r e presentation learning (SAVE) method for video-text retrieval . Given a short video $\mathcal{V}$ associated with a sound track $\mathcal{A}$ , SAVE uses a tri-branch network to embed the video frames to a set of visual tokens $\{v_{i}\}$ , the sound to a set of audio tokens $\{a_{i}\}$ , and the speech to a set of textual tokens $\{s_{i}\}$ . Gated fusion conditioned on the visual tokens is performed on the audio and textual tokens, yielding fused tokens $\{\hat{a}_{i}\}$ and $\{\hat{s}_{i}\}$ , respectively. By aggregating $\{v_{i}\}$ , $\{\hat{a}_{i}\}$ and $\{\hat{s}_{i}\}$ , we obtain $\{\hat{v}_{i}\}$ as a speech-aware video representation. The video’s relevance w.r.t. a specific query $\mathcal{T}$ is computed as a multi-grained similarity between $\{\hat{v}_{i}\}$ and the query embedding $t$ . Soft-ALBEF is used only during training for better alignment between the visual and audio tokens.

Figure 4: Proportion of videos with audio / ASR available .

Benchmark	Train	Val.	Test
MSRVTT-9k [ 53 ]	9,000	–	1,000
MSRVTT-7k [ 49 ]	6,513	497	2,990
VATEX [ 47 ]	25,991	1,500	1,500
Charades [ 43 ]	7,985	–	1,863
LSMDC [ 39 ]	101,079	7,408	1,000

Table 2: Number of videos per benchmark .

Model	MSRVTT-9k				MSRVTT-7k				VATEX				Charades				LSMDC				mR1
Model	R1	R5	R10	SumR	R1	R5	R10	SumR	R1	R5	R10	SumR	R1	R5	R10	SumR	R1	R5	R10	SumR	mR1
Vision-only:
CLIP4Clip, Neucom22 [ 32 ]	44.5	71.4	81.6	197.5	29.4	54.9	65.8	150.1	61.6	91.1	95.8	248.5	17.7	39.5	50.4	107.6	22.6	41.0	49.1	112.7	35.1
X-Pool, CVPR22 [ 10 ]	46.9	72.8	82.2	201.9	-	-	-	-	-	-	-	-	-	-	-	-	25.2	43.7	53.5	122.4	-
TS2-Net, ECCV22 [ 30 ]	47.0	74.5	83.8	205.3	29.9	56.4	67.3	153.6	59.1	90.0	95.2	244.3	16.9	38.5	49.5	104.9	23.4	42.3	50.9	116.6	35.3
X-CLIP, MM22 [ 33 ]	46.1	73.0	83.1	202.2	31.2	57.4	68.1	156.7	62.2	90.9	95.4	248.5	18.9	41.3	53.0	113.2	23.3	43.0	-	-	36.3

Table 3: Comparing with visual and audiovisual models for text-to-video retrieval . Per baseline, we cite numbers from its original paper where applicable. The proposed SAVE model outperforms visual and audiovisual baselines on all datasets.

Visual:
Methods	Computational complexity	Inference time (ms) $\downarrow$	SumR $\uparrow$
CLIP4Clip [ 32 ]	$\mathcal{O}(n_{\mathcal{V}}+n_{\mathcal{T}})$	9.76	197.5
X-Pool [ 10 ]	$\mathcal{O}(n_{\mathcal{V}}n_{\mathcal{T}})$	66.31	201.9
PIG [ 23 ]	$\mathcal{O}(n_{\mathcal{V}}+n_{\mathcal{T}})$	9.76	203.0
Audiovisual:
TEFAL [ 17 ]	$\mathcal{O}(n_{\mathcal{A}}n_{\mathcal{T}}+n_{\mathcal{V}}n_{\mathcal{T}})$	140.57	209.2

Table 4: Efficiency analysis . Inference time includes both query feature extraction and video-text similarity computation. Test set: MSRVTT-9k . GPU: NVIDIA RTX 3090.

Panel of Figure 5 — Figure 5: Comparison per group . Our SAVE consistently outperforms PIG (best visual model) and AVIGATE (best audiovisual model) across all groups, with the largest gain obtained in the Sound-Speech-related group.

#	Setup	MSRVTT-9k		MSRVTT-7k
#	Setup	R1	SumR	R1	SumR
0	Full	51.3	216.2	33.5	165.8
Speech branch:
1	w/o Speech Branch	50.9	211.9( 4.3 $\downarrow$ )	33.0	164.1( 1.7 $\downarrow$ )
2	ASR: Whisper $\rightarrow$ SenseVoice	51.2	215.4( 0.8 $\downarrow$ )	33.5	165.4( 0.4 $\downarrow$ )
3	$\mbox{CLIP}_{txt}$ : w/o parameter sharing	51.0	214.8( 1.4 $\downarrow$ )	33.4	165.0( 0.8 $\downarrow$ )

Table 5: Ablation studies of SAVE .

Figure 6: Soft-ALBEF vs ALBEF with low-score data filtered out.

Zero-shot:
Methods	R1	R5	R10	SumR
ImageBind-v	39.1	62.8	73.1	175.0
ImageBind-va	39.4	63.2	73.7	176.3
ImageBind-vas	40.6	62.6	74.1	177.3
Finetuned:
ImageBind-vas	47.2	72.9	82.0	202.1

Table 6: Comparison with ImageBind . Dataset: MSRVTT-9k .

Panel of Figure 13 — Figure S1 : t-SNE visualizations of feature spaces across three audio encoders . All three audio encoders struggle to capture speech semantics.

Setup	Audio	Speech	Soft-ALBEF	MSRVTT-9k		MSRVTT-7k
Setup	Audio	Speech	Soft-ALBEF	R1	SumR	R1	SumR
SAVE	AST	Whisper	✓	51.3	216.2	33.5	165.8
SAVE-2	Whisper	Whisper	✓	50.9	215.8	33.8	166.8
SAVE-3	Whisper	✗	✓	50.2	213.0	32.9	164.0

Table S1 : Ablation study of using Whisper as the audio encoder . Backbone: CLIP (ViT-B/32).

Vision-only:
Model	R1	R5	R10	SumR
CLIP4Clip, Neucom22 [ 32 ]	43.1	70.5	81.2	194.8
X-Pool, CVPR22 [ 10 ]	44.4	73.3	84.0	201.7
TS2-Net, ECCV22 [ 30 ]	45.3	74.1	83.7	203.1
X-CLIP, MM22 [ 33 ]	46.8	73.3	84.0	204.1
DiCoSA, IJCAI23 [ 20 ]	46.7	75.2	84.3	206.2

Table S2 : The video-to-text retrieval performance of different methods . Dataset: MSRVTT-9k. Backbone: ViT-B/32.

Vision-only:
Model	Text-to-Video Retrieval				Video-to-Text Retrieval
Model	R1	R5	R10	SumR	R1	R5	R10	SumR
CLIP4Clip, Neucom22 [ 32 ]	46.4	72.1	82.0	200.5	45.4	73.4	82.4	201.2
X-Pool, CVPR22 [ 10 ]	48.2	73.7	82.6	204.5	46.4	73.9	84.1	204.4
TS2-Net, ECCV22 [ 30 ]	49.4	75.6	85.3	210.3	46.6	75.9	84.9	207.4
X-CLIP, MM22 [ 33 ]	49.3	75.8	84.8	209.9	48.9	76.3	85.4	210.6

Table S3 : The retrieval performance of different methods on CLIP ViT-B/16 backbone . Dataset: MSRVTT-9k.

Figure S2 : Qualitative comparison of the Top-1 retrieved videos from PIG , AVIGATE , and SAVE , with the same color-coded blocks indicating how each model ranks these candidate videos. Benefiting from the speech branch, SAVE consistently assigns the ground-truth video (with green box) the best rank, showing clear advantages on speech-related queries. Best viewed on screen.

Audio-Visual Representation Learning

Audio-visual representation learning develops models that connect sound and video for recognition, retrieval, localization, and robust multimodal understanding.

See all 31 papers

OmniRetriever: Any-to-Any Audio-Video-Text Retrieval via Fusion-as-Teacher Distillation
May 26, 2026Yunze Liu, Chi-Hao Wu, Enmin Zhou +1Cross-Modal Learning Cross-Modal Knowledge Distillation
Memories.ai Research
VEDJE: Video-Efficient Discriminative Joint Encoder for Scalable Video-Text Retrieval
Oct 8, 2026Shahaf Wagner, Gabriele Serussi, Dan Ben Ami +2Video Retrieval Vision-Language Retrieval
INSIGHT Lab, Ben-Gurion University of the Negev, Israel · Decart AI · Texas A&M University, College Station
Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedy
Sep 22, 2025Geewook Kim, Minjoon SeoVLM Evaluation Audio-Visual Understanding
NAVER Cloud AI, South Korea · KAIST AI, South Korea

May 26, 2026J/K move · Enter open · S save

OmniRetriever: Any-to-Any Audio-Video-Text Retrieval via Fusion-as-Teacher Distillation

Yunze Liu, Chi-Hao Wu, Enmin Zhou, Junxiao Shen

Memories.ai Research

Cross-Modal Learning Cross-Modal Knowledge Distillation Multimodal Embedding Cross-Modal Retrieval Audio-Text Retrieval Multimodal Foundation Models+3

Open paper AskPDF arXiv

Unified multimodal embedding spaces have become the standard interface for cross-modal retrieval and multimodal RAG, and recent audio-video-text (AVT) encoders extend this setting to three modalities. Such encoders can produce a joint (T,V,A) embedding whenever all three modalities are available, but standard pairwise InfoNCE objectives leave this signal unused during training. We close this gap with fusion-as-teacher distillation, which treats a stop-gradient copy of the fused embedding as a teacher signal for the single-modal embeddings, paired with a Tuple-InfoNCE term that supervises the fused embedding directly. We instantiate this objective as OmniRetriever-7B. Across six zero-shot retrieval benchmarks, OmniRetriever-7B surpasses the closed-source Gemini Embedding 2 by 13.3-18.0 R@1 on Clotho and SoundDescs, and reaches the contemporary zero-shot specialist band of open video-text encoders on MSR-VTT and MSVD. To stress-test joint representations, we further release OmniRetriever-Bench, a 12-direction AVT retrieval benchmark totaling 3782 triples; on it OmniRetriever-7B attains AVG-all 34.84, improving over Gemini Embedding 2 by 1.72 and over the best prior open-source AVT method by 8.03.

Abstract

Audio-Visual Representation Learning

Explore similar work

OmniRetriever: Any-to-Any Audio-Video-Text Retrieval via Fusion-as-Teacher Distillation

VEDJE: Video-Efficient Discriminative Joint Encoder for Scalable Video-Text Retrieval

Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedy