cs.CVApr 13, 2026

Multimodal Large Language Model-Enabled Video Translation: A Role-Oriented Survey

Authors: Bingzheng Qu, Kehai Chen, Xuefeng Bai, Min Zhang

Organizations: School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen 518055, Guangdong, China.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Recent progress in multimodal large language models (MLLMs) is reshaping video translation from a cascaded pipeline of automatic speech recognition, machine translation, text-to-speech, and lip synchronization into a unified multimodal reasoning and generation problem. High-quality video translation requires not only semantic fidelity, but also temporal alignment, speaker consistency, and emotional expressiveness across visual, acoustic, and linguistic streams. This survey provides a focused review of MLLM-enabled video translation through a role-oriented taxonomy. We organize MLLM-enabled and MLLM-relevant studies into three functional roles: Semantic Reasoner, which grounds translation in video understanding, temporal reasoning, and multimodal fusion; Expressive Performer, which supports controllable and context-aware speech generation; and Visual Synthesizer, which enables lip synchronization and visually coherent speaker rendering. We further summarize representative datasets, benchmarks, and metrics for each role, and discuss how current evaluation protocols fall short of end-to-end video translation requirements. Finally, we identify open challenges in long-form video understanding, temporal modeling, multimodal alignment, multilingual robustness, and responsible deployment, outlining future directions for natural and trustworthy cross-lingual video communication.

Figures & tables

SurveyYearMain ScopeSREPVSMissing Link
Shen et al. [ 19 ]2024Multimodal MT△\triangle––Multimodal translation without speech or visual synthesis
Das et al. [ 20 ]2025Video-guided MT✓\checkmark––Video-guided text translation only
Yin et al. [ 21 ]2024General MLLMs✓\checkmark△\triangle△\triangleBroad MLLM scope; not video-translation-specific
Tang et al. [ 22 ]2025Video Understanding✓\checkmark––Video comprehension without cross-lingual generation
Wu et al. [ 23 ]2025Temporal Grounding✓\checkmark––Temporal localization without translation/generation
Gupta et al. [ 24 ]2024Speech-to-speech Translation△\triangle✓\checkmark–Speech translation with limited visual synchronization
TABLE I: Comparison with existing surveys related to MLLM-enabled video translation.
Fig. 2: Typical architecture of an MLLM-enabled video understanding model. Text, audio, and video inputs are encoded by trainable or frozen modality-specific encoders. The resulting embeddings are projected, temporally and spatially pooled, and concatenated into multimodal tokens.
Fig. 3: A typical architecture of MLLM-enabled text-to-speech synthesis.
Fig. 4: Comparison between UNet-based and DiT-based enabling visual backbones. UNet-based models commonly employ convolutional encoder–decoder denoisers with temporal modules, whereas DiT-based models represent videos as spatio-temporal latent tokens and perform denoising with Transformer blocks. In video translation, these backbones are mainly relevant when adapted for audio-driven lip motion, identity-preserving talking-head animation, and temporally coherent speaker rendering.
ModelMSVD-QA [ 147 ]MSRVTT-QA [ 147 ]ActivityNet-QA [ 148 ]Avg.
Audio-visual and omni-modal MLLMs
FAVOR [ 149 ]67.859.3–63.5
Dolphin [ 150 ]72.762.649.161.5
Video-SALMONN [ 151 ]67.959.5–63.7
AV-LLM [ 152 ]67.353.747.256.1
OneLLM [ 40 ]56.553.8–55.1
TABLE II: Zero-shot open-ended VideoQA accuracy of representative MLLMs. Higher scores are better.
MethodCategoryZero-shotMain ControlKey Architecture
Speech-token Language Models
CosyVoice [ 89 ]Speech-token LM✓\checkmarkSpeaker, multilingual styleSemantic tokenizer + text-to-token LLM + flow matching decoder
CosyVoice 2 [ 15 ]Speech-token LM✓\checkmarkStreaming, speaker styleFSQ tokenizer + streamlined LLM + chunk-aware flow matching
Spark-TTS [ 90 ]Speech-token LM✓\checkmarkSpeaker, acoustic attributesQwen-based LLM + BiCodec tokenizers
VALL-E R [ 88 ]Speech-token LM✓\checkmarkSpeaker promptNeural codec LM + monotonic alignment
HALL-E [ 92 ]Speech-token LM✓\checkmarkLong-form speechHierarchical neural codec LM
TABLE III: Representative Expressive Performer models for speech synthesis and controllable speech generation.
DatasetLanguageScalePrimary Use
LibriLight [ 161 ]English60k hSpeech Pretraining
LibriTTS [ 162 ]English585 hSpeaker Synthesis
LibriSpeech [ 163 ]English1k hASR and Intelligibility
MLS [ 164 ]MultilingualLarge-scaleMultilingual Speech
Emilia [ 165 ]MultilingualLarge-scaleZero-shot TTS
TABLE IV: Representative speech datasets for Expressive Performer evaluation.
More in this topic

Lip Synchronization

See all 19 papers

Explore similar work

CardsList

  1. Video-guided Machine Translation with Global Video Context

    Apr 8, 2026Jian Chen, JinZe Lv, Zi Long +1Machine TranslationCross-Modal Attention

    Shenzhen University · Shenzhen Technology University

  2. Watch, Remember, Reason: Human-View Video Understanding with MLLMs

    Jun 5, 2026Jiahao Meng, Yue Tan, Qi Xu +12Streaming Video UnderstandingVideo-Language Models

    School of Intelligence Science and Technology, Peking University · Wuhan University · Shanghai Jiao Tong University +7

  3. LVSum: A Benchmark for Timestamp-Aware Long Video Summarization

    Apr 11, 2026Alkesh Patel, Melis Ozyildirim, Ying-Chang Cheng +1Video SummarizationVideo-Language Model Evaluation

    Apple