While state-of-the-art audio-video generation models like Veo3 and Sora2 demonstrate remarkable capabilities, their closed-source nature makes their architectures and training paradigms inaccessible. To bridge this gap in accessibility and performance, we introduce UniTalking, a unified, end-to-end diffusion framework for generating high-fidelity speech and lip-synchronized video. At its core, our framework employs Multi-Modal Transformer Blocks to explicitly model the fine-grained temporal correspondence between audio and video latent tokens via a shared self-attention mechanism. By leveraging powerful priors from a pre-trained video generation model, our framework ensures state-of-the-art visual fidelity while enabling efficient training. Furthermore, UniTalking incorporates a personalized voice cloning capability, allowing the generation of speech in a target style from a brief audio reference. Qualitative and quantitative results demonstrate that our method produces highly realistic talking portraits, achieving superior performance over existing open-source approaches in lip-sync accuracy, audio naturalness, and overall perceptual quality.
Figures & tables
Figure 1: Illustration of our unified audio-video framework for talking portrait generation. UniTalking facilitates the generation of synchronized audio and video through multiple input modalities.
Figure 2: The data processing pipeline for curating our human-centric audio-video dataset. The pipeline employs sequential filtering, beginning with single-modality (video and audio) checks, followed by a cross-modal filtering stage. The final, high-quality audio-video pairs are then annotated with multi-level, multi-modal captions.
Figure 3: The architecture of UniTalking. Our framework jointly generates synchronized audio and video for talking portraits using a dual-stream Multi-Modal DiT (MM-DiT) backbone, which is trained as a Continuous Normalizing Flow via Flow Matching. The core of the model is the Multi-Modal Transformer Block, detailed on the right. It employs Joint Attention on concatenated audio-video tokens to ensure precise temporal alignment, and Cross Attention to incorporate multi-modal conditions such as text and acoustic style. Frozen components are marked with a snowflake, while trainable parts are marked with a flame.
Figure 4: Visualization of video frames and Mel-spectrograms for generated audio-visual talking portraits and ground-truth audio-visual data. At the top is the corresponding text prompt, with certain keywords highlighted in bold and red . The generated video and audio are semantically consistent with the ground truth.
Figure 5: Visualization of attention maps between video tokens and audio tokens in Joint Attention. Subfigures (a) and (b) show two different random samples. The first row in the subgraph represents audio-to-video attention, while the second row represents video-to-audio attention. Different columns correspond to weights derived from distinct inference sampling steps and different transformer blocks.
Figure 6: Results of the blind preference study for three metrics: video quality, audio quality, and audio-video synchronization. Ovi’s results are normalized to 100 as the baseline, with other methods reporting relative scores.
Method
Sync-C ↑
Sync-D ↓
Universe-1 [ 29 ]
1.85
11.97
OVI [ 21 ]
6.56
8.6
Sora2
5.35
7.78
UniTalking (Ours)
4.87
8.05
Table 1: Results of Lip-sync evaluation on T2AV
Method
Speaker Similarity ↑
English
Chinese
ElevenLabs
0.613
0.677
MiniMax [ 37 ]
0.756
0.780
Qwen3-Omni [ 34 ]
0.773
0.772
UniTalking (Ours)
0.703
0.662
Table 2: Results of Speaker Similarity on TR2AV. The text prompts from UniTalking supplement the video descriptions.
Type
Method
WER ↓
TTS
Fish Speech [ 17 ]
0.008
F5-TTS [ 3 ]
0.018
CosyVoice [ 7 ]
0.034
FireRedTTS [ 10 ]
0.038
Unified
Ovi-Aud [ 21 ]
0.035
UniTalking (Ours)
0.038
Table 3: Results of TTS evaluation. We follow the evaluation protocol of OVI [ 21 ] .
Joint audio-video generation models have shown that unified generation yields stronger cross-modal coherence than cascaded approaches. However, existing models couple modalities throughout denoising via pervasive attention, treating high-level semantics and low-level details in a fully entangled manner. This is suboptimal for talking head synthesis: while audio and facial motion are semantically correlated, their low-level realizations (acoustic signals and visual textures) follow distinct rendering processes. Enforcing joint modeling across all levels causes unnecessary entanglement and reduces efficiency. We propose Talker-T2AV, an autoregressive diffusion framework where high-level cross-modal modeling occurs in a shared backbone, while low-level refinement uses modality-specific decoders. A shared autoregressive language model jointly reasons over audio and video in a unified patch-level token space. Two lightweight diffusion transformer heads decode the hidden states into frame-level audio and video latents. Experiments on talking portrait benchmarks show Talker-T2AV outperforms dual-branch baselines in lip-sync accuracy, video quality, and audio quality, achieving stronger cross-modal consistency than cascaded pipelines.
Zhen Ye, Xu Tan, Aoxiong Yin +8
Hong Kong University of Science and Technology · Independent Researcher · Zhejiang University +3
Long-form and real-time talking-head generation remains challenging due to a latency-quality trade-off: inefficient multi-step diffusion prohibits streaming generation, whereas real-time autoregressive approaches suffer from error accumulation and identity drift. To address this drawback, we propose LeapTalk, a novel framework that achieves stable and real-time talking-head generation with a single forward step, scaling to arbitrarily long videos. At the heart of our approach lies a single-step bridge distillation scheme. On the one hand, departing from the conventional noise-to-data paradigm, we introduce a data-to-data transport formulation based on a Brownian bridge. Anchored by a persistent reference, this strategy effectively mitigates identity drift and enhances long-term temporal stability. On the other hand, to enable smooth knowledge transfer from a pre-trained diffusion teacher to the student bridge model, we explore a heterogeneous distillation framework with an SNR-aligned time transformation Φ(τ), which bridges the functional discrepancy between the two models. Moreover, we propose an audio-driven classifier-free guidance mechanism to maintain fine-grained lip synchronization under extreme step reduction. Extensive experiments demonstrate that our method achieves high-fidelity and temporally consistent video generation with only 1 step at up to 200 FPS, significantly outperforming existing approaches in both efficiency and stability. Project Page: https://zhangrongxiang.github.io/leaptalk-page/
Rongxiang Zhang, Songhua Liu
School of Artificial Intelligence, Shanghai Jiao Tong University · 2Harbin Institute of Technology
Generating realistic talking-head videos remains challenging due to persistent issues such as imperfect lip synchronization, unnatural motion, and evaluation metrics that correlate poorly with human perception. We propose FlowPortrait, a reinforcement-learning framework for audio-driven portrait animation built on a multimodal backbone for autoregressive audio-to-video generation. FlowPortrait introduces a human-aligned evaluation system based on Multimodal Large Language Models (MLLMs) to assess lip-sync accuracy, expressiveness, and motion quality. These signals are combined with perceptual and temporal consistency regularizers to form a stable composite reward, which is used to post-train the generator via Group Relative Policy Optimization (GRPO). Extensive experiments, including both automatic evaluations and human preference studies, demonstrate that FlowPortrait consistently produces higher-quality talking-head videos, highlighting the effectiveness of reinforcement learning for portrait animation.