cs.CVOct 5, 2026

Talk Like You: Imitating How You Speak in Real-Time Talking Head Generation

Authors: Baiqin Wang, Zhixing Ding, Jijie Li, Jiankuo Zhao, Zhen Lei, Xiangyu Zhu

Organizations: MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · CAIR, HKISI, Chinese Academy of Sciences · SCSE, FIE, M.U.S.T.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

In daily life, each person exhibits unique speaking habits, leading to subtle yet consistent lip-shape variations even when pronouncing the same word. Although recent talking head generation methods have achieved impressive visual fidelity and lip synchronization, they largely overlook user-specific customization, especially the motion patterns that characterize individual speaking habits. These habits are difficult to model and capture, as their motion patterns are highly fine-grained and often similar across individuals. As a result, many approaches produce overly uniform facial motions and fail to capture diverse, person-specific articulation patterns. To address this, we propose TalkLikeYou, an efficient framework that imitates how a target person speaks in talking head generation. Our method models habit in motion-space and achieves real-time performance through Flow Matching with only one sampling step during inference. We further adopt a two-stage imitation learning strategy to capture subtle distinctions between habits, allowing users to specify a target habit through either a preset style from the dataset or a reference video. In addition, we introduce a new metric PLAD that projects mouth motions onto representative articulation axes to evaluate imitation accuracy and generation diversity. Extensive experiments demonstrate that TalkLikeYou generates high-quality talking heads in real-time and significantly improves speaking habit imitation compared with prior methods. The code is available at: https://github.com/BQ-Wang0511/TalkLikeYou

Figures & tables

Figure 1: Overview of TalkLikeYou. Previous methods often generate uniform motions with limited control over speaking habits. We propose TalkLikeYou for efficient habit imitation in talking head generation, together with a new metric PLAD to evaluate habit similarity and diversity.
Figure 2: Framework of TalkLikeYou. Our method first generates habit-aware motion using the Flow Matching Motion Generator with optional habit sources through one-step sampling, and then renders them into a video. We further adopt a two-stage habit imitation learning strategy for more generalized and stable habit imitation. In the unified habit space, ⋆\star denotes the one-hot habit code and ∙\bullet denotes reference-video samples. During the first training stage, we use an alignment loss between reference video features and their corresponding one-hot habit codes.
Figure 3: Comparison Results. We compare our method with advanced baselines under three selected speaking habits, showing better visual quality and lip synchronization. Please zoom in for details. Video results are provided in the supplementary material.
MethodHDTFCelebV-HQ
Sync-C ↑\uparrowSync-D ↓\downarrowFID ↓\downarrowNIQE↓\text{NIQE}\downarrowFVD↓\text{FVD}\downarrowCSIM↑\text{CSIM}\uparrowSync-C ↑\uparrowSync-D ↓\downarrowFID ↓\downarrowNIQE↓\text{NIQE}\downarrowFVD↓\text{FVD}\downarrowCSIM↑\text{CSIM}\uparrow
SadTalk Zhang et al. (2023)7.667.3287.6045.72534.140.856.368.36109.1140.82829.090.80
Hallo2 Cui et al. (2025)7.337.9725.4113.02272.570.836.328.7363.9513.13688.030.75
Sonic Ji et al. (2025)8.626.7023.4212.91195.210.847.687.3566.3712.88704.120.76
Float Ki et al. (2025)7.048.1684.7012.39382.930.826.448.64137.2311.89854.890.70
Ditto Li et al. (2025)5.239.9421.6112.62319.030.875.809.1962.6712.13743.160.82
Table 1: Quantitative comparisons with state-of-the-art methods.
Figure 5
Figure 6
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: (a) Comparison of generation tendency under DDPM and Flow Matching. (b) PLAD histograms of different methods. (c) t-SNE visualization of the unified habit space, where stars denote preset one-hot habit features and dots denote reference-based habit features.
MethodsLip-syncImageTemporalHabitOverall
AccuracyQualityConsistencyImitationRealistic
Ditto Li et al. (2025)2.22.42.7—-2.5
Hallo2 Cui et al. (2025)2.22.83.3—-3.0
Sonic Ji et al. (2025)4.02.43.5—-3.6
SadTalker Zhang et al. (2023)1.21.92.4—-1.9
StyleTalk Ma et al. (2023a)1.31.91.61.51.7
Appendix
Table 3: User Study. The rating is on scale of 1-5 where higher rank demonstrate better results.
Figure 8: More results on habit imitation.
Figure 9: More comparison on Lip Synchronization with other baselines.
Figure 10: More visualization results on stylize faces.

Explore similar work

CardsList

  1. LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation

    Jul 29, 2026Rongxiang Zhang, Songhua LiuPrecise Lip SynchronizationTalk

    School of Artificial Intelligence, Shanghai Jiao Tong University · 2Harbin Institute of Technology

  2. SDTalk: Structured Facial Priors and Dual-Branch Motion Fields for Generalizable Gaussian Talking Head Synthesis

    May 11, 2026Peng Jia, Zhen Xiao, Jia Li +3Precise Lip SynchronizationTalk

    Hefei University of Technology, Hefei, China · University of Science and Technology of China, Hefei, China

  3. EmbodiedHead: Real-Time Listening and Speaking Avatar for Conversational Agents

    Apr 19, 2026Yu Zhang, Kaiyuan Shen, Yang LiAvatarsTurn-Taking

    School of Computer Science and Technology, East China Normal University, Shanghai, China · Garabido Shanghai Technology Co., Ltd., Shanghai, China