Generating realistic talking-head videos remains challenging due to persistent issues such as imperfect lip synchronization, unnatural motion, and evaluation metrics that correlate poorly with human perception. We propose FlowPortrait, a reinforcement-learning framework for audio-driven portrait animation built on a multimodal backbone for autoregressive audio-to-video generation. FlowPortrait introduces a human-aligned evaluation system based on Multimodal Large Language Models (MLLMs) to assess lip-sync accuracy, expressiveness, and motion quality. These signals are combined with perceptual and temporal consistency regularizers to form a stable composite reward, which is used to post-train the generator via Group Relative Policy Optimization (GRPO). Extensive experiments, including both automatic evaluations and human preference studies, demonstrate that FlowPortrait consistently produces higher-quality talking-head videos, highlighting the effectiveness of reinforcement learning for portrait animation.
Figures & tables
Figure 1: Overview of FlowPortrait . We build upon a pretrained MLLM to enable audio-to-video generation for portrait animation. Using an improved Flow-GRPO objective, we post-train the AR-Flow generator with a composite reward that integrates MLLM-based evaluations with perceptual and temporal consistency terms, leading to consistent improvements in generation quality.
Base Model
Method
Acc
Acc nt
Cov. (%)
Feature / Pixel Models
Inception-V3 [ 25 ]
FID
27.5
–
–
SyncNet [ 14 ]
LSE-C
53.4
–
–
SyncNet
LSE-D
51.6
–
–
Pixel-based
SSIM
61.2
–
–
Comparison Models
Table 1: Evaluation against human preferences. For score-based methods, Acc assigns half credit to ties. Cov. is the proportion of non-tied predictions, and Acc nt is accuracy among non-tied samples.
Model
Evaluation System
FVD ↓
FID ↓
LSE-D ↓
LSE-C ↑
SSIM ↑
SAS-SA ↑
MAS-SA ↑
MAS-MA ↑
Human ↑
Original
–
–
–
–
–
3.32
3.76
3.93
4.77
Sonic
62.25
44.14
12.05
2.91
0.633
2.94
3.43
3.58
3.56
Memo
74.94
41.12
13.65
0.92
0.668
2.72
3.31
3.37
3.21
Echomimics
97.06
60.15
13.18
1.51
0.573
2.55
3.16
3.26
2.54
Table 2: Scores assigned to different generation models by automated metrics, MLLM-based evaluators, and human annotators. Arrows indicate whether higher ( ↑ ) or lower ( ↓ ) values are better. Vertical separators partition evaluation systems into (i) automated metrics, (ii) MLLM-based evaluators, and (iii) human judgments.
Figure 2: Error rates of different generator types. Each bar represents the error rate for a specific generator type, with annotations indicating the number of error cases.
Figure 3: Score distribution histograms for the three evaluation aspects: lip-sync quality, facial expressiveness, and motion smoothness. Each histogram illustrates the frequency of scores assigned by Human (blue) and MAS-MA evaluation system (orange) across the range of possible scores (1 to 5).
In-domain Test Set
Out-domain Test Set
Generator
Lip-sync
Expressive
Motion
Avg
Lip-sync
Expressive
Motion
Avg
Original
4.03
4.33
3.43
3.93
4.74
4.42
2.93
4.03
Previous Generators
Echo
3.37
3.72
2.67
3.26
4.12
3.97
2.44
3.51
Memo
3.53
3.88
2.70
3.37
4.45
4.10
2.66
3.74
Sonic
3.73
4.07
2.93
3.58
4.45
4.15
2.63
3.74
Table 3: MLLM-based evaluation of different video generators on in-domain and out-domain test sets.
Generator
Lip-sync
Expressive
Motion
Avg
Orig
4.78
4.76
4.68
4.74
Sonic
3.59
3.47
3.21
3.42
SFT
3.87
3.75
3.55
3.72
RL
4.16
4.07
3.85
4.03
Table 4: Human annotation results comparing different generators, showing that RL post-training substantially improves generation quality and alignment with human preferences.
Figure 4: Cherrypicked failure cases from our SFT model, including blurred textures and hallucinated artifacts. RL post-training effectively mitigates these issues. The black masks are used to anonymize identities.
Figure 5: Comparison of reward curve during post-training for various noise levels during sampling stage.
Setting
W
Noise
Lip-sync
Expressive
Motion
Avg
SFT Baseline
N/A
N/A
3.74
4.07
3.16
3.66
RL Baseline ( RMLLM )
1
0.5
4.40
4.40
3.50
4.10
Ablation 1: Noise Level
Noise = 0.8
1
0.8
4.29 ( -0.11 )
4.33 ( -0.07 )
3.48 ( -0.02 )
4.03 ( -0.07 )
Noise = 1.0
1
1.0
4.12 ( -0.28 )
4.29 ( -0.11 )
3.41 ( -0.09 )
3.94 ( -0.16 )
Ablation 2: Window Size
Table 5: All ablations report score differences relative to the RL baseline. We found that small window size, small noise, and adding perceptual+consistency rewards with MLLM’s three aspect rewards produce the best results.
Figure 6: Comparing training-time metrics— Rperceptual and Rconsistency —reveals a clear gap between using only the MLLM-based reward and augmenting it with perceptual and consistency terms. Without these additional rewards, both metrics deteriorate sharply, aligning with the jittering and color drift observed in our qualitative results.
Precision: Mouth shapes perfectly match all spoken phonemes. Timing: Lip movements are perfectly synchronized with the audio, with no noticeable delay or lead. Naturalness: Movements are fluid and human-like, with no warping or visual artifacts.
4
Good
Precision: Mouth shapes are correct for the vast majority of sounds, with only subtle inaccuracies. Timing: Synchronization is excellent, with only an almost imperceptible lag or lead. Naturalness: Movements look natural but may lack fine-grained nuances.
3
Fair
Precision: Mouth shapes are generally correct but imprecise or simplified. Timing: A slight but consistent delay or lead is noticeable. Naturalness: Movements appear somewhat robotic or overly smooth; minor blurring artifacts may be present.
2
Poor
Precision: Mouth shapes are frequently incorrect. Timing: Audio–visual synchronization is clearly off and distracting. Naturalness: Movements are jerky or exhibit unnatural “flapping,” with obvious warping.
1
Bad
Precision: Lip movements have little to no correlation with speech. Timing: Audio and lip movements are severely out of sync. Naturalness: Mouth motion is highly distorted, nonsensical, or completely static.
Table 6: Human evaluation rubric for lip-sync quality (1–5).
Score
Guideline
Detailed Criteria (Emotional Match & Realism)
5
Excellent
Emotional Match: Facial expressions perfectly reflect the emotional tone of the audio. Realism: Includes natural micro-expressions and appears fully human and genuine.
4
Good
Emotional Match: Expressions largely match the audio’s emotion, with minor inconsistencies. Realism: Looks realistic but may feel slightly acted or less nuanced.
3
Fair
Emotional Match: Emotion is generally appropriate but muted, delayed, or underspecified. Realism: The face appears mildly uncanny or artificial.
2
Poor
Emotional Match: Expression mismatches the audio emotion or remains stuck in a neutral state. Realism: Expressions look highly unnatural and artificial.
1
Bad
Emotional Match: Expression is completely incorrect, nonsensical, or emotionless. Realism: Face resembles a static mask or mannequin.
Table 7: Human evaluation rubric for facial expressiveness (1–5).
Head Movement: Smooth, fluid, and purposeful. Subtle Motion: Natural blinks, head tilts, and minor posture shifts are present. Consistency: Motion is coherent and human-like throughout.
4
Good
Head Movement: Smooth and stable, but somewhat simple or repetitive. Subtle Motion: Blinking appears natural; other subtle motions may be limited. Consistency: Motion remains plausible with no major artifacts.
3
Fair
Head Movement: Slightly stiff, jerky, or floaty. Subtle Motion: Blinking is unnatural in cadence or absent for periods. Consistency: Occasional robotic or locked head poses occur.
2
Poor
Head Movement: Robotic, jittery, or laggy. Subtle Motion: No natural blinks or fine motions are observed. Consistency: Motion is highly unnatural and distracting.
1
Bad
Head Movement: Head is completely static. Subtle Motion: No blinking or facial motion at all. Consistency: The lack of motion is constant and clearly artificial.
Table 8: Human evaluation rubric for video smoothness and motion quality (1–5).
You are a specialized analyst focusing on audio-visual synchronization in digital media. Your expertise lies in phonetics and the precise mapping of mouth movements to speech.
Context
You will analyze a single video. Your sole focus is to evaluate and score its lip-sync quality. You must ignore all other aspects like emotional expression or head movement.
Primary Objective
Conduct a rigorous analysis and scoring of the lip-sync quality for this video. Your analysis must justify the score given and return a JSON dictionary with the score.
Evaluation Criteria
• Precision: How accurately do the lip movements match vowel/consonant shapes (visemes)? • Timing: Any noticeable delay/lead between audio and corresponding lip movements? • Naturalness: Human/fluid vs. artificial/robotic/flappy; warping or artifacts around the mouth?
Scoring Rubric (1–5)
• 5 (Perfect): Precision matches all phonemes; timing perfectly synchronized; natural, fluid, no artifacts. • 4 (Excellent): Mostly correct visemes with subtle inaccuracies; near-imperceptible lag/lead; slightly scripted. • 3 (Good/Acceptable): Generally correct but simplified/puppet-like; slight consistent lag/lead; minor artifacts possible. • 2 (Poor): Frequently incorrect visemes; clearly off-sync and distracting; robotic/jerky, obvious warping/flapping. • 1 (Failure): No correlation to speech; severely out of sync; bizarre/distorted/nonsensical or static mouth.
Output Constraints
1. Put all reasoning and analysis inside <think>...</think> . 2. After </think> , output only a valid JSON object with key "score" and an integer 1--5 . 3. Do not include natural language, commentary, markdown, or extra quotes outside the <think> block. 4. The final JSON must be valid and contain only the score.
Table 9: Prompt used for lip-sync evaluation. The model is instructed to assess only audio-visual synchronization and to return a single 1–5 score in JSON format.
You are a specialized analyst in human emotion and facial expression, with expertise in facial action coding systems (FACS) and psychological realism in digital actors.
Context
You will analyze a single video. Your sole focus is to evaluate and score its facial expressiveness and emotional authenticity. You must apply the scoring rubric below to assign a score from 1 to 5.
Primary Objective
Conduct a rigorous analysis of the video’s facial expressiveness, assign a score based on the provided rubric, and return a JSON dictionary with the score.
Evaluation Criteria
• Emotional Match: Does the facial expression accurately reflect the audio’s tone, sentiment, and emphasis? • Realism & Subtlety: Are there subtle micro-expressions (e.g., brow movements, eye crinkling, cheek twitches), or does the face appear uncanny or plastic? • Natural Dynamics: Do expressions transition smoothly with speech, or appear robotic, delayed, or disconnected?
Scoring Rubric (1–5)
• 5 (Excellent): Perfect emotional alignment with the audio; rich micro-expressions; fully genuine and human-like. • 4 (Good): Strong emotional match with minor inconsistencies; realistic but slightly acted or less nuanced. • 3 (Average): Generally appropriate emotion but muted, delayed, or underspecified; mildly uncanny or plastic. • 2 (Poor): Clear emotional mismatch or fixed neutral/awkward state; highly unnatural and artificial. • 1 (Very Poor): Completely incorrect or absent emotion; static, mask-like, or mannequin-like face.
Output Constraints
1. Put all reasoning and analysis inside <think>...</think> . 2. After </think> , output only a valid JSON object with key "score" and an integer 1--5 . 3. Do not include natural language, commentary, markdown, or extra quotes outside the <think> block. 4. The final JSON must be valid and contain only the score.
Table 10: Prompt used for evaluating facial expressiveness and emotional authenticity. The model assesses emotional alignment and realism of facial expressions and returns a single 1–5 score in JSON format.
Prompt Name
Motion Smoothness and Background Quality Evaluation Prompt (Single-Video, Single-Aspect)
Role
You are a specialized analyst in motion quality and video composition. Your expertise lies in identifying motion artifacts, assessing movement naturalness, and spotting background anomalies in synthesized media.
Context
You will analyze a single video. Your sole focus is to evaluate and score its motion smoothness and background integrity. You must ignore all other aspects such as lip-sync or facial emotion.
Primary Objective
Conduct a rigorous analysis of the video’s motion quality and background stability, assign a score using the provided rubric, and return a JSON dictionary with the score.
Evaluation Criteria
• Head Movement: Are head motions smooth, fluid, and purposeful, or stiff, jittery, and robotic? • Subtle Motions: Are natural micro-movements (e.g., blinking, slight tilts, posture shifts) present and realistic? • Consistency: Does motion remain coherent and human-like throughout, without locking or abrupt artifacts? • Background Quality: Is the background stable, with clean edges around hair and shoulders, or does it exhibit warping, shimmering, or distortion?
Scoring Rubric (1–5)
• 5 (Excellent): Perfectly smooth and purposeful head motion; rich subtle movements; fully consistent and human-like; background entirely stable with no artifacts. • 4 (Good): Smooth and consistent motion but somewhat simple or repetitive; natural blinking; minor, non-distracting background artifacts. • 3 (Average): Slightly stiff, jerky, or floaty motion; unnatural blinking cadence or frequent stillness; noticeable but moderate background warping or blur. • 2 (Poor): Clearly robotic or jittery motion with unnatural jerks; no subtle movements; distracting instability; obvious background distortion or melting. • 1 (Failure): Completely static or locked head; frozen face with no movement; entirely artificial appearance; severe background and subject distortion.
Output Constraints
1. Put all reasoning and analysis inside <think>...</think> . 2. After </think> , output only a valid JSON object with key "score" and an integer 1--5 . 3. Do not include natural language, commentary, markdown, or extra quotes outside the <think> block. 4. The final JSON must be valid and contain only the score.
Table 11: Prompt used for evaluating motion smoothness and background integrity. The model assesses head motion dynamics, subtle movements, and background stability, and returns a single 1–5 score in JSON format.
Figure 7: Qualitative examples comparing prior methods with our SFT and RL post-trained models. Our approach generates videos with more natural movement, better lip-sync, and improved expressiveness. Black masks are used to anonymize identities.
Figure 8: Examples of generated video frames illustrating the effect of different post-training configurations. Good cases (Rows 1 and 3): Our final post-trained model (all three aspect rewards with perceptual and consistency rewards, W=1 , η=0.5 ) produces consistent identity, stable colors, and smooth temporal dynamics. Bad case 1 (Row 2): Training with a large stochastic window leads to strong temporal color drift and bad texture. Bad case 2 (Row 4): Training with only the lip-sync reward introduces jittering and unnatural motion, including exaggerated eye movements. Black masks are used to anonymize identities.
Speech-driven talking character animation seeks to generate life-like portrait videos that convey natural conversation behavior, aligning facial motion with spoken audio. Although recent advances in video generation have substantially improved realism in video-based animation, achieving both accurate lip articulation and expressive behavior remains challenging. Existing approaches typically trade off precise phoneme-to-lip synchronization against dynamic facial expressions and head motion, yielding animations that are either accurate yet rigid, or expressive but poorly synchronized. We address this challenge by proposing ReFree-S2V, a flow-matching speech-to-portrait animation framework that builds upon a pretrained video generation model to achieve fine-grained speech articulation and high-level expressive cues in speech-driven portrait animation. This model introduces a multi-level speech representation capturing phonetic and prosodic information at both local and global granularities. These representations are selectively injected into transformer blocks via learnable level selectors, enabling both accurate lip synchronization and natural expressive motion. To achieve natural head movements, we further introduce a novel reward-free reinforcement learning scheme into flow-matching training to discourage perceptually implausible motion without relying on handcrafted synchronization metrics or reward models, or the high cost of human preference annotation. Extensive experiments demonstrate that ReFree-S2V achieves state-of-the-art performance, significantly outperforming existing methods in both quantitative lip-sync accuracy and qualitative human evaluations of naturalness and expressivity.
Salaheldin Mohamed, M. Hamza Mughal, Rishabh Dabral +1
Télécom Paris, Institut Polytechnique de Paris · Max Planck Institute for Informatics
While state-of-the-art audio-video generation models like Veo3 and Sora2 demonstrate remarkable capabilities, their closed-source nature makes their architectures and training paradigms inaccessible. To bridge this gap in accessibility and performance, we introduce UniTalking, a unified, end-to-end diffusion framework for generating high-fidelity speech and lip-synchronized video. At its core, our framework employs Multi-Modal Transformer Blocks to explicitly model the fine-grained temporal correspondence between audio and video latent tokens via a shared self-attention mechanism. By leveraging powerful priors from a pre-trained video generation model, our framework ensures state-of-the-art visual fidelity while enabling efficient training. Furthermore, UniTalking incorporates a personalized voice cloning capability, allowing the generation of speech in a target style from a brief audio reference. Qualitative and quantitative results demonstrate that our method produces highly realistic talking portraits, achieving superior performance over existing open-source approaches in lip-sync accuracy, audio naturalness, and overall perceptual quality.
Hebeizi Li, Zihao Liang, Benyuan Sun +4
Central Media Technology Institute, Huawei · School of Computer Science and Engineering, Beihang University
Recent video generative models have shown strong capability in synthesizing high-quality and temporally coherent videos. Yet, adapting them for controllable portrait animation remains nontrivial, as existing methods typically rely on architectural changes or extra conditioning modules, increasing complexity and reducing compatibility with pretrained models. We present CatPortrait, a simple and effective framework that enables controllable portrait animation via 3D face guidance as in-context conditioning. CatPortrait preserves the original video generation backbone and injects structured 3D facial guidance into the input space through spatial concatenation, allowing pretrained image-to-video models to animate portraits without sacrificing their pretrained priors. CatPortrait supports both video-driven and audio-driven portrait animation within a unified framework. Extensive experiments demonstrate that CatPortrait achieves competitive or superior results across both settings.
Ziqi Ni, Zhou Fang, Shijian Wang +2
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education, China