VidDub: Replacement-Aware Distillation for Two-Step Video Dubbing
Organizations: HKUST · Tencent · ZJU · Independent Researcher ·
Abstract
Video dubbing aims to synthesize lip movements that match target speech while preserving the speaker identity, head pose, and background of a source video. Existing editors are predominantly trained via self-reconstruction under original paired audio, even though the source mouth becomes a conflicting target when driving speech is replaced. We introduce replacement-aware cross-audio supervision: paired-audio samples use full-frame reconstruction, whereas unpaired replacement-audio samples mask out source-mouth reconstruction losses and supervise the mouth solely through audio-visual synchronization while preserving unmasked facial regions and background. Because offline dubbing provides the complete source video and target speech beforehand, we present VidDub, a full-context diffusion editor distilled into two denoising steps (2 NFE). Our replacement-aware teacher first learns accurate lip synchronization without regressing toward conflicting source mouth movements. Matched trajectory regression then transfers the teacher's CFG-guided trajectories to a two-step student, and distribution matching distillation recovers visual quality after step compression. We will release the curated training dataset, VidDub-Bench, fixed paired- and cross-audio evaluation manifests, preprocessing outputs, and unified evaluation code. Across three training seeds, masking out conflicting source-mouth reconstruction improves cross-audio Sync-C by 0.833; the final two-step model combines strong lip synchronization with superior visual quality, identity preservation, and visual speech recognition across four benchmarks.
Figures & tables
| Reconstruction | Replacement Dubbing | |||||||||
| Method | Params | NFE | FPS | Sync-C | Sync-D | SSIM | PSNR | Sync-C | Sync-D | FVD |
| Unedited source | – | 0 | – | 8.723 | 6.666 | – | – | 1.283 | 14.483 | – |
| Feed-forward and one-step editors | ||||||||||
| Wav2Lip [ 31 ] | 36M | 1 | 5.2 | 9.177 | 6.555 | 0.9233 | 28.97 | 8.782 | 6.972 | 129.51 |
| VideoReTalking [ 5 ] | 484M | 1 | 1.3 | 8.801 | 6.593 | 0.8981 | 27.85 | 8.312 | 6.988 | 105.72 |
| MuseTalk 1.5 [ 55 ] | 971M | 1 | 1.1 | 8.471 | 6.895 | 0.9489 | 33.74 | 7.116 | 7.880 | 40.10 |
| Reconstruction | Replacement Dubbing | ||||||||||
| Method | Params | NFE | Scored S/C | FPS | Sync-C | Sync-D | SSIM | PSNR | Sync-C | Sync-D | FVD |
| Feed-forward and one-step editors | |||||||||||
| Wav2Lip [ 31 ] | 36M | 1 | 499/499 | 5.2 | 7.209 | 7.420 | 0.9218 | 29.83 | 6.750 | 7.932 | 111.19 |
| VideoReTalking [ 5 ] | 484M | 1 | 402/402 | 1.3 | 6.787 | 7.552 | 0.8948 | 28.00 | 6.365 | 7.868 | 84.51 |
| MuseTalk 1.5 [ 55 ] | 971M | 1 | 500/499 | 1.1 | 4.968 | 9.106 | 0.9305 | 29.60 | 4.371 | 9.625 | 36.09 |
| Bidirectional diffusion editors (20–50 NFE) | |||||||||||
| Reconstruction | Replacement Dubbing | |||||||||
| Method | Params | NFE | FPS | Sync-C | Sync-D | SSIM | PSNR | Sync-C | Sync-D | FVD |
| Feed-forward and one-step editors | ||||||||||
| Wav2Lip | 36M | 1 | 5.2 | 8.269 | 7.193 | 0.9063 | 29.44 | 6.922 | 8.536 | 98.62 |
| VideoReTalking | 484M | 1 | 1.3 | 8.097 | 7.088 | 0.8773 | 26.77 | 6.873 | 8.069 | 65.43 |
| MuseTalk 1.5 | 971M | 1 | 1.1 | 6.430 | 8.673 | 0.9078 | 28.27 | 4.184 | 10.670 | 56.73 |
| Bidirectional diffusion editors (20–50 NFE) | ||||||||||
| Benchmark | Paired-audio only | Replacement-aware |
|---|---|---|
| HDTF-30 | 7.986 / 26.40 | 9.422 / 27.49 |
| TalkVid-Bench-50 | 6.460 / 26.14 | 7.414 / 26.51 |
| H3-Val-50 | 6.952 / 37.68 | 8.086 / 38.09 |
| VoxCeleb2-30 | 5.729 / 39.95 | 7.382 / 39.11 |
| Factor | Sync-C | Sync-D | FVD |
|---|---|---|---|
| Full objective paired only | |||
| Source-mouth mask (M) | |||
| Audio dropout (D) | |||
| Timestep range (T) |
| Variant | SSIM | Sync-C | FVD |
|---|---|---|---|
| Trajectory only (no DMD) | 0.9369 | 8.594 | 10.84 |
| DMD, unguided real score | 0.9402 | 8.517 | 7.00 |
| DMD, no preservation loss | 0.9381 | 8.733 | 7.69 |
| Complete DMD recipe | 0.9382 | 8.731 | 7.76 |
| w/o replacement audio | 0.9398 | 8.577 | 7.15 |
| Method | HDTF | Vox2 | VDB |
|---|---|---|---|
| Real (same audio) | 0.171 | 0.579 | 0.711 |
| Wav2Lip [ 31 ] | 1.093 | 1.009 | 1.020 |
| VideoReTalking [ 5 ] | 0.915 | 1.056 | 0.975 |
| Diff2Lip [ 28 ] | 0.957 | 0.963 | 0.990 |
| MuseTalk [ 55 ] | 0.987 | 1.068 | 1.014 |
| LatentSync [ 21 ] | 1.019 | 1.049 | 0.953 |
| Method | Sync-C | Upper-L1 | ID | Pose | |
|---|---|---|---|---|---|
| Wav2Lip | 6.922 | 1.820 | 0.909 | 0.459 | 256 |
| Diff2Lip | 7.575 | 1.283 | 0.862 | 0.518 | 256 |
| LatentSync | 7.544 | 1.605 | 0.913 | 0.422 | 254 |
| Lip Forcing | 5.192 | 3.761 | 0.900 | 0.928 | 256 |
| Full-context teacher | 8.718 | 1.207 | 0.872 | 0.573 | 256 |
| VidDub | 8.709 | 1.199 | 0.875 | 0.579 | 256 |
| Opponent | Sync | Real. | Identity | Stability | Overall |
|---|---|---|---|---|---|
| Teacher | 10/10/130 | 9/12/129 | 4/8/138 | 4/14/132 | 12/20/118 |
| LatentSync | 123/2/25 | 122/6/22 | 75/3/72 | 123/2/25 | 143/2/5 |
| Lip Forcing | 142/0/8 | 90/15/45 | 61/2/87 | 87/2/61 | 146/1/3 |
| Method | Valid | S-C | S-D | Last-C | Keep-L1 | Chroma | Keep |
|---|---|---|---|---|---|---|---|
| Wav2Lip [ 31 ] | 20/20 | 7.861 | 7.484 | 8.032 | 3.045 | 1.045 | +0.009 |
| LatentSync [ 21 ] | 20/20 | 7.771 | 7.083 | 7.844 | 4.018 | 0.397 | +0.685 |
| VidDub | 20/20 | 9.399 | 6.162 | 9.585 | 2.493 | 0.250 |
| Execution Design | First-Out | FPS | Peak GiB | VSR Wins | ID |
|---|---|---|---|---|---|
| Causal two-step | 1.052 s | 9.4 | 21.96 | 0/6 | 0.861 |
| Full-context (VidDub) | 7.165 s | 11.3 | 13.77 | 6/6 | 0.885 |
Appendix figures & tables66 assets
Supplementary material from the paper’s appendix.
Appendix
| Design | First output | Frames | 81-frame time | Peak GiB | VSR wins | Upper-L1 | ID | Pose |
|---|---|---|---|---|---|---|---|---|
| Causal two-step | 1.052 s | 9 | 8.662 s | 21.96 | 0/6 | 1.079 | 0.861 | 0.614 |
| Full-context two-step (VidDub) | 7.165 s | 81 | 7.195 s | 13.77 | 6/6 | 1.014 | 0.885 | 0.567 |
| Statistic | Value |
|---|---|
| Structured prompts | 100,000 |
| Frozen generated source videos | 256 |
| Same-audio tracks / replacement pairs | 256 / 256 |
| Frames / duration / frame rate | 81 / 3.375 s / 24 FPS |
| Model-ready spatial resolution | |
| Total unique-source duration | 14.4 min |
| Source set | Non-mouth | Center motion | Scale CV | |
|---|---|---|---|---|
| HDTF | 30 | 1.30 | 1.03 | 6.17 |
| VoxCeleb2 | 30 | 4.83 | 1.82 | 5.86 |
| TalkVid | 500 | 2.14 | 1.45 | 8.94 |
| VidDub-Bench | 256 | 4.57 | 2.29 | 6.31 |
| Benchmark | Same Sync-C | Same SSIM |
|---|---|---|
| HDTF | 0.92 | -0.10 |
| VoxCeleb2 | 0.98 | -0.30 |
| TalkVid | 0.98 | 0.23 |
| VidDub-Bench | 0.85 | 0.60 |
| Captured AIGC cross Sync-C | ||
| Stage | Initialized from | Frames | GPUs | Updates | Adapter LR | DiT | VACE LR | Cross audio | Auxiliary losses |
| Audio-conditioning warm-up | scratch adapter | 49 | 48 | 6,000+2,000+9,700 | frozen | frozen | 0 | flow | |
| Joint editing adaptation | warm-up | 81 | 48 | 3,000+2,500 | full, | 0 | flow | ||
| Replacement-aware teacher refinement | joint adaptation | 81 | 16 | 300 | LoRA-64, | 0.30 | Sync, LPIPS, repr. alignment | ||
| Auxiliary ablations | |||||||||
| Quality-only refinement | joint adaptation | 81 | 48 | 2,000 | LoRA-64, | 0 | LPIPS, repr. alignment | ||
| Replacement-after-quality branch | quality-only EMA@500 | 81 | 48 | 300 | LoRA-64, | 0.30 | Sync, LPIPS, repr. alignment | ||
| Checkpoint | SSIM | Same C/D | Cross C/D | Cross FVD |
|---|---|---|---|---|
| Joint editing adaptation | 0.9230 | 7.355 / 8.086 | 5.825 / 9.603 | 33.50 |
| Quality-only refinement, EMA | 0.9233 | 7.561 / 7.845 | 6.104 / 9.256 | 29.56 |
| Replacement-after-quality branch, EMA | 0.9216 | 9.371 / 6.464 | 9.135 / 6.631 | 31.59 |
| Replacement-aware teacher, EMA | 0.9279 | 9.760 / 5.900 | 9.422 / 6.127 | 27.49 |
| Teacher refinement | HDTF-30 | TalkVid-Bench-50 | H3-Val-50 | VoxCeleb2-30 |
|---|---|---|---|---|
| No replacement branch | 7.986 / 26.40 | 6.460 / 26.14 | 6.952 / 37.68 | 5.729 / 39.95 |
| Replacement-aware (0.30) | 9.422 / 27.49 | 7.414 / 26.51 | 8.086 / 38.09 | 7.382 / 39.11 |
| Comparison | Isolated factor | Primary metrics | Required direction | Interpretation |
|---|---|---|---|---|
| Warm-up joint adaptation | backbone/VACE adaptation | same/cross C/D, FVD | cross C up | audio features become usable by the generator |
| Joint adaptation quality-only | LPIPS+repr. alignment+LoRA+EMA | SSIM, mouth LPIPS, FVD | visual criteria met | improves visual metrics while reporting synchronization |
| No-replacement replacement-aware refinement | implemented replacement branch | cross C/D, FVD | cross C improves | matched replacement-sampling comparison; dropout semantics disclosed |
| live EMA | parameter averaging | all metrics, failures | lower variance | separates optimization noise from method gain |
| reference/dropout sweep | conditioning strength | cross-audio synchronization, identity | Pareto improvement | tests appearance-prior competition |
| Causal adaptation | Two-step initialization | Guided DMD | |
| Architecture and sampling | |||
| Backbone | Wan-1.3B + VACE | same | same |
| Audio module | Whisper-small + 287M adapter | same | same |
| Input clip | 81 frames, , 24 fps | same | same |
| Latent layout | 1 reference + 21 video latents | same | same |
| Causal blocks | 7 blocks 3 latents | same | same |
| Causal adaptation | Two-step initialization | Guided DMD | |
| Optimization | |||
| Optimizer / schedule | AdamW / cosine | AdamW / cosine | AdamW / cosine |
| Adapter or student learning rate | |||
| Wan/VACE learning rate | |||
| Fake-score learning rate (adapter / base) | – | – | / |
| Quantity | Median |
|---|---|
| Condition preparation | 0.014 s |
| Time to first block | 0.678 s |
| Denoising | 5.632 s |
| VAE decode | 2.960 s |
| Prepared-clip total | 8.606 s |
| Throughput | 9.412 fps |
| Duration (s) | Sources | RTF | Peak GiB | Sync-C / D | Keep L1 | Boundary mouth | Sync failures |
|---|---|---|---|---|---|---|---|
| 10 | 20 | 4.46 | 23.04 | 9.577 / 6.604 | 2.634 | 4.373 | 0 |
| 30 | 20 | 4.49 | 23.06 | 9.539 / 6.760 | 2.474 | 4.510 | 0 |
| 60 | 20 | 4.48 | 23.10 | 9.577 / 6.814 | 2.430 | 4.383 | 0 |
| Method | NFE | Peak GiB | Sync-C | Sync-D | Keep L1 | Color | Sync failures |
|---|---|---|---|---|---|---|---|
| Causal AR-48 (CFG 2) | 48 | 29.09 [29.09,29.09] | 9.153 [8.774,9.505] | 6.995 [6.832,7.172] | 2.701 [2.232,3.264] | 0.689 [0.627,0.748] | 0 |
| Trajectory ODE-2 | 2 | 23.27 [23.27,23.27] | 9.019 [8.638,9.376] | 7.130 [6.961,7.313] | 2.930 [2.474,3.478] | 0.867 [0.803,0.931] | 0 |
| Guided DMD-2 | 2 | 23.10 [23.10,23.10] | 9.163 [8.773,9.523] | 7.003 [6.831,7.193] | 2.821 [2.358,3.376] | 0.713 [0.659,0.770] | 0 |
| ID | Intervention | Diagnostic purpose | Blocks per sample |
|---|---|---|---|
| A3 | one uniformly sampled block | sparse-coverage reference | 1/7 |
| B3 | sample without replacement | block-coverage sensitivity | 3/7 |
| B7 | supervise every block | complete block coverage | 7/7 |
| S08 | noise-range sensitivity | 3/7 | |
| SW06 | loss-weight sensitivity | 3/7 | |
| X30 | S08 plus 30% counterfactual audio | replacement-audio supervision | 3/7 |
| Intervention | SSIM | Mouth LPIPS | Same C/D | Cross C/D | FVD S/C | Primary change |
|---|---|---|---|---|---|---|
| Three-block reference | 0.9310 | 0.1441 | 7.610 / 7.866 | 6.181 / 8.758 | 41.35 / 57.15 | 3/7 blocks |
| Broader Sync exposure | 0.9311 | 0.1439 | 7.981 / 7.582 | 6.807 / 8.232 | 41.36 / 56.20 | |
| Larger Sync weight | 0.9310 | 0.1435 | 7.827 / 7.714 | 6.507 / 8.532 | 40.93 / 56.38 | |
| Counterfactual speech | 0.9217 | 0.1707 | 8.619 / 7.251 | 7.921 / 7.492 | 54.41 / 63.82 | 30% cross audio |
| Weaker mouth history | 0.9323 | 0.1382 | 7.539 / 7.917 | 6.227 / 8.712 | 39.79 / 56.00 | 50% suppression |
| Shorter chunks | 0.9259 | 0.1551 | 7.563 / 7.843 | 6.337 / 8.615 | 45.46 / 57.05 | two latent frames |
| CFG | Sync-C | Sync-D | offset | Mouth |
|---|---|---|---|---|
| 0.0 | 1.878 | 13.260 | 10.19 | 9.04 |
| 1.0 | 6.181 | 8.758 | 0.00 | 0.00 |
| 1.5 | 7.245 | 7.915 | 0.06 | 3.82 |
| 2.0 | 7.607 | 7.638 | 0.07 | 6.03 |
| 3.0 | 7.889 | 7.536 | 0.00 | 9.69 |
| Method | SSIM | Mouth LPIPS | Same C | Cross C | FVD same | FVD cross |
|---|---|---|---|---|---|---|
| Full-context parent | 0.951154 | 0.096476 | 8.505424 | 8.050768 | 6.676098 | 11.701082 |
| A0 Flow | 0.939031 | 0.126039 | 7.628839 | 6.194756 | 8.257593 | 12.564611 |
| A1 Flow+Q | 0.938839 | 0.125288 | 7.631446 | 6.158610 | 8.078579 | 12.433051 |
| A2 Flow+Sync | 0.939122 | 0.125797 | 7.796448 | 6.468818 | 7.990434 | 12.506445 |
| A3 Flow+Q+Sync | 0.938953 | 0.124887 | 7.776736 | 6.450379 | 7.767604 | 12.377647 |
| A4 + representation alignment | 0.938046 | 0.126844 | 7.829486 | 6.560639 | 7.800708 | 12.940533 |
| Model | Updates | SSIM | Mouth LPIPS | Same C | Cross C | Cross D | FVD S/C |
|---|---|---|---|---|---|---|---|
| CF re-noise parent | 0 | 0.94580 | 0.11099 | 7.923 | 6.418 | 8.622 | 30.82 / 48.62 |
| CD0 raw | 200 | 0.94409 | 0.11232 | 7.930 | 6.395 | 8.612 | 31.68 / 49.41 |
| CD0 EMA | 200 | 0.94496 | 0.11201 | 7.939 | 6.333 | 8.663 | 31.97 / 49.10 |
| CD1 raw | 200 | 0.94379 | 0.11197 | 7.965 | 6.325 | 8.670 | 31.40 / 49.35 |
| CD1 EMA | 200 | 0.94538 | 0.11176 | 7.915 | 6.346 | 8.658 | 32.16 / 50.31 |
| Study | NFE | Real | Branches | SSIM | Mouth LPIPS | Cross C/D | FVD S/C |
|---|---|---|---|---|---|---|---|
| AR parent | 50 | – | 1 | 0.9388 | 0.1261 | 7.951 / 7.603 | 9.19 / 13.99 |
| Four-step ODE | 4 | – | 1 | 0.9326 | 0.1403 | 8.033 / 7.562 | 10.38 / 14.45 |
| Four-step DMD | 4 | 2.0 | 1 | 0.9377 | 0.1288 | 8.547 / 7.264 | 9.13 / 14.30 |
| Unified ODE/DMD | 2, 4 | 2.0 | 1 | selected under common quality, sync, and failure criteria | |||
| Trajectory teacher | Teacher CFG | Sync-C | Sync-D | Keep L1 | FVD |
|---|---|---|---|---|---|
| Unguided trajectories | 1 | 8.302 [8.159,8.449] | 7.135 [7.035,7.236] | 1.996 [1.952,2.042] | 10.52 |
| Guided trajectories | 2 | 8.641 [8.499,8.788] | 7.069 [6.971,7.168] | 1.974 [1.931,2.020] | 11.48 |
| Stage | Br. | SSIM | Sync-C | FVD |
|---|---|---|---|---|
| Teacher | 2 | 0.9476 | 8.751 | 6.66 |
| Direct causalization | 1 | 0.9387 | 7.993 | 8.21 |
| + replacement audio | 1 | 0.9390 | 8.121 | 8.33 |
| + guidance | 2 | 0.9369 | 8.726 | 8.91 |
| Variant | Real | Init. Consistency | SSIM | Mouth LPIPS | Cross C/D | FVD S/C |
|---|---|---|---|---|---|---|
| Complete causal DMD | 2.0 | 0.1 | 0.9382 | 0.1348 | 8.731 / 7.022 | 6.86 / 7.76 |
| Unguided real score ( ) | 1.0 | 0.1 | 0.9402 | 0.1303 | 8.517 / 6.953 | 5.26 / 7.00 |
| No ODE retention | 2.0 | 0 | 0.9381 | 0.1350 | 8.733 / 7.018 | 6.83 / 7.69 |
| No replacement audio | 2.0 | 0.1 | 0.9398 | 0.1305 | 8.577 / 7.080 | 6.09 / 7.15 |
| Cell | Retention | Cross Sync-C | Cross Sync-D | Cross FVD | ||
|---|---|---|---|---|---|---|
| F11 | 1 | 1 | 0.1 | 8.371 | 7.070 | 7.21 |
| F12 | 1 | 2 | 0.1 | 8.578 | 7.120 | 7.66 |
| F21 | 2 | 1 | 0.1 | 8.512 | 6.954 | 7.26 |
| F22 (complete) | 2 | 2 | 0.1 | 8.732 | 7.023 | 7.74 |
| Endpoint | Cross Sync-C | Cross Sync-D | Cross FVD |
|---|---|---|---|
| Matched control P1 | |||
| Complete P4 |
| Model | Correct C | Shifted C | Unrelated C | Null T C | Null S C | Keep SSIM | Correct FVD |
|---|---|---|---|---|---|---|---|
| 48-step teacher | 8.112 | 7.584 | 8.032 | 1.813 | 3.499 | 0.97994 | 28.96 |
| Matched control P1 | 8.262 | 7.706 | 8.236 | 1.606 | 3.168 | 0.97837 | 35.41 |
| F21 | 8.431 | 7.916 | 8.370 | 1.583 | 3.140 | 0.97848 | 35.38 |
| F22 | 8.625 | 8.183 | 8.592 | 1.639 | 3.281 | 0.97845 | 39.83 |
| Stage | Variants | Question | Selection criterion |
|---|---|---|---|
| AR coverage | A3, B3, B7 | Does supervision cover temporal/audio positions? | better mouth LPIPS and FVD without cross-sync loss |
| AR audio | B7, +S08, +cross 5/10%, warmup | Can audio evidence increase without X30 artifacts? | cross C/D improve; visual tolerance passes |
| AR history | GT, suppression, resample, audio-consistent rollout | How does history construction affect causal generation? | long-horizon and cross metrics improve jointly |
| Initializer | Euler4, re-noise4, ODE, CD0, CD1 | Which four-step objective retains the AR parent? | joint development quality/sync Pareto frontier |
| DMD guidance | real | Can teacher CFG be absorbed into one branch? | cross C/D improve at inference CFG 1 |
| DMD initializer consistency | none, ODE anchor, +quality, +cross keep/Sync | What prevents color and mouth artifacts? | FVD/LPIPS/color improve without synchronization loss |
| Benchmark / method | Upper-L1 | ID cosine | Pose error ( ∘ ) | |
|---|---|---|---|---|
| HDTF / Wav2Lip | 2.580 | 0.928 | 0.432 | 30 |
| HDTF / Diff2Lip | 1.753 | 0.888 | 0.464 | 30 |
| HDTF / LatentSync | 1.646 | 0.970 | 0.228 | 30 |
| HDTF / Lip Forcing | 1.669 | 0.934 | 0.340 | 30 |
| HDTF / Teacher | 0.873 | 0.911 | 0.415 | 30 |
| HDTF / Full-context 2-step | 0.914 | 0.902 | 0.449 | 30 |
| Method | Slope (s/s) | Fixed (s) | 4 s clip | 8 s clip |
|---|---|---|---|---|
| Wav2Lip (CLI) [ 31 ] | 4.77 | 137.1 | 162.3 | 170.4 |
| VideoReTalking (CLI) [ 5 ] | 19.16 | 90.6 | 167.2 | 243.8 |
| MuseTalk 1.5 (CLI) [ 55 , 41 ] | 22.02 | 67.1 | 155.2 | |
| Diff2Lip (CLI) [ 28 ] | 12.55 | 5.2 | 55.3 | 105.5 |
| LatentSync 1.6 [ 21 , 1 ] | 11.38 | 13.5 | 59.3 | 103.5 |
| X-Dub (CLI) [ 14 ] | 56.11 | 236.4 | 460.8 | 685.2 |
| Method | Valid | Sync-C | Sync-D | First C | Last C | Slope/min | Keep-L1 | Chroma | Keep | Chroma |
|---|---|---|---|---|---|---|---|---|---|---|
| Wav2Lip [ 31 ] | 20/20 | 7.861 | 7.484 | 8.147 | 8.032 | 3.045 | 1.045 | +0.009 | +0.003 | |
| LatentSync 1.6 [ 21 ] | 20/20 | 7.771 | 7.083 | 8.029 | 7.844 | 4.018 | 0.397 | +0.685 | +0.006 | |
| VidDub | 20/20 | 9.399 | 6.162 | 9.755 | 9.585 | 2.493 | 0.250 | +0.006 |
| Metric | Persistent | Re-anchored |
|---|---|---|
| Sync-C | 9.578 | 9.747 |
| Sync-D | 6.772 | 6.106 |
| Peak memory (GiB) | 23.10 | 22.38 |
| Skin chroma MAE | 7.48 | 2.43 |
| Lower-face stability | no | yes |
| Benchmark | Sources | Source duration / native FPS | Tracks | Cross pairing | VidDub execution |
|---|---|---|---|---|---|
| HDTF-30 | 30 | 8.0s / 25 | same + cross | fixed cyclic, different speaker | 81-frame windows, shifted tail |
| VoxCeleb2-30 | 30 | 4.0s / 25 | same + cross | fixed cyclic, different speaker | 81-frame windows, shifted tail |
| TalkVid-Bench | 500 | 5.042s / 24 | same + cross | deterministic within stratum | 81-frame windows, shifted tail |
| VidDub-Bench | 256 | 3.375s / 24 | same + cross | fixed multilingual pairs | one 81-frame full-context window |
| Frozen-50 | 50 | 3.375s / 24 | cross | identical clips for compared methods | matched blinded comparison |
| Long-20 | 20 | 60s / 25 | cross primary | frozen contiguous pairs | 81-frame windows, 17-frame overlap |
| Method | Scoreable | Sync-C | Sync-D | FVD |
|---|---|---|---|---|
| Wav2Lip | 255/256 | 6.930 | 8.514 | 99.05 |
| VideoReTalking | 252/256 | 6.883 | 8.052 | 66.18 |
| MuseTalk 1.5 | 253/256 | 4.193 | 10.650 | 57.24 |
| Diff2Lip | 255/256 | 7.588 | 7.286 | 59.17 |
| LatentSync 1.6 | 250/256 | 7.541 | 7.732 | 46.40 |
| X-Dub | 251/256 | 6.254 | 9.050 | 39.09 |
| Same Audio | Cross Audio | |||||||||
| Method | Params | NFE | FPS | Sync-C | Sync-D | SSIM | PSNR | Sync-C | Sync-D | FVD |
| Feed-forward and one-step editors | ||||||||||
| Wav2Lip [ 31 ] | 36M | 1 | 5.2 | 8.362 | 6.779 | 0.9297 | 31.34 | 7.800 | 7.173 | 71.73 |
| VideoReTalking [ 5 ] | 484M | 1 | 1.3 | 7.697 | 6.971 | 0.9087 | 28.95 | 7.124 | 7.394 | 100.75 |
| MuseTalk 1.5 [ 55 ] | 971M | 1 | 1.1 | 5.698 | 8.464 | 0.9317 | 32.19 | 4.305 | 9.673 | 54.59 |
| Bidirectional diffusion editors (20–50 NFE) | ||||||||||
| Same Audio | Cross Audio | |||||||||
| Method | Params | NFE | FPS | Sync-C | Sync-D | SSIM | PSNR | Sync-C | Sync-D | FVD |
| Feed-forward and one-step editors | ||||||||||
| Wav2Lip † [ 31 ] | 36M | 1 | 5.2 | 7.209 | 7.420 | 0.9218 | 29.83 | 6.750 | 7.932 | 111.19 |
| VideoReTalking [ 5 ] | 484M | 1 | 1.3 | 6.787 | 7.552 | 0.8948 | 28.00 | 6.365 | 7.868 | 84.51 |
| MuseTalk 1.5 [ 55 ] | 971M | 1 | 1.1 | 4.968 | 9.106 | 0.9305 | 29.60 | 4.371 | 9.625 | 36.09 |
| Bidirectional diffusion editors (20–50 NFE) | ||||||||||
| Method | Success | Sync-C [95% CI] | Sync-D [95% CI] | FVD |
|---|---|---|---|---|
| Wav2Lip | 499/500 | 6.686 [6.521,6.848] | 7.964 [7.856,8.075] | 112.21 |
| VideoReTalking | 402/500 | 6.361 [6.215,6.509] | 7.859 [7.772,7.953] | 87.80 |
| MuseTalk 1.5 | 498/500 | 4.360 [4.195,4.535] | 9.624 [9.495,9.754] | 36.67 |
| Diff2Lip | 403/500 | 6.904 [6.745,7.067] | 7.021 [6.925,7.116] | 70.21 |
| LatentSync 1.6 | 499/500 | 6.774 [6.611,6.943] | 7.553 [7.450,7.656] | 57.24 |
| X-Dub | 499/500 | 5.095 [4.916,5.275] | 9.391 [9.256,9.523] | 35.21 |
| Method | NFE | HDTF-30 | VoxCeleb2-30 | TalkVid-Bench |
|---|---|---|---|---|
| Wav2Lip [ 31 ] | 1 | 8.782 / 129.51 | 7.800 / 71.73 | 6.750 / 111.19 |
| VideoReTalking [ 5 ] | 1 | 8.312 / 105.72 | 7.124 / 100.75 | 6.365 / 84.51 |
| Diff2Lip [ 28 ] | 25 | 8.680 / 99.70 | 7.533 / 67.15 | 6.944 / 73.37 |
| MuseTalk 1.5 [ 55 ] | 1 | 7.116 / 40.10 | 4.305 / 54.59 | 4.371 / 36.09 |
| X-Dub [ 14 ] | 50 | 5.027 / 66.05 | 5.375 / 97.45 | 5.087 / 32.63 |
| LatentSync 1.6 [ 21 ] | 20 | 8.160 / 28.05 | 6.239 / 28.89 | 6.860 / 59.27 |
| Method | Backbone | NFE | Br. | Sync-C | Sync-D | FVD |
|---|---|---|---|---|---|---|
| Full-context parent, EMA@200 | 1.3B | 48 | 2 | 8.816 | 6.950 | 43.86 |
| LatentSync 1.6 [ 21 , 1 ] | – | 20 | 1 | 7.629 | 7.589 | 69.93 |
| Lip Forcing [ 6 ] | 14B | 2 | 1 | 3.973 | 11.164 | 34.70 |
| Causal two-step VidDub | 1.3B | 2 | 1 | 8.772 | 7.087 | 49.04 |
| Opponent | Lip sync | Mouth realism | Identity | Stability | Overall |
|---|---|---|---|---|---|
| Teacher | 10/10/130 | 9/12/129 | 4/8/138 | 4/14/132 | 12/20/118 |
| LatentSync | 123/2/25 | 122/6/22 | 75/3/72 | 123/2/25 | 143/2/5 |
| Lip Forcing | 142/0/8 | 90/15/45 | 61/2/87 | 87/2/61 | 146/1/3 |
Video Diffusion Models
Video diffusion models generate or edit temporally coherent video by iteratively denoising latent sequences conditioned on text, images, motion, or other signals.
See all 783 papers