Editing a local segment of a talking head video without reshooting the scene remains challenging and underexplored. Local edits, such as insertion, deletion, or substitution, require synthesizing a replacement segment while leaving the rest of the video unchanged. This involves variable-duration local rewriting while preserving identity, unedited regions, and boundary continuity, which standard speech-driven generation and lip synchronization methods do not directly address. Moreover, direct supervision is impractical, as it would require paired videos of the same person and scene that differ only in a local spoken segment. We instead formulate talking head video editing as facial motion infilling, a self-supervised pretext task that recovers masked facial motion from speech and surrounding motion context using ordinary video--speech pairs. Our key insight is that local video editing can be simulated during training by masking a motion span and reconstructing it from speech and visible motion context. Building on this formulation, we introduce FacEDiT, a mask-controlled talking head model with a local temporal attention bias and temporal smoothness regularization to improve lip--speech alignment and transition continuity. We also introduce FacEDiTBench, the first benchmark for talking head video editing, covering diverse edit types and lengths with dedicated evaluation metrics. Extensive experiments show that FacEDiT produces accurate, speech-aligned edits with strong identity preservation and seamless boundary transitions. Beyond editing, the same facial motion infilling model extends to portrait animation and lip synchronization by simply changing the mask pattern, establishing FacEDiT as a unified framework for talking head video editing and generation.
Figures & tables
Figure 1: Overview of FacEDiT. FacEDiT performs mask-controlled facial motion infilling for local talking head video editing, regenerating only the edited span from speech and visible motion context. By changing the mask pattern, the same model supports portrait animation and lip synchronization.
Method / Setting
Duration-changing local edit
Unedited preservation
Boundary continuity
Portrait animation
Lip sync
Portrait animation Zhang et al. (2023) ; Wei et al. (2024) ; Xu et al. (2024) ; Cui et al. (2025a) ; Bigata et al. (2025) ; Wang et al. (2024) ; Chen et al. (2025c) ; Meng et al. (2025) ; Cui et al. (2025b)
No
No
No
Yes
Partial
Lip synchronization Prajwal et al. (2019) ; Prajwal et al. (2020) ; Zhou et al. (2021) ; Mukhopadhyay et al. (2024) ; Li et al. (2024) ; Peng et al. (2025) ; Ma et al. (2025)
Partial
Partial
Partial
No
Yes
FacEDiT
Yes
Yes
Yes
Yes
Yes
Table 1: Comparison of talking head editing and synthesis settings. Duration-changing local edit denotes modifying a local segment in an existing video while allowing the edited segment length to change, such as insertion or deletion. Unedited preservation indicates whether the method is designed to keep regions outside the edit unchanged. Boundary continuity refers to modeling smooth transitions between edited and unedited regions. Portrait animation and lip synchronization denote support for full-sequence speech-driven generation and mouth-motion retargeting, respectively.
Figure 2: Overview of FacEDiT . (a) During training, FacEDiT learns facial motion infilling from paired speech and visible motion context using flow matching ( LCFM ) and temporal smoothness regularization ( LTS ). (b) During inference, the edit span or task setting defines the motion mask, and speech guides the masked motion prediction. (c) The predicted motion drives the source appearance features through the motion decoder to render the final video.
Table 2: Statistics and example of FacEDiTBench. (a) FacEDiTBench includes three types of edits with span lengths ranging from short to long. (b) An example of transcript editing shows a phrase insertion while preserving the surrounding content.
Table 3: Comparison on talking head video editing. We adapt speech-driven portrait animation and lip synchronization models as editing baselines (source task → editing). † uses boundary frames from the original video as keyframes for smooth interpolation while editing, and ∗ conditions on the original head pose and predicts only facial expression.
Figure 3: Qualitative comparison. The insertion edit adds the new word “darker” ( /dArk/ ) to the original sentence. The green arrow indicates the edited region, while the others are original frames. Our model achieves accurate lip-sync, preserves identity, and transitions smoothly to the original frames. In contrast, V-Express fails to preserve identity; EchoMimicV2 shows a front-facing bias; SadTalker and lip synchronization models produce mumbled motion; and most methods introduce discontinuities at boundary transitions ( e.g ., between // and /s/). Full results are in Appendix B .
\RowStyle Edit duration
N
Avg. dur.
LSE-D ↓
LSE-C ↑
P continuity↓
M continuity↓
IDSIM ↑
10–20 s
18
15.89 s
7.419
6.061
1.259
0.360
0.9474
20–25 s
13
21.64 s
7.740
5.975
1.204
0.312
0.9469
25–30 s
9
26.93 s
7.582
5.583
1.772
0.388
0.9290
≥ 30 s
12
33.13 s
7.673
5.178
1.501
0.549
0.9200
Table 4: Robustness to edit duration. We evaluate 52 edits ranging from 10 to approximately 40 seconds, grouped by edit duration, where N denotes the number of edits in each group.
\RowStyle
Fusion
Speech Enc.
Biased Attn.
LTS
LSE-D ↓
LSE-C ↑
IDSIM ↑
FVD ↓
P continuity↓
M continuity↓
(a)
Early
WavLM
✓
✗
7.292
6.581
0.965
63.229
2.447
0.825
(b)
CA
WavLM
✓
✗
7.179
6.673
0.966
61.955
2.466
0.829
(c)
CA
Wav2Vec
✓
✗
7.223
6.639
0.965
62.171
2.448
0.821
(d)
CA
WavLM
✗
✗
7.301
6.655
0.955
65.559
2.816
1.097
(e)
CA
WavLM
✓
✓
7.135
6.670
0.966
61.930
2.420
0.800
Table 5: Ablation results on FacEDiTBench. We evaluate the impact of different design choices by comparing various configurations. “Early” and “CA” indicate the speech feature integration strategy (early fusion vs. cross-attention). (e) is the final model used in all experiments.
\RowStyle Method
LSE-D ↓
LSE-C ↑
IDSIM ↑
FVD ↓
LPIPS ↓
V-Express Wang et al. (2024)
7.470
7.842
0.906
65.158
0.341
AniPortrait Wei et al. (2024)
11.168
2.866
0.914
102.414
0.310
EchoMimic Chen et al. (2025c)
9.085
5.555
0.929
89.129
0.320
EchoMimicV2 Meng et al. (2025)
9.966
4.051
0.890
131.603
0.322
SadTalker Zhang et al. (2023)
7.557
7.421
0.848
81.572
0.309
Hallo Xu et al. (2024)
7.824
7.080
0.925
42.825
0.283
Table 6: Results on cross-task generalization. Our unified model generalizes to speech-driven portrait animation and lip synchronization, achieving favorable performance.
3 Task
Ours
SadTalker Zhang et al. (2023)
Hallo Xu et al. (2024)
Hallo2 Cui et al. (2025a)
Hallo3 Cui et al. (2025b)
KeyFace Bigata et al. (2025)
Editing
Overall ↓
1.57 ± 1.40
3.93 ± 1.38
3.58 ± 1.29
3.90 ± 1.40
3.32 ± 1.51
4.70 ± 1.51
Boundary ↓
1.39 ± 1.25
4.19 ± 1.29
3.35 ± 1.07
3.90 ± 1.34
3.23 ± 1.49
4.93 ± 1.39
Portrait animation
Overall ↓
2.32 ± 1.60
4.83 ± 1.50
3.04 ± 1.35
2.91 ± 1.41
3.68 ± 1.59
4.23 ± 1.47
Table 7: Human study comparison. We report the average rank ( ± standard deviation) assigned by participants (lower is better) for talking head video editing and the corresponding generation task. Boundary continuity is evaluated for editing only. Ranks are computed independently within each comparison pool and are therefore not directly comparable across panels.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Edit type
Original transcript
Edited transcript
Substitution (5 words)
However, if that naturally suits you , then do
However, if your heart tells you to , then do
Insertion (6 words)
And I forget the story behind it, but it is really cool to see
And I forget the story behind it, about how the festival first began , but it is really cool to see
Deletion (6 words)
So we are not looking at a murder, we are looking at negligence
So we are not looking at negligence
Appendix
Table S1: Examples of edit types in FacEDiTBench. Each example shows the edit type and the number of edited words, along with the corresponding original and edited transcripts. The edited transcripts are used to generate edited speech for the dataset using VoiceCraft [ 29 ] .
Figure S1: Evaluation setup for boundary continuity. Frames 1 to (p−1) and (q+1) to the last frame are treated as unedited, while frames p through q constitute the edited segment. Based on these segments, we compute the boundary continuity metric at the transition frames. Our model produces smooth transitions at these boundaries, whereas the comparison model (Hallo2) exhibits noticeable discontinuities, particularly between frames q and q+1 .
Method
LSE-D ( ↓ )
LSE-C ( ↑ )
IDSIM ( ↑ )
FVD ( ↓ )
LPIPS ( ↓ )
V-Express [ 20 ]
7.718
7.264
0.756
196.974
0.426
AniPortrait [ 19 ]
11.421
2.651
0.847
151.998
0.330
EchoMimic [ 6 ]
9.511
4.890
0.832
143.894
0.350
EchoMimicV2 [ 10 ]
10.136
3.819
0.817
124.219
0.347
SadTalker [ 4 ]
7.578
7.119
0.779
132.646
0.324
Hallo [ 5 ]
8.210
6.591
0.842
116.429
0.315
Appendix
Table S2: Comparison on talking head video editing. Note that the compared models are designed for speech-driven portrait animation and thus serve only as baselines, as they are not directly applicable to editing. We highlight the best results in bold and underline the second best.
Method
Inference (seconds)
Ours
73.787
V-Express [ 20 ]
639.694
AniPortrait [ 19 ]
280.817
EchoMimic [ 6 ]
808.065
EchoMimicV2 [ 10 ]
512.349
Appendix
Table S3: Comparison of inference time. We measure the inference time of each model when generating a talking head video from the same 15 seconds of speech.
Figure S2: Qualitative comparison. Our method produces accurate lip synchronization with the edited speech while preserving identity and maintaining smooth transitions between edited and unedited frames compared to baseline models.
Variant
Velocity ↓
Acceleration ↓
w/o LTS
0.001983
0.001924
w/ LTS
0.001872
0.001671
Relative reduction
5.6%
13.2%
Appendix
Table S4: Motion smoothness analysis. RMS velocity and acceleration of predicted facial-motion trajectories with and without the temporal smoothness loss.
Figure S3: Qualitative comparison of frame resampling methods. When downsampling a video ( e.g ., from 124 frames to 7), an effective resampling strategy is required. Sparse frame selection introduces aliasing, while interpolation between only the first and last frames produces noticeable blur. In contrast, our resampling method produces smoother transitions with significantly fewer artifacts.
Method
LSE-D ( ↓ )
LSE-C ( ↑ )
IDSIM ( ↑ )
P continuity ( ↓ )
M continuity ( ↓ )
Sparse selection
7.769
5.137
0.968
3.609
1.587
Interpolation
7.716
5.215
0.954
3.139
0.829
Our resampling
7.336
5.588
0.960
3.118
0.825
Appendix
Table S5: Quantitative comparison of resampling methods. Using our frame resampling approach during downsampling improves lip-sync accuracy and boundary continuity compared to sparse frame selection and interpolation-based methods.
Figure S4: Instruction and example for the human study. We provide a sample template used in the talking head video editing evaluation.