Video Editing

Follow this topic Ask about this topic

AI video editing studies how to change footage—such as removing or inserting objects, altering motion or appearance, and assembling clips—while preserving the content that should remain. Common approaches adapt video diffusion and flow-matching models, using text, masks, reference images, keyframes, tracks, or 3D geometry to localize and guide edits; agentic systems also plan edits and operate professional timelines. Work increasingly targets streaming and long-video editing, alongside joint audio-visual changes and physics-aware scene effects. A central challenge is balancing strong instruction following with source fidelity, temporal and cross-shot coherence, and physically plausible results, while benchmarks still expose failures on complex instructions and secondary effects.

Latest papers

178 papers23 papers in the last four weeks, up 156% on the four weeks before. 0.1% of all new papers.

Mar 23, 2026

CLEAR: Context-Aware Learning with End-to-End Mask-Free Inference for Adaptive Video Subtitle Removal

Video subtitle removal aims to distinguish text overlays from background content while preserving temporal coherence. Existing diffusion-based methods necessitate explicit mask sequences during both training and inference phases, which restricts their practical deployment. In this paper, we present CLEAR (Context-aware Learning for End-to-end Adaptive Video Subtitle Removal), a mask-free framework that achieves truly end-to-end inference through context-aware adaptive learning. Our two-stage design decouples prior extraction from generative refinement: Stage I learns disentangled subtitle representations via self-supervised orthogonality constraints on dual encoders, while Stage II employs LoRA-based adaptation with generation feedback for dynamic context adjustment. Notably, our method only requires 0.77% of the parameters of the base diffusion model for training. On Chinese subtitle benchmarks, CLEAR outperforms mask-dependent baselines by + 6.77dB PSNR and -74.7% VFID, while demonstrating superior zero-shot generalization across six languages (English, Korean, French, Japanese, Russian, German), a performance enabled by our generation-driven feedback mechanism that ensures robust subtitle removal without ground-truth masks during inference.

Qingdong He, Chaoyi Wang, Peng Tang +2

University of Electronic Science and Technology of China · University of Chinese Academy of Sciences · Technical University of Munich +2

Video Editing Video Diffusion Models

Mar 20, 2026

PerformRecast: Expression and Head Pose Disentanglement for Portrait Video Editing

This paper primarily investigates the task of expression-only portrait video performance editing based on a driving video, which plays a crucial role in animation and film industries. Most existing research mainly focuses on portrait animation, which aims to animate a static portrait image according to the facial motion from the driving video. As a consequence, it remains challenging for them to disentangle the facial expression from head pose rotation and thus lack the ability to edit facial expression independently. In this paper, we propose PerformRecast, a versatile expression-only video editing method which is dedicated to recast the performance in existing film and animation. The key insight of our method comes from the characteristics of 3D Morphable Face Model (3DMM), which models the face identity, facial expression and head pose of 3D face mesh with separate parameters. Therefore, we improve the keypoints transformation formula in previous methods to make it more consistent with 3DMM model, which achieves a better disentanglement and provides users with much more fine-grained control. Furthermore, to avoid the misalignment around the boundary of face in generated results, we decouple the facial and non-facial regions of input portrait images and pre-train a teacher model to provide separate supervision for them. Extensive experiments show that our method produces high-quality results which are more faithful to the driving video, outperforming existing methods in both controllability and efficiency. Our code, data and trained models are available at https://youku-aigc.github.io/PerformRecast.

Jiadong Liang, Bojun Xiong, Jie Tian +4

HUJING Digital Media & Entertainment Group

Video Editing Representation Disentanglement

Mar 19, 2026

SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editing

Current instruction-guided video editing models struggle to simultaneously balance precise semantic modifications with faithful motion preservation. While existing approaches rely on injecting explicit external priors (e.g., VLM features or structural conditions) to mitigate these issues, this reliance severely bottlenecks model robustness and generalization. To overcome this limitation, we present SAMA (factorized Semantic Anchoring and Motion Alignment), a framework that factorizes video editing into semantic anchoring and motion modeling. First, we introduce Semantic Anchoring, which establishes a reliable visual anchor by jointly predicting semantic tokens and video latents at sparse anchor frames, enabling purely instruction-aware structural planning. Second, Motion Alignment pre-trains the same backbone on motion-centric video restoration pretext tasks (cube inpainting, speed perturbation, and tube shuffle), enabling the model to internalize temporal dynamics directly from raw videos. SAMA is optimized with a two-stage pipeline: a factorized pre-training stage that learns inherent semantic-motion representations without paired video-instruction editing data, followed by supervised fine-tuning on paired editing data. Remarkably, the factorized pre-training alone already yields strong zero-shot video editing ability, validating the proposed factorization. SAMA achieves state-of-the-art performance among open-source models and is competitive with leading commercial systems (e.g., Kling-Omni). Code, models, and datasets will be released.

Xinyao Zhang, Wenkai Dong, Yuxin Song +10

Baidu · Tsinghua University · City University of Hong Kong +1

Video Editing Instruction-Based Video Editing Video Representation Learning

Mar 19, 2026

EffectErase: Joint Video Object Removal and Insertion for High-Quality Effect Erasing

Video object removal aims to eliminate dynamic target objects and their visual effects, such as deformation, shadows, and reflections, while restoring seamless backgrounds. Recent diffusion-based video inpainting and object removal methods can remove the objects but often struggle to erase these effects and to synthesize coherent backgrounds. Beyond method limitations, progress is further hampered by the lack of a comprehensive dataset that systematically captures common object effects across varied environments for training and evaluation. To address this, we introduce VOR (Video Object Removal), a large-scale dataset that provides diverse paired videos, each consisting of one video where the target object is present with its effects and a counterpart where the object and effects are absent, with corresponding object masks. VOR contains 60K high-quality video pairs from captured and synthetic sources, covers five effects types, and spans a wide range of object categories as well as complex, dynamic multi-object scenes. Building on VOR, we propose EffectErase, an effect-aware video object removal method that treats video object insertion as the inverse auxiliary task within a reciprocal learning scheme. The model includes task-aware region guidance that focuses learning on affected areas and enables flexible task switching. Then, an insertion-removal consistency objective that encourages complementary behaviors and shared localization of effect regions and structural cues. Trained on VOR, EffectErase achieves superior performance in extensive experiments, delivering high-quality video object effect erasing across diverse scenarios.

Yang Fu, Yike Zheng, Ziyun Dai +1

Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University, China

Video Object Insertion Object Removal Video Editing Video Restoration

Mar 18, 2026

Versatile Editing of Video Content, Actions, and Dynamics without Training

Controlled video generation has seen drastic improvements in recent years. However, editing actions and dynamic events, or inserting contents that should affect the behaviors of other objects in real-world videos, remains a major challenge. Existing trained models struggle with complex edits, likely due to the difficulty of collecting relevant training data. Similarly, existing training-free methods are inherently restricted to structure- and motion-preserving edits and do not support modification of motion or interactions. Here, we introduce DynaEdit, a training-free editing method that unlocks versatile video editing capabilities with pretrained text-to-video flow models. Our method relies on the recently introduced inversion-free approach, which does not intervene in the model internals, and is thus model-agnostic. We show that naively attempting to adapt this approach to general unconstrained editing results in severe low-frequency misalignment and high-frequency jitter. We explain the sources for these phenomena and introduce novel mechanisms for overcoming them. Through extensive experiments, we show that DynaEdit achieves state-of-the-art results on complex text-based video editing tasks, including modifying actions, inserting objects that interact with the scene, and introducing global effects.

Vladimir Kulikov, Roni Paiss, Andrey Voynov +3

Google DeepMind · Technion – Israel Institute of Technology · The Weizmann Institute of Science

Video Editing Instruction-Based Video Editing

Mar 16, 2026

ViFeEdit: A Video-Free Tuner of Your Video Diffusion Transformer

Diffusion Transformers (DiTs) have demonstrated remarkable scalability and quality in image and video generation, prompting growing interest in extending them to controllable generation and editing tasks. However, compared to the image counterparts, progress in video control and editing remains limited, mainly due to the scarcity of paired video data and the high computational cost of training video diffusion models. To address this issue, in this paper, we propose a video-free tuning framework termed ViFeEdit for video diffusion transformers. Without requiring any forms of video training data, ViFeEdit achieves versatile video generation and editing, adapted solely with 2D images. At the core of our approach is an architectural reparameterization that decouples spatial independence from the full 3D attention in modern video diffusion transformers, which enables visually faithful editing while maintaining temporal consistency with only minimal additional parameters. Moreover, this design operates in a dual-path pipeline with separate timestep embeddings for noise scheduling, exhibiting strong adaptability to diverse conditioning signals. Extensive experiments demonstrate that our method delivers promising results of controllable video generation and editing with only minimal training on 2D image data. Codes are available https://github.com/Lexie-YU/ViFeEdit.

Ruonan Yu, Zhenxiong Tan, Zigeng Chen +2

National University of Singapore · Shanghai Jiao Tong University

Diffusion-Based Video Editing Video Editing Video Diffusion Models Controllable Video Generation Diffusion Transformer Temporal Consistency in Video Generation

Mar 10, 2026

When to Lock Attention: Training-Free KV Control in Video Diffusion

Maintaining background consistency while enhancing foreground quality remains a core challenge in video editing. Injecting full-image information often leads to background artifacts, whereas rigid background locking severely constrains the model's capacity for foreground generation. To address this issue, we propose KV-Lock, a training-free framework tailored for DiT-based video diffusion models. Our core insight is that the hallucination metric (variance of denoising prediction) directly quantifies generation diversity, which is inherently linked to the classifier-free guidance (CFG) scale. Building upon this, KV-Lock leverages diffusion hallucination detection to dynamically schedule two key components: the fusion ratio between cached background key-values (KVs) and newly generated KVs, and the CFG scale. When hallucination risk is detected, KV-Lock strengthens background KV locking and simultaneously amplifies conditional guidance for foreground generation, thereby mitigating artifacts and improving generation fidelity. As a training-free, plug-and-play module, KV-Lock can be easily integrated into any pre-trained DiT-based models. Extensive experiments validate that our method outperforms existing approaches in improved foreground quality with high background fidelity across various video editing tasks.

Tianyi Zeng, Jincheng Gao, Tianyi Wang +8

Shanghai Jiao Tong University, Shanghai, China · Tongji University, Shanghai, China · The University of Texas at Austin, Austin, TX, USA +3

Video Diffusion Models Video Editing Attention Control in Diffusion Models Diffusion Model Guidance Diffusion Transformer

Mar 10, 2026

From Ideal to Real: Stable Video Object Removal under Imperfect Conditions

Removing objects from videos remains difficult in the presence of real-world imperfections such as shadows, abrupt motion, and defective masks. Existing diffusion-based video inpainting models often struggle to maintain temporal stability and visual consistency under these challenges. We propose Stable Video Object Removal (SVOR), a robust framework that achieves shadow-free, flicker-free, and mask-defect-tolerant removal through three key designs: (1) Mask Union for Stable Erasure (MUSE), a windowed union strategy applied during temporal mask downsampling to preserve all target regions observed within each window, effectively handling abrupt motion and reducing missed removals; (2) Denoising-Aware Segmentation (DA-Seg), a lightweight segmentation head on a decoupled side branch equipped with Denoising-Aware AdaLN and trained with mask degradation to provide an internal diffusion-aware localization prior without affecting content generation; and (3) Curriculum Two-Stage Training: where Stage I performs self-supervised pretraining on unpaired real-background videos with online random masks to learn realistic background and temporal priors, and Stage II refines on synthetic pairs using mask degradation and side-effect-weighted losses, jointly removing objects and their associated shadows/reflections while improving cross-domain robustness. Extensive experiments show that SVOR attains new state-of-the-art results across multiple datasets and degraded-mask benchmarks, advancing video object removal from ideal settings toward real-world applications. Project page: https://xiaomi-research.github.io/svor/.

Jiagao Hu, Yuxuan Chen, Fuhao Li +4

MiLM Plus, Xiaomi Inc.

Video Editing Video Diffusion Models Temporal Consistency Image Inpainting

Mar 10, 2026

SyncEdit: Rethinking Lip Synchronization as Editing with Audio-Driven Diffusion Models

Lip synchronization refers to the task of modifying facial lip movements such that they are temporally aligned with a given audio signal. It is a fundamental problem in audio-visual synthesis for generating realistic talking-head videos. Recent progress in diffusion-based generative models has led to remarkable advances in lip synchronization. Nevertheless, existing methods typically achieve audio-visual alignment by fine-tuning pre-trained diffusion models on large scale datasets, resulting in substantial computational costs and significant data requirements. Inspired by FlowEdit, we reformulate lip synchronization as a video editing problem. Building upon a pre-trained audio-driven diffusion model, our approach achieves lip synchronization in a training-free manner, without additional fine-tuning or paired data. In this paper, we present SyncEdit, a training-free framework designed for lip synchronization. We reformulate the editing paradigm by substituting the edit sequence in FlowEdit with the target sequence, yielding an unbiased estimation of the desired output. Moreover, we propose annealed noise alignment, which progressively alignes the sampled Gaussian noise with diffusion-model-estimated noise during iterative editing, producing a smooth and stable editing trajectory. Extensive experimental results validate the effectiveness and robustness of the proposed framework. Code is available at \href[]{https://github.com/l1346792580123/SyncEdit}{here}.

Lixiang Lin, Siyuan Jin, Jinshan Zhang

HiThink Research · University of Science and Technology of China · Zhejiang University

Lip Synchronization Video Editing Audio-Visual Synchronization Audio-Driven Facial Animation Diffusion-Based Video Editing

Mar 6, 2026

Place-it-R1: Unlocking Environment-aware Reasoning Potential of MLLM for Video Object Insertion

Video object insertion is fundamental to video editing, yet existing diffusion methods often produce visually plausible but physically inconsistent results. We present Place-it-R1, an end-to-end framework for physically plausible video object insertion driven by environment-aware MLLM reasoning. Rather than treating reasoning as a generic text prompt, Place-it-R1 uses the MLLM to analyze the target environment, infer object-scene interactions, and determine where an insertion is physically valid. The resulting reasoning is translated into two complementary forms of guidance for video diffusion: semantic guidance that describes the intended physical interaction and spatial guidance that provides a valid insertion region in each frame. To further align generation with local physical realism, we introduce Spatial Direct Preference Optimization, which leverages an MLLM to rank generated candidates, and introduces a region-aware preference objective that explicitly localizes physical-violation penalties to the inserted-object region. Place-it-R1 further offers flexible and standard modes to trade off environment adaptation and scene preservation. Extensive experiments show that Place-it-R1 produces more physically coherent and visually natural insertions than state-of-the-art methods and achieves competitive results against commercial systems.

Bohai Gu, Taiyi Wu, Dazhao Du +5

The Hong Kong University of Science and Technology · AI Technology Center, Tencent Video, Tencent · Peking University

Video Object Insertion Video Editing Physical Consistency in Video Generation Video Diffusion Models Direct Preference Optimization

Mar 3, 2026

NOVA: Sparse Control, Dense Synthesis for Pair-Free Video Editing

Recent video editing models have achieved impressive results, but most still require large-scale paired datasets. Collecting such naturally aligned pairs at scale remains highly challenging and constitutes a critical bottleneck, especially for local video editing data. Existing workarounds transfer image editing to video through global motion control for pair-free video editing, but such designs struggle with background and temporal consistency. In this paper, we propose NOVA: Sparse Control & Dense Synthesis, a new framework for unpaired video editing. Specifically, the sparse branch provides semantic guidance through user-edited keyframes distributed across the video, and the dense branch continuously incorporates motion and texture information from the original video to maintain high fidelity and coherence. Moreover, we introduce a degradation-simulation training strategy that enables the model to learn motion reconstruction and temporal consistency by training on artificially degraded videos, thus eliminating the need for paired data. Our extensive experiments demonstrate that NOVA outperforms existing approaches in edit fidelity, motion preservation, and temporal coherence.

Tianlin Pan, Jiayi Dai, Chenpu Yuan +7

Nanjing University · University of Chinese Academy of Sciences · The University of Hong Kong +1

Mar 2, 2026

Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidance

Instruction-based video editing has witnessed rapid progress, yet current methods often struggle with precise visual control, as natural language is inherently limited in describing complex visual nuances. Although reference-guided editing offers a robust solution, its potential is currently bottlenecked by the scarcity of high-quality paired training data. To bridge this gap, we introduce a scalable data generation pipeline that transforms existing video editing pairs into high-fidelity training quadruplets, leveraging image generative models to create synthesized reference scaffolds. Using this pipeline, we construct RefVIE, a large-scale dataset tailored for instruction-reference-following tasks, and establish RefVIE-Bench for comprehensive evaluation. Furthermore, we propose a unified editing architecture, Kiwi-Edit, that synergizes learnable queries and latent visual features for reference semantic guidance. Our model achieves significant gains in instruction following and reference fidelity via a progressive multi-stage training curriculum. Extensive experiments demonstrate that our data and architecture establish a new state-of-the-art in controllable video editing. All datasets, models, and code is released at https://github.com/showlab/Kiwi-Edit.

Yiqi Lin, Guoqiang Liang, Ziyun Zeng +3

Show Lab, National University of Singapore

Video Editing Instruction-Based Video Editing Video Editing Benchmarks Controllable Video Generation Synthetic Data Generation

Mar 1, 2026

FREE-Edit: Using Editing-aware Injection in Rectified Flow Models for Zero-shot Image-Driven Video Editing

Image-driven video editing aims to propagate edit contents from the modified first frame to the remaining frames. Existing methods usually invert the source video to noise using a pre-trained image-to-video (I2V) model and then guide the sampling process using the edited first frame. Generally, a popular choice for maintaining motion and layout from the source video is intervening in the denoising process by injecting attention during reconstruction. However, such injection often leads to unsatisfactory results, where excessive injection leads to conflicting semantics with the source video while insufficient injection brings limited source representation. Recognizing this, we propose an Editing-awaRE (REE) injection method to modulate the injection intensity of each token. Specifically, we first compute the pixel difference between the source and edited first frame to form a corresponding editing mask. Next, we track the editing area throughout the entire video by using optical flow to warp the first-frame mask. Then, editing-aware feature injection intensity for each token is generated accordingly, where injection is not conducted in editing areas. Building upon REE injection, we further propose a zero-shot image-driven video editing framework with recent-emerging rectified-Flow models, dubbed FREE-Edit. Without fine-tuning or training, our FREE-Edit demonstrates effectiveness in various image-driven video editing scenarios, showing its capability to produce higher-quality outputs compared with existing techniques. Project page: https://free-edit.github.io/page/.

Maomao Li, Yunfei Liu, Yu Li

The University of Hong Kong · International Digital Economy Academy (IDEA)

Feb 27, 2026

CamDirector: Towards Long-Term Coherent Video Trajectory Editing

Video (camera) trajectory editing aims to synthesize new videos that follow user-defined camera paths while preserving scene content and plausibly inpainting previously unseen regions, upgrading amateur footage into professionally styled videos. Existing VTE methods struggle with precise camera control and long-range consistency because they either inject target poses through a limited-capacity embedding or rely on single-frame warping with only implicit cross-frame aggregation in video diffusion models. To address these issues, we introduce a new VTE framework that 1) explicitly aggregates information across the entire source video via a hybrid warping scheme. Specifically, static regions are progressively fused into a world cache then rendered to target camera poses, while dynamic regions are directly warped; their fusion yields globally consistent coarse frames that guide refinement. 2) processes video segments jointly with their history via a history-guided autoregressive diffusion model, while the world cache is incrementally updated to reinforce already inpainted content, enabling long-term temporal coherence. Finally, we present iPhone-PTZ, a new VTE benchmark with diverse camera motions and large trajectory variations, and achieve state-of-the-art performance with fewer parameters.

Zhihao Shi, Kejia Yin, Weilin Wan +5

McMaster University · University of Toronto · The University of Hong Kong +3

Video Editing Camera-Controlled Video Generation Trajectory-Guided Video Generation Diffusion-Based Video Editing Temporal Consistency in Video Generation Video Diffusion Models

Feb 25, 2026

SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model

SkyReels V4 is a unified multi modal video foundation model for joint video audio generation, inpainting, and editing. The model adopts a dual stream Multimodal Diffusion Transformer (MMDiT) architecture, where one branch synthesizes video and the other generates temporally aligned audio, while sharing a powerful text encoder based on the Multimodal Large Language Models (MLLM). SkyReels V4 accepts rich multi modal instructions, including text, images, video clips, masks, and audio references. By combining the MLLMs multi modal instruction following capability with in context learning in the video branch MMDiT, the model can inject fine grained visual guidance under complex conditioning, while the audio branch MMDiT simultaneously leverages audio references to guide sound generation. On the video side, we adopt a channel concatenation formulation that unifies a wide range of inpainting style tasks, such as image to video, video extension, and video editing under a single interface, and naturally extends to vision referenced inpainting and editing via multi modal prompts. SkyReels V4 supports up to 1080p resolution, 32 FPS, and 15 second duration, enabling high fidelity, multi shot, cinema level video generation with synchronized audio. To make such high resolution, long-duration generation computationally feasible, we introduce an efficiency strategy: Joint generation of low resolution full sequences and high-resolution keyframes, followed by dedicated super-resolution and frame interpolation models. To our knowledge, SkyReels V4 is the first video foundation model that simultaneously supports multi-modal input, joint video audio generation, and a unified treatment of generation, inpainting, and editing, while maintaining strong efficiency and quality at cinematic resolutions and durations.

Guibin Chen, Dixuan Lin, Jiangping Yang +48

Multimodal Video Generation Audio-Video Generation Multimodal Diffusion Models Video Editing Multi-Shot Video Generation Long Video Generation+1

Feb 24, 2026

PropFly: Learning to Propagate via On-the-Fly Supervision from Pre-trained Video Diffusion Models

Propagation-based video editing enables precise user control by propagating a single edited frame into following frames while maintaining the original context such as motion and structures. However, training such models requires large-scale, paired (source and edited) video datasets, which are costly and complex to acquire. Hence, we propose the PropFly, a training pipeline for Propagation-based video editing, relying on on-the-Fly supervision from pre-trained video diffusion models (VDMs) instead of requiring off-the-shelf or precomputed paired video editing datasets. Specifically, our PropFly leverages one-step clean latent estimations from intermediate noised latents with varying Classifier-Free Guidance (CFG) scales to synthesize diverse pairs of 'source' (low-CFG) and 'edited' (high-CFG) latents on-the-fly. The source latent serves as structural information of the video, while the edited latent provides the target transformation for learning propagation. Our pipeline enables an additional adapter attached to the pre-trained VDM to learn to propagate edits via Guidance-Modulated Flow Matching (GMFM) loss, which guides the model to replicate the target transformation. Our on-the-fly supervision ensures the model to learn temporally consistent and dynamic transformations. Extensive experiments demonstrate that our PropFly significantly outperforms the state-of-the-art methods on various video editing tasks, producing high-quality editing results.

Wonyong Seo, Jaeho Moon, Jaehyup Lee +2

KAIST · Kyungpook National University · Adobe Research

Video Editing Diffusion-Based Video Editing Temporal Consistency in Video Generation

Feb 16, 2026

EditCtrl: Disentangled Local and Global Control for Real-Time Generative Video Editing

High-fidelity generative video editing has seen significant quality improvements by leveraging pre-trained video foundation models. However, their computational cost is a major bottleneck, as they are often designed to inefficiently process the full video context regardless of the inpainting mask's size, even for sparse, localized edits. In this paper, we introduce EditCtrl, an efficient video inpainting control framework that focuses computation only where it is needed. Our approach features a novel local video context module that operates solely on masked tokens, yielding a computational cost proportional to the edit size. This local-first generation is then guided by a lightweight temporal global context embedder that ensures video-wide context consistency with minimal overhead. Not only is EditCtrl 10 times more compute efficient than state-of-the-art generative editing methods, it even improves editing quality compared to methods designed with full-attention. Finally, we showcase how EditCtrl unlocks new capabilities, including multi-region editing with text prompts and autoregressive content propagation.

Yehonathan Litman, Shikun Liu, Dario Seyb +5

Meta Reality Labs · Carnegie Mellon University · Meta AI

Video Editing Instruction-Based Video Editing Image Inpainting

Feb 9, 2026

PISCO: Precise Video Instance Insertion with Sparse Control

The landscape of AI video generation is undergoing a pivotal shift: moving beyond general generation - which relies on exhaustive prompt-engineering and "cherry-picking" - towards fine-grained, controllable generation and high-fidelity post-processing. In professional AI-assisted filmmaking, it is crucial to perform precise, targeted modifications. A cornerstone of this transition is video instance insertion, which requires inserting a specific instance into existing footage while maintaining scene integrity. Unlike traditional video editing, this task demands several requirements: precise spatial-temporal placement, physically consistent scene interaction, and the faithful preservation of original dynamics - all achieved under minimal user effort. In this paper, we propose PISCO, a video diffusion model for precise video instance insertion with arbitrary sparse keyframe control. PISCO allows users to specify a single keyframe, start-and-end keyframes, or sparse keyframes at arbitrary timestamps, and automatically propagates object appearance, motion, and interaction. To address the severe distribution shift induced by sparse conditioning in pretrained video diffusion models, we introduce Variable-Information Guidance for robust conditioning and Distribution-Preserving Temporal Masking to stabilize temporal generation, together with geometry-aware conditioning for realistic scene adaptation. We further construct PISCO-Bench, a benchmark with verified instance annotations and paired clean background videos, and evaluate performance using both reference-based and reference-free perceptual metrics. Experiments demonstrate that PISCO consistently outperforms strong inpainting and video editing baselines under sparse control, and exhibits clear, monotonic performance improvements as additional control signals are provided. Project page: xiangbogaobarry.github.io/PISCO.

Xiangbo Gao, Renjie Li, Xinghao Chen +4

Texas A&M University · KAIST · Stanford University +1

Physical Consistency in Video Generation Video Diffusion Models Video Editing Controllable Video Generation Video Object Insertion

Jan 17, 2026

Learning Stochastic Bridges for Video Object Removal via Video-to-Video Translation

Existing video object removal methods predominantly rely on diffusion models following a noise-to-data paradigm, where generation starts from uninformative Gaussian noise. This approach discards the rich structural and contextual priors present in the original input video. Consequently, such methods often lack sufficient guidance, leading to incomplete object erasure or the synthesis of implausible content that conflicts with the scene's physical logic. In this paper, we reformulate video object removal as a video-to-video translation task via a stochastic bridge model. Unlike noise-initialized methods, our framework establishes a direct stochastic path from the source video (with objects) to the target video (objects removed). This bridge formulation effectively leverages the input video as a strong structural prior, guiding the model to perform precise removal while ensuring that the filled regions are logically consistent with the surrounding environment. To address the trade-off where strong bridge priors hinder the removal of large objects, we propose a novel adaptive mask modulation strategy. This mechanism dynamically modulates input embeddings based on mask characteristics, balancing background fidelity with generative flexibility. Extensive experiments demonstrate that our approach significantly outperforms existing methods in both visual quality and temporal consistency. The project page is https://bridgeremoval.github.io/.

Zijie Lou, Xiangwei Feng, Jiaxin Wang +7

MT Lab, Meitu Inc., Beijing 100083, China · Beijing Jiaotong University, Beijing 100044, China

Video Editing Video-to-Video Translation Object Removal

Dec 19, 2025

InsertAnywhere: Geometrically Grounded and Optics-Aware Video Object Insertion

Recent advances in diffusion models have enabled impressive video editing capabilities, yet production-grade Video Object Insertion (VOI) remains challenging due to inadequate 4D scene understanding and a lack of proper optical interactions, such as shadows and reflections. To address these limitations, we present InsertAnywhere, a comprehensive VOI framework that achieves geometrically grounded object placement and optics-aware video synthesis. Our approach first leverages a 4D-aware mask generation module that allows users to anchor an object's 3D pose in a single frame. The framework automatically propagates this placement across the video, accurately handling local scene dynamics and occlusions. To synthesize realistic physical lighting interactions, we introduce Optics-Aware Representation Alignment, a novel strategy that utilizes an extended mask to guide feature extraction, enabling optical effects to seamlessly extend beyond the inserted object's boundary. Finally, to overcome the lack of training data for such phenomena, we construct and open-source ROSE++, a specialized quadruplet dataset tailored for the supervised learning of optical effects. Extensive experiments demonstrate that InsertAnywhere produces geometrically plausible and photometrically realistic insertions in complex real-world scenarios, significantly outperforming existing research and commercial generative tools.

Hoiyeong Jin, Hyojin Jang, Junha Hyung +6

KAIST AI · SK Telecom

Video Diffusion Models Video Editing Video Object Insertion

Dec 10, 2025

DirectSwap: Paired, Mask-Free Video Head Swapping with Full-Reference Evaluation

Head swapping replaces an entire head while preserving pose, expression, body motion, and scene. Progress is limited by the lack of cross-identity paired videos: real footage cannot provide different identities performing exactly the same motion, leaving the task without paired supervision or frame-aligned ground truth. Existing methods therefore rely on same-identity masked reconstruction, which restricts supervision to predefined editable regions. To address this, we introduce an identity-expression decoupled synthesis pipeline that constructs expression-synchronized cross-identity video pairs from real footage. Expression-bearing facial regions are retained under small clip-consistent geometric perturbations, while the surrounding head is regenerated with a new identity as synthesized swapping input. These pairs form a cross-identity benchmark with frame-aligned real-world video as ground-truth targets, enabling full-reference evaluation of identity, expression, pose, reconstruction fidelity, and temporal stability. This yields HeadSwapBench, the first cross-identity paired dataset for video head swapping, supporting both training (20,278 videos) and benchmarking (1,040 videos). With the cross-identity supervision enabled by this paired dataset, we propose DirectSwap, a mask-free video head swapping training paradigm. Under otherwise identical settings, this formulation outperforms head- and rectangle-masked same-identity reconstruction, particularly for bidirectional head-silhouette changes. At inference, driving-output divergence and emergent reference attention estimate the edit support, allowing unchanged non-head content to be restored from the driving video without external segmentation or additional training. On 1,040 clips from 104 unseen subjects, the resulting model demonstrates that the proposed paired supervision supports effective whole-head swapping.

Yanan Wang, Shengcai Liao, Panwen Hu +4

Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE · United Arab Emirates University (UAEU), Al Ain, UAE · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE +1

Video Editing Video Generation Evaluation

Nov 5, 2025

Unified Long Video Inpainting and Outpainting via Overlapping High-Order Co-Denoising

Diffusion-based text-to-video models are increasingly capable, but mask-based editing over hundreds of frames remains challenging: naïve long-video generation suffers from memory blow-up, window seams, and temporal drift, while existing editors often require specialized modules or heavy fine-tuning. We present Overlapping High-Order Co-Denoising, a lightweight framework that turns a single pre-trained text-to-video model into a unified inpainting-outpainting editor. We train only LoRA adapters using mixed interior and border masks together with a dual-region loss that improves synthesis inside the mask while explicitly preserving known content. At inference, we denoise long latent sequences using overlapping windows, apply second-order Heun sampling within each window, and fuse overlaps with Hamming-weighted blending to reduce boundary artifacts and improve temporal coherence. On InpaintBench (30 real-world videos, 81--300 frames), our method outperforms Wan 2.1 variants and VACE in background faithfulness (SSIM/LPIPS), temporal consistency (tLPIPS), and text alignment (CLIP), and scales to long horizons, demonstrated up to 800 frames, with memory bounded by the chosen window size.

Shuangquan Lyu, Jian Mao, Yue Ma

Carnegie Mellon University · Jilin University · Tsinghua University

Video Diffusion Models Long Video Generation Video Editing Image Inpainting Temporal Consistency in Video Generation Diffusion-Based Video Editing

Oct 15, 2025

Edit-Your-Interest: Efficient Video Editing via Feature Most-Similar Propagation

Text-to-image diffusion models have advanced video editing. However, existing video editing methods are limited by their high computational overhead and memory consumption. Furthermore, these approaches often sacrifice visual fidelity, leading to temporal inconsistencies and artifacts such as blurring and mosaic-like patterns. To address these dual challenges and strike a balance between computational efficiency and visual fidelity, we propose Edit-Your-Interest, a lightweight, text-driven, zero-shot video editing method. Edit-Your-Interest introduces a Spatio-Temporal Feature Memory bank (SFM) to cache spatial-attention features from previous frames under a fixed memory budget. The SFM maintains a representative candidate pool, reducing the computational overhead of full-sequence spatio-temporal modeling. To query this memory, we propose Feature Most-Similar Propagation (FMP), a similarity-based retrieval rule that propagates relevant cached tokens to the current frame. Together, SFM and FMP form a bounded-memory retrieval-based propagation design that improves temporal consistency while reducing blurring and mosaic-like artifacts. Finally, we introduce a feature-diversity-based SFM update algorithm to keep the memory representative and non-redundant throughout the video sequence. Furthermore, to enable precise object editing, we leverage cross-attention maps to automatically extract masks for the instances of interest. These masks are seamlessly integrated into the diffusion denoising process, enabling fine-grained control over target objects and allowing Edit-Your-Interest to perform highly accurate edits while robustly preserving the background integrity. Extensive experiments demonstrate that Edit-Your-Interest outperforms state-of-the-art methods in both efficiency and visual fidelity. Project page: https://edit-your-interest.github.io/.

Yi Zuo, Zitao Wang, Lingling Li +3

Xidian University,Xi’an, 710071, Shaanxi Province, China

Diffusion-Based Video Editing Video Editing Instruction-Based Video Editing Temporal Consistency in Video Generation

Oct 9, 2025

UniVideo: Unified Understanding, Generation, and Editing for Videos

Unified multimodal models have shown promising results in multimodal content generation and editing but remain largely limited to the image domain. In this work, we present UniVideo, a versatile framework that extends unified modeling to the video domain. UniVideo adopts a dual-stream design, combining a Multimodal Large Language Model (MLLM) for instruction understanding with a Multimodal DiT (MMDiT) for video generation. This design preserves the MLLM's original text generation capabilities, enables accurate interpretation of complex multimodal instructions, and maintains visual consistency in the generated content. Built on this architecture, UniVideo unifies diverse video generation and editing tasks under a single multimodal instruction paradigm and is jointly trained across them. Extensive experiments demonstrate that UniVideo matches or surpasses state-of-the-art task-specific baselines in text/image-to-video generation, in-context video generation and in-context video editing. Notably, the unified design of UniVideo enables two forms of generalization. First, UniVideo supports task composition, such as combining editing with style transfer, by integrating multiple capabilities within a single instruction. Second, even without explicit training on free-form video editing, UniVideo transfers its editing capability from large-scale image editing data to this setting, handling unseen instructions such as changing the environment or altering materials within a video. Beyond these core capabilities, UniVideo also supports visual-prompt-based video generation, where the MLLM interprets visual prompts and guides the MMDiT during synthesis. To foster future research, we released our model and code.

Cong Wei, Quande Liu, Zixuan Ye +5

University of Waterloo · Kling Team, Kuaishou Technology

Video Generation Unified Multimodal Models Video Editing Image-to-Video Generation Multimodal Diffusion Models Instruction-Based Video Editing+1

Sep 30, 2025

VRWKV-Editor: Reducing quadratic complexity in transformer-based video editing

In light of recent progress in video editing, deep learning models focusing on both spatial and temporal dependencies have emerged as the primary method. However, these models suffer from the quadratic computational complexity of traditional attention mechanisms, making them difficult to adapt to long-duration and high-resolution videos. This limitation restricts their applicability in practical contexts such as real-time video processing. To tackle this challenge, we introduce a method to reduce both time and space complexity of these systems by proposing VRWKV-Editor, a novel video editing model that integrates a linear spatio-temporal aggregation module into video-based diffusion models. VRWKV-Editor leverages bidirectional weighted key-value recurrence mechanism of the RWKV transformer to capture global dependencies while preserving temporal coherence, achieving linear complexity without sacrificing quality. Extensive experiments demonstrate that the proposed method achieves up to 3.7x speedup and 60% lower memory usage compared to state-of-the-art diffusion-based video editing methods, while maintaining competitive performance in frame consistency and text alignment. Furthermore, a comparative analysis we conducted on videos with different sequence lengths confirms that the gap in editing speed between our approach and architectures with self-attention becomes more significant with long videos.

Abdelilah Aitrouga, Youssef Hmamouche, Amal El Fallah Seghrouchni

International Artificial Intelligence Center of Morocco University Mohammed VI Polytechnic Rabat, Morocco · Sorbonne University, LIP6 - UMR 7606 CNRS, France

Video Diffusion Models Video Editing Diffusion Model Acceleration Temporal Consistency in Video Generation

Jun 1, 2025

MoCA-Video: Motion-Aware Concept Alignment for Consistent Video Editing

Unlike traditional video editing or inpainting, video semantic mixing fuses a reference concept with a moving target entity to produce a hybrid while preserving the source video's motion and layout. We propose MoCA-Video, a training-free framework that steers a frozen video-diffusion denoising trajectory through concept-localized reference injection. At selected low-noise steps, MoCA-Video uses concept attention to localize the target object and injects the reference latent into the localized region, where object structure has formed but appearance remains editable. A momentum-based correction carries the injected prediction across frames to encourage coherent concept integration through the sequence. We further introduce CASS, a CLIP-based metric that measures the output's directional alignment shift toward the reference and away from the source prompt. Using the denoiser's internal attention avoids an external localization model; in our A100 FP16 setup, MoCA-Video takes 3.2 seconds per output frame, excluding preprocessing. Across the evaluated baselines, MoCA-Video achieves the highest CASS, rel-CASS, and ImageReward, while LPIPS-T and FVD expose separate temporal-coherence and video-quality trade-offs.

Tong Zhang, Victor Escorcia, Juan C Leon Alcazar +1

Department of Computer Science King Abdullah University of Science and Technology Thuwal, Saudi Arabia · Independent Researcher

Diffusion Model Guidance Video Editing Temporal Consistency in Video Generation Diffusion-Based Video Editing

Dec 17, 2024

PRIMEdit: Probability Redistribution for Instance-Aware Multi-Object Video Editing With a New Benchmark Dataset

Recent Artificial Intelligence (AI)-based video editing has enabled users to edit videos through simple text prompts, significantly simplifying the editing process. However, recent zero-shot video editing techniques focus on global or single-object edits, which can lead to unintended changes in other parts of the video. When editing multiple objects only within their localized regions, existing methods face challenges, such as unfaithful editing, editing leakage, and lack of suitable evaluation datasets and metrics. To overcome these limitations, we propose

\textbf{P}

robability

\textbf{R}

edistribution for

\textbf{I}

nstance-aware

\textbf{M}

ulti-object Video

\textbf{Edit}

ing (

\textbf{PRIMEdit}

). PRIMEdit is a zero-shot framework that introduces two key modules: (i) Instance-centric Probability Redistribution (IPR) to ensure precise localization and faithful editing and (ii) Disentangled Multi-instance Sampling (DMS) to prevent editing leakage. Additionally, we present the MIVE dataset for multi-instance video editing, including its subsets categorized by instance counts and sizes for controlled analysis across different levels of scene complexity. We also introduce the Cross-Instance Accuracy (CIA) Score that is firstly proposed to evaluate the editing leakage in multi-instance video editing tasks, which exhibits strong correlation with human evaluation as measured by Spearman's

ρ= 0.86

. Our extensive qualitative, quantitative, and user study evaluations demonstrate that PRIMEdit significantly outperforms recent state-of-the-art methods in terms of editing faithfulness, accuracy, and leakage prevention, setting a new benchmark for multi-instance video editing.

Samuel Teodoro, Agus Gunawan, Soo Ye Kim +2

KAIST · Adobe Research · Chung-Ang University

Video Editing Instruction-Based Video Editing Video Editing Benchmarks

Jun 26, 2024

Diffusion Model-Based Video Editing: A Survey

The rapid development of diffusion models (DMs) has significantly advanced image and video applications, making "what you want is what you see" a reality. Among these, video editing has gained substantial attention and seen a swift rise in research activity, necessitating a comprehensive and systematic review of the existing literature. This paper reviews diffusion model-based video editing techniques, including theoretical foundations and practical applications. We begin by overviewing the mathematical formulation and image domain's key methods. Subsequently, we categorize video editing approaches by the inherent connections of their core technologies, depicting evolutionary trajectory. This paper also dives into novel applications, including point-based editing and pose-guided human video editing. Additionally, we present a comprehensive comparison using our newly introduced V2VBench. Building on the progress achieved to date, the paper concludes with ongoing challenges and potential directions for future research.

Wenhao Sun, Rong-Cheng Tu, Jingyi Liao +1

College of Computing and Data Science, Nanyang Technological University.

Video Diffusion Models Video Editing Diffusion-Based Video Editing