PISCO: Precise Video Instance Insertion with Sparse Control

Xiangbo Gao; Renjie Li; Xinghao Chen; Yuheng Wu; Suofei Feng; Qing Yin; Zhengzhong Tu

cs.CVFeb 9, 2026

PISCO: Precise Video Instance Insertion with Sparse Control

Authors: Xiangbo Gao, Renjie Li, Xinghao Chen, Yuheng Wu, Suofei Feng, Qing Yin, Zhengzhong Tu

Organizations: Texas A&M University · KAIST · Stanford University · Visko Platform

Abstract

The landscape of AI video generation is undergoing a pivotal shift: moving beyond general generation - which relies on exhaustive prompt-engineering and "cherry-picking" - towards fine-grained, controllable generation and high-fidelity post-processing. In professional AI-assisted filmmaking, it is crucial to perform precise, targeted modifications. A cornerstone of this transition is video instance insertion, which requires inserting a specific instance into existing footage while maintaining scene integrity. Unlike traditional video editing, this task demands several requirements: precise spatial-temporal placement, physically consistent scene interaction, and the faithful preservation of original dynamics - all achieved under minimal user effort. In this paper, we propose PISCO, a video diffusion model for precise video instance insertion with arbitrary sparse keyframe control. PISCO allows users to specify a single keyframe, start-and-end keyframes, or sparse keyframes at arbitrary timestamps, and automatically propagates object appearance, motion, and interaction. To address the severe distribution shift induced by sparse conditioning in pretrained video diffusion models, we introduce Variable-Information Guidance for robust conditioning and Distribution-Preserving Temporal Masking to stabilize temporal generation, together with geometry-aware conditioning for realistic scene adaptation. We further construct PISCO-Bench, a benchmark with verified instance annotations and paired clean background videos, and evaluate performance using both reference-based and reference-free perceptual metrics. Experiments demonstrate that PISCO consistently outperforms strong inpainting and video editing baselines under sparse control, and exhibits clear, monotonic performance improvements as additional control signals are provided. Project page: xiangbogaobarry.github.io/PISCO.

Figures & tables

Figure 1 : PISCO enables precise video instance insertion with arbitrary sparse keyframe control. Given a clean input video and a few user-provided instance cutouts at selected timestamps, PISCO inserts the instance with coherent temporal propagation and physical effects while preserving the original background dynamics.

Figure 2 : Overview of PISCO pipeline. We train a conditional video diffusion model with sparse keyframe control via Variable-Information Guidance (VIG), and stabilize sparse conditioning under pretrained temporal VAEs using Distribution-Preserving Temporal Masking (DPTM): pixel-space nearest-frame interpolation followed by token-level masking, with spatial mask and availability signals injected alongside RGB/depth conditions.

Figure 3 : Visualization of the depth-aware insertion . Conditioning on depth improves depth ordering and occlusion handling, reducing foreground/background blending artifacts compared to a depth-agnostic variant.

Figure 4 : Effectiveness of DPTM under sparse guidance. We compare results given segmented instance inputs only on odd frames. Naïve masking leads to distribution shifts and temporal artifacts, whereas DPTM preserves encoder input statistics and significantly improves temporal stability.

Figure 5 : Visualization of amodal instance completion and relighting augmentations. We complete occluded instance cutouts to form pseudo-amodal inputs and apply moderate relighting to improve illumination compatibility with the target scene while preserving keyframe appearance controllability.

Figure 6 : Effectiveness of relighting augmentation. Without relighting, the model over-preserves the input instance appearance, causing illumination mismatch after insertion. Relighting augmentation encourages lighting adaptation, yielding more coherent blending with the surrounding scene.

Method		Instance		Instance Mask
Method	Video	First	Last	First	Last	All	Text
Image Editing + I2V Generation
First	✓	✓		✓			✓
First&Last	✓	✓	✓	✓	✓		✓
Video Inpainting
CoCoCo	✓			✓	✓	✓	✓

Table 1 : Input conditions for all compared methods. We indicate whether each approach uses the clean video, reference instance frames (first/last), and instance masks (sparse or dense), as well as an optional text prompt. Notably, several baselines require dense per-frame masks, whereas PISCO operates with sparse keyframe masks (first-only or first&last).

Method	Whole Video				Foreground
Method	FVD $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	FVD $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
Image Editing + I2V Generation
First Only	826	0.451	15.47	0.55	297	0.030	30.24	0.97
First + Last	624	0.392	16.44	0.56	250	0.030	30.38	0.98
Video Inpainting
CoCoCo	590	0.191	23.62	0.80	398	0.031	30.26	0.97

Table 2 : Quantitative Results (FVD, LPIPS, PSNR, SSIM). Comparisons on Whole Video and Foreground Region. Blue and red indicate best and second-best among standard settings. Gray rows indicate the Five-Frame reference setting (not included in ranking).

Method	Background Consistency	Subject Consistency	Aesthetic Quality	Imaging Quality	Motion Smoothness	Overall Consistency	Temporal Flickering	Temporal Style	Average
Image Editing + I2V Generation
First Only	91.43	83.86	48.83	59.01	97.89	15.11	95.89	15.11	63.39
First + Last	92.28	85.09	49.47	59.71	98.18	15.58	96.26	15.58	64.02
Video Inpainting
CoCoCo	94.63	89.55	47.81	55.76	98.67	14.61	97.69	14.61	64.17
VideoPainter	94.51	89.14	47.82	57.68	98.97	13.85	97.77	13.85	64.20

Table 3 : VBench Quantitative Comparison. We compare PISCO against recent baselines. To ensure precise evaluation for instance insertion, Background and Subject Consistency are computed using masked regions to isolate foreground/background signals. Blue and red indicate the best and second-best results among standard settings. Gray rows indicate the Five-Frame setting.

Figure 7 : Qualitative Comparison . Visual results of instance insertion in two dynamic scenes (A) and (B). Existing methods struggle with background preservation (Agentic), instance identity (Inpainting), or scale consistency (UniVideo/VACE). Our method, PISCO , achieves superior visual fidelity and spatiotemporal alignment, particularly when utilizing sparse "First & Last" frame control.

Figure 8 : Broader applications beyond instance insertion. Leveraging the same instance-level conditioning and temporal propagation, PISCO supports diverse instance-centric video edits and simulation under controllable spatiotemporal guidance: Background Change , Instance Repositioning , Speed Change , Instance Rescaling , and Dynamics Simulation .

Explore similar work

May 22, 2026cs.CV

SimInsert: Seamless Video Object Insertion via Regional Sparse Attention Fusion

Video object insertion requires ensuring spatio-temporal coherence and interactive realism, extending far beyond simple content placement. However, current approaches are often hindered by a reliance on explicit motion engineering or resource-intensive retraining, restricting their flexibility and generalization. To bridge this gap, we present \textit{SimInsert}, a training-free paradigm that efficiently decouples the task into intuitive single-frame editing and semantic motion description. By harnessing the robust generative priors of image-to-video diffusion models, SimInsert propagates edits temporally, strictly preserving background invariance while enabling plausible, text-driven interactions between the inserted object and the dynamic environment. Our approach hinges on non-invasive guidance mechanisms that enforce structural consistency, facilitate seamless boundary fusion, and counteract the fidelity drift that typically accumulates during the denoising trajectory. Extensive quantitative experiments validate our efficacy: SimInsert surpasses state-of-the-art methods with an 18.8% gain in PSNR, 20.1% in SSIM, and a 44.1% decrease in LPIPS, offering a streamlined solution for high-fidelity video editing.

Xinyu Chen, Yuyi Qian, Jiang Lin +9

1State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, China · 4Xi’an Jiaotong-Liverpool University +3

Video Editing Diffusion-Based Image Editing Spatial Coherence Objects

Ask PDF

May 31, 2026cs.GR

AlbedoEdit: Unified Instance-Level Video Editing with Albedo Guidance

Video generative models have achieved remarkable progress in synthesizing photorealistic video sequences. However, enabling broader and more creative downstream applications requires fine-grained instance-level video editing, including object insertion, object removal, and texture editing, which has emerged as a prominent yet challenging problem. Existing approaches either propose unified generative frameworks with only coarse semantic control, or design task-specific frameworks for individual editing tasks, limiting their flexibility and applicability across diverse real-world scenarios. To address these limitations, we propose AlbedoEdit, a unified generative video editing framework that jointly supports object insertion, object removal, and texture editing. Our key insight is that the intrinsic albedo map, which is invariant to lighting and contains no specularity, shadowing and inter-reflection effects, provides an effective and user-friendly mechanism for specifying fine-grained appearance edits. Built upon video foundation models, AlbedoEdit is fine-tuned to translate source RGB videos into edited RGB videos, conditioned on a user-edited first-frame albedo. Trained on a new paired synthetic dataset covering all three editing tasks, AlbedoEdit implicitly learns to harmonize edited contents and simulate complex real-world visual effects triggered by editing operations, including specular highlights, soft shadows, and mirror reflections. AlbedoEdit demonstrates superior performance over state-of-the-art video editing approaches, both qualitatively and quantitatively. Project webpage is https://vcai.mpi-inf.mpg.de/projects/AlbedoEdit/.

Xilong Zhou, Bao-Huy Nguyen, Zheng Zeng +6

Max Planck Institute for Informatics · University of California Santa Barbara · NVIDIA Research +1

Video Editing Video Foundation Models

Ask PDF

Dec 19, 2025cs.CV

InsertAnywhere: Geometrically Grounded and Optics-Aware Video Object Insertion

Recent advances in diffusion models have enabled impressive video editing capabilities, yet production-grade Video Object Insertion (VOI) remains challenging due to inadequate 4D scene understanding and a lack of proper optical interactions, such as shadows and reflections. To address these limitations, we present InsertAnywhere, a comprehensive VOI framework that achieves geometrically grounded object placement and optics-aware video synthesis. Our approach first leverages a 4D-aware mask generation module that allows users to anchor an object's 3D pose in a single frame. The framework automatically propagates this placement across the video, accurately handling local scene dynamics and occlusions. To synthesize realistic physical lighting interactions, we introduce Optics-Aware Representation Alignment, a novel strategy that utilizes an extended mask to guide feature extraction, enabling optical effects to seamlessly extend beyond the inserted object's boundary. Finally, to overcome the lack of training data for such phenomena, we construct and open-source ROSE++, a specialized quadruplet dataset tailored for the supervised learning of optical effects. Extensive experiments demonstrate that InsertAnywhere produces geometrically plausible and photometrically realistic insertions in complex real-world scenarios, significantly outperforming existing research and commercial generative tools.

Hoiyeong Jin, Hyojin Jang, Junha Hyung +6

KAIST AI · SK Telecom

Optical Objects

Ask PDF