Physics-Aware Video Instance Removal Benchmark

Zirui Li; Xinghao Chen; Lingyu Jiang; Dengzhe Hou; Fangzhou Lin; Kazunori Yamada; Xiangbo Gao; Zhengzhong Tu

cs.CVApr 7, 2026

Physics-Aware Video Instance Removal Benchmark

Authors: Zirui Li, Xinghao Chen, Lingyu Jiang, Dengzhe Hou, Fangzhou Lin, Kazunori Yamada, Xiangbo Gao, Zhengzhong Tu

Organizations: Tohoku University · University of Washington · Texas A&M University

Abstract

Video Instance Removal (VIR) requires removing target objects while maintaining background integrity and physical consistency, such as specular reflections and illumination interactions. Despite advancements in text-guided editing, current benchmarks primarily assess visual plausibility, often overlooking the physical causalities, such as lingering shadows, triggered by object removal. We introduce the Physics-Aware Video Instance Removal (PVIR) benchmark, featuring 95 high-quality videos annotated with instance-accurate masks and removal prompts. PVIR is partitioned into Simple and Hard subsets, the latter explicitly targeting complex physical interactions. We evaluate four representative methods, PISCO-Removal, UniVideo, DiffuEraser, and CoCoCo, using a decoupled human evaluation protocol across three dimensions to isolate semantic, visual, and spatial failures: instruction following, rendering quality, and edit exclusivity. Our results show that PISCO-Removal and UniVideo achieve state-of-the-art performance, while DiffuEraser frequently introduces blurring artifacts and CoCoCo struggles significantly with instruction following. The persistent performance drop on the Hard subset highlights the ongoing challenge of recovering complex physical side effects.

Figures & tables

Figure 1: Qualitative comparison on the PVIR benchmark. Each row presents a specific instance removal scenario with its corresponding textual prompt. As a unified model, UniVideo demonstrates strong physics-awareness, successfully removing coupled side effects like ground shadows (e.g., rows b and c); however, it suffers from severe semantic hallucinations, occasionally generating unprompted artifacts to fill the void (e.g., the distorted figure generated in row d). PISCO-Removal consistently achieves clean erasure of both the target and its physical side effects while maintaining high background fidelity. DiffuEraser reliably masks the correct instance but frequently leaves unnatural residual shadows and spatial blurring (rows c and d). Finally, CoCoCo struggles significantly with instruction following, often leaving obvious “ghosting” silhouettes or failing to remove the object entirely (e.g., the white duck in row a).

Subset	#Videos	Avg. Frames	Key Properties
Simple	57	81	weaker interaction and simpler geometry
Hard	38	81	reflection/mirror/specular and stronger coupling
Total	95	81	mixed scenes and motions

Table 1: Summary of the PVIR Benchmark Dataset. The benchmark comprises 95 high-definition videos, categorized into Simple and Hard subsets based on the complexity of physical interactions (e.g., reflections, shadows, and dynamic fluid wakes).

Score	Instruction Following	Rendering Quality	Edit Exclusivity
Score 1	Target not removed or unrelated edit	Severe artifacts; unusable output	Uncontrolled edits across scene
Score 2	Partial removal with obvious residual traces	Obvious distortion/flicker; poor temporal consistency	Multiple non-target regions altered
Score 3	Target mostly removed with minor artifacts	Moderate quality degradation but viewable result	Minor non-target changes but structure mostly preserved
Score 4	Target precisely removed with no obvious residuals	High visual quality and stable temporal consistency	Non-target regions preserved with only negligible differences

Table 2: Decoupled Human Evaluation Rubric. Each of the three dimensions—Instruction Following (IF), Rendering Quality (RQ), and Edit Exclusivity (EE)—is scored independently on a 1–4 scale, enabling granular diagnosis of model failure modes.

Subset	Method	Instruction Following $\uparrow$	Rendering Quality $\uparrow$	Edit Exclusivity $\uparrow$	Overall $\uparrow$
Overall	CoCoCo	1.60	1.84	3.07	2.17
	UniVideo	3.06	3.45	3.53	3.35
	DiffuEraser	2.89	2.63	3.52	3.01
	PISCO-Removal	3.62	3.28	3.58	3.49
Simple	CoCoCo	1.75	1.98	3.33	2.35
	UniVideo	3.21	3.34	3.53	3.36

Table 3: Performance comparison on the comprehensive benchmark (95 videos), including overall results and the Simple/Hard split breakdown. Metric scores range from 1 to 4, where higher values indicate superior performance. Dark green and light green denote the best and second-best results within each subset, respectively.

Figure 2: Cross-metric trade-off analysis across Instruction Following (IF), Rendering Quality (RQ), and Edit Exclusivity (EE). Each point represents a single video-model pair. Markers distinguish between the Simple ( $\bullet$ ) and Hard ( $\times$ ) subsets. The tight clustering in the top-right quadrant for PISCO-Removal and UniVideo indicates high-fidelity, balanced performance. Conversely, the wide dispersion of DiffuEraser and CoCoCo reveals inherent trade-offs between reliable target removal and background preservation.

Video Editing

Video editing uses AI to alter, compose, relight, or reorganize moving footage while controlling how changes persist across frames.

See all 178 papers

BeyondMasks: Evaluating Causal and Physical Consistency in Video Object Removal
Aug 20, 2026Yigit Ekin, Enes Sanli, Aykut Erdem +2Object Removal Video Editing
Bilkent University, Department of Computer Engineering, Ankara, Türkiye · Koç University, Department of Computer Engineering, Istanbul, Türkiye · KUIS AI Center, Istanbul, Türkiye +1
VOID: Video Object and Interaction Deletion
Apr 2, 2026Saman Motamed, William Harvey, Benjamin Klein +3Object Removal Video Editing
Netflix · INSAIT, Sofia University “St. Kliment Ohridski”
EffectLearner: World-Aware Object-Effect Reasoning for Real-World Video Object Removal
Aug 6, 2026Feier Wu, Wanke Xia, Xu He +8Video Diffusion Models Video Editing
Tsinghua University · China Agricultural University · Jiangnan University +1

Aug 20, 2026J/K move · Enter open · S save

BeyondMasks: Evaluating Causal and Physical Consistency in Video Object Removal

Yigit Ekin, Enes Sanli, Aykut Erdem, Erkut Erdem+1

Bilkent University, Department of Computer Engineering, Ankara, Türkiye · Koç University, Department of Computer Engineering, Istanbul, Türkiye · KUIS AI Center, Istanbul, Türkiye+1

Object Removal Video Editing Physical Reasoning in Video Causal Reasoning in Videos Video Generation Evaluation

Open paper AskPDF arXiv

Recent advances in generative video models have significantly improved visual realism in video object removal, yet evaluation protocols still focus on masked region fidelity, treating removal as local inpainting. In real scenes, object removal is a causal intervention: eliminating an object also requires removing its induced physical effects, such as shadows, reflections, illumination changes, translucency, and dynamic traces. Existing benchmarks lack aligned clean references or remain limited to simplified synthetic settings, preventing systematic evaluation of causal consistency. We introduce BeyondMasks, a paired benchmark for causally consistent video object removal, consisting of temporally aligned synthetic and real world video pairs with clean background references. The dataset spans diverse photometric, geometric, volumetric, and dynamic interactions, and supports both mask based and instruction driven editing. We further propose CORE, a structured vision language model based evaluation protocol that jointly measures object disappearance and after effect consistency, aligning more closely with human judgments than existing metrics. Benchmarking state of the art methods reveals systematic failures in removing secondary physical effects despite high masked region fidelity, exposing a gap between visual plausibility and causal correctness. BeyondMasks reframes video object removal as causal scene consistency rather than local reconstruction and provides a unified framework for its evaluation.

Abstract

Video Editing

Explore similar work

BeyondMasks: Evaluating Causal and Physical Consistency in Video Object Removal

VOID: Video Object and Interaction Deletion

EffectLearner: World-Aware Object-Effect Reasoning for Real-World Video Object Removal