BeyondMasks: Evaluating Causal and Physical Consistency in Video Object Removal

Yigit Ekin; Enes Sanli; Aykut Erdem; Erkut Erdem; Aysegul Dundar

cs.CVAug 20, 2026

BeyondMasks: Evaluating Causal and Physical Consistency in Video Object Removal

Authors: Yigit Ekin, Enes Sanli, Aykut Erdem, Erkut Erdem, Aysegul Dundar

Organizations: Bilkent University, Department of Computer Engineering, Ankara, Türkiye · Koç University, Department of Computer Engineering, Istanbul, Türkiye · KUIS AI Center, Istanbul, Türkiye · Hacettepe University, Department of AI and Data Engineering, Ankara, Türkiye

Abstract

Recent advances in generative video models have significantly improved visual realism in video object removal, yet evaluation protocols still focus on masked region fidelity, treating removal as local inpainting. In real scenes, object removal is a causal intervention: eliminating an object also requires removing its induced physical effects, such as shadows, reflections, illumination changes, translucency, and dynamic traces. Existing benchmarks lack aligned clean references or remain limited to simplified synthetic settings, preventing systematic evaluation of causal consistency. We introduce BeyondMasks, a paired benchmark for causally consistent video object removal, consisting of temporally aligned synthetic and real world video pairs with clean background references. The dataset spans diverse photometric, geometric, volumetric, and dynamic interactions, and supports both mask based and instruction driven editing. We further propose CORE, a structured vision language model based evaluation protocol that jointly measures object disappearance and after effect consistency, aligning more closely with human judgments than existing metrics. Benchmarking state of the art methods reveals systematic failures in removing secondary physical effects despite high masked region fidelity, exposing a gap between visual plausibility and causal correctness. BeyondMasks reframes video object removal as causal scene consistency rather than local reconstruction and provides a unified framework for its evaluation.

Figures & tables

Figure 1: Video object removal requires eliminating both objects and their physical after-effects. Objects interact with their environment through shadows, reflections, translucency, illumination changes, volumetric effects (e.g., steam), and dynamic physical traces. Removing the object alone does not restore the scene if these effects persist. BeyondMasks introduces a benchmark to evaluate such causal object-environment interactions using temporally aligned video pairs, enabling systematic assessment of whether methods recover the scene as if the object had never been present.

Dataset	Size	Paired GT	After-effects	Causal effects	Avg. video length	Source
DAVIS	90	✗	✗	✗	$\sim$ 3 secs	Real
YouTube-VOS	508	✗	✗	✗	$\sim$ 4.5 secs	Real
HQVI	10	✓	✗	✗	$\sim$ 3.5 secs	Real
ROSE-Bench	60	✓	✓	✗	$\sim$ 3 secs	Synthetic
Ours	180	✓	✓	✓	$\sim$ 5.5 secs	Both

Table 1: Comparison of video object removal benchmarks. Unlike prior datasets, BeyondMasks provides temporally aligned paired ground truth while explicitly modeling diverse causal after-effects in both synthetic and real-world scenes.

Figure 2: Overview of the BeyondMasks construction pipeline. Top: Paired intervention-based generation, where an object-present video is derived from a clean background video, ensuring temporal alignment. Bottom: Representative categories of object-induced causal after-effects, including illumination, reflection, volumetric, and dynamic interaction effects.

Figure 3: Pixel metrics fail to capture causal correctness. Although LPIPS scores are similar across methods, their ability to remove object-induced effects (e.g., shadows or reflections) differs substantially. CORE separates these failure modes by evaluating object removal (CORE-OS) and elimination of causal after-effects (CORE-AES). $\downarrow$ better for LPIPS; $\uparrow$ better for CORE scores.

Figure 4: Overview of CORE (Causal Object Removal Evaluation) . Given an edited video $\hat{V}^{bg}$ and its temporally aligned clean reference $V^{bg}$ , CORE performs structured VLM-based comparison to assess interventional correctness. The evaluation jointly measures (i) object disappearance and (ii) consistency of object-induced physical after-effects, such as shadows, reflections, and dynamic perturbations. By conditioning semantic reasoning on aligned references, CORE moves beyond masked-region fidelity toward evaluation of causal and physical consistency.

Method	Type	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$	CORE-OS $\uparrow$	CORE-AES $\uparrow$
Lucy Edit (arXiv 2025) [ 23 ]	T	18.5884	0.7239	0.2758	480.08	1.443	1.551
VACE (ICCV 2025) [ 11 ]	T+M	19.9932	0.7856	0.2077	418.79	1.799	1.698
CoCoCo (AAAI 2025) [ 44 ]	T+M	20.8565	0.7287	0.2404	381.84	1.986	2.043
Gen. Omnimatte (CVPR 2025) [ 13 ]	T+M	23.8106	0.8175	0.1968	250.76	3.771	3.021
ProPainter (ICCV 2023) [ 41 ]	M	23.5979	0.8429	0.1635	225.28	3.063	2.368
DiffuEraser (arXiv 2025) [ 15 ]	M	25.1880	0.8769	0.1124	164.26	3.494	2.625

Table 2: Quantitative comparison on our benchmark. We evaluate representative video object removal methods using standard reconstruction metrics (PSNR, SSIM, LPIPS), the video realism metric FVD, and our VLM-based CORE evaluation. Type indicates the supervision required by each method: T denotes text-instruction-based editing, M denotes mask-guided removal, and T+M indicates methods that support both inputs. Bold indicates the best score.

After-effect	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	CORE-OS $\uparrow$	CORE-AES $\uparrow$
Shadow	23.6870	0.8298	0.1621	3.498	2.520
Reflection	23.9842	0.8558	0.1580	3.332	2.446
Light source	22.4233	0.8201	0.1849	3.554	2.248
Steam	22.8375	0.8466	0.1624	3.236	2.032
Translucent	22.9326	0.8156	0.2162	3.256	2.701
Causal	22.7912	0.8163	0.1867	3.486	2.141

Table 3: Average performance across after-effect categories. Results are averaged across the top-performing six models for each after-effect type. Higher values are better for PSNR, SSIM, CORE-OS, and CORE-AES, while lower values indicate better performance for LPIPS.

Figure 5: Qualitative comparison across six after-effect categories. Each category shows two temporal frames.

Figure 6: Context loss in mask-based video object removal . Many mask-based pipelines zero out the masked region prior to inference, eliminating potentially informative visual evidence (e.g., background partially visible through translucent structures). This removal of observable context forces the model to hallucinate missing content rather than recover available information, resulting in degraded restoration and incomplete causal consistency relative to the ground truth.

After-Effect Category	Count
Shadow	127
Reflection	61
Translucent effects	46
Causal physical effects	37
Light source effects	38
Steam / scattering	20

Table 4: Distribution of after-effect categories in BeyondMasks. Categories are not mutually exclusive; a single video may exhibit multiple interaction types.

Panel of Figure 7 — Figure 7: Localized differences in synthetic pairs.

Method	Type	Shadow	Reflection	Light	Steam	Translucent	Causal
Lucy Edit (arXiv) [ 23 ]	T	0.659	0.647	0.586	0.636	0.675	0.669
VACE (ICCV 2025) [ 11 ]	T+M	0.658	0.645	0.597	0.630	0.679	0.666
CoCoCo (AAAI 2025) [ 44 ]	T+M	0.662	0.651	0.593	0.648	0.685	0.661
Gen. Omnimatte (CVPR 2025) [ 13 ]	T+M	0.676	0.663	0.622	0.638	0.705	0.684
ProPainter (ICCV 2023) [ 41 ]	M	0.668	0.669	0.614	0.636	0.701	0.665
DiffuEraser (arXiv) [ 15 ]	M	0.664	0.667	0.615	0.647	0.696	0.679

Table 5: Evaluation using the ReMOVE metric on BeyondMasks across causal after-effect categories. Higher values indicate better performance. Type denotes the supervision modality: T indicates text-instruction-based editing, M indicates mask-guided removal, and T+M denotes methods supporting both inputs. Bold numbers indicate the best score in each column. Bold numbers indicate the best score in each column.

Method	Type	Shadow	Reflection	Light	Steam	Translucent	Causal
		CFD Score $\uparrow$
Lucy Edit (arXiv) [ 23 ]	T	0.677	0.652	0.667	0.763	0.805	0.668
VACE (ICCV 2025) [ 11 ]	T+M	0.558	0.585	0.520	0.631	0.670	0.594
CoCoCo (AAAI 2025) [ 44 ]	T+M	0.326	0.358	0.368	0.317	0.450	0.292
Gen. Omnimatte (CVPR 2025) [ 13 ]	T+M	0.285	0.350	0.359	0.376	0.404	0.260
ProPainter (ICCV 2023) [ 41 ]	M	0.253	0.248	0.341	0.275	0.301	0.249

Table 6: Evaluation using the CFD metric on BeyondMasks across causal after-effect categories. Higher values indicate better performance. Type denotes the supervision modality: T indicates text-instruction-based editing, M indicates mask-guided removal, and T+M denotes methods supporting both inputs. Bold numbers indicate the best score in each column. Bold numbers indicate the best score in each column.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	CORE-OS $\uparrow$	CORE-AES $\uparrow$
Lucy Edit (arXiv) [ 23 ]	18.8356 / 18.3282	0.7529 / 0.6942	0.2228 / 0.3285	1.069 / 1.813	1.276 / 1.824
VACE (ICCV 2025) [ 11 ]	18.9362 / 18.6913	0.7547 / 0.7236	0.2209 / 0.3010	1.415 / 2.176	1.404 / 1.989
CoCoCo (AAAI 2025) [ 44 ]	21.6499 / 20.1249	0.7625 / 0.6950	0.1962 / 0.2809	1.670 / 2.297	1.875 / 2.209
Gen. Omnimatte (CVPR 2025) [ 13 ]	25.0218 / 22.6120	0.8561 / 0.7794	0.1369 / 0.2562	3.865 / 3.670	2.966 / 3.055
ProPainter (ICCV 2023) [ 41 ]	24.6675 / 22.6555	0.8816 / 0.8060	0.1202 / 0.2044	3.326 / 2.800	2.371 / 2.361
DiffuEraser (arXiv) [ 15 ]	25.3617 / 25.1622	0.8932 / 0.8621	0.0942 / 0.1290	3.640 / 3.344	2.461 / 2.794

Table 7: Split-wise quantitative evaluation on BeyondMasks. Performance of representative video object removal and image object removal methods on the synthetic and real subsets of the dataset. Each entry reports Synthetic / Real results for reconstruction metrics (PSNR, SSIM, LPIPS) and the proposed CORE evaluation scores (CORE-OS and CORE-AES), which are measured on a discrete 1–5 scale. Image-based methods (OmniPaint and Object Clear) are applied frame-wise by processing one frame every 15 frames. Bold indicates the best value for each metric across all methods.

Metric	ObjectScore	AfterEffectScore
PSNR	0.15	0.28
SSIM	-0.14	-0.10
LPIPS	-0.02	0.10
CORE ${}_{\text{Gemini}}$	0.62	0.73
CORE ${}_{\text{GPT}}$	0.60	0.76
Inter Rater	0.69	0.79

Table 8: Correlation with human evaluation. Spearman correlation between automatic metrics and human ratings for ObjectScore and AfterEffectScore. Pixel-based metrics (PSNR, SSIM, LPIPS) are compared with VLM-based CORE evaluations using two judge models (Gemini and GPT). The Inter Rater row reports agreement between human annotators, serving as a reference level for achievable alignment.

Figure 8: Failure cases in real-world videos. For each example, the first row shows frames from the input video and the second row shows the output of a representative mask-based removal method. Although the target object is largely removed, residual object-induced effects remain visible. Red dashed boxes highlight typical artifacts such as persistent shadows, reflections in mirrors or glass surfaces, steam or smoke traces, and lighting inconsistencies that reveal incomplete removal of object-environment interactions.

Figure 9: Failure cases in synthetic scenes. The first row shows input frames and the second row shows the corresponding outputs after object removal. Despite successful removal of the foreground object, the edited results often retain interaction effects such as fence shadows, motion-induced dust trails, illumination artifacts, or water disturbances. Red dashed boxes indicate regions where the model reconstructs a visually plausible background but fails to remove the full causal footprint of the object.

Object Removal

See all 27 papers

VOID: Video Object and Interaction Deletion
Apr 2, 2026Saman Motamed, William Harvey, Benjamin Klein +3Object Removal Video Editing
Netflix · INSAIT, Sofia University “St. Kliment Ohridski”
From Understanding to Erasing: Towards Complete and Stable Video Object Removal
Apr 2, 2026Dingming Liu, Wenjing Wang, Chen Li +2Object Removal Video Editing
Peking University · WeChat Vision, Tencent Inc.
VOR-Bench: A Human Perception-Driven Benchmark for Video Object Removal
Sep 15, 2026Haonan Huang, Tianrui Qiu, Xianghao Zang +10Video Editing Video Quality Assessment
Beijing University of Posts and Telecommunications · China Telecom Artificial Intelligence Technology Co. Ltd · State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University

Apr 2, 2026J/K move · Enter open · S save

VOID: Video Object and Interaction Deletion

Saman Motamed, William Harvey, Benjamin Klein, Luc Van Gool+2

Netflix · INSAIT, Sofia University “St. Kliment Ohridski”

Object Removal Video Editing Physical Consistency in Video Generation Causal Reasoning in Videos

Open paper AskPDF arXiv

Existing video object removal methods excel at inpainting content "behind" the object and correcting appearance-level artifacts such as shadows and reflections. However, when the removed object has more significant interactions, such as collisions with other objects, current models fail to correct them and produce implausible results. We present VOID, a video object removal framework designed to perform physically-plausible inpainting in these complex scenarios. To train the model, we generate a new paired dataset of counterfactual object removals using Kubric and HUMOTO, where removing an object requires altering downstream physical interactions. During inference, a vision-language model identifies regions of the scene affected by the removed object. These regions are then used to guide a video diffusion model that generates physically consistent counterfactual outcomes. Experiments on both synthetic and real data show that our approach better preserves consistent scene dynamics after object removal compared to prior video object removal methods. We hope this framework sheds light on how to make video editing models better simulators of the world through high-level causal reasoning.

Abstract

Object Removal

Explore similar work

VOID: Video Object and Interaction Deletion

From Understanding to Erasing: Towards Complete and Stable Video Object Removal

VOR-Bench: A Human Perception-Driven Benchmark for Video Object Removal