Organizations: Bilkent University, Department of Computer Engineering, Ankara, Türkiye · Koç University, Department of Computer Engineering, Istanbul, Türkiye · KUIS AI Center, Istanbul, Türkiye · Hacettepe University, Department of AI and Data Engineering, Ankara, Türkiye
Recent advances in generative video models have significantly improved visual realism in video object removal, yet evaluation protocols still focus on masked region fidelity, treating removal as local inpainting. In real scenes, object removal is a causal intervention: eliminating an object also requires removing its induced physical effects, such as shadows, reflections, illumination changes, translucency, and dynamic traces. Existing benchmarks lack aligned clean references or remain limited to simplified synthetic settings, preventing systematic evaluation of causal consistency. We introduce BeyondMasks, a paired benchmark for causally consistent video object removal, consisting of temporally aligned synthetic and real world video pairs with clean background references. The dataset spans diverse photometric, geometric, volumetric, and dynamic interactions, and supports both mask based and instruction driven editing. We further propose CORE, a structured vision language model based evaluation protocol that jointly measures object disappearance and after effect consistency, aligning more closely with human judgments than existing metrics. Benchmarking state of the art methods reveals systematic failures in removing secondary physical effects despite high masked region fidelity, exposing a gap between visual plausibility and causal correctness. BeyondMasks reframes video object removal as causal scene consistency rather than local reconstruction and provides a unified framework for its evaluation.
Figures & tables
Figure 1: Video object removal requires eliminating both objects and their physical after-effects. Objects interact with their environment through shadows, reflections, translucency, illumination changes, volumetric effects (e.g., steam), and dynamic physical traces. Removing the object alone does not restore the scene if these effects persist. BeyondMasks introduces a benchmark to evaluate such causal object-environment interactions using temporally aligned video pairs, enabling systematic assessment of whether methods recover the scene as if the object had never been present.
Dataset
Size
Paired GT
After-effects
Causal effects
Avg. video length
Source
DAVIS
90
✗
✗
✗
∼ 3 secs
Real
YouTube-VOS
508
✗
✗
✗
∼ 4.5 secs
Real
HQVI
10
✓
✗
✗
∼ 3.5 secs
Real
ROSE-Bench
60
✓
✓
✗
∼ 3 secs
Synthetic
Ours
180
✓
✓
✓
∼ 5.5 secs
Both
Table 1: Comparison of video object removal benchmarks. Unlike prior datasets, BeyondMasks provides temporally aligned paired ground truth while explicitly modeling diverse causal after-effects in both synthetic and real-world scenes.
Figure 2: Overview of the BeyondMasks construction pipeline. Top: Paired intervention-based generation, where an object-present video is derived from a clean background video, ensuring temporal alignment. Bottom: Representative categories of object-induced causal after-effects, including illumination, reflection, volumetric, and dynamic interaction effects.
Figure 3: Pixel metrics fail to capture causal correctness. Although LPIPS scores are similar across methods, their ability to remove object-induced effects (e.g., shadows or reflections) differs substantially. CORE separates these failure modes by evaluating object removal (CORE-OS) and elimination of causal after-effects (CORE-AES). ↓ better for LPIPS; ↑ better for CORE scores.
Figure 4: Overview of CORE (Causal Object Removal Evaluation) . Given an edited video V^bg and its temporally aligned clean reference Vbg , CORE performs structured VLM-based comparison to assess interventional correctness. The evaluation jointly measures (i) object disappearance and (ii) consistency of object-induced physical after-effects, such as shadows, reflections, and dynamic perturbations. By conditioning semantic reasoning on aligned references, CORE moves beyond masked-region fidelity toward evaluation of causal and physical consistency.
Method
Type
PSNR ↑
SSIM ↑
LPIPS ↓
FVD ↓
CORE-OS ↑
CORE-AES ↑
Lucy Edit (arXiv 2025) [ 23 ]
T
18.5884
0.7239
0.2758
480.08
1.443
1.551
VACE (ICCV 2025) [ 11 ]
T+M
19.9932
0.7856
0.2077
418.79
1.799
1.698
CoCoCo (AAAI 2025) [ 44 ]
T+M
20.8565
0.7287
0.2404
381.84
1.986
2.043
Gen. Omnimatte (CVPR 2025) [ 13 ]
T+M
23.8106
0.8175
0.1968
250.76
3.771
3.021
ProPainter (ICCV 2023) [ 41 ]
M
23.5979
0.8429
0.1635
225.28
3.063
2.368
DiffuEraser (arXiv 2025) [ 15 ]
M
25.1880
0.8769
0.1124
164.26
3.494
2.625
Table 2: Quantitative comparison on our benchmark. We evaluate representative video object removal methods using standard reconstruction metrics (PSNR, SSIM, LPIPS), the video realism metric FVD, and our VLM-based CORE evaluation. Type indicates the supervision required by each method: T denotes text-instruction-based editing, M denotes mask-guided removal, and T+M indicates methods that support both inputs. Bold indicates the best score.
After-effect
PSNR ↑
SSIM ↑
LPIPS ↓
CORE-OS ↑
CORE-AES ↑
Shadow
23.6870
0.8298
0.1621
3.498
2.520
Reflection
23.9842
0.8558
0.1580
3.332
2.446
Light source
22.4233
0.8201
0.1849
3.554
2.248
Steam
22.8375
0.8466
0.1624
3.236
2.032
Translucent
22.9326
0.8156
0.2162
3.256
2.701
Causal
22.7912
0.8163
0.1867
3.486
2.141
Table 3: Average performance across after-effect categories. Results are averaged across the top-performing six models for each after-effect type. Higher values are better for PSNR, SSIM, CORE-OS, and CORE-AES, while lower values indicate better performance for LPIPS.
Figure 5: Qualitative comparison across six after-effect categories. Each category shows two temporal frames.
Figure 6: Context loss in mask-based video object removal . Many mask-based pipelines zero out the masked region prior to inference, eliminating potentially informative visual evidence (e.g., background partially visible through translucent structures). This removal of observable context forces the model to hallucinate missing content rather than recover available information, resulting in degraded restoration and incomplete causal consistency relative to the ground truth.
After-Effect Category
Count
Shadow
127
Reflection
61
Translucent effects
46
Causal physical effects
37
Light source effects
38
Steam / scattering
20
Table 4: Distribution of after-effect categories in BeyondMasks. Categories are not mutually exclusive; a single video may exhibit multiple interaction types.
Figure 7: Localized differences in synthetic pairs.
Method
Type
Shadow
Reflection
Light
Steam
Translucent
Causal
Lucy Edit (arXiv) [ 23 ]
T
0.659
0.647
0.586
0.636
0.675
0.669
VACE (ICCV 2025) [ 11 ]
T+M
0.658
0.645
0.597
0.630
0.679
0.666
CoCoCo (AAAI 2025) [ 44 ]
T+M
0.662
0.651
0.593
0.648
0.685
0.661
Gen. Omnimatte (CVPR 2025) [ 13 ]
T+M
0.676
0.663
0.622
0.638
0.705
0.684
ProPainter (ICCV 2023) [ 41 ]
M
0.668
0.669
0.614
0.636
0.701
0.665
DiffuEraser (arXiv) [ 15 ]
M
0.664
0.667
0.615
0.647
0.696
0.679
Table 5: Evaluation using the ReMOVE metric on BeyondMasks across causal after-effect categories. Higher values indicate better performance. Type denotes the supervision modality: T indicates text-instruction-based editing, M indicates mask-guided removal, and T+M denotes methods supporting both inputs. Bold numbers indicate the best score in each column. Bold numbers indicate the best score in each column.
CFD Score ↑
Method
Type
Shadow
Reflection
Light
Steam
Translucent
Causal
Lucy Edit (arXiv) [ 23 ]
T
0.677
0.652
0.667
0.763
0.805
0.668
VACE (ICCV 2025) [ 11 ]
T+M
0.558
0.585
0.520
0.631
0.670
0.594
CoCoCo (AAAI 2025) [ 44 ]
T+M
0.326
0.358
0.368
0.317
0.450
0.292
Gen. Omnimatte (CVPR 2025) [ 13 ]
T+M
0.285
0.350
0.359
0.376
0.404
0.260
ProPainter (ICCV 2023) [ 41 ]
M
0.253
0.248
0.341
0.275
0.301
0.249
Table 6: Evaluation using the CFD metric on BeyondMasks across causal after-effect categories. Higher values indicate better performance. Type denotes the supervision modality: T indicates text-instruction-based editing, M indicates mask-guided removal, and T+M denotes methods supporting both inputs. Bold numbers indicate the best score in each column. Bold numbers indicate the best score in each column.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
CORE-OS ↑
CORE-AES ↑
Lucy Edit (arXiv) [ 23 ]
18.8356 / 18.3282
0.7529 / 0.6942
0.2228 / 0.3285
1.069 / 1.813
1.276 / 1.824
VACE (ICCV 2025) [ 11 ]
18.9362 / 18.6913
0.7547 / 0.7236
0.2209 / 0.3010
1.415 / 2.176
1.404 / 1.989
CoCoCo (AAAI 2025) [ 44 ]
21.6499 / 20.1249
0.7625 / 0.6950
0.1962 / 0.2809
1.670 / 2.297
1.875 / 2.209
Gen. Omnimatte (CVPR 2025) [ 13 ]
25.0218 / 22.6120
0.8561 / 0.7794
0.1369 / 0.2562
3.865 / 3.670
2.966 / 3.055
ProPainter (ICCV 2023) [ 41 ]
24.6675 / 22.6555
0.8816 / 0.8060
0.1202 / 0.2044
3.326 / 2.800
2.371 / 2.361
DiffuEraser (arXiv) [ 15 ]
25.3617 / 25.1622
0.8932 / 0.8621
0.0942 / 0.1290
3.640 / 3.344
2.461 / 2.794
Table 7: Split-wise quantitative evaluation on BeyondMasks. Performance of representative video object removal and image object removal methods on the synthetic and real subsets of the dataset. Each entry reports Synthetic / Real results for reconstruction metrics (PSNR, SSIM, LPIPS) and the proposed CORE evaluation scores (CORE-OS and CORE-AES), which are measured on a discrete 1–5 scale. Image-based methods (OmniPaint and Object Clear) are applied frame-wise by processing one frame every 15 frames. Bold indicates the best value for each metric across all methods.
Metric
ObjectScore
AfterEffectScore
PSNR
0.15
0.28
SSIM
-0.14
-0.10
LPIPS
-0.02
0.10
CORE Gemini
0.62
0.73
CORE GPT
0.60
0.76
Inter Rater
0.69
0.79
Table 8: Correlation with human evaluation. Spearman correlation between automatic metrics and human ratings for ObjectScore and AfterEffectScore. Pixel-based metrics (PSNR, SSIM, LPIPS) are compared with VLM-based CORE evaluations using two judge models (Gemini and GPT). The Inter Rater row reports agreement between human annotators, serving as a reference level for achievable alignment.
Figure 8: Failure cases in real-world videos. For each example, the first row shows frames from the input video and the second row shows the output of a representative mask-based removal method. Although the target object is largely removed, residual object-induced effects remain visible. Red dashed boxes highlight typical artifacts such as persistent shadows, reflections in mirrors or glass surfaces, steam or smoke traces, and lighting inconsistencies that reveal incomplete removal of object-environment interactions.
Figure 9: Failure cases in synthetic scenes. The first row shows input frames and the second row shows the corresponding outputs after object removal. Despite successful removal of the foreground object, the edited results often retain interaction effects such as fence shadows, motion-induced dust trails, illumination artifacts, or water disturbances. Red dashed boxes indicate regions where the model reconstructs a visually plausible background but fails to remove the full causal footprint of the object.
Existing video object removal methods excel at inpainting content "behind" the object and correcting appearance-level artifacts such as shadows and reflections. However, when the removed object has more significant interactions, such as collisions with other objects, current models fail to correct them and produce implausible results. We present VOID, a video object removal framework designed to perform physically-plausible inpainting in these complex scenarios. To train the model, we generate a new paired dataset of counterfactual object removals using Kubric and HUMOTO, where removing an object requires altering downstream physical interactions. During inference, a vision-language model identifies regions of the scene affected by the removed object. These regions are then used to guide a video diffusion model that generates physically consistent counterfactual outcomes. Experiments on both synthetic and real data show that our approach better preserves consistent scene dynamics after object removal compared to prior video object removal methods. We hope this framework sheds light on how to make video editing models better simulators of the world through high-level causal reasoning.
Saman Motamed, William Harvey, Benjamin Klein +3
Netflix · INSAIT, Sofia University “St. Kliment Ohridski”
Video object removal aims to erase target objects while reconstructing visually plausible and temporally coherent content. However, target objects often induce shadows, reflections, illumination changes and other effects that extend beyond the provided mask, making conventional mask-conditioned completion prone to visible residuals. We therefore formulate side-effect-aware object removal as an understanding-guided process that integrates object--effect relations, affected-region localization, and context-aware reconstruction. Specifically, we introduce Object-Induced Relation Distillation to transfer token-level object--effect relations from a pretrained vision foundation model to the video diffusion model. We then design Object-aware Framewise Context Cross-Attention to combine target-object semantics with per-frame background context for removal and reconstruction, and propose Attention-guided Region Localization to derive a soft spatial prior over the target object and its affected regions. Extensive experiments across multiple benchmarks demonstrate that our method achieves more complete object-and-effect removal and outperforms existing approaches in removal quality, side-effect suppression, and temporal consistency.
Despite its crucial role in video object removal (VOR), existing evaluation paradigms face two critical limitations: questionable references and a misalignment between tradi- tional metrics and human preference. To address these challenges, we introduce VOR- Bench, which advances VOR evaluation through three integrated components. First, we present the VOR Dataset (VORD), the first benchmark dataset providing both paired edited videos and graffiti masks. Its unique strength lies in a diverse data spectrum, which encompasses model-generated, tool-rendered, and camera-captured data, ensuring robust assessment across real-world scenarios. Second, we develop rMPAF, a realistic Motion- capable Paired-video Acquisition Framework. By combining the strengths of image- based object removal and fine-tuned video generation models, rMPAF automatically generates realistic, motion-coherent paired videos. Finally, we propose three evaluation dimensions and introduce VOR-MDSM, the first perception-driven VLM-based scoring model specifically designed for mask-guided VOR. It bridges the gap between arithmetic metrics and human perception by covering the essential visual attributes and matching nuanced human judgment. Extensive experiments demonstrate that VOR-Bench yields evaluation results that align closely with human perception, achieving a remarkable cor- relation (\r{ho} > 0.9) with subjective assessments. We will release VOR-Bench along with its documentation to ensure full reproducibility.
Haonan Huang, Tianrui Qiu, Xianghao Zang +10
Beijing University of Posts and Telecommunications · China Telecom Artificial Intelligence Technology Co. Ltd · State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University