From Ideal to Real: Stable Video Object Removal under Imperfect Conditions
Organizations: MiLM Plus, Xiaomi Inc.
Abstract
Removing objects from videos remains difficult in the presence of real-world imperfections such as shadows, abrupt motion, and defective masks. Existing diffusion-based video inpainting models often struggle to maintain temporal stability and visual consistency under these challenges. We propose Stable Video Object Removal (SVOR), a robust framework that achieves shadow-free, flicker-free, and mask-defect-tolerant removal through three key designs: (1) Mask Union for Stable Erasure (MUSE), a windowed union strategy applied during temporal mask downsampling to preserve all target regions observed within each window, effectively handling abrupt motion and reducing missed removals; (2) Denoising-Aware Segmentation (DA-Seg), a lightweight segmentation head on a decoupled side branch equipped with Denoising-Aware AdaLN and trained with mask degradation to provide an internal diffusion-aware localization prior without affecting content generation; and (3) Curriculum Two-Stage Training: where Stage I performs self-supervised pretraining on unpaired real-background videos with online random masks to learn realistic background and temporal priors, and Stage II refines on synthetic pairs using mask degradation and side-effect-weighted losses, jointly removing objects and their associated shadows/reflections while improving cross-domain robustness. Extensive experiments show that SVOR attains new state-of-the-art results across multiple datasets and degraded-mask benchmarks, advancing video object removal from ideal settings toward real-world applications. Project page: https://xiaomi-research.github.io/svor/.
Figures & tables
| DAVIS | ROSE Bench | RORD-50 | |||||||||||||
| bgPSNR↑ | bgSSIM↑ | TF↓ | ReMove↑ | GPT↑ | PSNR↑ | SSIM↑ | TF↓ | ReMove↑ | GPT↑ | PSNR↑ | SSIM↑ | TF↓ | ReMove↑ | GPT↑ | |
| FuseFormer [ 22 ] | 29.51 | 0.8600 | 0.9921 | 0.8763 | 9.379 | 25.75 | 0.8847 | 0.9921 | 0.9058 | 9.776 | 27.04 | 0.8576 | 0.9961 | 0.9167 | 10.79 |
| FGT [ 40 ] | 30.74 | 0.9025 | 0.9537 | 0.8731 | 9.465 | 26.35 | 0.9059 | 0.9916 | 0.8927 | 10.89 | 27.51 | 0.8804 | 0.9961 | 0.9117 | 10.73 |
| Propainter [ 45 ] | 36.12 | 0.9753 | 0.9529 | 0.8607 | 9.446 | 25.14 | 0.9241 | 0.9903 | 0.8204 | 7.780 | 29.54 | 0.9367 | 0.9958 | 0.9040 | 10.16 |
| DiffuEraser [ 17 ] | 33.76 | 0.9467 | 0.9522 | 0.8626 | 10.25 | 26.83 | 0.9040 | 0.9917 | 0.8837 | 10.72 | 29.84 | 0.9345 | 0.9958 | 0.9126 | 11.53 |
| VACE [ 10 ] | 27.06 | 0.8656 | 0.9471 | 0.7081 | 5.263 | 22.71 | 0.8802 | 0.9913 | 0.7154 | 7.617 | 19.21 | 0.8622 | 0.9856 | 0.6842 | 3.962 |
| Propainter [ 45 ] | gen-omni [ 16 ] | minimax [ 46 ] | ROSE [ 23 ] | Ours | |
|---|---|---|---|---|---|
| Erasure | 0.897 | 0 .904 | 0.926 | 0.934 | 0.978 |
| Completion | 0.228 | 0.588 | 0.412 | 0.353 | 0 .551 |
| Average | 0.563 | 0 .746 | 0.669 | 0.644 | 0.765 |
| Stage I | Stage II | ROSE Bench | |||||||
|---|---|---|---|---|---|---|---|---|---|
| bg_train | Base | MUSE | MaskD | Seg | w-loss | PSNR | SSIM | LPIPS | Remove |
| 25.94 | 0.909 | 0.077 | 0.864 | ||||||
| 27.33 | 0.918 | 0.066 | 0.901 | ||||||
| 27.87 | 0.921 | 0.061 | 0.902 | ||||||
| 28.12 | 0.922 | 0.064 | 0.906 | ||||||
| 28.35 | 0.919 | 0.060 | 0.904 | ||||||
| DAVIS | ROSE Bench | RORD-50 | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bgLPIPS | TC | CFD | LPIPS | bgPSNR | bgSSIM | bgLPIPS | TC | CFD | LPIPS | bgPSNR | bgSSIM | bgLPIPS | TC | CFD | |
| DiffuEraser [ 17 ] | 0.0211 | 0.9676 | 0.4453 | 0.0885 | 28.5813 | 0.9214 | 0.0635 | 0.9870 | 0.3043 | 0.0469 | 31.2622 | 0.9506 | 0.0289 | 0.9878 | 0.3369 |
| gen-omni [ 16 ] | 0.1048 | 0.9746 | 0.4608 | 0.1013 | 28.4689 | 0.9000 | 0.0791 | 0.9901 | 0.3073 | 0.0442 | 31.7696 | 0.9308 | 0.0305 | 0.9973 | 0.3133 |
| minimax [ 46 ] | 0.0628 | 0.9739 | 0.4484 | 0.0917 | 28.0605 | 0.9144 | 0.0678 | 0.9903 | 0.3134 | 0.0507 | 30.3690 | 0.9433 | 0.0318 | 0.9912 | 0.3117 |
| ROSE [ 23 ] | 0.0631 | 0.9683 | 0.4805 | 0.0770 | 32.5895 | 0.9292 | 0.0417 | 0.9894 | 0.3486 | 0.0440 | 32.2127 | 0.9361 | 0.0272 | 0.9915 | 0.3150 |
| Ours | 0.0576 | 0.9746 | 0.4464 | 0.0451 | 33.2219 | 0.9443 | 0.0341 | 0.9902 | 0.3336 | 0.0440 | 32.4764 | 0.9511 | 0.0273 | 0.9931 | 0.2855 |
| Category | Method | PSNR | SSIM | LPIPS | ReMove |
|---|---|---|---|---|---|
| Common | ROSE | 36.60 | 0.952 | 0.041 | 0.859 |
| Ours | 36.37 | 0.952 | 0.022 | 0.853 | |
| Shadow | ROSE | 33.79 | 0.923 | 0.063 | 0.937 |
| Ours | 33.61 | 0.939 | 0.038 | 0.938 | |
| Light | ROSE | 30.07 | 0.921 | 0.086 | 0.900 |
| Ours | 31.16 | 0.932 | 0.041 | 0.896 |
| ROSE Bench | RORD-50 | |||||||
| PSNR | SSIM | LPIPS | Remove | PSNR | SSIM | LPIPS | Remove | |
| raw VACE | 22.71 | 0.8802 | 0.1175 | 0.7154 | 19.21 | 0.8622 | 0.1395 | 0.6842 |
| copy-paste data | 25.78 | 0.9014 | 0.0770 | 0.8547 | 27.39 | 0.9310 | 0.0665 | 0.8420 |
| Background data | 25.94 | 0.9086 | 0.0773 | 0.8640 | 28.99 | 0.9350 | 0.0519 | 0.9135 |
| Only Stage II | 28.68 | 0.9268 | 0.0522 | 0.9071 | 30.02 | 0.9357 | 0.0479 | 0.9163 |
| Stage I + II | 31.47 | 0.9335 | 0.0451 | 0.9082 | 31.26 | 0.9378 | 0.0440 | 0.9179 |
| PSNR | SSIM | LPIPS | Remove | PSNR | SSIM | LPIPS | Remove | PSNR | SSIM | LPIPS | Remove | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| gen-omni | 27.08 | 0.8831 | 0.1013 | 0.8998 | 26.78 | 0.8761 | 0.1151 | 0.9011 | 26.46 | 0.8698 | 0.1251 | 0.9014 |
| gen-omni w/ MUSE | 27.08 | 0.8838 | 0.1014 | 0.9003 | 26.80 | 0.8761 | 0.1152 | 0.9020 | 26.47 | 0.8697 | 0.1250 | 0.9027 |
| minimax | 26.30 | 0.8950 | 0.0917 | 0.8960 | 26.00 | 0.8884 | 0.0988 | 0.8968 | 25.89 | 0.8831 | 0.1042 | 0.9001 |
| minimax w/ MUSE | 26.33 | 0.8958 | 0.0918 | 0.8955 | 26.05 | 0.8885 | 0.0990 | 0.8987 | 25.97 | 0.8833 | 0.1042 | 0.9032 |
| ROSE | 31.12 | 0.9170 | 0.0770 | 0.9081 | 29.81 | 0.9019 | 0.0671 | 0.9060 | 29.08 | 0.8920 | 0.0745 | 0.9016 |
| Mask Drop 0 | Mask Drop 50% | |||||||
|---|---|---|---|---|---|---|---|---|
| PSNR | SSIM | LPIPS | Remove | PSNR | SSIM | LPIPS | Remove | |
| Vanilla Seg | 29.50 | 0.9360 | 0.0478 | 0.9170 | 25.07 | 0.9141 | 0.0835 | 0.8286 |
| DA-Seg | 30.02 | 0.9357 | 0.0479 | 0.9179 | 27.42 | 0.9220 | 0.0708 | 0.8599 |
Video Editing
Video editing uses AI to alter, compose, relight, or reorganize moving footage while controlling how changes persist across frames.
See all 171 papers