Real-VAS: a Real-World Video Amodal Segmentation Dataset
Organizations: Department of Computer Science, University of York, York, United Kingdom
Abstract
Amodal video object segmentation is fundamentally limited by the absence of datasets that combine real-world complexity with precise ground-truth annotations. To address this, we present Real Video Amodal Segmentation (Real-VAS), a new large-scale, zero-shot evaluation dataset built from composited real-image video. We introduce a novel data generation pipeline that enables the creation of programmatically consistent amodal ground truth for real-world footage, in which the object mask, occlusion boundary, and composited image are all derived from a single unoccluded source, without relying on human estimation or expensive 3D reconstruction. Our dataset is structured into two challenging scenarios: dynamic Occlusion, created by compositing two clips of moving objects, and a unique Container category featuring complex, physically constrained interactions. These container scenarios--On Surface Containment, Articulated Containment, and Mobile Containment--allow us to generate precise ground truth by simulating an object's motion based on its container's tracked transformation. As a result, Real-VAS provides a diverse and challenging benchmark for evaluating amodal segmentation models on realistic video with the precision of synthetic data. The dataset and generation code are publicly available at https://finlay-hudson.github.io/real_vas
Figures & tables
| Real Datasets | Synthetic Datasets | ||||||||
| Real-VAS | COCOA | COCOA-cls | D2S | MOVi-MC-AC | SAIL-VOS 3D | SAIL-VOS | DYCE | ||
| Metric | Ours | [ 59 ] | [ 14 ] | [ 13 ] | [ 33 ] | [ 19 ] | [ 20 ] | [ 11 ] | |
| Image or Video | Video | Image | Image | Image | Video | Video | Video | Image | |
| Accurate Annotations | ✓ | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ | |
| Scene Images | 14,677 | 5,073 | 3,499 | 5,600 | 293,904 | 237,611 | 111,654 | 5,500 | |
| Unique Objects | 160 | - | 80 | 60 | 1,033 | 178 | 162 | 79 | |
| Rubric | Rubric | Rubric | |
| Office | Cup Games | DAV/YTB | |
| TCOW | 69.4 | 38.3 | 52.8 |
| SAM2 | 72.7 | 55.8 | 68.9 |
| SAM2 | TABE | Diffusion VAS | |
| mIoU | 0.621 | 0.542 | 0.655 |
| 0.066 | 0.207 | 0.262 |
| Method | Inputs | Run Type | |||
| pix2gestalt [ 35 ] | Modal Seg. | Online | 0.411 | 0.060 | 0.136 |
| TCOW [ 45 ] | / | Offline | 0.426 | 0.179 | 0.165 |
| SDAmodal [ 51 ] | Modal Seg. & GT BBox | Online | 0.526 | 0.059 | 0.146 |
| Generative Omnimatte [ 25 ] | Modal Seg. | Offline | 0.416 | 0.045 | 0.062 |
| TABE [ 21 ] | Modal Seg. | Offline | 0.501 | 0.209 | 0.207 |
| PCNet [ 52 ] | Modal Seg. & GT BBox | Online | 0.452 | 0.030 | 0.011 |
| Real-VAS Containment | Real-VAS Occlusion | ||||||
| Method | |||||||
| pix2gestalt [ 35 ] | 0.203 | 0.016 | 0.053 | 0.478 | 0.096 | 0.163 | |
| TCOW [ 45 ] | 0.325 | 0.108 | 0.128 | 0.458 | 0.236 | 0.177 | |
| SDAmodal [ 51 ] | 0.335 | 0.022 | 0.076 | 0.587 | 0.088 | 0.168 | |
| Generative Omnimatte [ 25 ] | 0.250 | 0.000 | 0.003 | 0.470 | 0.082 | 0.081 | |
| TABE [ 21 ] | 0.225 | 0.044 | 0.118 | 0.590 | 0.340 | 0.236 | |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Variant | TCOW | Diffusion VAS |
| Real-VAS (full) | 0.426 | 0.604 |
| w/o Generative Omnimatte | 0.524 | 0.619 |
| w/o camera motion | 0.547 | 0.613 |
| w/o both | 0.560 | 0.628 |
| TAO-Amodal (Top-1) | Real-VAS | ||||
| AP 25 | AP 50 | AP 75 | |||
| Baseline | 0.973 | 0.893 | 0.720 | 0.628 | 0.288 |
| Fine-tuned | 0.974 | 0.899 | 0.742 | 0.652 | 0.293 |
| Clip Length (frames) | Frames Per Second | Resolution | ||||||
| Min | Max | Avg | Min | Max | Avg | Min | Max | Mode |
| 3 | 132 | 37 | 10 | 60 | 16 | |||
| Num. Occlusion Clips | Num. Containment Clips | |||
| Total | On Surface | Articulated | Mobile | |
| 300 | 100 | 38 | 33 | 29 |