Unlocking Cross-Attention in Label-Scarce Regimes: A Weakly Supervised Approach for Video Semantic Segmentation
Organizations: DEIB, Politecnico di Milano, Italy · Bocconi University, Italy
Abstract
Weakly Supervised Video Semantic Segmentation (WS-VSS) aims to learn temporally consistent semantic object masks in videos using only video-level labels. While fully supervised models successfully learn temporal relationships by means of cross-attention, training these temporal weights under weak supervision is unstable and even pushes the model below its single-frame baseline. Therefore, existing WS-VSS methods exploit temporal consistency by relying on external priors, such as optical flow. Unfortunately these cues are non-semantic, computationally expensive and noisy, degrading the supervisory signal. Instead of supervising attention weights directly or using external priors, we propose MemCAM, a WS-VSS transformer framework that learns temporal cross-attention directly through feature-space self-supervision. MemCAM introduces a CAM Propagation Algorithm that synthesizes a supervisory signal by propagating past CAMs according to patch-level feature similarities in the latent space. Specifically, we constrain their induced CAMs through a consistency objective between predicted and propagated maps, stabilizing temporal attention learning. Experiments on the YouTube-Objects v2.2 and WS datasets demonstrate that MemCAM consistently outperforms the temporal localization and segmentation performance of WS-VSS alternatives, showing that cross-attention can be effectively trained in label-scarce video settings through internal self-supervision.
Figures & tables
| Memory (M) | CAM Loss ( ) | Frozen (F) | Test CorLoc (%) | Params (M) | Time (ms) |
| ✗ | ✗ | ✗ | 68.6 | 5.7 | 10.9 |
| ✓ | ✗ | ✗ | 67.7 | 6.2 | 11.2 |
| ✗ | ✓ | ✗ | 74.8 | 5.7 | 12.9 |
| ✓ | ✓ | ✗ | 76.8 | 6.2 | 13.3 |
| ✓ | ✓ | ✓ | 70.7 | 6.2 | 13.3 |
| Method | CorLoc | Params | Time |
|---|---|---|---|
| Static, image-based | |||
| CAM [ 43 ] | 51.5 | 23.5 | 0.2 |
| GradCAM [ 32 ] | 54.5 | 23.5 | 27.8 |
| GradCAM++ [ 9 ] | 61.2 | 23.5 | 28.0 |
| Smooth-GradCAM++ [ 29 ] | 59.5 | 23.5 | 136.2 |
| XGradCAM [ 15 ] | 54.6 | 23.5 | 14.2 |
| Method | mIoU | Params | Time |
|---|---|---|---|
| Static, image-based | |||
| GradCAM [ 32 ] | 19.86 | 23.5 | 27.8 |
| GradCAM++ [ 9 ] | 16.40 | 23.5 | 28.0 |
| LayerCAM [ 19 ] | 17.69 | 23.5 | 17.9 |
| PuzzleCAM [ 20 ] | 33.52 | 24.8 | 45.7 |
| MCTformer+-tiny [ 38 ] | 36.90 | 5.7 | 10.9 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Weakly Supervised Semantic Segmentation
Weakly supervised semantic segmentation learns pixel-level class masks from image-level labels, points, scribbles, or other sparse annotations.
See all 36 papers