cs.CVApr 1, 2026
SaveVideo Patch Pruning: Efficient Video Instance Segmentation via Early Token Reduction
Organizations: Institute for Information Processing (tnt), L3S - Leibniz University Hannover, Germany
Abstract
Vision Transformers (ViTs) have demonstrated state-ofthe-art performance in several benchmarks, yet their high computational costs hinders their practical deployment. Patch Pruning offers significant savings, but existing approaches restrict token reduction to deeper layers, leaving early-stage compression unexplored. This limits their potential for holistic efficiency. In this work, we present a novel Video Patch Pruning framework (VPP) that integrates temporal prior knowledge to enable efficient sparsity within early ViT layers. Our approach is motivated by the observation that prior features extracted from deeper layers exhibit strong foreground selectivity. Therefore we propose a fully differentiable module for temporal mapping to accurately select the most relevant patches in early network stages. Notably, the proposed method enables a patch reduction of up to 60% in dense prediction tasks, exceeding the capabilities of conventional image-based patch pruning, which typically operate around a 30% patch sparsity. VPP excels the high-sparsity regime, sustaining remarkable performance even when patch usage is reduced below 55%. Specifically, it preserves stable results with a maximal performance drop of 0.6% on the Youtube-VIS 2021 dataset.
Figures & tables
Figure 1: Pruning masks for image-based (SViT) and video-based (VPP) pruning at 60% avg. patch reduction. For instance segmentation, only the highlighted objects must be segmented, while the background can be ignored. VPP uses temporal information for early-stage pruning, effectively preserving the foreground.
| Method | pruning before layer 3 | pruning- stages | additional prune param | dynamic pruning |
| DynamicViT [ 24 ] | ✗ | 3 | 0.723 M | ✗ |
| TPS [ 36 ] | ✗ | 3 | 0.723 M | ✗ |
| SViT [ 20 ] | ✗ | 9 | 0.341 M | ✓ |
| VPP (ours) | ✓ | 4 | 0.189 M | ✓ |
Table 1: Categorical overview Patch Pruning methods. VPP leverages video context for early feature reduction, ensuring minimal computational overhead. The method also enables adaptive mask sizing to better adapt the mask to varying instance sizes.
Figure 2: Foreground Selectivity (FGS) across layers. This plot shows the ability of feature x to identify patches belonging to an object instance. Initial features (idx: 0) and early blocks (idx: 1-3) significantly lack in Foreground Selectivity.
Figure 3: Mapping-Selective Module (Map-SM). Using the features from preceding frame ( ), Map-SM calculates an association matrix to establish patch correspondences in order to map high-level features onto the current frame. Mask ensures that only the remaining patches from the preceding frame are considered for the mask selection process.
Figure 4: Performance Loss in AP vs. Pruned Patches on Youtube-VIS 2019 and 2021, using model size small.
| model | tiny | small | ||||||||||
| dataset | goal PKR (%) | method | PKR (%) | PKR (%) | AP | |||||||
| YTVIS 21 | 100 | ViT-Adapter [ 3 ] | 100 | 49.1 | 44.8 | 42.2 | - | 100 | 57.3 | 52.3 | 50.4 | - |
| 55 | DynViT [ 24 ] | 54.4 | 44.1 | 40.4 | 36.9 | -5.3 | 54.4 | 53.8 | 48.8 | 45.2 | -5.2 | |
| TPS [ 36 ] | 54.4 | 46.9 | 42.8 | 38.9 | -3,3 | 54.4 | 54.9 | 50.2 | 46.2 | -4.2 | ||
| SViT [ 20 ] | 54.0 | 46.7 | 42.9 | 39.2 | -3.0 | 54.2 | 55.8 | 51.1 | 46.9 | -3.5 | ||
| VPP (ours) | 52.7 | 49.9 | 45.7 | 42.3 | +0.1 | 51.5 | 56.1 | 51.3 | 49.8 | -0.6 | ||
Table 2: Average Precision on YTVIS 19/21 Dataset. The results are grouped into similar sparsity levels, measured by PKR. Instance segmentation metrics (bbox and segm) are reported, independent of tracking ID association. The best scores are highlighted as bold.
| PKr (%) | method | ||||
| 55% | SViT | 73.7% | 77.2% | 68.8% | 65.8% |
| VPP (ours) | 82.3% | 87.3% | 76.2% | 72.2% | |
| 40% | SViT | 62.7% | 67.5% | 56.0% | 52.1% |
| VPP (ours) | 73.4% | 79.5% | 65.9% | 60.7% |
Table 3: Intersection over Instance (IoI) of ViT-Small on YoutubeVIS 2021. Scores are separated by instance size: S mall ( ), M edium , and L arge .
| No. | Map-SM index i | SM idx index i | PKR (%) | AP | Note |
| 1 | - | 0,3,6,9 | 55.9 | 40.1 | |
| 2 | - | 1,3,6,9 | 52.4 | 40.3 | |
| 3 | 0 | 3,6,9 | 51.6 | 41.5 | |
| 4 | 1 | - | 54.2 | 40.6 | const Kr |
| 5 | 1 | 3,6,9 | 52.7 | 42.3 | (default) |
Table 4: Positioning ablation of Map-SM and SM on dataset YoutubeVIS 2021 at PKR=55%. Early patch initialization with Map-SM (no. 5) improves AP by , compared to image-based patch selection (no. 1+2).
Figure 5: Patch density per layer. VPP removes patches after layer 1, while maintaining high patch density within the deepest layers (7-12). In contrast, image-based SViT requires dense initial layers, leaving the deepest layers with insufficient patch density.
Figure 6: Mask adaptation of VPP at scene switches. Patch activity is visualized through shading: less activated patches are darker, and fully black patches are skipped. White, textureless inputs ( ) led to a centrally sampled masks. After the video is enabled at , VPP is able to reallocate the foreground mask within a single timestep ( ). The last column shows the GT-mask as reference.
| method | AP | FLOPS | ||
| ViT-Ada [ 3 ] | 50.4 | 88.0 G | 17.14 | 6.58 |
| SViT [ 20 ] | 46.9 | 59.6 G | 24.87 | 8.19 |
| VPP (ours) | 49.8 | 59.9 G | 25.94 | 8.13 |
Table 5: FLOPS and Throughput, measured in Frames per Second (FPS). All experiments utilize the YTVIS-2021 dataset and involve pruning the ViT-Ada-small model with a 55% PKR.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| name hyperparameter | value |
| pos idx Map-SM | 1 |
| ref-layer Map-SM | 6 |
| pos idx SM | 3,6,9 |
| 10 | |
| scale | 10 |
| scale | 40 |
Table 6: Hyperparameters used in Video Patch Pruning
| PKR (%) | AP | |
| 0.3 | 51.4 | 41.8 |
| 0.5 | 52.7 | 42.3 |
| 0.7 | 54.1 | 39.5 |
Table 7: Evaluation of model performance across different settings for . ViT-Adapter Tiny is trained on Youtube-VIS 2021 at a goal-PKR of 55%. shows the best performance.
Figure 7: Examples for the initial pruning mask , generated by Mapping Selective Module (Map-SM). This mask removes 40% of patches after the first transformer block. All masks demonstrate that patches from foreground objects are sampled more densely, while background regions are effectively pruned during the initial mask generation.
| PKR (%) | dataset Youtube-VIS | method | ||||
| 55% | 2021 | SViT | 73.7% | 77.2% | 68.8% | 65.8% |
| VPP (ours) | 82.3% | 87.3% | 76.2% | 72.2% | ||
| 2019 | SViT | 63.1% | 67.9% | 61.0% | 58.5% | |
| VPP (ours) | 77.3% | 84.4% | 75.0% | 71.4% | ||
| 40% | 2021 | SViT | 62.7% | 67.5% | 56.0% | 52.1% |
| VPP (ours) | 73.4% | 79.5% | 65.9% | 60.7% |
Table 8: Intersection over Instance (IoI) over datasets Youtube-VIS 2019 and 2021. VPP shows superior patch coverage every setting, improving the IoI scores by compared to SViT.
Figure 8: Comparison of token usage ( PKR) for image-based ( SViT ) and video-based ( VPP ) Pruning. Unlike image-based patch pruning, VPP computes foreground features almost fully dense, while sparsely activating background patches for context preservation. For Video Instance Segmentation, this is essential to ensure the model focuses on the instances to be segmented, thereby minimizing redundant background computation.
Figure 9: Exemplary pruning masks for videos 1–4 at a 40% Patch Keep Ratio (PKR). Results are shown for timesteps . Patch activity is represented via grayscale shading, where darker patches are activated less. Fully black patches are removed by Map-SM after the first layer.
Figure 10: Exemplary pruning masks for videos 5-8 at a 40% Patch Keep Ratio (PKR). Results are shown for timesteps . Patch activity is represented via grayscale shading, where darker patches are activated less. Fully black patches are removed by Map-SM after the first layer.