VidEoMT: Your ViT is Secretly Also a Video Segmentation Model

Narges Norouzi; Idil Esen Zulfikar; Niccolò Cavagnero; Tommie Kerssies; Bastian Leibe; Gijs Dubbelman; Daan de Geus

cs.CVFeb 19, 2026

VidEoMT: Your ViT is Secretly Also a Video Segmentation Model

Authors: Narges Norouzi, Idil Esen Zulfikar, Niccolò Cavagnero, Tommie Kerssies, Bastian Leibe, Gijs Dubbelman, Daan de Geus

Organizations: Eindhoven University of Technology · RWTH Aachen University

Abstract

Existing online video segmentation models typically combine a per-frame segmenter with complex specialized tracking modules. While effective, these modules introduce significant architectural complexity and computational overhead. Recent studies suggest that plain Vision Transformer (ViT) encoders, when scaled with sufficient capacity and large-scale pre-training, can conduct accurate image segmentation without requiring specialized modules. Motivated by this observation, we propose the Video Encoder-only Mask Transformer (VidEoMT), a simple encoder-only video segmentation model that eliminates the need for dedicated tracking modules. To enable temporal modeling in an encoder-only ViT, VidEoMT introduces a lightweight query propagation mechanism that carries information across frames by reusing queries from the previous frame. To balance this with adaptability to new content, it employs a query fusion strategy that combines the propagated queries with a set of temporally-agnostic learned queries. As a result, VidEoMT attains the benefits of a tracker without added complexity, achieving competitive accuracy while being 5x-10x faster, running at up to 160 FPS with a ViT-L backbone. Code: https://www.tue-mps.org/videomt/

Figures & tables

Panel of Figure 1 — Figure 1: CAVIS vs. VidEoMT (Ours). VidEoMT is much faster than both CAVIS [ 21 ] and a combination of EoMT [ 19 ] and CAVIS, while maintaining competitive AP across different sizes of DINOv2 [ 30 ] . Evaluated on YouTube-VIS 2019 val [ 38 ] .

Figure 2: Current State-of-the-Art Video Segmentation Methods vs. VidEoMT (Ours). We compare the architectures of current state-of-the-art video segmentation methods – using CAVIS [ 21 ] as a representative example – and our encoder-only VidEoMT method. VidEoMT streamlines the video segmentation framework, relying on the power of large-scale pre-training with vision foundation models rather than handcrafted task-specific components. TF means Transformer and CA means context-aware .

Figure 3: VidEoMT architecture. For the initial video frame at $t=0$ , learnable queries are concatenated to the patch tokens after the first $L_{1}$ ViT blocks. Both sets of tokens are then jointly processed in the last $L_{2}$ blocks, outputting predictions and track queries. For successive frames, learnable queries and previous-frame track queries are fed to the query fusion module before being processed by the ViT blocks.

Panel of Table 1 — Table 1: From CAVIS to VidEoMT. Stepwise removal of CAVIS modules toward EoMT, and modifications extending it to our VidEoMT. Evaluated on YouTube-VIS 2019 val [ 38 ] .

Method	Backbone	Pre-training	YouTube-VIS 2019 val [ 38 ]					YouTube-VIS 2021 val [ 38 ]
Method	Backbone	Pre-training	AP	AP 75	AR 10	GFLOPs	FPS	AP	AP 75	AR 10	GFLOPs	FPS
MinVIS [ 17 ]	Swin-L	IN21K	61.6	68.6	66.6	401	29	55.3	62.0	60.8	255	30
DVIS [ 39 ]	Swin-L	IN21K	63.9	70.4	69.0	411	23	58.7	66.6	64.6	405	24
DVIS-DAQ [ 41 ]	Swin-L	IN21K	65.7	73.6	70.7	415	13	61.1	68.2	66.6	410	11
DVIS++ [ 40 ]	ViT-L	DINOv2	67.7	75.3	73.7	846	18	62.3	70.2	68.0	830	17
DVIS-DAQ [ 41 ]	ViT-L	DINOv2	68.3	76.1	73.5	851	10	62.4	70.8	68.0	836	10

Table 2: Online VIS on YouTube-VIS 2019 and 2021.

Method	Backbone	Pre-training	YouTube-VIS 2022 val [ 38 ]					OVIS val [ 31 ]
Method	Backbone	Pre-training	AP ${}^{\text{L}}$	AP ${}^{\text{L}}_{\text{75}}$	AR ${}^{\text{L}}_{\text{10}}$	GFLOPs	FPS	AP	AP 75	AR 10	GFLOPs	FPS
MinVIS [ 17 ]	Swin-L	IN21K	33.1	33.7	36.6	224	31	39.4	41.3	43.3	408	30
DVIS [ 39 ]	Swin-L	IN21K	39.9	42.6	44.9	401	23	45.9	48.3	51.5	419	24
DVIS-DAQ [ 41 ]	Swin-L	IN21K	–	–	–	–	–	49.5	51.7	54.9	423	12
DVIS++ [ 40 ]	ViT-L	DINOv2	37.5	39.4	43.5	820	18	49.6	55.0	54.6	868	17
CAVIS [ 21 ]	ViT-L	DINOv2	39.5	40.5	44.9	815	15	53.2	59.1	58.2	863	15

Table 3: Online VIS on YouTube-VIS 2022 and OVIS. † Input resolution of 544 (shortest image side) for OVIS, default for others.

Method	Backbone	Pre-training	VIPSeg val [ 27 ]
Method	Backbone	Pre-training	VPQ	STQ	GFLOPs	FPS
DVIS [ 39 ]	Swin-L	IN21K	54.7	47.7	879	20
DVIS++ [ 40 ]	ViT-L	DINOv2	56.0	49.8	2290	13
CAVIS [ 21 ]	ViT-L	DINOv2	56.9	51.0	2612	10
DVIS-DAQ [ 41 ]	ViT-L	DINOv2	57.4	52.0	2315	4
VidEoMT	ViT-L	DINOv2	55.2	48.9	1897	75

Table 4: Online VPS on VIPSeg.

Method	Backbone	Pre-training	VSPW val [ 28 ]
Method	Backbone	Pre-training	mVC 16	mIoU	GFLOPs	FPS
DVIS [ 39 ]	Swin-L	IN21K	94.3	61.3	879	22
DVIS++ [ 40 ]	ViT-L	DINOv2	94.2	62.8	2290	13
VidEoMT	ViT-L	DINOv2	95.0	64.9	1909	73

Table 5: Online VSS on VSPW.

Segmenter	Tracker	AP	AP 75	AR 10	GFLOPs	FPS
EoMT	CAVIS [ 21 ]	68.1	76.3	73.6	699	42
EoMT	DVIS++ [ 40 ]	67.0	73.8	73.2	683	69
EoMT	DVIS-DAQ [ 41 ]	67.3	74.8	73.4	703	28
VidEoMT	–	68.6	75.6	73.9	566	160

Table 6: Alternative approaches: EoMT as a segmenter. Comparison of EoMT equipped with state-of-the-art trackers and our proposed VidEoMT. Evaluated on YouTube-VIS 2019 val .

Encoder	Decoder	Temporal Modeling	AP	GFLOPs	FPS
ViT-Adapter	M2F [ 8 ]	TrackFormer [ 25 ]	67.8	739	22
ViT-Adapter	M2F [ 8 ]	Query Fusion	68.0	718	32
VidEoMT	–	Query Fusion	68.6	566	160

Table 7: Alternative approaches: Query propagation in the decoder. Comparison of ViT-Adapter + Mask2Former (M2F) equipped with TrackFormer or our query fusion strategy and the proposed VidEoMT. All methods use a ViT-L backbone with DINOv2 pre-training. Evaluated on YouTube-VIS 2019 val .

Model	Pre-training	AP	GFLOPs	FPS
CAVIS	DINOv3	$\downarrow$ 0.0 68.8	838	13
VidEoMT	DINOv3	$\downarrow$ 0.0 68.9 $\uparrow$ 0.1	566	133
CAVIS	DINOv2	$\downarrow$ 0.0 68.9	838	15
VidEoMT	DINOv2	$\downarrow$ 0.0 68.6 $\downarrow$ 0.3	566	160
CAVIS	EVA-02	$\downarrow$ 0.0 68.0	835	12
VidEoMT	EVA-02	$\downarrow$ 0.0 67.8 $\downarrow$ 0.2	564	119

Table 8: Impact of pre-training. VidEoMT performs better with larger-scale pre-training. Evaluated on YouTube-VIS 2019 val .

Method	Size	AP	Params	GFLOPs	FPS
CAVIS	L	$\downarrow$ 0.0 68.9	358M	838	15
VidEoMT	L	$\downarrow$ 0.0 68.6 $\downarrow$ 0.3	318M	566	160
CAVIS	B	$\downarrow$ 0.0 59.5	131M	390	18
VidEoMT	B	$\downarrow$ 0.0 58.2 $\downarrow$ 1.3	95M	182	251
CAVIS	S	$\downarrow$ 0.0 55.5	57M	251	19
VidEoMT	S	$\downarrow$ 0.0 52.8 $\downarrow$ 2.7	25M	56	294

Table 9: Impact of model size. VidEoMT performs better as ViT [ 13 ] size increases. Evaluated on YouTube-VIS 2019 val .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 13 — Figure A : Removing specialized components. This figure visualizes the step-by-step removal of complex, specialized components from the CAVIS [ 21 ] model, as reported in the results in Tab. 6 of the main manuscript.

Method	YouTube-VIS 2019 val [ 38 ]
Method	AP	GFLOPs	FPS
No propagation	61.3	565	162
Propagation only	63.9	565	162
Non-object reset	67.8	565	157
TrackFormer	67.7	571	117
Fusion $\;\Rightarrow\;$ VidEoMT	68.6	566	160

Appendix

Table A : Query propagation methods. Comparison of alternative strategies for temporal propagation.

Method	Size	AP	Params	GFLOPs	FPS
CAVIS		68.9	358M	838	15
EoMT + CAVIS	L	68.1	328M	699	42
VidEoMT		68.6	318M	566	160
CAVIS		59.5	131M	390	18
EoMT + CAVIS	B	57.4	103M	284	67
VidEoMT		58.2	95M	182	251

Appendix

Table B : Impact of model size. VidEoMT performs better as ViT [ 13 ] size increases. Evaluated on YouTube-VIS 2019 val .

Panel of Figure 16 — Figure B : Qualitative results for video instance segmentation. We compare CAVIS [ 21 ] to VidEoMT on the YouTube-VIS 2019 dataset [ 38 ] .

Panel of Figure 17 — Figure C : Qualitative results for video instance segmentation. We compare CAVIS [ 21 ] to VidEoMT on the OVIS dataset [ 31 ] .

Panel of Figure 18 — Figure D : Qualitative results for video panoptic segmentation. We compare CAVIS [ 21 ] to VidEoMT on the VIPSeg dataset [ 27 ] .

Explore similar work

PMT: Plain Mask Transformer for Image and Video Segmentation with Frozen Vision Encoders
Mar 26, 2026Niccolò Cavagnero, Narges Norouzi, Gijs Dubbelman +1Image Segmentation Video Object Segmentation
Eindhoven University of Technology
LVMT: Video Mask Transformer for Long-term Video Segmentation
Sep 28, 2026Narges Norouzi, Niccolò Cavagnero, Idil Esen Zulfikar +3Video Object Segmentation
Eindhoven University of Technology · RWTH Aachen University
TrajTok: Learning Trajectory Tokens enables better Video Understanding
Feb 26, 2026Chenhao Zheng, Jieyu Zhang, Jianing Zhang +6Visual Tokenization Video Representation Learning
University of Washington · Allen Institute for Artificial Intelligence · Woven by Toyota, Inc +1

Mar 26, 2026J/K move · Enter open · S save

PMT: Plain Mask Transformer for Image and Video Segmentation with Frozen Vision Encoders

Niccolò Cavagnero, Narges Norouzi, Gijs Dubbelman, Daan de Geus

Eindhoven University of Technology

Image Segmentation Video Object Segmentation Vision Foundation Model Adaptation Efficient ViTs

Open paper AskPDF arXiv

Vision Foundation Models (VFMs) pre-trained at scale enable a single frozen encoder to serve multiple downstream tasks simultaneously. Recent VFM-based encoder-only models for image and video segmentation, such as EoMT and VidEoMT, achieve competitive accuracy with remarkably low latency, yet they require finetuning the encoder, sacrificing the multi-task encoder sharing that makes VFMs practically attractive for large-scale deployment. To reconcile encoder-only simplicity and speed with frozen VFM features, we propose the Plain Mask Decoder (PMD), a fast Transformer-based segmentation decoder that operates on top of frozen VFM features. The resulting model, the Plain Mask Transformer (PMT), preserves the architectural simplicity and low latency of encoder-only designs while keeping the encoder representation unchanged and shareable. The design seamlessly applies to both image and video segmentation, inheriting the generality of the encoder-only framework. On standard image segmentation benchmarks, PMT matches the frozen-encoder state of the art while running up to ~3x faster. For video segmentation, it even performs on par with fully finetuned methods, while being up to 8x faster than state-of-the-art frozen-encoder models. Code: https://github.com/tue-mps/pmt.

Abstract

Explore similar work

PMT: Plain Mask Transformer for Image and Video Segmentation with Frozen Vision Encoders

LVMT: Video Mask Transformer for Long-term Video Segmentation

TrajTok: Learning Trajectory Tokens enables better Video Understanding