PMT: Plain Mask Transformer for Image and Video Segmentation with Frozen Vision Encoders

Niccolò Cavagnero; Narges Norouzi; Gijs Dubbelman; Daan de Geus

cs.CVMar 26, 2026

PMT: Plain Mask Transformer for Image and Video Segmentation with Frozen Vision Encoders

Authors: Niccolò Cavagnero, Narges Norouzi, Gijs Dubbelman, Daan de Geus

Organizations: Eindhoven University of Technology

Abstract

Vision Foundation Models (VFMs) pre-trained at scale enable a single frozen encoder to serve multiple downstream tasks simultaneously. Recent VFM-based encoder-only models for image and video segmentation, such as EoMT and VidEoMT, achieve competitive accuracy with remarkably low latency, yet they require finetuning the encoder, sacrificing the multi-task encoder sharing that makes VFMs practically attractive for large-scale deployment. To reconcile encoder-only simplicity and speed with frozen VFM features, we propose the Plain Mask Decoder (PMD), a fast Transformer-based segmentation decoder that operates on top of frozen VFM features. The resulting model, the Plain Mask Transformer (PMT), preserves the architectural simplicity and low latency of encoder-only designs while keeping the encoder representation unchanged and shareable. The design seamlessly applies to both image and video segmentation, inheriting the generality of the encoder-only framework. On standard image segmentation benchmarks, PMT matches the frozen-encoder state of the art while running up to ~3x faster. For video segmentation, it even performs on par with fully finetuned methods, while being up to 8x faster than state-of-the-art frozen-encoder models. Code: https://github.com/tue-mps/pmt.

Figures & tables

Panel of Figure 1 — Figure 1: ViT-Adapter + Mask2Former vs . PMT (Ours). PMT exhibits a better trade-off between Panoptic Quality and FPS across different sizes of frozen DINOv3 [ 41 ] pre-trained ViTs [ 14 ] . Evaluated on COCO val2017 [ 25 ] , see Tab. 4 .

Figure 2: Plain Mask Transformer (PMT) Architecture. Instead of injecting the query tokens within the ViT encoder as in the encoder-only framework of EoMT and VidEoMT, we extract features at multiple encoder levels and feed them into an efficient segmentation decoder that processes queries and patch tokens in parallel. $\oplus$ denotes element-wise addition. $\odot$ denotes the dot product.

Panel of Table 1 — Table 1: From ViT-Adapter + Mask2Former to EoMT. Stepwise removal of Mask2Former modules toward EoMT with frozen DINOv3 [ 41 ] encoder. Evaluated on COCO val2017 [ 25 ] .

Panel of Table 2 — Table 2: From EoMT to PMT. Stepwise addition of modules from EoMT with frozen DINOv3 [ 41 ] and DINOv2 [ 39 ] encoders to our PMT. Evaluated on COCO val2017 [ 25 ] .

Model	Pre-train	Params	GFLOPs	FPS	PQ
ViT-Adapter + M2F	DINOv3	340M	804	48	56.4
PMT (Ours)		357M	767	141	56.1
ViT-Adapter + M2F	DINOv2	342M	804	49	55.7
PMT (Ours)		359M	767	146	55.5
ViT-Adapter + M2F	IN21K	342M	804	49	51.0
PMT (Ours)		359M	767	146	49.7

Table 3: Pre-training. Impact of pre-training on segmentation quality. Evaluated on COCO val2017 [ 25 ] .

Model	Size	Params	GFLOPs	FPS	PQ
ViT-Adapter + M2F	L	340M	804	48	56.4
PMT (Ours)		357M	767	141	56.1
ViT-Adapter + M2F	B	116M	334	64	53.6
PMT (Ours)		116M	281	262	52.8
ViT-Adapter + M2F	S	45M	163	73	49.6
PMT (Ours)		29M	91	400	45.3

Table 4: Model Size. Impact of model size on segmentation quality and inference speed. Evaluated on COCO val2017 [ 25 ] .

Blocks	Params	GFLOPs	FPS	PQ
2	332M	695	153	54.0
4	344M	731	146	55.6
6	357M	767	141	56.1
8	370M	803	133	56.1

Table 5: Decoder Layers. Impact of number of decoder layers on segmentation quality and inference speed. Evaluated on COCO val2017 [ 25 ] .

Method	Backbone	Pre-training	Input size	Params	GFLOPs	FPS	PQ	AP
Mask2Former † [ 11 ]	Swin-L [ 26 ]	IN21K	$800^{2}$	216M	868	24	57.8	50.1
kMaX-DeepLab [ 49 ]	ConvNext-L [ 27 ]	IN21K	$1281^{2}$	232M	–	–	58.0	–
OneFormer † [ 19 ]	DiNAT-L [ 15 ]	IN21K	$800^{2}$	223M	736	20	58.0	49.2
MaskDINO † [ 24 ]	Swin-L [ 26 ]	IN21K	$800^{2}$	223M	1326	14	58.3	52.3
EoMT [ 20 ]	ViT-L [ 14 ]	DINOv2	$640^{2}$	316M	669	172	56.0	45.2
EoMT [ 20 ]	ViT-L [ 14 ]	DINOv3	$640^{2}$	315M	668	162	56.8	45.9

Table 6: PMT for Panoptic and Instance Segmentation on COCO val2017 [ 25 ] . † During inference, these models resize the shortest side of images to the indicated scale, while preserving the aspect ratio.

Method	Backbone	Pre-training	Encoder	Input size	Params	GFLOPs	FPS	mIoU
Mask2Former † [ 11 ]	Swin-L [ 26 ]	IN21K		$640^{2}$	216M	–	33	56.1
MaskDINO † [ 24 ]	Swin-L [ 26 ]	IN21K		$640^{2}$	223M	–	–	56.6
OneFormer † [ 19 ]	ConvNext-XL [ 27 ]	IN21K		$640^{2}$	373M	607	21	57.4
OneFormer † [ 19 ]	DiNAT-L [ 15 ]	IN21K		$896^{2}$	223M	678	19	58.1
EoMT [ 20 ]	ViT-L [ 14 ]	DINOv2		$512^{2}$	316M	721	154	58.4
EoMT [ 20 ]	ViT-L [ 14 ]	DINOv3		$512^{2}$	315M	721	151	59.5

Table 7: PMT for Semantic Segmentation on ADE20K [ 52 ] . † These models resize the shortest side of images to the indicated scale during inference, while preserving the aspect ratio.

Method	Backbone	Pre-training	Encoder	YouTube-VIS 2019 val [ 47 ]					YouTube-VIS 2021 val [ 47 ]
Method	Backbone	Pre-training	Encoder	GFLOPs	FPS	AP	AP 75	AR 10	GFLOPs	FPS	AP	AP 75	AR 10
MinVIS [ 17 ]	Swin-L [ 26 ]	IN21K		401	29	61.6	68.6	66.6	255	30	55.3	62.0	60.8
DVIS [ 50 ]	Swin-L [ 26 ]	IN21K		411	23	63.9	70.4	69.0	405	24	58.7	66.6	64.6
DVIS-DAQ [ 53 ]	Swin-L [ 26 ]	IN21K		415	13	65.7	73.6	70.7	410	11	61.1	68.2	66.6
DVIS++ [ 51 ]	ViT-Adapter-L [ 9 ]	DINOv2		846	18	67.7	75.3	73.7	830	17	62.3	70.2	68.0
DVIS-DAQ [ 53 ]	ViT-Adapter-L [ 9 ]	DINOv2		851	10	68.3	76.1	73.5	836	10	62.4	70.8	68.0

Table 8: PMT for Online VIS on YouTube-VIS 2019 and 2021 [ 47 ] .

Method	Backbone	PT	Encoder	VIPSeg val [ 34 ]
Method	Backbone	PT	Encoder	GFLOPs	FPS	VPQ	STQ
DVIS [ 50 ]	Swin-L [ 26 ]	I		879	20	54.7	47.7
DVIS++ [ 51 ]	ViT-Adapter-L [ 9 ]	D2		2290	13	56.0	49.8
DVIS-DAQ [ 53 ]	ViT-Adapter-L [ 9 ]	D2		2315	4	57.4	52.0
CAVIS [ 23 ]	ViT-Adapter-L [ 9 ]	D2		2612	10	56.9	51.0
VidEoMT [ 38 ]	ViT-L [ 14 ]	D2		1897	75	55.2	48.9

Table 9: PMT for Online VPS on VIPSeg [ 21 ] . PT : Pre-training. I : ImageNet-21K [ 13 ] . D2 : DINOv2 [ 39 ] ). D3 : DINOv3 [ 41 ] .

Method	Backbone	PT	Encoder	VSPW val [ 33 ]
Method	Backbone	PT	Encoder	GFLOPs	FPS	mVC 16	mIoU
DVIS [ 50 ]	Swin-L [ 26 ]	I		879	22	94.3	61.3
DVIS++ [ 51 ]	ViT-Adapter-L [ 9 ]	D2		2290	13	94.2	62.8
VidEoMT [ 38 ]	ViT-L [ 14 ]	D2		1909	73	95.0	64.9
VidEoMT [ 38 ]	ViT-L [ 14 ]	D3		1909	71	94.4	64.0
PMT (Ours)	ViT-L [ 14 ]	D2		2049	60	94.6	64.3

Table 10: PMT for Online VSS on VSPW [ 33 ] . PT : Pre-training. I : ImageNet-21K [ 13 ] . D2 : DINOv2 [ 39 ] ). D3 : DINOv3 [ 41 ] .

Explore similar work

VidEoMT: Your ViT is Secretly Also a Video Segmentation Model
Feb 19, 2026Narges Norouzi, Idil Esen Zulfikar, Niccolò Cavagnero +4Video Object Segmentation Vision Transformer
Eindhoven University of Technology · RWTH Aachen University
MPM: Mutual Pair Merging for Efficient Vision Transformers
Apr 7, 2026Simon Ravé, Pejman Rasti, David RousseauVisual Token Merging Efficient ViTs
LARIS University of Angers Angers, France · UMR INRAe-IRHS Angers, France
LUMA: Benchmarking Segmentation via a Lightweight Universal Mask Adapter
Jul 1, 2026Tobias Christian Nauen, Anosh Billimoria, Federico Raue +3Efficient ViTs Semantic Segmentation
RPTU University Kaiserslautern-Landau, Kaiserslautern, Germany · German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany

Feb 19, 2026J/K move · Enter open · S save

VidEoMT: Your ViT is Secretly Also a Video Segmentation Model

Narges Norouzi, Idil Esen Zulfikar, Niccolò Cavagnero, Tommie Kerssies+3

Eindhoven University of Technology · RWTH Aachen University

Video Object Segmentation Vision Transformer

Open paper AskPDF arXiv

Existing online video segmentation models typically combine a per-frame segmenter with complex specialized tracking modules. While effective, these modules introduce significant architectural complexity and computational overhead. Recent studies suggest that plain Vision Transformer (ViT) encoders, when scaled with sufficient capacity and large-scale pre-training, can conduct accurate image segmentation without requiring specialized modules. Motivated by this observation, we propose the Video Encoder-only Mask Transformer (VidEoMT), a simple encoder-only video segmentation model that eliminates the need for dedicated tracking modules. To enable temporal modeling in an encoder-only ViT, VidEoMT introduces a lightweight query propagation mechanism that carries information across frames by reusing queries from the previous frame. To balance this with adaptability to new content, it employs a query fusion strategy that combines the propagated queries with a set of temporally-agnostic learned queries. As a result, VidEoMT attains the benefits of a tracker without added complexity, achieving competitive accuracy while being 5x-10x faster, running at up to 160 FPS with a ViT-L backbone. Code: https://www.tue-mps.org/videomt/

Abstract

Explore similar work

VidEoMT: Your ViT is Secretly Also a Video Segmentation Model

MPM: Mutual Pair Merging for Efficient Vision Transformers

LUMA: Benchmarking Segmentation via a Lightweight Universal Mask Adapter