PMT: Plain Mask Transformer for Image and Video Segmentation with Frozen Vision Encoders

Niccolò Cavagnero; Narges Norouzi; Gijs Dubbelman; Daan de Geus

cs.CVMar 26, 2026

PMT: Plain Mask Transformer for Image and Video Segmentation with Frozen Vision Encoders

Authors: Niccolò Cavagnero, Narges Norouzi, Gijs Dubbelman, Daan de Geus

Organizations: Eindhoven University of Technology

Abstract

Vision Foundation Models (VFMs) pre-trained at scale enable a single frozen encoder to serve multiple downstream tasks simultaneously. Recent VFM-based encoder-only models for image and video segmentation, such as EoMT and VidEoMT, achieve competitive accuracy with remarkably low latency, yet they require finetuning the encoder, sacrificing the multi-task encoder sharing that makes VFMs practically attractive for large-scale deployment. To reconcile encoder-only simplicity and speed with frozen VFM features, we propose the Plain Mask Decoder (PMD), a fast Transformer-based segmentation decoder that operates on top of frozen VFM features. The resulting model, the Plain Mask Transformer (PMT), preserves the architectural simplicity and low latency of encoder-only designs while keeping the encoder representation unchanged and shareable. The design seamlessly applies to both image and video segmentation, inheriting the generality of the encoder-only framework. On standard image segmentation benchmarks, PMT matches the frozen-encoder state of the art while running up to ~3x faster. For video segmentation, it even performs on par with fully finetuned methods, while being up to 8x faster than state-of-the-art frozen-encoder models. Code: https://github.com/tue-mps/pmt.

Figures & tables

Panel of Figure 1 — Figure 1: ViT-Adapter + Mask2Former vs . PMT (Ours). PMT exhibits a better trade-off between Panoptic Quality and FPS across different sizes of frozen DINOv3 [ 41 ] pre-trained ViTs [ 14 ] . Evaluated on COCO val2017 [ 25 ] , see Tab. 4 .

Figure 2: Plain Mask Transformer (PMT) Architecture. Instead of injecting the query tokens within the ViT encoder as in the encoder-only framework of EoMT and VidEoMT, we extract features at multiple encoder levels and feed them into an efficient segmentation decoder that processes queries and patch tokens in parallel. $\oplus$ denotes element-wise addition. $\odot$ denotes the dot product.

Panel of Table 1 — Table 1: From ViT-Adapter + Mask2Former to EoMT. Stepwise removal of Mask2Former modules toward EoMT with frozen DINOv3 [ 41 ] encoder. Evaluated on COCO val2017 [ 25 ] .

Panel of Table 2 — Table 2: From EoMT to PMT. Stepwise addition of modules from EoMT with frozen DINOv3 [ 41 ] and DINOv2 [ 39 ] encoders to our PMT. Evaluated on COCO val2017 [ 25 ] .

Model	Pre-train	Params	GFLOPs	FPS	PQ
ViT-Adapter + M2F	DINOv3	340M	804	48	56.4
PMT (Ours)		357M	767	141	56.1
ViT-Adapter + M2F	DINOv2	342M	804	49	55.7
PMT (Ours)		359M	767	146	55.5
ViT-Adapter + M2F	IN21K	342M	804	49	51.0
PMT (Ours)		359M	767	146	49.7

Table 3: Pre-training. Impact of pre-training on segmentation quality. Evaluated on COCO val2017 [ 25 ] .

Model	Size	Params	GFLOPs	FPS	PQ
ViT-Adapter + M2F	L	340M	804	48	56.4
PMT (Ours)		357M	767	141	56.1
ViT-Adapter + M2F	B	116M	334	64	53.6
PMT (Ours)		116M	281	262	52.8
ViT-Adapter + M2F	S	45M	163	73	49.6
PMT (Ours)		29M	91	400	45.3

Table 4: Model Size. Impact of model size on segmentation quality and inference speed. Evaluated on COCO val2017 [ 25 ] .

Blocks	Params	GFLOPs	FPS	PQ
2	332M	695	153	54.0
4	344M	731	146	55.6
6	357M	767	141	56.1
8	370M	803	133	56.1

Table 5: Decoder Layers. Impact of number of decoder layers on segmentation quality and inference speed. Evaluated on COCO val2017 [ 25 ] .

Method	Backbone	Pre-training	Input size	Params	GFLOPs	FPS	PQ	AP
Mask2Former † [ 11 ]	Swin-L [ 26 ]	IN21K	$800^{2}$	216M	868	24	57.8	50.1
kMaX-DeepLab [ 49 ]	ConvNext-L [ 27 ]	IN21K	$1281^{2}$	232M	–	–	58.0	–
OneFormer † [ 19 ]	DiNAT-L [ 15 ]	IN21K	$800^{2}$	223M	736	20	58.0	49.2
MaskDINO † [ 24 ]	Swin-L [ 26 ]	IN21K	$800^{2}$	223M	1326	14	58.3	52.3
EoMT [ 20 ]	ViT-L [ 14 ]	DINOv2	$640^{2}$	316M	669	172	56.0	45.2
EoMT [ 20 ]	ViT-L [ 14 ]	DINOv3	$640^{2}$	315M	668	162	56.8	45.9

Table 6: PMT for Panoptic and Instance Segmentation on COCO val2017 [ 25 ] . † During inference, these models resize the shortest side of images to the indicated scale, while preserving the aspect ratio.

Method	Backbone	Pre-training	Encoder	Input size	Params	GFLOPs	FPS	mIoU
Mask2Former † [ 11 ]	Swin-L [ 26 ]	IN21K		$640^{2}$	216M	–	33	56.1
MaskDINO † [ 24 ]	Swin-L [ 26 ]	IN21K		$640^{2}$	223M	–	–	56.6
OneFormer † [ 19 ]	ConvNext-XL [ 27 ]	IN21K		$640^{2}$	373M	607	21	57.4
OneFormer † [ 19 ]	DiNAT-L [ 15 ]	IN21K		$896^{2}$	223M	678	19	58.1
EoMT [ 20 ]	ViT-L [ 14 ]	DINOv2		$512^{2}$	316M	721	154	58.4
EoMT [ 20 ]	ViT-L [ 14 ]	DINOv3		$512^{2}$	315M	721	151	59.5

Table 7: PMT for Semantic Segmentation on ADE20K [ 52 ] . † These models resize the shortest side of images to the indicated scale during inference, while preserving the aspect ratio.

Method	Backbone	Pre-training	Encoder	YouTube-VIS 2019 val [ 47 ]					YouTube-VIS 2021 val [ 47 ]
Method	Backbone	Pre-training	Encoder	GFLOPs	FPS	AP	AP 75	AR 10	GFLOPs	FPS	AP	AP 75	AR 10
MinVIS [ 17 ]	Swin-L [ 26 ]	IN21K		401	29	61.6	68.6	66.6	255	30	55.3	62.0	60.8
DVIS [ 50 ]	Swin-L [ 26 ]	IN21K		411	23	63.9	70.4	69.0	405	24	58.7	66.6	64.6
DVIS-DAQ [ 53 ]	Swin-L [ 26 ]	IN21K		415	13	65.7	73.6	70.7	410	11	61.1	68.2	66.6
DVIS++ [ 51 ]	ViT-Adapter-L [ 9 ]	DINOv2		846	18	67.7	75.3	73.7	830	17	62.3	70.2	68.0
DVIS-DAQ [ 53 ]	ViT-Adapter-L [ 9 ]	DINOv2		851	10	68.3	76.1	73.5	836	10	62.4	70.8	68.0

Table 8: PMT for Online VIS on YouTube-VIS 2019 and 2021 [ 47 ] .

Method	Backbone	PT	Encoder	VIPSeg val [ 34 ]
Method	Backbone	PT	Encoder	GFLOPs	FPS	VPQ	STQ
DVIS [ 50 ]	Swin-L [ 26 ]	I		879	20	54.7	47.7
DVIS++ [ 51 ]	ViT-Adapter-L [ 9 ]	D2		2290	13	56.0	49.8
DVIS-DAQ [ 53 ]	ViT-Adapter-L [ 9 ]	D2		2315	4	57.4	52.0
CAVIS [ 23 ]	ViT-Adapter-L [ 9 ]	D2		2612	10	56.9	51.0
VidEoMT [ 38 ]	ViT-L [ 14 ]	D2		1897	75	55.2	48.9

Table 9: PMT for Online VPS on VIPSeg [ 21 ] . PT : Pre-training. I : ImageNet-21K [ 13 ] . D2 : DINOv2 [ 39 ] ). D3 : DINOv3 [ 41 ] .

Method	Backbone	PT	Encoder	VSPW val [ 33 ]
Method	Backbone	PT	Encoder	GFLOPs	FPS	mVC 16	mIoU
DVIS [ 50 ]	Swin-L [ 26 ]	I		879	22	94.3	61.3
DVIS++ [ 51 ]	ViT-Adapter-L [ 9 ]	D2		2290	13	94.2	62.8
VidEoMT [ 38 ]	ViT-L [ 14 ]	D2		1909	73	95.0	64.9
VidEoMT [ 38 ]	ViT-L [ 14 ]	D3		1909	71	94.4	64.0
PMT (Ours)	ViT-L [ 14 ]	D2		2049	60	94.6	64.3

Table 10: PMT for Online VSS on VSPW [ 33 ] . PT : Pre-training. I : ImageNet-21K [ 13 ] . D2 : DINOv2 [ 39 ] ). D3 : DINOv3 [ 41 ] .

Explore similar work

Feb 19, 2026

VidEoMT: Your ViT is Secretly Also a Video Segmentation Model

Existing online video segmentation models typically combine a per-frame segmenter with complex specialized tracking modules. While effective, these modules introduce significant architectural complexity and computational overhead. Recent studies suggest that plain Vision Transformer (ViT) encoders, when scaled with sufficient capacity and large-scale pre-training, can conduct accurate image segmentation without requiring specialized modules. Motivated by this observation, we propose the Video Encoder-only Mask Transformer (VidEoMT), a simple encoder-only video segmentation model that eliminates the need for dedicated tracking modules. To enable temporal modeling in an encoder-only ViT, VidEoMT introduces a lightweight query propagation mechanism that carries information across frames by reusing queries from the previous frame. To balance this with adaptability to new content, it employs a query fusion strategy that combines the propagated queries with a set of temporally-agnostic learned queries. As a result, VidEoMT attains the benefits of a tracker without added complexity, achieving competitive accuracy while being 5x-10x faster, running at up to 160 FPS with a ViT-L backbone. Code: https://www.tue-mps.org/videomt/

Narges Norouzi, Idil Esen Zulfikar, Niccolò Cavagnero +4

Eindhoven University of Technology · RWTH Aachen University

Video Object Segmentation Vision Transformer

Ask PDF

Apr 7, 2026

MPM: Mutual Pair Merging for Efficient Vision Transformers

Decreasing sequence length is a common way to accelerate transformers, but prior token reduction work often targets classification and reports proxy metrics rather than end-to-end latency. For semantic segmentation, token reduction is further constrained by the need to reconstruct dense, pixel-aligned features, and on modern accelerators the overhead of computing merge maps can erase expected gains. We propose Mutual Pair Merging (MPM), a training-free token aggregation module that forms mutual nearest-neighbor pairs in cosine space, averages each pair, and records a merge map enabling a gather-based reconstruction before the decoder so that existing segmentation heads can be used unchanged. MPM introduces no learned parameters and no continuous compression knob (no keep-rate or threshold). The speed-accuracy trade-off is set by a discrete insertion schedule. We benchmark end-to-end latency on an NVIDIA H100 GPU (with and without FlashAttention-2) and a Raspberry Pi 5 across standard segmentation datasets. On ADE20K, MPM reduces per-image latency by up to 60% for ViT-Tiny on Raspberry Pi 5, and increases throughput by up to 20% on H100 with FlashAttention-2 while keeping the mIoU drop below 3%. These results suggest that simple, reconstruction-aware, training-free token merging can translate into practical wall-clock gains for segmentation when overhead is explicitly accounted for.

Simon Ravé, Pejman Rasti, David Rousseau

LARIS University of Angers Angers, France · UMR INRAe-IRHS Angers, France

Visual Token Merging Efficient ViTs Vision Transformer Semantic Segmentation

Ask PDF

Jul 1, 2026

LUMA: Benchmarking Segmentation via a Lightweight Universal Mask Adapter

Comparing transformer backbones for image segmentation is confounded: each is paired with a different decoder, recipe, and pretraining, so reported differences rarely reflect the backbone itself. We introduce the Lightweight Universal Mask Adapter (LUMA), a lightweight, backbone-agnostic mask-transformer head that treats any backbone as a black-box feature extractor, letting a set of queries read from its features through cheap cross-attention. LUMA matches the accuracy of EoMT, the state-of-the-art efficient ViT-segmenter, at lower cost, while attaching unchanged to isotropic, hierarchical, convolutional, and mixture-of-experts backbones alike. Holding this head fixed, we benchmark 20 backbones, 11 pretraining schemes and a range of resolutions on ADE20K and Cityscapes under one modern recipe. We find that ``efficient'' token mixers fail to deliver efficiency even at the high resolutions that motivate them, with plain ViT holding the throughput Pareto-front at every resolution. Additionally, the pretraining objective, not the architecture, the lever the field has tuned hardest, governs segmentation quality.

Tobias Christian Nauen, Anosh Billimoria, Federico Raue +3

RPTU University Kaiserslautern-Landau, Kaiserslautern, Germany · German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany

Efficient ViTs Semantic Segmentation

Ask PDF