LVMT: Video Mask Transformer for Long-term Video Segmentation

Narges Norouzi; Niccolò Cavagnero; Idil Esen Zulfikar; Bastian Leibe; Gijs Dubbelman; Daan de Geus

cs.CVSep 28, 2026

LVMT: Video Mask Transformer for Long-term Video Segmentation

Authors: Narges Norouzi, Niccolò Cavagnero, Idil Esen Zulfikar, Bastian Leibe, Gijs Dubbelman, Daan de Geus

Organizations: Eindhoven University of Technology · RWTH Aachen University

Abstract

Existing online video segmentation methods struggle to track objects in long, complex videos with long-term occlusions. We hypothesize that this limitation is caused by (i) the inability of their temporal propagation mechanism to adaptively select the object information that is propagated across time, and (ii) their inability to be trained on long videos due to memory requirements and vanishing gradients. To address the first limitation, we propose to use a lightweight GRU-based temporal propagation module that can learn to select which information it keeps in memory and propagates across time. Second, to allow training on long videos, we introduce Truncated Query Propagation (TQP), a training strategy in which the model processes a video in chunks of frames, where information about tracked objects is propagated between chunks but backpropagation is only conducted in individual chunks, enabling longer temporal supervision without out-of-memory issues, inference overhead, or vanishing gradients. The resulting model is called the Long-term Video Mask Transformer (LVMT). Extensive experiments on six benchmarks show that LVMT sets a new state of the art across a range of video segmentation tasks, while retaining the speed of the highly efficient model it is based on, making it 10X faster than the prior state of the art. Code: https://www.tue-mps.org/lvmt

Figures & tables

Panel of Figure 1 — Figure 1 : PMT vs. LVMT. Mean AP $\pm$ std. dev. over five runs. Across ViT-L/B/S, LVMT improves AP by at least $\mathbf{+4.6}$ over the efficient PMT [ 3 ] baseline at similar FPS. Evaluated on OVIS val .

Figure 2 : LVMT architecture. The Plain Mask Decoder (PMD) takes input queries and projected patch features from a frozen ViT encoder to produce segmentation queries for mask and class prediction. At $t=0$ , learnable queries $\mathbf{Q}^{\mathrm{lrn}}$ are used to initialize both the decoder input and the GRU hidden state. Thereafter, a GRU cell adaptively updates the hidden state from the current segmentation queries to yield propagation queries for the next frame. Truncated Query Propagation (TQP), visualized in Fig. 3 , is applied at training time.

Figure 3 : Truncated Query Propagation (TQP). A video of $T$ frames is partitioned into $M$ chunks of $F$ frames, and processed in order in a single iteration. For each chunk, we compute a loss and backpropagate it to calculate the gradients. The optimizer is updated with the average, accumulated gradients after all chunks are consumed. Queries are detached and propagated across chunk boundaries, bounding peak memory to a single chunk and preventing the model from suffering from vanishing gradients caused by long-horizon recurrence.

Method	Step	GRU	TQP	$T_{\mathrm{train}}$	Mean AP $\uparrow$	Params $\downarrow$	GFLOPs $\downarrow$	FPS $\uparrow$
PMT [ 3 ]	(0)	$\times$	$\times$	5	$51.8\pm 0.3$	358M	1014	97
	(1)	✓	$\times$	5	$54.2\pm 0.4$	363M	1015	95
	(2)	✓	$\times$	10	$52.3\pm 0.3$	363M	1015	95
	(3)	✓	✓	10	$56.1\pm 0.3$	363M	1015	95
LVMT (Ours)	(4)	✓	✓	15	$\mathbf{56.4\pm 0.4}$	363M	1015	95

Table 1 : Stepwise modifications from PMT to LVMT on OVIS val [ 31 ] . Mean AP

\pm

standard deviation are reported over five independent runs.

T_{\mathrm{train}}

is the training-clip length in frames.

Method	Backbone	Pre-training	Encoder	OVIS val [ 31 ]					YouTube-VIS 2022 val [ 42 ]
Method	Backbone	Pre-training	Encoder	AP	AP 75	AR 10	GFLOPs	FPS	AP ${}^{\text{L}}$	AP ${}^{\text{L}}_{\text{75}}$	AR ${}^{\text{L}}_{\text{10}}$	GFLOPs	FPS
DVIS++ [ 44 ]	ViT-Adapter-L [ 5 ]	DINOv2		49.6	55.0	54.6	868	17	37.5	39.4	43.5	820	18
CAVIS [ 20 ]	ViT-Adapter-L [ 5 ]	DINOv2		53.2	59.1	58.2	863	15	39.5	40.5	44.9	815	15
DVIS-DAQ [ 45 ] †	ViT-Adapter-L [ 5 ]	DINOv2		54.3	60.2	59.8	1173	8	42.0	43.0	48.4	826	10
LOMM [ 19 ]	ViT-Adapter-L [ 5 ]	DINOv2		51.7	57.5	56.2	899	12	48.2	53.2	52.6	842	12
VidEoMT [ 28 ] †	ViT-L [ 12 ]	DINOv2		52.5	57.2	57.5	934	115	42.6	46.1	48.1	557	161

Table 2 : LVMT for VIS on OVIS and YouTube-VIS 2022 [ 31 , 42 ] . † Input resolution of 544 shortest image side for OVIS.

Method	IDF1 (%) $\uparrow$	AssA (%) $\uparrow$	MT (%) $\uparrow$	ML (%) $\downarrow$	Total IDS $\downarrow$
CAVIS [ 20 ]	79.2	74.7	76.3	6.1	2683
VidEoMT [ 28 ]	78.7	73.3	75.5	6.2	2801
PMT [ 3 ]	78.9	73.3	75.7	6.3	2755
LVMT (Ours)	81.8	78.4	78.8	5.5	2001

Table 3 : Tracking quality on OVIS val [ 31 ] . We compare LVMT with existing methods using specialized tracking metrics.

Method	Backbone	PT	Encoder	VIPSeg val [ 25 ]
Method	Backbone	PT	Encoder	VPQ	STQ	GFLOPs	FPS
DVIS++ [ 44 ]	ViT-Adapter-L [ 5 ]	D2		56.0	49.8	2290	13
DVIS-DAQ [ 45 ]	ViT-Adapter-L [ 5 ]	D2		57.4	52.0	2315	4
CAVIS [ 20 ]	ViT-Adapter-L [ 5 ]	D2		56.9	51.0	2612	10
VidEoMT [ 28 ]	ViT-L [ 12 ]	D2		55.2	48.9	1897	75
VidEoMT [ 28 ]	ViT-L [ 12 ]	D3		55.1	48.1	1897	71

Table 4 : LVMT for VPS on VIPSeg [ 25 ] . PT : Pre-training. D2 : DINOv2 [ 29 ] . D3 : DINOv3 [ 34 ] .

Method	Backbone	PT	Encoder	VSPW val [ 26 ]
Method	Backbone	PT	Encoder	mVC 16	mIoU	GFLOPs	FPS
DVIS++ [ 44 ]	ViT-Adapter-L [ 5 ]	D2		94.2	62.8	2290	13
VidEoMT [ 28 ]	ViT-L [ 12 ]	D2		95.0	64.9	1909	73
VidEoMT [ 28 ]	ViT-L [ 12 ]	D3		94.4	64.0	1909	71
PMT [ 3 ]	ViT-L [ 12 ]	D2		94.6	64.3	2049	60
LVMT (Ours)	ViT-L [ 12 ]	D2		95.2	65.4	2050	59

Table 5 : LVMT for VSS on VSPW [ 26 ] . PT : Pre-training. D2 : DINOv2 [ 29 ] . D3 : DINOv3 [ 34 ] .

Memory	AP	AP 75	AR 10	GFLOPs	FPS
LSTM [ 14 ]	56.0	61.3	60.9	1016	93
S5 [ 35 ]	55.8	61.8	60.5	1014	96
DeepGRU [ 46 ]	56.5	61.8	61.5	1019	91
Memory Bank [ 13 ]	54.5	59.8	59.9	1017	86
GRU [ 9 ]	56.7	61.8	61.6	1015	95

Table 6 : Effect of memory mechanism on OVIS val [ 31 ] . We compare various memory mechanisms for temporal modeling.

$T_{\textrm{train}}$	GRU	TQP	AP	AP 75	AR 10	GFLOPs	FPS
5	$\times$	$\times$	52.0	56.0	57.7	1014	97
5	✓	$\times$	54.7	60.6	59.7	1015	95
10	$\times$	$\times$	51.1	56.0	56.7	1014	97
10	✓	$\times$	52.6	56.6	58.4	1015	95
10	$\times$	✓	54.5	59.6	59.9	1014	97
10	✓	✓	56.5	61.5	61.5	1015	95

Table 7 : Effect of GRU and TQP on OVIS val [ 31 ] . We evaluate the effects of GRU-based query propagation and TQP across different training clip lengths

T_{\textrm{train}}

. All results are measured using 8 NVIDIA H100 (94GB) with batch size 8 (one clip per GPU) and multi-scale resolution (320–640px shortest side, capped at 768).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Backbone	Pre-training	Encoder	YouTube-VIS 2019 val [ 42 ]					YouTube-VIS 2021 val [ 42 ]
Method	Backbone	Pre-training	Encoder	AP	AP 75	AR 10	GFLOPs	FPS	AP	AP 75	AR 10	GFLOPs	FPS
DVIS++ [ 44 ]	ViT-Adapter-L [ 5 ]	DINOv2		67.7	75.3	73.7	846	18	62.3	70.2	68.0	830	17
DVIS-DAQ [ 45 ]	ViT-Adapter-L [ 5 ]	DINOv2		68.3	76.1	73.5	851	10	62.4	70.8	68.0	836	10
CAVIS [ 20 ]	ViT-Adapter-L [ 5 ]	DINOv2		68.9	76.2	73.6	838	15	64.6	72.5	69.3	824	15
LOMM [ 19 ]	ViT-Adapter-L [ 5 ]	DINOv2		69.1	76.5	73.5	842	12	65.0	72.7	69.1	842	12
VidEoMT [ 28 ]	ViT-L [ 12 ]	DINOv2		68.6	75.6	73.9	566	160	63.1	69.3	68.1	560	160

Appendix

Table A : LVMT for VIS on YouTube-VIS 2019 and 2021 [ 42 ] .

GT Metric	High-IDS	Low-IDS
Mean objects per video	$10$	$3$
Disappear rate † (%)	$63$	$13$
Mean disappearance length (frames)	13	6

Appendix

Table B : Analysis of high- and low-IDS videos on OVIS val . High- and low-IDS groups are defined as the 20 videos with the highest and lowest number of identity switches, respectively, measured using the GRU-based model from step (1) in Tab. 1 in the main paper. IDS denotes identity switches. † Fraction of frames where

\geq

1 object is absent due to occlusion or leaving the scene. All statistics are computed from ground-truth (GT) annotations.

Model	GRU	TQP	AP	AP 75	AR 10	GFLOPs	FPS
VidEoMT [ 28 ]	$\times$	$\times$	50.1	53.9	55.8	934	104
LVMT (Ours)	$\times$	$\times$	51.1	56.0	56.7	1014	97
VidEoMT [ 28 ]	✓	$\times$	51.8	56.5	57.3	935	102
LVMT (Ours)	✓	$\times$	52.6	56.6	58.4	1015	95
VidEoMT [ 28 ]	$\times$	✓	52.1	56.7	57.5	934	104
LVMT (Ours)	$\times$	✓	54.5	59.6	59.9	1014	97

Appendix

Table C : Effect of GRU and TQP. Impact of GRU-based propagation and TQP training on VidEoMT [ 28 ] and LVMT on OVIS val with 10-frame training clips.

Method	Size	AP	Params	GFLOPs	FPS
VidEoMT [ 28 ]	L	51.9	316M	934	104
PMT [ 3 ]		52.0	358M	1014	97
LVMT (Ours)		56.7	363M	1015	95
VidEoMT [ 28 ]	B	42.7	93M	304	178
PMT [ 3 ]		42.9	116M	350	160
LVMT (Ours)		48.1	120M	351	155

Appendix

Table D : Impact of model size on OVIS val [ 31 ] . We compare VidEoMT [ 28 ] , PMT [ 3 ] , and LVMT across different model sizes.

Method	Backbone	Encoder	AP	AP 75	AR 10
		DINOv2 [ 29 ]
PMT [ 3 ]	ViT-L [ 12 ]		51.8	57.7	56.0
LVMT (Ours)	ViT-L [ 12 ]		55.6	61.6	60.5
PMT [ 3 ]	ViT-L [ 12 ]		53.8	56.0	58.8
LVMT (Ours)	ViT-L [ 12 ]		56.5	61.2	61.7
		DINOv3 [ 34 ]

Appendix

Table E : Effect of encoder fine-tuning on OVIS val [ 31 ] . We compare frozen and fine-tuned encoders for PMT [ 3 ] and LVMT using DINOv2 and DINOv3 pre-training.

Panel of Figure 16 — Figure A : Temporal gradient flow with and without TQP. TQP shortens the backward path and increases within-chunk gradient retention from approximately 5% to 37%.

Panel of Figure 17 — Figure B : GRU memory retention around object occlusion. Occluded-object queries rely more strongly on memory during the hidden interval.

Chunk size	AP	AP 75	AR 10	GFLOPs	FPS
1	50.9	54.2	55.4	1015	95
3	55.8	60.9	61.0	1015	95
5	56.7	61.8	61.6	1015	95
10	54.3	59.7	60.0	1015	95

Appendix

Table F : Effect of chunk size on OVIS val [ 31 ] . We explore different chunk sizes used by TQP during training.

Hidden state initialization	AP	AP 75	AR 10
Zero Init	55.2	59.6	60.6
Random Gaussian Init	55.8	61.5	60.8
Learnable State	55.7	61.8	60.6
Learnable Object Queries	56.7	61.8	61.6

Appendix

Table G : Effect of hidden-state initialization on OVIS val [ 31 ] . We evaluate different GRU hidden-state initialization strategies.

Panel of Figure 20 — Figure C : Qualitative results on OVIS [ 31 ] . Comparison between PMT [ 3 ] , LVMT, and the ground-truth annotations on selected frames $t=\{0,4,8,10,11\}$ . PMT suffers from identity switches at $t=4$ and $t=8$ under heavy occlusion among similar motorcycles and riders, while LVMT preserves more consistent identities across the sequence.

Explore similar work

MoVISA: Multi-Token Reasoning for Video Object Segmentation
Sep 24, 2026Ruining Zhao, Ho Kei Cheng, Alexander G SchwingVideo Object Segmentation Multimodal Large Language Models
University of Illinois Urbana-Champaign
Open-World Video Segmentation
Jun 14, 2026Qing Su, Kaiyang Li, Yuan Zhuang +2Video Object Segmentation Discovery
University of Connecticut
VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMs
Sep 15, 2026Haoyu Guo, Yuan Feng, Junlin Lv +3Video Multimodal Large Language Models Multimodal Large Language Models
School of Biomedical Engineering, University of Science and Technology of China · Data Darkness Lab, MIRACLE Center, Suzhou Institute for Advanced Research · China +1

Sep 24, 2026, cs.CVJ/K move · Enter open · S save

MoVISA: Multi-Token Reasoning for Video Object Segmentation

Ruining Zhao, Ho Kei Cheng, Alexander G Schwing

University of Illinois Urbana-Champaign

Video Object Segmentation Multimodal Large Language Models

Open paper AskPDF arXiv

Recent advances in video object segmentation with Multimodal Large Language Model (MLLM) reasoning have demonstrated the effectiveness of using a single textual token, such as SEG, to predict segmentation masks across images and videos. However, we observe that this single-token strategy lacks the granularity required to precisely localize multiple objects across time in video segmentation tasks. To address this limitation, we develop Multi-Token Reasoning for Video Object Segmentation, or MoVISA. MoVISA uses multiple segmentation tokens, such as SEG0 and SEG1, to represent an object across different frames. This design enables more fine-grained alignment between language prompts and spatio-temporal mask predictions, improving both performance and interpretability. On the challenging MeViS, DAVIS17, ReVOS, and Ref-Youtube-VOS benchmarks, our model achieves a 13.2 percent J and F improvement on MeViS and an 8.4 percent J and F improvement on ReVOS. Code and models will be released.

Abstract

Explore similar work

MoVISA: Multi-Token Reasoning for Video Object Segmentation

Open-World Video Segmentation

VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMs