Video segmentation models recognize and track objects over time, but they do not indicate whether each segmented region moves independently of the observing camera. This dynamicity attribute cannot be inferred from semantics alone and is confounded by camera ego-motion. We introduce \method, an online framework that augments query-based video segmentation with region-level dynamicity prediction. It jointly produces the original segmentation outputs and a dynamic or static state for each predicted region. At inference, DynEoMT uses only the current frame and propagated queries, without optical flow, depth, camera pose, previous RGB frames, or feature maps. Because established video segmentation benchmarks do not annotate this attribute, we also introduce a class-agnostic offline supervision pipeline using camera-compensated optical flow and confidence-aware temporal filtering. Across VIPSeg, OVIS, YouTube-VIS 2022, and VSPW, DynEoMT achieves balanced accuracies of 84.3, 68.0, 68.6, and 87.6, respectively, while largely preserving segmentation performance. These results show that segmentation-region dynamicity can be learned from propagated queries, enabling its online prediction without a dedicated motion-processing pipeline at inference. The complete code will be released as open source to enable full reproduction of the method and experiments.
Figures & tables
Figure 1: Overview of DynEoMT. The current image and the queries propagated from the previous frame are jointly processed by a ViT. The resulting current queries feed the original segmentation heads and the added dynamicity head, producing segmentation outputs and dynamic/static predictions.
Figure 2: Offline dynamicity pseudo-label construction. MemFlow supplies bidirectional optical flow between paired frames. Forward–backward consistency filtering removes unreliable correspondences. A camera-induced flow model is then robustly selected, and its predicted flow is subtracted from the observed flow. Residual-motion evidence is aggregated within each annotated mask. Confidence-aware temporal filtering assigns a dynamic, static, or uncertain target. The procedure is class-agnostic and is used only to construct training supervision.
Figure 3: DynEoMT architecture. Current image patches and object queries propagated from the preceding frame are jointly processed by the ViT encoder. The resulting queries feed the original classification and mask heads together with the proposed query-level dynamicity head, and are then propagated to the next frame. The propagated queries constitute the only temporal context available at inference. The dynamicity branch is the sole architectural addition to VidEoMT; no previous or future RGB frames, optical flow, depth, or camera pose are required.
Dataset
Videos
Frames
Dynamic
Static
Uncertain
Dyn. rate
VIPSeg
2,806
66,767
110,529
514,550
79,138
17.7%
OVIS
607
42,149
106,619
57,608
39,186
64.9%
YTVIS22
2,985
90,160
90,633
45,789
33,437
66.4%
VSPW
2,806
197,253
242,566
2,744,462
242,459
8.1%
Table 1: Offline dynamicity supervision on training splits. Dynamic rate is measured among valid labels.
Aggregation
Coverage
Prec.
Rec.
BAcc
F1
MCC
Majority
97.1
80.5
58.9
76.0
68.0
57.2
Any-dynamic
97.1
66.8
72.8
77.7
69.7
54.3
Table 2: Human audit of VIPSeg pseudo-labels.
Dataset
Backbone
Metric
VidEoMT
DynEoMT
VIPSeg
S
VPQ
4.45
4.32
STQ
8.24
8.05
B
VPQ
7.57
6.75
STQ
12.66
11.55
L
VPQ
55.7
54.5
STQ
49.4
47.6
Table 3: Native video segmentation metrics. Values are percentages. Each benchmark is evaluated with its standard task metrics.
Dataset
Backbone
BAcc
MCC
Coverage
DynSeg-F1
VIPSeg
S
83.36
59.37
25.77
29.79
B
84.30
59.46
32.98
36.48
L
83.80
58.00
60.70
55.00
OVIS
S
64.11
30.22
82.27
20.65
B
67.98
36.23
85.96
22.74
L
67.60
36.59
90.48
25.22
Table 4: Dynamicity and backbone capacity. BAcc and MCC are conditional on valid mask matches; Coverage and DynSeg-F1 are end-to-end. Values are percentages.
Dataset
ViT
FPS
ms/frame
VidEoMT
DynEoMT
VidEoMT
DynEoMT
VIPSeg
S
28.89
28.48
34.61
35.12
B
13.16
13.03
75.98
76.76
L
4.59
4.74
217.72
210.87
OVIS
S
42.70
41.89
23.42
23.87
B
21.64
21.65
46.21
46.18
Table 5: Backbone runtime on a Jetson AGX Orin across datasets and ViT scales. Higher FPS and lower ms/frame are better.
Setting
BAcc
MCC
Cov.
DynSeg-F1
Frozen + linear probe
81.56
50.48
61.91
51.53
Frozen + MLP probe
83.75
55.46
61.91
53.83
Frozen + MLP, no propagation
84.82
57.67
54.47
51.92
Full DynEoMT
83.80
58.00
60.70
55.00
Table 6: Probing dynamicity information in VidEoMT queries on VIPSeg with ViT-L. BAcc and MCC are conditional on valid mask matches. Coverage and DynSeg-F1 are end-to-end.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter group
Default values
Setting
Coverage
BAcc
F1
MCC
Default
–
Default
97.1
76.0
68.0
57.2
FB consistency
(1.5,0.05)
0.8×
96.9
75.9
67.9
57.1
1.2×
97.1
75.9
67.8
56.9
Camera-model selection
(0.20,0.035)
0.8×
97.0
76.0
68.0
57.1
1.2×
97.0
76.1
68.1
57.6
Residual-motion threshold
(3,0.20,1.25,0.0015)
0.8×
96.9
76.7
69.0
57.2
Appendix
Table 7: Sensitivity of pseudo-label quality to the main parameter groups on the manually audited VIPSeg subset. Results are reported using majority track-level aggregation. For multiplicative perturbations, all constants in the corresponding group are jointly scaled. The MAD normalization factor 1.4826 is kept fixed.
Variant
Aggregation
Coverage
Prec.
Rec.
BAcc
F1
MCC
Full pipeline
Majority
97.1
80.5
58.9
76.0
68.0
57.2
Any-dynamic
97.1
66.8
72.8
77.7
69.7
54.3
No temporal smoothing
Majority
97.5
80.7
57.9
75.6
67.4
56.7
Any-dynamic
97.5
61.4
77.3
76.9
68.4
51.3
No confidence weighting
Majority
98.4
80.3
59.3
76.1
68.2
57.2
Any-dynamic
98.4
65.6
73.4
77.3
69.2
53.1
Appendix
Table 8: Pseudo-label generation ablation on the manually audited VIPSeg subset.Metrics are computed on confidently labeled tracks, while coverage is measured over all manually annotated tracks.
Dataset
ViT
FPS
ms/frame
VidEoMT
DynEoMT
VidEoMT
DynEoMT
VIPSeg
S
94.95
97.41
10.53
10.27
B
69.91
69.62
14.30
14.36
L
28.44
30.38
35.16
32.92
OVIS
S
79.66
85.60
12.55
11.68
B
62.52
81.90
15.99
12.21
Appendix
Table 9: Backbone runtime of VidEoMT and DynEoMT on an NVIDIA A100-PCIE-40GB GPU.
Figure 4: VIPSeg qualitative predictions. Each row shows one sampled time step. From left to right: first sequence mask/class prediction, first sequence dynamicity prediction, second sequence mask/class prediction, and second sequence dynamicity prediction.
Figure 5: OVIS qualitative predictions. The four columns show mask/class and dynamicity predictions for two representative sequences, sampled at matched temporal positions.
Figure 6: YouTube-VIS 2022 qualitative predictions. Two representative sequences are shown with uniformly sampled time steps.
Figure 7: VSPW qualitative predictions. Since VSPW is semantic, dynamicity is displayed on predicted semantic regions rather than instance-specific masks.
Motion segmentation in dynamic scenes is highly challenging, as conventional methods heavily rely on estimating camera poses and point correspondences from inherently noisy motion cues. Existing statistical inference or iterative optimization techniques that struggle to mitigate the cumulative errors in multi-stage pipelines often lead to limited performance or high computational cost. In contrast, we propose a fully learning-based approach that directly infers moving objects from latent feature representations via attention mechanisms, thus enabling end-to-end feed-forward motion segmentation. Our key insight is to bypass explicit correspondence estimation and instead let the model learn to implicitly disentangle object and camera motion. Supported by recent advances in 4D scene geometry reconstruction (e.g., π3), the proposed method leverages reliable camera poses and rich spatial-temporal priors, which ensure stable training and robust inference for the model. Extensive experiments demonstrate that by eliminating complex pre-processing and iterative refinement, our approach achieves state-of-the-art motion segmentation performance with high efficiency. The code is available at:https://github.com/zjutcvg/GeoMotion.
Xiankang He, Peile Lin, Ying Cui +3
College of Computer Science and Technology, Zhejiang University of Technology · Zhejiang Key Laboratory of Visual Information Intelligent Processing · State Key Lab of CAD & CG, Zhejiang University
Existing online video segmentation models typically combine a per-frame segmenter with complex specialized tracking modules. While effective, these modules introduce significant architectural complexity and computational overhead. Recent studies suggest that plain Vision Transformer (ViT) encoders, when scaled with sufficient capacity and large-scale pre-training, can conduct accurate image segmentation without requiring specialized modules. Motivated by this observation, we propose the Video Encoder-only Mask Transformer (VidEoMT), a simple encoder-only video segmentation model that eliminates the need for dedicated tracking modules. To enable temporal modeling in an encoder-only ViT, VidEoMT introduces a lightweight query propagation mechanism that carries information across frames by reusing queries from the previous frame. To balance this with adaptability to new content, it employs a query fusion strategy that combines the propagated queries with a set of temporally-agnostic learned queries. As a result, VidEoMT attains the benefits of a tracker without added complexity, achieving competitive accuracy while being 5x-10x faster, running at up to 160 FPS with a ViT-L backbone. Code: https://www.tue-mps.org/videomt/
Reasoning video object segmentation predicts pixel-level masks in videos from natural-language queries that may involve implicit and temporally grounded references. However, existing methods are developed and evaluated in an offline regime, where the entire video is available at inference time and future frames can be exploited for retrospective disambiguation, deviating from real-world deployments that require strictly causal, frame-by-frame decisions. We study Online Reasoning Video Object Segmentation (ORVOS), where models must incrementally interpret queries using only past and current frames without revisiting previous predictions, while handling referent shifts as events unfold. To support evaluation, we introduce ORVOSB, a benchmark with frame-level causal annotations and referent-shift labels, comprising 210 videos, 12,907 annotated frames, and 512 queries across five reasoning categories. We further propose a baseline with continually-updated segmentation prompts and a structured temporal token reservoir for long-horizon reasoning under bounded computation. Experiments show that existing methods struggle under strict causality and referent shifts, while our baseline establishes a strong foundation for future research.
Jinyuan Liu, Yang Wang, Zeyu Zhao +3
Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced · School of Computer Science and Engineering, Beihang University · School of Electrical and Information Engineering, Northeast Petroleum University