Video segmentation models recognize and track objects over time, but they do not indicate whether each segmented region moves independently of the observing camera. This dynamicity attribute cannot be inferred from semantics alone and is confounded by camera ego-motion. We introduce \method, an online framework that augments query-based video segmentation with region-level dynamicity prediction. It jointly produces the original segmentation outputs and a dynamic or static state for each predicted region. At inference, DynEoMT uses only the current frame and propagated queries, without optical flow, depth, camera pose, previous RGB frames, or feature maps. Because established video segmentation benchmarks do not annotate this attribute, we also introduce a class-agnostic offline supervision pipeline using camera-compensated optical flow and confidence-aware temporal filtering. Across VIPSeg, OVIS, YouTube-VIS 2022, and VSPW, DynEoMT achieves balanced accuracies of 84.3, 68.0, 68.6, and 87.6, respectively, while largely preserving segmentation performance. These results show that segmentation-region dynamicity can be learned from propagated queries, enabling its online prediction without a dedicated motion-processing pipeline at inference. The complete code will be released as open source to enable full reproduction of the method and experiments.
Figures & tables
Figure 1: Overview of DynEoMT. The current image and the queries propagated from the previous frame are jointly processed by a ViT. The resulting current queries feed the original segmentation heads and the added dynamicity head, producing segmentation outputs and dynamic/static predictions.
Figure 2: Offline dynamicity pseudo-label construction. MemFlow supplies bidirectional optical flow between paired frames. Forward–backward consistency filtering removes unreliable correspondences. A camera-induced flow model is then robustly selected, and its predicted flow is subtracted from the observed flow. Residual-motion evidence is aggregated within each annotated mask. Confidence-aware temporal filtering assigns a dynamic, static, or uncertain target. The procedure is class-agnostic and is used only to construct training supervision.
Figure 3: DynEoMT architecture. Current image patches and object queries propagated from the preceding frame are jointly processed by the ViT encoder. The resulting queries feed the original classification and mask heads together with the proposed query-level dynamicity head, and are then propagated to the next frame. The propagated queries constitute the only temporal context available at inference. The dynamicity branch is the sole architectural addition to VidEoMT; no previous or future RGB frames, optical flow, depth, or camera pose are required.
Dataset
Videos
Frames
Dynamic
Static
Uncertain
Dyn. rate
VIPSeg
2,806
66,767
110,529
514,550
79,138
17.7%
OVIS
607
42,149
106,619
57,608
39,186
64.9%
YTVIS22
2,985
90,160
90,633
45,789
33,437
66.4%
VSPW
2,806
197,253
242,566
2,744,462
242,459
8.1%
Table 1: Offline dynamicity supervision on training splits. Dynamic rate is measured among valid labels.
Aggregation
Coverage
Prec.
Rec.
BAcc
F1
MCC
Majority
97.1
80.5
58.9
76.0
68.0
57.2
Any-dynamic
97.1
66.8
72.8
77.7
69.7
54.3
Table 2: Human audit of VIPSeg pseudo-labels.
Dataset
Backbone
Metric
VidEoMT
DynEoMT
VIPSeg
S
VPQ
4.45
4.32
STQ
8.24
8.05
B
VPQ
7.57
6.75
STQ
12.66
11.55
L
VPQ
55.7
54.5
STQ
49.4
47.6
Table 3: Native video segmentation metrics. Values are percentages. Each benchmark is evaluated with its standard task metrics.
Dataset
Backbone
BAcc
MCC
Coverage
DynSeg-F1
VIPSeg
S
83.36
59.37
25.77
29.79
B
84.30
59.46
32.98
36.48
L
83.80
58.00
60.70
55.00
OVIS
S
64.11
30.22
82.27
20.65
B
67.98
36.23
85.96
22.74
L
67.60
36.59
90.48
25.22
Table 4: Dynamicity and backbone capacity. BAcc and MCC are conditional on valid mask matches; Coverage and DynSeg-F1 are end-to-end. Values are percentages.
Dataset
ViT
FPS
ms/frame
VidEoMT
DynEoMT
VidEoMT
DynEoMT
VIPSeg
S
28.89
28.48
34.61
35.12
B
13.16
13.03
75.98
76.76
L
4.59
4.74
217.72
210.87
OVIS
S
42.70
41.89
23.42
23.87
B
21.64
21.65
46.21
46.18
Table 5: Backbone runtime on a Jetson AGX Orin across datasets and ViT scales. Higher FPS and lower ms/frame are better.
Setting
BAcc
MCC
Cov.
DynSeg-F1
Frozen + linear probe
81.56
50.48
61.91
51.53
Frozen + MLP probe
83.75
55.46
61.91
53.83
Frozen + MLP, no propagation
84.82
57.67
54.47
51.92
Full DynEoMT
83.80
58.00
60.70
55.00
Table 6: Probing dynamicity information in VidEoMT queries on VIPSeg with ViT-L. BAcc and MCC are conditional on valid mask matches. Coverage and DynSeg-F1 are end-to-end.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter group
Default values
Setting
Coverage
BAcc
F1
MCC
Default
–
Default
97.1
76.0
68.0
57.2
FB consistency
(1.5,0.05)
0.8×
96.9
75.9
67.9
57.1
1.2×
97.1
75.9
67.8
56.9
Camera-model selection
(0.20,0.035)
0.8×
97.0
76.0
68.0
57.1
1.2×
97.0
76.1
68.1
57.6
Residual-motion threshold
(3,0.20,1.25,0.0015)
0.8×
96.9
76.7
69.0
57.2
Appendix
Table 7: Sensitivity of pseudo-label quality to the main parameter groups on the manually audited VIPSeg subset. Results are reported using majority track-level aggregation. For multiplicative perturbations, all constants in the corresponding group are jointly scaled. The MAD normalization factor 1.4826 is kept fixed.
Variant
Aggregation
Coverage
Prec.
Rec.
BAcc
F1
MCC
Full pipeline
Majority
97.1
80.5
58.9
76.0
68.0
57.2
Any-dynamic
97.1
66.8
72.8
77.7
69.7
54.3
No temporal smoothing
Majority
97.5
80.7
57.9
75.6
67.4
56.7
Any-dynamic
97.5
61.4
77.3
76.9
68.4
51.3
No confidence weighting
Majority
98.4
80.3
59.3
76.1
68.2
57.2
Any-dynamic
98.4
65.6
73.4
77.3
69.2
53.1
Appendix
Table 8: Pseudo-label generation ablation on the manually audited VIPSeg subset.Metrics are computed on confidently labeled tracks, while coverage is measured over all manually annotated tracks.
Dataset
ViT
FPS
ms/frame
VidEoMT
DynEoMT
VidEoMT
DynEoMT
VIPSeg
S
94.95
97.41
10.53
10.27
B
69.91
69.62
14.30
14.36
L
28.44
30.38
35.16
32.92
OVIS
S
79.66
85.60
12.55
11.68
B
62.52
81.90
15.99
12.21
Appendix
Table 9: Backbone runtime of VidEoMT and DynEoMT on an NVIDIA A100-PCIE-40GB GPU.
Figure 4: VIPSeg qualitative predictions. Each row shows one sampled time step. From left to right: first sequence mask/class prediction, first sequence dynamicity prediction, second sequence mask/class prediction, and second sequence dynamicity prediction.
Figure 5: OVIS qualitative predictions. The four columns show mask/class and dynamicity predictions for two representative sequences, sampled at matched temporal positions.
Figure 6: YouTube-VIS 2022 qualitative predictions. Two representative sequences are shown with uniformly sampled time steps.
Figure 7: VSPW qualitative predictions. Since VSPW is semantic, dynamicity is displayed on predicted semantic regions rather than instance-specific masks.
College of Computer Science and Technology, Zhejiang University of Technology · Zhejiang Key Laboratory of Visual Information Intelligent Processing · State Key Lab of CAD & CG, Zhejiang University
Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced · School of Computer Science and Engineering, Beihang University · School of Electrical and Information Engineering, Northeast Petroleum University