Moving Regions, Not Moving Objects: Segmenting What Actually Moves under Camera Motion

Wenhua Jin; Tiancheng Chen; Sheng Tang; Jiahao Li; Yupeng Liu; Weiqi Zhang; Juntong Fang; Zesong Li

cs.CVOct 22, 2026

Moving Regions, Not Moving Objects: Segmenting What Actually Moves under Camera Motion

Authors: Wenhua Jin, Tiancheng Chen, Sheng Tang, Jiahao Li, Yupeng Liu, Weiqi Zhang, Juntong Fang, Zesong Li

Organizations: Kosmo Research · Automotive Engineering Department, Jilin University · Xiamen University · School of Software, Tsinghua University

Abstract

Moving object segmentation is conventionally formulated at the sequence and object level, marking an entire instance as moving once it moves anywhere in a video. Recent instantaneous formulations distinguish moving from stationary intervals, but still generate masks at the object level through frozen segmentation foundation models. Because the mask is derived from an instance-level proposal, motion is typically attributed to the entire instance rather than localized to the regions that actually move. Moreover, such instance-level predictions can produce masks even when no part of the instance is actually moving, revealing a more fundamental issue: motion is not an intrinsic property of an instance; it can only be determined by comparing at least two frames. Building on this premise, we introduce MoRSe, which jointly interprets two frames to predict a moving-region mask, distinguishing actual scene motion from apparent motion induced by the camera. A single RGB encoder replaces multi-stage foundation stacks, so no optical flow, point tracks, or separate mask generator is needed at inference. We release MoRSe-5K, a benchmark of 42 sequences with Static, Dynamic-Static, and Dynamic settings designed to evaluate the model's ability to distinguish and localize moving and stationary regions under different camera and scene motion conditions. MoRSe outperforms both specialized moving object segmentation methods and dense 4D reconstruction methods with motion-thresholded predictions, while producing the fewest false positives on sequences where nothing moves.

Figures & tables

Figure 2: MoRSe architecture. Both frames pass jointly through a UniQuery4R-initialized ViT and a shallow convolutional branch; the two are fused and split into source and target feature pyramids. A camera head reads the ViT camera tokens and predicts per-view pose and field of view as auxiliary outputs that never enter the mask path. Sampled source coordinates form the decoder queries; target features provide keys and values, and four cross-attention blocks refine each query before a shared MLP predicts the motion logit.

Figure 3: Inference granularity. Target selection sets the granularity without changing the weights: one pair, a target window, or a promptable segmenter.

		Static		Dynamic-Static $+$ Dynamic
				Moving region			Moving object
Method	Native output	PER $\!\downarrow$	FER $\!\downarrow$	$\mathcal{J}\!\uparrow$	$\mathcal{F}\!\uparrow$	$\mathcal{J}\&\mathcal{F}\!\uparrow$	$\mathcal{J}\!\uparrow$	$\mathcal{F}\!\uparrow$	$\mathcal{J}\&\mathcal{F}\!\uparrow$
Dense 4D reconstruction: predict scene flow, converted to a mask by thresholding
Trace Anything †	trajectory field	21.01	100.00	32.10	35.90	34.00	30.18 ∗	38.23 ∗	34.21 ∗
Any4D †	scene flow	0 0.46	0 18.70	27.30	29.80	28.60	33.80 ∗	38.28 ∗	36.04 ∗
4RC †	scene flow	0 0.53	0 18.20	40.50	41.50	41.00	39.56 ∗	43.69 ∗	41.62 ∗

Table 1: Main results on MoRSe-5K (%). Metrics and the two ground-truth variants follow Section 4.3 . “Scene flow” rows are thresholded into masks by each model’s own released or automatic criterion, frozen before evaluation and never re-tuned per variant.

\dagger

: matched-window rerun from released code.

\ast

: the moving-object score comes from the shared SAM 3 readout; unmarked rows are scored on their own instance output. best , second , third per column.

Figure 4: Moving regions under a moving camera. Top (Static): the camera moves while the person stands still; every baseline paints the whole person, while MoRSe leaves the frame empty. Bottom (Dynamic-Static): only the child’s hand moves; MoRSe marks that hand alone, whereas the baselines cover the whole child.

$\|\mathcal{T}\|$	Precision $\uparrow$	Recall $\uparrow$	$\mathcal{J}\&\mathcal{F}\uparrow$
2	86.7	80.4	72.1
4	86.0	82.2	72.9
8	84.8	83.3	73.0

Table 2: Union-mask inference. Mean precision, recall, and

\mathcal{J}\&\mathcal{F}

over five benchmarks when masks from

|\mathcal{T}|

target frames are unioned per source (Equation 1 ).

	DAVIS16-M			FBMS-59			MoCA
Method	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$
OCLR † ( Xie et al., 2022 )	70.25	69.60	69.92	61.71	62.87	62.29	12.08	17.86	14.97
ABR † ( Xie et al., 2024a )	74.02	71.81	72.91	73.48	75.32	74.40	0 8.80	14.71	11.75
FlowSAM † ( Xie et al., 2024b )	37.56	38.18	37.87	40.00	41.89	40.95	29.81	24.54	27.17
SegAnyMo † ( Huang et al., 2025 )	75.89	80.82	78.35	68.80	76.07	72.44	58.81	66.10	62.46
RoMo † ( Goli et al., 2025 )	70.11	73.87	71.99	58.86	61.07	59.96	11.50	12.39	11.94

Table 3: Comparison with moving object segmentation (MOS) baselines. Conventional foreground–background MOS under one protocol across three benchmarks, with instance outputs unioned before evaluation. Every baseline is a controlled rerun from released code, so the numbers are mutually comparable but need not match the original papers. Within each column, best , second best , and third best .

\ddagger

: MoRSe emits a moving region rather than instances, so its row is shown in grey for reference only and is left out of the ranking; the object-level entry is MoRSe

+

SAM 3.

Low-level branch
Variant	wJF $\uparrow$	PER $\downarrow$	FER $\downarrow$
		( $\times 10^{-4}$ )	(%)
Default (MoRSe)	77.39	0.273	1.45
ResNet-18	77.27	0.348	2.07
ResNet-34	77.41	1.448	2.64
ResNet-50	78.23	2.428	2.80

Table 4: Ablations on MoRSe-5K (%). Each row changes one component. Entries average two seeds; the default row averages repeated runs, unlike its single run in Table 1 . All ablations are trained on a single A800 GPU. Per-seed spread (

\pm

std) in Table 9 .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

3D point tracking — 19 entries, 137,103 sequences, 19,260,636 frames; 92.3% of training samples
Source	Seq.	Frames	Source	Seq.	Frames
PointOdyssey	111	207,968	Kubric4D	808	48,480
CoTracker3-Kubric	5,827	699,240	Flock4D	950	434,003
TAPIP3D-Kubric	10,950	262,800	DynamicReplica	8	2,400
SynthVerse	26,187	4,934,809	Stereo4D	81,962	12,128,259
Syn4D	565	168,787	Waymo	5,497	131,928

Appendix

Table 5: Training corpus inventory. Family headers carry the normalized sampling-weight share. LASIESTA and MannequinChallenge are listed for evaluation provenance and excluded from training.

Group	Quantity	Value
Query sampler	Motion-region share $\alpha_{m}$	$1/6$ (171 of 1,024)
	Boundary-band share $\alpha_{b}$	0 (band disabled)
	Static/background share $\alpha_{h}$	$5/6$ (uniform, static first)
Objectives	Dice weight $\lambda_{\mathrm{dice}}$	0.5
	BCE weight $\lambda_{\mathrm{bce}}$	0.5
	Mask loss scale $\lambda_{\mathrm{mask}}$	3.0

Appendix

Table 6: Hyperparameters and protocol constants.

Method	Criterion	Value	Provenance and rationale
Any4D	Scene-flow magnitude	0.1	Official default. Released custom-data path ( demo_inference.py , FLOW_THRESH ) with use_scene_flow_mask_refined . Its gates are kept: MoGe mask eroded $3\times$ , $\pm 0.5\sigma$ depth consistency, one dilation.
4RC	Max world displacement $\max_{t}\\|\mathbf{x}(t)-\mathbf{x}(0)\\|$	0.025	Official default. Released track visualizer ( viser_visualizer_track.py , DISP_THRESH ).
MoVieS	Window maximum of predicted motion	0.05	Substituted. The released mask_by_motion value of 0.005 is calibrated for rendered splat flow and floods the frame under window-maximum aggregation. We take 0.05 from an 11-value sweep over $[0.04,0.06]$ , a plateau minimizing full-resolution PER (0.0154 at 0.05 versus 0.1791 at 0.005).
Trace Anything	Control-point variance map, segment-wise log-Otsu	Auto	Automatic. Released inference protocol ( infer.py ); no manual parameter.
VGGT4D	QK-clustering score, window-wise adaptive multi-Otsu, then RefineDynMask pruning	Auto	Automatic. Released demo pipeline; no manual parameter.
UniQuery4R (backbone)	warp3d-delta magnitude	0.005	Swept. A smooth sweep over $[0.0005,0.02]$ peaks at the same value under both ground-truth variants of Table 1 .

Appendix

Table 7: Scene-flow-to-mask conversion provenance. “Auto” rows follow the released pipeline and expose no hand-set threshold.

Precision
$\|\mathcal{T}\|$	2	4	8	16	32
DAVIS16-M	87.4	87.1	86.6	86.0	85.2
DAVIS17-M	88.2	87.8	87.3	86.8	85.9
SegTrackv2 ( Li et al., 2013 )	80.8	79.8	77.8	75.1	72.6
FBMS-59	89.8	89.5	89.2	89.0	89.0
MoCA	87.5	86.0	83.2	80.3	76.6

Appendix

Table 8: Union over targets , on the five conventional MOS benchmarks. Recall is monotone in

|\mathcal{T}|

by construction; precision is not.

Low-level branch
Variant	wJF $\uparrow$	PER $\downarrow$ ( $\times 10^{-4}$ )	FER $\downarrow$ (%)
Default (MoRSe)	77.39 $\pm$ 0.85	0.273 $\pm$ 0.263	1.45 $\pm$ 0.99
ResNet-18	77.27 $\pm$ 0.70	0.348 $\pm$ 0.049	2.07 $\pm$ 0.51
ResNet-34	77.41 $\pm$ 0.74	1.448 $\pm$ 1.605	2.64 $\pm$ 0.90
ResNet-50	78.23 $\pm$ 0.55	2.428 $\pm$ 2.985	2.80 $\pm$ 1.46
removed	76.38 $\pm$ 0.03	1.898 $\pm$ 1.212	2.52 $\pm$ 0.68

Appendix

Table 9: Per-seed spread behind Table 4 . Mean

\pm

std over two seeds; the default row averages repeated runs.

Model	Total params. (M) $\!\downarrow$	Precompute (ms) $\!\downarrow$	Core (ms) $\!\downarrow$	Total (ms) $\!\downarrow$	FPS $\!\uparrow$	Peak mem. (GB) $\!\downarrow$
MoRSe	1,651	30.5	42.4	72.9	13.7	9.4

Appendix

Table 10: End-to-end inference cost on one A800-80GB at bf16. Precompute is joint encoding and target-memory construction, amortized over a four-frame window; Core is coordinate decoding and mask assembly.

Setting	Precompute (ms) $\!\downarrow$	Core (ms) $\!\downarrow$	Total (ms) $\!\downarrow$	FPS $\!\uparrow$
Processed shape ( $W_{\mathrm{enc}}\!=\!4$ , stride 4)
$280\!\times\!504$	26.1	35.4	0 61.5	16.3
$504\!\times\!280$	26.2	35.4	0 61.7	16.2
$\mathbf{378\!\times\!504}$	33.5	47.1	0 80.5	12.4
$406\!\times\!504$	35.2	50.6	0 85.8	11.7
$504\!\times\!504$	42.9	62.7	105.6	0 9.5

Appendix

Table 11: One factor varied at a time , on one A800-80GB at bf16. Default setting in bold; query counts in the stride block are per frame.

Figure 5: Failure on small-displacement motion. Ground truth (green, left) and MoRSe prediction (red, right) over five consecutive frames. Top: a swaying branch. Bottom: a small animal behind a rock.

Figure 6: Qualitative results on MoRSe-5K. Left: source frames with their ground-truth moving regions (green). Right: MoRSe predictions (red). Within a group the source is shared and the target changes.

Figure 7: Qualitative results on the conventional MOS benchmarks. Columns alternate input and prediction. Top: MoRSe. Bottom (dashed): the same sources under MoRSe (left) and MoRSe $+$ SAM 3 (right).

Motion Segmentation

See all 17 papers

May 28, 2026

GMOS: Grounding Moving Object Segmentation in 3D Space and Time

Moving Object Segmentation (MOS) aims to discover, segment, and track objects that move independently of the camera. Current MOS methods, however, exhibit two fundamental limitations: they rely on pre-computed 2D auxiliary modalities such as optical flow or point trajectories that lack 3D geometric information, and they treat motion as a sequence-level attribute, overlooking the instantaneous motion state of each object. We address both by grounding MOS in 3D space and time, and propose GMOS, a framework that operates directly on RGB video to produce 3D-aware, temporally fine-grained segmentation of multiple moving objects, alongside a foreground--background variant GMOS-S for faster deployment. To support training and evaluation in this regime, we curate GMOS-2K, a dataset of 2,210 real-world videos with per-object temporal motion annotations drawn from five established Video Object Segmentation (VOS) benchmarks, and formalise MOS-I ("I" for instantaneous), a temporally fine-grained evaluation protocol with three complementary metrics. GMOS achieves state-of-the-art results across MOS, MOS-I, and Unsupervised VOS benchmarks, while running significantly faster than prior multi-object MOS methods and supporting online inference for streaming deployment.

Junyu Xie, Tengda Han, Weidi Xie +1

Visual Geometry Group, Department of Engineering Science, University of Oxford, UK · SAI, Shanghai Jiao Tong University, China

Video Object Segmentation Multi-Object Tracking

Ask PDF

Feb 25, 2026

GeoMotion: Rethinking Motion Segmentation via Latent 4D Geometry

Motion segmentation in dynamic scenes is highly challenging, as conventional methods heavily rely on estimating camera poses and point correspondences from inherently noisy motion cues. Existing statistical inference or iterative optimization techniques that struggle to mitigate the cumulative errors in multi-stage pipelines often lead to limited performance or high computational cost. In contrast, we propose a fully learning-based approach that directly infers moving objects from latent feature representations via attention mechanisms, thus enabling end-to-end feed-forward motion segmentation. Our key insight is to bypass explicit correspondence estimation and instead let the model learn to implicitly disentangle object and camera motion. Supported by recent advances in 4D scene geometry reconstruction (e.g.,

π^3

), the proposed method leverages reliable camera poses and rich spatial-temporal priors, which ensure stable training and robust inference for the model. Extensive experiments demonstrate that by eliminating complex pre-processing and iterative refinement, our approach achieves state-of-the-art motion segmentation performance with high efficiency. The code is available at:https://github.com/zjutcvg/GeoMotion.

Xiankang He, Peile Lin, Ying Cui +3

College of Computer Science and Technology, Zhejiang University of Technology · Zhejiang Key Laboratory of Visual Information Intelligent Processing · State Key Lab of CAD & CG, Zhejiang University

Motion Segmentation

Ask PDF

Sep 13, 2026

DynEoMT: Learning Object Dynamicity from Online Segmentation Queries

Video segmentation models recognize and track objects over time, but they do not indicate whether each segmented region moves independently of the observing camera. This dynamicity attribute cannot be inferred from semantics alone and is confounded by camera ego-motion. We introduce \method, an online framework that augments query-based video segmentation with region-level dynamicity prediction. It jointly produces the original segmentation outputs and a dynamic or static state for each predicted region. At inference, DynEoMT uses only the current frame and propagated queries, without optical flow, depth, camera pose, previous RGB frames, or feature maps. Because established video segmentation benchmarks do not annotate this attribute, we also introduce a class-agnostic offline supervision pipeline using camera-compensated optical flow and confidence-aware temporal filtering. Across VIPSeg, OVIS, YouTube-VIS 2022, and VSPW, DynEoMT achieves balanced accuracies of 84.3, 68.0, 68.6, and 87.6, respectively, while largely preserving segmentation performance. These results show that segmentation-region dynamicity can be learned from propagated queries, enabling its online prediction without a dedicated motion-processing pipeline at inference. The complete code will be released as open source to enable full reproduction of the method and experiments.

Calvin Galagain, Martyna Poreba, François Goulette

Université Paris-Saclay, CEA, List, F-91120 Palaiseau, France · U2IS, ENSTA Paris, Institut Polytechnique de Paris, 91120, Palaiseau, France

Video Object Segmentation Video Instance Segmentation

Ask PDF

Abstract

Motion Segmentation

Explore similar work

GMOS: Grounding Moving Object Segmentation in 3D Space and Time

GeoMotion: Rethinking Motion Segmentation via Latent 4D Geometry

DynEoMT: Learning Object Dynamicity from Online Segmentation Queries