Moving Regions, Not Moving Objects: Segmenting What Actually Moves under Camera Motion

Wenhua Jin; Tiancheng Chen; Sheng Tang; Jiahao Li; Yupeng Liu; Weiqi Zhang; Juntong Fang; Zesong Li

cs.CVOct 22, 2026

Moving Regions, Not Moving Objects: Segmenting What Actually Moves under Camera Motion

Authors: Wenhua Jin, Tiancheng Chen, Sheng Tang, Jiahao Li, Yupeng Liu, Weiqi Zhang, Juntong Fang, Zesong Li

Organizations: Kosmo Research · Automotive Engineering Department, Jilin University · Xiamen University · School of Software, Tsinghua University

Abstract

Moving object segmentation is conventionally formulated at the sequence and object level, marking an entire instance as moving once it moves anywhere in a video. Recent instantaneous formulations distinguish moving from stationary intervals, but still generate masks at the object level through frozen segmentation foundation models. Because the mask is derived from an instance-level proposal, motion is typically attributed to the entire instance rather than localized to the regions that actually move. Moreover, such instance-level predictions can produce masks even when no part of the instance is actually moving, revealing a more fundamental issue: motion is not an intrinsic property of an instance; it can only be determined by comparing at least two frames. Building on this premise, we introduce MoRSe, which jointly interprets two frames to predict a moving-region mask, distinguishing actual scene motion from apparent motion induced by the camera. A single RGB encoder replaces multi-stage foundation stacks, so no optical flow, point tracks, or separate mask generator is needed at inference. We release MoRSe-5K, a benchmark of 42 sequences with Static, Dynamic-Static, and Dynamic settings designed to evaluate the model's ability to distinguish and localize moving and stationary regions under different camera and scene motion conditions. MoRSe outperforms both specialized moving object segmentation methods and dense 4D reconstruction methods with motion-thresholded predictions, while producing the fewest false positives on sequences where nothing moves.

Figures & tables

Figure 2: MoRSe architecture. Both frames pass jointly through a UniQuery4R-initialized ViT and a shallow convolutional branch; the two are fused and split into source and target feature pyramids. A camera head reads the ViT camera tokens and predicts per-view pose and field of view as auxiliary outputs that never enter the mask path. Sampled source coordinates form the decoder queries; target features provide keys and values, and four cross-attention blocks refine each query before a shared MLP predicts the motion logit.

Figure 3: Inference granularity. Target selection sets the granularity without changing the weights: one pair, a target window, or a promptable segmenter.

		Static		Dynamic-Static $+$ Dynamic
				Moving region			Moving object
Method	Native output	PER $\!\downarrow$	FER $\!\downarrow$	$\mathcal{J}\!\uparrow$	$\mathcal{F}\!\uparrow$	$\mathcal{J}\&\mathcal{F}\!\uparrow$	$\mathcal{J}\!\uparrow$	$\mathcal{F}\!\uparrow$	$\mathcal{J}\&\mathcal{F}\!\uparrow$
Dense 4D reconstruction: predict scene flow, converted to a mask by thresholding
Trace Anything †	trajectory field	21.01	100.00	32.10	35.90	34.00	30.18 ∗	38.23 ∗	34.21 ∗
Any4D †	scene flow	0 0.46	0 18.70	27.30	29.80	28.60	33.80 ∗	38.28 ∗	36.04 ∗
4RC †	scene flow	0 0.53	0 18.20	40.50	41.50	41.00	39.56 ∗	43.69 ∗	41.62 ∗

Table 1: Main results on MoRSe-5K (%). Metrics and the two ground-truth variants follow Section 4.3 . “Scene flow” rows are thresholded into masks by each model’s own released or automatic criterion, frozen before evaluation and never re-tuned per variant.

\dagger

: matched-window rerun from released code.

\ast

: the moving-object score comes from the shared SAM 3 readout; unmarked rows are scored on their own instance output. best , second , third per column.

Figure 4: Moving regions under a moving camera. Top (Static): the camera moves while the person stands still; every baseline paints the whole person, while MoRSe leaves the frame empty. Bottom (Dynamic-Static): only the child’s hand moves; MoRSe marks that hand alone, whereas the baselines cover the whole child.

$\|\mathcal{T}\|$	Precision $\uparrow$	Recall $\uparrow$	$\mathcal{J}\&\mathcal{F}\uparrow$
2	86.7	80.4	72.1
4	86.0	82.2	72.9
8	84.8	83.3	73.0

Table 2: Union-mask inference. Mean precision, recall, and

\mathcal{J}\&\mathcal{F}

over five benchmarks when masks from

|\mathcal{T}|

target frames are unioned per source (Equation 1 ).

	DAVIS16-M			FBMS-59			MoCA
Method	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$
OCLR † ( Xie et al., 2022 )	70.25	69.60	69.92	61.71	62.87	62.29	12.08	17.86	14.97
ABR † ( Xie et al., 2024a )	74.02	71.81	72.91	73.48	75.32	74.40	0 8.80	14.71	11.75
FlowSAM † ( Xie et al., 2024b )	37.56	38.18	37.87	40.00	41.89	40.95	29.81	24.54	27.17
SegAnyMo † ( Huang et al., 2025 )	75.89	80.82	78.35	68.80	76.07	72.44	58.81	66.10	62.46
RoMo † ( Goli et al., 2025 )	70.11	73.87	71.99	58.86	61.07	59.96	11.50	12.39	11.94

Table 3: Comparison with moving object segmentation (MOS) baselines. Conventional foreground–background MOS under one protocol across three benchmarks, with instance outputs unioned before evaluation. Every baseline is a controlled rerun from released code, so the numbers are mutually comparable but need not match the original papers. Within each column, best , second best , and third best .

\ddagger

: MoRSe emits a moving region rather than instances, so its row is shown in grey for reference only and is left out of the ranking; the object-level entry is MoRSe

+

SAM 3.

Low-level branch
Variant	wJF $\uparrow$	PER $\downarrow$	FER $\downarrow$
		( $\times 10^{-4}$ )	(%)
Default (MoRSe)	77.39	0.273	1.45
ResNet-18	77.27	0.348	2.07
ResNet-34	77.41	1.448	2.64
ResNet-50	78.23	2.428	2.80

Table 4: Ablations on MoRSe-5K (%). Each row changes one component. Entries average two seeds; the default row averages repeated runs, unlike its single run in Table 1 . All ablations are trained on a single A800 GPU. Per-seed spread (

\pm

std) in Table 9 .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

3D point tracking — 19 entries, 137,103 sequences, 19,260,636 frames; 92.3% of training samples
Source	Seq.	Frames	Source	Seq.	Frames
PointOdyssey	111	207,968	Kubric4D	808	48,480
CoTracker3-Kubric	5,827	699,240	Flock4D	950	434,003
TAPIP3D-Kubric	10,950	262,800	DynamicReplica	8	2,400
SynthVerse	26,187	4,934,809	Stereo4D	81,962	12,128,259
Syn4D	565	168,787	Waymo	5,497	131,928

Appendix

Table 5: Training corpus inventory. Family headers carry the normalized sampling-weight share. LASIESTA and MannequinChallenge are listed for evaluation provenance and excluded from training.

Group	Quantity	Value
Query sampler	Motion-region share $\alpha_{m}$	$1/6$ (171 of 1,024)
	Boundary-band share $\alpha_{b}$	0 (band disabled)
	Static/background share $\alpha_{h}$	$5/6$ (uniform, static first)
Objectives	Dice weight $\lambda_{\mathrm{dice}}$	0.5
	BCE weight $\lambda_{\mathrm{bce}}$	0.5
	Mask loss scale $\lambda_{\mathrm{mask}}$	3.0

Appendix

Table 6: Hyperparameters and protocol constants.

Method	Criterion	Value	Provenance and rationale
Any4D	Scene-flow magnitude	0.1	Official default. Released custom-data path ( demo_inference.py , FLOW_THRESH ) with use_scene_flow_mask_refined . Its gates are kept: MoGe mask eroded $3\times$ , $\pm 0.5\sigma$ depth consistency, one dilation.
4RC	Max world displacement $\max_{t}\\|\mathbf{x}(t)-\mathbf{x}(0)\\|$	0.025	Official default. Released track visualizer ( viser_visualizer_track.py , DISP_THRESH ).
MoVieS	Window maximum of predicted motion	0.05	Substituted. The released mask_by_motion value of 0.005 is calibrated for rendered splat flow and floods the frame under window-maximum aggregation. We take 0.05 from an 11-value sweep over $[0.04,0.06]$ , a plateau minimizing full-resolution PER (0.0154 at 0.05 versus 0.1791 at 0.005).
Trace Anything	Control-point variance map, segment-wise log-Otsu	Auto	Automatic. Released inference protocol ( infer.py ); no manual parameter.
VGGT4D	QK-clustering score, window-wise adaptive multi-Otsu, then RefineDynMask pruning	Auto	Automatic. Released demo pipeline; no manual parameter.
UniQuery4R (backbone)	warp3d-delta magnitude	0.005	Swept. A smooth sweep over $[0.0005,0.02]$ peaks at the same value under both ground-truth variants of Table 1 .

Appendix

Table 7: Scene-flow-to-mask conversion provenance. “Auto” rows follow the released pipeline and expose no hand-set threshold.

Precision
$\|\mathcal{T}\|$	2	4	8	16	32
DAVIS16-M	87.4	87.1	86.6	86.0	85.2
DAVIS17-M	88.2	87.8	87.3	86.8	85.9
SegTrackv2 ( Li et al., 2013 )	80.8	79.8	77.8	75.1	72.6
FBMS-59	89.8	89.5	89.2	89.0	89.0
MoCA	87.5	86.0	83.2	80.3	76.6

Appendix

Table 8: Union over targets , on the five conventional MOS benchmarks. Recall is monotone in

|\mathcal{T}|

by construction; precision is not.

Low-level branch
Variant	wJF $\uparrow$	PER $\downarrow$ ( $\times 10^{-4}$ )	FER $\downarrow$ (%)
Default (MoRSe)	77.39 $\pm$ 0.85	0.273 $\pm$ 0.263	1.45 $\pm$ 0.99
ResNet-18	77.27 $\pm$ 0.70	0.348 $\pm$ 0.049	2.07 $\pm$ 0.51
ResNet-34	77.41 $\pm$ 0.74	1.448 $\pm$ 1.605	2.64 $\pm$ 0.90
ResNet-50	78.23 $\pm$ 0.55	2.428 $\pm$ 2.985	2.80 $\pm$ 1.46
removed	76.38 $\pm$ 0.03	1.898 $\pm$ 1.212	2.52 $\pm$ 0.68

Appendix

Table 9: Per-seed spread behind Table 4 . Mean

\pm

std over two seeds; the default row averages repeated runs.

Model	Total params. (M) $\!\downarrow$	Precompute (ms) $\!\downarrow$	Core (ms) $\!\downarrow$	Total (ms) $\!\downarrow$	FPS $\!\uparrow$	Peak mem. (GB) $\!\downarrow$
MoRSe	1,651	30.5	42.4	72.9	13.7	9.4

Appendix

Table 10: End-to-end inference cost on one A800-80GB at bf16. Precompute is joint encoding and target-memory construction, amortized over a four-frame window; Core is coordinate decoding and mask assembly.

Setting	Precompute (ms) $\!\downarrow$	Core (ms) $\!\downarrow$	Total (ms) $\!\downarrow$	FPS $\!\uparrow$
Processed shape ( $W_{\mathrm{enc}}\!=\!4$ , stride 4)
$280\!\times\!504$	26.1	35.4	0 61.5	16.3
$504\!\times\!280$	26.2	35.4	0 61.7	16.2
$\mathbf{378\!\times\!504}$	33.5	47.1	0 80.5	12.4
$406\!\times\!504$	35.2	50.6	0 85.8	11.7
$504\!\times\!504$	42.9	62.7	105.6	0 9.5

Appendix

Table 11: One factor varied at a time , on one A800-80GB at bf16. Default setting in bold; query counts in the stride block are per frame.

Figure 5: Failure on small-displacement motion. Ground truth (green, left) and MoRSe prediction (red, right) over five consecutive frames. Top: a swaying branch. Bottom: a small animal behind a rock.

Figure 6: Qualitative results on MoRSe-5K. Left: source frames with their ground-truth moving regions (green). Right: MoRSe predictions (red). Within a group the source is shared and the target changes.

Figure 7: Qualitative results on the conventional MOS benchmarks. Columns alternate input and prediction. Top: MoRSe. Bottom (dashed): the same sources under MoRSe (left) and MoRSe $+$ SAM 3 (right).

Motion Segmentation

See all 17 papers

GMOS: Grounding Moving Object Segmentation in 3D Space and Time
May 28, 2026Junyu Xie, Tengda Han, Weidi Xie +1Video Object Segmentation Multi-Object Tracking
Visual Geometry Group, Department of Engineering Science, University of Oxford, UK · SAI, Shanghai Jiao Tong University, China
GeoMotion: Rethinking Motion Segmentation via Latent 4D Geometry
Feb 25, 2026Xiankang He, Peile Lin, Ying Cui +3Motion Segmentation
College of Computer Science and Technology, Zhejiang University of Technology · Zhejiang Key Laboratory of Visual Information Intelligent Processing · State Key Lab of CAD & CG, Zhejiang University
DynEoMT: Learning Object Dynamicity from Online Segmentation Queries
Sep 13, 2026Calvin Galagain, Martyna Poreba, François GouletteVideo Object Segmentation Video Instance Segmentation
Université Paris-Saclay, CEA, List, F-91120 Palaiseau, France · U2IS, ENSTA Paris, Institut Polytechnique de Paris, 91120, Palaiseau, France

May 28, 2026J/K move · Enter open · S save

GMOS: Grounding Moving Object Segmentation in 3D Space and Time

Junyu Xie, Tengda Han, Weidi Xie, Andrew Zisserman

Visual Geometry Group, Department of Engineering Science, University of Oxford, UK · SAI, Shanghai Jiao Tong University, China

Video Object Segmentation Multi-Object Tracking

Open paper AskPDF arXiv

Moving Object Segmentation (MOS) aims to discover, segment, and track objects that move independently of the camera. Current MOS methods, however, exhibit two fundamental limitations: they rely on pre-computed 2D auxiliary modalities such as optical flow or point trajectories that lack 3D geometric information, and they treat motion as a sequence-level attribute, overlooking the instantaneous motion state of each object. We address both by grounding MOS in 3D space and time, and propose GMOS, a framework that operates directly on RGB video to produce 3D-aware, temporally fine-grained segmentation of multiple moving objects, alongside a foreground--background variant GMOS-S for faster deployment. To support training and evaluation in this regime, we curate GMOS-2K, a dataset of 2,210 real-world videos with per-object temporal motion annotations drawn from five established Video Object Segmentation (VOS) benchmarks, and formalise MOS-I ("I" for instantaneous), a temporally fine-grained evaluation protocol with three complementary metrics. GMOS achieves state-of-the-art results across MOS, MOS-I, and Unsupervised VOS benchmarks, while running significantly faster than prior multi-object MOS methods and supporting online inference for streaming deployment.

Abstract

Motion Segmentation

Explore similar work

GMOS: Grounding Moving Object Segmentation in 3D Space and Time

GeoMotion: Rethinking Motion Segmentation via Latent 4D Geometry

DynEoMT: Learning Object Dynamicity from Online Segmentation Queries