Object Concepts Emerge from Motion

Boshi Li; Xiaohui Wang; Xiaoyang Wu; Zhichao Li; Ya Yang; Naiyan Wang

cs.CVSep 3, 2026

Object Concepts Emerge from Motion

Authors: Boshi Li, Xiaohui Wang, Xiaoyang Wu, Zhichao Li, Ya Yang, Naiyan Wang

Organizations: Beijing University of Posts and Telecommunications · The University of Hong Kong, Hong Kong · Department of Computer Science, The University of Hong Kong, Hong Kong

Abstract

Object-centric visual representations are important for physical-world perception, but existing visual pretraining methods often capture semantic categories without preserving the identity and coherence of individual instances. We present a biologically inspired framework that learns object-centric representations for single images from raw videos. Our approach uses motion boundaries as a source of object-level grouping: off-the-shelf optical flow and clustering produce pseudo-instance masks, which supervise a single-image encoder with pixel-level pairwise metric learning. The framework requires neither human annotations nor camera calibration. We first obtain 195 million pseudo-labeled frames from 7,163 hours of driving and web videos, then expand the supervision to 421 million frames with Motion-Verified Self-Training, which combines model proposals with motion evidence. We train encoders up to Swin-H and distill the learned representations into a family of Swin backbones. Across monocular depth estimation, 3D object detection, 3D occupancy prediction, and end-to-end planning, the resulting models achieve competitive or superior performance relative to supervised and self-supervised pretraining baselines, with particularly strong transfer on geometry- and instance-sensitive tasks. These results show that motion-derived supervision can teach static image encoders to represent visual instances, providing a complementary direction for scalable visual pretraining.

Figures & tables

Fig. 2: Cycle-1 motion-derived object supervision. The first row converts raw video frames into pseudo-instance labels: a flow estimator produces optical flow, and pixel clustering over reliable flow regions yields pseudo-instance labels. The second row trains an image encoder from these labels: a single image is encoded into dense features and optimized with pairwise metric learning, pulling pixels from the same pseudo-instance together and pushing pixels from different instances apart.

Panel of Figure 2 — Fig. 3: Examples of Cycle-1 pseudo-label generation. We estimate optical flow from neighboring frames and cluster reliable pixels with coherent motion into category-agnostic pseudo-instance labels. Colors distinguish motion clusters, not semantic categories.

Fig. 4: Motion-Verified Self-Training. We freeze the Cycle-1 Swin-H encoder and train a MaskFormer head on Cycle-1 pseudo-labels. The resulting proposal model generates candidate masks, which are not accepted directly: optical flow is reused as an independent signal to refine and verify them, producing Cycle-2 pseudo-labels and expanding the training set from 195M to 421M pseudo-labeled frames on the same video sources.

Panel of Figure 4 — Fig. 5: Qualitative visualization of the Cycle-2 motion-guided refinement and verification rules. Each row follows a proposal $P$ through adaptive erosion, constrained growth, and flow-based verification. In the growth panel, orange denotes the seed, green denotes added pixels, and cyan denotes flow-magnitude edges. In the boundary-support panel, green and red denote supported and unsupported boundary pixels, respectively; magenta in the interior-conflict panel marks flow edges inside the refined mask. Proposals are accepted when $\rho_{\mathrm{bdry}}\geq 0.4$ and $\rho_{\mathrm{int}}\leq 0.05$ .

Fig. 6: Reference-point similarity maps of our Swin-H encoder. Each panel places four colored reference points on the input image and visualizes, around the image, the cosine-similarity map induced by each point. High responses remain localized around coherent object regions, indicating that the learned features encode instance-level affinity rather than only category-level appearance.

Pretraining	Swin-T			ViT-S/Swin-S			ViT-B/Swin-B			ViT-L/Swin-L			Swin-H
	Abs Rel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$	Abs Rel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$	Abs Rel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$	Abs Rel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$	Abs Rel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$
ImageNet-22K	0.055	2.182	0.969	0.052	2.093	0.973	0.052	2.088	0.975	0.051	2.044	0.977	–	–	–
SimMIM [ 9 ]	–	–	–	–	–	–	0.050	2.031	0.975	0.048	1.941	0.979	–	–	–
Semantic-SAM [ 86 ]	0.055	2.169	0.971	–	–	–	–	–	–	0.049	2.007	0.979	–	–	–
DINOv2 [ 7 ]	–	–	–	0.052	2.153	0.974	0.050	2.081	0.976	0.048	1.971	0.980	–	–	–
DINOv3 [ 56 ]	–	–	–	0.052	2.161	0.974	0.047	1.958	0.981	0.044	1.787	0.986	–	–	–

TABLE I: Monocular depth estimation on the KITTI Eigen split using DCDepth [ 83 ] . Rows denote pretraining methods and columns denote backbone variants. In the shared S/B/L groups, DINO uses ViT backbones, while the other methods use Swin backbones.

Method	Pretraining	Abs Rel $\downarrow$	Sq Rel $\downarrow$	iRMSE $\downarrow$	SILog $\downarrow$
NeW CRFs [ 87 ]	ImageNet sup.	8.37	1.83	11.03	10.39
VA-DepthNet [ 88 ]	ImageNet sup.	7.96	1.66	10.44	9.63
IEBins [ 89 ]	MIM [ 90 ]	7.82	1.60	10.68	9.84
NDDepth [ 91 ]	MIM [ 90 ]	7.75	1.59	10.62	9.62
DCDepth [ 83 ]	Semantic-SAM [ 86 ]	7.83	1.54	10.12	9.60
DCDepth	Ours (Swin-L)	6.44	1.07	8.66	8.07

TABLE II: Monocular depth estimation on the official KITTI online benchmark. Metrics follow the leaderboard convention.

Panel of Figure 8 — Fig. 7: Qualitative comparison of depth geometry under a 1.5 m lateral novel-view rendering. Our Swin-L produces more coherent surfaces and fewer depth artifacts than DINOv2 ViT-L.

Pretraining	Swin-T		Swin-S		Swin-B		Swin-L		Swin-H
	NDS $\uparrow$	mAP $\uparrow$	NDS $\uparrow$	mAP $\uparrow$	NDS $\uparrow$	mAP $\uparrow$	NDS $\uparrow$	mAP $\uparrow$	NDS $\uparrow$	mAP $\uparrow$
ImageNet-22K	51.69	42.12	53.62	45.22	53.98	45.48	54.59	45.91	–	–
SimMIM [ 9 ]	–	–	–	–	54.03	45.18	54.98	46.52	–	–
Ours	51.56	42.49	54.35	45.16	55.69	47.67	55.89	47.59	56.92	48.87

TABLE III: Quantitative evaluation of BEVFormer V2 [ 93 ] on the nuScenes validation set at an input resolution of

1600\times 900

. Rows denote pretraining methods and columns denote Swin backbone variants.

Pretraining	Swin-T		ViT-S/Swin-S		ViT-B/Swin-B		ViT-L/Swin-L
	NDS $\uparrow$	mAP $\uparrow$	NDS $\uparrow$	mAP $\uparrow$	NDS $\uparrow$	mAP $\uparrow$	NDS $\uparrow$	mAP $\uparrow$
ImageNet-22K	47.42	36.34	48.78	38.00	50.42	40.71	50.48	40.09
DINOv2 [ 7 ]	–	–	46.24	34.88	49.08	38.36	51.91	42.05
DINOv3 [ 56 ]	–	–	47.17	36.08	52.36	42.68	56.40	48.30
Ours	46.52	36.45	50.76	40.54	51.94	42.23	52.88	43.26

TABLE IV: Quantitative evaluation of BEVFormer V2 [ 93 ] on the nuScenes validation set at an input resolution of

704\times 256

. Rows denote pretraining methods and columns denote backbone variants. In the shared S/B/L groups, DINOv2 and DINOv3 use ViT backbones, while the other methods use Swin backbones.

Pretraining	Swin-T		ViT-S/Swin-S		ViT-B/Swin-B		ViT-L/Swin-L		Swin-H
	RayIoU $\uparrow$	$\mathrm{RayIoU}_{1\mathrm{m}}\uparrow$	RayIoU $\uparrow$	$\mathrm{RayIoU}_{1\mathrm{m}}\uparrow$	RayIoU $\uparrow$	$\mathrm{RayIoU}_{1\mathrm{m}}\uparrow$	RayIoU $\uparrow$	$\mathrm{RayIoU}_{1\mathrm{m}}\uparrow$	RayIoU $\uparrow$	$\mathrm{RayIoU}_{1\mathrm{m}}\uparrow$
ImageNet-22K	35.50	29.40	36.80	30.40	37.60	31.30	37.60	31.40	–	–
SimMIM [ 9 ]	–	–	–	–	38.00	31.70	38.60	32.60	–	–
DINOv2 [ 7 ]	–	–	35.90	29.50	37.10	31.00	39.00	32.80	–	–
DINOv3 [ 56 ]	–	–	36.73	30.40	38.79	32.65	41.02	34.93	–	–
Ours	36.72	30.50	38.45	32.40	39.41	33.13	40.04	33.99	40.03	33.97

TABLE V: RayIoU and

\mathrm{RayIoU}_{1\mathrm{m}}

evaluation of SparseOcc [ 94 ] on the nuScenes validation set. Rows denote pretraining methods and columns denote backbone variants. In the shared S/B/L groups, DINO uses ViT backbones, while the other methods use Swin backbones.

Backbone	NC $\uparrow$	DAC $\uparrow$	DDC $\uparrow$	TL $\uparrow$	EP $\uparrow$	TTC $\uparrow$	LK $\uparrow$	HC $\uparrow$	EC $\uparrow$	EPDMS $\uparrow$
DA-ViT-L [ 99 ]	98.4	98.6	99.6	99.8	90.5	97.8	97.0	98.3	78.6	90.5
DINOv2-L [ 7 ]	97.9	97.0	99.3	99.7	87.7	97.0	95.5	98.2	77.1	87.2
DINOv3-L [ 56 ]	98.1	97.8	99.5	99.7	89.5	97.4	96.2	98.3	77.3	89.0
Ours, Swin-L	98.6	97.1	99.5	99.7	89.4	98.1	96.2	98.3	77.6	88.9

TABLE VI: Planning results on NAVSIMv2 using the corrected NAVSIM evaluator. Baseline results are from Xu et al. [ 98 ] ; the Ours row is our evaluation. All variants use the same DriveSuprim planning decoder and training configuration and differ only in the visual backbone. NC: no-at-fault collision; DAC: drivable-area compliance; DDC: driving-direction compliance; TL: traffic-light compliance; EP: ego progress; TTC: time-to-collision; LK: lane keeping; HC: history comfort; EC: extended comfort.

Pipeline	Pretraining supervision	Model	KITTI SILog $\downarrow$	nuScenes NDS $\uparrow$	nuScenes RayIoU $\uparrow$
Cycle-1	Motion-derived (195M)	Cycle-1 Swin-H	5.90	56.80	39.71
Cycle-2	Motion-verified (421M)	Ours (Swin-H)	5.69	56.92	40.03

TABLE VII: Complete Cycle-2 pipeline comparison under the Swin-H architecture. The Cycle-2 row reports our final Swin-H and therefore includes the motion-verified pseudo-label set, continued Cycle-2 optimization, and the final same-architecture feature-distillation stage. The pretraining-set sizes count pseudo-labeled frames. Downstream architectures, training schedules, and evaluation protocols are identical between rows.

Variant	Pretraining schedule	Additional exposures	Abs Rel $\downarrow$	Sq Rel $\downarrow$	RMSE $\downarrow$	RMSE log $\downarrow$	$\delta_{1}\uparrow$	SILog $\downarrow$
Cycle-1	50 epochs on Cycle-1 labels	–	0.0509	0.1409	2.0084	0.0769	0.976	6.9709
Cycle-1, longer	75 epochs on Cycle-1 labels	4.88B	0.0512	0.1446	2.0052	0.0773	0.973	6.9885
Cycle-2	50 Cycle-1 + 10 Cycle-2 epochs	4.21B	0.0498	0.1385	1.9766	0.0756	0.978	6.8761

TABLE VIII: Swin-T direct-pretraining control for Cycle-2 supervision on KITTI depth estimation with DCDepth. Additional sample exposures are measured relative to the 50-epoch Cycle-1 reference. The Cycle-1 continuation and Cycle-2 branch use comparable additional pretraining budgets, while all downstream settings are identical. This control intentionally excludes the final Swin-H feature-distillation stage.

Subset	Labels	Inst./frame	Ref. coverage (%)	IoU (%)	Matched frames (%)
Paired frames	Cycle-1	1.6	13.8	83.8	45.0
Paired frames	Cycle-2	2.4	16.6	81.6	54.0
Cycle-2-only frames	Cycle-2	2.1	14.6	81.2	47.5

TABLE IX: Pseudo-label coverage and agreement with independently generated SAM 2 reference masks. Ref. coverage measures the reference foreground covered by pseudo-labels.

Panel of Figure 16 — Fig. 8: Qualitative comparison of Cycle-1 and Cycle-2 pseudo-labels. Each row shows an input frame, its optical flow, flow boundaries used as motion evidence, and the pseudo-labels generated in the two cycles. Conservative flow clustering in Cycle-1 captures only instances supported by sufficiently complete motion regions, whereas Cycle-2 combines model proposals with motion-based verification to recover additional instances. Colors distinguish instances within each label panel and do not denote semantic categories.

Ours	DINO	IN	SILog $\downarrow$	Abs Rel $\downarrow$	RMSE $\downarrow$
✓			6.714	0.049	1.958
	✓		7.119	0.052	2.153
		✓	7.455	0.055	2.182
	✓	✓	7.071	0.052	2.117
✓	✓		6.622	0.048	1.948

TABLE X: Feature-level complementarity on the KITTI Eigen split using DCDepth [ 83 ] . Ours and IN use Swin-T, while DINO uses DINOv2 ViT-S. For dual-encoder variants, the two feature maps are concatenated along the channel dimension before the depth decoder.

Object-Centric Representation Learning

Object-Centric Representation Learning studies visual models that organize scenes into distinct objects and their properties, positions, and relations.

See all 89 papers

W2Rep: Learning Visual Representations by Watching the World Change
Sep 28, 2026Wen Huang, Hang Guo, Jiarui Yang +3Video Representation Learning Self-Supervised Visual Representation Learning
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Nankai University, Tianjin, China · Shenzhen University, Shenzhen, China
TT-VidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining
Sep 27, 2026Shih-Ying Yeh, Daniel Z. Kaplan, Xuehai Wang +3Efficient ViTs Video Representation Learning
National Tsing Hua University · Comfy Org Research · Karolinska Institutet +2
Self-Supervised Learning of Structured Dynamics from Videos
Jul 23, 2026Lukas Knobel, Andrew Zisserman, Yuki M. AsanoVideo Representation Learning Self-Supervised Visual Representation Learning
Fundamental AI Lab, UTN · VGG, University of Oxford

Sep 28, 2026J/K move · Enter open · S save

W2Rep: Learning Visual Representations by Watching the World Change

Wen Huang, Hang Guo, Jiarui Yang, Zheng Liu+2

Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Nankai University, Tianjin, China · Shenzhen University, Shenzhen, China

Video Representation Learning Self-Supervised Visual Representation Learning Temporal Representation Learning

Open paper AskPDF arXiv

Images capture the world at one moment, whereas video reveals how it changes. Image self-supervision learns spatial structure from a single moment, while video methods commonly learn temporal relationships inside a representation computed jointly from several frames. We ask whether watching a scene change can instead improve features available from one image without sacrificing the ability to represent video. We introduce W2Rep, a masked feature-prediction framework in which an independently encoded source image participates in prediction at the same or another moment. The predictor is conditioned on visible video context, the queried location, and the signed time interval between source and target. This gives the cross-frame objective two complementary roles: the image path learns features that remain useful across time, while the video path must gather evidence that is missing from the source image. Across model scales and downstream tasks, W2Rep improves frozen and fine-tuned recognition under our comparison protocol, while joint video encoding provides further gains over frame-wise aggregation. Controlled experiments show that these gains depend on directly updating the source-image features and on using both video context and temporal displacement. Overall, change across a video can supervise a visual encoder whose representations remain useful at either image or video granularity. Code is available at~https://wenooi.github.io/W2Rep.

Abstract

Object-Centric Representation Learning

Explore similar work

W2Rep: Learning Visual Representations by Watching the World Change

TT-VidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining

Self-Supervised Learning of Structured Dynamics from Videos