SceneScribe-1M: A Large-Scale Video Dataset with Comprehensive Geometric and Semantic Annotations

Yunnan Wang; Kecheng Zheng; Jianyuan Wang; Minghao Chen; David Novotny; Christian Rupprecht; Yinghao Xu; Xing Zhu

cs.CVApr 9, 2026

SceneScribe-1M: A Large-Scale Video Dataset with Comprehensive Geometric and Semantic Annotations

Authors: Yunnan Wang, Kecheng Zheng, Jianyuan Wang, Minghao Chen, David Novotny, Christian Rupprecht, Yinghao Xu, Xing Zhu,

Organizations: Shanghai Jiao Tong University · Ant Group · Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Zhejiang Key Laboratory of Industrial Intelligence and Digital Twin ·

Abstract

The convergence of 3D geometric perception and video synthesis has created an unprecedented demand for large-scale video data that is rich in both semantic and spatio-temporal information. While existing datasets have advanced either 3D understanding or video generation, a significant gap remains in providing a unified resource that supports both domains at scale. To bridge this chasm, we introduce SceneScribe-1M, a new large-scale, multi-modal video dataset. It comprises one million in-the-wild videos, each meticulously annotated with detailed textual descriptions, precise camera parameters, dense depth maps, and consistent 3D point tracks. We demonstrate the versatility and value of SceneScribe-1M by establishing benchmarks across a wide array of downstream tasks, including monocular depth estimation, scene reconstruction, and dynamic point tracking, as well as generative tasks such as text-to-video synthesis, with or without camera control. By open-sourcing SceneScribe-1M, we aim to provide a comprehensive benchmark and a catalyst for research, fostering the development of models that can both perceive the dynamic 3D world and generate controllable, realistic video content.

Figures & tables

Type	Dataset	Domain	Dynamic	Sem. Ann.	3D. Ann.	#Scene Clips	#Frames
3D Perception	RealEstate10K [ 73 ]	Indoor-Real	✗	N/A	C.	80K	10M
	BlendedMVS [ 66 ]	Open-Synthetic	✗	Single Label	D. C.	113	17K
	CO3Dv2 [ 38 ]	Object-Real	✗	Single Label	C.	19K	1.5M
	PointOdyssey [ 72 ]	Object-Synthetic	✔	N/A	D. C. P.	159	200K
	CamVid-30K [ 71 ]	Open-Real	✔	N/A	C.	30K	-
	Multi-Cam Video	Open-Synthetic	✔	Single Label	C.	136K	11M

Table 1: Comparisons with Previous Works . SceneScribe-1M is a large-scale video dataset with comprehensive geometric and semantic annotations. In the Geometric Annotation column, Depth map, Camera Pose, and 3D Tracks are abbreviated as D., C., and P., respectively.

Figure 2: Curation Pipeline for SceneScribe-1M consist of: (a) We begin by collecting large-scale videos from various sources; (b) Raw videos undergo specification and content inspection, with temporal segmentation models employed to ensure continuity; and (c) We integrate Qwen2.5-VL-72B [ 5 ] , MegaSaM [ 32 ] , and TAPIP3D [ 68 ] to perform comprehensive geometric and semantic annotations.

Panel of Figure 3 — Figure 3: Statistics of Raw Video Specification after filtering, including Resolution, Frame Per Second (FPS), and Duration.

Figure 4: Statistics of Raw Video Content after filtering. These charts demonstrate that the raw videos exhibit sufficient diversity of motion while eliminating the lighting interference.

Panel of Figure 5 — Figure 5: Caption Statistics : (a) The average caption length is adequate to capture the details within each scene, and (b) Key words ( e.g., atmosphere , subject , and take place ) effectively cover aspects such as the scene context, primary objects, and actions.

Panel of Figure 6 — Figure 6: Statistics of Object Motion Metrics . It can be observed that both object motion metrics in SceneScribe-MVS after applying the sampling strategy exhibit a greater static degree than the thresholds. This demonstrates that our sampling not only facilitates effective dynamic mask generation within SceneScribe-1M, but also improves control over the proportion of dynamics.

Panel of Figure 7 — Figure 7: Statistics of Camera Motion Metrics . The similar distributions of camera motion metrics in SceneScribe-1M and SceneScribe-MVS indicate that we disentangle camera and object motion, enabling control over object dynamics while preserving camera diversity.

Figure 8: Visualization Results of Downstream Tasks. We conduct various downstream task on SceneScribe-1M, i.e. , MoGe [ 54 ] (monocular depth estimation), VGGT [ 51 ] (3D reconstruction), MonST3R [ 69 ] (4D reconstruction), CoTracker3 [ 24 ] (2D Point Tracking), SpatialTrackerV2 [ 58 ] (3D Point Tracking) and A3CD [ 2 ] . These results highlight the robust applicability of SceneScribe-1M in 3D perception and video generation, offering a unified resource that effectively supports both domains at scale.

Method	NYUv2 [ 42 ]		KITTI [ 47 ]		ETH3D [ 41 ]		iBims-1 [ 26 ]		GSO [ 15 ]		Sintel [ 8 ]		DDAD [ 18 ]		DIODE [ 48 ]		Average
Method	Rel $\downarrow$	$\delta_{1}\uparrow$	Rel $\downarrow$	$\delta_{1}\uparrow$	Rel $\downarrow$	$\delta_{1}\uparrow$	Rel $\downarrow$	$\delta_{1}\uparrow$	Rel $\downarrow$	$\delta_{1}\uparrow$	Rel $\downarrow$	$\delta_{1}\uparrow$	Rel $\downarrow$	$\delta_{1}\uparrow$	Rel $\downarrow$	$\delta_{1}\uparrow$	Rel $\downarrow$	$\delta_{1}\uparrow$
Scale-invariant depth map
Moge (w/o SceneScribe)	3.44	98.4	4.25	97.8	3.36	98.9	3.46	97.0	1.47	100	19.3	73.4	9.17	90.5	4.89	94.7	6.17	93.8
Moge (w SceneScribe-1M)	3.42	98.3	4.13	97.9	3.45	98.7	3.26	98.0	1.47	100	19.6	72.0	8.95	91.5	4.82	95.3	6.14	94.0
Affine-invariant depth map
Moge (w/o SceneScribe)	2.92	98.6	3.94	98.0	2.69	99.2	2.74	97.9	0.94	100	13.0	83.2	8.40	92.1	3.16	97.5	4.72	95.8

Table 2: Evaluation of Monocular Depth Estimation on Representative Benchmarks .

	Pose Estimation		Point Map Estimation
Method	AUC 30 $\uparrow$	AUC 15 $\uparrow$	ACC. $\downarrow$	Comp. $\downarrow$	Overall $\downarrow$
VGGT (w/o SceneScribe-1M)	89.5	83.4	0.873	0.482	0.677
VGGT (w SceneScribe-1M)	89.9	83.8	0.890	0.504	0.697

Table 3: Evaluation of Scene Reconstruction on Representative Benchmarks .

	Kinetics			RGB-S			DAVIS			Mean
Method	AJ $\uparrow$	$\delta_{avg}^{vis}$ $\uparrow$	OA $\uparrow$	AJ $\uparrow$	$\delta_{avg}^{vis}$ $\uparrow$	OA $\uparrow$	AJ $\uparrow$	$\delta_{avg}^{vis}$ $\uparrow$	OA $\uparrow$	$\delta_{avg}^{vis}$ $\uparrow$
CoTracker3 (w/o SceneScribe)	54.7	67.8	87.4	74.3	85.2	92.4	64.4	76.9	91.2	76.6
CoTracker3 (w SceneScribe)	55.5	68.4	88.2	74.9	86.3	92.8	64.5	77.6	92.0	77.4

Table 4: Evaluation of Dynamic Point Tracking on Representative Benchmarks.

Method	TransErr $\downarrow$	RotErr $\downarrow$	FID $\downarrow$	FVD $\downarrow$	CLIP $\uparrow$
AC3D (w/o SceneScribe-1M)	0.374	0.039	1.27	38.20	28.62
AC3D (w SceneScribe-1M)	0.318	0.026	1.19	35.15	29.98

Table 5: Text/Pose-to-Video Evaluation on RealEstate10K [ 73 ] .

T2V Generation

Text-to-video generation creates moving visual scenes from natural-language prompts, with growing control over motion, identity, events, and style.

See all 110 papers

Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding
Mar 19, 2026Xianjin Wu, Dingkang Liang, Tianrui Feng +53D Spatial Reasoning 3D Scene Understanding
Huazhong University of Science and Technology, China · Baidu Inc., China
OmniHuman: A Large-scale Dataset and Benchmark for Human-Centric Video Generation
Apr 20, 2026Lei Zhu, Xing Cai, Yingjie Chen +6Audio-Video Generation Video Generation
Peking University, Beijing, China · WeChat Lab, Tencent, China · Chinese Academy of Sciences, Beijing, China
Syn4D: A Multiview Synthetic 4D Dataset
May 6, 2026Zeren Jiang, Yushi Lan, Yihang Luo +83D Point Tracking 4D Reconstruction
VGG, University of Oxford · Nanyang Technological University · Naver Labs Europe

Mar 19, 2026J/K move · Enter open · S save

Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding

Xianjin Wu, Dingkang Liang, Tianrui Feng, Kui Xia+4

Huazhong University of Science and Technology, China · Baidu Inc., China

3D Spatial Reasoning 3D Scene Understanding 3D VLMs

Open paper AskPDF arXiv

While Multimodal Large Language Models demonstrate impressive semantic capabilities, they often suffer from spatial blindness, struggling with fine-grained geometric reasoning and physical dynamics. Existing solutions typically rely on explicit 3D modalities or complex geometric scaffolding, which are limited by data scarcity and generalization challenges. In this work, we propose a paradigm shift by leveraging the implicit spatial prior within large-scale video generation models. We posit that to synthesize temporally coherent videos, these models inherently learn robust 3D structural priors and physical laws. We introduce VEGA-3D (Video Extracted Generative Awareness), a plug-and-play framework that repurposes a pre-trained video diffusion model as a Latent World Simulator. By extracting spatiotemporal features from intermediate noise levels and integrating them with semantic representations via a token-level adaptive gated fusion mechanism, we enrich MLLMs with dense geometric cues without explicit 3D supervision. Extensive experiments across 3D scene understanding, spatial reasoning, and embodied manipulation benchmarks demonstrate that our method outperforms state-of-the-art baselines, validating that generative priors provide a scalable foundation for physical-world understanding. Code is publicly available at https://github.com/H-EmbodVis/VEGA-3D.

Abstract

T2V Generation

Explore similar work

Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding

OmniHuman: A Large-scale Dataset and Benchmark for Human-Centric Video Generation

Syn4D: A Multiview Synthetic 4D Dataset