Organizations: State Key Laboratory of General Artificial Intelligence, BIGAI · Beijing University of Posts and Telecommunications · Peking University · Beijing Institute of Technology · Tsinghua University
Annotated 3D scene data is scarce and expensive to acquire, while abundant unlabeled videos are readily available on the internet. In this paper, we demonstrate that carefully designed data engines can leverage web-curated, unlabeled videos to automatically generate training data, to facilitate end-to-end models in 3D scene understanding alongside human-annotated datasets. We identify and analyze bottlenecks in automated data generation, revealing critical factors that determine the efficiency and effectiveness of learning from unlabeled data. To validate our approach across different perception granularities, we evaluate on three tasks spanning low-level perception, i.e., 3D object detection and instance segmentation, to high-evel reasoning, i.e., 3D spatial Visual Question Answering (VQA) and Vision-Lanugage Navigation (VLN). Models trained on our generated data demonstrate strong zero-shot performance and show further improvement after finetuning. This demonstrates the viability of leveraging readily available web data as a path toward more capable scene understanding systems.
Figures & tables
Figure 2: Statistics comparison. SceneVerse++ encompasses more scenes, larger areas, and greater object diversity compared with existing real-world datasets.
Figure 3: Overview of data generation. The pipeline leverages a modular design for automatic 3D reconstruction and segmentation.
Figure 4: Reconstruction and segmentation comparison , where SceneVerse++ features a balance in quality and efficiency.
Table 1: Testing SpatialLM on 3D object detection. Performance is reported under different pretraining and finetuning configurations with the same model architecture.
Benchmark
Pretrain
Finetune
AP 25
AP 50
AP
ScanNet
-
ScanNet
36.1
31.8
22.8
SceneVerse++
-
15.4
13.0
8.3
SceneVerse++
ScanNet
38.5
32.9
23.6
Table 2: Testing Mask3D on 3D instance segmentation . It presents reliance on data-specific bias that hurts model scaling.
Table 6
Figure 6: Overview of the VLN data generation pipeline. We construct VLN data from room-tour videos by (i) preprocessing trajectories to eliminate redundant local rotations and segmenting long paths into sub-paths suitable for instruction generation; (ii) converting camera transitions within each sub-path into R2R-style navigation actions; and (iii) generating instructions for each sub-path using VLMs.
Figure 7: Trajectory comparison. Top: Room-tour videos show irregular and redundant camera motions. Middle: R2R trajectories are smooth and goal-directed. Bottom: raw videos are converted into VLN-compatible data. Different colors indicate sub-paths.
Pretrain
Finetune
SR↑
OS↑
SPL↑
Dist↓
PL
-
R2R
0.088
0.133
0.076
8.031
5.222
R2R + SceneVerse++
-
0.188
0.262
0.150
8.117
10.496
SceneVerse++
–
0.107
0.194
0.074
9.418
14.097
SceneVerse++
R2R
0.228
0.315
0.191
7.65
11.642
SceneVerse++ (w/o IE)
–
0.022
0.043
0.016
8.978
2.333
SceneVerse++ (w/o IE)
R2R
0.074
0.111
0.062
8.175
5.009
Table 4: VLN evaluation under different training settings . TR denotes trajectory refinement, and IE for instruction enrichment.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S.1 : Example of quality check. The data samples from SceneVerse++ and ScanNet are mixed and anonymous.
Criterion
SceneVerse++
ScanNet
Scene Item Richness
4.43
3.68
Scene Reconstruction Completeness
4.25
3.09
Object Reconstruction Completeness
4.16
3.23
Object Segmentation Completeness
3.93
3.26
Object Segmentation Granularity
3.89
3.24
Average
4.13
3.30
Appendix
Table S.1 : The quality check of SceneVerse++ and ScanNet.
Figure S.2 : Overview of data curation pipeline .
Figure S.3 : Qualitative example of sensitivity tests on different segment distributions. We evaluate the model trained with k=0.01 and s=20 on other segment settings. The first row shows the input segments, and the second row shows the 3D instance prediction. As the segments become smaller, over-segmentation gradually appears (highlighted by the blue boxes). Conversely, as the segments become larger, under-segmentation becomes increasingly evident (see the red boxes). The AP is reported as the average over the whole ScanNet.
kThresh
segMinVerts
AP 25
AP 50
AP
10−2
20
36.1
31.8
22.8
10−1
20
34.6
28.1
18.4
10−3
20
35.9
30.4
21.3
10−2
100
30.8
24.6
15.8
10−2
500
17.9
12.6
7.2
10−1
1000
11.2
7.7
4.2
Appendix
Table S.2: Evaluation sensitivity on different segment settings on 3D instance segmentation. The model is trained with kThresh= 10−2 and segMinVerts=20, and performance degrades if the distribution of the testing segments diverges from training.
Task Type
Count
Object Relative Direction
155,199
Object Absolute Distance
137,397
Object Relative Distance
226,639
Object Size Estimate
44,050
Object Count
53,200
Route Plan
9,588
Appendix
Table S.3 : 3D Spatial VQA Data Distribution.
Category
Setting
Hardware
4 × NVIDIA A100 GPUs
Precision
BF16
LoRA Rank
128
LoRA Scaling Factor
256
Per-device Batch Size
1
Gradient Accumulation Steps
32
Appendix
Table S.4 : Training Details for 3D Spatial VQA.
Data Source
Setting
SR↑
OS↑
SPL↑
Dist↓
PL
NaVILA
Zero-shot
0.09
0.132
0.08
9.406
8.505
SV++
Zero-shot
0.09
0.145
0.063
9.439
11.274
R2R + NaVILA
Mix
0.29
0.424
0.213
7.960
16.013
R2R + SV++
Mix
0.32
0.402
0.258
7.447
12.918
Appendix
Table S.5 : Comparison between NaVILA and SV++ . Experiments use Qwen2.5-VL-7B under zero-shot and mixed-training.
Figure S.4 : 3D spatial VQA answer distribution.
Figure S.5 : Data scaling effects.
You are an embodied AI agent making task summaries for a navigation task. Your goal is to generate faithful, human-readable navigation instructions. — Input —
• A sequence of first-person images and a stepwise action sequence. • Each image corresponds to the visual observation immediately before the action in that frame. • Alignment is strictly one-to-one: image[i] always pairs with action[i] . • An action entry may describe a single action or a composite action (e.g., “turn left and move forward”), but it still corresponds to the visual state in the paired image. — Core reasoning rule —
• Always rely primarily on visual observations when determining how to move. • Use actions only as fallback when the image is unclear. • Maintain consistent spatial logic: if an object is on the left, turning left should bring it to the center view. — Language and Output Style —
• Avoid first-person narration; use third-person, objective instructions such as “A sofa is on the right; turn right to face it.” • Avoid narrative openings (e.g., “The journey begins…”). • Use direct commands: “Turn right into the hallway.”, “Walk straight past the sofa.”. • Always include all necessary turning/movement instructions. • Mention only key orientation-relevant landmarks (sofa, table, doorway, window). — Responsibilities —
0. Trajectory summarization: • Summarize overall motion, room types, and representative objects. • Briefly describe the starting location. • Provide a concise step-by-step movement description consistent with images. • End with a clear final position description. 1. Per-step reasoning: • Think in first-person as the agent (camera aligned with orientation). • Base reasoning on visible evidence in the current frame. • Mention only representative, orientation-relevant objects. • Use diverse spatial expressions: “to the right”, “just ahead”, “past the table”, etc. • Ensure geometric consistency between viewpoint and actions. • If actions conflict with geometry, trust the image. — Action rules —
• Actions may be single or composite (joined by “and”). • Allowed actions: TurnLeft , TurnRight , MoveForward , Move , Stop . • “Move” alone means a small forward motion without rotation. • Composite actions operate sequentially: turn first, then move. — Special Requirement: Three Reformulations —
Rewrite the trajectory summary into three distinct linguistic styles with identical semantic content: Formal Instructional Style, Natural Conversational Style and Narrative Descriptive Style. Guidelines: • All three must preserve identical spatial logic and landmarks. • No conflicts are allowed. • All must fully cover the entire trajectory. — Examples of the Three Styles — • Instruction 1 (Formal): “Turn right into the hallway. Advance straight past the dining table. Enter the bedroom and stop in front of the bed.” • Instruction 2 (Conversational): “Take a right into the hallway and keep walking until you pass the dining table on your left. Go into the bedroom and stop by the bed.” • Instruction 3 (Narrative): “Turning right, you move into the hallway, the dining table sliding by on your left. The hall opens into a bedroom, where you halt just before the bed.”
Appendix
Table S.6 : Prompts for Navigation instruction generation in SceneVerse++ .
You are an AI assistant tasked with generating Fill-in-the-blank Action Completion MCQ for robot navigation. Your job is to output a multiple-choice question (with blanks) and its correct answer. — Input —
A sequence of continuous key frames from a room-tour video (the frames are consecutive and represent a smooth camera/robot trajectory). — High-level rule (priority order) — 1. ALL reasoning must be grounded purely on the visual evidence from the frames 2. Use visual cues such as object appearance/disappearance, relative positions, scaling, and viewpoint rotation to infer the robot’s movements and turns. 3. When describing places, objects, or targets, use detailed and specific visual anchors — not just generic room names. For example: “the blue sofa on the right,” “the black dining table ahead,” “the kitchen counter with sink,” or “the hallway with a white door at the end.” 4. If any step or turn cannot be confidently inferred from visual evidence, skip or merge it rather than guessing. Do NOT fabricate movements. — Core Procedure (must follow this order) — 1. Construct a concise, numbered Trajectory Summary : • Carefully analyze the continuous frame sequence to extract a minimal yet complete trajectory. • Each entry in the summary should be a single action step, such as: "1. Go forward until [object/room]", "2. Turn left", "3. Go forward until [object/room]". • Determine steps by tracking: – Appearance/disappearance or scaling of landmarks (for “Go forward”) – Change in viewing direction or lateral movement (for “Turn left/right/back”) • The summary should form a coherent navigation sequence from the starting viewpoint to the final destination. • Explicitly describe both start place and end place in visually grounded detail: e.g., “You are a robot beginning at the living room, facing the blue sofa.” e.g., “You want to navigate to the kitchen with a table on your left.” • When describing each “Go forward” anchor, be as specific as visually supported: – Include object appearance (color, size, material), or scene context (e.g., furniture type or nearby area). Example: “Go forward until the blue sofa.” • Only include meaningful transitions — skip redundant minor movements or rotations that don’t correspond to clear spatial change. • Ensure geometric reasoning consistency. • Make the trajectory alternate logically between “Go forward” and “Turn”. • Example of a trajectory summary: – 1. Go forward until the 3-seater sofa (evidence: frames X–Y) – 2. Turn right (evidence: frames X–Y) – 3. Go forward until the dining table (evidence: frames X–Y) – 4. Turn left (evidence: frames X–Y) – 5. Go forward until the kitchen counter with sink (evidence: frames X–Y) 2. QA Generation (based on Trajectory Summary only) : • Normalize steps to alternate between “Go forward…” and “Turn …”. • Determine where to place [please fill in] blanks: – Every turn step must become a blank. • “Go forward” must mention detailed visible landmarks. • Use the strict template: Q: You are a robot beginning at [start place, with visual details]. You want to navigate to [end place, with visual details]. You will perform the following actions: 1. Go forward until [object/room] 2. [please fill in] 3. Go forward until [object/room] ... N. Go forward until [object/room]. You have reached the final destination. (Note: for each [please fill in], choose either ’turn back,’ ’turn left,’ or ’turn right.’)
Appendix
Table S.7 : Prompts to generate route plan VQA in SceneVerse++ - part1.
3. Options generation: • For each blank, permissible options: – ’turn back’, ’turn left’, ’turn right’ • If one blank: produce A–C. • If >= 2 blanks: produce A–D. • Each option is a full sequence of turns. 4. Correct answer determination: • Must match turns inferred from visual trajectory summary. • No guessing ambiguous turns. — Output format (STRICT JSON SCHEMA — all keys required) — • "trajectory summary": a list of strings. • "question": full question string. • "options": dictionary with keys A, B, C, D. • "answer": one correct option. — Strict behaviour notes (must obey) — • Only use frame-based evidence. • Build a reliable Trajectory Summary. • The first step may be a Turn or a Go-forward action. • Never guess ambiguous turns.
Appendix
Table S.8 : Prompts to generate route plan VQA in SceneVerse++ - part2
Large language models are increasingly being applied to 3D scene understanding. However, existing 3D-aware LLMs largely operate as black boxes: they produce bounding boxes or textual responses without exposing the reasoning process, and they typically depend on pre-trained 3D detectors for object proposals. We present Scene-R1, a video-grounded framework for 3D scene reasoning that avoids both pre-trained 3D object detectors and dense point-cloud backbones. The framework combines reinforcement-learning-based reasoning with a two-stage grounding pipeline. In the first stage, Scene-R1 performs temporal grounding by identifying the video segments most relevant to an open-ended query. In the second stage, it grounds the target in image space by predicting a 2D bounding box. This prediction is then refined into a precise 3D localization using high-quality candidates from a zero-shot segmentation module, enabling fine-grained geometric reasoning without detector-based proposals. Scene-R1 can also be adapted to 3D visual question answering, generating free-form answers directly from RGB-D video. Our training pipeline uses lightweight 2D supervision projected from sparse scans, avoiding the computational overhead required to train 3D instance segmentation models. Across multiple datasets, Scene-R1 outperforms existing open-vocabulary baselines while also providing interpretable intermediate reasoning steps. These results suggest that reinforcement-learning-based reasoning over RGB-D video can support accurate 3D scene understanding with task-level supervision. The training, inference, and 3D refinement code is available at https://github.com/819346675/Scene-R1.
Junjie Wang, Zhihao Yuan, Shuyi Jiang +5
FNii-Shenzhen, CUHKSZ · SSE, CUHKSZ · Singapore University of Technology and Design +1
Reliable 3D dynamic perception requires models that can anticipate motion beyond predefined categories, yet progress is hindered by the scarcity of dense, high-quality motion annotations. While self-supervision on unlabeled real data offers a path forward, empirical evidence suggests that scaling unlabeled data fails to close the performance gap due to noisy proxy signals. In this paper, we propose learning robust real-world motion priors entirely from scalable simulation. We introduce SynFlow, a data generation pipeline for large-scale synthetic LiDAR scene flow. Unlike prior works that prioritize sensor-specific realism, SynFlow employs a motion-oriented strategy to synthesize diverse kinematic patterns across 4,000 sequences (∼940k frames), termed SynFlow-4k. This represents a 34× scale-up in annotated volume over existing real-world benchmarks. Our experiments demonstrate that SynFlow-4k provides a highly domain-invariant motion prior. In a zero-shot regime, models trained only on our synthetic data generalize across multiple real-world benchmarks, comparable to in-domain supervised baselines on nuScenes and outperforming state-of-the-art methods on TruckScenes by 31.8%. Furthermore, SynFlow-4k serves as a label-efficient foundation: fine-tuning with only 5% of real-world labels surpasses models trained from scratch on the full available budget. We open-source the pipeline and dataset to facilitate research in generalizable 3D motion estimation. More detail can be found at https://kin-zhang.github.io/SynFlow.
Qingwen Zhang, Xiaomeng Zhu, Chenhan Jiang +1
KTH Royal Institute of Technology, Stockholm, Sweden · Hong Kong University of Science and Technology, Hong Kong, China
High-quality 3D scene assets are critical for embodied applications such as robotic manipulation, navigation, and simulation. Despite their strong object priors, recent single-image 3D generation models such as SAM3D remain insufficient for real-world scenes, where severe occlusions, redundant observations, and cross-view inconsistencies make reliable scene generation challenging. We introduce Scene-SAM3D, a training-free framework that extends SAM3D from single-view object generation to calibrated multi-view scene asset generation. Scene-SAM3D selects a compact set of complementary views, reducing observation redundancy while providing additional evidence for regions occluded in individual views. Based on the selected views, it performs step-efficient latent velocity fusion to integrate multi-view evidence and suppress cross-view conflicts in canonical space. Finally, a lightweight rigid-object Gaussian optimization refines the scene layout within 200 iterations while preserving the generated object geometry. Experiments on Replica and ScanNet++ demonstrate consistent improvements at both instance and scene levels, with our method reducing scene-level CD by 43.8% on Replica and 30.9% on ScanNet++, while cutting flow-model sampling FLOPs and wall-time latency by nearly 20% under the same multi-view setting. Code will be released at https://github.com/xibi777/Scene-SAM3D.
Yuqi Zhang, Yadan Luo, Xiangyu Sun +3
The University of Queensland · Shanghai AI Laboratory · East China Normal University