cs.CVApr 2, 2026
SaveLifting Unlabeled Internet-level Data for 3D Scene Understanding
Organizations: State Key Laboratory of General Artificial Intelligence, BIGAI · Beijing University of Posts and Telecommunications · Peking University · Beijing Institute of Technology ·
Abstract
Annotated 3D scene data is scarce and expensive to acquire, while abundant unlabeled videos are readily available on the internet. In this paper, we demonstrate that carefully designed data engines can leverage web-curated, unlabeled videos to automatically generate training data, to facilitate end-to-end models in 3D scene understanding alongside human-annotated datasets. We identify and analyze bottlenecks in automated data generation, revealing critical factors that determine the efficiency and effectiveness of learning from unlabeled data. To validate our approach across different perception granularities, we evaluate on three tasks spanning low-level perception, i.e., 3D object detection and instance segmentation, to high-evel reasoning, i.e., 3D spatial Visual Question Answering (VQA) and Vision-Lanugage Navigation (VLN). Models trained on our generated data demonstrate strong zero-shot performance and show further improvement after finetuning. This demonstrates the viability of leveraging readily available web data as a path toward more capable scene understanding systems.
Figures & tables
Figure 2: Statistics comparison. SceneVerse++ encompasses more scenes, larger areas, and greater object diversity compared with existing real-world datasets.
Figure 3: Overview of data generation. The pipeline leverages a modular design for automatic 3D reconstruction and segmentation.
Figure 4: Reconstruction and segmentation comparison , where SceneVerse++ features a balance in quality and efficiency.
| Benchmark | Pretrain | Finetune | [email protected] | [email protected] |
| ARKitScenes | SpatialLM | - | 35.1 | 21.2 |
| SceneVerse++ | - | 35.8 | 20.7 | |
| ScanNet | - | ScanNet | 2.9 | 0.7 |
| SpatialLM | - | 29.0 | 19.7 | |
| SceneVerse++ | - | 30.9 | 21.3 | |
| SpatialLM | ScanNet | 38.0 | 28.7 |
Table 1: Testing SpatialLM on 3D object detection. Performance is reported under different pretraining and finetuning configurations with the same model architecture.
| Benchmark | Pretrain | Finetune | AP 25 | AP 50 | AP |
| ScanNet | - | ScanNet | 36.1 | 31.8 | 22.8 |
| SceneVerse++ | - | 15.4 | 13.0 | 8.3 | |
| SceneVerse++ | ScanNet | 38.5 | 32.9 | 23.6 |
Table 2: Testing Mask3D on 3D instance segmentation . It presents reliance on data-specific bias that hurts model scaling.
Figure 6: Overview of the VLN data generation pipeline. We construct VLN data from room-tour videos by (i) preprocessing trajectories to eliminate redundant local rotations and segmenting long paths into sub-paths suitable for instruction generation; (ii) converting camera transitions within each sub-path into R2R-style navigation actions; and (iii) generating instructions for each sub-path using VLMs.
Figure 7: Trajectory comparison. Top: Room-tour videos show irregular and redundant camera motions. Middle: R2R trajectories are smooth and goal-directed. Bottom: raw videos are converted into VLN-compatible data. Different colors indicate sub-paths.
| Pretrain | Finetune | SR↑ | OS↑ | SPL↑ | Dist↓ | PL |
| - | R2R | 0.088 | 0.133 | 0.076 | 8.031 | 5.222 |
| R2R + SceneVerse++ | - | 0.188 | 0.262 | 0.150 | 8.117 | 10.496 |
| SceneVerse++ | – | 0.107 | 0.194 | 0.074 | 9.418 | 14.097 |
| SceneVerse++ | R2R | 0.228 | 0.315 | 0.191 | 7.65 | 11.642 |
| SceneVerse++ (w/o IE) | – | 0.022 | 0.043 | 0.016 | 8.978 | 2.333 |
| SceneVerse++ (w/o IE) | R2R | 0.074 | 0.111 | 0.062 | 8.175 | 5.009 |
Table 4: VLN evaluation under different training settings . TR denotes trajectory refinement, and IE for instruction enrichment.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S.1 : Example of quality check. The data samples from SceneVerse++ and ScanNet are mixed and anonymous.
| Criterion | SceneVerse++ | ScanNet |
| Scene Item Richness | 4.43 | 3.68 |
| Scene Reconstruction Completeness | 4.25 | 3.09 |
| Object Reconstruction Completeness | 4.16 | 3.23 |
| Object Segmentation Completeness | 3.93 | 3.26 |
| Object Segmentation Granularity | 3.89 | 3.24 |
| Average | 4.13 | 3.30 |
Table S.1 : The quality check of SceneVerse++ and ScanNet.
Figure S.2 : Overview of data curation pipeline .
Figure S.3 : Qualitative example of sensitivity tests on different segment distributions. We evaluate the model trained with k=0.01 and s=20 on other segment settings. The first row shows the input segments, and the second row shows the 3D instance prediction. As the segments become smaller, over-segmentation gradually appears (highlighted by the blue boxes). Conversely, as the segments become larger, under-segmentation becomes increasingly evident (see the red boxes). The AP is reported as the average over the whole ScanNet.
| kThresh | segMinVerts | AP 25 | AP 50 | AP |
| 20 | 36.1 | 31.8 | 22.8 | |
| 20 | 34.6 | 28.1 | 18.4 | |
| 20 | 35.9 | 30.4 | 21.3 | |
| 100 | 30.8 | 24.6 | 15.8 | |
| 500 | 17.9 | 12.6 | 7.2 | |
| 1000 | 11.2 | 7.7 | 4.2 |
Table S.2: Evaluation sensitivity on different segment settings on 3D instance segmentation. The model is trained with kThresh= and segMinVerts=20, and performance degrades if the distribution of the testing segments diverges from training.
| Task Type | Count |
| Object Relative Direction | 155,199 |
| Object Absolute Distance | 137,397 |
| Object Relative Distance | 226,639 |
| Object Size Estimate | 44,050 |
| Object Count | 53,200 |
| Route Plan | 9,588 |
Table S.3 : 3D Spatial VQA Data Distribution.
| Category | Setting |
| Hardware | 4 NVIDIA A100 GPUs |
| Precision | BF16 |
| LoRA Rank | 128 |
| LoRA Scaling Factor | 256 |
| Per-device Batch Size | 1 |
| Gradient Accumulation Steps | 32 |
Table S.4 : Training Details for 3D Spatial VQA.
| Data Source | Setting | SR↑ | OS↑ | SPL↑ | Dist↓ | PL |
| NaVILA | Zero-shot | 0.09 | 0.132 | 0.08 | 9.406 | 8.505 |
| SV++ | Zero-shot | 0.09 | 0.145 | 0.063 | 9.439 | 11.274 |
| R2R + NaVILA | Mix | 0.29 | 0.424 | 0.213 | 7.960 | 16.013 |
| R2R + SV++ | Mix | 0.32 | 0.402 | 0.258 | 7.447 | 12.918 |
Table S.5 : Comparison between NaVILA and SV++ . Experiments use Qwen2.5-VL-7B under zero-shot and mixed-training.
Figure S.4 : 3D spatial VQA answer distribution.
Figure S.5 : Data scaling effects.
| You are an embodied AI agent making task summaries for a navigation task. Your goal is to generate faithful, human-readable navigation instructions. — Input — |
| • A sequence of first-person images and a stepwise action sequence. • Each image corresponds to the visual observation immediately before the action in that frame. • Alignment is strictly one-to-one: image[i] always pairs with action[i] . • An action entry may describe a single action or a composite action (e.g., “turn left and move forward”), but it still corresponds to the visual state in the paired image. — Core reasoning rule — |
| • Always rely primarily on visual observations when determining how to move. • Use actions only as fallback when the image is unclear. • Maintain consistent spatial logic: if an object is on the left, turning left should bring it to the center view. — Language and Output Style — |
| • Avoid first-person narration; use third-person, objective instructions such as “A sofa is on the right; turn right to face it.” • Avoid narrative openings (e.g., “The journey begins…”). • Use direct commands: “Turn right into the hallway.”, “Walk straight past the sofa.”. • Always include all necessary turning/movement instructions. • Mention only key orientation-relevant landmarks (sofa, table, doorway, window). — Responsibilities — |
| 0. Trajectory summarization: • Summarize overall motion, room types, and representative objects. • Briefly describe the starting location. • Provide a concise step-by-step movement description consistent with images. • End with a clear final position description. 1. Per-step reasoning: • Think in first-person as the agent (camera aligned with orientation). • Base reasoning on visible evidence in the current frame. • Mention only representative, orientation-relevant objects. • Use diverse spatial expressions: “to the right”, “just ahead”, “past the table”, etc. • Ensure geometric consistency between viewpoint and actions. • If actions conflict with geometry, trust the image. — Action rules — |
| • Actions may be single or composite (joined by “and”). • Allowed actions: TurnLeft , TurnRight , MoveForward , Move , Stop . • “Move” alone means a small forward motion without rotation. • Composite actions operate sequentially: turn first, then move. — Special Requirement: Three Reformulations — |
| Rewrite the trajectory summary into three distinct linguistic styles with identical semantic content: Formal Instructional Style, Natural Conversational Style and Narrative Descriptive Style. Guidelines: • All three must preserve identical spatial logic and landmarks. • No conflicts are allowed. • All must fully cover the entire trajectory. — Examples of the Three Styles — • Instruction 1 (Formal): “Turn right into the hallway. Advance straight past the dining table. Enter the bedroom and stop in front of the bed.” • Instruction 2 (Conversational): “Take a right into the hallway and keep walking until you pass the dining table on your left. Go into the bedroom and stop by the bed.” • Instruction 3 (Narrative): “Turning right, you move into the hallway, the dining table sliding by on your left. The hall opens into a bedroom, where you halt just before the bed.” |
Table S.6 : Prompts for Navigation instruction generation in SceneVerse++ .
| You are an AI assistant tasked with generating Fill-in-the-blank Action Completion MCQ for robot navigation. Your job is to output a multiple-choice question (with blanks) and its correct answer. — Input — |
| A sequence of continuous key frames from a room-tour video (the frames are consecutive and represent a smooth camera/robot trajectory). — High-level rule (priority order) — 1. ALL reasoning must be grounded purely on the visual evidence from the frames 2. Use visual cues such as object appearance/disappearance, relative positions, scaling, and viewpoint rotation to infer the robot’s movements and turns. 3. When describing places, objects, or targets, use detailed and specific visual anchors — not just generic room names. For example: “the blue sofa on the right,” “the black dining table ahead,” “the kitchen counter with sink,” or “the hallway with a white door at the end.” 4. If any step or turn cannot be confidently inferred from visual evidence, skip or merge it rather than guessing. Do NOT fabricate movements. — Core Procedure (must follow this order) — 1. Construct a concise, numbered Trajectory Summary : • Carefully analyze the continuous frame sequence to extract a minimal yet complete trajectory. • Each entry in the summary should be a single action step, such as: "1. Go forward until [object/room]", "2. Turn left", "3. Go forward until [object/room]". • Determine steps by tracking: – Appearance/disappearance or scaling of landmarks (for “Go forward”) – Change in viewing direction or lateral movement (for “Turn left/right/back”) • The summary should form a coherent navigation sequence from the starting viewpoint to the final destination. • Explicitly describe both start place and end place in visually grounded detail: e.g., “You are a robot beginning at the living room, facing the blue sofa.” e.g., “You want to navigate to the kitchen with a table on your left.” • When describing each “Go forward” anchor, be as specific as visually supported: – Include object appearance (color, size, material), or scene context (e.g., furniture type or nearby area). Example: “Go forward until the blue sofa.” • Only include meaningful transitions — skip redundant minor movements or rotations that don’t correspond to clear spatial change. • Ensure geometric reasoning consistency. • Make the trajectory alternate logically between “Go forward” and “Turn”. • Example of a trajectory summary: – 1. Go forward until the 3-seater sofa (evidence: frames X–Y) – 2. Turn right (evidence: frames X–Y) – 3. Go forward until the dining table (evidence: frames X–Y) – 4. Turn left (evidence: frames X–Y) – 5. Go forward until the kitchen counter with sink (evidence: frames X–Y) 2. QA Generation (based on Trajectory Summary only) : • Normalize steps to alternate between “Go forward…” and “Turn …”. • Determine where to place [please fill in] blanks: – Every turn step must become a blank. • “Go forward” must mention detailed visible landmarks. • Use the strict template: Q: You are a robot beginning at [start place, with visual details]. You want to navigate to [end place, with visual details]. You will perform the following actions: 1. Go forward until [object/room] 2. [please fill in] 3. Go forward until [object/room] ... N. Go forward until [object/room]. You have reached the final destination. (Note: for each [please fill in], choose either ’turn back,’ ’turn left,’ or ’turn right.’) |
Table S.7 : Prompts to generate route plan VQA in SceneVerse++ - part1.
| 3. Options generation: • For each blank, permissible options: – ’turn back’, ’turn left’, ’turn right’ • If one blank: produce A–C. • If >= 2 blanks: produce A–D. • Each option is a full sequence of turns. 4. Correct answer determination: • Must match turns inferred from visual trajectory summary. • No guessing ambiguous turns. — Output format (STRICT JSON SCHEMA — all keys required) — • "trajectory summary": a list of strings. • "question": full question string. • "options": dictionary with keys A, B, C, D. • "answer": one correct option. — Strict behaviour notes (must obey) — • Only use frame-based evidence. • Build a reliable Trajectory Summary. • The first step may be a Turn or a Go-forward action. • Never guess ambiguous turns. |
Table S.8 : Prompts to generate route plan VQA in SceneVerse++ - part2
More in this topic
3D Scene Understanding
3D scene understanding builds geometric and semantic representations of environments for spatial reasoning, perception, and interaction.
See all 295 papers