We present LucidLyra, a framework that makes generatively reconstructed 3D scenes directly queryable for semantics and instances in a feed-forward manner from video latents. While generative 3D reconstruction can synthesize virtual environments from a single image, downstream applications in robotics and physical AI require not only visual realism but also access to scene semantics and object identities. Existing 3D understanding approaches, however, require captured multi-view scenes, have limited generalization capabilities, or rely on expensive per-scene processing and optimization. To bridge this gap, our method distills complementary knowledge from a camera-controlled video diffusion model and frozen 2D vision foundation models into a unified model to jointly infer 3D Gaussians encoding geometry, appearance, open-vocabulary semantics, and class-agnostic instance cues. The resulting scenes support novel-view rendering, open-vocabulary segmentation, and object grouping without per-scene optimization, while training requires neither manual 3D annotations nor captured multi-view data. Our method outperforms baselines combining state-of-the-art 3DGS segmentation approaches with generative 3D scene reconstruction on curated synthetic and real-image-conditioned benchmarks, while preserving reconstruction quality. Furthermore, our latent formulation naturally extends to monocular video, reconstructing structured dynamic 3D scenes across time. By turning renderable but unstructured outputs into queryable 3D worlds, LucidLyra enables downstream applications including 3D scene editing, 3D asset mining, and robotic simulation.
Figures & tables
Figure 2: LucidLyra framework. A frozen camera-controlled video generation model produces multi-view latents. Its RGB decoder and frozen 2D foundation models construct reconstruction, semantic, and instance targets. The targets supervise our Gaussian decoder which maps the latents, camera rays, and input image features to a shared queryable 3D scene. The right panel details our 3DGS decoder. Source- and target-time inputs (grayed out) are used only for the dynamic extension. Snowflakes and flames denote frozen and trainable components, respectively.
LucidLyra-SDG Static
LucidLyra-ADE20K
Reconstruction
Semantics
Instance
Reconstruction
Semantics
Instance
Method
PSNR ↑
LPIPS ↓
mIoU ↑
mAcc ↑
mTrack PR ↑
Track PR 25 ↑
PSNR ↑
LPIPS ↓
mIoU ↑
mAcc ↑
mTrack PR ↑
Track PR 25 ↑
Lyra + Lifted Teachers
24.49
0.211
60.5
75.3
22.8
33.2
24.59
0.217
50.1
64.5
17.3
26.8
Lyra
24.49
0.211
–
–
–
–
24.59
0.217
–
–
–
–
Lyra + LangSplatV2
24.49
0.211
32.4
52.7
–
–
24.59
0.217
37.1
53.7
–
–
Lyra + SFS
24.49
0.211
31.9
49.5
–
–
24.59
0.217
35.2
51.3
–
–
Table 1: Static evaluation on LucidLyra-SDG and LucidLyra-ADE20K. We report PSNR (dB), LPIPS (–), and mIoU, mAcc, mTrackPR, and TrackPR 25 (all in % ). † : uses video model RGB output.
Reconstruction
Semantics
Instance
Method
PSNR ↑
LPIPS ↓
mIoU ↑
mAcc ↑
mTrack PR ↑
Track PR 25 ↑
LucidLyra (Ours)
25.02
0.186
64.1
79.2
Table 2: Ablations on LucidLyra-SDG. PSNR (dB), LPIPS (–), and mIoU, mAcc, mTrackPR, and TrackPR25 (all in % ).
Table 2: LucidLyra qualitative results. We visualize input- and novel-view 3DGS [ 28 ] renderings across RGB, semantic/instance feature PCAs, and open-vocabulary/instance segmentation using generated input images.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
H×W
L
V
S
B
Steps
Static
1
704$\times$1\,280
121
1
9
1
10000
2
704$\times$1\,280
121
1
9
1
20000
3
704$\times$1\,280
121
1
9
1
20000
4
704$\times$1\,280
121
1 – 6
$$3V
1
10000
Dynamic
Appendix
Table 3: LucidLyra training setup.
Reconstruction
Semantics
Instance
Method
PSNR ↑
LPIPS ↓
mIoU ↑
mAcc ↑
mTrack PR ↑
Track PR 25 ↑
Lyra + Lifted Teachers
22.16
0.248
53.2
69.4
12.1
15.1
Lyra
22.16
0.248
–
–
–
–
Lyra † + 4D LangSplat
23.03
0.354
30.3
47.8
–
–
Lyra † + TRASE
22.77
0.309
–
–
7.7
12.5
Lyra † + IGGT 4D
–
–
36.1
46.0
7.8
12.9
Appendix
Table 4: Dynamic reconstruction and understanding on LucidLyra-SDG Dynamic. We report PSNR (dB), LPIPS (–), and mIoU, mAcc, mTrackPR, and TrackPR 25 (all in % ). † denotes using the video diffusion model’s RGB output.
ImageNet
RADSeg on ADE20K
Method
Feature Dim.
Top-1 ↑
Top-5 ↑
Low ↑
Mid ↑
High ↑
SigLIP2
1152
83.50
96.86
27.16
28.96
28.94
+ Projection PVL
256
80.69
95.89
25.55
27.06
27.09
+ Projection PVL
128
76.61
94.02
24.22
25.34
25.14
+ Projection PVL
64
70.16
90.43
22.13
23.66
23.61
Appendix
Table 5: PVL projection quality. SigLIP2 ImageNet val zero-shot classification accuracy and RADSeg open-vocabulary semantic segmentation on ADE20K eval mIoU. Both metrics in % ↑ .
Figure 4: LucidLyra qualitative results. We visualize input- and novel-view 3DGS [ 28 ] renderings across RGB, semantic/instance feature PCAs, and open-vocabulary/instance segmentation for generated input images.
Figure 5: Dynamic LucidLyra qualitative results. We visualize input- and novel-view 3DGS renderings across RGB, semantic/instance feature PCAs, and open-vocabulary/instance segmentation for generated input videos.
Figure 6: Waymo qualitative examples. Each two-row block shows one scene from four viewpoints. Paired columns show RGB, semantic/instance feature PCA, and semantic/instance segmentation. Insets show the original input images.
Figure 7: Qualitative comparison on LucidLyra-SDG Static eval. We show three viewpoints per scene, with semantic predictions above and instance predictions below. † denotes directly using generated multi-view RGB [ 2 ] .
Figure 8: Qualitative comparison on LucidLyra-ADE20K static eval. We show three viewpoints per scene, with semantic predictions above and instance predictions below. † denotes directly using generated multi-view RGB.
Figure 9: Qualitative comparison on the dynamic evaluation set. We show times t=0,40,80,120$$ along an orbiting camera trajectory. Both scene time and camera pose change. From top to bottom: RGB reconstruction, semantic segmentation, and instance segmentation. IGGT4D uses LSeg [ 39 ] for semantic predictions. Dynamic Lyra supplies the reconstruction used by the lifted-teacher baseline. Instance ID colors are random per method.
Figure 10: Object- and part-level instance segmentation. We visualize 3 views per scene, showing 3DGS RGB render, object- and part-level 3DGS feature PCA, 3DGS object clusters, and within-object part clusters. Random colors denote cluster IDs.
Figure 11: Robot simulation in NVIDIA Isaac Sim using LucidLyra scenes. Four different robot-simulation scenarios built on LucidLyra reconstructions. Extracted geometry provides collision surfaces, semantic labels support task planning, and Gaussian-splat rendering preserves the scene appearance. Overlaid robot poses show motion over time, with increasing opacity indicating temporal progression.
Figure 12: LucidLyra 3DGS Scene Editing. Left: Our GUI provides an interface for natural interaction with the generated Gaussian scenes. Gaussians are selected via text prompts or via clicking and can then be manipulated interactively. Right: Editing examples showing the initial 3DGS scene, the applied edits and the final scene.
Figure 13: Asset extraction and articulation. (a) Selected instances alongside their textured reconstructions. Display scales are independent. (b) Selected single-view (top) and multi-view (bottom) Pixal3D results. Each result pairs a textured view with an alternate neutral-material view. (c) Articulated assets shown closed and open, with original textures and part segmentation color. Part boundaries and hinges are obtained through PARTICULATE [ 44 ] on the generated asset meshes. Doors open to 85$^{\circ}$ and lids to 75∘ for illustration, not as measured joint limits.
Figure 14: LucidLyra-SDG static training data. RGB frames, instance tracks, and PCA visualizations of semantic teacher features across generated camera trajectories.
Figure 15: LucidLyra-SDG static evaluation data. RGB frames and reference semantic and instance annotations for the conditioning view and generated viewpoints.
Figure 16: LucidLyra-ADE20K evaluation data. Real conditioning images and generated viewpoints with semantic and instance annotations propagated from official ADE20K labels.
Figure 17: LucidLyra-SDG dynamic training data. RGB frames, instance tracks, and PCA visualizations of semantic teacher features across viewpoints and time.
Figure 18: LucidLyra-SDG dynamic evaluation data. RGB frames with reference semantic and instance annotations from dynamic evaluation scenes.
Recent advances in video generation enable a new paradigm for 3D scene creation: generating camera-controlled videos that simulate scene walkthroughs, then lifting them to 3D via feed-forward reconstruction techniques. This generative reconstruction approach combines the visual fidelity and creative capacity of video models with 3D outputs ready for real-time rendering and simulation. Scaling to large, complex environments requires 3D-consistent video generation over long camera trajectories with large viewpoint changes and location revisits, a setting where current video models degrade quickly. Existing methods for long-horizon generation are fundamentally limited by two forms of degradation: spatial forgetting and temporal drifting. As exploration proceeds, previously observed regions fall outside the model's temporal context, forcing the model to hallucinate structures when revisited. Meanwhile, autoregressive generation accumulates small synthesis errors over time, gradually distorting scene appearance and geometry. We present Lyra 2.0, a framework for generating persistent, explorable 3D worlds at scale. To address spatial forgetting, we maintain per-frame 3D geometry and use it solely for information routing -- retrieving relevant past frames and establishing dense correspondences with the target viewpoints -- while relying on the generative prior for appearance synthesis. To address temporal drifting, we train with self-augmented histories that expose the model to its own degraded outputs, teaching it to correct drift rather than propagate it. Together, these enable substantially longer and 3D-consistent video trajectories, which we leverage to fine-tune feed-forward reconstruction models that reliably recover high-quality 3D scenes.
Recent video world models have increasingly adopted 3D memory mechanisms to integrate historical observations into a lightweight and persistent scene representation. This naturally raises a complementary question: if 3D representations can serve as an effective mechanism for memorizing observed scenes, can they also be directly used as the latent space for reconstructing and generating scenes? Motivated by this idea, we represent a scene as a compact and unstructured 3D-anchored latent space (SceneAE). Each latent combines an explicit 3D position with an implicit content feature and is spatially grounded to describe its local scene region, forming a sparse latent point cloud. Unlike conventional video-VAE latents that describe how a scene appears from particular viewpoints, our representation directly describes the scene itself, making it naturally applicable to both reconstruction and generation. For reconstruction, once the scene has been generated, our persistent 3D representation can be projected and decoded into RGB-D observations by different target cameras, without the need to regenerate a video for each new camera trajectory. For generation, we leverage the geometry-content separation to first generate camera-controlled 3D geometry and then synthesize content conditioned on it, reducing camera-appearance entanglement and improving camera controllability. Experiments show that this 3D latent achieves superior generation quality, computational cost, and camera controllability.
Shuofeng Sun, Wenzhao Zheng, Yunpeng Zhang +3
Beijing University of Posts and Telecommunications · University of California, Berkeley · PhiGent Robotics +1
Large language models are increasingly being applied to 3D scene understanding. However, existing 3D-aware LLMs largely operate as black boxes: they produce bounding boxes or textual responses without exposing the reasoning process, and they typically depend on pre-trained 3D detectors for object proposals. We present Scene-R1, a video-grounded framework for 3D scene reasoning that avoids both pre-trained 3D object detectors and dense point-cloud backbones. The framework combines reinforcement-learning-based reasoning with a two-stage grounding pipeline. In the first stage, Scene-R1 performs temporal grounding by identifying the video segments most relevant to an open-ended query. In the second stage, it grounds the target in image space by predicting a 2D bounding box. This prediction is then refined into a precise 3D localization using high-quality candidates from a zero-shot segmentation module, enabling fine-grained geometric reasoning without detector-based proposals. Scene-R1 can also be adapted to 3D visual question answering, generating free-form answers directly from RGB-D video. Our training pipeline uses lightweight 2D supervision projected from sparse scans, avoiding the computational overhead required to train 3D instance segmentation models. Across multiple datasets, Scene-R1 outperforms existing open-vocabulary baselines while also providing interpretable intermediate reasoning steps. These results suggest that reinforcement-learning-based reasoning over RGB-D video can support accurate 3D scene understanding with task-level supervision. The training, inference, and 3D refinement code is available at https://github.com/819346675/Scene-R1.
Junjie Wang, Zhihao Yuan, Shuyi Jiang +5
FNii-Shenzhen, CUHKSZ · SSE, CUHKSZ · Singapore University of Technology and Design +1