LucidLyra: Feed-Forward Generative 3D Scene Reconstruction and Joint Understanding via Multi-Task Distillation

Oliver Hahn; Rodrigo Marcuzzi; Sangeetha Grama Srinivasan; Alessandro Burzio; Sherwin Bahmani; Jiahui Huang; Stefan Roth; Shengyu Huang

cs.CVOct 29, 2026

LucidLyra: Feed-Forward Generative 3D Scene Reconstruction and Joint Understanding via Multi-Task Distillation

Authors: Oliver Hahn, Rodrigo Marcuzzi, Sangeetha Grama Srinivasan, Alessandro Burzio, Sherwin Bahmani, Jiahui Huang, Stefan Roth, Shengyu Huang,

Organizations: NVIDIA · TU Darmstadt · Università degli Studi di Modena e Reggio Emilia · ELIZA

Abstract

We present LucidLyra, a framework that makes generatively reconstructed 3D scenes directly queryable for semantics and instances in a feed-forward manner from video latents. While generative 3D reconstruction can synthesize virtual environments from a single image, downstream applications in robotics and physical AI require not only visual realism but also access to scene semantics and object identities. Existing 3D understanding approaches, however, require captured multi-view scenes, have limited generalization capabilities, or rely on expensive per-scene processing and optimization. To bridge this gap, our method distills complementary knowledge from a camera-controlled video diffusion model and frozen 2D vision foundation models into a unified model to jointly infer 3D Gaussians encoding geometry, appearance, open-vocabulary semantics, and class-agnostic instance cues. The resulting scenes support novel-view rendering, open-vocabulary segmentation, and object grouping without per-scene optimization, while training requires neither manual 3D annotations nor captured multi-view data. Our method outperforms baselines combining state-of-the-art 3DGS segmentation approaches with generative 3D scene reconstruction on curated synthetic and real-image-conditioned benchmarks, while preserving reconstruction quality. Furthermore, our latent formulation naturally extends to monocular video, reconstructing structured dynamic 3D scenes across time. By turning renderable but unstructured outputs into queryable 3D worlds, LucidLyra enables downstream applications including 3D scene editing, 3D asset mining, and robotic simulation.

Figures & tables

Figure 2: LucidLyra framework. A frozen camera-controlled video generation model produces multi-view latents. Its RGB decoder and frozen 2D foundation models construct reconstruction, semantic, and instance targets. The targets supervise our Gaussian decoder which maps the latents, camera rays, and input image features to a shared queryable 3D scene. The right panel details our 3DGS decoder. Source- and target-time inputs (grayed out) are used only for the dynamic extension. Snowflakes and flames denote frozen and trainable components, respectively.

	LucidLyra-SDG Static						LucidLyra-ADE20K
	Reconstruction		Semantics		Instance		Reconstruction		Semantics		Instance
Method	PSNR $\uparrow$	LPIPS $\downarrow$	mIoU $\uparrow$	mAcc $\uparrow$	mTrack PR $\uparrow$	Track PR 25 $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	mIoU $\uparrow$	mAcc $\uparrow$	mTrack PR $\uparrow$	Track PR 25 $\uparrow$
Lyra + Lifted Teachers	24.49	0.211	60.5	75.3	22.8	33.2	24.59	0.217	50.1	64.5	17.3	26.8
Lyra	$24.49$	$0.211$	–	–	–	–	$24.59$	$0.217$	–	–	–	–
Lyra + LangSplatV2	$24.49$	$0.211$	$32.4$	$52.7$	–	–	$24.59$	$0.217$	$37.1$	$53.7$	–	–
Lyra + SFS	$24.49$	$0.211$	$31.9$	$49.5$	–	–	$24.59$	$0.217$	$35.2$	$51.3$	–	–

Table 1: Static evaluation on LucidLyra-SDG and LucidLyra-ADE20K. We report PSNR (dB), LPIPS (–), and mIoU, mAcc, mTrackPR, and TrackPR 25 (all in

\%

\dagger

: uses video model RGB output.

	Reconstruction		Semantics		Instance
Method	PSNR $\uparrow$	LPIPS $\downarrow$	mIoU $\uparrow$	mAcc $\uparrow$	mTrack PR $\uparrow$	Track PR 25 $\uparrow$
LucidLyra (Ours)
25.02
0.186
64.1
79.2

Table 2: Ablations on LucidLyra-SDG. PSNR (dB), LPIPS (–), and mIoU, mAcc, mTrackPR, and TrackPR25 (all in

\%

Panel of Table 2 — Table 2: LucidLyra qualitative results. We visualize input- and novel-view 3DGS [ 28 ] renderings across RGB, semantic/instance feature PCAs, and open-vocabulary/instance segmentation using generated input images.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	$\mathbf{H\times W}$	$\mathbf{L}$	$\mathbf{V}$	$\mathbf{S}$	$\mathbf{B}$	Steps
Static
$1$	$704$\times$1\,280$	$121$	$1$	$9$	$1$	$10\,000$
$2$	$704$\times$1\,280$	$121$	$1$	$9$	$1$	$20\,000$
$3$	$704$\times$1\,280$	$121$	$1$	$9$	$1$	$20\,000$
$4$	$704$\times$1\,280$	$121$	$1$ – $6$	$$3 $V$	$1$	$10\,000$
Dynamic

Appendix

Table 3: LucidLyra training setup.

	Reconstruction		Semantics		Instance
Method	PSNR $\uparrow$	LPIPS $\downarrow$	mIoU $\uparrow$	mAcc $\uparrow$	mTrack PR $\uparrow$	Track PR 25 $\uparrow$
Lyra + Lifted Teachers	$22.16$	$0.248$	$53.2$	$69.4$	$12.1$	$15.1$
Lyra	$22.16$	0.248	–	–	–	–
Lyra $\dagger$ + 4D LangSplat	23.03	$0.354$	$30.3$	$47.8$	–	–
Lyra $\dagger$ + TRASE	$22.77$	$0.309$	–	–	$7.7$	$12.5$
Lyra $\dagger$ + IGGT 4D	–	–	$36.1$	$46.0$	$7.8$	$12.9$

Appendix

Table 4: Dynamic reconstruction and understanding on LucidLyra-SDG Dynamic. We report PSNR (dB), LPIPS (–), and mIoU, mAcc, mTrackPR, and TrackPR 25 (all in

\%

\dagger

denotes using the video diffusion model’s RGB output.

		ImageNet		RADSeg on ADE20K
Method	Feature Dim.	Top-1 $\uparrow$	Top-5 $\uparrow$	Low $\uparrow$	Mid $\uparrow$	High $\uparrow$
SigLIP2	$1\,152$	$83.50$	$96.86$	$27.16$	$28.96$	$28.94$
+ Projection $P_{\mathrm{VL}}$	$256$	$80.69$	$95.89$	$25.55$	$27.06$	$27.09$
+ Projection $P_{\mathrm{VL}}$	$128$	$76.61$	$94.02$	$24.22$	$25.34$	$25.14$
+ Projection $P_{\mathrm{VL}}$	$64$	$70.16$	$90.43$	$22.13$	$23.66$	$23.61$

Appendix

Table 5:

P_{\mathrm{VL}}

projection quality. SigLIP2 ImageNet val zero-shot classification accuracy and RADSeg open-vocabulary semantic segmentation on ADE20K eval mIoU. Both metrics in %

\uparrow

Panel of Figure 4 — Figure 4: LucidLyra qualitative results. We visualize input- and novel-view 3DGS [ 28 ] renderings across RGB, semantic/instance feature PCAs, and open-vocabulary/instance segmentation for generated input images.

Panel of Figure 5 — Figure 5: Dynamic LucidLyra qualitative results. We visualize input- and novel-view 3DGS renderings across RGB, semantic/instance feature PCAs, and open-vocabulary/instance segmentation for generated input videos.

Panel of Figure 6 — Figure 6: Waymo qualitative examples. Each two-row block shows one scene from four viewpoints. Paired columns show RGB, semantic/instance feature PCA, and semantic/instance segmentation. Insets show the original input images.

Panel of Figure 7 — Figure 7: Qualitative comparison on LucidLyra-SDG Static eval. We show three viewpoints per scene, with semantic predictions above and instance predictions below. $\dagger$ denotes directly using generated multi-view RGB [ 2 ] .

Panel of Figure 8 — Figure 8: Qualitative comparison on LucidLyra-ADE20K static eval. We show three viewpoints per scene, with semantic predictions above and instance predictions below. $\dagger$ denotes directly using generated multi-view RGB.

Panel of Figure 9 — Figure 9: Qualitative comparison on the dynamic evaluation set. We show times $t=$ 0 $,$ 40 $,$ 80 $,$ 120$$ along an orbiting camera trajectory. Both scene time and camera pose change. From top to bottom: RGB reconstruction, semantic segmentation, and instance segmentation. IGGT4D uses LSeg [ 39 ] for semantic predictions. Dynamic Lyra supplies the reconstruction used by the lifted-teacher baseline. Instance ID colors are random per method.

Panel of Figure 10 — Figure 10: Object- and part-level instance segmentation. We visualize $3$ views per scene, showing 3DGS RGB render, object- and part-level 3DGS feature PCA, 3DGS object clusters, and within-object part clusters. Random colors denote cluster IDs.

Panel of Figure 11 — Figure 11: Robot simulation in NVIDIA Isaac Sim using LucidLyra scenes. Four different robot-simulation scenarios built on LucidLyra reconstructions. Extracted geometry provides collision surfaces, semantic labels support task planning, and Gaussian-splat rendering preserves the scene appearance. Overlaid robot poses show motion over time, with increasing opacity indicating temporal progression.

Panel of Figure 12 — Figure 12: LucidLyra 3DGS Scene Editing. Left: Our GUI provides an interface for natural interaction with the generated Gaussian scenes. Gaussians are selected via text prompts or via clicking and can then be manipulated interactively. Right: Editing examples showing the initial 3DGS scene, the applied edits and the final scene.

Panel of Figure 13 — Figure 13: Asset extraction and articulation. (a) Selected instances alongside their textured reconstructions. Display scales are independent. (b) Selected single-view (top) and multi-view (bottom) Pixal3D results. Each result pairs a textured view with an alternate neutral-material view. (c) Articulated assets shown closed and open, with original textures and part segmentation color. Part boundaries and hinges are obtained through PARTICULATE [ 44 ] on the generated asset meshes. Doors open to $85$^{\circ}$ and lids to$ 75 $^{\circ}$ for illustration, not as measured joint limits.

Panel of Figure 14 — Figure 14: LucidLyra-SDG static training data. RGB frames, instance tracks, and PCA visualizations of semantic teacher features across generated camera trajectories.

Panel of Figure 15 — Figure 15: LucidLyra-SDG static evaluation data. RGB frames and reference semantic and instance annotations for the conditioning view and generated viewpoints.

Panel of Figure 16 — Figure 16: LucidLyra-ADE20K evaluation data. Real conditioning images and generated viewpoints with semantic and instance annotations propagated from official ADE20K labels.

Panel of Figure 17 — Figure 17: LucidLyra-SDG dynamic training data. RGB frames, instance tracks, and PCA visualizations of semantic teacher features across viewpoints and time.

Panel of Figure 18 — Figure 18: LucidLyra-SDG dynamic evaluation data. RGB frames with reference semantic and instance annotations from dynamic evaluation scenes.

3D Scene Understanding

3D scene understanding builds geometric and semantic representations of environments for spatial reasoning, perception, and interaction.

See all 296 papers

Lyra 2.0: Explorable Generative 3D Worlds
Apr 14, 2026Tianchang Shen, Sherwin Bahmani, Kai He +123D Scene Generation 3D-Aware Video Generation
NVIDIA
SceneAE: Scene Generation and Reconstruction in Queryable 3D-Anchored Latent Space
Oct 14, 2026Shuofeng Sun, Wenzhao Zheng, Yunpeng Zhang +33D Scene Generation 3D Scene Representation
Beijing University of Posts and Telecommunications · University of California, Berkeley · PhiGent Robotics +1
Scene-R1: Video-Grounded Large Language Models for 3D Scene Reasoning with Lightweight Annotations
Jun 21, 2025Junjie Wang, Zhihao Yuan, Shuyi Jiang +53D Scene Understanding 3D VLMs
FNii-Shenzhen, CUHKSZ · SSE, CUHKSZ · Singapore University of Technology and Design +1

Apr 14, 2026J/K move · Enter open · S save

Lyra 2.0: Explorable Generative 3D Worlds

Tianchang Shen, Sherwin Bahmani, Kai He, Sangeetha Grama Srinivasan+11

NVIDIA

3D Scene Generation 3D-Aware Video Generation Long-Horizon Video Generation Temporal Consistency in Video Generation Camera-Controlled Video Generation Memory-Augmented Video Generation

Open paper AskPDF arXiv

Recent advances in video generation enable a new paradigm for 3D scene creation: generating camera-controlled videos that simulate scene walkthroughs, then lifting them to 3D via feed-forward reconstruction techniques. This generative reconstruction approach combines the visual fidelity and creative capacity of video models with 3D outputs ready for real-time rendering and simulation. Scaling to large, complex environments requires 3D-consistent video generation over long camera trajectories with large viewpoint changes and location revisits, a setting where current video models degrade quickly. Existing methods for long-horizon generation are fundamentally limited by two forms of degradation: spatial forgetting and temporal drifting. As exploration proceeds, previously observed regions fall outside the model's temporal context, forcing the model to hallucinate structures when revisited. Meanwhile, autoregressive generation accumulates small synthesis errors over time, gradually distorting scene appearance and geometry. We present Lyra 2.0, a framework for generating persistent, explorable 3D worlds at scale. To address spatial forgetting, we maintain per-frame 3D geometry and use it solely for information routing -- retrieving relevant past frames and establishing dense correspondences with the target viewpoints -- while relying on the generative prior for appearance synthesis. To address temporal drifting, we train with self-augmented histories that expose the model to its own degraded outputs, teaching it to correct drift rather than propagate it. Together, these enable substantially longer and 3D-consistent video trajectories, which we leverage to fine-tune feed-forward reconstruction models that reliably recover high-quality 3D scenes.

Abstract

3D Scene Understanding

Explore similar work

Lyra 2.0: Explorable Generative 3D Worlds

SceneAE: Scene Generation and Reconstruction in Queryable 3D-Anchored Latent Space

Scene-R1: Video-Grounded Large Language Models for 3D Scene Reasoning with Lightweight Annotations