SEM-ROVER: Semantic Voxel-Guided Diffusion for Large-Scale Driving Scene Generation
Authors: Hiba Dahmani, Nathan Piasco, Moussab Bennehar, Luis Roldão, Dzmitry Tsishkou, Laurent Caraffa, Jean-Philippe Tarel, Roland Brémond
Organizations: Noah’s Ark, Huawei Paris Research Center, France · COSYS, Gustave Eiffel University, France · LASTIG, IGN-ENSG, Gustave Eiffel University, France
Scalable generation of outdoor driving scenes requires 3D representations that remain consistent across multiple viewpoints and scale to large areas. Existing solutions either rely on image or video generative models distilled to 3D space, harming the geometric coherence and restricting the rendering to training views, or are limited to small-scale 3D scene or object-centric generation. In this work, we propose a 3D generative framework based on Σ-Voxfield grid, a discrete representation where each occupied voxel stores a fixed number of colorized surface samples. To generate this representation, we train a semantic-conditioned diffusion model that operates on local voxel neighborhoods and uses 3D positional encodings to capture spatial structure. We scale to large scenes via progressive spatial outpainting over overlapping regions. Finally, we render the generated Σ-Voxfield grid with a deferred rendering module to obtain photorealistic images, enabling large-scale multiview-consistent 3D scene generation without per-scene optimization. Extensive experiments show that our approach can generate diverse large-scale urban outdoor scenes, renderable into photorealistic images with various sensor configurations and camera trajectories while maintaining moderate computation cost compared to existing approaches.
Figures & tables
Figure 1: Our model generates large-scale 3D driving scenes given a coarse semantic voxel grid. Here we show a generated large-scale driving scene spanning ≈ 100 000 m2
Method
Prior
Pipeline
Feed Forward
Urban Architect
3D Layout
2D Diff + NeRF
×
LSD-3D
PC + Boxes
2D Diff + 3DGS
×
GEN3C
Text/Image + PC
Video Diff
✓
MagicDrive3D
HDMap + Text + Traj
Video Diff. + 3DGS
×
InfiniCube
HDMap + Text + Traj.
Vox Diff + Video Diff + FF 3DGS
✓
Ours
Semantic Voxels
Vox Diff. + Deferred rendering
✓
Table 1: Driving-scene generation methods. We review relevant works in terms of required priors and computation pipelines. We explicitly distinguish per-scene optimization from feed-forward diffusion pipelines.
Figure 2: Overview of SEM-ROVER. Our framework performs generation directly in 3D to ensure consistent geometry and appearance. It represents the scene with a Σ -Voxfield grid and applies transformer-based diffusion over 3D tokens. To scale to large environments efficiently, we use an iterative outpainting strategy. Finally, a deferred rendering engine converts the generated 3D scene into photorealistic views.
Figure 3: Simplified illustration of 2D Σ Voxfield. a) A 2D voxel containing a continuous colorized surface field. b) 2D Σ Voxfield points, uniformly sampled on the surface. c) Rendering method of the Σ Voxfield, where each point is replaced by a 2D Gaussians aligned with the implicit surface.
Figure 4: Qualitative results on WOD. We show three WOD scenes (a–c). For each scene, the top strip visualizes the semantic voxel rendering used for conditioning, and the bottom strip shows the corresponding generated scene from 5 camera views.
Figure 5: Qualitative results on PandaSet. We show three PandaSet scenes (a–c). For each scene, the top strip visualizes the semantic voxel rendering used for conditioning, and the bottom strip shows the corresponding generated scene from 5 camera views.
Figure 6: Qualitative comparison to InfiniCube [ 17 ] . For two waymo scenes (a) and (b), we show five camera of the semantic conditioning and below the rendered generated scenes for Our method (Ours) and Infinicube (InfC).
Figure 7: Qualitative results: generative capabilities. , the top row visualizes the conditioning signal, and the two rows below show different generations sampled from the same conditioning, highlighting output diversity under fixed structure.
Method
FID (seen) ↓
FID (novel) ↓
KID (seen) ↓
KID (novel) ↓
Min. VRAM
InfiniCube [ 17 ]
84.14
99.13
0.03
0.06
75 GB
GEN3C [ 14 ]
113.27
117.63
0.08
0.09
43 GB
Ours (ASD)
81.98
89.20
0.05
0.06
8 GB
Table 2: Quantitative Evaluation of Generation Quality for 3D Scene Generation with our proposed method and existing approaches.
Method
E3D ↓
MMDS ↓
w/o sem. cond.
3.927
0.105
w/o ordering
3.585
0.093
Ours
3.523
0.091
Table 3: Feature-space ablation of our 3D diffusion model.
Figure 8: Semantic editing : We remove existing cars by editing the semantic grid (red boxes), then regenerate the content conditioned on the updated semantics. The generated results remain coherent and consistent with the surrounding scene.
Figure 9: Scene inpainting We mask a voxel region as shows (1) (darker voxel colors for unmasked) and re-generate only this region while keeping the rest of the Σ -Voxfield grid fixed. (2) and (3) show two inpainted results (red box) that remain coherent with the background and vary in both structure and appearance.
Figure 10: Textured meshes after data processing. Top line shows 2 scenes from Pandaset [ 31 ] , bottom are 2 scenes from WOD [ 28 ] . Orange triangles in the colorized mesh denote faces not textured during the procedure.
Figure 11: Visualization of Σ -VoxFields. (a) Large-scene Σ -VoxFields with their corresponding semantic layouts. (b) Examples of Σ -VoxFields training samples, each containing 50–150 neighboring Σ -VoxFields used to train our model.
Figure 12: Architecture of the Σ -Voxfield diffusion model.
Figure 13: Deferred rendering architectures. (a) Autoregressive Stable Diffusion: our renderer based on SD model, with additional conditioning for autoregressive generation (previous frame) and consistent 3D generation (3D buffer from Σ -Voxfield rendering and sky mask); (b) Video Stable Diffusion: our renderer based on VSD model, with additional conditioning from 3D buffers and sky masks.
Figure 14: Top-view visualization of progressive region selection on a semantic voxel grid. Each panel shows the region selected at one iteration of Algorithm 1 , with the overlaid number indicating the selection order. The extraction expands outward from already covered regions, producing overlapping local sets suitable for conditional outpainting.
Figure 15: Top-view visualization of progressive outpainting. The first row shows the semantic voxel grid layout conditioning, while the second row shows the corresponding generated 3D Σ -Voxfield. The panels correspond to different stages of region expansion and illustrate that our Repaint [ 18 ] based outpainting preserves spatial coherence across neighboring local generations.
Figure 17: Additional Qualitative results on WOD [ 28 ] . We show six WOD scenes (a–f). For each scene, the top strip visualizes the semantic voxel rendering used for conditioning, and the bottom strip shows the corresponding generated scene from 5 camera views.
Figure 18: Additional Qualitative results on Pandaset [ 31 ] . We show six Pandaset scenes (a–f). For each scene, the top strip visualizes the semantic voxel rendering used for conditioning, and the bottom strip shows the corresponding generated scene from 5 camera views.
Figure 19: Comparison with baselines. We show four scenes (a–d), each rendered from the front camera comparing our method to Infinicube [ 17 ] and GEN3C [ 14 ]
Figure 20: 3D Buffers. Rendering of our generated Σ -Voxelfield grids along with the generated scene geometry as normal map.
Figure 21: Large scene generation. The semantic voxel grid exhibit the conditioning used to create a large driving sequence. We also show a top view rendering of the generated Σ -Voxfield grid.
Scene-level 3D generation has long been dominated by 2D multi-view or video diffusion models. Typically, these 2D-based approaches perform scene generation in a 2D latent space, which introduces two fundamental issues: (i) representing 3D scenes via 2D views leads to significant representation redundancy, and (ii) latent space rooted in 2D inherently limits the spatial consistency of the generated scenes. In this paper, we propose to perform 3D scene generation directly within a high-dimensional implicit 3D latent space derived from powerful 2D foundation models (e.g., DINOv2, SigLIP2, DA3). Then we tame diffusion transformers to perform diffusion modeling directly in the 3D latent space, enabling 3D-grounded scene generation. In our experiments, we not only demonstrate the advantages of our method over 2D-based approaches in terms of efficiency and spatial consistency, but also comprehensively analyze the impact of 3D latent representation on 3D scene generation performance. Furthermore, we validate the favorable scalability of our method with respect to representation and model capacity.
Dongxu Wei, Qi Xu, Zhiqi Li +6
Westlake University · Afari Intelligent Drive · Zhejiang University
Generating complete 3D scenes from sparse, unconstrained views is a fundamental challenge in 3D vision which requires reasoning beyond observed content while remaining computationally tractable. Existing feed-forward reconstruction methods are inherently limited to content visible in the input images, while 3D generative modeling is hindered by the high computational cost of dense volumetric representations and the scarcity of large-scale 3D supervision. We introduce SPAR3S, a sparse voxel-aligned 3D latent generative model for conditional scene completion without requiring ground-truth 3D data for supervision. Our key insight is to formulate 3D scene generation in a structured, compact, voxel-aligned 3D latent space where only occupied voxels are represented. We learn this sparse latent space directly from multi-view images using photometric supervision via differentiable 3D Gaussian Splatting. Given a partial set of observed voxels encoded from sparse input views, scene completion reduces to predicting the missing latent tokens and their spatial support within the voxel grid. To this end, we train a masked autoregressive transformer that jointly models voxel occupancy and latent token values, enabling efficient and spatially consistent generation of unseen regions. We demonstrate the effectiveness of our method on synthetic indoor scenes, achieving higher novel-view quality than prior work. We further validate its generalization on RealEstate10k, highlighting its applicability to real-world data.
Thomas Lucas, Maxime Pietrantoni, Philippe Weinzaepfel +4
1NAVER LABS Europe, France · 2NAVER LABS, South Korea · 3Carnegie Mellon University, USA
We present SynCity 3000, a framework for generating 3D scenes that are globally coherent while enabling fine-grained layout control. Building on the ability of current image-to-3D generators to produce complex 3D assets from a single image, we extend this capability to the scale of entire scenes by adapting the generator to be applicable as a convolutional operator. We achieve this by fine-tuning the model on scene-like data generated by a new synthetic data engine, which we propose to address the scarcity of 3D scene data for training. The convolutional generator is then applied to a dimetric image of the entire scene, generated from the user prompt, resulting in 3D scenes of arbitrary size and complexity. Across diverse prompts and layouts, SynCity 3000 produces large, coherent, and detailed scenes, addressing the shortcomings of prior approaches to 3D scene generation.