cs.CVApr 7, 2026

SEM-ROVER: Semantic Voxel-Guided Diffusion for Large-Scale Driving Scene Generation

Authors: Hiba Dahmani, Nathan Piasco, Moussab Bennehar, Luis Roldão, Dzmitry Tsishkou, Laurent Caraffa, Jean-Philippe Tarel, Roland Brémond

Organizations: Noah’s Ark, Huawei Paris Research Center, France · COSYS, Gustave Eiffel University, France · LASTIG, IGN-ENSG, Gustave Eiffel University, France

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Scalable generation of outdoor driving scenes requires 3D representations that remain consistent across multiple viewpoints and scale to large areas. Existing solutions either rely on image or video generative models distilled to 3D space, harming the geometric coherence and restricting the rendering to training views, or are limited to small-scale 3D scene or object-centric generation. In this work, we propose a 3D generative framework based on ΣΣ-Voxfield grid, a discrete representation where each occupied voxel stores a fixed number of colorized surface samples. To generate this representation, we train a semantic-conditioned diffusion model that operates on local voxel neighborhoods and uses 3D positional encodings to capture spatial structure. We scale to large scenes via progressive spatial outpainting over overlapping regions. Finally, we render the generated ΣΣ-Voxfield grid with a deferred rendering module to obtain photorealistic images, enabling large-scale multiview-consistent 3D scene generation without per-scene optimization. Extensive experiments show that our approach can generate diverse large-scale urban outdoor scenes, renderable into photorealistic images with various sensor configurations and camera trajectories while maintaining moderate computation cost compared to existing approaches.

Figures & tables

Figure 1: Our model generates large-scale 3D driving scenes given a coarse semantic voxel grid. Here we show a generated large-scale driving scene spanning ≈\approx 100 000 m2
MethodPriorPipelineFeed Forward
Urban Architect3D Layout2D Diff + NeRF×\times
LSD-3DPC + Boxes2D Diff + 3DGS×\times
GEN3CText/Image + PCVideo Diff✓\checkmark
MagicDrive3DHDMap + Text + TrajVideo Diff. + 3DGS×\times
InfiniCubeHDMap + Text + Traj.Vox Diff + Video Diff + FF 3DGS✓\checkmark
OursSemantic VoxelsVox Diff. + Deferred rendering✓\checkmark
Table 1: Driving-scene generation methods. We review relevant works in terms of required priors and computation pipelines. We explicitly distinguish per-scene optimization from feed-forward diffusion pipelines.
Figure 2: Overview of SEM-ROVER. Our framework performs generation directly in 3D to ensure consistent geometry and appearance. It represents the scene with a Σ\Sigma -Voxfield grid and applies transformer-based diffusion over 3D tokens. To scale to large environments efficiently, we use an iterative outpainting strategy. Finally, a deferred rendering engine converts the generated 3D scene into photorealistic views.
Figure 3: Simplified illustration of 2D Σ\Sigma Voxfield. a) A 2D voxel containing a continuous colorized surface field. b) 2D Σ\Sigma Voxfield points, uniformly sampled on the surface. c) Rendering method of the Σ\Sigma Voxfield, where each point is replaced by a 2D Gaussians aligned with the implicit surface.
Figure 4: Qualitative results on WOD. We show three WOD scenes (a–c). For each scene, the top strip visualizes the semantic voxel rendering used for conditioning, and the bottom strip shows the corresponding generated scene from 5 camera views.
Figure 5: Qualitative results on PandaSet. We show three PandaSet scenes (a–c). For each scene, the top strip visualizes the semantic voxel rendering used for conditioning, and the bottom strip shows the corresponding generated scene from 5 camera views.
Figure 6: Qualitative comparison to InfiniCube [ 17 ] . For two waymo scenes (a) and (b), we show five camera of the semantic conditioning and below the rendered generated scenes for Our method (Ours) and Infinicube (InfC).
Figure 7: Qualitative results: generative capabilities. , the top row visualizes the conditioning signal, and the two rows below show different generations sampled from the same conditioning, highlighting output diversity under fixed structure.
MethodFID (seen) ↓\downarrowFID (novel) ↓\downarrowKID (seen) ↓\downarrowKID (novel) ↓\downarrowMin. VRAM
InfiniCube [ 17 ]84.1499.130.030.0675 GB
GEN3C [ 14 ]113.27117.630.080.0943 GB
Ours (ASD)81.9889.200.050.068 GB
Table 2: Quantitative Evaluation of Generation Quality for 3D Scene Generation with our proposed method and existing approaches.
MethodE3D ↓\downarrowMMDS ↓\downarrow
w/o sem. cond.3.9270.105
w/o ordering3.5850.093
Ours3.5230.091
Table 3: Feature-space ablation of our 3D diffusion model.
Figure 8: Semantic editing : We remove existing cars by editing the semantic grid (red boxes), then regenerate the content conditioned on the updated semantics. The generated results remain coherent and consistent with the surrounding scene.
Figure 9: Scene inpainting We mask a voxel region as shows (1) (darker voxel colors for unmasked) and re-generate only this region while keeping the rest of the Σ\Sigma -Voxfield grid fixed. (2) and (3) show two inpainted results (red box) that remain coherent with the background and vary in both structure and appearance.
Figure 10: Textured meshes after data processing. Top line shows 2 scenes from Pandaset [ 31 ] , bottom are 2 scenes from WOD [ 28 ] . Orange triangles in the colorized mesh denote faces not textured during the procedure.
Figure 11: Visualization of Σ\Sigma -VoxFields. (a) Large-scene Σ\Sigma -VoxFields with their corresponding semantic layouts. (b) Examples of Σ\Sigma -VoxFields training samples, each containing 50–150 neighboring Σ\Sigma -VoxFields used to train our model.
Figure 12: Architecture of the Σ\Sigma -Voxfield diffusion model.
Figure 13: Deferred rendering architectures. (a) Autoregressive Stable Diffusion: our renderer based on SD model, with additional conditioning for autoregressive generation (previous frame) and consistent 3D generation (3D buffer from Σ\Sigma -Voxfield rendering and sky mask); (b) Video Stable Diffusion: our renderer based on VSD model, with additional conditioning from 3D buffers and sky masks.
Figure 14: Top-view visualization of progressive region selection on a semantic voxel grid. Each panel shows the region selected at one iteration of Algorithm 1 , with the overlaid number indicating the selection order. The extraction expands outward from already covered regions, producing overlapping local sets suitable for conditional outpainting.
Figure 15: Top-view visualization of progressive outpainting. The first row shows the semantic voxel grid layout conditioning, while the second row shows the corresponding generated 3D Σ\Sigma -Voxfield. The panels correspond to different stages of region expansion and illustrate that our Repaint [ 18 ] based outpainting preserves spatial coherence across neighboring local generations.
Figure 17: Additional Qualitative results on WOD [ 28 ] . We show six WOD scenes (a–f). For each scene, the top strip visualizes the semantic voxel rendering used for conditioning, and the bottom strip shows the corresponding generated scene from 5 camera views.
Figure 18: Additional Qualitative results on Pandaset [ 31 ] . We show six Pandaset scenes (a–f). For each scene, the top strip visualizes the semantic voxel rendering used for conditioning, and the bottom strip shows the corresponding generated scene from 5 camera views.
Figure 19: Comparison with baselines. We show four scenes (a–d), each rendered from the front camera comparing our method to Infinicube [ 17 ] and GEN3C [ 14 ]
Figure 20: 3D Buffers. Rendering of our generated Σ\Sigma -Voxelfield grids along with the generated scene geometry as normal map.
Figure 21: Large scene generation. The semantic voxel grid exhibit the conditioning used to create a large driving sequence. We also show a top view rendering of the generated Σ\Sigma -Voxfield grid.
More in this topic

3D Scene Generation

3D scene generation creates structured, explorable environments from text, images, maps, or other inputs.

See all 190 papers

Explore similar work

CardsList

  1. Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale

    Apr 13, 2026Dongxu Wei, Qi Xu, Zhiqi Li +63D Scene Generation3D Diffusion Models

    Westlake University · Afari Intelligent Drive · Zhejiang University

  2. Sparse auto-regressive modeling for scene generation from multi-view images

    Sep 3, 2026Thomas Lucas, Maxime Pietrantoni, Philippe Weinzaepfel +4Sparse-View 3D Reconstruction3D Scene Generation

    1NAVER LABS Europe, France · 2NAVER LABS, South Korea · 3Carnegie Mellon University, USA

  3. SynCity 3000: Bootstrapping Scene-Scale 3D Diffusion

    Jul 6, 2026Paul Engstler, Iro Laina, Christian Rupprecht +13D Scene Generation3D Scene Layout Generation

    Visual Geometry Group, University of Oxford