DrivePTS: A Progressive Learning Framework with Textual and Structural Enhancement for Driving Scene Generation

Zhechao Wang; Yiming Zeng; Lufan Ma; Zeqing Fu; Chen Bai; Ziyao Lin; Cheng Lu

cs.CVFeb 26, 2026

DrivePTS: A Progressive Learning Framework with Textual and Structural Enhancement for Driving Scene Generation

Authors: Zhechao Wang, Yiming Zeng, Lufan Ma, Zeqing Fu, Chen Bai, Ziyao Lin, Cheng Lu

Organizations: XPeng Motors

Abstract

Synthesis of diverse driving scenes serves as a crucial data augmentation technique for validating the robustness and generalizability of autonomous driving systems. Current methods aggregate high-definition (HD) maps and 3D bounding boxes as geometric conditions in diffusion models for conditional scene generation. However, implicit inter-condition dependency causes generation failures when control conditions change independently. Additionally, these methods suffer from insufficient details in both semantic and structural aspects. Specifically, brief and view-invariant captions restrict semantic contexts, resulting in weak background modeling. Meanwhile, the standard denoising loss with uniform spatial weighting neglects foreground structural details, causing visual distortions and blurriness. To address these challenges, we propose DrivePTS, which incorporates three key innovations. Firstly, our framework adopts a progressive learning strategy to mitigate inter-dependency between geometric conditions, reinforced by an explicit mutual information constraint. Secondly, a Vision-Language Model is utilized to generate multi-view hierarchical descriptions across six semantic aspects, providing fine-grained textual guidance. Thirdly, a frequency-guided structure loss is introduced to strengthen the model's sensitivity to high-frequency elements, improving foreground structural fidelity. Extensive experiments demonstrate that our DrivePTS achieves state-of-the-art fidelity and controllability in generating diverse driving scenes. Notably, DrivePTS successfully generates rare scenes where prior methods fail, highlighting its strong generalization ability.

Figures & tables

Figure 1: Comparison of various scene generation methods on the modified map layouts. The first column highlights the modified regions of each map. While MagicDrive fails to adapt to map modifications, our DrivePTS successfully generates scenes aligning with the updated map configurations.

Figure 2: The overall architecture of our proposed DrivePTS. The left part illustrates our adopted generative network structure, while the center depicts the training process corresponding to the proposed progressive learning strategy. The right part highlights the implementation of frequency-guided structure loss.

Method	Synthesis resolution	FID $\downarrow$	mAP $\uparrow$	NDS $\uparrow$	mAOE $\downarrow$	Road	Vehicle
Method	Synthesis resolution	FID $\downarrow$	mAP $\uparrow$	NDS $\uparrow$	mAOE $\downarrow$	mIoU $\uparrow$	mIoU $\uparrow$
Oracle	–	–	35.54	41.20	0.56	73.68	34.80
BEVGen	–	25.54	–	–	–	50.20	5.89
BEVControl	224 $\times$ 400	24.85	–	–	–	60.80	26.80
MagicDrive	224 $\times$ 480	16.20	12.30	23.32	–	61.05	27.01
Panacea*	256 $\times$ 512	16.96	11.65	22.40	0.82	57.11	22.77

Table 1: Comparison of generation fidelity and controllability across different driving-view generation methods on the NuScenes validation set. Panacea* denotes our reproduced results based on the officially provided generated dataset. Outcomes demonstrating superior performance are highlighted in bold .

\uparrow

\downarrow

indicates that a higher/lower value is better.

Training	Road mIoU $\uparrow$
train	65.83
train + Real val	67.53
train + Syn. val (MagicDrive)	66.12 (-1.41)
train + Syn. val (Panacea)	66.60 (-0.93)
train + Syn. val (PerLDiff)	65.74 (-1.79)
train + Syn. val (Ours)	67.49 (-0.04)

Table 2: Performance comparison for the boosting performance of BEV segmentation models using synthesized dataset on the NuScenes test set using CVT. The “ train + Real val ” configuration serves as a benchmark, representing the ideal upper performance limit achievable. The numbers in parentheses indicate the performance disparity relative to the “ train + Real val ” configuration.

MHD	FGSL	MIC	FID $\downarrow$	Road mIoU $\uparrow$	Vehicle mIoU $\uparrow$
–	–	–	15.10	59.77	25.80
✓	–	–	12.03	61.22	26.49
–	✓	–	14.47	62.92	26.95
✓	✓	–	11.68	63.60	27.16
✓	✓	✓	11.45	63.95	27.82

Table 3: Ablation study on the effectiveness of Multi-View Hierarchical Descriptions (MHD), Frequency-Guided Structure Loss (FGSL), and Mutual Information Constraint (MIC) in driving scene generation.

$\lambda_{\text{freq}}$	Road mIoU $\uparrow$	Vehicle mIoU $\uparrow$
0	61.22	25.49
0.25	63.47	26.31
0.5	63.60	27.16
0.75	63.01	26.84
1	61.95	26.17

Table 4: Ablation study on coefficients

\lambda_{\text{freq}}

and

\lambda_{\text{MI}}

. The ablation on

\lambda_{\text{freq}}

is conducted in Stage 1, and the optimal

\lambda_{\text{freq}}

is then applied to stage 2 for the

\lambda_{\text{MI}}

ablation.

Figure 3: Impact of different iterative steps on the FID and geometry controllability of generated images.

Figure 4: Examples of controllable road layout generation via HD map editing. Each row shows a case where the map is modified to introduce new road structures, with the generated image accurately reflecting these changes.

Figure 5: Visualization of the impact of textual descriptions on scene reconstruction. Rows with the same index denote the same scene. Improvements brought by multi-view hierarchical descriptions are highlighted with various colored circles or rectangles.

Figure 6: Qualitative comparison of scene generation with and without frequency-guided structure loss. Columns 1-2 show vehicle structure improvements, while columns 3-4 show road boundary enhancements. Red boxes highlight structural distortions without the proposed loss, whereas green boxes indicate successful mitigation with the loss.

Metric	MagicDrive	DriveDreamer	Panacea	Drive-WM	Ours (SD-2.1)	Ours (SD-3.5)
FVD $\downarrow$	221	353	139	122	128	110

Table 5: Comparison of temporal consistency for different video generation methods on the NuScenes dataset. Lower FVD scores indicate better temporal coherence.

Figure 7: Comparison of original dataset captions and VLM-driven multi-view hierarchical descriptions. Each group shows: (top) six-view images, (middle) original dataset captions, and (bottom) our multi-view hierarchical descriptions.

Figure 8: Qualitative comparison of scene reconstruction quality between original dataset captions and our multi-view hierarchical descriptions. For each example: original image (top), reconstruction from original captions (middle), and our results (bottom). Red regions indicate missing details with original captions, while green regions highlight successful recovery through our fine-grained descriptions.

Figure 9: Qualitative comparison of scene reconstruction quality between original dataset captions and our multi-view hierarchical descriptions. Notably, in night scene generation, original captions tend to produce hallucinated illuminated areas due to insufficient contextual information, while our multi-view hierarchical descriptions faithfully reconstruct authentic nighttime atmospheres.

Figure 10: Examples of targeted road removal in driving scene generation using our DrivePTS framework. Each group shows: (1) original geometric conditions, (2) original scene generation, (3) modified geometric conditions after road removal, and (4) updated scene generation. Green boxes indicate areas corresponding to the geometric modifications.

Figure 11: Examples of targeted road addition in driving scene generation using our DrivePTS framework. The visualization follows the same four-row format as above, where the third row shows geometric conditions with added roads.

Figure 12: Additional examples of targeted road addition using our DrivePTS framework. The visualization format follows the same structure as above.

Explore similar work

Jun 18, 2026

FrozenDrive: Zero-Shot Text-Guided Driving Scene Generation and Data Augmentation with Parameter-Free Frozen Diffusion Model

Synthetic data for autonomous driving is surging, powered by diffusion models that promise scalable scene generation. Yet key obstacles remain, as enforcing multi-view and temporal consistency often relies on backbone fine-tuning or added layers, which erodes pre-trained knowledge and weakens text alignment. Models also stay close to the training distribution, struggling under adverse weather and unseen configurations, and fidelity favors frequent over rare classes. We address these gaps with FrozenDrive, a controllable generative framework that preserves a pretrained diffusion models knowledge while achieving strong consistency. FrozenDrive conditions on rich driving-stack signals and text prompts, and introduces knowledge-preserving spatio-temporal attention to impose cross-view alignment and temporal coherence in a single pass within a parameter-free frozen diffusion backbone. An additional object-focused constraint improves per-object fidelity for rare categories. Without any weather- or scene-specific fine-tuning, our model synthesizes globally coherent multi-view driving scenes from text, particularly under adverse and rare conditions, and surpasses prior baselines. On nuScenes, FrozenDrive augmented data significantly improves AD models performance, especially at night and in rain, demonstrating stronger robustness when trained with our scenario-targeted data.

Yuhwan Jeong, Hyeonseong Kim, Daehyun We +5

KAIST, Visual Intelligence Lab

Synthetic Data Augmentation Video Diffusion Models Multi-View Video Generation Autonomous Driving Autonomous Driving Simulation Temporal Consistency in Video Generation+1

Ask PDF

May 25, 2026

AnyScene: Towards Highly Controllable Driving Scene Generation at Anywhere and Beyond

Generating high-fidelity and controllable synthetic data is critical for advancing end-to-end autonomous driving, particularly for addressing the long tail of rare safety-critical scenarios. Existing occupancy-guided methods typically rely on shallow conditioning mechanisms and reference-frame-dependent video synthesis, which limits fine-grained controllability from arbitrary BEV layouts and restricts their applicability for scalable simulation. In this paper, we propose AnyScene, a unified occupancy-centric framework for driving scene generation. AnyScene generates semantic occupancy sequences from BEV layouts through a Spatial-Temporal Occupancy Diffusion Transformer that jointly tokenizes BEV and occupancy features in an autoregressive manner. This design enables precise controllability from cross-dataset and user-defined BEV inputs while naturally supporting long-horizon generation. Building upon the generated occupancy, a Geometry-Grounded View Expansion module treats occupancy as the canonical spatial representation and synthesizes temporally consistent multi-view driving videos in a reference-free and autoregressive fashion, supporting flexible camera configurations at inference time. Extensive experiments demonstrate that AnyScene achieves state-of-the-art performance in both occupancy and video generation. It exhibits strong generalization to unseen and customized layouts, and provides measurable benefits for downstream tasks such as sparse-view 3D reconstruction.

Haiming Zhang, Junfei Zhou, Feng Jiang +6

Li Auto · Southwest Jiaotong University · Tsinghua University

Long-Horizon Video Generation 3D Scene Generation Controllable Video Generation Autonomous Driving Simulation Autonomous Driving Scenario Generation

Ask PDF

Apr 6, 2026

HorizonWeaver: Generalizable Multi-Level Semantic Editing for Driving Scenes

Ensuring safety in autonomous driving requires scalable generation of realistic, controllable driving scenes beyond what real-world testing provides. Yet existing instruction guided image editors, trained on object-centric or artistic data, struggle with dense, safety-critical driving layouts. We propose HorizonWeaver, which tackles three fundamental challenges in driving scene editing: (1) multi-level granularity, requiring coherent object- and scene-level edits in dense environments; (2) rich high-level semantics, preserving diverse objects while following detailed instructions; and (3) ubiquitous domain shifts, handling changes in climate, layout, and traffic across unseen environments. The core of HorizonWeaver is a set of complementary contributions across data, model, and training: (1) Data: Large-scale dataset generation, where we build a paired real/synthetic dataset from Boreas, nuScenes, and Argoverse2 to improve generalization; (2) Model: Language-Guided Masks for fine-grained editing, where semantics-enriched masks and prompts enable precise, language-guided edits; and (3) Training: Content preservation and instruction alignment, where joint losses enforce scene consistency and instruction fidelity. Together, HorizonWeaver provides a scalable framework for photorealistic, instruction-driven editing of complex driving scenes, collecting 255K images across 13 editing categories and outperforming prior methods in L1, CLIP, and DINO metrics, achieving +46.4% user preference and improving BEV segmentation IoU by +33%. Project page: https://msoroco.github.io/horizonweaver/

Mauricio Soroco, Francesco Pittaluga, Zaid Tasneem +5

Simon Fraser University · NEC Labs America

Text-Guided Image Editing Autonomous Driving Scenario Generation Domain Generalization

Ask PDF