DrivePTS: A Progressive Learning Framework with Textual and Structural Enhancement for Driving Scene Generation

Zhechao Wang; Yiming Zeng; Lufan Ma; Zeqing Fu; Chen Bai; Ziyao Lin; Cheng Lu

cs.CVFeb 26, 2026

DrivePTS: A Progressive Learning Framework with Textual and Structural Enhancement for Driving Scene Generation

Authors: Zhechao Wang, Yiming Zeng, Lufan Ma, Zeqing Fu, Chen Bai, Ziyao Lin, Cheng Lu

Organizations: XPeng Motors

Abstract

Synthesis of diverse driving scenes serves as a crucial data augmentation technique for validating the robustness and generalizability of autonomous driving systems. Current methods aggregate high-definition (HD) maps and 3D bounding boxes as geometric conditions in diffusion models for conditional scene generation. However, implicit inter-condition dependency causes generation failures when control conditions change independently. Additionally, these methods suffer from insufficient details in both semantic and structural aspects. Specifically, brief and view-invariant captions restrict semantic contexts, resulting in weak background modeling. Meanwhile, the standard denoising loss with uniform spatial weighting neglects foreground structural details, causing visual distortions and blurriness. To address these challenges, we propose DrivePTS, which incorporates three key innovations. Firstly, our framework adopts a progressive learning strategy to mitigate inter-dependency between geometric conditions, reinforced by an explicit mutual information constraint. Secondly, a Vision-Language Model is utilized to generate multi-view hierarchical descriptions across six semantic aspects, providing fine-grained textual guidance. Thirdly, a frequency-guided structure loss is introduced to strengthen the model's sensitivity to high-frequency elements, improving foreground structural fidelity. Extensive experiments demonstrate that our DrivePTS achieves state-of-the-art fidelity and controllability in generating diverse driving scenes. Notably, DrivePTS successfully generates rare scenes where prior methods fail, highlighting its strong generalization ability.

Figures & tables

Figure 1: Comparison of various scene generation methods on the modified map layouts. The first column highlights the modified regions of each map. While MagicDrive fails to adapt to map modifications, our DrivePTS successfully generates scenes aligning with the updated map configurations.

Figure 2: The overall architecture of our proposed DrivePTS. The left part illustrates our adopted generative network structure, while the center depicts the training process corresponding to the proposed progressive learning strategy. The right part highlights the implementation of frequency-guided structure loss.

Method	Synthesis resolution	FID $\downarrow$	mAP $\uparrow$	NDS $\uparrow$	mAOE $\downarrow$	Road	Vehicle
Method	Synthesis resolution	FID $\downarrow$	mAP $\uparrow$	NDS $\uparrow$	mAOE $\downarrow$	mIoU $\uparrow$	mIoU $\uparrow$
Oracle	–	–	35.54	41.20	0.56	73.68	34.80
BEVGen	–	25.54	–	–	–	50.20	5.89
BEVControl	224 $\times$ 400	24.85	–	–	–	60.80	26.80
MagicDrive	224 $\times$ 480	16.20	12.30	23.32	–	61.05	27.01
Panacea*	256 $\times$ 512	16.96	11.65	22.40	0.82	57.11	22.77

Table 1: Comparison of generation fidelity and controllability across different driving-view generation methods on the NuScenes validation set. Panacea* denotes our reproduced results based on the officially provided generated dataset. Outcomes demonstrating superior performance are highlighted in bold .

\uparrow

\downarrow

indicates that a higher/lower value is better.

Training	Road mIoU $\uparrow$
train	65.83
train + Real val	67.53
train + Syn. val (MagicDrive)	66.12 (-1.41)
train + Syn. val (Panacea)	66.60 (-0.93)
train + Syn. val (PerLDiff)	65.74 (-1.79)
train + Syn. val (Ours)	67.49 (-0.04)

Table 2: Performance comparison for the boosting performance of BEV segmentation models using synthesized dataset on the NuScenes test set using CVT. The “ train + Real val ” configuration serves as a benchmark, representing the ideal upper performance limit achievable. The numbers in parentheses indicate the performance disparity relative to the “ train + Real val ” configuration.

MHD	FGSL	MIC	FID $\downarrow$	Road mIoU $\uparrow$	Vehicle mIoU $\uparrow$
–	–	–	15.10	59.77	25.80
✓	–	–	12.03	61.22	26.49
–	✓	–	14.47	62.92	26.95
✓	✓	–	11.68	63.60	27.16
✓	✓	✓	11.45	63.95	27.82

Table 3: Ablation study on the effectiveness of Multi-View Hierarchical Descriptions (MHD), Frequency-Guided Structure Loss (FGSL), and Mutual Information Constraint (MIC) in driving scene generation.

$\lambda_{\text{freq}}$	Road mIoU $\uparrow$	Vehicle mIoU $\uparrow$
0	61.22	25.49
0.25	63.47	26.31
0.5	63.60	27.16
0.75	63.01	26.84
1	61.95	26.17

Table 4: Ablation study on coefficients

\lambda_{\text{freq}}

and

\lambda_{\text{MI}}

. The ablation on

\lambda_{\text{freq}}

is conducted in Stage 1, and the optimal

\lambda_{\text{freq}}

is then applied to stage 2 for the

\lambda_{\text{MI}}

ablation.

Figure 3: Impact of different iterative steps on the FID and geometry controllability of generated images.

Figure 4: Examples of controllable road layout generation via HD map editing. Each row shows a case where the map is modified to introduce new road structures, with the generated image accurately reflecting these changes.

Figure 5: Visualization of the impact of textual descriptions on scene reconstruction. Rows with the same index denote the same scene. Improvements brought by multi-view hierarchical descriptions are highlighted with various colored circles or rectangles.

Figure 6: Qualitative comparison of scene generation with and without frequency-guided structure loss. Columns 1-2 show vehicle structure improvements, while columns 3-4 show road boundary enhancements. Red boxes highlight structural distortions without the proposed loss, whereas green boxes indicate successful mitigation with the loss.

Metric	MagicDrive	DriveDreamer	Panacea	Drive-WM	Ours (SD-2.1)	Ours (SD-3.5)
FVD $\downarrow$	221	353	139	122	128	110

Table 5: Comparison of temporal consistency for different video generation methods on the NuScenes dataset. Lower FVD scores indicate better temporal coherence.

Figure 7: Comparison of original dataset captions and VLM-driven multi-view hierarchical descriptions. Each group shows: (top) six-view images, (middle) original dataset captions, and (bottom) our multi-view hierarchical descriptions.

Figure 8: Qualitative comparison of scene reconstruction quality between original dataset captions and our multi-view hierarchical descriptions. For each example: original image (top), reconstruction from original captions (middle), and our results (bottom). Red regions indicate missing details with original captions, while green regions highlight successful recovery through our fine-grained descriptions.

Figure 9: Qualitative comparison of scene reconstruction quality between original dataset captions and our multi-view hierarchical descriptions. Notably, in night scene generation, original captions tend to produce hallucinated illuminated areas due to insufficient contextual information, while our multi-view hierarchical descriptions faithfully reconstruct authentic nighttime atmospheres.

Figure 10: Examples of targeted road removal in driving scene generation using our DrivePTS framework. Each group shows: (1) original geometric conditions, (2) original scene generation, (3) modified geometric conditions after road removal, and (4) updated scene generation. Green boxes indicate areas corresponding to the geometric modifications.

Figure 11: Examples of targeted road addition in driving scene generation using our DrivePTS framework. The visualization follows the same four-row format as above, where the third row shows geometric conditions with added roads.

Figure 12: Additional examples of targeted road addition using our DrivePTS framework. The visualization format follows the same structure as above.

Explore similar work

FrozenDrive: Zero-Shot Text-Guided Driving Scene Generation and Data Augmentation with Parameter-Free Frozen Diffusion Model
Jun 18, 2026Yuhwan Jeong, Hyeonseong Kim, Daehyun We +5Synthetic Data Augmentation Video Diffusion Models
KAIST, Visual Intelligence Lab
AnyScene: Towards Highly Controllable Driving Scene Generation at Anywhere and Beyond
May 25, 2026Haiming Zhang, Junfei Zhou, Feng Jiang +6Long-Horizon Video Generation 3D Scene Generation
Li Auto · Southwest Jiaotong University · Tsinghua University
HorizonWeaver: Generalizable Multi-Level Semantic Editing for Driving Scenes
Apr 6, 2026Mauricio Soroco, Francesco Pittaluga, Zaid Tasneem +5Text-Guided Image Editing Autonomous Driving Scenario Generation
Simon Fraser University · NEC Labs America

Jun 18, 2026J/K move · Enter open · S save

FrozenDrive: Zero-Shot Text-Guided Driving Scene Generation and Data Augmentation with Parameter-Free Frozen Diffusion Model

Yuhwan Jeong, Hyeonseong Kim, Daehyun We, Seonkyu Song+4

KAIST, Visual Intelligence Lab

Synthetic Data Augmentation Video Diffusion Models Multi-View Video Generation Autonomous Driving Autonomous Driving Simulation Temporal Consistency in Video Generation+1

Open paper AskPDF arXiv

Synthetic data for autonomous driving is surging, powered by diffusion models that promise scalable scene generation. Yet key obstacles remain, as enforcing multi-view and temporal consistency often relies on backbone fine-tuning or added layers, which erodes pre-trained knowledge and weakens text alignment. Models also stay close to the training distribution, struggling under adverse weather and unseen configurations, and fidelity favors frequent over rare classes. We address these gaps with FrozenDrive, a controllable generative framework that preserves a pretrained diffusion models knowledge while achieving strong consistency. FrozenDrive conditions on rich driving-stack signals and text prompts, and introduces knowledge-preserving spatio-temporal attention to impose cross-view alignment and temporal coherence in a single pass within a parameter-free frozen diffusion backbone. An additional object-focused constraint improves per-object fidelity for rare categories. Without any weather- or scene-specific fine-tuning, our model synthesizes globally coherent multi-view driving scenes from text, particularly under adverse and rare conditions, and surpasses prior baselines. On nuScenes, FrozenDrive augmented data significantly improves AD models performance, especially at night and in rain, demonstrating stronger robustness when trained with our scenario-targeted data.

Abstract

Explore similar work

FrozenDrive: Zero-Shot Text-Guided Driving Scene Generation and Data Augmentation with Parameter-Free Frozen Diffusion Model

AnyScene: Towards Highly Controllable Driving Scene Generation at Anywhere and Beyond

HorizonWeaver: Generalizable Multi-Level Semantic Editing for Driving Scenes