NavGen: Visual Generative Models as a Scalable Data Engine for Embodied 3D Navigation

Xijie Huang; Yongyang Wan; Chengbin Dong; Zimo Ding; Mo Zhu; Yijin Wang; Zhiyang Liu; Fei Gao

cs.ROSep 25, 2026

NavGen: Visual Generative Models as a Scalable Data Engine for Embodied 3D Navigation

Authors: Xijie Huang, Yongyang Wan, Chengbin Dong, Zimo Ding, Mo Zhu, Yijin Wang, Zhiyang Liu, Fei Gao, +2 more

Organizations: Zhejiang University. · Differential Robotics.

Abstract

General-purpose robot models increasingly rely on large and diverse datasets. For embodied 3D navigation, however, existing data sources face a fundamental trade-off: simulated data can be generated at scale but often suffer from the visual sim-to-real gap, whereas real-world flight data provide realistic observations but are costly to collect. This paper studies another direction: the use of high-fidelity visual generative models as scalable data engines for embodied 3D navigation. We introduce NavGen, a text-to-video data generation pipeline that produces diverse vision-language navigation (VLN) episodes across indoor and outdoor scenes. We also propose a style-diversification method that scales up long-tail data that are difficult and costly to collect. The resulting dataset contains approximately 400K navigation episodes. We evaluate our dataset against existing UAV navigation datasets across multiple metrics, and find that the model trained on our data generally improves with scale, outperforming those trained on existing datasets. To validate real-world transferability, we deploy the trained model in world-action-model paradigm to real-world flying experiments. The final model achieves a 75% success rate across different navigation tasks and environments.

Figures & tables

Fig. 1: Overview of the NavGen data generation pipeline. The upper left part uses the open-source model to generate 300K common task samples. The upper right part uses the more powerful but closed-source model to generate 10K long-tail task samples with a style-diversification pipeline to scale up to 100K.

Variant	T5	Enc.	DiT	Dec.	Total	Speedup
	(s)	(s)	(s)	(s)	(s)	( $\times$ )
Original	0.116	0.189	156.583	13.520	171.568	1.00 $\times$
Sparse	0.165	0.057	9.585	1.421	12.250	14.00 $\times$
+ Distilled	0.058	0.051	0.553	1.040	1.751	97.98 $\times$
+ Distilled + TAE	0.058	0.049	0.510	0.055	0.702	244.40 $\times$

TABLE I: Inference latency breakdown of the proposed system. T5, Enc., DiT, and Dec. denote text-encoding, VAE-encoding, diffusion-transformer, and VAE-decoding latency, respectively.

Fig. 2: Bidirectional long-video generation via clean-prefix history conditioning.

Parameter	Stage 1: SFT	Stage 2: sCM	Stage 3: DMD2
LoRA rank	64	64	64
LoRA alpha	64	64	64
Global batch size	32	32	32
Optimizer	AdamW	AdamW	AdamW
Generator LR	1e-4	1e-5	2e-5
Critic LR	–	–	4e-6

TABLE II: Training hyperparameters across the three-stage pipeline.

Fig. 3: Waypoints decoding from RGB frames.

Fig. 4: Qualitative analysis of our dataset. Figure (a) is the motion distribution. Figure (b) is the word cloud of the text description. Figure (c), (d), (e) are the analyses of the specific dimension.

Metric	TravelUAV [ 3 ]	IndoorUAV [ 4 ]	OpenFly [ 5 ]	UAV-Flow [ 6 ]	Ours
Aesthetic	$0.471\pm 0.057$	$0.546\pm 0.053$	$0.551\pm 0.052$	$0.549\pm 0.047$	$\mathbf{0.586\pm 0.037}$
Smoothness	$0.888\pm 0.026$	$0.897\pm 0.021$	$0.927\pm 0.022$	$\mathbf{0.987\pm 0.006}$	$0.952\pm 0.016$
Sem. Cons.	$0.710\pm 0.048$	$0.858\pm 0.056$	$0.883\pm 0.037$	$\mathbf{0.979\pm 0.008}$	$0.952\pm 0.013$
I2V Cons. S	$0.776\pm 0.051$	$0.913\pm 0.044$	$0.912\pm 0.062$	$\mathbf{0.985\pm 0.006}$	$0.977\pm 0.010$
I2V Cons. B	$0.797\pm 0.044$	$0.930\pm 0.029$	$0.923\pm 0.041$	$\mathbf{0.974\pm 0.006}$	$0.973\pm 0.007$
Avg	$0.7284\pm 0.032$	$0.8288\pm 0.034$	$0.8392\pm 0.032$	$\mathbf{0.8948\pm 0.011}$	$0.888\pm 0.012$

TABLE III: VBench grouped scores at the best-performing checkpoint per dataset (bold = best, underline = second best per row). Aesthetic: Aesthetic Quality; Smoothness: Motion Smoothness and Temporal Flickering; Sem. Cons.: Subject and Background Consistency; I2V Cons.: alignment to the input first frame, reported separately for Subject (S) and Background (B).

Fig. 5: Dataset diversity analysis. Figure (a) and (b) show the Vendi Score and FD CLIP. Figure (c) is the visualization of the clip feature distribution of the datasets.

Fig. 6: Navigation performance scaling with dataset size. The ar and bi mean autoregressive and bidirectional, respectively.

Fig. 7: Ablation study on style diversification. Performance comparison between training on original 10K samples versus 100K diversified samples on unseen real-world tasks.

Stage	Prompt rewrite	Video generation	Action decode	Ego-planner
Time	1 s	0.7 s	0.5 s	10 ms

TABLE IV: Inference time of each stage in our pipeline.

Fig. 8: Short-horizon navigation experiments in real-world environments.

Fig. 9: Long-horizon navigation demonstration.

Explore similar work

FlyMirage: A Fully Automated Generation Pipeline for Diverse and Scalable UAV Flight Data via Generative World Model
May 19, 2026Jinhan Li, Xijie Huang, Weiqi Gai +9UAV Navigation Synthetic Data Generation for Robotics
State Key Laboratory of Industrial Control Technology, Zhejiang University, Hangzhou 310027, China. · Differential Robotics, Hangzhou 311121, China.
ImagiNav: Scalable Embodied Navigation via Generative Visual Prediction and Inverse Dynamics
Mar 14, 2026Jie Chen, Yuxin Cai, Yizhuo Wang +5Robot Navigation Vision-Language Navigation
Department of Mechanical Engineering, National University of Singapore, Singapore · Institute for Infocomm Research (I2R), Agency for Science, Technology and Research (A*STAR), Singapore · Nanyang Technological University (NTU), Singapore
GN0: Toward a Unified Paradigm for Generation, Evaluation, and Policy Learning in Visual-Language Navigation
Jun 2, 2026Xinhai Li, Xiaotao Zhang, Yuehao Huang +10Robot Policy Learning Embodied Navigation
Institute of Artificial Intelligence, China Telecom · Shanghai Jiao Tong University · Zhejiang University +3

May 19, 2026J/K move · Enter open · S save

FlyMirage: A Fully Automated Generation Pipeline for Diverse and Scalable UAV Flight Data via Generative World Model

Jinhan Li, Xijie Huang, Weiqi Gai, Zhaoqi Wang+8

State Key Laboratory of Industrial Control Technology, Zhejiang University, Hangzhou 310027, China. · Differential Robotics, Hangzhou 311121, China.

UAV Navigation Synthetic Data Generation for Robotics Vision-Language Navigation 3D Scene Generation Aerial VLN

Open paper AskPDF arXiv

In the field of Vision-Language Navigation (VLN), aerial datasets remain limited in their ability to combine scale, diversity, and realism, often relying on either costly real-world scenes or visually limited simulations. To address these challenges, we introduce FlyMirage, a highly scalable and fully automated data generation pipeline for aerial VLN. Our approach leverages large language models (LLM) as an environment designer to promote scene diversity, paired with a generative world model that instantiates these designs into high-fidelity 3D Gaussian Splatting (3DGS) scenes. To substantially reduce human labor and ensure the feasibility of flight data, FlyMirage automates scene exploration and semantic information acquisition, and further integrates a dynamically feasible planner for uncrewed aerial vehicle (UAV) trajectory generation. Utilizing this toolchain, we generate a large-scale, diverse, and photorealistic aerial VLN dataset. We further fine-tune the widely used pi_0 model on this dataset and demonstrate the improvement our data brings to the model's ability to navigate in completely unseen environments, both in simulation and in the real world. Project website: https://lij1500.github.io/flymirage_website/ .

Abstract

Explore similar work

FlyMirage: A Fully Automated Generation Pipeline for Diverse and Scalable UAV Flight Data via Generative World Model

ImagiNav: Scalable Embodied Navigation via Generative Visual Prediction and Inverse Dynamics

GN0: Toward a Unified Paradigm for Generation, Evaluation, and Policy Learning in Visual-Language Navigation