Zero-Shot Cross-City Generalization in End-to-End Autonomous Driving: Self-Supervised versus Supervised Representations

Fatemeh Naeinian; Ali Hamza; Haoran Zhu; Anna Choromanska

cs.CVMar 12, 2026

Zero-Shot Cross-City Generalization in End-to-End Autonomous Driving: Self-Supervised versus Supervised Representations

Authors: Fatemeh Naeinian, Ali Hamza, Haoran Zhu, Anna Choromanska

Organizations: Department of Electrical and Computer Engineering, NYU Tandon School of Engineering, Brooklyn, NY, USA.

Abstract

End-to-end autonomous driving models are typically trained on multi-city datasets using supervised ImageNet-pretrained backbones, yet their ability to generalize to unseen cities remains largely unexamined. When training and evaluation data are geographically mixed, models may implicitly rely on city-specific cues, masking failure modes that would occur under real-world domain shifts when generalizing to new locations. In this work, we formulate zero-shot cross-city transfer as a controlled representation-level stress test for end-to-end autonomous driving and ask how visual pretraining affects transfer behavior under geographic domain shift. We conduct a comprehensive study by integrating self-supervised backbones I-JEPA, DINOv2, and MAE into planning frameworks. We evaluate performance under strict geographic splits on nuScenes in the open-loop setting and on NAVSIM in the closed-loop evaluation protocol. Our experiments reveal a substantial generalization gap when transferring models across cities with different road topologies, traffic conventions, and visual environments. In open-loop evaluation, a supervised backbone exhibits severe degradation when transferring between cities, yet some domain-specific self-supervised methods can substantially reduce both displacement and collision degradation. In closed-loop evaluation, self-supervised pretraining improves average out-of-distribution PDMS in several single-city training settings. Our results provide empirical evidence that representation learning influences the robustness of cross-city planning and motivate zero-shot geographic transfer as an important stress test for evaluating end-to-end autonomous driving systems.

Figures & tables

Fig. 1: Overview of the evaluation framework. Left: region-specific self-supervised pretraining of I-JEPA, DINOv2, and MAE on balanced NAVSIM NavTrain imagery. Center: integration of the pretrained visual backbones into LAW and TransFuser. Right: zero-shot geographic evaluation on unseen cities or regions without target-domain adaptation.

Region	Available planning samples	Planning used	Eligible SSL frames	SSL frames used
Las Vegas	67,683	10,821	99,690	16,303
Boston	12,954	10,821	19,851	16,303
Pittsburgh	10,821	10,821	16,651	16,303
Singapore	11,830	10,821	16,303	16,303

TABLE I: NAVSIM NavTrain data volume and balanced subsets. Downstream training uses 10,821 windowed samples per region. SSL pretraining uses 16,303 eligible three-camera frames per region, corresponding to 48,909 individual images.

Pretraining	Backbone	Multi-Domain B+S $\rightarrow$ B+S		In Domain B $\rightarrow$ B		Cross Domain B $\rightarrow$ S		In Domain S $\rightarrow$ S		Cross Domain S $\rightarrow$ B
Pretraining	Backbone	L2 @avg $\downarrow$	Coll @avg $\downarrow$	L2 @avg $\downarrow$	Coll @avg $\downarrow$	L2 @avg $\downarrow$	Coll @avg $\downarrow$	L2 @avg $\downarrow$	Coll @avg $\downarrow$	L2 @avg $\downarrow$	Coll @avg $\downarrow$
Supervised	Swin-Tiny (original LAW)	0.72 $\pm$ 0.05	0.35 $\pm$ 0.08	0.67 $\pm$ 0.03	0.32 $\pm$ 0.02	4.06 $\pm$ 0.51	5.66 $\pm$ 0.35	0.97 $\pm$ 0.17	0.54 $\pm$ 0.17	1.11 $\pm$ 0.19	0.84 $\pm$ 0.52
Self- Supervised	DINOv2 (ViT-g/14, public, frozen)	0.85 $\pm$ 0.04	0.31 $\pm$ 0.02	0.68 $\pm$ 0.04	0.27 $\pm$ 0.03	3.00 $\pm$ 0.16	3.91 $\pm$ 0.69	1.37 $\pm$ 0.04	0.60 $\pm$ 0.05	2.17 $\pm$ 0.46	1.48 $\pm$ 1.10
	I-JEPA (ViT-H/14, public, frozen)	0.72 $\pm$ 0.03	0.41 $\pm$ 0.33	0.65 $\pm$ 0.02	0.26 $\pm$ 0.04	3.53 $\pm$ 1.23	2.64 $\pm$ 1.22	0.98 $\pm$ 0.04	0.29 $\pm$ 0.01	1.99 $\pm$ 0.64	1.29 $\pm$ 0.49
	MAE (ViT-H/14, public, frozen)	0.67 $\pm$ 0.01	0.34 $\pm$ 0.17	0.59 $\pm$ 0.01	0.32 $\pm$ 0.02	3.06 $\pm$ 0.97	3.92 $\pm$ 1.52	0.79 $\pm$ 0.03	0.29 $\pm$ 0.05	3.61 $\pm$ 1.23	1.94 $\pm$ 1.12
	DINOv2 (ViT-S/14, navbal, frozen)	1.67 $\pm$ 0.15	1.06 $\pm$ 0.48	1.07 $\pm$ 0.03	0.37 $\pm$ 0.02	4.04 $\pm$ 0.14	5.12 $\pm$ 0.36	1.64 $\pm$ 0.02	1.00 $\pm$ 0.10	3.82 $\pm$ 0.01	3.53 $\pm$ 0.12

TABLE II: Zero-shot cross-city transfer on nuScenes using LAW. We report joint-city, in-domain, and cross-city evaluation under Boston–Singapore splits. L2 (m) and collision rate (Coll, %) are averaged over the 3 s prediction horizon; lower is better. Results are mean

\pm

std over three downstream-training seeds. Swin denotes the original LAW backbone, Public denotes large-scale pretrained models, and navbal denotes our ViT-S/14 models pretrained on balanced NAVSIM NavTrain subsets. The final row reports the difference between the Swin baseline and the best SSL result in each column (Swin

-

Best SSL); positive values indicate an improvement from SSL.

Backbone	Flipped B $\rightarrow$ S		Flipped S $\rightarrow$ B
Backbone	L2 @avg $\downarrow$	Coll @avg $\downarrow$	L2 @avg $\downarrow$	Coll @avg $\downarrow$
Swin-Tiny (supervised, original LAW)	4.02 $\pm$ 0.48	5.68 $\pm$ 0.75	1.15 $\pm$ 0.19	0.84 $\pm$ 0.63
DINOv2 (ViT-g/14, public, frozen)	2.86 $\pm$ 0.09	3.34 $\pm$ 0.97	2.15 $\pm$ 0.44	1.75 $\pm$ 1.31
I-JEPA (ViT-H/14, public, frozen)	2.99 $\pm$ 0.24	2.38 $\pm$ 0.61	2.00 $\pm$ 0.76	1.29 $\pm$ 0.51
MAE (ViT-H/14, public, frozen)	2.96 $\pm$ 0.85	2.46 $\pm$ 2.16	3.61 $\pm$ 1.25	1.83 $\pm$ 1.11
DINOv2 (ViT-S/14, navbal, frozen)	4.04 $\pm$ 0.15	5.98 $\pm$ 0.33	3.82 $\pm$ 0.04	6.58 $\pm$ 0.09

TABLE III: Test-time horizontal-flip diagnostic for the cross-city LAW evaluation in Table II . At test time, camera images are horizontally flipped before inference, and the predicted trajectories are horizontally flipped back to the original coordinate frame before evaluation. Metrics and all other aspects of the evaluation protocol are unchanged.

Pretraining	Backbone	Las Vegas		Boston		Pittsburgh		Singapore		4-region joint
Pretraining	Backbone	TF	LTF	TF	LTF	TF	LTF	TF	LTF	TF	LTF
Supervised	ResNet34	$46.8{\pm}2.7$	$46.5{\pm}1.6$	$53.1{\pm}0.5$	$50.0{\pm}2.6$	$58.2{\pm}2.0$	$56.6{\pm}1.1$	$\mathbf{46.7{\pm}3.7}$	$44.5{\pm}2.9$	$72.8{\pm}1.6$	$69.6{\pm}1.7$
Self- Supervised	DINOv2 (ViT-g/14, public)	$\mathbf{52.2{\pm}0.6}$	$\mathbf{52.7{\pm}1.3}$	$52.8{\pm}4.3$	$\mathbf{59.5{\pm}3.2}$	$55.9{\pm}5.6$	$\mathbf{60.4{\pm}1.0}$	$43.7{\pm}5.2$	$\mathbf{47.9{\pm}4.3}$	$69.9{\pm}7.2$	$\mathbf{72.6{\pm}3.5}$
	I-JEPA (ViT-H/14, public)	$46.8{\pm}1.1$	$48.4{\pm}1.9$	$52.1{\pm}2.1$	$51.9{\pm}3.5$	$58.6{\pm}0.9$	$59.6{\pm}1.3$	$40.2{\pm}4.6$	$45.7{\pm}2.9$	$68.4{\pm}5.4$	$69.4{\pm}4.2$
	MAE (ViT-H/14, public)	$49.4{\pm}3.2$	$51.0{\pm}2.9$	$54.2{\pm}0.6$	$52.0{\pm}6.0$	$57.1{\pm}6.3$	$60.3{\pm}0.7$	$38.0{\pm}6.1$	$45.7{\pm}3.3$	$70.5{\pm}2.6$	$68.5{\pm}5.6$
	DINOv2 (ViT-S/14, navbal)	$50.0{\pm}2.2$	$49.0{\pm}2.8$	$52.7{\pm}2.1$	$51.2{\pm}3.2$	$58.5{\pm}2.3$	$50.2{\pm}4.3$	$38.1{\pm}2.2$	$45.0{\pm}2.8$	$70.6{\pm}2.3$	$70.2{\pm}3.0$

TABLE IV: NAVSIM representation comparison with frozen pretrained backbones. Under single-region training, values report mean OOD EPDMS over the three unseen regions; under joint four-region training, values report Total EPDMS on NavTest. Results are mean

\pm

std over three downstream-training seeds; higher is better. Best performance in each column is shown in bold.

Fig. 2: EPDMS change under the test-time horizontal-flip diagnostic across 360 source–target evaluations with opposite driving-side conventions.

	Boston	Las Vegas	Pitts.	Sing.
Vehicles within 20 m	4.96	4.84	2.57	0.72
Vehicles in ego corridor	0.63	0.81	0.44	0.10
Stationary share	3.7%	4.0%	2.0%	0.5%
Turning share	41%	31%	46%	67%
Ego-speed proxy (m/s)	4.95	3.98	5.68	5.37
Nearest vehicle (m)	11.2	10.6	27.4	51.3

TABLE V: Scenario composition of the balanced NAVSIM training subsets. Vehicle statistics are computed from the current frame, while stationary, turning, and ego-speed measures are derived from the future ego trajectory. The ego-speed proxy is forward displacement divided by the 4 s horizon.

Fig. 3: Source scenario content and NAVSIM transfer degradation. (a) Singapore-source EPDMS penalty across sparse-, mid-, dense-traffic, and stationary test scenes on Boston, Las Vegas, and Pittsburgh; the stationary subset contains 437 samples. (b) Relationship between the percentage of stopped scenes in the source training data and stationary-scene performance, averaged over target regions and backbone families.

Fig. 4: Source content design experiment, EPDMS results on NAVSIM. Training on the sparse Las Vegas subset produces the largest degradation on stationary scenes, with smaller changes on moving scenes.

Fig. 5: Representative low-speed behavior across benchmarks using the same frozen MAE ViT-S/14 architecture with source-specific SSL checkpoints. (a) LAW on nuScenes. (b) TransFuser on NAVSIM. In both examples, the Singapore-trained model predicts excessive forward motion in a stationary scene.

Backbone	Train	Eval region	NC	DAC	DDC	TLC	EP	TTC	LK	HC	TFEC	EPDMS
DINOv2 ViT-g/14	Pittsburgh	Pittsburgh (ID)	$\mathbf{93.6\pm 1.3}$	$75.9\pm 5.8$	$91.4\pm 2.3$	$\mathbf{99.8\pm 0.2}$	$86.1\pm 1.5$	$92.3\pm 1.0$	$84.8\pm 4.1$	$96.2\pm 0.4$	$35.2\pm 5.1$	$\mathbf{60.8\pm 4.4}$
		Las Vegas	$\mathbf{91.1\pm 1.2}$	$\mathbf{87.6\pm 2.4}$	$98.8\pm 0.4$	$\mathbf{100.0\pm 0.0}$	$87.1\pm 2.0$	$86.9\pm 1.1$	$\mathbf{89.9\pm 2.2}$	$98.7\pm 0.5$	$42.0\pm 2.0$	$\mathbf{72.7\pm 2.4}$
		Boston	$88.3\pm 1.1$	$77.7\pm 3.0$	$\mathbf{93.8\pm 1.6}$	$99.6\pm 0.1$	$85.4\pm 1.0$	$84.9\pm 2.0$	$87.4\pm 3.1$	$97.9\pm 0.2$	$43.9\pm 3.0$	$\mathbf{61.8\pm 2.3}$
		Singapore	$94.8\pm 0.6$	$\mathbf{60.6\pm 4.8}$	$\mathbf{90.5\pm 2.7}$	$\mathbf{99.8\pm 0.0}$	$\mathbf{72.3\pm 2.0}$	$93.6\pm 0.5$	$78.0\pm 2.6$	$98.2\pm 0.7$	$24.2\pm 7.9$	$\mathbf{46.8\pm 3.0}$
	Singapore	Singapore (ID)	$96.2\pm 0.7$	$59.6\pm 4.4$	$\mathbf{94.9\pm 1.0}$	$\mathbf{99.8\pm 0.0}$	$73.1\pm 3.6$	$94.8\pm 1.4$	$80.7\pm 7.2$	$98.7\pm 0.2$	$27.4\pm 2.8$	$47.3\pm 2.0$
		Las Vegas	$75.7\pm 5.2$	$\mathbf{88.5\pm 2.3}$	$\mathbf{98.1\pm 1.0}$	$\mathbf{99.1\pm 0.3}$	$89.7\pm 3.9$	$71.4\pm 6.0$	$82.0\pm 7.3$	$99.0\pm 0.2$	$28.4\pm 11.4$	$\mathbf{57.9\pm 4.5}$

TABLE VI: EPDMS sub-component breakdown for Pittsburgh- and Singapore-trained Latent TransFuser models. Values are mean

\pm

std over three downstream-training seeds. Models are trained on one source region and evaluated both in-domain (ID) and zero-shot on the remaining regions. DINOv2 and I-JEPA use frozen public pretrained backbones; ResNet34 is the supervised reference. Best performance for each fixed train–evaluation pair and metric is shown in bold.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Backbone	Adaptation	Train	TransFuser EPDMS (%)						Latent TransFuser EPDMS (%)
Backbone	Adaptation	Train	LV	B	P	S	OOD	Total	LV	B	P	S	OOD	Total
ResNet34	Supervised	4-region	82.5	74.6	68.1	54.7	–	72.8	79.4	70.2	62.1	57.1	–	69.6
		LV	79.2	59.0	44.1	37.4	46.8	59.4	78.6	58.8	43.5	37.2	46.5	59.0
		B	70.0	62.7	43.5	45.8	53.1	58.6	65.6	59.2	40.5	43.8	50.0	55.2
		P	70.7	62.7	61.1	41.2	58.2	61.6	68.1	61.6	60.6	40.0	56.6	60.2
		S	57.9	42.8	39.3	50.0	46.7	48.3	55.8	40.4	37.4	48.7	44.5	46.2

Appendix

TABLE VII: Detailed NAVSIM cross-region EPDMS (%) for TransFuser and Latent TransFuser. Models are trained either jointly on all four regions or on one source region and evaluated on each NavTest region. For single-region training, OOD is the unweighted mean EPDMS over the three unseen regions, whereas Total is the population-weighted EPDMS over the full NavTest split. OOD is not defined for four-region training and is shown as “–”. For single-region training, bold values denote in-domain performance; for four-region training, bold denotes Total EPDMS. Higher is better.

End-to-End Autonomous Driving

End-to-end autonomous driving learns to turn sensor observations and driving context into vehicle actions or trajectories.

See all 227 papers

May 28, 2026

CityGen: Structure-Guided City-Style Synthesis for Cross-City Autonomous Driving

Autonomous driving systems are commonly trained and evaluated within limited geographic regions, which hinders their scalability when deployed in new cities. However, significant domain shifts in appearance, road topology, and traffic patterns often cause severe performance degradation under cross-city deployment. Existing approaches based on domain adaptation, data augmentation, or synthetic data generation typically rely on labeled target data, city-specific annotations, or task-specific designs, limiting their scalability and effectiveness for holistic evaluation. In this paper, we introduce CityTransfer-Bench, a geographically disjoint benchmark for evaluating cross-city generalization across perception, segmentation, and planning, and propose CityGen, a diffusion-based generative framework that performs zero-label city adaptation via HD-map-conditioned synthesis guided by city-level visual prompts. Extensive experiments demonstrate that CityGen consistently improves cross-city robustness across multiple tasks, establishing a scalable and label-efficient foundation for generalizable autonomous driving.

Zezhong Qian, Zhao Yang, Lu Tan +4

1Jiangsu Cytoderm Intelligent Technology Co., Ltd., China · 2Xi’an Jiaotong University, Xi’an, China · 3Tsinghua University, Beijing, China +1

Domain Adaptation Conditional Image Generation Synthetic Data Generation Spatial Generalization Autonomous Driving Autonomous Driving Scenario Generation

Ask PDF

Jul 5, 2026

Geographic Diversity Beats Data Volume for Cross-Domain Generalization in Zero-Label JEPA Driving World Models

Self-supervised latent world models can assign a surprise score to driving scenarios without any human labels. A natural follow-up question is whether such a model, trained on driving data from one geographic region, can generalize its notion of complexity to unseen cities and sensor configurations. We study this question through a controlled transfer experiment: we train JEPA-based world models on nuPlan data (Pittsburgh, Boston, Singapore) and evaluate zero-shot on held-out Argoverse 2 validation scenarios from Miami and Austin. We find that models trained on geographically diverse data generalize significantly better than models trained on equal amounts of single-geography data. In a matched-scale ablation at 63,000 scenarios per condition (n=3 seeds each), combined training reduces mean surprise score by 16.5% relative to nuPlan-only training (0.228 +/- 0.015 vs 0.273 +/- 0.008). Notably, training on 200,000 AV2-only scenarios (3x more data from one geography) still produces higher surprise (0.264) than the combined 63K model, suggesting that geographic diversity is a stronger predictor of cross-domain generalization than raw data volume.

Santosh Jaiswal

Autonomous Driving Datasets Zero-Shot Learning World Model Learning Spatial Generalization Cross-Domain Generalization World Models for Autonomous Driving

Ask PDF

Jan 29, 2026

Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving

End-to-end autonomous driving increasingly leverages self-supervised video pretraining to learn transferable planning representations. However, pretraining video world models for scene understanding has so far brought only limited improvements. This limitation is compounded by the inherent ambiguity of driving: each scene typically provides only a single human trajectory, making it difficult to learn multimodal behaviors. In this work, we propose Drive-JEPA, a framework that integrates Video Joint-Embedding Predictive Architecture (V-JEPA) with multimodal trajectory distillation for end-to-end driving. First, we adapt V-JEPA for end-to-end driving, pretraining a ViT encoder on large-scale driving videos to produce predictive representations aligned with trajectory planning. Second, we introduce a proposal-centric planner that distills diverse simulator-generated trajectories alongside human trajectories, with a momentum-aware selection mechanism to promote stable and safe behavior. When evaluated on NAVSIM, the V-JEPA representation combined with a simple transformer-based decoder outperforms prior methods by 3 PDMS in the perception-free setting. The complete Drive-JEPA framework achieves 93.3 PDMS on v1 and 87.8 EPDMS on v2, setting a new state-of-the-art.

Linhan Wang, Zichong Yang, Chen Bai +6

Virginia Tech · Purdue University · XPENG Motors +1

End-to-End Autonomous Driving NavSim Joint-Embedding Predictive Architecture Video Representation Learning Autonomous Driving Planning

Ask PDF

Abstract

End-to-End Autonomous Driving

Explore similar work

CityGen: Structure-Guided City-Style Synthesis for Cross-City Autonomous Driving

Geographic Diversity Beats Data Volume for Cross-Domain Generalization in Zero-Label JEPA Driving World Models

Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving