TerraTransfer++: Demonstration-Free End-to-End Planner

Zikang Xiong; Weixin Li; Zhouchonghao Wu; Akshay Rangesh; Saarth Bonde; Grantland Hall; Chen Tang; Yihan Hu

cs.ROOct 21, 2026

TerraTransfer++: Demonstration-Free End-to-End Planner

Authors: Zikang Xiong, Weixin Li, Zhouchonghao Wu, Akshay Rangesh, Saarth Bonde, Grantland Hall, Chen Tang, Yihan Hu,

Organizations: Applied Intuition · University of California, Los Angeles · University of California, Berkeley

Abstract

End-to-end autonomous driving has achieved state-of-the-art performance on benchmarks and real-world deployments. Its standard training recipe, however, is expensive across all existing paradigms: imitation learning requires fleet-scale driving logs whose behavior distribution only thinly covers the safety-critical tail; closed-loop reinforcement learning in a photorealistic simulator, while offering diverse behaviors, is bottlenecked by sample-efficiency, per-step cost of rendering, and a forward pass through a large vision backbone. Self-play in vectorized simulators changes the economics: at millions of rollout steps per second, one can obtain a state distribution with natural diversity in collisions, near-misses, and recoveries that no driving log contains, and a highly reactive policy emerges from a basic driving-quality reward. Our approach exploits this asymmetry. We pretrain a single policy by self-play, distill its rollouts into a flow-matching trajectory teacher that also plans, and train lightweight adapters on a frozen pretrained vision backbone to reproduce the latent space of either teacher through the action KL divergence, a batch-relational low-rank structural loss, and, for the planner, a trajectory-matching loss. Every target traces back to the self-play policy, so alignment never supervises against a logged trajectory: a paired dataset of (image, scene-state) frames suffices, with no need for the curated expert demonstrations that imitation pretraining is built on. From this one self-play teacher we obtain two end-to-end students: a closed-loop policy that outperforms every published imitation-based end-to-end method on the photorealistic 3D Gaussian splatting closed-loop benchmark HUGSim, and a planner that reaches 59.4 EPDMS on NAVSIM navhard, above every published method including DriveZero-Scale (57.1), and 95.5 PDMS on navtest, without human demonstrations.

Figures & tables

Panel of Figure 1 — Figure 1. Conventional training pipeline vs. our reinforcement-learning pretraining pipeline. Conventional recipes imitate fleet logs and fine-tune on curated demonstrations, logged trajectories, or rendered simulation. They therefore need human data or rendering at every stage. Phase 1 (top right) trains a self-play policy $\pi_{\theta}$ with proximal policy optimization (PPO) in a multi-agent vectorized simulator. Phase 2 (middle right) freezes the policy as the self-play teacher and distills its rollouts into a trajectory teacher that also outputs a planned trajectory; both teachers read vector state. Phase 3 (bottom right) aligns an image-conditioned student to either frozen teacher: the self-play teacher yields the closed-loop student, and the trajectory teacher yields the planning student. The student keeps the teacher’s ego encoder unchanged and is not aligned on it; only the teacher’s scene features (road and partner, plus traffic for the trajectory teacher) enter the structural loss. An action loss matches the teacher’s action distribution, while a structural loss matches its pairwise relationships among scenes; for the planning student, an L2 loss also matches the plan the trajectory teacher selects among $16$ samples. Alignment uses paired (image, scene-state) frames; it needs neither demonstrations nor online rendering during training. Blue modules are trainable; outlined modules are frozen. In the blue DINOv3 + Adapter block, only the adapter trains; the backbone stays frozen. The trajectory head and the L2 loss apply to the planning student only.

	Closed-loop HD-Score					Native HD-Score
Method	Easy	Medium	Hard	Extreme	All	Easy	Medium	Hard	Extreme	All
nuScenes
UniAD ( Hu et al., 2023 )	0.423	0.231	0.286	0.122	0.259	0.412	0.234	0.290	0.129	0.260
VAD ( Jiang et al., 2023 )	0.443	0.256	0.270	0.108	0.267	0.424	0.246	0.257	0.113	0.257
LTF ( Chitta et al., 2023 )	0.708	0.440	0.326	0.111	0.404	0.702	0.440	0.331	0.124	0.407
ECO ( Zhang et al., 2025a )	0.764	0.416	0.405	0.255	0.452	0.846	0.431	0.403	0.263	0.476

Table 1. Closed-loop and native HD-Score across all four HUGSim dataset splits. The left block reports closed-loop HD-Score, and the right block reports native planned-trajectory HD-Score; both blocks score the same re-run rollouts. UniAD, VAD, and LTF are re-run from their released checkpoints on the current scenes and maps; ECO uses its released checkpoint. Results are grouped by difficulty tier for nuScenes (

88

scenarios), PandaSet (

127

), KITTI-360 (

113

), and Waymo (

108

). The student emits a single jerk action per step and is scored natively with the decel stand-in plan. The italicized teacher row is the privileged reference and is excluded from best-per-column counts. Higher is better; bold marks the best comparison entry per column within each split.

Figure 2. Native HD-Score on the shared

345

-scenario protocol. Our student exceeds both concurrent self-play-to-pixel systems; the teacher upper-bounds the student. Our rows score the decel stand-in plan, which satisfies the comfort bounds by construction; GigaPixel and DriveZero are scored on their own plans.

Figure 3. Visually implausible extreme-tier scenarios. Some HUGSim Extreme scenarios begin from implausible configurations: other views are occluded, and vehicles appear to pass through one another. Configurations like these trigger the student’s conservative yielding, which sacrifices route completion and pulls the Extreme scores of Table 1 down for every method.

			navtest	navhard
Method	Demo	Priv.	PDMS $\uparrow$	Stage 1 $\uparrow$	Stage 2 $\uparrow$	EPDMS $\uparrow$
Privileged reference (ground-truth scene state)
Human driver	–	–	94.8	–	–	–
PDM-Closed ( Dauner et al., 2023 )	✓	✓	89.1	–	–	56.6
DriveRL ( He et al., 2026 )	✗	✓	95.8	84.1	65.4	55.6
Trajectory teacher (ours)	✗	✓	95.9	92.2	67.5	62.1

Table 2. Pseudo closed-loop planning on NAVSIM. navtest reports PDMS (NAVSIMv1); navhard reports the per-stage scores and the two-stage EPDMS (NAVSIMv2). Baseline rows are the published results compiled by He et al. (2026) ; – marks a score that compilation does not list. Demo marks whether human trajectory demonstrations were used as training supervision; Priv. marks whether the planner reads privileged ground-truth scene state (vectorized agents and map) at test time rather than sensor input. Italicized rows are privileged references, not comparison entries. Bold marks the best score in each group. Our rows are the trajectory teacher of § 2.3 and the trajectory student of § 2.4 ; neither uses human trajectory supervision. Both teacher rows keep the best of

16

sampled plans (§ 3.1 ); the student row scores a single sampled plan with no selection, so it reads no ground-truth state; IDM selection forecasts the other agents with the intelligent driver model instead of constant-velocity extrapolation.

Training paradigm	CL HD-Score
Imitation (nuPlan logged actions)	$0.206$
Modular cascade (SparseDrive)	$0.361$
Self-play pretraining + alignment (ours)	0.574
Action-only distillation	$0.434$
Structural-only distillation	$0.373$

Table 4. Paradigm ablation under matched data and compute. Closed-loop HD-Score on the nuScenes split. Each row uses the same

1.83

M nuPlan paired frames without the HUGSim domain-adaptation frames and the same compute budget; the cascade replaces the vision encoders with a SparseDrive perception module. The second block holds out one alignment loss term at a time, trained on the same data and compute budget.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Reward parameter	Coefficient setting	Effect
Collision	$\alpha_{\text{collision}}\sim\mathcal{U}(0.0,3.0)$	Collision avoidance
Collision speed	$\alpha_{\text{coll-speed}}=0.1$	Impact severity
Boundary	$\alpha_{\text{boundary}}\sim\mathcal{U}(0.0,3.0)$	Drivable-area compliance
Comfort	$\alpha_{\text{comfort}}\sim\mathcal{U}(0.0,0.1)$	Acceleration/jerk comfort
Lane alignment	$\alpha_{l\text{-align}}\sim\mathcal{U}(2.5{\times}10^{-4},2.5{\times}10^{-2})$	Lane heading
Velocity alignment	$\alpha_{v\text{-align}}\sim\mathcal{U}(0.0,1.0)$	Forward progress

Appendix

Table 5. Vehicle reward coefficients used during self-play. Ranges denote independent per-vehicle, per-episode uniform sampling.

Sub	Native HUGSim HD-Score	Closed-loop variant (this work)
NC	OBB ego–partner collision per keyframe; vehicle-only partner filter; output $\in\{0,1\}$ .	Same.
DAC	Four-corner-inside check against per-scene drivable-area polylines; output $\in\{0,0.5,1\}$ .	Same.
TTC	Entire planned trajectory ( $7$ poses, $\leq\!3.5\,\mathrm{s}$ ) rigidly shifted by ego velocity at two horizons $\Delta\in\{0.5,1.0\}\,\mathrm{s}$ ; collision of every shifted ego pose vs. frozen partners $+$ scene geometry ( $14$ probes); binary $\{0,1\}$ .	Realized ego and all valid partners velocity-propagated on a fine grid (stride $\Delta_{\text{TTC}}=0.2\,\mathrm{s}$ , horizon $H_{\text{TTC}}=1.0\,\mathrm{s}$ ); earliest ego collision vs. propagated partners $+$ 3DGS background ( $>\!N_{\text{bg}}$ enclosed points) gives fractional $\min(1,\,\tau^{\star}_{t}/H_{\text{TTC}})\in[0,1]$ (Eq. 19 ); $\delta{=}0$ collision $\Rightarrow$ TTC ${}_{t}=0$ .
COM	Kinematic comfort bounds (lon. accel., lon. jerk, yaw accel., yaw rate; a lateral bound is coded but inert) on the keyframe’s planned trajectory poses; binary $\{0,1\}$ .	Same active bounds on the realized ego trajectory over a seconds-parameterized window of duration $W_{\text{COM}}=1.0\,\mathrm{s}$ (Eq. 20 ).
$R_{c}$	$\min(1,\max_{t}r_{t})$ with $r_{t}=(k_{t}{+}1)/(0.9N)$ : a camera-pose- index fraction ( $k_{t}$ = nearest pose index, $N$ = pose count) that saturates at $90\%$ of the poses.	Same (Eq. 21 ); retained so all rows share one route-completion protocol.
Agg.	$\mathrm{mean}_{t}[\mathrm{NC}_{t}\cdot\mathrm{DAC}_{t}\cdot(5\,\mathrm{TTC}_{t}+2\,\mathrm{COM}_{t})/7]\cdot R_{c}$ .	Same (Eq. 16 ).

Appendix

Table 6. Native HD-Score vs. closed-loop HD-Score per subscore. NC, DAC,

R_{c}

, and the aggregation formula are unchanged; TTC and COM are redefined to operate on the realized rollout.

			Native (decel stand-in plan)						Closed-loop (realized rollout)
Set	N	$R_{c}$	NC	DAC	TTC	COM	PDMS	HD	NC	DAC	TTC	COM	PDMS	HD
nuScenes
Easy	18	0.949	0.944	0.997	0.934	1.000	0.925	0.875	0.980	0.999	0.967	0.978	0.954	0.906
Medium	34	0.716	0.923	0.999	0.907	1.000	0.903	0.657	0.978	1.000	0.964	0.971	0.950	0.684
Hard	18	0.732	0.832	0.997	0.835	1.000	0.817	0.653	0.965	0.999	0.942	0.947	0.920	0.685
Extreme	18	0.301	0.743	0.997	0.713	1.000	0.707	0.250	0.959	0.999	0.925	0.931	0.889	0.279

Appendix

Table 7. Self-play teacher across all

436

HUGSim scenarios and difficulty tiers, under both scorers. The teacher uses privileged vectorized scene state; background traffic is replayed. The Native block scores the decel stand-in plan with HUGSim’s planned-trajectory subscores; the Closed-loop block scores the realized rollout with the redefined TTC and COM.

R_{c}

is shared. PDMS is the NC, DAC, TTC, and COM aggregate before route completion (HD-PDMS); HD

=

PDMS

\times R_{c}

			Native (decel stand-in plan)						Closed-loop (realized rollout)
Set	N	$R_{c}$	NC	DAC	TTC	COM	PDMS	HD	NC	DAC	TTC	COM	PDMS	HD
nuScenes
Easy	18	0.934	0.957	1.000	0.950	1.000	0.946	0.884	0.981	1.000	0.972	0.975	0.962	0.901
Medium	34	0.721	0.916	0.998	0.900	1.000	0.897	0.650	0.974	1.000	0.956	0.974	0.943	0.682
Hard	18	0.690	0.853	0.997	0.850	1.000	0.838	0.620	0.963	0.999	0.946	0.985	0.934	0.641
Extreme	18	0.295	0.760	1.000	0.736	1.000	0.728	0.251	0.968	1.000	0.940	0.945	0.913	0.278

Appendix

Table 8. Aligned end-to-end student across all

436

HUGSim scenarios and difficulty tiers, under both scorers. The student consumes camera images and emits one jerk action per step. The Native block scores the decel stand-in plan (§ A.2 ); the Closed-loop block scores the realized rollout.

R_{c}

is shared. PDMS is the NC, DAC, TTC, and COM aggregate before route completion (HD-PDMS); HD

=

PDMS

\times R_{c}

Figure 5. Vision-backbone ablation. Closed-loop HD-Score under a fixed alignment recipe for each image backbone, trained on the headline data including the

300

K HUGSim domain-adaptation frames.

Method	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	C $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
Privileged reference (ground-truth scene state)
Human driver	100	100	100	99.9	87.5	94.8
PDM-Closed ( Dauner et al., 2023 )	94.6	99.8	89.9	86.9	99.9	89.1
DriveRL ( He et al., 2026 )	99.8	99.9	99.1	99.0	91.5	95.8
Trajectory teacher (ours)	99.0	99.6	93.2	99.8	97.2	95.9
Camera & LiDAR input, human demonstrations

Appendix

Table 9. Per-subscore breakdown on the NAVSIMv1 navtest split. PDMS aggregates NC, DAC, TTC, C, and EP (one decimal). Baseline rows are published numbers; italicized rows are privileged references. Bold marks the best result in each input group. GigaPixel ( Rowe et al., 2026 ) is omitted because it reports navhard results only.

			Subscores (Stage 1 / Stage 2)
Method	Demo	Stage	NC	DAC	DDC	TLC	EP	TTC	LK	HC	EC	S.	EPDMS
Privileged reference (ground-truth scene state)
PDM-Closed ( Dauner et al., 2023 )	✓	S1	94.4	98.8	100	99.5	100	93.5	99.3	87.7	36.0	–	56.6
		S2	90.5	90.6	95.4	98.4	100	86.6	74.2	91.9	29.7	–	56.6
DriveRL ( He et al., 2026 )	✗	S1	99.1	99.6	100.0	99.6	91.1	98.7	99.3	90.9	18.2	84.1	55.6
		S2	94.4	92.0	96.4	98.1	89.1	89.1	73.8	88.5	10.9	65.4	55.6

Appendix

Table 10. Per-subscore breakdown on the NAVSIMv2 navhard split. EPDMS aggregates NC, DAC, DDC, TLC, EP, TTC, LK, HC, and EC over two stages; S. is the per-stage EPDM score. Stage 2 re-scores under Gaussian-Splatting-based ego-state perturbations. Baseline rows are published numbers; italicized rows are privileged references. Our rows are measured under the standard protocol.

End-to-End Autonomous Driving

End-to-end autonomous driving learns to turn sensor observations and driving context into vehicle actions or trajectories.

See all 227 papers

TerraTransfer: Learning End-to-End Driving Policies Without Expert Demonstrations
Jun 16, 2026Zikang Xiong, Weixin Li, Zhouchonghao Wu +6End-to-End Autonomous Driving RL for Autonomous Driving
1Applied Intuition · 2UCLA · 3UC Berkeley
Scaling Self-Play for End-to-End Driving
Jun 17, 2026Luke Rowe, Roger Girgis, Rodrigue de Schaetzen +6Self-Play RL End-to-End Autonomous Driving
Mila · Université de Montréal · Polytechnique Montréal +4
DriveZero: End-to-End Driving Beyond Human Demonstrations
Sep 5, 2026Hao He, Chengcheng Hu, Zirun Su +17End-to-End Autonomous Driving RL for Autonomous Driving
Xiaomi EV AD & Robotics, L3 Team

Jun 16, 2026J/K move · Enter open · S save

TerraTransfer: Learning End-to-End Driving Policies Without Expert Demonstrations

Zikang Xiong, Weixin Li, Zhouchonghao Wu, Akshay Rangesh+5

1Applied Intuition · 2UCLA · 3UC Berkeley

End-to-End Autonomous Driving RL for Autonomous Driving Autonomous Driving Simulation

Open paper AskPDF arXiv

End-to-end autonomous driving has achieved state-of-the-art performance on benchmarks and real-world deployments. Its standard training recipe, however, is expensive across all stages: collecting and labeling millions of driving frames is costly, and closed-loop RL on images is bottlenecked by the per-step cost of photorealistic rendering plus a forward pass through a large vision backbone. Self-play in vectorized simulators changes the economics: millions of rollout steps per second, and a state distribution naturally rich in collisions, near-misses, and recoveries that no driving log contains. Our approach exploits this asymmetry by decoupling learning to drive from learning to see. We pretrain a single policy by self-play, then align its latent space with a pretrained vision backbone, through the action KL divergence and a batch-relational low-rank structural loss. The action target comes from the self-play policy, so alignment never supervises against a logged trajectory: a paired dataset of (image, scene-state) frames suffices, with no need for the curated expert demonstrations that imitation pretraining is built on. On photorealistic 3D Gaussian splatting closed-loop scenarios, the resulting end-to-end policy matches or exceeds prior end-to-end methods.

Abstract

End-to-End Autonomous Driving

Explore similar work

TerraTransfer: Learning End-to-End Driving Policies Without Expert Demonstrations

Scaling Self-Play for End-to-End Driving

DriveZero: End-to-End Driving Beyond Human Demonstrations