Beyond Planning: Differentiable Control-in-the-loop End-to-End Autonomous Driving

Zhidong Wang; Zirui Li; Zitong Shan; Zhiqi Mao; Qihang Huang; Zhan Chen; Jingsong Liang; Han Yu

cs.ROOct 13, 2026

Beyond Planning: Differentiable Control-in-the-loop End-to-End Autonomous Driving

Authors: Zhidong Wang, Zirui Li, Zitong Shan, Zhiqi Mao, Qihang Huang, Zhan Chen, Jingsong Liang, Han Yu,

Organizations: School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore · Collaborative Initiative, Interdisciplinary Graduate Programme, Nanyang Technological University, Singapore · College of Computing and Data Science, Nanyang Technological University, Singapore

Abstract

End-to-end autonomous driving has achieved remarkable progress by planning trajectories directly from raw sensor data. Yet the step from planning to control still relies on hand-tuned tracking controllers that block the gradient path from vehicle motion back to the network. This gap stems from non-differentiable vehicle dynamics, aliased supervision from coarsely sampled expert labels, rate mismatch between high-rate control and low-rate re-planning, and vehicle model discrepancies between training and deployment. This paper proposes DIVA (Differentiable Identified Vehicle dynamics for end-to-end Autonomous driving) to close this gap. An identified actuator-level vehicle model is embedded in the training loop of the end-to-end driving network, forming a differentiable chain from perception to vehicle states. A learning-based control decoder replaces the external controller and is supervised on the rolled-out vehicle states, bypassing the aliased control-input labels. A rate-consistent control-input parameterization reconciles the heterogeneous sampling rates and enables asynchronous execution with a 3.8 ms control-only cycle. On-policy re-identification and an online disturbance observer address training-time distribution shift and deployment-time model mismatch respectively. On the closed-loop Bench2Drive benchmark, DIVA achieves a driving score of 86.7, surpassing the widely adopted state-of-the-art approach by 3.3 points while nearly halving mean jerk and reducing brake switching by 64%, and maintains superior path tracking accuracy under significant vehicle model mismatch at deployment.

Figures & tables

Fig. 1: Overview of DIVA: the control decoder cross-attends to memory tokens that combine the scene and planning features, the explicit route and target-speed reference, the state feedback and the disturbance estimate, and its control-input knots pass through a first-order hold into the identified differentiable vehicle dynamics, whose longitudinal and lateral actuator channels drive a single-track model integrated at 20 Hz. Under imitation learning in state space, the state loss propagates along the gradient path back through the vehicle dynamics into the control and planning decoders, and at deployment the control decoder runs on a fast path over cached perception, executing receding-horizon control inputs conditioned on the disturbance estimate of the online observer.

Fig. 2: On-policy re-identification: (a) coverage of the identification excitations (gray) and of the base controller (red, 50% and 85% contours); (b) map at throttle 0.7; (c) 141 on-policy upshifts; (d) 49 on-policy launches; (e) speed bias of both models under the controller’s commands; (f) position error on held-out drives; kinematic bicycle: the constant-parameter model of [ 31 ] shipped with TF++.

Method	Planner	DS $\uparrow$	SR (%) $\uparrow$	Effi. $\uparrow$	Comf. $\uparrow$	Over- taking	Merg- ing	Emerg. brake	Give way	Traffic signs	Jerk (m/s 3 )	$\|a_{x}\|$ (m/s 2 )	WOT frac.	Brake sw. (1/min)	Steering rate (1/s)	Lat. off. p95 (m)	Speed err. (m/s)	Speed (m/s)
		Official metrics				Ability (%) $\uparrow$					Execution metrics $\downarrow$
TF++ [ 30 ]	baseline	83.34	67.6	250.97	33.9	55.56	58.75	78.33	50.00	86.84	16.42	4.54	0.823	42.7	0.173	0.159	3.36	8.67
MPC+DOB	baseline	79.59	61.2	259.22	38.5	37.78	53.75	76.67	50.00	82.63	10.92	3.74	0.113	15.8	0.097	0.220	3.33	8.67
DIVA+PID	joint	83.80	67.1	248.31	31.4	60.00	56.25	78.33	50.00	82.63	16.72	4.57	0.821	42.8	0.169	0.126	3.04	8.72
DIVA (ours)	joint	86.73	71.7	256.29	36.6	55.56	65.82	90.00	40.00	87.89	8.86	3.19	0.203	15.4	0.098	0.100	2.96	8.63

TABLE I: Closed-Loop Driving and Execution Results on Bench2Drive-220

Fig. 3: Execution metrics on Bench2Drive-220: (a) jerk RMS and deceleration p99, (b) lateral acceleration and mean sideslip angle, (c) comfort-limit exceedance fraction for

a_{x}

and jerk, (d) throttle total variation and steering-rate. TF++ and MPC+DOB: baseline planner; DIVA+PID and DIVA: joint planner.

	Closed loop									Offline, 2 s horizon
						WOT	Lat.	Brake	Speed	Lat.		$u_{s,0}$
		$\Delta$ DS $\uparrow$	SR $\uparrow$		Jerk $\downarrow$	frac.	offset $\downarrow$	sw. $\downarrow$	err. $\downarrow$	p90 $\downarrow$	Speed $\downarrow$	error
Variant	DS $\uparrow$	[95% CI]	(%)	RC $\uparrow$	(m/s 3 )	$\downarrow$	p95 (m)	(1/min)	(m/s)	(m)	(m/s)	$\downarrow$
DIVA (ours)	69.3	–	36.5	91.69	10.84	0.21	0.109	17.8	4.65	0.306	1.110	0.0145
DIVA+PID	60.1	$-9.27$ [ $-16.6$ , $-2.0$ ] †	17.3	90.78	15.87	0.82	0.143	33.5	4.64	–	–	–
w/o vehicle model, control-input BC	62.0	$-7.36$ [ $-14.3$ , $-0.5$ ] †	21.2	91.78	12.78	0.11	0.159	24.2	4.54	0.543	1.345	0.0094

TABLE II: Ablation on Hard-52: Driving Score, Closed-Loop and Offline Metrics

Fig. 4: Lateral offset (p95, top) and mean jerk (bottom) under sustained vehicle-side mismatch on the traffic-free maneuver suite (16 routes). Columns: perturbed quantity with nominal at factor 1, bias 0 or dead time 0. All controllers on the joint planner.

Fig. 5: DS and tracking under vehicle-side mismatch, all controllers on the joint planner. (a) Maneuver suite: DS (top) and speed error (bottom) vs. perturbed quantity. (b) Hard-52 with traffic: DS, lateral offset p95, speed error and jerk for the nominal vehicle (shaded) and one strength of four perturbation families.

Fig. 6: Steering bias $+0.10$ on suite route 2084 (junction left turn at night). (a) Paths of the five controllers. (b) Front camera of DIVA+PID at 5.8 s, 0.5 s before its collision. (c) Observer estimate $\hat{b}_{s}$ against the injected bias. (d) Steering command; dotted: exact compensation $c_{s}=-b_{s}$ . (e) Lateral offset from the lane center; dotted: lane edge. (f) Speed. DIVA+PID and DIVA w/o observer use no estimate.

Fig. 7: DS (a) and lateral offset p95 (b) on Hard-52 at reduced perception rates with 0.1 s latency; at 20 Hz, the synchronous runs without latency, where hold and fast path coincide. Hold: command held at the perception rate; fast path: asynchronous DIVA; DIVA+PID on held path: its PID at 20 Hz on the last perception result transformed to the current pose.

Fig. 8: Case studies on six routes where all controllers score 100; TF++ and MPC+DOB run on the baseline planner. (a) T-junction stop and turn. (b) Hard-braking lead vehicle. (c) Control-loss disturbance. (d) Pedestrian stepping out. (e) Parking exit into traffic. (f) Vehicle cut-in. Bottom row: front camera of DIVA on each route.

Configuration	Mean (ms)	p95 (ms)	Rate (Hz)
DIVA+PID	17.28	17.97	58
DOB-PID	18.08	20.60	55
MPC+DOB, CPU	45.08	45.92	22
MPC+DOB, GPU	81.47	83.79	12
DIVA, synchronous	21.30	23.93	47
DIVA, async., 20 Hz perception	18.44	19.19	54

TABLE III: Latency of the Complete Control Cycle

Fig. 9: Identification experiments: measured vehicle against model roll-outs under the same commands. (a) Acceleration map; (b) launch from rest; (c) full-brake step; (d) steering slew-rate limiter; (e) yaw-rate steps up to a lateral acceleration

a_{y}\approx 6

m/s 2 ; (f) steady-state yaw-rate gain in all 86 step-steer tests (dots) and of the model at 5, 12 and 20 m/s (lines; shaded: benchmark range).

Symbol	Quantity	Value
Specification
$m$	Mass	1696 kg
$l_{f}$ , $l_{r}$	Center of mass to front, rear axle	1.172, 1.689 m
$W$	Track width	1.593 m
$x_{o}$	Center of mass ahead of reference point	0.30 m
$\delta_{\max}(v_{x})$	At 0, 20, 60, 120 km/h	70, 63, 56, 49 ∘

TABLE IV: Parameters of the Identified Vehicle Model

Item	Value
Token width, embedding MLP width	256, 128
Planning decoder: heads, FFN, GRU width	8, 2048, 64
Control decoder: heads, FFN, dropout	8, 1024, 0.1
$w_{y}$ , $w_{x}$ , $w_{\psi}$ , $w_{v}$ , $w_{R}$	1.0, 0.5, 2.0, 0.5, 0.5
$w_{a}$ , $w_{h}$ , $w_{p}$	0.5, 0.2, 1.0
Anchor $\Omega_{j}$ : decay per knot, factor at standstill	0.8, 4

TABLE V: Implementation Details and Hyperparameters

End-to-End Autonomous Driving

End-to-end autonomous driving learns to turn sensor observations and driving context into vehicle actions or trajectories.

See all 227 papers

DIVER: Reinforced Diffusion Breaks Imitation Bottlenecks in End-to-End Autonomous Driving
Jul 5, 2025Ziying Song, Lin Liu, Hongyu Pan +7End-to-End Autonomous Driving Diffusion-Based Planning
Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China · School of Artificial Intelligence (School of Software), Yanshan University, Qinhuangdao, China · Horizon Robotics +3
Learning Direct Control Policies with Flow Matching for Autonomous Driving
May 14, 2026Marcello Ceresini, Federico Pirazzoli, Andrea Bertogalli +5Flow Matching End-to-End Autonomous Driving
1Università degli Studi di Parma · 2Alma Mater Studiorum - Università di Bologna · 3VisLab (an Ambarella Inc. company)
Unleashing the Potential of Diffusion Models for End-to-End Autonomous Driving
Feb 26, 2026Yinan Zheng, Tianyi Tan, Bin Huang +11Diffusion-Based Planning End-to-End Autonomous Driving
Institute for AI Industry Research (AIR), Tsinghua University · Xiaomi EV

Jul 5, 2025J/K move · Enter open · S save

DIVER: Reinforced Diffusion Breaks Imitation Bottlenecks in End-to-End Autonomous Driving

Ziying Song, Lin Liu, Hongyu Pan, Bencheng Liao+6

Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China · School of Artificial Intelligence (School of Software), Yanshan University, Qinhuangdao, China · Horizon Robotics+3

End-to-End Autonomous Driving Diffusion-Based Planning Imitation Learning RL for Diffusion Models RL for Autonomous Driving Trajectory Prediction

Open paper AskPDF arXiv

Most end-to-end autonomous driving methods rely on imitation learning from single expert demonstrations, often leading to conservative and homogeneous behaviors that limit generalization in complex real-world scenarios. In this work, we propose DIVER, an end-to-end driving framework that integrates reinforcement learning with diffusion-based generation to produce diverse and feasible trajectories. At the core of DIVER lies a reinforced diffusion-based generation mechanism. First, the model conditions on map elements and surrounding agents to generate multiple reference trajectories from a single ground-truth trajectory, alleviating the limitations of imitation learning that arise from relying solely on single expert demonstrations. Second, reinforcement learning is employed to guide the diffusion process, where reward-based supervision enforces safety and diversity constraints on the generated trajectories, thereby enhancing their practicality and generalization capability. Furthermore, to address the limitations of L2-based open-loop metrics in capturing trajectory diversity, we propose a novel Diversity metric to evaluate the diversity of multi-mode predictions.Extensive experiments on the closed-loop NAVSIM and Bench2Drive benchmarks, as well as the open-loop nuScenes dataset, demonstrate that DIVER significantly improves trajectory diversity, effectively addressing the mode collapse problem inherent in imitation learning.

Abstract

End-to-End Autonomous Driving

Explore similar work

DIVER: Reinforced Diffusion Breaks Imitation Bottlenecks in End-to-End Autonomous Driving

Learning Direct Control Policies with Flow Matching for Autonomous Driving

Unleashing the Potential of Diffusion Models for End-to-End Autonomous Driving