End-to-end driving policies are commonly trained through open-loop behavior cloning, yet they must ultimately operate in closed-loop when deployed on a vehicle, creating a fundamental mismatch between training and execution. Beyond the commonly studied effects of covariate shift and causal confusion, we identify a complementary factor for this open-loop/closed-loop gap: waypoint-based supervision and displacement metrics do not ensure that the intermediate trajectory is physically coherent or easy for the controller to track. We observe that these inconsistencies concentrate primarily at intermediate waypoints, while the predicted endpoint remains comparatively reliable. Based on this observation, we introduce Endpoint-Constrained Optimization (ECO), a lightweight postprocessing layer that anchors the trajectory to the vehicle's executed history, preserves the policy's predicted endpoint, and reshapes the intermediate waypoints to improve feasibility. ECO requires no map, privileged simulator state, or additional training, and can be inserted between a broad range of waypoint-emitting policies and their controllers. Across two closed-loop simulators, it improves the aggregate closed-loop score of all six evaluated generative and regression-based driving policies, and the gains tend to increase with how often the base plans violate motion limits. On HUGSIM, ECO improves VaVAM from 18.1 to 31.0 HD-Score (+71%), achieving 1st place on the HUGSIM Closed-Loop Driving Challenge. Similarly, on AlpaSim, ECO increases the scene scores of VaVAM and DiffusionDrive by 123% and 22%, respectively. These results show that for a broad collection of end-to-end driving models, repairing the intermediate geometry of predicted trajectories without changing the policy's predicted endpoint can substantially improve closed-loop performance.
Figures & tables
Fig. 2 : Pipeline. The frozen end-to-end driving model receives camera inputs from the simulator and samples a trajectory for the next N steps. ECO anchors K=2 past poses and the last predicted waypoint, then solves a trajectory optimization problem to refine the waypoints between them (Sec. IV ). The controller then tracks the refined plan.
TABLE I : Components of Eq. ( 1 ). Indices run over the extended trajectory z (Sec. IV ).
open loop (m)
paired Δ after ECO
closed loop
Sent plan
ADE ↓
FDE ↓
Δ ADE / Δ FDE (m)
deficit [%] ↓
HD-Score ↑
VaVAM
2.53
3.27
–
44
23.1
VaVAM + ECO
1.66
2.42
−0.03 / +0.00
0
39.6
UniAD
1.86
2.25
–
16
27.8
UniAD + ECO
2.02
2.61
+0.03 / +0.00
0
37.3
TABLE II : Small open-loop changes can produce large closed-loop gains. Results on the 34-scene nuScenes subset of the HUGSIM test set. ADE/FDE compare the plan to the ground truth, whereas paired Δ measures the change from each raw plan to its ECO-processed version. Deficit is the fraction of waypoints violating a motion limit. ECO changes paired ADE/FDE by only centimeters while improving HD-Score by +16.5 for VaVAM and +9.5 for UniAD. The ADE/FDE columns are computed over the plans encountered in each closed-loop rollout and therefore reflect different visited-state distributions. Paired Δ ADE/ Δ FDE instead compares each raw plan with its ECO-processed version at the same state, isolating the direct effect of post-processing.
Fig. 3 : Collisions and limit violations concentrate in the middle of a predicted plan. Measured with VaVAM, on the nuScenes subset of the HUGSIM test set. (a) Fraction of predicted waypoints whose ego footprint overlaps an agent box recorded in the rollout at the frame each waypoint is meant for (i.e., waypoints in collision). (b) Waypoints violating a motion limit for a single sampled plan, for the average of eight sampled plans, and for one plan after ECO.
HUGSIM [ 11 ] Test Set (185 scenes), mean over 3 evaluation seeds
HD-Score ↑
Policy
Waymo
nuScenes
KITTI-360
PandaSet
Overall
Rc↑
NC ↑
TTC ↑
COM ↑
VaVAM [ 5 ]
25.8
23.1
4.0
23.1
18.1 ± 0.5
40.9
52
38
14
VaVAM + ECO
40.0
39.6
22.2
30.3
31.0 ± 0.6
45.2
54
46
97
Δ (ECO − base)
+14.2
+16.5
+18.2
+7.2
+12.9 ± 0.8
+4.3
+2
+8
+83
LTF [ 9 ]
34.5
17.8
3.7
22.6
17.9 ± 1.1
31.6
44
39
100
TABLE III : Cross-policy and cross-simulator evaluation of ECO. Scores are percentages unless a unit is shown. ± is the standard deviation over passes; Δ rows combine the base and ECO deviations in quadrature. Top: HD-Score on the 185-scene HUGSIM test set by domain and overall, with route completion, no-collision, TTC, and comfort. Bottom: Evaluation in AlpaSim, an independent simulator with no comfort score; lin./nonlin. MPC: AlpaSim’s linear/nonlinear model-predictive controller. ECO consistently improves the aggregate scores of all base policies in both simulators.
VaVAM [ 5 ]
UniAD [ 7 ]
DrivoR (w/ SimScale [ 42 ] ) [ 10 ]
Train
Test
Train
Test
Train
Test
base policy
20.0
18.1
32.7 ⋆
26.8
35.7 ⋆
34.6
+ ECO
32.1
31.0
34.9
32.4
38.8
36.2
TABLE IV : HD-Score on both HUGSIM splits. Train (345 scenes) and test (185 scenes); ⋆ taken from [ 10 ] .
Fig. 4 : Results by HUGSIM difficulty tier. (a) Every scene of the 185-scene test set as one point: HD-Score without and with ECO. Points above the diagonal improve. (b) ECO helps most on easier scenes where the base policy is already competent.
Fig. 5 : ECO benefit tends to increase with base-plan deficit. Paired Δ HD-Score with ECO against the fraction of base-plan waypoints that violate a motion limit, for VaVAM-B (357M parameters) at 2, 5, 10 (deployed), and 20 flow matching steps, VaVAM-S (204M parameters), VaVAM-L (1.4B parameters), and four other policies on the same nuScenes test split in HUGSIM.
Start boundary
Endpoint xN−1
HD-Score
Δ vs. ECO
x0 pinned
free
32.1
− 7.5
history anchored
free
34.4
− 5.2
x0 pinned
pinned
36.9
− 2.7
history anchored
pinned
39.6
–
base policy
23.1
− 16.5
TABLE V: Boundary conditions. VaVAM with the ECO objective on the nuScenes split of the HUGSIM test set. Plans are either anchored to the executed history h−K:0 (first predicted waypoint x0 free) or pinned at x0 , with the endpoint xN−1 either free or pinned.
Fig. 6 : Trajectory-shaping operators on a sampled plan. All operators are applied with the same boundary conditions as ECO: the executed history is anchored, the endpoint is pinned (square), and the first predicted waypoint is free. Rings denote motion-limit violations; HD-Scores are the averages on the nuScenes split (Sec. V-A ).
As scaling pre-training data alone yields diminishing returns, post-training is becoming increasingly important across physical AI domains such as autonomous driving. End-to-end driving policies are pre-trained in open loop with behavior cloning on human demonstrations. However, compounding errors during closed-loop deployment can take the vehicle outside the training data distribution, increasing the risk of safety-critical incidents. Closed-loop post-training can mitigate this risk but requires costly simulation for sensor-based policies. We propose OPTED (on-policy fine-tuning for end-to-end driving) which decouples reinforcement learning from the post-training of the end-to-end policy: a privileged teacher is trained using RL on vectorized inputs (HD-map and bounding boxes). This teacher then provides supervision to the pre-trained student during closed-loop post-training. We apply OPTED to two camera-based models, TransFuser and VaVAM, and fine-tune them in AlpaSim, using neural reconstructions (3DGS) of real driving logs. Driving scores increase by factors of 1.6× and 9.5×, respectively. In controlled experiments OPTED matches closed-loop performance with approximately three orders of magnitude fewer simulator interactions than direct RL post-training, while staying closer to the human prior. Project page: https://01dami23.github.io/opted/
Current trajectory prediction models are primarily trained in an open-loop manner, which often leads to covariate shift and compounding errors when deployed in real-world, closed-loop settings. In this work, we propose a closed-loop training paradigm for receding-horizon ego prediction and identify two design choices that are critical for it to succeed. First, closed-loop supervision must be on-policy: each predicted mode is rolled out and supervised only from the states its own execution produced, preserving the prediction accuracy that cross-mode supervision would otherwise degrade. Second, gradients must not flow across the executed state transition, as backpropagating through these transitions allows the model to retroactively adjust an earlier prediction so that the resulting state is easier to predict from, mathematically regretting a mistake rather than learning to recover from it. We show empirically that this shortcut couples the learned policy to the specific replanning rate used during training and degrades at other rates, whereas detaching the gradient forces the model to rectify drift instead, allowing it to generalize across replanning rates unseen during training. Evaluated across three benchmarks (nuScenes, DeepScenario, and Argoverse 2) and two architectures, a decoder-only adaptation of LMFormer and an unmodified implementation of QCNet, our approach matches open-loop baselines under single-shot evaluation and improves with higher replanning rates, reducing collisions by up to 35% and off-road rates by up to 67%.
Closed-loop evaluation of end-to-end driving requires continuous rollouts that reveal how earlier decisions affect subsequent driving. However, existing benchmarks evaluate only short segments and fail to capture later consequences. Ambiguous directional commands also obscure the intended navigation objective. We introduce Odyssey, a closed-loop benchmark for long-horizon driving comprising 100 scenarios, each reconstructed from a 100-second nuPlan driving log to preserve the context of navigation maneuvers and traffic interactions. To provide a consistent navigation objective, Odyssey replaces directional commands with explicit standard-definition (SD) map routes that specify which roads to follow, while sensor-based planning determines local driving actions. Throughout these rollouts, diffusion-based refinement of 3DGS-rendered images reduces rendering artifacts along the ego trajectory. To assess how effectively planners follow these routes and prepare for upcoming maneuvers, we introduce SD Route Compliance and Pre-Lane Change Score. These assessments are complemented by RouteDS, which extends the Driving Score with penalties for SD-route deviations and failed lane preparation. We adapt state-of-the-art planners, including vision-language-action (VLA) models, and evaluate their navigation performance using these metrics. Odyssey highlights open questions in route representation and integration for E2E driving. Benchmark code and adapted baselines will be released publicly.