Not All History Helps: Velocity-Aware Selective Memory for Long-Horizon End-to-End Autonomous Driving

Yuchen Liu; Ziying Song; Shengkai Zhang; Jiannan Chen; Peiliang Wu; Lei Yang; Bin Sun; Yan Gong

cs.ROAug 16, 2026

Not All History Helps: Velocity-Aware Selective Memory for Long-Horizon End-to-End Autonomous Driving

Authors: Yuchen Liu, Ziying Song, Shengkai Zhang, Jiannan Chen, Peiliang Wu, Lei Yang, Bin Sun, Yan Gong,

Organizations: Nanyang Technological University, Singapore · North University of China, Taiyuan 030051, China · Yanshan University, Qinhuangdao 066004, China · Beijing Jiaotong University, Beijing 100044, China ·

Abstract

Reliable long-horizon planning remains a key challenge in end-to-end autonomous driving. By accounting for future motion evolution and potential consequences, it provides forward-looking guidance for safe and consistent driving in evolving traffic environments. Existing methods use historical planning states as temporal context. Self-generated history may become stale or conflict with the current motion stage, introducing unreliable priors. We propose StableDrive to address cross-cycle historical reliability and within-horizon motion-stage evolution. Selective Momentum Memory (SMM), implemented with a Mamba selective state-space operator, controls the influence of the preceding self-predicted planning state on the current cycle. Motion-Stage Training Scaffold (MSTS) uses motion-stage, long-horizon trajectory, and longitudinal-motion supervision to guide stage-aware future motion learning and is removed before inference. A fixed parameter midpoint between two architecture-aligned endpoints yields a single deployable SMM planner without model ensembling or extra inference-time computation. On nuScenes under the MomAD evaluation protocol, StableDrive achieves SOTA performance across all reported planning metrics from 1 to 6 s, reducing average collision rate by 23.3%, TPC by 30.9%, and L2 by 11.8% over the best previously reported value for each metric. On the curated Longitudinal-Transition nuScenes (LT-nuScenes), StableDrive reduces 6-s collision rate by 23.81%, TPC by 10.90%, and L2 by 6.37%. On NAVSIM v1 and v2, StableDrive achieves the highest PDMS/EPDMS in all three reported settings, including a 5.7-point EPDMS gain on v2 navhard over the previous best.

Figures & tables

Fig. 1: Selective planning memory enables safer long-horizon planning. (a) A nuScenes example comparing MomAD [ 7 ] and StableDrive with the ground-truth future trajectory. (b) Retaining a stale planning prior causes MomAD [ 7 ] to diverge after $3\text{\,}\mathrm{s}$ and collide at $6\text{\,}\mathrm{s}$ . (c) StableDrive suppresses unreliable history and remains collision-free. (d) StableDrive improves six-second L2, Col. Rate, and TPC by 5.83%, 13.11%, and 11.34%, respectively, over our local MomAD [ 7 ] reproduction under the same evaluation protocol. Lower is better for all metrics.

Fig. 2: Overall framework of StableDrive. StableDrive combines selective cross-cycle planning memory with a train-and-retire motion-stage scaffold. Multi-view images are encoded into a sparse scene representation, and joint scene interaction generates command-conditioned planning queries. SMM selectively updates the current queries using score-modulated one-cycle history before candidate generation and selection. MSTS provides horizon-wise motion-stage supervision during training and is removed at inference. A fixed parameter midpoint between the SMM-only and MSTS-trained endpoints yields a single deployable checkpoint. All visualizations correspond to the same nuScenes validation sample.

Fig. 3: Selective Momentum Memory (SMM). SMM gates the one-cycle cached query with its previous score, integrates the retained history with the current query through a causal Mamba [ 12 ] update, and injects the result as a residual. The refined query drives trajectory prediction, after which it and the updated score are detached and stored for the next cycle.

Fig. 4: Motion-Stage Training Scaffold (MSTS). The predicted motion-stage distribution conditions candidate-specific horizon tokens. A shared Horizon Mamba [ 12 ] processes each candidate sequence without cross-candidate state exchange, after which gated residual refinement produces the refined trajectory increments. MSTS is training-only, and ground truth is used only for supervision.

	$\displaystyle\mathbf{y}^{0}_{t,c,k,h}$	$\displaystyle=\sum_{\tau=1}^{h}\Delta\mathbf{y}^{0}_{t,c,k,\tau}=(x^{0}_{t,c,k,h},y^{0}_{t,c,k,h}),$
	$\displaystyle v^{0}_{t,c,k,h}$	$\displaystyle=\Delta y^{0}_{t,c,k,h}/\Delta t,$
	$\displaystyle a^{0}_{t,c,k,h}$	$\displaystyle=(v^{0}_{t,c,k,h}-v^{0}_{t,c,k,h-1})/\Delta t,\hskip 8.50012pta^{0}_{t,c,k,1}=0,$
	$\displaystyle\bm{\gamma}_{t,c,k,h}$	$\displaystyle=\operatorname{Concat}[\mathbf{y}^{0}_{t,c,k,h},\Delta\mathbf{y}^{0}_{t,c,k,h},$
		$\displaystyle y^{0}_{t,c,k,h},x^{0}_{t,c,k,h},v^{0}_{t,c,k,h},a^{0}_{t,c,k,h}]\in\mathbb{R}^{8}.$

Table 5

Method	Backb.	L2 (m) $\downarrow$							Col. Rate (%) $\downarrow$							TPC $\downarrow$	FPS $\uparrow$
		1s	2s	3s	4s	5s	6s	Avg.	1s	2s	3s	4s	5s	6s	Avg.	Avg.
UniAD [ 1 ]	R101	0.47	0.91	1.35	1.91	2.47	3.07	1.70	0.25	0.36	0.61	0.99	1.64	2.51	1.06	1.58	1.8 A100
VAD [ 2 ]	R50	0.41	0.70	1.05	1.82	2.23	3.01	1.54	0.03	0.19	0.43	0.89	1.71	2.41	0.94	1.23	4.5 3090
SparseDrive [ 3 ]	R50	0.43	0.87	1.23	1.75	2.32	2.95	1.59	0.19	0.31	0.56	0.87	1.54	2.33	0.97	1.46	9.0 4090
DiffusionDrive [ 23 ]	R50	0.35	0.81	0.99	1.65	1.94	2.40	1.36	0.20	0.32	0.45	0.82	1.51	2.23	0.92	1.31	8.2 4090
DIVER [ 26 ]	R50	0.38	0.75	1.10	1.53	1.98	2.49	1.37	0.13	0.31	0.44	0.80	1.41	2.11	0.87	1.35	–

TABLE I: Planning performance on the nuScenes validation set over the complete 1–6 s planning horizon. All methods use camera-only input. Avg. denotes the mean over the six horizons; for TPC, only the six-horizon average is reported due to space constraints. R50/R101 denote ResNet-50/101, and DiT denotes Diffusion Transformer. FPS is provided only as a hardware-specific efficiency reference; GPU subscripts indicate the device used for measurement, and FPS values are not compared across devices. “–” denotes unavailable results. Lower is better for L2, Col. Rate, and TPC, and the best planning results are shown in bold.

Method	L2 (m) $\downarrow$							Col. Rate (%) $\downarrow$							TPC (m) $\downarrow$
	1 s	2 s	3 s	4 s	5 s	6 s	Avg.	1 s	2 s	3 s	4 s	5 s	6 s	Avg.	Avg.
MomAD †	0.30	0.62	1.04	1.53	2.07	2.65	1.37	0.53	1.59	2.56	3.64	4.60	5.56	3.08	0.85
StableDrive (Ours)	$\mathbf{0.30}_{{\color[rgb]{1,0,0}\scriptscriptstyle 0.00}}$	$\mathbf{0.62}_{{\color[rgb]{1,0,0}\scriptscriptstyle 0.00}}$	$\mathbf{1.03}_{{\color[rgb]{1,0,0}\scriptscriptstyle-0.01}}$	$\mathbf{1.50}_{{\color[rgb]{1,0,0}\scriptscriptstyle-0.03}}$	$\mathbf{1.98}_{{\color[rgb]{1,0,0}\scriptscriptstyle-0.09}}$	$\mathbf{2.48}_{{\color[rgb]{1,0,0}\scriptscriptstyle-0.17}}$	$\mathbf{1.32}_{{\color[rgb]{1,0,0}\scriptscriptstyle-0.05}}$	$\mathbf{0.00}_{{\color[rgb]{1,0,0}\scriptscriptstyle-0.53}}$	$\mathbf{0.26}_{{\color[rgb]{1,0,0}\scriptscriptstyle-1.33}}$	$\mathbf{0.44}_{{\color[rgb]{1,0,0}\scriptscriptstyle-2.12}}$	$\mathbf{1.26}_{{\color[rgb]{1,0,0}\scriptscriptstyle-2.38}}$	$\mathbf{2.75}_{{\color[rgb]{1,0,0}\scriptscriptstyle-1.85}}$	$\mathbf{4.23}_{{\color[rgb]{1,0,0}\scriptscriptstyle-1.33}}$	$\mathbf{1.49}_{{\color[rgb]{1,0,0}\scriptscriptstyle-1.59}}$	$\mathbf{0.79}_{{\color[rgb]{1,0,0}\scriptscriptstyle-0.06}}$

TABLE II: Planning on LT-nuScenes over 1–6 s. MomAD † [ 7 ] is our local matched six-second reproduction. Avg. is the six-horizon mean; TPC is shown only as Avg. Red subscripts are StableDrive minus MomAD † ; negative values indicate improvement, and Col. Rate differences are percentage points. Lower is better; bold marks the best displayed values, including rounded ties.

Method	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	Conf. $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
VADv2 [ 57 ]	97.2	89.1	91.6	100	76.0	80.9
TransFuser [ 58 ]	97.7	92.8	92.8	100	79.2	84.0
${\operatorname{TransFuser}_{\operatorname{mmt}}}^{*}$ [ 58 ]	96.2	95.4	90.7	100	80.7	85.1
UniAD [ 1 ]	97.8	91.9	92.9	100	78.8	83.4
PARA-Drive [ 59 ]	97.9	92.4	93.0	99.8	79.3	84.0
DRAMA [ 60 ]	98.0	93.1	94.8	100	80.1	85.5

TABLE III: Comparison on planning-oriented NAVSIMv1 navtest split with

\operatorname{\textbf{Closed-Loop}}

metrics. ‘mmt’ refers multi-modal trajectory variant of

\operatorname{TransFuser}

and ∗ denotes the re-implementation.

Method	NC $\uparrow$	DAC $\uparrow$	DDC $\uparrow$	TL $\uparrow$	EP $\uparrow$	TTC $\uparrow$	LK $\uparrow$	HC $\uparrow$	EC $\uparrow$	EPDMS $\uparrow$
TransFuser [ 58 ]	96.9	89.9	97.8	99.7	87.1	95.4	92.7	98.3	87.2	76.7
DiffusionDrive [ 23 ]	98.2	95.9	99.4	99.8	87.5	97.3	96.8	98.3	87.7	84.5
Hydra-MDP++ [ 67 ]	97.2	97.5	99.4	99.6	83.1	96.5	94.4	98.2	70.9	81.4
DriveSuprim [ 64 ]	97.5	96.5	99.4	99.6	88.4	96.6	95.5	98.3	77.0	83.1
DiffusionDriveV2 [ 68 ]	97.7	96.6	99.2	99.8	88.9	97.2	96.0	97.8	91.0	87.5
DriveWorld-VLA [ 69 ]	98.6	99.1	99.6	99.8	87.4	97.9	97.0	97.8	78.6	86.8

TABLE IV: Comparison with SOTA methods on the NAVSIMv2 navtest split [ 66 ] . EPDMS ∗ denotes results computed with the original NAVSIMv2 evaluation code before the human-behavior filtering fix, while EPDMS denotes the corrected official implementation.

Method	Stage	NC $\uparrow$	DAC $\uparrow$	DDC $\uparrow$	TL $\uparrow$	EP $\uparrow$	TTC $\uparrow$	LK $\uparrow$	HC $\uparrow$	EC $\uparrow$	EPDMS $\uparrow$
TransFuser [ 58 ]	Stage 1	96.2	79.5	99.1	99.5	84.1	95.1	94.2	97.5	79.1	23.1
TransFuser [ 58 ]	Stage 2	77.7	70.2	84.2	98.0	85.1	75.6	45.4	95.7	75.9	23.1
DiffusionDrive [ 23 ]	Stage 1	96.0	79.7	97.4	99.5	81.3	93.1	90.8	96.8	73.8	24.2
DiffusionDrive [ 23 ]	Stage 2	82.1	72.2	88.5	98.7	85.1	78.8	49.2	89.3	71.2	24.2
GuideFlow [ 27 ]	Stage 1	96.6	80.5	96.3	99.3	82.3	94.9	91.5	97.7	67.8	27.1
GuideFlow [ 27 ]	Stage 2	87.3	76.7	88.8	99.2	84.3	85.1	49.7	93.1	44.5	27.1

TABLE V: Comparison with SOTA methods on the NAVSIMv2 navhard split [ 66 ] .

Method	1 s	2 s	3 s	4 s	5 s	6 s
L2 (m) $\downarrow$
MomAD †	0.29	0.56	0.93	1.38	1.91	2.50
SMM Planner	0.30	0.59	0.97	1.44	1.97	2.58
MSTS-Trained Planner	0.29	0.57	0.93	1.36	1.86	2.42
StableDrive (Ours)	0.28	0.55	0.89	1.32	1.80	2.35
Col. Rate (%) $\downarrow$

Table 11

Fig. 5: Training Dynamics under Matched Optimization Budgets. Both branches start from the same SMM checkpoint and receive 11,720 optimizer updates. (a) The shared base objective

\mathcal{L}_{\mathrm{base}}

remains comparable and bounded for SMM-only and MSTS-assisted training. (b) The unweighted MSTS auxiliary objectives remain bounded and settle to lower late-training levels, with

\mathcal{L}_{\mathrm{far}}

exhibiting larger intermediate fluctuations. Faint traces show raw rank-averaged audit values, while the emphasized curves show step-aware EMA trends. MSTS is retired before fixed-midpoint StableDrive construction.

Fig. 6: Cross-Cycle Drift under a Shared Spatiotemporal Reference. Plans issued at $t-1$ , $t$ , and $t+1$ are transformed into each displayed current ego frame and restricted to their shared future timestamps. MomAD [ 7 ] (top) shows increasing terminal dispersion across consecutive planning cycles, whereas StableDrive (bottom) keeps the aligned plans concentrated near the ground truth. Gray shading indicates terminal dispersion; GT denotes ground truth.

Fig. 7: Selective Reuse of One-Cycle History across Longitudinal Motion-Stage Transitions. Columns show the front-view scene, the current plan produced by MomAD [ 7 ] , pose- and time-aligned $t-1$ momentum modes colored by their exact SMM weights $\sigma(s)$ , and the StableDrive memory-fused plan. Rows show (a) restart, (b) sustained acceleration, (c) dense-traffic deceleration, and (d) stopping on a rainy curved road. Memory colors use a fixed scale, and annotations identify the maximum- and minimum-weight modes. The historical modes originate from the immediately preceding planning cycle, 0.5 s earlier, and all BEV panels within each row use the same metric crop. The non-uniform weights and the change in the dominant mode across the illustrated motion stages indicate that the contribution of one-cycle history is conditioned on the current planning state rather than applied uniformly across motion stages.

Variant	Construction role	$\alpha$	MSTS at inference	Extra params	L2 Avg. (m) $\downarrow$	Col. Avg. (%) $\downarrow$	TPC Avg. (m) $\downarrow$
T0	SMM endpoint	0.0	Off	0	2.00	1.24	1.23
T1-on	Retention ablation	–	On	+0.81M	1.89	1.35	1.23
T1-off	MSTS endpoint	1.0	Off	0	1.88	1.35	1.22
StableDrive	Fixed midpoint	0.5	Off	0	1.83	1.19	1.20

TABLE VIII: Train–retire, endpoint-to-midpoint, and matched inference-cost analysis. Panel A reports 4–6 s averages under equal training budgets. T0 is the SMM-only endpoint; T1-on and T1-off use the same MSTS-trained checkpoint and differ only in whether MSTS is retained at inference. StableDrive is the fixed arithmetic midpoint (

\alpha=0.5

) of the architecture-aligned T0 and T1-off endpoints. Panel B reports active parameters, FLOPs, latency, FPS, and peak allocated CUDA memory under steady-state, single-sample FP16 inference on one RTX 4090. T0, T1-off, and StableDrive share the same deployed SMM graph; their small throughput differences are therefore not ranked.

Variant	L2 (m) $\downarrow$							Col. Rate (%) $\downarrow$							TPC (m) $\downarrow$
	1 s	2 s	3 s	4 s	5 s	6 s	Avg.	1 s	2 s	3 s	4 s	5 s	6 s	Avg.	Avg.
StableDrive (1-frame )	0.28	0.55	0.89	1.32	1.80	2.35	1.20	0.01	0.09	0.30	0.65	1.16	1.78	0.66	0.85
StableDrive (2-frame )	0.29	0.57	0.94	1.39	1.91	2.51	1.27	0.12	0.24	0.43	0.81	1.36	2.03	0.83	0.93
StableDrive (4-frame )	0.29	0.56	0.92	1.36	1.87	2.45	1.24	0.11	0.25	0.48	0.88	1.42	2.10	0.87	0.89

TABLE IX: History-length comparison of StableDrive using one, two, or four preceding planning cycles over the complete 1–6 s horizon. L2 and Col. Rate are reported at each horizon and as six-horizon averages, while TPC is reported only as the six-horizon average. Lower is better for all metrics, and the best values are shown in bold.

Autonomous Driving Planning

Autonomous driving planning selects safe, feasible vehicle actions and trajectories amid traffic, uncertainty, road rules, and changing goals.

See all 384 papers

MomADv2: Reliable Temporal Memory for End-to-End Autonomous Driving
Aug 24, 2026Ziying Song, Shengkai Zhang, Lin Liu +8End-to-End Autonomous Driving Long-Horizon Planning
School of Artificial Intelligence (School of Software), Yanshan University · Beijing Jiaotong University · Nanyang Technological University +5
MomWorld: Momentum-Aware Latent World Model for Long-Horizon Autonomous Driving
Sep 27, 2026Ziying Song, Shengkai Zhang, Lei Yang +6Flow Matching World Model-Based Planning
Nanyang Technological University · Beijing Jiaotong University · North University of China +2
GraphWorld: Long-Horizon Planning with World Models for End-to-End Autonomous Driving
Jun 15, 2026Ziying Song, Caiyan Jia, Lin Liu +8World Model Learning End-to-End Autonomous Driving
Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China

Aug 24, 2026J/K move · Enter open · S save

MomADv2: Reliable Temporal Memory for End-to-End Autonomous Driving

Ziying Song, Shengkai Zhang, Lin Liu, Peiliang Wu+7

School of Artificial Intelligence (School of Software), Yanshan University · Beijing Jiaotong University · Nanyang Technological University+5

End-to-End Autonomous Driving Long-Horizon Planning Autonomous Driving Planning State Space Models

Open paper AskPDF arXiv

Long-horizon planning is critical for safe autonomous driving in complex scenarios. Existing methods improve planning continuity with temporal memory, but such memory may become invalid and mislead decisions when the driving command changes. Thus, selectively leveraging useful history while suppressing command-inconsistent memory remains a key challenge. To address this issue, we propose MomADv2, a reliable state-space memory framework for long-horizon end-to-end autonomous driving. At its core, MomADv2 introduces a Selective State-Space Planning Memory Query Module, which filters historical planning queries based on temporal continuity and command consistency, selects planning modes relevant to the current command, and models the evolution of planning intentions through a selective state-space mechanism. To further alleviate local trajectory deviations and error accumulation in long-horizon planning, we design a Flow-Matching Trajectory Residual Refiner. It learns a continuous residual correction field from the refined planning output to the expert trajectory, enabling fine-grained trajectory refinement while preserving the stability of anchor-based planning. Extensive experiments on closed-loop NAVSIM and Bench2Drive, as well as open-loop nuScenes, demonstrate that MomADv2 improves long-horizon planning consistency and reduces the average collision rate by 15.6% over MomAD under 6-second planning.

Abstract

Autonomous Driving Planning

Explore similar work

MomADv2: Reliable Temporal Memory for End-to-End Autonomous Driving

MomWorld: Momentum-Aware Latent World Model for Long-Horizon Autonomous Driving

GraphWorld: Long-Horizon Planning with World Models for End-to-End Autonomous Driving