Trajectory prediction models often fail in real-world automated driving due to distributional shifts between training and test conditions. Such distributional shifts, whether behavioural or environmental, pose a critical risk by causing the model to make incorrect forecasts in unfamiliar situations. We propose a self-supervised method that trains a decoder in a post-hoc fashion on the self-supervised task of forecasting the second half of observed trajectories from the first half. The L2 norm of the gradient of this forecasting loss with respect to the decoder's final layer defines a score to identify distribution shifts. Our approach, first, does not affect the trajectory prediction model, ensuring no interference with original prediction performance and second, demonstrates substantial improvements on distribution shift detection for trajectory prediction on the Shifts and Argoverse datasets. Moreover, we show that this method can also be used to early detect collisions of a deep Q-Network motion planner in the Highway simulator. Source code is available at https://github.com/Michedev/forecasting-the-past.
Figures & tables
Figure 1: Instead of (a) reconstructing the entire past trajectory Xpast and using the reconstruction loss Lreconstruct as a distribution shift score, we (b) forecast a portion of the observed trajectory Xpast and use the gradient of the forecasting loss, ∇Lforecast , as the distribution shift score S . Specifically, we divide Xpast into Xpast,1 and Xpast,2 and forecast X^past,2 given Xpast,1 . The gradient ∇Lforecast effectively identifies unknown scenarios in trajectory prediction.
Figure 2: Overview of our post-hoc gradient-based distribution shift detection method. The upper part shows the pre-trained encoder-decoder for trajectory prediction. Given its encoder eγ , we propose our forecast-the-past decoder dpast for distribution shift detection. The historical trajectory Xpast is split into two segments, Xpast,1 and Xpast,2 , which are resized via linear interpolation. The frozen encoder processes Xpast,1 to produce the latent representation zpast,1 , while our forecast-the-past decoder dpast predicts Xpast,2 . The gradient of the forecasting loss Lpast with respect to the input of the decoder’s last layer hlast serves as the distribution shift score S=∇Lpast .
Figure 3: Qualitative example of our gradient-based distribution shift detection method. The figure shows trajectory samples from in-distribution and distribution shifted scenarios. For each sample, we display the historical trajectory split into two segments ( Xpast,1 with the blue line and Xpast,2 with the green line), the forecasted second segment using our self-supervised decoder X^past,2 with a dashed line, and the corresponding gradient-based anomaly score S . Higher gradient magnitudes indicate distribution shift samples that deviate from learned motion dynamics.
Figure 4: Highway simulator scenarios depicting a merge crash, a roundabout crash, and normal roundabout navigation. We mark the start and end points of trajectories, highlight the colliding vehicle in a different color, and mark the crash point. A horizontal red line in the OOD score plots indicates the crash timestep, and a thin line denotes the anomaly threshold. Collisions in Figures 4(a) and 4(b) produce high gradient values, while normal behavior in Figure 4(c) shows gradients below the threshold. In Figure 4(a) , the method predicts stopping (black line), and the score increases immediately after step 20, successfully detecting the collision. In Figure 4(b) , the gradient score reaches a peak at timestep 26, preceding a collision after 4 steps. Figure 4(c) shows normal driving behavior in a roundabout, with a stable gradient score over time.
Figure 5: Distribution of rotation determinant and max velocity in Argoverse V1. Further details about their calculation are in Paragraphs 4.2 and 4.2 . We observe that trajectories are skewed towards turning right in Argoverse [ 6 ] .
Model
AUROC (%)
RIP-BC (K=1) [ 33 , 10 ]
52.8%
RIP-BC (K=5) [ 33 , 10 ]
52.1%
RIP-DIM (K=1) [ 33 , 29 ]
51.8%
RIP-DIM (K=5) [ 33 , 29 ]
50.9%
lGMM [ 38 ]
56.8%
Our Method
71.0 %
Table 1: Performance of Distribution shifts detection on Shifts [ 25 ]
KDE
OC-SVM
IF
Our Method
Turn right
56.1%
56.4%
61.0%
71.3 %
Turn left
55.2%
52.8%
53.2%
70.3 %
Max Velocity
70.8%
61.6%
65.4%
79.2 %
Table 2: OOD detection performance in terms of AUROC on the Argoverse Dataset.
Figure 6: Gradient Distribution for In-Distribution vs. Out-Of-Distribution samples for Shifts [ 25 ] and Argoverse [ 6 ] . We observe different distribution for ID (Blue area) and OOD (Red Area) in Shifts Dataset [ 25 ] .
Figure 7: Kernel density estimation (KDE) [ 8 ] of the last-layer gradients in the Highway environment [ 21 ] for the intersection driving task. We observed very distinct gradients between ID on OOD samples, leading to almost perfect collision detection.
Model
last
lat.
all
loss
AE
50.56%
56.26%
57.29%
50.00%
MAE
50.51%
56.08%
57.46%
48.95%
Our method
71.30%
60.20%
58.70%
51.00%
Table 3: Ablation study of model architectures and scoring strategies for distribution shift detection on the Shifts dataset [ 25 ] using the AUROC metric. We compare standard and masked autoencoders against our method using different scores: gradients w.r.t. the last layer ( last ), the latent space ( latent ), all model parameters ( all ), as well as the self-supervised loss function of the models ( loss ).
Method
Merge
Intersection
Roundabout
AE
99.9%
64.3%
85.1%
Masked AE
99.9%
65.5%
85.8%
IF
99.1%
67.9%
91.1%
Our method
99.9%
96.1%
95.8%
Table 4: Distribution Shifts detection performance (AUROC) across different scenarios in the Highway [ 21 ] Environment.
Figure 8: t-SNE visualization of the decoder’s last layer inputs for the Highway [ 21 ] Roundabout scenario. Safe (ID) and collision (OOD) trajectories form well-separated clusters, demonstrating the discriminative power of the learned representations.
Figure 9: Gradient vs loss-based OOD detection on the Shifts dataset [ 25 ] . While the loss function Lpast reaches 51.0 % AUROC, the gradient of the loss function ∇Lpast reaches 70.9 % AUROC showcasing the effectiveness of gradient-based OOD detection (Sec. 3.2 ).
Trajectory prediction is central to safety in autonomous driving, yet learning-based predictors tend to degrade sharply when encountering scenarios poorly represented by their training data. Many methods attempt to mitigate distribution shift degradation through data-centric or test-time adaptation approaches; however, they are typically validated along fragmented axes of generalization, leaving the field without a standardized way to compare robustness across shifts a model may encounter. To address this, we introduce ControlledShifts, a framework and benchmark suite that systematically re-splits existing trajectory datasets into in-distribution (seen) and out-of-distribution (unseen) partitions, via a shared characterization-and-splitting formulation, in which a characterization function fixes the axis of variation a benchmark probes and a splitting function fixes how the tail of that axis is withheld. The suite comprises three benchmarks targeting key topological and behavioral distribution shifts. Furthermore, to aggregate multi-dimensional performance metrics across these benchmarks, we propose a unified robustness score that evaluates models along two complementary dimensions: prediction quality (relative performance gain) and prediction stability (performance preservation under shift). We showcase ControlledShifts by benchmarking prominent transformer-based architectures, exposing critical differences in how models of varying capacities handle latent relevance and environmental structure.
Ingrid Navarro, Pablo Ortega-Kral, Yutong Duan +2
Robotics Institute, School of Computer Science, Carnegie Mellon University · Work done as part of an internship at Lavoro AI. · Stack AV +1
Accurate trajectory forecasting of surrounding traffic participants is a core capability for autonomous driving, enabling vehicles to anticipate behavior and plan safe maneuvers. We observe that current state-of-the-art forecasting models on Argoverse 2 and the Waymo Open Motion Dataset tailor their training objectives to the different benchmark metrics. Because these metrics encourage conflicting behavior, we propose a paradigm change for trajectory forecasting: training models with metric-agnostic probabilistic objectives and treating metric optimization as a downstream task applied to the predictive distribution. Concretely, we introduce Trajectory Distribution Evaluation (TraDiE) policies, metric-specific policies that map a predictive distribution to the set of K trajectories and confidences required by trajectory forecasting metrics. We evaluate this framework by introducing DONUT-NLL, which adapts the training objective of the state-of-the-art trajectory forecasting model DONUT to directly optimize the predictive distribution. Using our policies, DONUT-NLL achieves state-of-the-art results on all metrics of the Waymo motion prediction benchmark.
Markus Knoche, Daan de Geus, Bastian Leibe
RWTH Aachen University, Germany · Eindhoven University of Technology, Netherlands
Existing trajectory prediction methods exhibit significant performance degradation under distribution shifts during test time. Although test-time training techniques have been explored to enable adaptation, current approaches rely on an offline pre-trained predictor that lacks online learning flexibility. Moreover, they depend on fixed online model updating rules that do not accommodate the specific characteristics of test data. To address these limitations, we first propose a meta-learning framework to directly optimize the predictor for fast and accurate online adaptation, which performs bi-level optimization on the performance of simulated test-time adaptation tasks during pre-training. Furthermore, at test time, we introduce a data-adaptive model updating mechanism that dynamically adjusts the predefined learning rates and updating frequencies based on online partial derivatives and hard sample selection. This mechanism enables the online learning rate to suit the test data, and focuses on informative hard samples to enhance efficiency. Experiments are conducted on various challenging cross-dataset distribution shift scenarios, including nuScenes, Lyft, and Waymo. Results demonstrate that our method achieves superior adaptation accuracy, surpassing state-of-the-art test-time training methods for trajectory prediction. Additionally, our method excels under suboptimal learning rates and high FPS demands, showcasing its robustness and practicality.
Yuning Wang, Pu Zhang, Yuan He +2
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, IAIR, Xi’an Jiaotong University, China · KargoBot, China