ControlledShifts: Towards Standardizing Robustness Evaluation in Trajectory Prediction Under Distribution Shifts
Authors: Ingrid Navarro, Pablo Ortega-Kral, Yutong Duan, Jonathan Francis, Jean Oh
Organizations: Robotics Institute, School of Computer Science, Carnegie Mellon University · Work done as part of an internship at Lavoro AI. · Stack AV · Bosch Center for Artificial Intelligence
Trajectory prediction is central to safety in autonomous driving, yet learning-based predictors tend to degrade sharply when encountering scenarios poorly represented by their training data. Many methods attempt to mitigate distribution shift degradation through data-centric or test-time adaptation approaches; however, they are typically validated along fragmented axes of generalization, leaving the field without a standardized way to compare robustness across shifts a model may encounter. To address this, we introduce ControlledShifts, a framework and benchmark suite that systematically re-splits existing trajectory datasets into in-distribution (seen) and out-of-distribution (unseen) partitions, via a shared characterization-and-splitting formulation, in which a characterization function fixes the axis of variation a benchmark probes and a splitting function fixes how the tail of that axis is withheld. The suite comprises three benchmarks targeting key topological and behavioral distribution shifts. Furthermore, to aggregate multi-dimensional performance metrics across these benchmarks, we propose a unified robustness score that evaluates models along two complementary dimensions: prediction quality (relative performance gain) and prediction stability (performance preservation under shift). We showcase ControlledShifts by benchmarking prominent transformer-based architectures, exposing critical differences in how models of varying capacities handle latent relevance and environmental structure.
Figures & tables
Fig. 1: Representative seen (left, green panes) and unseen (right, red panes) scenarios for each benchmark: BackgroundAgents (top), EgoSafeShift (center), and Environments (bottom). seen scenarios are used for training and validation; unseen scenarios are held out for testing.
Benchmark
Model
In Distribution ( seen , ↓ )
Distribution Shift ( unseen , ↓ )
Robustness Scores ( ↑ )
BrierFDE
MinFDE 6
MinADE 6
MissRate
CollisionRate
BrierFDE
MinFDE 6
MinADE 6
MissRate
CollisionRate
Quality
Stability
Uniform
Naive
11.228
10.561
3.649
0.640
0.048
11.357 ( +1.15% )
10.692 ( +1.24% )
3.689 ( +1.09% )
0.632 ( -1.28% )
0.046 ( -4.18% )
1.000
1.000
AutoBot
3.213
2.519
2.480
0.368
0.017
3.195 ( -0.58% )
2.500 ( -0.74% )
2.453 ( -1.05% )
0.375 ( +1.85% )
0.016 ( -7.27% )
2.758
1.000
SceneTransformer
4.748
4.276
1.547
0.491
0.067
4.793 ( +0.94% )
4.330 ( +1.26% )
1.566 ( +1.24% )
0.490 ( -0.31% )
0.069 ( +3.96% )
1.837
1.000
Wayformer
2.792
2.306
0.880
0.385
0.060
2.806 ( +0.51% )
2.327 ( +0.91% )
0.887 ( +0.77% )
0.404 ( +4.99% )
0.061 ( +3.21% )
3.034
1.000
MTR
3.839
3.316
1.210
0.424
0.054
3.841 ( +0.04% )
3.323 ( +0.20% )
1.209 ( -0.09% )
0.428 ( +1.00% )
0.056 ( +3.31% )
2.306
1.000
TABLE I: Model performance under each controlled distribution shift, one block per benchmark. We show the seen (validation) split followed by the unseen (test) split, with the parenthesized gap relative to seen in percent (red = degradation, green = improvement). Uniform is the no-shift setting. The rightmost columns show per-benchmark robustness scores; a score of 1 represents the reference. By design: Naive scores 1.000 on quality, being the quality reference, and all models score 1.000 on stability under Uniform , being referenced against their own score there. Bold marks the best model per group column. Displacement error metrics are in meters, and CollisionRate uses a 0.25 m threshold. Shaded rows are per-benchmark means.
Fig. 2: Robustness Scores, showing quality (left) and stability (right) rankings across models from Table I . Higher is better.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. 3: Distribution of non-background (top) and background (bottom) agents per scenario across splits, comparing uniform splitting (left) with our background, density-based splitting (right) for the BackgroundAgents benchmark. While the non-background distribution is preserved across both strategies, our splitting concentrates high background agent density in the test set.
Fig. 4: Distribution of safety-relevant scores per scenario across splits, comparing uniform splitting (left) with our EgoSafeShift re-splitting (right). Under Uniform splitting, all three splits have similar score distributions. Under our splitting the train/val distributions concentrate on low-scored scenarios, whereas the test distribution concentrates on higher-scored scenarios where latent criticality is more relevant.
Fig. 5: Distribution of NetLSD descriptors for the Environments benchmark using a 2D t-SNE projection. The panes show the projection colored by cluster assignment (left) and by data split (right). Under our splitting, the clusters with the highest silhouette score are assigned to the test set, as this score is used as a proxy for separability of map topologies.
Fig. 6: Examples of road topologies illustrating the shift induced by Environments . Side-by-side, we show the road layout and NetLSD -encoded graph, both colored according to the legend. seen environments (left) show dense, highly connected layouts with multi-lane intersections and wide avenues. unseen environments (right) show sparse, weakly connected layouts of long parallel corridors requiring primarily straight-line travel. This graph density difference is what the NetLSD heat trace captures and what our clustering separates on.
Fig. 7: Pairwise Jaccard overlap of scenarios across benchmarks, shown separately for training, validation, and test splits. High training overlap reflects shared scenario pools, while consistently low validation and test overlap demonstrates that the benchmarks evaluate disentangled conditions.
Trajectory prediction models often fail in real-world automated driving due to distributional shifts between training and test conditions. Such distributional shifts, whether behavioural or environmental, pose a critical risk by causing the model to make incorrect forecasts in unfamiliar situations. We propose a self-supervised method that trains a decoder in a post-hoc fashion on the self-supervised task of forecasting the second half of observed trajectories from the first half. The L2 norm of the gradient of this forecasting loss with respect to the decoder's final layer defines a score to identify distribution shifts. Our approach, first, does not affect the trajectory prediction model, ensuring no interference with original prediction performance and second, demonstrates substantial improvements on distribution shift detection for trajectory prediction on the Shifts and Argoverse datasets. Moreover, we show that this method can also be used to early detect collisions of a deep Q-Network motion planner in the Highway simulator. Source code is available at https://github.com/Michedev/forecasting-the-past.
Michele De Vita, Julian Wiederer, Vasileios Belagiannis
The robustness of trajectory prediction models is crucial for developing safe autonomous driving systems. Adversarial attacks on trajectory prediction can significantly impair the accuracy of predicted trajectories, leading to hazardous driving behaviors. While heuristic defense strategies have been implemented to enhance the robustness of trajectory prediction models, these measures often fail against more sophisticated, targeted adversarial attacks. Hence, there is a pressing need to establish verifiable safety assurances for trajectory prediction models. In this paper, we extend the traditional Randomized Smoothing framework to "TrajRS", which provides a certified robust radius for smoothed trajectory predictors. We clarify and expand the formal definitions of robustness in trajectory prediction and tailor the practical TrajRS scheme specifically to "robustness for the optimal prediction" and "robustness for all possible predictions". An extensive set of experiments demonstrates that TrajRS effectively achieves robustness certification for all smoothed pedestrian trajectory predictors in this work.
Liang Zhang, Gaojie Jin, Yao Shi +4
Key Laboratory of System Software (Chinese Academy of Sciences) and State Key Laboratory of Computer Science, Institute of Software, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences, China · Hangzhou Institute for Advanced Study, UCAS, China +1
Existing trajectory prediction methods exhibit significant performance degradation under distribution shifts during test time. Although test-time training techniques have been explored to enable adaptation, current approaches rely on an offline pre-trained predictor that lacks online learning flexibility. Moreover, they depend on fixed online model updating rules that do not accommodate the specific characteristics of test data. To address these limitations, we first propose a meta-learning framework to directly optimize the predictor for fast and accurate online adaptation, which performs bi-level optimization on the performance of simulated test-time adaptation tasks during pre-training. Furthermore, at test time, we introduce a data-adaptive model updating mechanism that dynamically adjusts the predefined learning rates and updating frequencies based on online partial derivatives and hard sample selection. This mechanism enables the online learning rate to suit the test data, and focuses on informative hard samples to enhance efficiency. Experiments are conducted on various challenging cross-dataset distribution shift scenarios, including nuScenes, Lyft, and Waymo. Results demonstrate that our method achieves superior adaptation accuracy, surpassing state-of-the-art test-time training methods for trajectory prediction. Additionally, our method excels under suboptimal learning rates and high FPS demands, showcasing its robustness and practicality.
Yuning Wang, Pu Zhang, Yuan He +2
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, IAIR, Xi’an Jiaotong University, China · KargoBot, China