Uncertainty-Aware Trajectory Prediction: A Unified Framework Harnessing Positional and Semantic Uncertainties
Authors: Jintao Sun, Hu Zhang, Gangyi Ding, Zhedong Zheng
Organizations: School of Computer Science and Technology, Beijing Institute of Technology, China 100081. · CSIRO DATA61, Australia. · Faculty of Science and Technology and Institute of Collaborative Innovation, University of Macau, China 999078.
Trajectory prediction seeks to forecast the future motion of dynamic entities, such as vehicles and pedestrians, given a temporal horizon of historical movement data and environmental context. A central challenge in this domain is the inherent uncertainty in real-time maps, arising from two primary sources: (1) positional inaccuracies due to sensor limitations or environmental occlusions, and (2) semantic errors stemming from misinterpretations of scene context. To address these challenges, we propose a novel unified framework that jointly models positional and semantic uncertainties and explicitly integrates them into the trajectory prediction pipeline. Our approach employs a dual-head architecture to independently estimate semantic and positional predictions in a dual-pass manner, deriving prediction variances as uncertainty indicators in an end-to-end fashion. These uncertainties are subsequently fused with the semantic and positional predictions to enhance the robustness of trajectory forecasts. We evaluate our uncertainty-aware framework on the nuScenes real-world driving dataset, conducting extensive experiments across four map estimation methods and two trajectory prediction baselines. Results verify that our method (1) effectively quantifies map uncertainties through both positional and semantic dimensions, and (2) consistently improves the performance of existing trajectory prediction models across multiple metrics, including minimum Average Displacement Error (minADE), minimum Final Displacement Error (minFDE), and Miss Rate (MR). Code will available at https://github.com/JT-Sun/UATP.
Figures & tables
Fig. 1: Motivation. The 6 images on the left are captured by 6 different cameras on the vehicle. The map estimation remains challenging from RGB images, and thus inevitably contains noise, accumulating the error in the trajectory prediction. Comparing the ground-truth HD map (a) and the predicted map in (b), we can see that the error usually occurs in uncertain areas. Therefore, in this work, we intend to leverage two types of uncertainty, i.e. , positional uncertainty and semantic uncertainty, to indicate the map errors, mitigating the negative impacts. (c) shows the positional uncertainty for three categories shown in three colors: green for boundary, blue for pedestrian crossing, orange for divider, and red for the ego car. The greater the positional uncertainty of the three categories, the larger the ellipse centered on the map element. (d) shows the semantic uncertainty of our constructed high-definition map, where the purple error band indicates the likelihood of an area being misclassified as another category.
Fig. 2: Overall pipeline. Firstly, given six images from the vehicle camera, we extract 2D features via a visual backbone and transform them into BEV features. Specifically, we harness the primary and auxiliary heads to predict two BEV features given different-level feature maps. Secondly, during the map estimation stage (bottom), we perform location regression and semantic regression on the features from both primary and auxiliary heads. The resulting primary and auxiliary map element vectors ( μ and μ′ ) are used to calculate KL divergence as positional uncertainty β . Similarly, we obtain the semantic scores c and c′ from the corresponding MLPs, and then we could derive the semantic uncertainty Δc . Thirdly, we concatenate the high-definition map location information μ , the mean semantic score cˉ , and their uncertainties (β,Δc) , as the input of the downstream model (GNN or Transformer Encoder) to facilitate the scene understanding for trajectory prediction.
Online HD Map Method
Trajectory Prediction Method
HiVT [ 1 ]
DenseTNT [ 2 ]
minADE ↓
minFDE ↓
MR ↓
minADE ↓
minFDE ↓
MR ↓
MapTR [ 17 ]
0.4015
0.8418
0.0981
1.091
2.058
0.3543
MapTR [ 17 ] + [ 40 ]
0.3854
0.7909
0.0834
1.089
2.006
0.3499
MapTR [ 17 ] + Ours
0.3660 (−5%)
0.7564 (−5%)
0.0745 (−11%)
0.954 (−13%)
1.909 (−5%)
0.3429 (−2%)
MapTRv2 [ 18 ]
0.4057
0.8499
0.0992
1.214
2.312
0.4138
TABLE I: Quantitative results of eight experiments combining 4 high-definition map estimation models and 2 trajectory prediction models on the nuScenes [ 59 ] dataset are presented. Overall, we observe that our method, which integrates both positional and semantic uncertainties, outperforms previous approaches in enhancing the prediction model performance, with the most significant improvement seen in the MapTRv2-centerline method and StreamMapNet method.
Online HD
Trajectory
Unc
Unc
minADE ↓
minFDE ↓
MR ↓
Map Method
Prediction Method
pos
sem
MapTR [ 17 ]
HiVT [ 1 ]
0.4015
0.8418
0.0981
Baseline [ 40 ]
HiVT [ 1 ]
0.3854
0.7909
0.0834
Ours*
HiVT [ 1 ]
✓
0.3717
0.7820
0.0829
Ours*
HiVT [ 1 ]
✓
0.3643
0.7573
0.0812
Ours
HiVT [ 1 ]
✓
✓
0.3660
0.7564
0.0745
TABLE II: Ablation study on our main components, i.e. , positional uncertainty and semantic uncertainty. Unc_pos denotes the positional uncertainty method, while Unc_sem represents the semantic uncertainty method. We use a checkmark ✓ to indicate whether the method is applied. * means part of our uncertainty.
Method
Regression Loss Weight
LR
Gradient Norm
MapTR [ 17 ]
0.03
1.0E-4
3
MapTRv2 [ 18 ]
0.03
1.0E-4
3
MapTRv2-Centerline [ 18 ]
0.03
1.0E-4
3
StreamMapNet [ 49 ]
2.00
1.0E-4
3
TABLE III: Map prediction training hyperparameters.
Online HD Map Method
LR
Batch Size
Dropout
MapTR [ 17 ] + HiVT [ 1 ]
3.5E-4
32
0.1
MapTR [ 17 ] + DenseTN [ 2 ]
3.0E-3
16
0.1
MapTRv2 [ 18 ] + HiVT [ 1 ]
3.5E-4
32
0.1
MapTRv2 [ 18 ] + DenseTNT [ 2 ]
2.0E-3
16
0.1
MapTRv2-Centerline [ 18 ] + HiVT [ 1 ]
3.5E-4
32
0.1
MapTRv2-Centerline [ 18 ] + DenseTNT [ 2 ]
3.5E-3
16
0.1
TABLE IV: Hyperparameters chosen for different trajectory prediction methods.
Fig. 3: The left figure shows the effectiveness of our proposed method for estimating high-definition map positional uncertainty and semantic uncertainty in a normal road scenario in the test set. The right figure also shows the effectiveness of our proposed method for estimating high-definition map positional uncertainty and semantic uncertainty in test set scenarios involving curved roads and parking lots. Green denotes road boundaries, blue denotes pedestrian crossings, orange denotes lane dividers, purple indicates category semantic uncertainty, gray denotes lane centerlines, the red vehicle denotes the ego vehicle, and the gray vehicles denote other agents.
Fig. 4: Top Left: At busy intersections with dense map elements, our proposed uncertainty information improves vehicle trajectory prediction. Top Right: When turning, the camera perspective often fails to capture all surrounding road conditions, potentially causing trajectory predictions to extend beyond the road boundaries. Bottom Left: In complex environments with numerous occlusions, both types of uncertainties in map prediction increase. Bottom Right: When lane information is unclear, the environment is open, and map estimation is poor, our uncertainty information helps maintain high accuracy in trajectory prediction despite incomplete map input.
Fig. 5: Map visualization of our uncertainty method in the Argoversev2 sensor dataset. In the figure, green represents road boundaries, blue represents pedestrian crossings, orange represents lane dividers, purple indicates category semantic uncertainty, red vehicle denotes the ego vehicle.
Fig. 6: Visualization of our uncertainty estimation in challenging conditions: (a) rainy scenario with visibility distortions, and (b) nighttime scenario with limited visibility. Our method effectively highlights positional and label semantic uncertainties, estimates the map uncertainty information, and demonstrates robustness in diverse environments.
Fig. 7: Visualization of a failure case. The yellow box in the figure illustrates the discrepancy between the ground truth and our predicted trajectory, highlighting a case where an abnormal ground truth deviation under simple, unobstructed road conditions leads to prediction errors.
Most trajectory forecasting models are trained on clean annotated histories, and are often evaluated under the same idealized assumption, although practical deployments rely on trajectories produced by imperfect multi-object trackers. The real-world observations exhibit localization jitter, missed or unstable detections, and data-association ambiguity, which are usually either ignored or removed through denoising. This paper instead treats tracking-derived reliability cues as an informative signal to be propagated to the predictor. We propose a plug-in uncertainty-aware formulation in which each observed state is encoded as an uncertain state representation, modeled by a Gaussian distribution whose covariance combines detection-level localization uncertainty and association-level ambiguity through the law of total variance. Existing backbones are adapted with minimal architectural changes: input trajectories are represented as Gaussian observations, and predicted trajectories are produced as Gaussian forecasts rather than deterministic coordinates. To train predictors that remain robust under structured observation noise, we combine temporally correlated Ornstein-Uhlenbeck perturbations with response-based knowledge distillation from a teacher trained on clean trajectories. Experiments on Oxford Town Centre and VIRAT using real tracker outputs, together with a complementary ETH/UCY pseudo-detection protocol, show that the proposed formulation improves displacement accuracy and the reliability-sharpness trade-off of probabilistic forecasts.
Stephane Da Silva Martins, Victor Petrovic, Emanuel Aldea +1
SATIE – CNRS UMR 8029, Université Paris-Saclay, France · ENS Paris-Saclay, Université Paris-Saclay, France
Online map generation and trajectory prediction are critical components of the autonomous driving perception-prediction-planning pipeline. While modern vectorized mapping models achieve high geometric accuracy, they typically treat map estimation as a deterministic task, discarding structural uncertainty. Existing probabilistic approaches often rely on diagonal covariance matrices, which assume independence between points and fail to capture the strong spatial correlations inherent in road geometry. To address this, we propose a structured probabilistic formulation for online map generation. Our method explicitly models intra-element dependencies by predicting a dense covariance matrix, parameterized via a Low-Rank plus Diagonal (LRPD) covariance decomposition. This formulation represents uncertainty as a combination of a low-rank component, which captures global spatial structure, and a diagonal component representing independent local noise, thereby capturing geometric correlations without the prohibitive computational cost of full covariance matrices. Evaluations on the nuScenes dataset demonstrate that our uncertainty-aware framework yields consistent improvements in online map generation quality compared to deterministic baselines. Furthermore, our approach establishes new state-of-the-art performance for map-based motion prediction, highlighting the critical role of uncertainty in planning tasks. Code is published under link-available-soon.
Pritom Gogoi, Faris Janjoš, Bin Yang +1
Bosch Center for AI, Germany · University of Stuttgart, Germany · Coburg University of Applied Sciences, Germany
Accurate trajectory forecasting and well-defined predictive uncertainty are crucial for reliable, safety-critical applications such as autonomous driving. Most trajectory prediction approaches provide point estimates only, while uncertainty-aware approaches typically quantify uncertainty only in the trajectory space. In physics-aware approaches, uncertainty in the predicted motion variables should be explicitly modeled and propagated through the vehicle dynamics. Otherwise, the resulting trajectory-space uncertainty may not fully reflect the variability introduced by the underlying motion prediction. Therefore, in this work, uncertainty-aware extensions of X-TRACK (X-TRACK-DE and X-TRACK-MCD), a physics-aware trajectory prediction framework, are proposed. The proposed framework predicts future vehicle motion variables and models both aleatoric and epistemic uncertainties by propagating motion space uncertainty to trajectory space. Additionally, conformal prediction is applied to the trajectory space predictive covariance to construct uncertainty regions targeting a desired marginal coverage level. Evaluation on the highD dataset shows that X-TRACK-DE improves trajectory prediction accuracy over the deterministic baseline, while both uncertainty-aware variants provide predictive uncertainty that can be conformally calibrated to the desired marginal coverage level.
Aanchal Rajesh Chugh, Sebastian Dorn
Technische Hochschule Augsburg TTZ Landsberg am Lech