Uncertainty-Aware Trajectory Prediction: A Unified Framework Harnessing Positional and Semantic Uncertainties
Authors: Jintao Sun, Hu Zhang, Gangyi Ding, Zhedong Zheng
Organizations: School of Computer Science and Technology, Beijing Institute of Technology, China 100081. · CSIRO DATA61, Australia. · Faculty of Science and Technology and Institute of Collaborative Innovation, University of Macau, China 999078.
Trajectory prediction seeks to forecast the future motion of dynamic entities, such as vehicles and pedestrians, given a temporal horizon of historical movement data and environmental context. A central challenge in this domain is the inherent uncertainty in real-time maps, arising from two primary sources: (1) positional inaccuracies due to sensor limitations or environmental occlusions, and (2) semantic errors stemming from misinterpretations of scene context. To address these challenges, we propose a novel unified framework that jointly models positional and semantic uncertainties and explicitly integrates them into the trajectory prediction pipeline. Our approach employs a dual-head architecture to independently estimate semantic and positional predictions in a dual-pass manner, deriving prediction variances as uncertainty indicators in an end-to-end fashion. These uncertainties are subsequently fused with the semantic and positional predictions to enhance the robustness of trajectory forecasts. We evaluate our uncertainty-aware framework on the nuScenes real-world driving dataset, conducting extensive experiments across four map estimation methods and two trajectory prediction baselines. Results verify that our method (1) effectively quantifies map uncertainties through both positional and semantic dimensions, and (2) consistently improves the performance of existing trajectory prediction models across multiple metrics, including minimum Average Displacement Error (minADE), minimum Final Displacement Error (minFDE), and Miss Rate (MR). Code will available at https://github.com/JT-Sun/UATP.
Figures & tables
Fig. 1: Motivation. The 6 images on the left are captured by 6 different cameras on the vehicle. The map estimation remains challenging from RGB images, and thus inevitably contains noise, accumulating the error in the trajectory prediction. Comparing the ground-truth HD map (a) and the predicted map in (b), we can see that the error usually occurs in uncertain areas. Therefore, in this work, we intend to leverage two types of uncertainty, i.e. , positional uncertainty and semantic uncertainty, to indicate the map errors, mitigating the negative impacts. (c) shows the positional uncertainty for three categories shown in three colors: green for boundary, blue for pedestrian crossing, orange for divider, and red for the ego car. The greater the positional uncertainty of the three categories, the larger the ellipse centered on the map element. (d) shows the semantic uncertainty of our constructed high-definition map, where the purple error band indicates the likelihood of an area being misclassified as another category.
Fig. 2: Overall pipeline. Firstly, given six images from the vehicle camera, we extract 2D features via a visual backbone and transform them into BEV features. Specifically, we harness the primary and auxiliary heads to predict two BEV features given different-level feature maps. Secondly, during the map estimation stage (bottom), we perform location regression and semantic regression on the features from both primary and auxiliary heads. The resulting primary and auxiliary map element vectors ( μ and μ′ ) are used to calculate KL divergence as positional uncertainty β . Similarly, we obtain the semantic scores c and c′ from the corresponding MLPs, and then we could derive the semantic uncertainty Δc . Thirdly, we concatenate the high-definition map location information μ , the mean semantic score cˉ , and their uncertainties (β,Δc) , as the input of the downstream model (GNN or Transformer Encoder) to facilitate the scene understanding for trajectory prediction.
Online HD Map Method
Trajectory Prediction Method
HiVT [ 1 ]
DenseTNT [ 2 ]
minADE ↓
minFDE ↓
MR ↓
minADE ↓
minFDE ↓
MR ↓
MapTR [ 17 ]
0.4015
0.8418
0.0981
1.091
2.058
0.3543
MapTR [ 17 ] + [ 40 ]
0.3854
0.7909
0.0834
1.089
2.006
0.3499
MapTR [ 17 ] + Ours
0.3660 (−5%)
0.7564 (−5%)
0.0745 (−11%)
0.954 (−13%)
1.909 (−5%)
0.3429 (−2%)
MapTRv2 [ 18 ]
0.4057
0.8499
0.0992
1.214
2.312
0.4138
TABLE I: Quantitative results of eight experiments combining 4 high-definition map estimation models and 2 trajectory prediction models on the nuScenes [ 59 ] dataset are presented. Overall, we observe that our method, which integrates both positional and semantic uncertainties, outperforms previous approaches in enhancing the prediction model performance, with the most significant improvement seen in the MapTRv2-centerline method and StreamMapNet method.
Online HD
Trajectory
Unc
Unc
minADE ↓
minFDE ↓
MR ↓
Map Method
Prediction Method
pos
sem
MapTR [ 17 ]
HiVT [ 1 ]
0.4015
0.8418
0.0981
Baseline [ 40 ]
HiVT [ 1 ]
0.3854
0.7909
0.0834
Ours*
HiVT [ 1 ]
✓
0.3717
0.7820
0.0829
Ours*
HiVT [ 1 ]
✓
0.3643
0.7573
0.0812
Ours
HiVT [ 1 ]
✓
✓
0.3660
0.7564
0.0745
TABLE II: Ablation study on our main components, i.e. , positional uncertainty and semantic uncertainty. Unc_pos denotes the positional uncertainty method, while Unc_sem represents the semantic uncertainty method. We use a checkmark ✓ to indicate whether the method is applied. * means part of our uncertainty.
Method
Regression Loss Weight
LR
Gradient Norm
MapTR [ 17 ]
0.03
1.0E-4
3
MapTRv2 [ 18 ]
0.03
1.0E-4
3
MapTRv2-Centerline [ 18 ]
0.03
1.0E-4
3
StreamMapNet [ 49 ]
2.00
1.0E-4
3
TABLE III: Map prediction training hyperparameters.
Online HD Map Method
LR
Batch Size
Dropout
MapTR [ 17 ] + HiVT [ 1 ]
3.5E-4
32
0.1
MapTR [ 17 ] + DenseTN [ 2 ]
3.0E-3
16
0.1
MapTRv2 [ 18 ] + HiVT [ 1 ]
3.5E-4
32
0.1
MapTRv2 [ 18 ] + DenseTNT [ 2 ]
2.0E-3
16
0.1
MapTRv2-Centerline [ 18 ] + HiVT [ 1 ]
3.5E-4
32
0.1
MapTRv2-Centerline [ 18 ] + DenseTNT [ 2 ]
3.5E-3
16
0.1
TABLE IV: Hyperparameters chosen for different trajectory prediction methods.
Fig. 3: The left figure shows the effectiveness of our proposed method for estimating high-definition map positional uncertainty and semantic uncertainty in a normal road scenario in the test set. The right figure also shows the effectiveness of our proposed method for estimating high-definition map positional uncertainty and semantic uncertainty in test set scenarios involving curved roads and parking lots. Green denotes road boundaries, blue denotes pedestrian crossings, orange denotes lane dividers, purple indicates category semantic uncertainty, gray denotes lane centerlines, the red vehicle denotes the ego vehicle, and the gray vehicles denote other agents.
Fig. 4: Top Left: At busy intersections with dense map elements, our proposed uncertainty information improves vehicle trajectory prediction. Top Right: When turning, the camera perspective often fails to capture all surrounding road conditions, potentially causing trajectory predictions to extend beyond the road boundaries. Bottom Left: In complex environments with numerous occlusions, both types of uncertainties in map prediction increase. Bottom Right: When lane information is unclear, the environment is open, and map estimation is poor, our uncertainty information helps maintain high accuracy in trajectory prediction despite incomplete map input.
Fig. 5: Map visualization of our uncertainty method in the Argoversev2 sensor dataset. In the figure, green represents road boundaries, blue represents pedestrian crossings, orange represents lane dividers, purple indicates category semantic uncertainty, red vehicle denotes the ego vehicle.
Fig. 6: Visualization of our uncertainty estimation in challenging conditions: (a) rainy scenario with visibility distortions, and (b) nighttime scenario with limited visibility. Our method effectively highlights positional and label semantic uncertainties, estimates the map uncertainty information, and demonstrates robustness in diverse environments.
Fig. 7: Visualization of a failure case. The yellow box in the figure illustrates the discrepancy between the ground truth and our predicted trajectory, highlighting a case where an abnormal ground truth deviation under simple, unobstructed road conditions leads to prediction errors.