Authors: Nicolas Schischka, Nikhil Gosala, B Ravi Kiran, Senthil Yogamani, Abhinav Valada
Organizations: Department of Computer Science, University of Freiburg, Germany. · Qualcomm SARL France. · Automated Driving, Qualcomm Technologies, Inc.
Motion forecasting enables autonomous vehicles to anticipate scene evolution by predicting the future trajectories of dynamic agents. However, existing approaches typically assume a closed-world setting with a fixed object taxonomy and access to high-quality perception, limiting their applicability in the real world where perception is imperfect, and new object classes may emerge over time. In this work, we introduce class-incremental motion forecasting, a novel setting in which new object classes are sequentially introduced over time and future object trajectories are predicted directly from camera images. We propose the first end-to-end framework for this setting, which adapts to newly introduced classes while mitigating catastrophic forgetting of previously learned ones. Our method generates motion forecasting pseudo-labels for known classes and matches them with 2D instance masks from an open-vocabulary segmentation model. This 3D-to-2D keypoint voting mechanism filters inconsistent and overconfident predictions, while a query feature variance-based replay strategy samples informative past sequences to preserve prior knowledge. Extensive evaluations on nuScenes and Argoverse 2 show that our approach successfully preserves performance on known classes while effectively adapting to novel ones. We further demonstrate zero-shot transfer to real-world driving and show that the framework extends naturally to open- and closed-loop end-to-end class-incremental planning on nuScenes and NeuroNCAP. Code and models will be made publicly available at https://omen.cs.uni-freiburg.de.
Figures & tables
Fig. 1 : Our approach is the first to tackle the problem of class-incremental motion forecasting. In contrast to (a) traditional and (b) end-to-end motion forecasting, (c) the underlying model is trained incrementally, with access to labels only for a subset of all classes Ci and raw multi-view camera images. As a result, it continually learns to forecast the motion of all classes in an end-to-end manner, while handling imperfect object detections and successfully combating catastrophic forgetting.
Fig. 2 : Illustration of the proposed OMEN architecture. At each incremental step i , we create detection and motion forecasting pseudo-labels for the old categories with the old model Φi−1 , filter them in an offline process via a 3D-to-2D keypoint voting mechanism with the 2D instance masks output by a VLM, and add them to the detection ( △ ) and motion forecasting ( ∘ ) ground truth of Di , as detailed in Section III-B . Furthermore, a replay buffer is created as described in Section III-C , based on the latent motion query space of the old model.
Fig. 3 : Visualization of the VLM-guided pseudo-label filtering for objects of the car class. Whenever a bounding box is associated with a VLM instance mask, the mask is removed for the remaining objects. Therefore, object k is not added to the set of pseudo-labels since it refers to the same object as object i that has a higher confidence. Note that objects i and k do not overlap in 3D space, which means they cannot be filtered based on 3D IoU.
Method
EPA (%) ↑
AP_f static (%) ↑
AP_f linear (%) ↑
AP_f non-linear (%) ↑
mAP_f (%) ↑
(1)
(2)
All
(1)
(2)
All
(1)
(2)
All
(1)
(2)
All
(1)
(2)
All
Joint Training
45.92
35.49
30.26
54.27
31.99
34.23
29.01
31.10
18.78
14.97
14.11
6.60
32.75
25.73
19.87
Forgetting
0.0
0.0
2.31
0.0
0.0
3.39
0.0
0.0
1.28
0.0
0.0
0.64
0.0
0.0
1.77
LwF [ 47 ]
25.65
6.07
13.05
33.26
14.35
18.40
16.80
7.88
9.23
6.89
2.61
2.71
18.98
8.28
10.11
Pseudo-Labeling
3.25
-1.38
8.47
41.01
13.51
22.05
10.61
19.73
10.62
2.94
8.05
2.73
18.19
13.76
11.80
Pseudo-Labeling †
19.82
6.87
12.12
46.13
17.76
22.78
25.06
18.40
14.47
10.63
7.35
3.93
27.27
14.50
13.73
TABLE I : Class-incremental motion prediction results on the nuScenes validation set for the proposed per-class incremental setting. We report all metrics for the first class car (1), the second class pedestrian (2), and all classes (All). † means that for this experiment, our proposed pseudo-labeling from the future detections was used. The best results are marked bold , and the second-best underlined .
Method
mAP (%) ↑
NDS (%)↑
AMOTA (%)↑
(1)
All
(1)
All
(1)
All
Joint Training
59.6
37.1
68.1
48.2
49.9
30.7
Forgetting
0.0
4.6
0.0
6.1
0.0
5.2
LwF [ 47 ]
39.1
20.8
56.0
37.9
24.5
13.6
Pseudo-Labeling
45.9
25.6
58.1
41.0
29.2
19.4
Pseudo-Labeling †
49.9
25.8
61.7
40.8
41.3
20.8
TABLE II : Class-incremental 3D object detection and tracking results on the nuScenes validation set for the proposed per-class incremental setting. We report all metrics for the first class car (1) and all classes (All). † means that for this experiment, our proposed pseudo-labeling from the future detections was used. The best results are marked bold , and the second-best underlined .
Method
AP_f static (%) ↑
AP_f linear (%) ↑
AP_f non-linear (%) ↑
mAP_f (%) ↑
(1-4)
(5)
(6-7)
All
(1-4)
(5)
(6-7)
All
(1-4)
(5)
(6-7)
All
(1-4)
(5)
(6-7)
All
Joint Training
30.61
31.99
42.60
34.23
15.07
31.10
20.04
18.78
7.85
14.11
0.36
6.60
17.85
25.73
21.00
19.87
Forgetting
0.0
0.0
28.42
8.12
0.0
0.0
13.49
3.85
0.0
0.0
0.10
0.03
0.0
0.0
14.00
4.00
LwF [ 47 ]
15.61
17.63
27.71
19.36
3.88
8.58
9.16
6.06
1.59
4.31
0.18
1.58
7.03
10.17
12.35
9.00
Pseudo-Labeling
19.92
15.36
26.01
21.01
4.22
19.04
3.74
6.20
1.89
8.44
0.13
2.32
8.68
14.28
9.96
9.84
Pseudo-Labeling †
20.53
18.90
23.64
21.18
6.07
19.67
8.98
8.84
2.21
8.23
0.40
2.55
9.60
15.60
11.00
10.86
TABLE III : Class-incremental motion prediction results on the nuScenes validation set for the proposed group-incremental setting, which consists of a base training with four classes (1-4), an incremental step with one class (5), followed by a second incremental step with two classes (6-7). We report all metrics for the individual steps and all classes (All). † means that for this experiment, our proposed pseudo-labeling from the future detections was used. The best results are marked bold , and the second-best underlined .
Method
CDS (%) ↑
AP_f static (%) ↑
AP_f linear (%) ↑
AP_f non-linear (%) ↑
mAP_f (%) ↑
(1-20)
(21-26)
All
(1-20)
(21-26)
All
(1-20)
(21-26)
All
(1-20)
(21-26)
All
(1-20)
(21-26)
All
Joint Training
31.21
16.74
27.87
44.74
26.67
41.13
27.18
16.29
25.00
18.55
29.41
21.45
32.63
24.32
30.83
Forgetting
0.0
0.72
0.17
0.0
1.74
0.35
0.0
0.09
0.02
0.0
2.49
0.66
0.0
1.46
0.32
LwF [ 47 ]
7.73
11.75
8.66
10.85
21.08
12.89
7.88
9.79
8.26
6.90
14.86
9.02
8.91
15.69
10.38
Pseudo-Labeling
29.01
15.14
25.81
41.43
20.62
37.27
25.43
11.83
22.71
11.87
27.38
16.00
29.07
19.99
27.10
Pseudo-Labeling †
28.54
15.27
25.48
40.91
19.76
36.68
26.89
12.96
24.10
15.65
26.25
18.47
30.22
19.66
27.94
TABLE IV : Class-incremental 3D object detection and motion prediction results on the Argoverse 2 validation set for the proposed incremental 20-2 setting with a total of four steps. We set the evaluation range to 50 m , following the AV2 End-to-End Forecasting settings for motion prediction. † indicates that, for this experiment, we used our proposed pseudo-labeling based on future detections. The best results are marked bold , and the second-best underlined .
Method
nuScenes
Argoverse 2
7 steps
3 steps
Forgetting
28.92
10.15
32.93
LwF [ 47 ]
9.94
3.12
24.02
Pseudo-Labeling
10.73
1.47
3.86
Pseudo-Labeling †
1.65
0.55
2.71
CL-DETR † [ 39 ]
0.41
− 0.15
4.03
TABLE V : Class-incremental forgetting percentage points (FPP) regarding mAP_f for the first set of classes C0 on the three proposed splits. Negative values indicate improvement over the base training, positive values indicate forgetting. † means that for this experiment, we used our proposed pseudo-labeling from the future detections. The best results are marked bold , and the second-best underlined .
Fig. 4 : Class-incremental motion prediction results per class and per incremental step on the nuScenes validation set for the proposed per-class incremental setting. While the class-averaged forecasting metrics increase over incremental steps, it can be observed that classes introduced at later steps, representing rarer classes, are more prone to forgetting. Moreover, some classes experience an increase in forecasting performance for intermediate steps due to the coupling of agents from different classes.
Step
L2 (m)↓
Col. Rate (%)↓
1 s
2 s
3 s
Avg.
1 s
2 s
3 s
Avg.
Joint Training
0.32
0.63
1.04
0.66
0.02
0.08
0.34
0.14
1
0.38
0.76
1.27
0.80
0.03
0.12
0.60
0.25
2
0.36
0.71
1.18
0.75
0.02
0.10
0.45
0.19
3
0.33
0.64
1.05
0.67
0.02
0.10
0.41
0.18
4
0.35
0.67
1.10
0.70
0.03
0.16
0.46
0.21
TABLE VI : Class-incremental open-loop planning results after each training stage on the nuScenes validation set for the proposed per-class incremental setting. The best results are marked bold , and the second-best underlined .
Fig. 5 : Class-incremental closed-loop planning results on the NeuroNCAP evaluation for the proposed per-class incremental setting. Compared to jointly training the model with all classes, OMEN achieves a comparable NeuroNCAP score and an even slightly lower collision rate.
Forecasting PL
VLM Filt.
Var. Buffer
AMOTA (%) ↑ / mAP_f (%) ↑
pred.
fut det.
(1)
(2)
(3)
(4)
(5)
(6)
(7)
All
Joint Training
49.9 / 32.75
32.4 / 25.73
29.5 / 15.15
0.0 / 5.14
32.6 / 17.81
36.8 / 24.19
33.8 / 18.34
30.7 / 19.87
0.0 / 0.0
0.0 / 0.0
0.0 / 0.0
0.0 / 0.0
0.0 / 0.0
0.0 / 0.0
36.4 / 12.39
5.2 / 1.77
✓
29.2 / 18.19
17.4 / 13.76
5.2 / 6.71
0.0 / 1.50
25.2 / 11.01
25.0 / 15.08
33.8 / 16.36
19.4 / 11.80
✓
41.3 / 27.27
20.8 / 14.50
11.0 / 9.04
0.0 / 3.48
18.7 / 9.75
27.2 / 17.82
26.2 / 14.21
20.8 / 13.73
✓
✓
52.0 / 29.45
30.4 / 14.70
19.6 / 10.34
0.0 / 3.10
27.8 / 12.65
32.1 / 17.90
30.2 / 14.06
27.4 / 14.60
TABLE VII : Ablation study on the per-class incremental setting to study the influence of each contribution. The best results are marked bold .
Fig. 6 : Qualitative comparison of selected scenes per replay buffer strategy (CL-DETR [ 39 ] , DINOv3, Ours). We show one sample from each of the six highest-rated sequences per method after training the class car on the per-class incremental setting. As can be seen, the sequences selected by CL-DETR represent the data label distribution of the dataset well, but only contain a few moving objects without particularly interesting motion patterns. When using diverse DINO features, the scenes cover a variety of weather and lighting conditions in the image space, but they often only contain a few objects that are not necessarily moving. In contrast, our proposed sampling strategy selects scenarios with a large number of moving objects with diverse motions.
Setting
Selection
AP_f linear (%) ↑
AP_f non-linear (%) ↑
per-class (7 steps)
Classes
(1)
(2)
All
(1)
(2)
All
Random
27.32
19.26
15.68
12.90
6.37
4.29
DINOv3
26.97
18.99
15.25
12.75
6.69
4.01
CL-DETR
27.04
19.28
14.58
12.86
6.80
4.10
Ours
28.86
19.50
16.24
13.04
8.10
4.57
group (3 steps)
Classes
(1-4)
(5)
All
(1-4)
(5)
All
TABLE VIII : Ablation study for the exemplar selection strategy used on both proposed nuScenes settings. For the random exemplar selections, we run three trials with different random seeds and average the results. The best results are marked bold .
Buffer Size
AP_f non-linear (%) ↑
mAP_f (%) ↑
(1)
(2)
All
(1)
(2)
All
K=6
11.84
5.53
3.56
29.60
15.10
14.86
K=12
12.53
7.55
4.37
30.16
15.14
15.50
K=30
13.04
8.10
4.57
29.95
15.82
15.60
TABLE IX : Ablation study for the size of the replay buffer K on the per-class incremental setting. Note that we keep the buffer size constant across steps. The best results are marked bold .
Function
AP_f non-linear (%) ↑
mAP_f (%) ↑
(1)
(2)
All
(1)
(2)
All
STD
12.62
6.61
4.48
29.54
14.55
15.13
MAD
13.23
6.77
4.34
29.97
14.52
15.00
SS
13.04
8.10
4.57
29.95
15.82
15.60
TABLE X : Ablation study for the statistical function used to select the replay buffer samples based on the latent query space on the per-class incremental setting. The best results are marked bold .
Threshold
AP_f non-linear (%) ↑
mAP_f (%) ↑
(1)
(2)
All
(1)
(2)
All
θ=0.2
13.02
7.13
3.97
28.90
14.86
14.58
θ=0.3
13.04
8.10
4.57
29.95
15.82
15.60
θ=0.4
12.20
7.11
4.42
28.46
13.63
15.00
TABLE XI : Ablation study for the pseudo-labeling threshold θ on the per-class incremental setting. The best results are marked bold .
Method
Step 0 (hh:mm)↓
Step 1 (hh:mm)↓
Step 2 (hh:mm)↓
Total (hh:mm)↓
ttrain
treplay
tpseudo
ttrain
treplay
tpseudo
ttrain
ttotal
Joint Training
03:17
-
-
06:41
-
-
09:59
19:57
Ours w/o pre-computed VLM masks
03:17
00:04
01:18
03:21
00:04
01:18
03:08
12:30
Ours w/ pre-computed VLM masks
03:17
00:04
00:09
03:21
00:04
00:09
03:08
10:12
TABLE XII : Comparison of the training time t between OMEN and the joint re-training with all labels in each step (0-2) for the group-incremental nuScenes setting, divided into training time ttrain , replay inference & buffer generation time treplay , and pseudo labeling time tpseudo .
VLM
AMOTA (%) ↑
mAP_f (%) ↑
(1)
(2)
All
(1)
(2)
All
YOLOE-26l-seg [ 52 ]
52.1
31.3
27.8
30.18
15.53
15.59
Grounded SAM 2 [ 45 ]
52.7
32.5
28.0
29.95
15.82
15.60
TABLE XIII : Ablation study between Grounded SAM 2 and a light-weight open-vocabulary segmentation model (YOLOE-26l-seg) on the per-class incremental setting. The best results are marked bold .
OMG
AP_f non-linear (%) ↑
mAP_f (%) ↑
(1)
(2)
All
(1)
(2)
All
✓
11.94
6.68
3.52
28.27
15.09
14.31
✗
13.04
8.10
4.57
29.95
15.82
15.60
TABLE XIV : Ablation study for the online map generation (OMG) module on the per-class incremental setting. The best results are marked bold .
Class- aware
AP_f non-linear (%) ↑
mAP_f (%) ↑
(3)
(2)
(6)
All
(3)
(2)
(6)
All
✗
2.29
3.12
0.06
3.18
10.08
10.83
17.25
14.51
✓
2.91
8.10
0.14
4.57
11.26
15.82
19.62
15.60
TABLE XV : Comparison between a class-agnostic motion forecasting algorithm and OMEN on the per-class incremental setting. For this experiment, the motion forecasting head is frozen after training on cars. The best results are marked bold . The results are shown for the class truck (3) that has similar motion behavior as cars, and the classes pedestrian (2) and bicycle (6) with more dissimilar behavior.
Fig. 7 : Qualitative results on the nuScenes dataset for the proposed per-class incremental setting. The range of the birds-eye-view images is set to 40m in the longitudinal and lateral direction of the ego-vehicle. In the top two examples, it can be seen that CL-DETR [ 39 ] might predict movements for static objects and more often misses objects such as pedestrians than OMEN. Moreover, as seen in the residual rows, our approach predicts non-linear motions for objects of different classes more accurately, also under varying lighting conditions.
Fig. 8 : Qualitative open-loop planning results for the per-class incremental setting on the nuScenes dataset. The top- 3 modes for the planned future ego-trajectory over the next 6s are shown in orange from a birds-eye view, while the planning command is printed at the bottom left. The range of the birds-eye-view images is set to 40m in the longitudinal and lateral direction of the ego-vehicle. We compare the results from the first step of training (only with cars) against the final trained model and the ground truth (GT).
Fig. 9 : Extended qualitative zero-shot real-world results recorded with our in-house perception car. The range of the birds-eye-view images is set to 40m in the longitudinal and lateral direction of the ego-vehicle, while the prediction horizon is set to 6s , and the top- 3 predicted modes per object are shown. The camera’s field of view is highlighted in the bird’s-eye view projection.
Fig. 10 : The self-driving perception car that we used in the real-world experiments.
Motion forecasting is a core task in autonomous driving systems, aiming to accurately predict the future trajectories of surrounding agents to ensure driving safety. Existing methods typically process discrete driving scenes independently, neglecting the temporal continuity and historical context correlations inherent in real-world driving environments. This paper proposes PanguMotion, a motion forecasting framework for continuous driving scenarios that integrates Transformer blocks from the Pangu-1B large language model as feature enhancement modules into autonomous driving motion prediction architectures. We conduct experiments on the Argoverse 2 datasets processed by the RealMotion data reorganization strategy, transforming each independent scene into a continuous sequence to mimic real-world driving scenarios.
In dynamic traffic environments, motion forecasting models must be able to accurately estimate future trajectories continuously. Streaming-based methods are a promising solution, but despite recent advances, their performance often degrades when exposed to heterogeneous observation lengths. To address this, we propose a novel streaming-based motion forecasting framework that explicitly focuses on evolving scenes. Our method incrementally processes incoming observation windows and leverages an instance-aware context streaming to maintain and update latent agent representations across inference steps. A dual training objective further enables consistent forecasting accuracy across diverse observation horizons. Extensive experiments on Argoverse 2, nuScenes, and Argoverse 1 demonstrate the robustness of our approach under evolving scene conditions and also on the single-agent benchmarks. Our model achieves state-of-the-art performance in streaming inference on the Argoverse 2 multi-agent benchmark, while maintaining minimal latency, highlighting its suitability for real-world deployment.
Alexander Prutsch, Christian Fruhwirth-Reisinger, David Schinagl +1
Institute of Visual Computing, Graz University of Technology
Accurately anticipating how complex, diverse scenes will evolve requires models that represent uncertainty, simulate along extended interaction chains, and efficiently explore many plausible futures. Yet most existing approaches rely on dense video or latent-space prediction, expending substantial capacity on dense appearance rather than on the underlying sparse trajectories of points in the scene. This makes large-scale exploration of future hypotheses costly and limits performance when long-horizon, multi-modal motion is essential. We address this by formulating the prediction of open-set future scene dynamics as step-wise inference over sparse point trajectories. Our autoregressive diffusion model advances these trajectories through short, locally predictable transitions, explicitly modeling the growth of uncertainty over time. This dynamics-centric representation enables fast rollout of thousands of diverse futures from a single image, optionally guided by initial constraints on motion, while maintaining physical plausibility and long-range coherence. We further introduce OWM, a benchmark for open-set motion prediction based on diverse in-the-wild videos, to evaluate accuracy and variability of predicted trajectory distributions under real-world uncertainty. Our method matches or surpasses dense simulators in predictive accuracy while achieving orders-of-magnitude higher sampling speed, making open-set future prediction both scalable and practical. Project page: http://compvis.github.io/myriad.
Stefan Andreas Baumann, Jannik Wiese, Tommaso Martorella +2
Munich Center for Machine Learning (MCML) · Netflix