Class-Incremental Motion Forecasting

Nicolas Schischka; Nikhil Gosala; B Ravi Kiran; Senthil Yogamani; Abhinav Valada

cs.CVMar 10, 2026

Class-Incremental Motion Forecasting

Authors: Nicolas Schischka, Nikhil Gosala, B Ravi Kiran, Senthil Yogamani, Abhinav Valada

Organizations: Department of Computer Science, University of Freiburg, Germany. · Qualcomm SARL France. · Automated Driving, Qualcomm Technologies, Inc.

Abstract

Motion forecasting enables autonomous vehicles to anticipate scene evolution by predicting the future trajectories of dynamic agents. However, existing approaches typically assume a closed-world setting with a fixed object taxonomy and access to high-quality perception, limiting their applicability in the real world where perception is imperfect, and new object classes may emerge over time. In this work, we introduce class-incremental motion forecasting, a novel setting in which new object classes are sequentially introduced over time and future object trajectories are predicted directly from camera images. We propose the first end-to-end framework for this setting, which adapts to newly introduced classes while mitigating catastrophic forgetting of previously learned ones. Our method generates motion forecasting pseudo-labels for known classes and matches them with 2D instance masks from an open-vocabulary segmentation model. This 3D-to-2D keypoint voting mechanism filters inconsistent and overconfident predictions, while a query feature variance-based replay strategy samples informative past sequences to preserve prior knowledge. Extensive evaluations on nuScenes and Argoverse 2 show that our approach successfully preserves performance on known classes while effectively adapting to novel ones. We further demonstrate zero-shot transfer to real-world driving and show that the framework extends naturally to open- and closed-loop end-to-end class-incremental planning on nuScenes and NeuroNCAP. Code and models will be made publicly available at https://omen.cs.uni-freiburg.de.

Figures & tables

Fig. 1 : Our approach is the first to tackle the problem of class-incremental motion forecasting. In contrast to (a) traditional and (b) end-to-end motion forecasting, (c) the underlying model is trained incrementally, with access to labels only for a subset of all classes $C^{i}$ and raw multi-view camera images. As a result, it continually learns to forecast the motion of all classes in an end-to-end manner, while handling imperfect object detections and successfully combating catastrophic forgetting.

Fig. 2 : Illustration of the proposed OMEN architecture. At each incremental step $i$ , we create detection and motion forecasting pseudo-labels for the old categories with the old model $\Phi^{i-1}$ , filter them in an offline process via a 3D-to-2D keypoint voting mechanism with the 2D instance masks output by a VLM, and add them to the detection ( $\triangle$ ) and motion forecasting ( $\circ$ ) ground truth of $D^{i}$ , as detailed in Section III-B . Furthermore, a replay buffer is created as described in Section III-C , based on the latent motion query space of the old model.

Fig. 3 : Visualization of the VLM-guided pseudo-label filtering for objects of the car class. Whenever a bounding box is associated with a VLM instance mask, the mask is removed for the remaining objects. Therefore, object $k$ is not added to the set of pseudo-labels since it refers to the same object as object $i$ that has a higher confidence. Note that objects $i$ and $k$ do not overlap in 3D space, which means they cannot be filtered based on 3D IoU.

Method	EPA (%) $\uparrow$			AP_f static (%) $\uparrow$			AP_f linear (%) $\uparrow$			AP_f non-linear (%) $\uparrow$			mAP_f (%) $\uparrow$
Method	(1)	(2)	All	(1)	(2)	All	(1)	(2)	All	(1)	(2)	All	(1)	(2)	All
Joint Training	45.92	35.49	30.26	54.27	31.99	34.23	29.01	31.10	18.78	14.97	14.11	6.60	32.75	25.73	19.87
Forgetting	0.0	0.0	2.31	0.0	0.0	3.39	0.0	0.0	1.28	0.0	0.0	0.64	0.0	0.0	1.77
LwF [ 47 ]	25.65	6.07	13.05	33.26	14.35	18.40	16.80	7.88	9.23	6.89	2.61	2.71	18.98	8.28	10.11
Pseudo-Labeling	3.25	-1.38	8.47	41.01	13.51	22.05	10.61	19.73	10.62	2.94	8.05	2.73	18.19	13.76	11.80
Pseudo-Labeling $\dagger$	19.82	6.87	12.12	46.13	17.76	22.78	25.06	18.40	14.47	10.63	7.35	3.93	27.27	14.50	13.73

TABLE I : Class-incremental motion prediction results on the nuScenes validation set for the proposed per-class incremental setting. We report all metrics for the first class car (1), the second class pedestrian (2), and all classes (All).

\dagger

means that for this experiment, our proposed pseudo-labeling from the future detections was used. The best results are marked bold , and the second-best underlined .

Method	mAP (%) $\uparrow$		NDS $(\%)\uparrow$		AMOTA $(\%)\uparrow$
Method	(1)	All	(1)	All	(1)	All
Joint Training	59.6	37.1	68.1	48.2	49.9	30.7
Forgetting	0.0	4.6	0.0	6.1	0.0	5.2
LwF [ 47 ]	39.1	20.8	56.0	37.9	24.5	13.6
Pseudo-Labeling	45.9	25.6	58.1	41.0	29.2	19.4
Pseudo-Labeling $\dagger$	49.9	25.8	61.7	40.8	41.3	20.8

TABLE II : Class-incremental 3D object detection and tracking results on the nuScenes validation set for the proposed per-class incremental setting. We report all metrics for the first class car (1) and all classes (All).

\dagger

means that for this experiment, our proposed pseudo-labeling from the future detections was used. The best results are marked bold , and the second-best underlined .

Method	AP_f static (%) $\uparrow$				AP_f linear (%) $\uparrow$				AP_f non-linear (%) $\uparrow$				mAP_f (%) $\uparrow$
Method	(1-4)	(5)	(6-7)	All	(1-4)	(5)	(6-7)	All	(1-4)	(5)	(6-7)	All	(1-4)	(5)	(6-7)	All
Joint Training	30.61	31.99	42.60	34.23	15.07	31.10	20.04	18.78	7.85	14.11	0.36	6.60	17.85	25.73	21.00	19.87
Forgetting	0.0	0.0	28.42	8.12	0.0	0.0	13.49	3.85	0.0	0.0	0.10	0.03	0.0	0.0	14.00	4.00
LwF [ 47 ]	15.61	17.63	27.71	19.36	3.88	8.58	9.16	6.06	1.59	4.31	0.18	1.58	7.03	10.17	12.35	9.00
Pseudo-Labeling	19.92	15.36	26.01	21.01	4.22	19.04	3.74	6.20	1.89	8.44	0.13	2.32	8.68	14.28	9.96	9.84
Pseudo-Labeling $\dagger$	20.53	18.90	23.64	21.18	6.07	19.67	8.98	8.84	2.21	8.23	0.40	2.55	9.60	15.60	11.00	10.86

TABLE III : Class-incremental motion prediction results on the nuScenes validation set for the proposed group-incremental setting, which consists of a base training with four classes (1-4), an incremental step with one class (5), followed by a second incremental step with two classes (6-7). We report all metrics for the individual steps and all classes (All).

\dagger

means that for this experiment, our proposed pseudo-labeling from the future detections was used. The best results are marked bold , and the second-best underlined .

Method	CDS (%) $\uparrow$			AP_f static (%) $\uparrow$			AP_f linear (%) $\uparrow$			AP_f non-linear (%) $\uparrow$			mAP_f (%) $\uparrow$
Method	(1-20)	(21-26)	All	(1-20)	(21-26)	All	(1-20)	(21-26)	All	(1-20)	(21-26)	All	(1-20)	(21-26)	All
Joint Training	31.21	16.74	27.87	44.74	26.67	41.13	27.18	16.29	25.00	18.55	29.41	21.45	32.63	24.32	30.83
Forgetting	0.0	0.72	0.17	0.0	1.74	0.35	0.0	0.09	0.02	0.0	2.49	0.66	0.0	1.46	0.32
LwF [ 47 ]	7.73	11.75	8.66	10.85	21.08	12.89	7.88	9.79	8.26	6.90	14.86	9.02	8.91	15.69	10.38
Pseudo-Labeling	29.01	15.14	25.81	41.43	20.62	37.27	25.43	11.83	22.71	11.87	27.38	16.00	29.07	19.99	27.10
Pseudo-Labeling $\dagger$	28.54	15.27	25.48	40.91	19.76	36.68	26.89	12.96	24.10	15.65	26.25	18.47	30.22	19.66	27.94

TABLE IV : Class-incremental 3D object detection and motion prediction results on the Argoverse 2 validation set for the proposed incremental 20-2 setting with a total of four steps. We set the evaluation range to 50

m

, following the AV2 End-to-End Forecasting settings for motion prediction.

\dagger

indicates that, for this experiment, we used our proposed pseudo-labeling based on future detections. The best results are marked bold , and the second-best underlined .

Method	nuScenes		Argoverse 2
Method	7 steps	3 steps	Argoverse 2
Forgetting	28.92	10.15	32.93
LwF [ 47 ]	9.94	3.12	24.02
Pseudo-Labeling	10.73	1.47	3.86
Pseudo-Labeling $\dagger$	1.65	0.55	2.71
CL-DETR $\dagger$ [ 39 ]	0.41	$-$ 0.15	4.03

TABLE V : Class-incremental forgetting percentage points (FPP) regarding mAP_f for the first set of classes

C^{0}

on the three proposed splits. Negative values indicate improvement over the base training, positive values indicate forgetting.

\dagger

means that for this experiment, we used our proposed pseudo-labeling from the future detections. The best results are marked bold , and the second-best underlined .

Fig. 4 : Class-incremental motion prediction results per class and per incremental step on the nuScenes validation set for the proposed per-class incremental setting. While the class-averaged forecasting metrics increase over incremental steps, it can be observed that classes introduced at later steps, representing rarer classes, are more prone to forgetting. Moreover, some classes experience an increase in forecasting performance for intermediate steps due to the coupling of agents from different classes.

Step	L2 $(m)\downarrow$				Col. Rate $(\%)\downarrow$
Step	1 $s$	2 $s$	3 $s$	Avg.	1 $s$	2 $s$	3 $s$	Avg.
Joint Training	0.32	0.63	1.04	0.66	0.02	0.08	0.34	0.14
1	0.38	0.76	1.27	0.80	0.03	0.12	0.60	0.25
2	0.36	0.71	1.18	0.75	0.02	0.10	0.45	0.19
3	0.33	0.64	1.05	0.67	0.02	0.10	0.41	0.18
4	0.35	0.67	1.10	0.70	0.03	0.16	0.46	0.21

TABLE VI : Class-incremental open-loop planning results after each training stage on the nuScenes validation set for the proposed per-class incremental setting. The best results are marked bold , and the second-best underlined .

Fig. 5 : Class-incremental closed-loop planning results on the NeuroNCAP evaluation for the proposed per-class incremental setting. Compared to jointly training the model with all classes, OMEN achieves a comparable NeuroNCAP score and an even slightly lower collision rate.

Forecasting PL		VLM Filt.	AMOTA (%) $\uparrow$ / mAP_f (%) $\uparrow$
pred.	fut det.		(1)	(2)	(3)	(4)	(5)	(6)	(7)	All
Joint Training			49.9 / 32.75	32.4 / 25.73	29.5 / 15.15	0.0 / 5.14	32.6 / 17.81	36.8 / 24.19	33.8 / 18.34	30.7 / 19.87
			0.0 / 0.0	0.0 / 0.0	0.0 / 0.0	0.0 / 0.0	0.0 / 0.0	0.0 / 0.0	36.4 / 12.39	5.2 / 1.77
✓			29.2 / 18.19	17.4 / 13.76	5.2 / 6.71	0.0 / 1.50	25.2 / 11.01	25.0 / 15.08	33.8 / 16.36	19.4 / 11.80
	✓		41.3 / 27.27	20.8 / 14.50	11.0 / 9.04	0.0 / 3.48	18.7 / 9.75	27.2 / 17.82	26.2 / 14.21	20.8 / 13.73
	✓	✓	52.0 / 29.45	30.4 / 14.70	19.6 / 10.34	0.0 / 3.10	27.8 / 12.65	32.1 / 17.90	30.2 / 14.06	27.4 / 14.60

TABLE VII : Ablation study on the per-class incremental setting to study the influence of each contribution. The best results are marked bold .

Panel of Figure 13 — Fig. 6 : Qualitative comparison of selected scenes per replay buffer strategy (CL-DETR [ 39 ] , DINOv3, Ours). We show one sample from each of the six highest-rated sequences per method after training the class car on the per-class incremental setting. As can be seen, the sequences selected by CL-DETR represent the data label distribution of the dataset well, but only contain a few moving objects without particularly interesting motion patterns. When using diverse DINO features, the scenes cover a variety of weather and lighting conditions in the image space, but they often only contain a few objects that are not necessarily moving. In contrast, our proposed sampling strategy selects scenarios with a large number of moving objects with diverse motions.

Setting	Selection	AP_f linear (%) $\uparrow$			AP_f non-linear (%) $\uparrow$
per-class (7 steps)	Classes	(1)	(2)	All	(1)	(2)	All
	Random	27.32	19.26	15.68	12.90	6.37	4.29
	DINOv3	26.97	18.99	15.25	12.75	6.69	4.01
	CL-DETR	27.04	19.28	14.58	12.86	6.80	4.10
	Ours	28.86	19.50	16.24	13.04	8.10	4.57
group (3 steps)	Classes	(1-4)	(5)	All	(1-4)	(5)	All

TABLE VIII : Ablation study for the exemplar selection strategy used on both proposed nuScenes settings. For the random exemplar selections, we run three trials with different random seeds and average the results. The best results are marked bold .

Buffer Size	AP_f non-linear (%) $\uparrow$			mAP_f (%) $\uparrow$
Buffer Size	(1)	(2)	All	(1)	(2)	All
$K=6$	11.84	5.53	3.56	29.60	15.10	14.86
$K=12$	12.53	7.55	4.37	30.16	15.14	15.50
$K=30$	13.04	8.10	4.57	29.95	15.82	15.60

TABLE IX : Ablation study for the size of the replay buffer

K

on the per-class incremental setting. Note that we keep the buffer size constant across steps. The best results are marked bold .

Function	AP_f non-linear (%) $\uparrow$			mAP_f (%) $\uparrow$
Function	(1)	(2)	All	(1)	(2)	All
STD	12.62	6.61	4.48	29.54	14.55	15.13
MAD	13.23	6.77	4.34	29.97	14.52	15.00
SS	13.04	8.10	4.57	29.95	15.82	15.60

TABLE X : Ablation study for the statistical function used to select the replay buffer samples based on the latent query space on the per-class incremental setting. The best results are marked bold .

Threshold	AP_f non-linear (%) $\uparrow$			mAP_f (%) $\uparrow$
Threshold	(1)	(2)	All	(1)	(2)	All
$\theta=0.2$	13.02	7.13	3.97	28.90	14.86	14.58
$\theta=0.3$	13.04	8.10	4.57	29.95	15.82	15.60
$\theta=0.4$	12.20	7.11	4.42	28.46	13.63	15.00

TABLE XI : Ablation study for the pseudo-labeling threshold

\theta

on the per-class incremental setting. The best results are marked bold .

Method	Step 0 $(hh:mm)\downarrow$			Step 1 $(hh:mm)\downarrow$			Step 2 $(hh:mm)\downarrow$	Total $(hh:mm)\downarrow$
Method	$t_{\text{train}}$	$t_{\text{replay}}$	$t_{\text{pseudo}}$	$t_{\text{train}}$	$t_{\text{replay}}$	$t_{\text{pseudo}}$	$t_{\text{train}}$	$t_{\text{total}}$
Joint Training	03:17	-	-	06:41	-	-	09:59	19:57
Ours w/o pre-computed VLM masks	03:17	00:04	01:18	03:21	00:04	01:18	03:08	12:30
Ours w/ pre-computed VLM masks	03:17	00:04	00:09	03:21	00:04	00:09	03:08	10:12

TABLE XII : Comparison of the training time

t

between OMEN and the joint re-training with all labels in each step (0-2) for the group-incremental nuScenes setting, divided into training time

t_{\text{train}}

, replay inference & buffer generation time

t_{\text{replay}}

, and pseudo labeling time

t_{\text{pseudo}}

VLM	AMOTA (%) $\uparrow$			mAP_f (%) $\uparrow$
VLM	(1)	(2)	All	(1)	(2)	All
YOLOE-26l-seg [ 52 ]	52.1	31.3	27.8	30.18	15.53	15.59
Grounded SAM 2 [ 45 ]	52.7	32.5	28.0	29.95	15.82	15.60

TABLE XIII : Ablation study between Grounded SAM 2 and a light-weight open-vocabulary segmentation model (YOLOE-26l-seg) on the per-class incremental setting. The best results are marked bold .

OMG	AP_f non-linear (%) $\uparrow$			mAP_f (%) $\uparrow$
OMG	(1)	(2)	All	(1)	(2)	All
✓	11.94	6.68	3.52	28.27	15.09	14.31
✗	13.04	8.10	4.57	29.95	15.82	15.60

TABLE XIV : Ablation study for the online map generation (OMG) module on the per-class incremental setting. The best results are marked bold .

Class- aware	AP_f non-linear (%) $\uparrow$				mAP_f (%) $\uparrow$
Class- aware	(3)	(2)	(6)	All	(3)	(2)	(6)	All
✗	2.29	3.12	0.06	3.18	10.08	10.83	17.25	14.51
✓	2.91	8.10	0.14	4.57	11.26	15.82	19.62	15.60

TABLE XV : Comparison between a class-agnostic motion forecasting algorithm and OMEN on the per-class incremental setting. For this experiment, the motion forecasting head is frozen after training on cars. The best results are marked bold . The results are shown for the class truck (3) that has similar motion behavior as cars, and the classes pedestrian (2) and bicycle (6) with more dissimilar behavior.

Fig. 7 : Qualitative results on the nuScenes dataset for the proposed per-class incremental setting. The range of the birds-eye-view images is set to $40\text{\,}\mathrm{m}$ in the longitudinal and lateral direction of the ego-vehicle. In the top two examples, it can be seen that CL-DETR [ 39 ] might predict movements for static objects and more often misses objects such as pedestrians than OMEN. Moreover, as seen in the residual rows, our approach predicts non-linear motions for objects of different classes more accurately, also under varying lighting conditions.

Fig. 8 : Qualitative open-loop planning results for the per-class incremental setting on the nuScenes dataset. The top- $3$ modes for the planned future ego-trajectory over the next $6\text{\,}\mathrm{s}$ are shown in orange from a birds-eye view, while the planning command is printed at the bottom left. The range of the birds-eye-view images is set to $40\text{\,}\mathrm{m}$ in the longitudinal and lateral direction of the ego-vehicle. We compare the results from the first step of training (only with cars) against the final trained model and the ground truth (GT).

Fig. 9 : Extended qualitative zero-shot real-world results recorded with our in-house perception car. The range of the birds-eye-view images is set to $40\text{\,}\mathrm{m}$ in the longitudinal and lateral direction of the ego-vehicle, while the prediction horizon is set to $6\text{\,}\mathrm{s}$ , and the top- $3$ predicted modes per object are shown. The camera’s field of view is highlighted in the bird’s-eye view projection.

Fig. 10 : The self-driving perception car that we used in the real-world experiments.

Class-Incremental Learning

Class-incremental learning studies how models can add new categories over time while retaining the ability to recognize previously learned classes.

See all 157 papers

PanguMotion: Continuous Driving Motion Forecasting with Pangu Transformers
Mar 17, 2026Quanhao Ren, Yicheng Li, Nan SongTrajectory Prediction Vehicle Trajectory Prediction
School of Data Science, Fudan University
SHARP: Short-Window Streaming for Accurate and Robust Prediction in Motion Forecasting
Mar 30, 2026Alexander Prutsch, Christian Fruhwirth-Reisinger, David Schinagl +1Trajectory Prediction Vehicle Trajectory Prediction
Institute of Visual Computing, Graz University of Technology
Envisioning the Future, One Step at a Time
Apr 10, 2026Stefan Andreas Baumann, Jannik Wiese, Tommaso Martorella +2Trajectory Prediction Probabilistic Trajectory Prediction
Munich Center for Machine Learning (MCML) · Netflix

Mar 17, 2026J/K move · Enter open · S save

PanguMotion: Continuous Driving Motion Forecasting with Pangu Transformers

Quanhao Ren, Yicheng Li, Nan Song

School of Data Science, Fudan University

Trajectory Prediction Vehicle Trajectory Prediction Spatiotemporal Forecasting

Open paper AskPDF arXiv

Motion forecasting is a core task in autonomous driving systems, aiming to accurately predict the future trajectories of surrounding agents to ensure driving safety. Existing methods typically process discrete driving scenes independently, neglecting the temporal continuity and historical context correlations inherent in real-world driving environments. This paper proposes PanguMotion, a motion forecasting framework for continuous driving scenarios that integrates Transformer blocks from the Pangu-1B large language model as feature enhancement modules into autonomous driving motion prediction architectures. We conduct experiments on the Argoverse 2 datasets processed by the RealMotion data reorganization strategy, transforming each independent scene into a continuous sequence to mimic real-world driving scenarios.

Abstract

Class-Incremental Learning

Explore similar work

PanguMotion: Continuous Driving Motion Forecasting with Pangu Transformers

SHARP: Short-Window Streaming for Accurate and Robust Prediction in Motion Forecasting

Envisioning the Future, One Step at a Time