SHARP: Short-Window Streaming for Accurate and Robust Prediction in Motion Forecasting

Alexander Prutsch; Christian Fruhwirth-Reisinger; David Schinagl; Horst Possegger

cs.CVMar 30, 2026

SHARP: Short-Window Streaming for Accurate and Robust Prediction in Motion Forecasting

Authors: Alexander Prutsch, Christian Fruhwirth-Reisinger, David Schinagl, Horst Possegger

Organizations: Institute of Visual Computing, Graz University of Technology

Abstract

In dynamic traffic environments, motion forecasting models must be able to accurately estimate future trajectories continuously. Streaming-based methods are a promising solution, but despite recent advances, their performance often degrades when exposed to heterogeneous observation lengths. To address this, we propose a novel streaming-based motion forecasting framework that explicitly focuses on evolving scenes. Our method incrementally processes incoming observation windows and leverages an instance-aware context streaming to maintain and update latent agent representations across inference steps. A dual training objective further enables consistent forecasting accuracy across diverse observation horizons. Extensive experiments on Argoverse 2, nuScenes, and Argoverse 1 demonstrate the robustness of our approach under evolving scene conditions and also on the single-agent benchmarks. Our model achieves state-of-the-art performance in streaming inference on the Argoverse 2 multi-agent benchmark, while maintaining minimal latency, highlighting its suitability for real-world deployment.

Figures & tables

Figure 1: In real-world driving scenes, the available context history for different agents is heterogeneous. Therefore, forecasting models should be able to make accurate predictions for both long-term (brown car) and short-term (green car) contexts, where agents have been observed for a long time or have recently entered the field of view. While existing models often struggle with such varying context lengths, our SHARP is explicitly designed to provide accurate forecasts in such dynamically evolving scenes. Dotted lines indicate the available observations, while transparent paths show the corresponding future trajectory.

Figure 2: To jointly incorporate newly detected agents and long-term agent histories when forecasting motions in evolving scenes, we leverage a streaming-based motion forecasting model. The example at time step

t-1

illustrates the standard model pass without streaming context, consisting of separate agent and lane encoders (

f_{A}

and

f_{L}

), a scene encoder (

f_{S}

), and a trajectory decoder (

f_{D}

) with a streaming refinement module (

f_{R}

). At the next time step

t

, we integrate the previous scene context

S_{\text{enc}}^{t-1}

via an instance-aware context streamer (

f_{\text{IA}}

) and generate auxiliary target-centric features

C^{t}

by aggregating scene elements closely around the endpoints of the previous predictions

F^{t-1}

. To improve robustness, we also perform, only during training, a parallel model pass without streaming modules, producing

F^{t}_{\text{chunk}}

which is used to compute the

\mathcal{L}_{\text{chunk}}

objective.

Figure 3: Comparison of different evaluation setups on a motion forecasting dataset with

H_{t}

as historical context and

F_{t}

as future for the prediction task. The standard benchmark evaluation only considers a single history/future split per scenario (first row). In our experiments, we test the models at different time steps

t_{p}^{i}

into the scenario and using varying context lengths

T^{i}_{\text{cl}}

as model input and also evaluating for different future horizons

T^{i}_{f}

Table 1: Evaluation of streaming-based methods on the AV2 validation set at varying prediction time steps

t_{p}

and context lengths

T_{\text{cl}}

. To ensure a fair comparison, we test related work [ 36 , 50 ] using their standard streaming configuration (3 s input windows with 1 s displacement) and only adjust the number of inference passes to vary

T_{\text{cl}}

. The top block evaluates short observation lengths, while the second block extends the context to achieve streaming processing beyond the standard 3 iterative steps of [ 36 , 50 ] . The third block tests streaming with

T_{\text{cl}}

= 4 s at different prediction times. The bottom block corresponds to the standard AV2 evaluation. For predictions at

t_{p}

>

5 s, the evaluation is truncated as only 11 s are available. Further details on the experiment setup are provided in the supplementary material.

Method	Streaming	avgMinADE 1	avgMinFDE 1	actorMR 6	avgMinADE 6	avgMinFDE 6	avgBrierMinFDE 6
FJMP [ 33 ]	✗	1.52	4.00	0.23	0.81	1.89	2.59
Forecast-MAE [ 6 ]	✗	1.30	3.33	0.19	0.69	1.55	2.24
SRefiner [ 45 ]	✗	1.48	3.85	0.19	0.68	1.52	2.21
RealMotion [ 36 ]	✓	1.14	2.87	0.18	0.62	1.32	2.01
DeMo [ 50 ]	✓	1.12	2.78	0.16	0.58	1.24	1.93
SHARP (Ours)	✓	1.03	2.53	0.15	0.56	1.15	1.80

Table 2: Multi-agent benchmark on the AV2 test set. For all metrics lower values indicate better performance, table sorted in descending order by avgBrierMinFDE 6 . For all methods we report results without model ensembling.

Method	Streaming	MR 6	mADE 6	mFDE 6	b-mFDE 6
SIMPL [ 51 ]	✗	0.19	0.72	1.43	2.05
HPTR [ 52 ]	✗	0.19	0.73	1.43	2.03
Forecast-MAE [ 6 ]	✗	0.17	0.71	1.39	2.03
MTR [ 34 ]	✗	0.15	0.73	1.44	1.98
EMP-D [ 30 ]	✗	0.17	0.71	1.37	1.98
GANet [ 40 ]	✗	0.17	0.72	1.34	1.96

Table 3: Single-agent benchmark on the AV2 test set. For all metrics lower values indicate better, table sorted in descending order by brier-minFDE 6 . We report results for published works on the official leaderboard without model ensembling.

Dataset	$T_{\text{cl}}$	TCF	IA	DT	mADE 6	mFDE 6	b-mFDE 6
	5 s	✗	✗	✗	0.74	1.28	1.91
		✓	✗	✗	0.64	1.22	1.84
		✓	✓	✗	0.63	1.19	1.81
AV2		✓	✓	✓	0.64	1.20	1.82
Val	1 s	✗	✗	✗	1.18	2.57	3.28
		✓	✗	✗	1.09	2.49	3.18

Table 4: Ablation study on the impact of our target-centric features (TCF), Dual Training (DT) and Instance-Aware (IA) Context Streaming using varying context lengths (

T_{\text{cl}}

). For each dataset, we compare full-history and partial-history settings. All predictions adhere to the standard evaluation protocol:

t_{p}

=2\,s

T_{f}

=3\,s

for AV1 and

t_{p}

=5\,s

T_{f}

=6\,s

for AV2.

Method	Streaming	mFDE 1	MR 5	mADE 5
THOMAS [ 16 ]	✗	6.71	0.55	1.33
PGP [ 9 ]	✗	7.17	0.52	1.27
DeMo [ 50 ]	✗	6.60	0.43	1.22
MacFormer [ 13 ]	✗	7.50	0.57	1.21
LAFormer [ 23 ]	✗	6.95	0.48	1.19
Q-EANet [ 5 ]	✗	6.77	0.48	1.18

Table 5: Results on the nuScenes prediction challenge test set. Following the evaluation protocol, we set the model output to

K=5

. DeMo [ 50 ] reports nuScenes results without streaming.

Dataset	Input Steps	$T_{\text{cl}}$	Method	mADE 6	mFDE 6
Argoverse 1	5	0.5 s	POP-H [ 41 ]	0.74	1.10
			TSD-H [ 57 ]	0.72	1.04
			SHARP (Ours)	0.61	0.96
	10	1 s	HiVT-64-FLN [ 46 ]	0.81	1.25
			POP-H [ 41 ]	0.70	1.06
			TSD-H [ 57 ]	0.67	0.98

Table 6: Comparison to the state-of-the-art varying-length models [ 46 , 41 , 57 ] , following their evaluation protocol.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Table 7: Evaluation of SHARP on the AV2 validation set performed across different context lengths

T_{\text{cl}}

. Within each group, the prediction time step

t_{p}

and forecasting horizon

T_{f}

are held constant, while the observation length provided to the model is varied. We evaluate predictions at the standard

t_{p}

=5\,s

, as well as at

t_{p}

=7\,s

and

t_{p}

=8\,s

, to study the impact of longer observation lengths on forecasting performance.

Noise Type	Noise Ratio	mADE 6	mFDE 6	b-mFDE 6
Randomized	80%	0.65	1.23	1.85
	40%	0.65	1.22	1.84
	20%	0.64	1.21	1.83
Set to Zero (no match)	80%	0.65	1.22	1.84
	40%	0.64	1.20	1.83
	20%	0.64	1.20	1.82

Appendix

Table 8: We evaluate the robustness of our instance-aware context streamer under perturbed instance masks . In the first group, we partially randomize the masks, introducing incorrect correspondences and removing correct ones, resulting in both false positives and false negatives. In the second group, we remove associations by partially setting mask values to zero, producing false negatives while leaving other annotations intact.

Table 9: Evaluation of SHARP in the multi-agent setting on the AV2 validation set at varying prediction time steps

t_{p}

and context lengths

T_{\text{cl}}

. The evaluation scope is limited to our method due to missing multi-agent implementation details and code bases for related work [ 36 , 50 ] .

Endpoint Noise (meter)	mADE 6	mFDE 6	b-mFDE 6
$\mathcal{U}(-5,5)$	0.65	1.23	1.85
$\mathcal{N}(0,3)$	0.65	1.23	1.85
$\mathcal{U}(-3,3)$	0.64	1.21	1.83
$\mathcal{N}(0,1)$	0.61	1.20	1.82
$\mathcal{U}(-1,1)$	0.64	1.20	1.83
Clean Data	0.63	1.19	1.81

Appendix

Table 10: We evaluate the robustness of our target-centric features under perturbations of the predicted endpoints used for the token selection. Gaussian (

\mathcal{N}

) and uniform (

\mathcal{U}

) noise are applied to simulate degradation of predictions during streaming. This setup assesses how our model handles such errors and its ability to recover from inaccurate prior predictions.

Method	Streaming	MR 6	mADE 6	mFDE 6
mmTransformer [ 24 ]	✗	0.11	0.71	1.15
FRM [ 29 ]	✗	-	0.68	0.99
ADAPT [ 1 ]	✗	0.08	0.67	0.95
SIMPL [ 51 ]	✗	0.08	0.66	0.95
HiVT [ 55 ]	✗	0.09	0.66	0.96
R-Pred [ 7 ]	✗	0.09	0.66	0.95

Appendix

Table 11: Results on the Argoverse 1 prediction challenge validation set .

Table 12: We compare our SHARP to DeMo [ 50 ] using short context lengths

T_{\text{cl}}

of 1 s on the AV2 single-agent validation set.

Table 13: Evaluation of SHARP on extend prediction horizons using the AV2 single-agent validation set. Predictions of the agent’s final position at the end of each scenario are made at varying prediction times

t_{p}

. Even the initial predictions over long horizons provide reasonable estimates. As the available context

T_{\text{cl}}

increases and the prediction horizon

T_{f}

shortens, the forecasts naturally become more accurate.

$T_{\text{cl}}$	Loss $\mathcal{L}$ Configuration	mADE 6	mFDE 6	b-mFDE 6
	$2\cdot\mathcal{L}_{\text{stream}}+1\cdot\mathcal{L}_{\text{chunk}}$	0.66	1.23	1.85
5 s	$1\cdot\mathcal{L}_{\text{stream}}+2\cdot\mathcal{L}_{\text{chunk}}$	0.66	1.24	1.87
	$1\cdot\mathcal{L}_{\text{stream}}+1\cdot\mathcal{L}_{\text{chunk}}$	0.66	1.24	1.86
	$2\cdot\mathcal{L}_{\text{stream}}+1\cdot\mathcal{L}_{\text{chunk}}$	0.97	2.14	2.82
1 s	$1\cdot\mathcal{L}_{\text{stream}}+2\cdot\mathcal{L}_{\text{chunk}}$	0.87	1.84	2.52
	$1\cdot\mathcal{L}_{\text{stream}}+1\cdot\mathcal{L}_{\text{chunk}}$	0.94	2.03	2.70

Appendix

Table 14: We evaluate training SHARP using a weighted dual loss formulation , where the weights allow us to emphasize either long-term or short-term context performance. In our final model, we omit these weights, resulting in equal contributions from both loss terms. Note that this analysis is based on a preliminary experiment conducted with a reduced training schedule.

Table 15: Ablation study on AV2 for long input horizons using our instance-aware context streaming

f_{\text{IA}}

module compared to the standard context relay streaming module proposed by [ 36 ] .

$T_{h}$	Train Pass at $t_{p}$	IA	mADE 6	mFDE 6	b-mFDE 6
3 s	{3, 4, 5} s	✗	0.66	1.23	1.84
2 s	2 .. 5 s	✗	0.66	1.24	1.85
1 s	1 .. 5 s	✗	0.64	1.22	1.85
1 s	1 .. 8 s	✗	0.65	1.22	1.84
2 s	2 .. 5 s	✓	0.66	1.22	1.83
1 s	1 .. 5 s	✓	0.65	1.21	1.83

Appendix

Table 16: Ablation study on training our approach with varying input window sizes (

T_{h}

) and different numbers of back-propagation passes per scenario, with and without our instance-aware context streaming (IA). The displacement during consecutive train passes is 1 s for all experiments. The first row reports training results for our SHARP architecture using the baseline streaming setup of [ 36 ] . Reducing the input window size

T_{h}

enables performing more back-propagation passes over each scenario. However, existing streaming mechanisms fail to fully exploit this potential, which is effectively unlocked by our instance-aware context streamer. All evaluations follow the standard protocol on the AV2 validation set. Experiments are conducted without dual training to neglect confounding influences and isolate the effect of instance-aware context streaming.

Method	Streaming	$B=1$	$B=16$	$B=32$	$B=64$	$B=128$	Model Parameters	AV2 Test Set brier-minFDE 6
QCNet [ 54 ]	✗	141 ms	303 ms	565 ms	1092 ms	-	7.7M	1.91
RealMotion [ 36 ]	✓	19 ms	44 ms	0 79 ms	140 ms	273 ms	2.9M	1.89
DeMo [ 50 ]	✓	0 25 ms	35 ms	0 49 ms	00 88 ms	180 ms	5.9M	1.84
SHARP (Ours)	✓	0 21 ms	33 ms	0 35 ms	00 57 ms	105 ms	4.6M	1.83

Appendix

Table 17: Latency analysis for batches of size

B

when performing single-agent prediction , evaluated on the Argoverse 2 validation set with an NVIDIA RTX 3090 GPU. Larger batch sizes

B

correspond to more practically-relevant latency estimates, since real-world deployments must process multiple surrounding traffic agents, i.e .

B>1

, simultaneously. For QCNet we run out of GPU memory when testing

B=128

Dataset	RoI	Agents		Lanes			Input Window
Dataset	Radius	Features	Types	Features	Types	$P_{l}$	Length $T_{h}$
AV1 [ 4 ]	150 m	$xy$ -pos	Vehicle	$xy$ -pos	Standard Lane	10	0.5 s
AV2 [ 44 ]	150 m	$xy$ -pos, $xy$ -vel	Veh./Pedest./Cycl./Others	$xy$ -pos	Veh./Bike/Bus	20	1 s
nuScenes [ 2 ]	100 m	$xy$ -pos	Car/Pedest./Cycl./Other Veh./Motorcycl./Others	$xy$ -pos	Standard/Intersection	20	0.5 s

Appendix

Table 18: Data preprocessing and sampling parameters for loading data from Argoverse 1 (AV1), Argoverse 2 (AV2), and nuScenes. RoI radius denotes the region of interest radius used to sample surrounding agent and map data.

Figure 4: Comparison of motion forecasting processing schemes: standard snapshot-based forecasting, the streaming-based forecasting used by RealMotion [ 36 ] and Demo [ 50 ] (long, overlapping, observation windows) and our proposed approach (compact, subsequent, observation windows).

Figure 5: Details for evaluating related work on the evolving scene setting (main paper Table 1). The first example shows the standard execution on the AV2 benchmark. In the second example, we increase the input context by simply executing another streaming step. In the last example, we test the models with fewer context by evaluating after two streaming steps.

Figure 6: Qualitative single-agent prediction results of our SHARP on scenarios from the Argoverse 2 validation set. We visualize the predictions of our streaming-based method at

t_{p}

\in\{3,4,5\}

s. The visualizations also show ground truth future , historical agent observations , and surrounding agents . For comparison, the right column shows the predictions of DeMo [ 50 ] at the AV2 standard prediction time step

t_{p}

=5 s.

Figure 7: Failure cases of our SHARP on scenarios from the Argoverse 2 validation set. We visualize the predictions of our streaming-based method at

t_{p}

\in\{3,4,5\}

t_{p}

=5 s. In the first scenario, a vehicle is waiting at an intersection but will initiate a right turn later – an action that is not anticipated from the current observations. In the second scenario, a vehicle turns into a driveway that is not represented in the map data. In the final scenario, a vehicle stops in the future for a reason that is not captured in the current perception data.

Trajectory Prediction

Trajectory prediction forecasts how people, vehicles, robots, and other dynamic systems will move from observed histories and context.

See all 186 papers

Mar 2, 2026

Streaming Real-Time Trajectory Prediction Using Endpoint-Aware Modeling

Future trajectories of neighboring traffic agents have a significant influence on the path planning and decision-making of autonomous vehicles. While trajectory forecasting is a well-studied field, research mainly focuses on snapshot-based prediction, where each scenario is treated independently of its global temporal context. However, real-world autonomous driving systems need to operate in a continuous setting, requiring real-time processing of data streams with low latency and consistent predictions over successive timesteps. We leverage this continuous setting to propose a lightweight yet highly accurate streaming-based trajectory forecasting approach. We integrate valuable information from previous predictions with a novel endpoint-aware modeling scheme. Our temporal context propagation uses the trajectory endpoints of the previous forecasts as anchors to extract targeted scenario context encodings. Our approach efficiently guides its scene encoder to extract highly relevant context information without needing refinement iterations or segment-wise decoding. Our experiments highlight that our approach effectively relays information across consecutive timesteps. Unlike methods using multi-stage refinement processing, our approach significantly reduces inference latency, making it well-suited for real-world deployment. We achieve state-of-the-art streaming trajectory prediction results on the Argoverse~2 multi-agent and single-agent benchmarks, while requiring substantially fewer resources.

Alexander Prutsch, David Schinagl, Horst Possegger

Institute of Visual Computing, Graz University of Technology

Trajectory Prediction Multi-Agent Trajectory Prediction Autonomous Driving

Ask PDF

Mar 10, 2026

Class-Incremental Motion Forecasting

Motion forecasting enables autonomous vehicles to anticipate scene evolution by predicting the future trajectories of dynamic agents. However, existing approaches typically assume a closed-world setting with a fixed object taxonomy and access to high-quality perception, limiting their applicability in the real world where perception is imperfect, and new object classes may emerge over time. In this work, we introduce class-incremental motion forecasting, a novel setting in which new object classes are sequentially introduced over time and future object trajectories are predicted directly from camera images. We propose the first end-to-end framework for this setting, which adapts to newly introduced classes while mitigating catastrophic forgetting of previously learned ones. Our method generates motion forecasting pseudo-labels for known classes and matches them with 2D instance masks from an open-vocabulary segmentation model. This 3D-to-2D keypoint voting mechanism filters inconsistent and overconfident predictions, while a query feature variance-based replay strategy samples informative past sequences to preserve prior knowledge. Extensive evaluations on nuScenes and Argoverse 2 show that our approach successfully preserves performance on known classes while effectively adapting to novel ones. We further demonstrate zero-shot transfer to real-world driving and show that the framework extends naturally to open- and closed-loop end-to-end class-incremental planning on nuScenes and NeuroNCAP. Code and models will be made publicly available at https://omen.cs.uni-freiburg.de.

Nicolas Schischka, Nikhil Gosala, B Ravi Kiran +2

Department of Computer Science, University of Freiburg, Germany. · Qualcomm SARL France. · Automated Driving, Qualcomm Technologies, Inc.

Class-Incremental Learning Autonomous Driving Perception

Ask PDF

Mar 17, 2026

PanguMotion: Continuous Driving Motion Forecasting with Pangu Transformers

Motion forecasting is a core task in autonomous driving systems, aiming to accurately predict the future trajectories of surrounding agents to ensure driving safety. Existing methods typically process discrete driving scenes independently, neglecting the temporal continuity and historical context correlations inherent in real-world driving environments. This paper proposes PanguMotion, a motion forecasting framework for continuous driving scenarios that integrates Transformer blocks from the Pangu-1B large language model as feature enhancement modules into autonomous driving motion prediction architectures. We conduct experiments on the Argoverse 2 datasets processed by the RealMotion data reorganization strategy, transforming each independent scene into a continuous sequence to mimic real-world driving scenarios.

Quanhao Ren, Yicheng Li, Nan Song

School of Data Science, Fudan University

Trajectory Prediction Vehicle Trajectory Prediction Spatiotemporal Forecasting

Ask PDF

Abstract

Trajectory Prediction

Explore similar work

Streaming Real-Time Trajectory Prediction Using Endpoint-Aware Modeling

Class-Incremental Motion Forecasting

PanguMotion: Continuous Driving Motion Forecasting with Pangu Transformers