Streaming Real-Time Trajectory Prediction Using Endpoint-Aware Modeling

Alexander Prutsch; David Schinagl; Horst Possegger

cs.CVMar 2, 2026

Streaming Real-Time Trajectory Prediction Using Endpoint-Aware Modeling

Authors: Alexander Prutsch, David Schinagl, Horst Possegger

Organizations: Institute of Visual Computing, Graz University of Technology

Abstract

Future trajectories of neighboring traffic agents have a significant influence on the path planning and decision-making of autonomous vehicles. While trajectory forecasting is a well-studied field, research mainly focuses on snapshot-based prediction, where each scenario is treated independently of its global temporal context. However, real-world autonomous driving systems need to operate in a continuous setting, requiring real-time processing of data streams with low latency and consistent predictions over successive timesteps. We leverage this continuous setting to propose a lightweight yet highly accurate streaming-based trajectory forecasting approach. We integrate valuable information from previous predictions with a novel endpoint-aware modeling scheme. Our temporal context propagation uses the trajectory endpoints of the previous forecasts as anchors to extract targeted scenario context encodings. Our approach efficiently guides its scene encoder to extract highly relevant context information without needing refinement iterations or segment-wise decoding. Our experiments highlight that our approach effectively relays information across consecutive timesteps. Unlike methods using multi-stage refinement processing, our approach significantly reduces inference latency, making it well-suited for real-world deployment. We achieve state-of-the-art streaming trajectory prediction results on the Argoverse~2 multi-agent and single-agent benchmarks, while requiring substantially fewer resources.

Figures & tables

Figure 1: Traffic scenes are constantly changing, which is often neglected in trajectory prediction methods operating on snapshots rather than in a continuous setting. For example, for the trajectory prediction of the turning process of the focal agent (green), a possible interaction with a pedestrian (red) near the endpoint of the previous estimation (at time

t-1

) is important context that needs to be considered in the current prediction (

t

). We model this temporal context propagation with a novel endpoint streaming mechanism, achieving accurate predictions at minimal latencies.

Figure 2: Overview of our streaming-based trajectory prediction architecture SEAM. We assume that for frame

t-1

no previous prediction exists for our focal agent. Thus, only a standard trajectory prediction model pass is executed. In the next frame

t

, we incorporate the endpoints from the previous predictions to aggregate target-centric context information. We encode them using a second encoder path and provide it as additional input to our novel dual-context decoder.

Figure 3: Architecture of our novel dual-context attention decoder, leveraging both agent-centric

S^{t}

and target-centric

T^{t}

features.

Method	Streaming	minADE 1	minFDE 1	MR 6	minADE 6	minFDE 6	brier-minFDE 6
SIMPL [ 40 ]	✗	2.03	5.50	0.19	0.72	1.43	2.05
HPTR [ 41 ]	✗	1.84	4.61	0.19	0.73	1.43	2.03
Forecast-MAE [ 4 ]	✗	1.74	4.36	0.17	0.71	1.39	2.03
MTR [ 28 ]	✗	1.74	4.39	0.15	0.73	1.44	1.98
EMP-D [ 25 ]	✗	1.75	4.35	0.17	0.71	1.37	1.98
GANet [ 34 ]	✗	1.77	4.48	0.17	0.72	1.34	1.96

Table 1: Single-agent trajectory prediction results on the Argoverse 2 test set. For all metrics lower indicate better, table sorted in descending order by brier-minFDE 6 . We report results for published works on the official leaderboard without model ensembling. The streaming approach of Pang et al . [ 24 ] is not included (their custom dataset prevents direct comparison). *For RealMotion [ 30 ] we report the results for their official checkpoint, as well as the results from their paper, which is trained on the train and validation set.

Method	Streaming	avgMinADE 1	avgMinFDE 1	actorMR 6	avgMinADE 6	avgMinFDE 6	avgBrierMinFDE 6
FJMP [ 27 ]	✗	1.52	4.00	0.23	0.81	1.89	2.59
Forecast-MAE [ 4 ]	✗	1.30	3.33	0.19	0.69	1.55	2.24
RealMotion [ 30 ]	✓	1.14	2.97	0.18	0.62	1.32	2.01
DeMo [ 39 ]	✓	1.12	2.78	0.16	0.58	1.24	1.93
SEAM (Ours)	✓	1.05	2.57	0.16	0.62	1.21	1.85
QCNeXt [ 44 ]	✗	1.03	2.55	0.14	0.54	1.13	1.79

Table 2: Multi-agent trajectory prediction results on the Argoverse 2 test set. For all metrics lower values indicate better performance, table sorted in descending order by avgBrierMinFDE 6 . For all methods we report results without model ensembling.

Method	Streaming	Latency ( $B=1$ )		Latency ( $B=32$ )		Latency ( $B=64$ )		Model	AV2 Test Set
		Offline	Online	Offline	Online	Offline	Online	Parameters	brier-minFDE 6
SmartRefine [ 42 ]	✗	$>$ 118 ms*		$>$ 680 ms*		-		8.0M	1.86
QCNet [ 43 ]	✗	$>$ 118 ms		$>$ 680 ms		-		7.7M	1.91
RealMotion [ 30 ]	✓	0 86 ms	24 ms	247 ms	88 ms	436 ms	150 ms	2.9M	1.93
DeMo [ 39 ]	✓	139 ms	39 ms	275 ms	83 ms	466 ms	142 ms	5.9M	1.84
SEAM (Ours)	✓	0 91 ms	28 ms	140 ms	50 ms	246 ms	0 74 ms	4.6M	1.84

Table 3: Latency analysis for predicting

B

Argoverse 2 scenarios in the single-agent setting using a NVIDIA V100 GPU. Online (for streaming-based methods) is the time required to process a single sliding window; Offline is the time required to process the full stream (to generate the benchmark predictions). For streaming methods the online latency is relevant for practical applications. *SmartRefine does not provide an implementation for AV2—since its main results add refinement iterations to QCNet, latency is higher than for QCNet.

Figure 4: Qualitative results on two Argoverse 2 scenarios. We show the predictions of our streaming-based method at

t\in\{3,4,5\}

s. The visualizations also show ground truth future , agent histories , and neighboring agents . The right column shows the final predictions at

t=5

s for RealMotion [ 30 ] in the streaming-based setting. Top row: the focal agent is approaching an intersection where other traffic is currently passing by, making it difficult to identify possible movement. Bottom row: a pedestrian crosses the street at an intersection to the right. Our approach correctly predicts that the vehicle can either continue straight or turn right, either waiting before the crosswalk or proceeding directly if the pedestrian has already left the crosswalk.

Dataset	$h$	Stream	CS+TR [ 30 ]	EAM	mADE 6	mFDE 6	b-mFDE 6
	2s	✗	✗	✗	0.64	1.04	1.67
AV1	1s	✓	✓	✗	0.62	0.98	1.62
Val	1s	✓	✗	✓	0.60	0.95	1.55
	1s	✓	✓	✓	0.59	0.93	1.52
	5s	✗	✗	✗	0.77	1.39	2.00
AV2	3s	✓	✓	✗	0.75	1.29	1.93

Table 4: Comparison of different information forwarding mechanisms for streaming trajectory prediction. We compare the integration of context stream (CS) and trajectory relay (TR) introduced by [ 30 ] to our endpoint-aware modeling (EAM). The first line shows the traditional snapshot-based use. We highlight best values.

Dataset	$r$	mADE 6	mFDE 6	b-mFDE 6
	10m	0.68	1.27	1.89
AV2	15m	0.67	1.26	1.87
Val	30m	0.66	1.25	1.85
	45m	0.66	1.25	1.85

Table 5: Comparison of different radii

r

to select context elements for our target-centric region. We highlight best values.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Comparison between snapshot-based and streaming-based trajectory prediction paradigms. Both approaches operate on the same benchmark input data, ensuring a fair comparison without additional data. In the streaming paradigm, past observations are processed using a sliding window, closely resembling practical deployment conditions. To model the challenges of real-world operation information relay mechanisms are established in streaming processing. This enables more consistent and temporally coherent predictions compared to snapshot-based processing, which handles each frame independently. In this example, predictions from the third pass of the streaming model can be directly evaluated against those from the snapshot-based approach.

Figure 6: Modeling of target-centric coordinate system ( orange ) using the prediction ( green ) from the previous time frame. We also show the target context radius ( orange ) and the positional embedding ( blue ) used to model the relationship to the agent-centric origin (black).

Dataset	# Blocks	mADE 6	mFDE 6	b-mFDE 6
	1	0.66	1.26	1.86
AV2 Val	2	0.66	1.25	1.85
	3	0.66	1.26	1.85

Appendix

Table 6: Ablation study for different number of attention blocks in our target encoder

f_{T}

Method	minADE 1	minFDE 1	MR 6	minADE 6	minFDE 6	brier-minFDE 6
RealMotion [ 30 ]	1.65	4.10	0.16	0.67	1.30	1.94
DeMo [ 39 ]	1.48	3.73	0.13	0.61	1.19	1.86
SEAM (Ours)	1.60	3.96	0.15	0.66	1.25	1.85

Appendix

Table 7: Comparison of streaming-based methods for single-agent trajectory prediction on the Argoverse 2 validation set. For all models we report results without model ensembling.

Endpoint Noise	mADE 6	mFDE 6	b-mFDE 6
$\mathcal{N}(0,5)$	0.67	1.28	1.88
$\mathcal{U}(-5,5)$	0.67	1.27	1.87
$\mathcal{N}(0,3)$	0.67	1.26	1.87
$\mathcal{U}(-3,3)$	0.67	1.26	1.86
$\mathcal{N}(0,1)$	0.67	1.25	1.85
$\mathcal{U}(-1,1)$	0.67	1.25	1.85

Appendix

Table 8: Robustness of our endpoint-aware modeling on the AV2 [ 37 ] validation set by perturbing the prediction endpoints at

t\in\{3,4\}

s. These endpoints, which define the anchors for extracting our target-centric features, are modified using additive uniform noise (

\mathcal{U}

) or Gaussian noise (

\mathcal{N}

). The shown results correspond to the prediction errors at

t=5

Panel of Figure 7 — Figure 7: Density plots of prediction errors. Both plots compare the first prediction done at $t=3$ s (x-axis) versus the final prediction done at at $t=5$ s (y-axis). The top plot shows that even when the initial error is large, the model can recover and correct its prediction. The bottom plot provides a zoomed-in view near the origin. The higher density of points below the $x=y$ diagonal (dashed line) indicates that predictions generally improve over time.

Method	Latency ( $B=1$ )		Latency ( $B=32$ )		Latency ( $B=64$ )
	Offline	Online	Offline	Online	Offline	Online
RealMotion [ 30 ]	51 ms	13 ms	255 ms	85 ms	512 ms	172 ms
DeMo [ 39 ]	73 ms	18 ms	223 ms	71 ms	433 ms	143 ms
SEAM (Ours)	50 ms	13 ms	95 ms	39 ms	185 ms	0 65 ms

Appendix

Table 9: Latency analysis for predicting

B

Argoverse 2 single-agent scenarios using one NVIDIA A10 GPU. We compare streaming-based methods and report the offline and online inference latency. The online latency is relevant for practical application.

Model	Training Data	Global Consistency Module	actorMR 6	avgMinADE 6	avgMinFDE 6	avgBrierMinFDE 6
RealMotion [ 30 ]	Single-Agent Data	✗	0.727	1.413	4.060	4.716
SEAM (Ours)	Single-Agent Data	✗	0.693	1.324	3.643	4.256
SEAM (Ours)	Finetuned on Multi-Agent Data	✗	0.228	0.720	1.619	2.126
SEAM (Ours)	Finetuned on Multi-Agent Data	✓	0.155	0.600	1.180	1.814

Appendix

Table 10: Ablation study on adapting single-agent methods to the AV2 multi-agent validation set. The first two rows show that, without any model adaptations or finetuning, both our approach and related work [ 30 ] perform poorly in the multi-agent setting. Finetuning on the multi-agent dataset significantly improves performance, and incorporating a global consistency module for scene-level scoring yields the best results.

Method	Fluctuation
RealMotion [ 30 ]	0.347
SEAM (Ours)	0.341

Appendix

Table 11: We compare the trajectory fluctuation of our model to RealMotion [ 30 ] . The fluctuation metric defined by [ 24 ] gives an indication on the consistency of trajectories across multiple timesteps.

Figure 8: Qualitative results of our approach on scenarios from the Argoverse 2 validation set. We show the predictions of our streaming-based method at

t\in\{3,4,5\}

s. The visualizations also show ground truth future , agent histories , and neighboring agents . The right column shows the final predictions at

t=5

s for using RealMotion [ 30 ] in the streaming setting.

Figure 9: Qualitative results of our approach on scenarios from the Argoverse 2 validation set. We show the predictions of our streaming-based method at

t\in\{3,4,5\}

s. The visualizations also show ground truth future , agent histories , and neighboring agents . The right column shows the final predictions at

t=5

s for using RealMotion [ 30 ] in the streaming setting.

Figure 10: Failure cases of our approach on scenarios from the Argoverse 2 validation set. We show the predictions of our streaming-based method at

t\in\{3,4,5\}

s. The visualizations also show ground truth future , agent histories , and neighboring agents . The right column shows the final predictions at

t=5

s for using RealMotion [ 30 ] in the streaming setting. In the first scenario, a vehicle stops before an intersection that is fairly far away for a currently unknown reason. In the second scenario, a vehicle stops for a reason that is not detected at the moment. In the third scenario, the vehicle begins turning into a driveway that is not modeled in the lane data. The fourth, fifth, and sixth scenarios depict vehicles waiting at an intersection where their future path is unclear.

Trajectory Prediction

Trajectory prediction forecasts how people, vehicles, robots, and other dynamic systems will move from observed histories and context.

See all 187 papers

SHARP: Short-Window Streaming for Accurate and Robust Prediction in Motion Forecasting
Mar 30, 2026Alexander Prutsch, Christian Fruhwirth-Reisinger, David Schinagl +1Trajectory Prediction Vehicle Trajectory Prediction
Institute of Visual Computing, Graz University of Technology
MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
Apr 11, 2026Wenchang Duan, Zhenguo Gao, Jinguo Xian +1Model Compression Multi-Agent Trajectory Prediction
School of Mathematical Sciences, Shanghai Jiao Tong University, Shanghai 200240, China · Bio-X Institutes, Key Laboratory for the Genetics of Developmental and Neuropsychiatric Disorders, Shanghai Jiao Tong University, Shanghai 200030, China · Shanghai Key Laboratory of Psychotic Disorders, and Brain Science and Technology Research Center, Shanghai Jiao Tong University, Shanghai 200030, China
PanguMotion: Continuous Driving Motion Forecasting with Pangu Transformers
Mar 17, 2026Quanhao Ren, Yicheng Li, Nan SongTrajectory Prediction Vehicle Trajectory Prediction
School of Data Science, Fudan University

Mar 30, 2026J/K move · Enter open · S save

SHARP: Short-Window Streaming for Accurate and Robust Prediction in Motion Forecasting

Alexander Prutsch, Christian Fruhwirth-Reisinger, David Schinagl, Horst Possegger

Institute of Visual Computing, Graz University of Technology

Trajectory Prediction Vehicle Trajectory Prediction Multi-Agent Trajectory Prediction

Open paper AskPDF arXiv

In dynamic traffic environments, motion forecasting models must be able to accurately estimate future trajectories continuously. Streaming-based methods are a promising solution, but despite recent advances, their performance often degrades when exposed to heterogeneous observation lengths. To address this, we propose a novel streaming-based motion forecasting framework that explicitly focuses on evolving scenes. Our method incrementally processes incoming observation windows and leverages an instance-aware context streaming to maintain and update latent agent representations across inference steps. A dual training objective further enables consistent forecasting accuracy across diverse observation horizons. Extensive experiments on Argoverse 2, nuScenes, and Argoverse 1 demonstrate the robustness of our approach under evolving scene conditions and also on the single-agent benchmarks. Our model achieves state-of-the-art performance in streaming inference on the Argoverse 2 multi-agent benchmark, while maintaining minimal latency, highlighting its suitability for real-world deployment.

Abstract

Trajectory Prediction

Explore similar work

SHARP: Short-Window Streaming for Accurate and Robust Prediction in Motion Forecasting

MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction

PanguMotion: Continuous Driving Motion Forecasting with Pangu Transformers