Recover to Predict: Progressive Retrospective Learning for Variable-Length Trajectory Prediction

Hao Zhou; Lu Qi; Jason Li; Jie Zhang; Yi Liu; Xu Yang; Mingyu Fan; Fei Luo

cs.ROMar 11, 2026

Recover to Predict: Progressive Retrospective Learning for Variable-Length Trajectory Prediction

Authors: Hao Zhou, Lu Qi, Jason Li, Jie Zhang, Yi Liu, Xu Yang, Mingyu Fan, Fei Luo

Organizations: Great Bay University · Tsinghua SIGS · Wuhan University · NTU ·

Abstract

Trajectory prediction is critical for autonomous driving, enabling safe and efficient planning in dense, dynamic traffic. Most existing methods optimize prediction accuracy under fixed-length observations. However, real-world driving often yields variable-length, incomplete observations, posing a challenge to these methods. A common strategy is to directly map features from incomplete observations to those from complete ones. This one-shot mapping, however, struggles to learn accurate representations for short trajectories due to significant information gaps. To address this issue, we propose a Progressive Retrospective Framework (PRF), which gradually aligns features from incomplete observations with those from complete ones via a cascade of retrospective units. Each unit consists of a Retrospective Distillation Module (RDM) and a Retrospective Prediction Module (RPM), where RDM distills features and RPM recovers previous timesteps using the distilled features. Moreover, we propose a Rolling-Start Training Strategy (RSTS) that enhances data efficiency during PRF training. PRF is plug-and-play with existing methods. Extensive experiments on datasets Argoverse 2 and Argoverse 1 demonstrate the effectiveness of PRF. Code is available at https://github.com/zhouhao94/PRF.

Figures & tables

Panel of Figure 1 — Figure 1: Fig. 1(a) and Fig. 1(b) display two common scenarios that yield variable-length, incomplete trajectories. Fig. 1(c) and Fig. 1(d) respectively present the mADE 6 and mFDE 6 results for the original DeMo [ 47 ] , DeMo with Isolated Training (DeMo-IT), and DeMo with PRF (DeMo-PRF) under varying observation lengths.

Figure 2: Overview of the PRF. A cascade of retrospective units progressively distills features of varying-length inputs, aligning them with those from complete ones to improve prediction performance. Each unit includes a Retrospective Distillation Module (RDM) that distills features to longer observations and a Retrospective Prediction Module (RPM) that recovers omitted history from the distilled features.

Figure 3: Illustration of the (a) RDM and (b) RPM. RDM employs a residual-based distillation strategy, featuring a logit branch that generates a gating vector and a residual branch that learns from the omitted history. RPM employs a decoupled query strategy, utilizing mode queries for multimodal trajectory proposals and state queries for trajectory refinement, with the proposals serving as anchors.

Method	Argoverse 2 (mADE 6 /mFDE 6 )						Argoverse 1 (mADE 6 /mFDE 6 )
Method	10	20	30	40	50	Avg- $\Delta$ 50	5	10	15	20	Avg- $\Delta$ 20
QCNet-Ori	0.900/1.526	0.777/1.338	0.752/1.296	0.725/1.252	0.726/1.253	0.063/0.100	0.807/1.172	0.769/1.139	0.751/1.104	0.709/1.040	0.067/0.098
QCNet-IT	0.741/1.293	0.734/1.279	0.730/1.276	0.726/1.267	0.726/1.253	0.007 /0.034	0.747/1.083	0.721/1.058	0.714/1.043	0.709/1.040	0.018/0.021
QCNet-DTO	0.768/1.315	0.739/1.270	0.735/1.269	0.732/1.261	0.731/1.258	0.012/0.021	0.764/1.102	0.722/1.057	0.709/1.046	0.702/1.034	0.030/0.034
QCNet-FLN	0.752/1.274	0.735/1.253	0.731/1.243	0.729/1.231	0.724/1.231	0.013/0.019	0.760/1.088	0.719/1.041	0.710/1.027	0.699/1.017	0.031/0.035
QCNet-LaKD	0.739/1.259	0.725 /1.235	0.725 /1.232	0.721/1.227	0.718/1.219	0.010 /0.019	0.737/1.057	0.708 /1.044	0.699/1.034	0.696 /1.027	0.019/0.018

Table 1: Variable-length trajectory prediction comparison on Argoverse 2 (left) and Argoverse 1 (right) validation sets. For Argoverse 2, AVG–

\Delta

50 is the average difference between {10, 20, 30, 40} and 50. For Argoverse 1, AVG–

\Delta

20 is the average difference between {5, 10, 15} and 20. Best results are in bold and second best are underlined .

Panel of Figure 4 — Figure 4: Qualitative results on the Argoverse 2 validation set. Incomplete observations, predicted trajectories, and ground truth trajectories are shown in yellow, green, and pink, respectively. Our predictions align more closely with the ground truth than other methods.

Table 6

RDM	RPM	RSTS	mADE 6 /mFDE 6
RDM	RPM	RSTS	10	20	30	40	50
			0.876/1.455	0.769/1.337	0.756/1.286	0.726/1.252	0.725/1.256
✓			0.655/1.257	0.640/1.237	0.636/1.231	0.636/1.227	0.639/1.231
✓	✓		0.652/1.241	0.637/1.214	0.634/1.207	0.631/1.204	0.635/1.208
✓	✓	✓	0.617 / 1.183	0.603 / 1.155	0.598 / 1.143	0.599 / 1.145	0.596 / 1.142

Table 4: Ablation study of the core modules of our model on the Argoverse 2 validation set.

Table 8

Num	mADE 6 /mFDE 6
Num	10	20	30	40	50
GRU	0.662/1.286	0.644/1.256	0.640/1.245	0.639/1.242	0.640/1.245
Attn	0.653/1.261	0.639/1.235	0.635/1.229	0.634/1.225	0.634 /1.224
Mamba	0.652 / 1.241	0.637 / 1.214	0.634 / 1.207	0.631 / 1.204	0.635/ 1.208

Table 7: Ablation of the sequence modeling choices in RPM on the Argoverse 2 validation set.

[0,40]	[0,30]	[0,20]	mADE 6 /mFDE 6
[0,40]	[0,30]	[0,20]	10	20	30	40	50
			0.652/1.241	0.637/1.214	0.634/1.207	0.631/1.204	0.635/1.208
✓			0.631/1.211	0.618/1.189	0.613/1.178	0.615/1.185	0.613/1.182
✓	✓		0.624/1.201	0.608/1.174	0.606/1.167	0.606/1.167	0.606/1.165
✓	✓	✓	0.617 / 1.183	0.603 / 1.155	0.598 / 1.143	0.599 / 1.145	0.596 / 1.142

Table 8: Ablation study of data utilization in the RSTS on the Argoverse 2 validation set.

Strategy	mADE 6 /mFDE 6
Strategy	10	20	30	40	50
Direct	0.663/1.275	0.644/1.240	0.639/1.228	0.635/1.220	0.635/1.222
PRF (Our)	0.652 / 1.241	0.637 / 1.214	0.634 / 1.207	0.631 / 1.204	0.635/ 1.208

Table 9: Analysis of progressive distillation vs. direct distillation on the Argoverse 2 validation set. RTST is not used in training.

Panel of Figure 5 — Figure 5: t-SNE visualization of (a) direct and (b) progressive distillation strategies. Features distilled from 10 to 50 are shown in yellow, while features of the standard length 50 are shown in blue.

Length	10	20	30	40	50
Inference time (s)	0.268	0.236	0.203	0.172	0.140
FLOPs (G)	1.651	1.581	1.513	1.443	1.375

Table 10: Analysis of inference efficiency on the Argoverse 2 validation set. Results are measured with one multi-agent scenario per forward pass, using an NVIDIA GeForce RTX 4090 GPU.

Figure 6: Illustration of the RSTS on the Argoverse 2 dataset with a standard observation horizon of $T_{o}=50$ and a prediction horizon of $T_{f}=60$ . As the prediction start point shifts from 50 to 40, 30, and 20, additional training samples are generated to train the retrospective units and the decoder.

Panel of Figure 7 — Figure 7: More qualitative results on the Argoverse 2 validation set. Incomplete observations, predicted trajectories, and ground truth trajectories are shown in yellow, green, and pink, respectively. The absence of an observation trajectory indicates that the vehicle is stationary. Our predictions align more closely with the ground truth compared to other methods.

Panel of Figure 8 — Figure 8: Failure cases of DeMo-PRF on the Argoverse 2 validation set. The first and second rows visualize the predicted trajectories and ground-truth trajectories, respectively. Incomplete observations, predicted trajectories, and ground truth trajectories are shown in yellow, green, and pink, respectively. The absence of an observation trajectory indicates that the vehicle is stationary.

Figure 9: t-SNE visualization of features distilled by the progressive strategy. Green and orange points represent features extracted from trajectories with observation lengths of 10 and 50 timesteps, respectively. Red, purple, brown, and blue points correspond to features distilled from 10-step observations to those with 20, 30, 40, and 50 steps, which gradually shift from the manifold of 10-step observations toward that of 50-step observations.

Trajectory Prediction

Trajectory prediction forecasts how people, vehicles, robots, and other dynamic systems will move from observed histories and context.

See all 187 papers

TaPD: Temporal-adaptive Progressive Distillation for Observation-Adaptive Trajectory Forecasting in Autonomous Driving
Mar 6, 2026Mingyu Fan, Yi Liu, Hao Zhou +3Trajectory Prediction Multi-Agent Trajectory Prediction
College of Information and Intelligent Science, Donghua University, Shanghai 201620 China. · School of Computing and Information Technology, Great Bay Institute for Advanced Study/Great Bay University, Dongguan, China, and also with the Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China. · Chongqing Chang’an Technology Co., Ltd, Chongqing 401120 China. +1
SAIL: Scene-aware Adaptive Iterative Learning for Long-Tail Trajectory Prediction in Autonomous Vehicles
Apr 6, 2026Bin Rao, Haicheng Liao, Chengyue Wang +3Trajectory Prediction Vehicle Trajectory Prediction
State Key Laboratory of Internet of Things for Smart City and Department of Civil and Environmental Engineering, University of Macau, Macau SAR, China · State Key Laboratory of Internet of Things for Smart City and Department of Computer and Information Science, University of Macau, Macau SAR, China · Department of Automotive Engineering, Tsinghua University, Beijing, China +2
Streaming Real-Time Trajectory Prediction Using Endpoint-Aware Modeling
Mar 2, 2026Alexander Prutsch, David Schinagl, Horst PosseggerTrajectory Prediction Multi-Agent Trajectory Prediction
Institute of Visual Computing, Graz University of Technology

Mar 6, 2026J/K move · Enter open · S save

TaPD: Temporal-adaptive Progressive Distillation for Observation-Adaptive Trajectory Forecasting in Autonomous Driving

Mingyu Fan, Yi Liu, Hao Zhou, Deheng Qian+2

College of Information and Intelligent Science, Donghua University, Shanghai 201620 China. · School of Computing and Information Technology, Great Bay Institute for Advanced Study/Great Bay University, Dongguan, China, and also with the Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China. · Chongqing Chang’an Technology Co., Ltd, Chongqing 401120 China.+1

Trajectory Prediction Multi-Agent Trajectory Prediction Knowledge Distillation

Open paper AskPDF arXiv

Trajectory prediction is essential for autonomous driving, enabling vehicles to anticipate the motion of surrounding agents to support safe planning. However, most existing predictors assume fixed-length histories and suffer substantial performance degradation when observations are variable or extremely short in real-world settings (e.g., due to occlusion or a limited sensing range). We propose TaPD (Temporal-adaptive Progressive Distillation), a unified plug-and-play framework for observation-adaptive trajectory forecasting under variable history lengths. TaPD comprises two cooperative modules: an Observation-Adaptive Forecaster (OAF) for future prediction and a Temporal Backfilling Module (TBM) for explicit reconstruction of the past. OAF is built on progressive knowledge distillation (PKD), which transfers motion pattern knowledge from long-horizon "teachers" to short-horizon "students" via hierarchical feature regression, enabling short observations to recover richer motion context. We further introduce a cosine-annealed distillation weighting scheme to balance forecasting supervision and feature alignment, improving optimization stability and cross-length consistency. For extremely short histories where implicit alignment is insufficient, TBM backfills missing historical segments conditioned on scene evolution, producing context-rich trajectories that strengthen PKD and thereby improve OAF. We employ a decoupled pretrain-reconstruct-finetune protocol to preserve real-motion priors while adapting to backfilled inputs. Extensive experiments on Argoverse 1 and Argoverse 2 show that TaPD consistently outperforms strong baselines across all observation lengths, delivers especially large gains under very short inputs, and improves other predictors (e.g., HiVT) in a plug-and-play manner. Code will be available at https://github.com/zhouhao94/TaPD.

Abstract

Trajectory Prediction

Explore similar work

TaPD: Temporal-adaptive Progressive Distillation for Observation-Adaptive Trajectory Forecasting in Autonomous Driving

SAIL: Scene-aware Adaptive Iterative Learning for Long-Tail Trajectory Prediction in Autonomous Vehicles

Streaming Real-Time Trajectory Prediction Using Endpoint-Aware Modeling