What Is the Influence of Task Order in Continual Learning? An Empirical Study and Order-Aware Evaluation for Motion Forecasting

Zirui Li; Yunlong Lin; Haohan Yang; Xiaocong Zhao; Chen Lv

cs.LGNov 3, 2026

What Is the Influence of Task Order in Continual Learning? An Empirical Study and Order-Aware Evaluation for Motion Forecasting

Authors: Zirui Li, Yunlong Lin, Haohan Yang, Xiaocong Zhao, Chen Lv

Organizations: School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore 639798 · School of Mechanical Engineering, Beijing Institute of Technology, Beijing 100081, China, and also with the School of Computation, Information and Technology, Technical University of Munich, 85748 Garching, Germany · Department of Civil and Environmental Engineering, Samueli School of Engineering, University of California, Los Angeles, CA 90095 USA · Key Laboratory of Road and Traffic Engineering, Ministry of Education, Tongji University, Shanghai 201804, China

Abstract

Aggregate scores for continual learning algorithms rest on a distribution over task orders that is rarely stated. We formulate algorithm comparison as an expectation over a declared order distribution, which separates training variation from order variation, and apply it to motion forecasting with four road-layout tasks from the INTERACTION dataset, all 24 task orders, four algorithms, and three training replicates (288 runs). Task order accounts for 65% to 91% of the variation in current-task error and backward transfer, and 90% to 99% of this order effect comes from the identity of the starting task: the smallest task cannot be learned from a cold start. Beyond the first stage, the order effect is comparable to training variability. In most orders, the leading algorithm in current-task error and backward transfer is within one replicate standard deviation of the runner-up. Backward transfer depends on the order mainly through its reference term, which weights the first task most heavily, so relative improvement and final error identify different leading algorithms for displacement error. A transition-direction descriptor computed from task statistics tracks the position of the smallest task, and weighting orders by it leaves the algorithm ranking unchanged. These results call for benchmarks that declare their order distribution and report order-resolved, starting-task-resolved, warm-start, and final errors alongside aggregate scores. The code and data are available here.

Figures & tables

Fig. 1: Task order and algorithm comparison. (a) Current-task final displacement error (FDE-CT, m), averaged over three replicates, and algorithm ranking for two orders that differ only in their final two tasks. In both orders the leader lies within one pooled replicate standard deviation of the runner-up. (b) A common order policy $q(o)$ , uniform or weighted by the transition descriptor $D(o)$ , aggregates the order-resolved outcomes of every algorithm into a uniform mean or weighted score, shown with the range across orders.

Fig. 2: Order-aware evaluation workflow. (1) Tasks differ in size and speed distribution. (2) Each algorithm learns every order, starting cold in the first stage. (3) Testing all tasks after each stage gives the stage–task errors $R_{r,i}$ , summarised by CT, $\mathrm{CT}_{2:N}$ , $\mathrm{BWT}=P-B$ , and the final error. (4) W1 speed distances between consecutive tasks, $\Delta_{r}$ , define the descriptor $D(o)$ . (5) A common order policy $q(o)$ yields policy scores, and a sum-of-squares decomposition (FDE-CT) and a replicate bootstrap assess them. Networks, policy weights, and comparison marks are schematic.

Fig. 3: Task feature distributions. Dots, bands, and whiskers indicate medians, interquartile ranges, and 5th–95th percentiles. Acceleration is restricted to

[-5,5]

m/s 2 .

Fig. 4: UQnet backbone (adapted from [ 20 ] ): (a) encoder, (b) regularisation network, used during training and removed at inference, and (c) predictor, which decodes fused map and vehicle features with position features into an endpoint heatmap.

Algorithm	Stored examples	Output matching	Scenario partition	Gradient constraint	Gradient-based selection
Vanilla	$\times$	$\times$	$\times$	$\times$	$\times$
DER	$\checkmark$	$\checkmark$	$\times$	$\times$	$\times$
SyReM	$\checkmark$	$\times$	$\times$	$\checkmark$	$\checkmark$
GSM	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	$\times$

TABLE I: Published mechanisms of the compared algorithms. Stored examples serve output matching, rehearsal, or gradient constraints, and scenario partition allocates memory by past task.

	All 24 orders (%)					Excl. A-first (%)
Metric	Start	Order $\mid$ start	Alg.	Order $\times$ alg.	Within	Order	Alg.	Within
FDE-CT	$69.6$	$1.7$	$1.3$	$11.3$	$16.0$	$30.6$	$2.3$	$50.8$
MR-CT	$58.3$	$6.3$	$1.7$	$10.3$	$23.4$	$24.5$	$2.4$	$51.2$
FDE-BWT	$75.7$	$0.5$	$0.9$	$9.7$	$13.1$	$19.3$	$17.4$	$45.9$
MR-BWT	$89.8$	$1.5$	$3.4$	$1.3$	$4.0$	$39.5$	$26.5$	$25.6$
FDE-CT 2:N	$4.9$	$20.7$	$2.3$	$20.2$	$52.0$	$24.6$	$1.8$	$53.1$

TABLE II: Percentage of the total centred sum of squares per component. The order term is split into starting-task identity (Start) and order within starting task, and the last three columns give the 18 orders that do not start with A.

Order	Vanilla		DER		SyReM		GSM
	FDE-CT	MR-CT	FDE-CT	MR-CT	FDE-CT	MR-CT	FDE-CT	MR-CT
A $\to$ B $\to$ C $\to$ D	$8.25_{\pm 0.46}$	$32.83_{\pm 2.59}$	$6.17_{\pm 2.13}$	$31.75_{\pm 1.97}$	$7.17_{\pm 3.71}$	$31.74_{\pm 1.65}$	$6.08_{\pm 1.36}$	$31.84_{\pm 0.56}$
A $\to$ B $\to$ D $\to$ C	$4.98_{\pm 2.49}$	$30.83_{\pm 4.43}$	$6.67_{\pm 2.39}$	$32.20_{\pm 1.14}$	$6.55_{\pm 2.48}$	$28.76_{\pm 2.27}$	$8.00_{\pm 3.49}$	$34.38_{\pm 0.52}$
A $\to$ C $\to$ B $\to$ D	$7.33_{\pm 3.01}$	$33.28_{\pm 4.02}$	$6.69_{\pm 2.57}$	$29.44_{\pm 1.99}$	$3.31_{\pm 0.31}$	$25.33_{\pm 0.67}$	$6.23_{\pm 1.09}$	$31.86_{\pm 2.69}$
A $\to$ C $\to$ D $\to$ B	$9.26_{\pm 1.95}$	$29.96_{\pm 1.81}$	$3.48_{\pm 1.02}$	$28.85_{\pm 3.15}$	$4.71_{\pm 1.20}$	$27.92_{\pm 4.05}$	$5.00_{\pm 1.84}$	$30.21_{\pm 1.17}$
A $\to$ D $\to$ B $\to$ C	$9.20_{\pm 1.98}$	$31.51_{\pm 2.50}$	$3.38_{\pm 0.70}$	$28.18_{\pm 4.41}$	$3.90_{\pm 1.97}$	$26.08_{\pm 6.67}$	$9.17_{\pm 2.11}$	$35.83_{\pm 0.44}$

TABLE III: Current-task errors by task order (mean

\pm

sample standard deviation over three replicates). FDE is in metres and MR in percent. Bold: per-order minimum separable from the runner-up (Section IV-D ).

Order	Vanilla		DER		SyReM		GSM
	FDE-BWT	MR-BWT	FDE-BWT	MR-BWT	FDE-BWT	MR-BWT	FDE-BWT	MR-BWT
A $\to$ B $\to$ C $\to$ D	$-16.39_{\pm 1.57}$	$-46.19_{\pm 3.15}$	$-10.99_{\pm 5.05}$	$-41.41_{\pm 0.95}$	$-14.77_{\pm 8.97}$	$-52.95_{\pm 2.18}$	$-12.00_{\pm 3.63}$	$-54.18_{\pm 2.86}$
A $\to$ B $\to$ D $\to$ C	$-8.41_{\pm 5.26}$	$-42.47_{\pm 2.39}$	$-12.28_{\pm 5.50}$	$-44.11_{\pm 2.93}$	$-13.55_{\pm 5.89}$	$-54.58_{\pm 3.96}$	$-15.61_{\pm 9.07}$	$-48.23_{\pm 11.43}$
A $\to$ C $\to$ B $\to$ D	$-13.97_{\pm 7.56}$	$-47.29_{\pm 3.65}$	$-13.49_{\pm 6.02}$	$-46.46_{\pm 2.77}$	$-5.96_{\pm 0.70}$	$-50.20_{\pm 1.11}$	$-12.35_{\pm 2.34}$	$-49.80_{\pm 3.44}$
A $\to$ C $\to$ D $\to$ B	$-19.84_{\pm 4.77}$	$-42.47_{\pm 11.19}$	$-5.36_{\pm 2.77}$	$-37.45_{\pm 12.64}$	$-9.18_{\pm 3.12}$	$-53.14_{\pm 5.23}$	$-8.97_{\pm 5.24}$	$-48.22_{\pm 7.97}$
A $\to$ D $\to$ B $\to$ C	$-19.42_{\pm 4.94}$	$-45.86_{\pm 2.94}$	$-5.44_{\pm 1.35}$	$-40.02_{\pm 8.01}$	$-7.15_{\pm 4.53}$	$-45.12_{\pm 8.76}$	$-18.75_{\pm 5.52}$	$-50.08_{\pm 2.73}$

TABLE IV: Backward transfer by task order (mean

\pm

sample standard deviation over three replicates). FDE is in metres and MR in percentage points. Bold as in Table III .

Start	$R_{1,1}$	CT	CT 2:N	$P$	$B$	BWT	Final
A	$21.52$	$6.25$	$1.16$	$1.46$	$13.69$	$-12.23$	$1.28$
B	$2.77$	$1.81$	$1.48$	$2.34$	$2.46$	$-0.12$	$1.52$
C	$1.04$	$1.12$	$1.15$	$1.64$	$1.13$	$0.52$	$1.50$
D	$1.32$	$1.45$	$1.50$	$2.04$	$1.50$	$0.54$	$1.48$

TABLE V: FDE (m) by starting task over 72 runs per row: initial error

R_{1,1}

, CT with and without (CT 2:N ) the first stage, subsequent error

P

, reference

B

, BWT

=P-B

, and final error.

Algorithm	Metric	W1 speed	KL speed	JS speed	MMD speed	W1 accel.
Vanilla	FDE-CT	$-0.75$	$-0.23$	$-0.20$	$-0.75$	$+0.69$
	MR-CT	$-0.75$	$-0.09$	$-0.12$	$-0.75$	$+0.69$
	FDE-BWT	$+0.66$	$+0.13$	$+0.10$	$+0.66$	$-0.55$
	MR-BWT	$+0.81$	$+0.26$	$+0.32$	$+0.81$	$-0.46$
DER	FDE-CT	$-0.57$	$-0.29$	$-0.36$	$-0.57$	$+0.46$
	MR-CT	$-0.54$	$-0.22$	$-0.27$	$-0.54$	$+0.48$

TABLE VI: Spearman correlations between candidate transition-direction descriptors and order-level performance. W1: Wasserstein-1. Bold:

|\rho|\geq 0.7

Fig. 5: Transition-direction descriptor versus performance for each algorithm. Points are replicate means for individual orders, coloured by the position of task A, and

\rho

is the Spearman correlation with

D

. Grey guides mark the six distinct descriptor values.

Metric	$\rho(D)$	Excl. A-first	Start-adj.	$\rho(p_{A})$	$p_{A}$ -adj.
FDE-CT	$-0.64$	$-0.47$	$-0.48$	$-0.77$	$-0.23$
MR-CT	$-0.63$	$-0.43$	$-0.49$	$-0.79$	$-0.11$
FDE-BWT	$+0.61$	$+0.40$	$+0.29$	$+0.73$	$+0.22$
MR-BWT	$+0.74$	$+0.64$	$+0.62$	$+0.77$	$+0.53$
FDE-CT 2:N	$-0.37$	$-0.49$	$-0.58$	$-0.20$	$-0.27$
MR-CT 2:N	$-0.27$	$-0.47$	$-0.50$	$-0.10$	$-0.20$

TABLE VII: Mean Spearman associations across algorithms.

\rho(D)

: descriptor and order means. Excl. A-first: 18 orders. Start-adj.: rank residuals after removing starting-task means.

\rho(p_{A})

: position of task A.

p_{A}

-adj.: rank residuals after removing position-of-A means.

Algorithm	FDE: mean [min, max]	Prob.	MR: mean [min, max]	Prob.
Vanilla	1.63 [1.18, 2.14]	$0.00$	17.22 [10.84, 25.67]	$0.00$
DER	1.63 [1.17, 2.34]	$0.00$	17.80 [11.47, 29.09]	$0.00$
SyReM	1.08 [0.87, 1.99]	$1.00$	9.23 [4.38, 24.69]	$1.00$
GSM	1.43 [1.03, 2.44]	$0.00$	15.86 [8.05, 36.25]	$0.00$

TABLE VIII: Final errors: mean [minimum, maximum] over order-level replicate means, and the bootstrap probability (Prob.) of the smallest uniform mean. FDE is in metres and MR in percent. Bold: smallest mean.

Fig. 6: Paths with the largest and smallest total transition distance

L

for each starting task. Order strips identify the paths, and paired points compare algorithm-averaged scores, standardised separately for each metric across task orders.

Metric	Policy	Vanilla	DER	SyReM	GSM
FDE-CT	Uniform	$2.94$ (0.00)	$2.42$ (0.29)	$2.33$ (0.71)	$2.94$ (0.00)
	$D$ -weighted	$3.76$ (0.00)	$3.06$ (0.43)	$3.01$ (0.57)	$3.76$ (0.00)
	Excl. A-first	$1.43$ (0.06)	$1.48$ (0.04)	$1.31$ (0.90)	$1.62$ (0.00)
MR-CT	Uniform	$18.10$ (0.00)	$17.30$ (0.05)	$16.04$ (0.94)	$19.58$ (0.00)
	$D$ -weighted	$21.87$ (0.00)	$20.12$ (0.21)	$19.34$ (0.79)	$22.98$ (0.00)
	Excl. A-first	$13.86$ (0.01)	$13.32$ (0.09)	$11.98$ (0.89)	$15.19$ (0.00)

TABLE IX: Policy scores (units as in Tables III and IV ) under the uniform policy, the descriptor-weighted policy, and the uniform policy over the 18 orders that do not start with A, with the bootstrap probability of the smallest score under that policy in parentheses. Bold: smallest score per row.

Forecasting

See all 9 papers

From Order to Distribution: An Exact Operator Framework for Forgetting in Continual Learning
Apr 15, 2026Zonghuan Xu, Xingjun MaContinual Learning Catastrophic Forgetting
Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China
Temporal Taskification in Streaming Continual Learning: A Source of Evaluation Instability
Apr 23, 2026Nicolae Filat, Ahmed Hussain, Konstantinos Kalogiannis +1Continual Learning
Bitdefender, Romania · KTH Royal Institute of Technology · Stockholm, Sweden +2
Implicit Assumptions About Tasks in Continual Learning
Oct 30, 2026Kathrin Korte, Christian Metzner, Akshay Bedhotiya +3Continual Learning
IT University of Copenhagen Copenhagen, Denmark · Institute of Neuroinformatics University of Zurich and ETH Zurich · Peter Grünberg Institute Forschungszentrum Jülich & RWTH Aachen

Apr 15, 2026J/K move · Enter open · S save

From Order to Distribution: An Exact Operator Framework for Forgetting in Continual Learning

Zonghuan Xu, Xingjun Ma

Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China

Continual Learning Catastrophic Forgetting

Open paper AskPDF arXiv

A central challenge in continual learning is forgetting: the loss of performance on previously learned tasks after learning new ones. Prior theory has analyzed forgetting under random orderings of fixed task collections in overparameterized linear regression. We shift the focus from task order to task distribution, asking how its structure determines forgetting. In the linear setting with a shared solution, i.i.d. task sampling, and sequential exact fitting, we derive an exact operator identity expressing historical forgetting directly in terms of the task distribution. Building on this identity, we establish an exponential decay guarantee for expected historical forgetting under every fixed task distribution in finite dimensions, characterize its asymptotic behavior, and relate decay to the distribution's coverage of observable directions. For an individual learned task, we show that subsequent tasks can collectively support recovery without exact revisits. We derive a lower bound on recovery time and construct a task distribution attaining its inverse-coverage scaling.

Abstract

Forecasting

Explore similar work

From Order to Distribution: An Exact Operator Framework for Forgetting in Continual Learning

Temporal Taskification in Streaming Continual Learning: A Source of Evaluation Instability

Implicit Assumptions About Tasks in Continual Learning