Normative Loss Landscape Navigation: A Trajectory-Based Approach to Mitigating Forgetting in Incremental Learning

Isabelle Aguilar; Zayn Andre Zainal; Luis Fernando Herbozo Contreras; Zhaojing Huang; Omid Kavehei

cs.LGSep 28, 2026

Normative Loss Landscape Navigation: A Trajectory-Based Approach to Mitigating Forgetting in Incremental Learning

Authors: Isabelle Aguilar, Zayn Andre Zainal, Luis Fernando Herbozo Contreras, Zhaojing Huang, Omid Kavehei

Abstract

Continual learning models suffer from catastrophic forgetting when trained sequentially on non-stationary data distributions. Previously, this has been addressed through weight regularization. While preconditioning gradients offer a promising alternative to mitigate forgetting, current approaches are myopic. Conversely, standard regularization methods apply rigid, scalar Euclidean penalties that entirely ignore the underlying Riemannian geometry of the parameter space. To overcome this gap, we propose TMLN (Trajectory-Modulatory Landscape Navigation), a normative navigation policy that formalizes continual learning as an optimal control problem over a curved loss landscape. TMLN utilizes a memory-efficient diagonal empirical Fisher Information Matrix (FIM) to define a localized Riemannian manifold. To compensate for the spatial limitations of the diagonal approximation, TMLN dynamically modulates a preconditioner using the normalized historical trajectory of the network's parameter values. By integrating this trajectory-based preconditioning directly into the gradient update, we actively shield historically critical parameter directions without relying on additive penalties. Empirical evaluations on class- and domain-incremental benchmarks demonstrate that our method significantly reduces the loss barrier between consecutive tasks.

Figures & tables

Figure 1: TMLN Framework. (a) On a loss landscape, continual learning tackles both forgetting (in which the model moves out of task 1’s local minima and into task 2’s local minima) and intransigence (where the model struggles to move out of task 1’s minima). (b) Trajectory Modulated Landscape Navigation measures local curvature and historical trajectory, and incorporates it into a preconditioned gradient step. (c) Our method efficiently modulates optimization learning rules, allowing sequential learning while preventing movement in steep directions that harm past knowledge. This figure is produced with the aid of Gemini 3.

	Split CIFAR-100			CORe50
Method	ACC ( $\uparrow$ )	FM ( $\downarrow$ )	INT ( $\downarrow$ )	ACC ( $\uparrow$ )	FM ( $\downarrow$ )	INT ( $\downarrow$ )
Joint (oracle)	$94.39\pm 0.14$	$--$	$--$	$80.03\pm 0.00$	$--$	$--$
Naive SGD	$7.78\pm 0.40$	$66.53\pm 2.40$	$21.67\pm 2.55$	$63.72\pm 0.57$	$41.13\pm 0.69$	$23.88\pm 0.00$
ER	$9.23\pm 0.19$	$73.42\pm 3.37$	$\bf{14.99\pm 3.24}$	$77.36\pm 1.50$	$23.89\pm 1.68$	$1.57\pm 0.11$
EWC++	$6.01\pm 0.06$	$58.18\pm 0.59$	$33.64\pm 0.39$	$76.94\pm 0.52$	$16.44\pm 0.76$	$9.51\pm 0.32$
RWalk	$12.12\pm 1.35$	$70.01\pm 1.46$	$15.04\pm 0.38$	$67.76\pm 0.68$	$\bf{4.25\pm 0.96}$	$32.32\pm 0.20$

Table 1: Performance metrics on Split CIFAR-100 and CORe50. Final Accuracy (ACC), Average Forgetting (FM), and Intransigence (INT) are reported. Joint (oracle) serves as an upper bound for accuracy. All methods are averaged over five runs.

Method	Geometry (FIM)	Trajectory	ACC ( $\uparrow$ )	Forgetting ( $\downarrow$ )	Intransigence ( $\downarrow$ )
Standard SGD	–	–	$19.04\pm 0.15$	$89.89\pm 1.26$	$0.00\pm 0.48$
TMLN-Static	$\checkmark$	–	$46.97\pm 2.56$	$41.39\pm 4.30$	$14.56\pm 4.14$
TMLN-Euclidean	–	$\checkmark$	$50.35\pm 1.00$	$\mathbf{39.48\pm 1.85}$	$12.27\pm 0.78$
TMLN (Full)	$\checkmark$	$\checkmark$	$\mathbf{51.55\pm 0.56}$	$40.32\pm 3.67$	$\mathbf{9.94\pm 3.66}$

Table 2: Ablation results of TMLN. We evaluate the impact of the Riemannian metric tensor (FIM) and historical-trajectory modulation on Split CIFAR-10. Results are averaged over five runs.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Parameter	Value
TMLN (Ours)	$\lambda$	$0.1$
TMLN (Ours)	$\gamma$	$1.0$
ER	$\mathcal{B}$	$500$
EWC++	$\lambda$	$10000$
RWalk	$\lambda$	$10$
	$\alpha$	$0.8$

Appendix

Table 3: Hyperparameter specifications for baselines and the proposed method.

Method	Auxiliary Memory Form	Peak VRAM (GB)	Relative Training Time
Naive	None ( $0$ )	1.04	1.00x
ER (M=500)	Raw images ( $\sim 6$ MB)	1.94	1.04x
EWC++	Diagonal Fisher ( $\lVert\theta\rVert$ )	1.16	1.34x
OCAR	K-FAC matrices ( $\sum_{l=1}^{L}\big((d_{\mathrm{in}}^{(l)})^{2}+(d_{\mathrm{out}}^{(l)})^{2}\big)^{\ast}$ )	3.55	16.86x
TMLN (Ours)	Diagonal Fisher; stored $\theta_{\mathrm{prev}}$ ( $2\times\lVert\theta\rVert$ )	2.07	1.07x

Appendix

Table 4: Computational and Memory Overhead on CIFAR-100. Time is normalised relative to Naive SGD. Auxiliary memory indicates the additional state required beyond the standard model parameters (

\theta

	Task 1 Loss			Task 2 Loss
Method	Init	End of Task 1	End of Task 2	Init	End of Task 1	End of Task 2
Naive SGD	2.9338	1.3871	24.5702	11.1482	12.1247	6.3916
ER	2.3030	0.3775	22.5264	10.4751	13.3639	5.2835
TMLN	2.4773	1.1380	9.0663	8.7170	10.6379	1.0983

Appendix

Table 5: Optimization trajectory loss values across sequential tasks for Tasks 1 & 2. Init, End of Task 1, and End of Task 2 columns correspond to the loss landscape visualization markers. Best final loss is in bold.

Method	Configuration	ACC ( $\uparrow$ )	FM ( $\downarrow$ )	INT ( $\downarrow$ )
Joint (oracle)	$--$	$75.98\pm 0.56$	$--$	$--$
Naive	$\eta=0.001$	$19.04\pm 0.25$	$92.15\pm 1.94$	$-1.55\pm 1.64$
	$\eta=0.005$	$19.18\pm 0.05$	$91.28\pm 0.80$	$-0.74\pm 0.89$
	$\eta=0.01$	$19.04\pm 0.15$	$89.89\pm 1.26$	$0.00\pm 0.48$
	$\eta=0.05$	$16.78\pm 3.39$	$86.77\pm 1.70$	$5.09\pm 4.93$
	$\eta=0.1$	$12.14\pm 3.70$	$84.00\pm 0.69$	$13.92\pm 4.13$

Appendix

Table 6: Continual learning metrics for Split CIFAR-10 at various hyperparameter configurations.

Explore similar work

MANGO: Meta-Adaptive Network Gradient Optimization for Online Continual Learning
May 18, 2026Ankita Awasthi, Marco Apolinario, Kaushik RoyReplay-Based Continual Learning Plasticity
Purdue University, USA · TU Delft, Netherlands
Theoretical Foundations of Continual Learning via Drift-Plus-Penalty
Jun 7, 2026Nazreen Shah, Govinda Arya, Bharath B. N. +1Replay-Based Continual Learning Continual Learning
IIIT Delhi · IIT Dharwad
Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation
Jan 31, 2026Hao Gu, Mao-Lin Luo, Zi-Hao Zhou +3Replay-Based Continual Learning Forgetting
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China

May 18, 2026, cs.LGJ/K move · Enter open · S save

MANGO: Meta-Adaptive Network Gradient Optimization for Online Continual Learning

Ankita Awasthi, Marco Apolinario, Kaushik Roy

Purdue University, USA · TU Delft, Netherlands

Replay-Based Continual Learning Plasticity Parameter-Efficient Adaptation Regularization

Open paper AskPDF arXiv

In Online Continual Learning (OCL), a neural network sequentially learns from a non-stationary data stream in a single-pass with access only to a limited memory replay buffer. This contrasts sharply with off-line continual learning where training is multiple epoch dependent on large datasets. The main challenge faced by OCL is to overcome catastrophic forgetting of past tasks (stability) while learning new ones efficiently (plasticity). Existing methods counter forgetting via replay-based rehearsal, output level distillation, fixed regularization, or meta-learning on the current data. However, these methods have limitations: rehearsal introduces a stored sample bias; distillation operates on output-distributions without modulating parameter updates; fixed-regularization penalizes parameters irrespective of sensitivity; stream-only meta-learning lacks a feedback controlled parameter update. We propose Meta-Adaptive Network Gradient Optimization (MANGO), an OCL framework that balances stability-plasticity via gradient-gating and meta-learned regularization. Gradient-gating scales parameter updates based on sensitivity, preventing destructive updates. Meta-learned regularization adapts stability coefficients, evaluating the effect of parameter update on replay. In MANGO, replay acts as both a training signal and a forgetting evaluator. We evaluated our method on three standard OCL benchmark datasets. MANGO outperforms strong baselines, achieving state-of-the-art results with consistent performance across replay sizes. In domain incremental learning on CLEAR-10 and class incremental learning on CIFAR-100 and Tiny-ImageNet, it achieves highest accuracy among all baselines and achieves positive Backward Transfer, overcoming forgetting on CLEAR-10.

Abstract

Explore similar work

MANGO: Meta-Adaptive Network Gradient Optimization for Online Continual Learning

Theoretical Foundations of Continual Learning via Drift-Plus-Penalty

Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation