One-Step Curvature Probes Miss the Fitting Operator: Retained Capacity and Terminal Null-Space Correction for Continual Learning

Abu Sa-Adat Mohamed Moon-Im Al Ahsan; Ibne Farabi Shihab; Md Najmus Swaqeeb

cs.LGOct 2, 2026

One-Step Curvature Probes Miss the Fitting Operator: Retained Capacity and Terminal Null-Space Correction for Continual Learning

Authors: Abu Sa-Adat Mohamed Moon-Im Al Ahsan, Ibne Farabi Shihab, Md Najmus Swaqeeb

Organizations: Department of Computer Science and Engineering, BRAC University, Bangladesh · Department of Computer Science, Iowa State University, USA · BRAC University, Bangladesh

Abstract

A one-step curvature probe evaluates an initial direction, whereas continual learners are judged after reaching comparable new-task fit. In an overparameterized linearization, projected gradient descent converges to $Δ_P=PJ^\top(JPJ^\top)^{-1}r$ , and its squared-displacement inflation is exactly the reciprocal of the retained fitting capacity $c_P(r)$ . More generally, the terminal old-task quadratic ratio factorizes as $1/[c_P(r)G_{\rm end}(P,r)]$ , where $G_{\rm end}$ compares curvature along endpoint directions. In the rank-one case, $G_{\rm end}$ equals the one-step probe gain; for multiple outputs, the two gains can differ. The terminal quadratic also separates into a curvature-optimal fitting floor and an algorithm-dependent null-space excess, motivating terminal null-space correction, which preserves linearized new-task outputs on its Jacobian batch. Controlled checks validate the local quadratic and show that projection can greatly reduce matched-norm curvature while barely changing terminal forgetting. Across common-threshold configurations, projection yields the larger signed old-task loss change in 73/99 matched pairs, retained fitting capacity falls with rank, and fixed-rank comparisons separate terminal forgetting even when probe gain is approximately matched. On Permuted MNIST and Split CIFAR-100, terminal correction decreases signed old-task loss in 166/180 method--dataset--seed pairs under the all-seed intention-to-correct analysis; because acceptance and outcome reporting use the same held-out split, this result is descriptive and test-conditioned. Overall, terminal cost depends jointly on retained fitting capacity, endpoint-direction curvature, and the null-space component selected by optimization.

Figures & tables

Panel of Figure 1 — Figure 1: The operational distinction. Retention is limited by the information carried in the pre-request state. Retraction is limited by the residual uncertainty about the retraining target after conditioning on the state and the request. Changing what the request reveals changes the bound.

Panel of Figure 2 — Figure 2: Cache–delivery operating points. The dashed curve shows the leading-order achievable state–update interpolation for the request-hedging ridge family with $d=r=90$ , $p=d(d+1)/2$ , $\varepsilon=10^{-4}$ , and normalized radius $\rho=1$ ; its ideal bit rates are divided by eight to display bytes, and lower-order and serialization costs are omitted. The $M=0$ and $B=0$ endpoints are displayed at the one-byte plotting floor for log-scale visibility. Filled markers show approximate operating points for parameters, low-rank curvature, and packed fp32 sufficient statistics, ordered as California Housing, Concrete, YearPredictionMSD, CIFAR-10 ResNet features, and CIFAR-100 vision transformer (ViT) features. The first two sufficient-statistic points coincide. Nonzero delivery coordinates are reconstructed from rounded table means. Empirical points use $k=32$ value-exposing requests and their table-specific distortion criteria; points from different tasks should not be interpreted as a common Pareto frontier.

Method family	decoder state charged to $M$	request-time placement	YearPredictionMSD instantiation
Exact ridge sufficient statistics [ 27 ]	packed $(G,c)$	$B=0$	$M=16.35$ KiB, $B=0$
Dense-Hessian certified removal [ 15 , 29 ]	parameters + dense $d\times d$ curvature	local correction + native noise term	$M=31.99$ KiB; exact ridge algebra gives $B=0$ before any certification noise
Influence / approximate deletion [ 19 , 17 ]	parameters if Hessian remains source-side	one $d$ -vector correction	$M=0.352$ KiB, $B=220.5$ bytes under our residual codec
Hessian-free online [ 26 ]	one $d$ -vector statistic per training record	local vector aggregation	$M\approx 141$ MiB for the training split, $B\approx 0$

Table 1: Published methods under the paper’s decoder-state accounting. The state column is the object charged when the method is instantiated as a deployed decoder on the same tasks;

B

is the additional model-update packet under our codec. “Local” means the method can complete from cached state and request without a model-update transcript, while its native compute/noise guarantees remain those of the cited method.

$s$	predicted $d-s$	fitted slope	fitted/predicted
$0$	$64$	$64.41\pm 0.63$	$1.006$
$16$	$48$	$47.62\pm 0.48$	$0.992$
$32$	$32$	$32.27\pm 0.37$	$1.008$
$48$	$16$	$15.88\pm 0.21$	$0.993$

Table 2: Direct test of the partial-curvature coefficient. The

\pm

values are ordinary least-squares slope standard errors across nine codec operating points.

Objective	$d$	cache rank $s$	$s/d$	reduction
California ridge	$8$	$4$	$50.0\%$	$13.0\%$
Concrete ridge	$8$	$4$	$50.0\%$	$13.4\%$
YearPrediction ridge	$90$	$16$	$17.8\%$	$9.7\%$
MNIST logistic	$784$	$128$	$16.3\%$	$12.2\%$
ResNet-feature logistic	$512$	$128$	$25.0\%$	$19.6\%$
CIFAR-10 ResNet ridge head	$512$	$64$	$12.5\%$	$10.3\%$

Table 3: Natural-objective cache sweep. “Reduction” is the measured decrease in total post-request packet bytes relative to parameters only at the same distortion. The

s/d

column is the exact coefficient only for the orthogonal block interface of Corollary 3.2 ; it is shown here as a reference, not as a prediction.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

State	$M$	$B_{\rm id}$	$B_{\rm value}$	$D_{\max}^{\rm id}$	$D_{\max}^{\rm value}$
Current parameters	$3{,}152$	$3{,}184$	$3{,}184$	$3.2\!\times\!10^{-17}$	$3.2\!\times\!10^{-17}$
Sufficient statistics $(G,c)$	$17{,}856$	$3{,}184$	$0$	$3.2\!\times\!10^{-17}$	$4.7\!\times\!10^{-15}$
Indexed training rows	$554{,}784$	$0$	$0$	$0$	$0$

Appendix

Table 4: Ridge-interface accounting under a fixed finite-precision codec.

M

and

B

are realized bits under one uncompressed float64 codec;

D_{\max}

is the maximum held-out prediction RMSE over all conditions, fractions, seeds, and requests. Request costs

Q

are stated in the text and are common across states within each interface/fraction surface. The exact-real references are excluded from this finite table.

cache rank $s$	predicted $d-s$	fitted slope	max error $/\varepsilon$
$0$	$64$	$64.41\pm 0.63$	$0.914$
$16$	$48$	$47.62\pm 0.48$	$0.906$
$32$	$32$	$32.27\pm 0.37$	$0.899$
$48$	$16$	$15.88\pm 0.21$	$0.892$

Appendix

Table 5: Direct coefficient test for Corollary 3.2 . The predicted slope is the unresolved dimension

d-s

. The

\pm

term is the ordinary least-squares slope standard error across the

9

codec operating points.

Dataset	$d$	full-rank requests	max $\\|w_{v}-w\\|_{2}$	min packing / bits
California Housing	$8$	$250/250$	$2.4\times 10^{-13}$	$2{,}604$ / $11.35$
Concrete Strength	$8$	$250/250$	$3.1\times 10^{-13}$	$2{,}889$ / $11.50$
YearPredictionMSD	$90$	$250/250$	$7.6\times 10^{-13}$	$5{,}911$ / $12.53$

Appendix

Table 6: Natural-data witnesses for Proposition A.1 . “Full-rank requests” counts deletion requests for which

\operatorname{rank}(\mathcal{J}_{Q})=d

. The packing column is the minimum greedy

2\varepsilon

-packing size across the

250

requests for each dataset at parameter RMSE

\varepsilon=10^{-4}

. Because the candidate pool has only

8{,}192

members, the associated bit lower bound cannot exceed

13

bits.

Dataset	min $\sigma_{\min}(\mathcal{J}_{Q})$	median $\sigma_{\min}(\mathcal{J}_{Q})$	min local gain
California Housing	$2.1\times 10^{-3}$	$8.4\times 10^{-3}$	$1.8\times 10^{-3}$
Concrete Strength	$1.7\times 10^{-3}$	$7.9\times 10^{-3}$	$1.4\times 10^{-3}$
YearPredictionMSD	$6.2\times 10^{-4}$	$2.8\times 10^{-3}$	$5.0\times 10^{-4}$

Appendix

Table 7: Conditioning of the natural-data target map. Singular values use orthonormal coordinates in the feasible reweighting subspace. “Local gain” is the minimum finite-difference ratio over

10{,}000

local pairs per request within radius

0.02

Dataset	$B_{k=1}$	$B_{k=8}$	$B_{k=32}$	$\Delta$ bits / coordinate	$\widehat{\alpha}$
California Housing	$24.6\pm 1.1$	$26.0\pm 1.1$	$27.0\pm 1.2$	$2.4$	$0.47$
Concrete Strength	$25.1\pm 1.4$	$26.8\pm 1.4$	$27.7\pm 1.5$	$2.6$	$0.51$
YearPredictionMSD	$192.4\pm 3.8$	$209.0\pm 4.0$	$220.5\pm 4.3$	$2.5$	$0.49$

Appendix

Table 8: Residual-codec growth with deletion batch size. Payloads are mean bytes for the parameters-only decoder. The fourth numeric column converts the increase from

k=1

k=32

into additional bits per parameter coordinate. The final column fits the median maximum residual index to

k^{\alpha}

across

k\in\{1,8,32\}

. The

\pm

values are sample standard deviations across five split-level mean payloads.

Dataset	$d$	sketch rank	$M_{\rm param}$ (KiB)	$M_{\rm sketch}$ (KiB)	$B_{\rm param}$	$B_{\rm sketch}$
California Housing	$8$	$4$	$0.031$	$0.176$	$27.0\pm 1.2$	$23.5\pm 1.1$
Concrete Strength	$8$	$4$	$0.031$	$0.176$	$27.7\pm 1.5$	$24.0\pm 1.2$
YearPredictionMSD	$90$	$16$	$0.352$	$6.04$	$220.5\pm 4.3$	$199.1\pm 4.0$

Appendix

Table 9: Natural-data cache–delivery tradeoff at held-out NRMSE at most

10^{-4}

relative to direct retraining.

B

is the mean post-request payload in bytes for group deletion

k=32

; the

\pm

term is the standard deviation across the five split-level means.

quantity	theorem	fitted exponent	rate discrepancy
$\\|v_{Q}\\|_{2}$ versus $k$	$k^{1/2}$	$k^{0.50\pm 0.03}$	–
population radius versus $m$	$m^{-1}$	$m^{-1.01\pm 0.03}$	–
known- $\Sigma$ fluctuation SD versus $m$	$m^{-3/2}$	$m^{-1.49\pm 0.04}$	$0.38$ bits/active coordinate

Appendix

Table 10: Gaussian-design scaling audit. Exponents are fitted on log–log axes across the declared grid; the final column compares reverse-water-filling rate with the empirical Gaussian code length at the same squared-error distortion. The

\pm

values are ordinary least-squares standard errors of the fitted exponents across the stated grid.

Dataset	predicted bits/coordinate	observed bits/coordinate	difference
California Housing	8.7	9.0	0.3
Concrete Strength	9.3	9.7	0.4
YearPredictionMSD	17.7	18.0	0.3

Appendix

Table 11: Per-dataset body-width check at

k=32

. The predicted column evaluates the Gaussian approximation using each training split’s empirical covariance and validation-selected residual scale; the observed column removes the fixed 18-byte packet header before dividing by dimension.

Task	$d$	parameters only	diagonal	rank- $32$	rank- $128$
MNIST $3$ vs. $8$	$784$	$1.48$	$1.45$	$1.44$	$1.30$
CIFAR-10 cat vs. dog, ResNet-18 features	$512$	$1.12$	$1.09$	$1.08$	$0.90$

Appendix

Table 12: Nonquadratic strongly-convex objectives. Mean request-time payload

B

in KiB at parameter RMSE at most

10^{-4}

from direct retraining, averaged over

500

deletion requests per split and

5

splits.

Representation	classes	$M_{\rm param}$	$M_{\rm rank64}$	$M_{\rm stat}$	$B_{\rm param}$	$B_{\rm rank64}$
CIFAR-10 / ResNet-18	$10$	$20.0$ KiB	$148.3$ KiB	$533.0$ KiB	$7.8$ KiB	$7.0$ KiB
CIFAR-100 / ViT-B/16	$100$	$300.0$ KiB	$492.3$ KiB	$1.42$ MiB	$116.2$ KiB	$108.3$ KiB

Appendix

Table 13: Deep-representation ridge heads. State is fp32; update payloads are means over

5{,}000

value-exposing deletion requests.

projection rank $r$	trainable state	sufficient-stat state	$B_{\rm param}$	crossover requests
8	3.13 KiB	3.27 KiB	1.71 KiB	1
16	6.25 KiB	6.78 KiB	3.34 KiB	1
32	12.50 KiB	14.56 KiB	6.55 KiB	1

Appendix

Table 14: Fixed-downprojection adapter accounting on CIFAR-100 ViT-B/16 features. Parameter and sufficient-statistic states use fp32;

B_{\rm param}

is the mean residual packet for

k=32

value-exposing deletions at logit NRMSE at most

10^{-4}

Model	zero deletion	masked-batch loss	$\delta$	body bits (P/F/K)
Fashion-MNIST MLP	bit-identical	retained-example mean	$2.9\times 10^{-4}$	$10/10/9$
CIFAR-10 ResNet-18	bit-identical	retained-example mean	$3.0\times 10^{-4}$	$10/10/9$

Appendix

Table 15: Nonconvex protocol controls and residual codec. Zero deletion checks exact replay of the declared training trajectory. Body widths are the modal signed-integer widths selected by the request-level codec; the 18-byte packet header is charged separately in every update. P/F/K denote parameters-only, diagonal-Fisher, and K-FAC.

Model	decoder state	$M$	$B$
Fashion-MNIST MLP	parameters only	$0.90$ MiB	$0.280\pm 0.004$ MiB
	+ diagonal Fisher	$1.80$ MiB	$0.280\pm 0.004$ MiB
	+ K-FAC	$2.39$ MiB	$0.252\pm 0.004$ MiB
CIFAR-10 ResNet-18	parameters only	$42.66$ MiB	$13.34\pm 0.06$ MiB
	+ diagonal Fisher	$85.32$ MiB	$13.34\pm 0.06$ MiB
	+ K-FAC	$222.10$ MiB	$12.01\pm 0.05$ MiB

Appendix

Table 16: Nonconvex decoder-state accounting under same-initialization, slot-preserving retraining.

M

is serialized pre-request state and

B

is mean post-request residual delivery. The

\pm

term is SD across

3

seed-level means.

State format	payload $M$ (bits)	median error	99th percentile	envelope
float64 entries of $H$	$8{,}704$	$0$	$0$	$32{,}000/32{,}000$
float32 entries of $H$	$4{,}352$	$8.81\!\times\!10^{-9}$	$2.73\!\times\!10^{-8}$	$32{,}000/32{,}000$
float16 entries of $H$	$2{,}176$	$7.19\!\times\!10^{-5}$	$2.23\!\times\!10^{-4}$	$32{,}000/32{,}000$
uniform 12-bit	$1{,}632$	$2.55\!\times\!10^{-4}$	$3.24\!\times\!10^{-4}$	$32{,}000/32{,}000$
uniform 8-bit	$1{,}088$	$4.11\!\times\!10^{-3}$	$5.20\!\times\!10^{-3}$	$32{,}000/32{,}000$
uniform 6-bit	$816$	$1.66\!\times\!10^{-2}$	$2.11\!\times\!10^{-2}$	$32{,}000/32{,}000$

Appendix

Table 17: State-only precision sweep at

d=16

. Each format stores

p=d(d+1)/2=136

upper-triangular entries of

H

. Errors are Euclidean parameter errors over

2{,}000

matrices and all

16

singleton deletions per matrix (

32{,}000

requested targets per format). The final column counts targets satisfying the derived local perturbation envelope.

Fit	prediction	$d=8$	$d=16$	$d=32$
$M$ at $B=0$	$36,136,528$	$36.31\!\pm\!0.51$	$134.69\!\pm\!0.72$	$538.73\!\pm\!7.80$
$B$ at $M=0$	$8,16,32$	$8.07\!\pm\!0.11$	$15.85\!\pm\!0.09$	$32.27\!\pm\!0.45$
$\theta=1/2$ : $M/p$	$0.5$	$0.510\!\pm\!0.012$	$0.495\!\pm\!0.003$	$0.495\!\pm\!0.003$
$\theta=1/2$ : $B/d$	$0.5$	$0.495\!\pm\!0.003$	$0.495\!\pm\!0.003$	$0.495\!\pm\!0.003$

Appendix

Table 18: Fitted coefficients against

\log_{2}(1/\varepsilon)

. Here

p=d(d+1)/2

. Intermediate slopes are normalized by

p

d

. The

\pm

values are ordinary least-squares slope standard errors across the eight codec operating points.

State format	median error	99th percentile	max error/bound	valid trials
float64	$0$	$0$	$0$	$1{,}000/1{,}000$
float32	$3.12\!\times\!10^{-8}$	$5.89\!\times\!10^{-8}$	$0.160$	$1{,}000/1{,}000$
float16	$2.57\!\times\!10^{-4}$	$5.10\!\times\!10^{-4}$	$0.172$	$1{,}000/1{,}000$
uniform 12-bit	$1.37\!\times\!10^{-3}$	$2.11\!\times\!10^{-3}$	$0.168$	$1{,}000/1{,}000$
uniform 8-bit	$2.20\!\times\!10^{-2}$	$3.41\!\times\!10^{-2}$	$0.152$	$1{,}000/1{,}000$
uniform 6-bit	$9.04\!\times\!10^{-2}$	$1.49\!\times\!10^{-1}$	$0.070$	$1{,}000/1{,}000$

Appendix

Table 19: Finite-statistic perturbation for

d=16

n=128

, and

\lambda=0.2

, over

1{,}000

instances per format. The ratio column is the maximum observed parameter error divided by the bound in Equation ( 36 ).

Dataset	$d$	deletions	$M_{\rm param}$ (KiB)	$B_{\rm param}$ (bytes)	$M_{\rm stat}$ (KiB)	max NRMSE
California Housing	$8$	$5{,}000$	$0.031$	$24.6\pm 1.1$	$0.172$	$8.7\!\times\!10^{-5}$
Concrete Compressive Strength	$8$	$5{,}000$	$0.031$	$25.1\pm 1.4$	$0.172$	$9.1\!\times\!10^{-5}$
YearPredictionMSD	$90$	$5{,}000$	$0.352$	$192.4\pm 3.8$	$16.35$	$9.4\!\times\!10^{-5}$

Appendix

Table 20: External ridge validation for singleton deletion.

M

is frozen decoder-state size and

B

is the mean parameters-only update size; sufficient statistics use

B=0

under the value-exposing interface. The

\pm

term is the SD across five split-level mean update sizes.

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 301 papers

Why Zeroth-Order Adaptation May Forget Less: A Randomized Shaping Theory
May 11, 2026Yao Shu, Jian Mu, Zhongxiang DaiZeroth-Order Optimization Continual Learning
HKUST(GZ)
Curvature-Aligned Probing for Local Loss-Landscape Stabilization
Apr 16, 2026Nikita Kiselev, Andrey GrabovoyNeural Network Optimization Neural Network Training Dynamics
Department of Intelligent Systems Moscow Institute of Physics and Technology
Learning in the Null Space: Small Singular Values for Continual Learning
Feb 25, 2026Cuong Anh Pham, Praneeth Vepakomma, Samuel HorváthContinual Learning Orthogonal Representation Learning
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE · Massachusetts Institute of Technology (MIT), USA

May 11, 2026J/K move · Enter open · S save

Why Zeroth-Order Adaptation May Forget Less: A Randomized Shaping Theory

Yao Shu, Jian Mu, Zhongxiang Dai

HKUST(GZ)

Zeroth-Order Optimization Continual Learning Gradient Descent Dynamics Catastrophic Forgetting

Open paper AskPDF arXiv

Continual learning requires new-task adaptation without damaging previously acquired capabilities. Recent forward-pass and zeroth-order (ZO) results show that low-query adaptation may retain better than first-order (FO) descent, but the usual view of ZO as noisy FO estimation does not explain why. We give a local randomized gradient-shaping analysis: finite differences expose a raw shape that is mean-aligned with FO, while the norm-matched comparator fixes the expected squared adaptation norm. Under this controlled comparison, forgetting depends on how the adaptation shape exposes retention curvature. For norm-matched ZO, the expected shaped retention curvature obeys an exact identity that preserves the isotropic retention floor while contracting only the anisotropic component. Projecting this identity onto the incoming gradient yields the observable FO--ZO quadratic forgetting gap: ZO improves mean forgetting precisely when the FO direction has above-average retention curvature, by a query-dependent fraction of that curvature excess. A practical finite-query accounting separates the mean mechanism from one-batch sampling and smoothing perturbations. As an algorithmic transfer, RISE applies the calibrated ZO shape to exact FO gradients inside parameter blocks. Its target is a stability--plasticity tradeoff: randomized shaping may reduce the retention exposure paid by FO, exact gradients remove finite-smoothing bias from finite-difference ZO, and blockwise sampling supplies many local shaping directions after one gradient computation. The blockwise analysis separates mean-step damage from centered random exposure, showing how block-diagonal curvature, cross-block coupling, and local shaping diagnostics specify where this exact-gradient transfer is most likely to be visible.

Abstract

Continual Learning

Explore similar work

Why Zeroth-Order Adaptation May Forget Less: A Randomized Shaping Theory

Curvature-Aligned Probing for Local Loss-Landscape Stabilization

Learning in the Null Space: Small Singular Values for Continual Learning