What Must Replay Preserve? Separating Correctable Bias from Class Correspondence

BoRen Deng; Xiangyue Ma; Chenglong Li; Xiaoting Du

cs.LGOct 5, 2026

What Must Replay Preserve? Separating Correctable Bias from Class Correspondence

Authors: BoRen Deng, Xiangyue Ma, Chenglong Li, Xiaoting Du

Organizations: Beijing University of Posts and Telecommunications · Beihang University · Beijing University of Technology

Abstract

Class-incremental learning must recognize all classes seen so far without task labels. Logit replay methods such as DER and DER++ mitigate forgetting by matching the model's past predictions on stored examples. Deleting this matching reveals its benefit, but the resulting accuracy cost cannot show whether the stored scores themselves are needed, or whether the cost survives correction of the classifier's bias toward recent classes. We propose a diagnostic framework that treats a cached prediction as temporally heterogeneous supervision: it separates classes known when an example was stored from classes learned afterward, edits each group, and evaluates every model before and after a task-level offset that leaves within-task predictions unchanged. On CIFAR-100 with DER++, suitable fixed constants replace the unrefreshed stored scores of later-learned classes within an equivalence margin of 1 percentage point, and the offset reduces the cost of deleting their matching from 14.9 to 1.8 points. Reassigning the non-gold scores of classes known at storage, which preserves their values and each task's target probability, costs 4.3 points before and 4.0 after the offset, and a parallel cost persists in image distillation. In the tested fixed-head setting, the large cost of deleting later-class matching is thus mostly correctable by this offset, whereas the smaller cost of disrupting class correspondence persists. Code and data are available at anonymous.4open.science/r/replay-preserve-E22B.

Figures & tables

Figure 1: Different class groups require different replay information. A stored tiger example illustrates classes known at storage ( $H$ ), learned afterward ( $N$ ), and unseen at replay ( $F$ ); the photograph is illustrative. The cards show dataset-level DER++ costs of deleting $N$ matching, replacing $N$ scores with a constant, and reassigning $H$ scores, as trained and after a task-level offset (withheld-image cohort, Section 4.3 ).

Policy	Reference on $N$	Selected-block loss
Full	Cached vector $q_{i,N}$	$\ell_{N}(z_{N},q_{i,N})$
Record mean	$\mu_{q_{i},N}\mathbf{1}$	$\ell_{\rm flat}(z_{N};\mu_{q_{i},N})$
Fixed anchor	$a\mathbf{1}$	$\ell_{\rm flat}(z_{N};a)$
Delete $N$	None	$0$

Table 1: Target replacement retains a constraint that deletion removes. Definitions on the support

N=N_{i,t}

. All DER++ policies keep replay labels and

H/F

matching;

\ell_{\rm flat}

as in Equation 4 .

Figure 2: Deleting

N

matching loses accuracy while simplified references retain it. Glyphs show what the stored

N

scores become. (a) Final accuracy over 20 paired seeds (dots) and means. (b) Full minus replacement, with 90% paired intervals and the

\pm 1

pp equivalence band; each constant is tested separately on seeds 1–8 (circles) and 9–20 (diamonds) and is grey unless both intervals lie in the band; single-marker rows use all seeds of their cohort; “elsewhere” rows reuse

a=-2

in other settings, and TinyImageNet passes only its original rule.

Figure 3: Changing which class holds which value is costly on $H$ and not on $N$ . Dots are seeds; whiskers are 95% paired intervals for $H$ and 90% for $N$ , whose band is the $\pm 1$ pp margin. (a) Replacing values by a mean (record mean on $N$ , task-block mean on $H$ ; 20 seeds each) and reassigning them (a cyclic shift across $N$ , 12 seeds; within each $H$ task block, 20 seeds). (b) Reassignment cost after rescaling each record’s residual to the energies of Section 3.3 ; the $H$ points and the $N$ point at $e_{H}$ come from one 12-seed cohort, in which $N$ values are reassigned within task blocks; the $N$ point at $e_{N}$ is the across- $N$ shift of panel (a).

Figure 4: A task-level offset removes most of the deletion cost; the reassignment cost remains. Hollow markers show models as trained and filled markers the same models after the offset, with 95% paired intervals. (a) Cost relative to Full in the prespecified withheld-image cohort (lines are seeds). (b) Gain of Both over Neither with $a=-2$ (eight seeds each; offsets cross-fitted on test halves): a within-task part that no offset changes (dashed) plus a cross-task part (bars).

Figure 5: The offset separates correctable from persistent gains beyond target interventions. Markers and intervals as in Figure 4 . (a) Gain over ER of three replay methods (32 paired seeds). (b) Gain of distillation students over label-only training. (c) Corrected gain over label-only training for original and edited within-block targets in a separate E1 cohort; left, schematic.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Policy	Reference on $N$	Accuracy (%)
Full	Cached target vector	37.62
Record mean (Niso)	Record-specific scalar	37.30
Generic	Independently calibrated anchor	37.09
Delete $N$	Matching term removed	22.46

Appendix

Table S1: Simplified

N

targets retain accuracy, whereas deletion is costly. Terminal class-incremental accuracy on CIFAR-100 with ResNet-18; twenty paired units. Untreated supports and label losses remain supervised.

Dataset ( $n$ )	Contrast	Older tasks	Latest task
CIFAR-100 (12)	Full $-$ Delete $N$	$+16.168$	$-6.617$
		$[15.150,17.185]$	$[-8.492,-4.741]$
CIFAR-100 (20)	Full $-$ Delete $N$	$+17.832$	$-8.855$
		$[17.034,18.630]$	$[-10.765,-6.945]$
CIFAR-100 (20)	Full $-$ Perm $H$	$+5.447$	$-1.925$
		$[4.815,6.080]$	$[-3.146,-0.704]$

Appendix

Table S2: Retaining the constraint benefits older tasks at a latest-task cost. Paired accuracy differences in pp with 95% intervals. Rows use separate cohorts; the component studies use Generic on CIFAR-100 and record mean on TinyImageNet.

Comparison	Endpoint	Mean	95% interval	Positive/ $n$
Full – Niso	J	+0.329	$[-0.056,+0.715]$	12/20
Full – Niso	$A_{\rm task}$	+0.236	$[-0.221,+0.694]$	12/20
Full – Niso	$C_{\rm task}$	+0.092	$[-0.190,+0.374]$	10/20
Full – Niso	A	-0.009	$[-0.457,+0.439]$	9/20
Full – Niso	C	+0.246	$[-0.024,+0.516]$	12/20
Niso – Delete N	J	+14.836	$[+14.212,+15.460]$	20/20

Appendix

Table S3: Record-mean replacement: all accuracy endpoints (20 paired units). Differences in pp, with paired 95% intervals.

Comparison	Endpoint	Mean	95% interval	Positive/ $n$
Full – Delete N	J	+13.886	$[+13.107,+14.665]$	12/12
Full – Delete N	$A_{\rm task}$	+15.750	$[+14.855,+16.645]$	12/12
Full – Delete N	$C_{\rm task}$	-5.116	$[-5.654,-4.578]$	0/12
Pooled – Full	J	-0.220	$[-0.813,+0.373]$	3/12
Pooled – Full	$A_{\rm task}$	-0.348	$[-0.985,+0.288]$	3/12
Pooled – Full	$C_{\rm task}$	-0.058	$[-0.572,+0.455]$	6/12

Appendix

Table S4: Deletion and anchor interventions: task-coordinate endpoints (12 paired units). Differences in pp, with paired 95% intervals. Only Pooled–Full (

J

\pm 1

pp TOST) and the two-member

C_{\rm task}

family were prespecified. The diagnostic Full–Delete

J

value here is 13.886 pp, versus 13.889 pp in the frozen training endpoint.

Comparison	Endpoint	Mean	95% interval	Positive/ $n$
Pooled – Full	old	-0.249	$[-0.955,+0.457]$	3/12
Pooled – Full	latest	-0.025	$[-0.894,+0.844]$	7/12
Self – Shuffle	old	-0.484	$[-1.228,+0.260]$	5/12
Self – Shuffle	latest	+0.517	$[-0.888,+1.922]$	7/12
Shuffle – Pooled	old	+0.528	$[-0.257,+1.312]$	9/12
Shuffle – Pooled	latest	+0.250	$[-0.708,+1.208]$	7/12

Appendix

Table S5: Anchor interventions evaluated on older and newest tasks, recovered from the same sixty terminal per-task vectors (12 paired units). Differences in pp, with paired 95% intervals. Full–Delete population contrasts are in Table S2 .

Method	Anchor	$n$	Acc.	Cost	90% CI	Family	Holm $p$	Old/new cost
DER++	$a=-1$	8	37.7	$+0.19$	$[-0.10,+0.47]$	later B	.001	$+0.36/-1.40$
DER++	$a=-2$	8	37.5	$+0.44$	$[+0.04,+0.85]$	initial A	.035	$+0.52/-0.26$
DER++	$a=-4$	8	37.9	$+0.03$	$[-0.58,+0.64]$	later B	.010	$-0.08/+1.01$
DER++	$a=0$	8	37.1	$+0.80$	$[+0.39,+1.21]$	initial A	.189	$+1.07/-1.61$
DER++	$N/F$ fixed, $a=-2$	8	37.1	$+0.83$	$[+0.40,+1.25]$	C	.234	$+1.15/-2.05$
DER++, remaining 12	$a=-2$	12	37.4	$+0.08$	$[-0.71,+0.87]$	E	.031	—

Appendix

Table S6: Fixed

N

references retain the tested benefit for selected constants. Accuracy and Full-minus-variant cost are in percentage points. Intervals are paired 90% intervals for

\pm 1

pp equivalence; Holm

p

is adjusted within the specified family, not across families. Old/new costs are descriptive; — where only the training-log endpoint exists. Tiny

a=-2

and memory rows use their own Full controls. Families E–H were specified before their runs; E and “one” are single tests.

	Retained terms		CIFAR-100	TinyImageNet
Policy	Level	Dispersion	Generic, $n=20$	Record mean, $n=8$
Neither	–	–	22.46	10.73
Level only	$\checkmark$	–	24.78	11.67
Dispersion only	–	$\checkmark$	28.98	12.06
Both	$\checkmark$	$\checkmark$	37.09	19.45
Both $-$ Level only			$12.304$ $[11.661,12.946]$	$7.780$ $[6.803,8.757]$

Appendix

Table S7: Both retained components outperform either alone. Accuracy (%) under fixed coefficients; bold marks the highest observed mean among the four component policies, not a significance test. Paired contrasts below use pp and 95% intervals.

Dataset	Endpoint	Interaction	95% CI
CIFAR-100	Older	$+7.179$	$[6.378,7.980]$
CIFAR-100	Latest	$-6.800$	$[-8.233,-5.367]$
TinyImageNet	Older	$+7.843$	$[6.547,9.139]$
TinyImageNet	Latest	$-6.138$	$[-8.795,-3.480]$

Appendix

Table S8: Level and dispersion interact positively on older tasks and negatively on the latest task. Differences in pp with paired 95% intervals.

Setting	$n$	$I_{J}$ (pp)	$I_{C_{\rm task}}$ (pp)	$I_{L_{\rm task}}$ (nats)	$I_{L_{\rm within}}$ (nats)
CIFAR-100, Generic	20	$+5.78\ [5.14,6.43]$	$-1.45\ [-1.78,-1.12]$	$+0.64\ [0.58,0.69]$	$-0.02\ [-0.03,-0.01]$
TinyImageNet, record mean	8	$+6.44\ [5.45,7.44]$	$+1.15\ [0.48,1.81]$	$+1.41\ [1.25,1.57]$	$+0.09\ [0.06,0.13]$

Appendix

Table S9: Level–dispersion interactions on accuracy and exact loss terms (mean and paired 95% interval). Positive loss interaction denotes a larger reduction by the joint policy.

Comparison	$J$ (pp)	$L_{\rm task}$ (nats)	$L_{\rm within}$ (nats)
Both $-$ doubled level	$+12.49\ [11.43,13.55]$	$+1.59\ [1.46,1.71]$	$-0.17\ [-0.19,-0.15]$
Both $-$ doubled dispersion	$+7.32\ [5.76,8.87]$	$+0.75\ [0.60,0.90]$	$+0.02\ [0.01,0.04]$

Appendix

Table S10: Doubling a single term does not recover the joint policy on CIFAR-100 DER++ (

n=8

). Both uses two terms at the original coefficient; each single-term policy uses twice the original coefficient. Differences are paired means with 95% intervals; loss columns are reductions by Both, so positive values mean lower loss under Both.

Comparison	Endpoint	Mean	95% interval	Positive/ $n$
Full – Perm H	J	+4.714	$[+4.213,+5.214]$	20/20
Full – Perm H	$A_{\rm task}$	+4.689	$[+4.172,+5.206]$	20/20
Full – Perm H	$C_{\rm task}$	+12.771	$[+12.288,+13.255]$	20/20
Full – Perm H	A	+5.098	$[+4.517,+5.678]$	20/20
Full – Perm H	C	+5.517	$[+5.111,+5.923]$	20/20
Full – Iso H*	J	+3.405	$[+3.036,+3.774]$	20/20

Appendix

Table S11: Historical-class reassignment readouts and the three-arm deployment comparison (20 paired units). Differences in pp, with paired 95% intervals. Starred contrasts cross implementations and are descriptive.

Comparison	Endpoint	Mean	90% CI	95% CI	Role
Original – Permute N	J	-0.011	$[-0.502,+0.481]$	$[-0.613,+0.592]$	primary, $\pm 1$ pp
Original – Mean N	J	+0.222	$[-0.181,+0.624]$	$[-0.272,+0.715]$	secondary, $\pm 1.5$ pp
Perm. – Orig. N	old	+0.088	$[-0.490,+0.666]$	$[-0.621,+0.797]$	descr. population
Perm. – Orig. N	latest	-0.683	$[-1.470,+0.103]$	$[-1.647,+0.281]$	descr. population
Orig. – Perm. N (instr.)	$A_{\rm task}$	-0.294	$[-0.831,+0.243]$	$[-0.952,+0.364]$	instr. projection
Orig. – Perm. N (instr.)	$C_{\rm task}$	+0.013	$[-0.220,+0.247]$	$[-0.273,+0.300]$	instr. projection

Appendix

Table S12: Coordinate reassignment on N: frozen primary and separately identified secondary readouts.

Reassignment restricted to	Units	Cost	95% CI	Positive/ $n$
Own task block only	first 8	$+1.79$	$[1.08,2.49]$	8/8
Own task block only	remaining 12	$+1.76$	$[1.09,2.42]$	11/12
Other seen task blocks only	first 8	$+3.72$	$[2.66,4.79]$	8/8
Other seen task blocks only	remaining 12	$+3.86$	$[3.11,4.61]$	12/12

Appendix

Table S13: Both the record’s own task block and other seen blocks carry

H

correspondence. CIFAR-100 DER++; the first eight units and, specified before their runs, the remaining twelve units of the main cohort. Full-minus-variant cost in pp with 95% intervals; one-sided replication tests Holm-adjusted over the two restrictions (

p<10^{-4}

each).

Figure S1:

H

assignment remains costly after reducing its residual energy, while raised-energy

N

reassignment remains equivalent. Original-minus-reassigned accuracy on CIFAR-100, with twelve paired units per comparison. The

H

rows and “

N

: matched to

H

” use within-task-block reassignment in one cohort; “

N

: native” is the across-

N

shift, from a separate cohort.

H

intervals are 95%;

N

intervals are 90%, with shading indicating the

\pm 1

pp equivalence band.

Comparison	$n$	$J$ (pp)	$L_{\rm task}$ reduction	$L_{\rm within}$ reduction
Full minus Delete $N$
C100 DER++	20	$+15.16$	$+1.77\ [1.69,1.85]$	$-0.17\ [-0.18,-0.17]$
C100 DER++, second cohort	12	$+13.89$	$+1.67\ [1.60,1.74]$	$-0.19\ [-0.20,-0.17]$
C100 DER	12	$+23.74$	$+17.86\ [17.32,18.41]$	$-0.27\ [-0.28,-0.25]$
TinyImageNet DER++	8	$+8.41$	$+2.77\ [2.67,2.87]$	$-0.31\ [-0.34,-0.28]$
Pooled reference minus Delete $N$

Appendix

Table S14: Exact cross-entropy split: first-policy accuracy gain

J

(pp) and loss reductions (nats), with paired 95% intervals. C100 denotes CIFAR-100; “res.” denotes residual energy.

Setting	Comparison	$n$	Before	After	Change [95% CI]
C100 DER++	Full vs Delete $N$	20	15.16	15.67	$+0.51\ [0.08,0.93]$
C100 DER	Full vs Delete $N$	12	23.74	15.05	$-8.70\ [-9.33,-8.06]$
TinyImageNet DER++	Full vs Delete $N$	8	8.41	6.60	$-1.81\ [-2.00,-1.62]$
C100 DER++	Full vs Perm $H$	20	4.71	4.77	$+0.06\ [-0.08,0.20]$
C100 DER	Full vs Perm $H$	8	4.56	4.60	$+0.04\ [-0.18,0.25]$
TinyImageNet DER++	Full vs Perm $H$	4	6.09	5.49	$-0.60\ [-0.92,-0.27]$

Appendix

Table S15: Terminal task-block row-norm equalization does not remove the tested gaps. Full-minus-edited accuracy gaps in pp before and after correction; the last column is the paired 95% interval for the change in the gap.

Setting	Objective	Contrast	$n$	Difference [CI]
CIFAR-100	DER	Full $-$ Delete $N$	12	$23.7\ [22.9,24.6]_{95\%}$
CIFAR-100	DER	Full $-$ Record mean	12	$0.13\ [-0.28,0.54]_{90\%}$ †
CIFAR-100	DER	Full $-$ Fixed $N$ , $a=-1$	12	$0.05\ [-0.44,0.53]_{90\%}$ †
CIFAR-100	DER	Full $-$ Fixed $N$ , $a=-2$	12	$0.16\ [-0.40,0.72]_{90\%}$ †
CIFAR-100	DER	Full $-$ Fixed $N$ , $a=-4$	12	$-0.30\ [-0.82,0.21]_{90\%}$ †
CIFAR-100	DER	Full $-$ Perm $H$	8	$4.56\ [3.71,5.42]_{95\%}$

Appendix

Table S16: Selected replacement, deletion, and reassignment effects extend beyond the canonical comparison. Accuracy differences in pp with paired intervals at the stated level; † : 90% interval within

\pm 1

pp. Rows may use separate seeds; all use ResNet-18 except the ResNet-32 rows. Mean/Perm

H

are the block-mean and reassignment edits of Section 3.3 .

Memory	Full $-$ Delete $N$ [95% CI]	Pooled $-$ Full [90% CI]
200	$9.129\ [8.059,10.198]$	$0.344\ [-0.272,0.959]$
500	$14.329\ [13.368,15.290]$	$0.209\ [-0.591,1.008]$
2,000	$11.741\ [10.991,12.491]$	$0.121\ [-0.210,0.453]$

Appendix

Table S17: Deletion costs persist, but pooled-reference equivalence is unresolved at memory 500. Eight paired units per memory. Differences in pp; 95% intervals for deletion and 90% intervals for equivalence against

\pm 1

pp. Full is shared with the correspondence sweep.

Memory	Comparison	$J$	$A_{\rm task}$	$C_{\rm task}$
200	Full – Iso H*	$+1.166$ $[-0.919,+3.251]$	$+1.397$ $[-0.630,+3.425]$	$+6.719$ $[+5.111,+8.326]$
200	Iso – Perm H*	$+1.450$ $[+0.526,+2.374]$	$+1.378$ $[+0.437,+2.318]$	$+6.619$ $[+6.096,+7.142]$
200	Full – Perm H	$+2.616$ $[+1.075,+4.157]$	$+2.775$ $[+1.205,+4.345]$	$+13.337$ $[+11.781,+14.894]$
500	Full – Iso H*	$+3.326$ $[+2.707,+3.945]$	$+3.709$ $[+3.075,+4.343]$	$+4.929$ $[+4.001,+5.857]$
500	Iso – Perm H*	$+0.421$ $[-0.410,+1.253]$	$+0.027$ $[-0.843,+0.898]$	$+6.986$ $[+6.166,+7.806]$
500	Full – Perm H	$+3.747$ $[+3.000,+4.495]$	$+3.736$ $[+2.794,+4.679]$	$+11.915$ $[+11.032,+12.798]$

Appendix

Table S18: Historical correspondence across memory budgets (eight paired units per memory). Each cell gives a difference in pp and a paired 95% interval. Starred contrasts cross implementations.

Memory	Comparison	$A_{\rm task}$	$C_{\rm task}$
200	Full – Delete N	$+10.259$ $[+8.798,+11.719]$	$-5.713$ $[-6.579,-4.846]$
200	Pooled – Full	$+0.316$ $[-0.669,+1.302]$	$+0.394$ $[-0.227,+1.014]$
500	Full – Delete N	$+16.020$ $[+14.957,+17.083]$	$-4.895$ $[-5.352,-4.438]$
500	Pooled – Full	$+0.176$ $[-0.860,+1.212]$	$+0.260$ $[-0.537,+1.057]$
2000	Full – Delete N	$+13.976$ $[+13.105,+14.847]$	$-4.019$ $[-4.238,-3.799]$
2000	Pooled – Full	$+0.260$ $[-0.256,+0.776]$	$-0.226$ $[-0.573,+0.120]$

Appendix

Table S19: Task-coordinate readouts of the pooling sweep, computed by the diagnostic pipeline (eight paired units per memory). Each cell gives a difference in pp and a paired 95% interval, not an equivalence interval.

Memory	Comparison	Older tasks	Latest task
200	Full – Delete N	$+10.664$ $[+9.363,+11.965]$	$-4.688$ $[-8.199,-1.176]$
200	Pooled – Full	$+0.424$ $[-0.478,+1.325]$	$-0.375$ $[-1.690,+0.940]$
500	Full – Delete N	$+16.789$ $[+15.548,+18.030]$	$-7.812$ $[-10.092,-5.533]$
500	Pooled – Full	$+0.249$ $[-0.952,+1.450]$	$-0.150$ $[-1.538,+1.238]$
2000	Full – Delete N	$+14.199$ $[+13.255,+15.142]$	$-10.375$ $[-13.040,-7.710]$
2000	Pooled – Full	$+0.046$ $[-0.394,+0.486]$	$+0.800$ $[-0.906,+2.506]$

Appendix

Table S20: Older-task benefits coexist with latest-task costs across memory capacities (eight paired units per memory). Each cell gives a frozen-endpoint difference in pp and a descriptive paired 95% interval, not an equivalence interval.

Contrast	Difference	CI	Positive/ $n$
Full $-$ Flat (primary)	$+6.36$	$[4.87,7.84]_{90\%}$	8/8
Full $-$ None	$+6.05$	$[4.70,7.40]_{95\%}$	8/8
Flat $-$ None	$-0.31$	$[-1.84,1.22]_{95\%}$	5/8
Arm means: Full 49.58, Flat 43.22, None 43.53 (seed SD 0.99, 1.90, 1.86).

Appendix

Table S21: A refreshed later-class target is not replaceable by its mean. X-DER on CIFAR-100, eight paired seeds; final class-incremental accuracy and paired differences in pp. The primary Full

-

Flat test uses a 90% interval (equivalence margin

\pm 1

pp); the secondary contrasts use 95% intervals.

Cohort	Contrast	$n$	As trained	After offset	Block	Cond. CE	Spec.
Withheld-image cohort (Fig. 4 a)	Full $-$ Delete $N$	8	$+14.89\ [13.81,15.97]$	$+1.76\ [0.74,2.79]$	$+1.99$	$+0.04$	P
	Full $-$ Fixed	8	$+0.19\ [-0.47,0.85]$	$+0.34\ [-0.13,0.80]$	$+0.35$	$-0.02$	P
	Full $-$ Perm $H$	8	$+4.29\ [3.45,5.12]$	$+3.98\ [3.47,4.49]$	$+4.62$	$-0.46$	P
	Perm $H-$ Delete $N$	8	$+10.60\ [9.23,11.97]$	$-2.22\ [-3.18,-1.25]$	$-2.63$	$+0.50$
Withheld-image cohort, ResNet-32	Full $-$ Delete $N$	8	$+6.06\ [5.34,6.79]$	$-1.55\ [-2.17,-0.93]$	$-1.76$	$-0.08$	P
	Full $-$ Fixed	8	$-0.49\ [-1.44,0.46]$	$+0.39\ [-0.18,0.95]$	$+0.01$	$+0.04$	P

Appendix

Table S22: Intervention costs before and after the task-level offset. DER++ on CIFAR-100 (ResNet-18 unless stated) and TinyImageNet. Accuracy costs in pp with paired 95% intervals; “Block” is the difference in group-restricted accuracy (pp) and “Cond. CE” in conditional cross-entropy (nats), both after the offset and unchanged by it. The fixed anchor uses

a=-2

. The first three cohorts fit the offset on 500 training images withheld from training; the others use five-fold test cross-fitting.

Contrast	Backbone	One offset	Offset per task	Task oracle
Full $-$ Perm $H$	ResNet-18 †	$+3.98\ [3.47,4.49]$	$+3.82\ [3.21,4.43]$	$+12.38\ [11.15,13.60]$
	ResNet-32	$+4.19\ [3.68,4.70]$	$+4.35\ [3.95,4.76]$	$+13.41\ [12.95,13.87]$
	Reduced ResNet-18	$+3.19\ [2.21,4.17]$	$+3.24\ [2.36,4.12]$	$+11.14\ [10.50,11.79]$
Full $-$ Delete $N$	ResNet-18 †	$+1.76\ [0.74,2.79]$	$-2.45\ [-3.39,-1.50]$	$-5.03\ [-5.41,-4.64]$
	ResNet-32	$-1.55\ [-2.17,-0.93]$	$-2.98\ [-3.52,-2.44]$	$-7.10\ [-7.72,-6.47]$
	Reduced ResNet-18	$-0.59\ [-1.46,0.29]$	$-2.54\ [-3.24,-1.84]$	$-6.12\ [-6.89,-5.34]$

Appendix

Table S23: Withheld-image cohorts under one offset and one offset per task. DER++ on CIFAR-100, eight seeds per backbone; accuracy costs in pp with paired 95% intervals. Both offsets are fitted on the same 500 withheld training images; the task oracle restricts prediction to the true task and is unchanged by any per-task offset. † Per-task offsets and task oracle computed post hoc; prespecified for the other two backbones.

Cohort	$I_{J}$ as trained	$I_{J}$ after offset	Both $-$ Neither as trained	Both $-$ Neither after offset
ResNet-18, $a=-2$	$+6.38\ [5.33,7.43]$	$-0.42\ [-0.94,0.11]$	$+14.23\ [13.39,15.07]$	$+0.77\ [0.09,1.46]$
ResNet-18, $a=-4$	$+7.01\ [5.67,8.35]$	$+0.45\ [-0.36,1.25]$	$+14.69\ [13.69,15.68]$	$+0.58\ [-0.11,1.27]$
ResNet-18, $a=-2$ , fresh (cohort of Fig. 4 b)	$+5.72\ [4.20,7.24]$	$+1.23\ [0.29,2.17]$	$+14.31\ [13.01,15.61]$	$+1.81\ [0.86,2.76]$
ResNet-32, $a=-2$	$+4.39\ [3.16,5.61]$	$-0.71\ [-1.51,0.09]$	$+6.86\ [6.29,7.43]$	$-1.55\ [-2.27,-0.83]$
ResNet-32, $a=-2$ , fresh (cohort of Fig. 4 b)	$+3.95\ [2.37,5.54]$	$-1.09\ [-2.25,0.08]$	$+7.35\ [6.26,8.43]$	$-1.54\ [-2.28,-0.79]$

Appendix

Table S24: The level–dispersion interaction depends on the prediction rule. Fixed-anchor factorial cohorts (DER++, CIFAR-100, eight seeds each); pp with paired 95% intervals. The raw interaction was prespecified in every cohort; the offset readout was prespecified in the second-anchor cohort (secondary) and in both fresh cohorts, where Both

<

Neither after the offset was also a prespecified secondary test on ResNet-32 (one-sided

p=.0009

Contrast	As trained	After offset	Block	Cond. CE
DER++ $-$ ER	$+15.18\ [14.64,15.72]$	$+5.70\ [5.25,6.14]$	$+5.86$	$-1.19$
DER $-$ ER	$+13.46\ [13.01,13.91]$	$+4.85\ [4.45,5.24]$	$+5.11$	$-1.16$
ER-ACE $-$ ER	$+12.78\ [12.43,13.13]$	$+0.27\ [-0.14,0.68]$	$+0.96$	$-0.83$
DER++ $-$ DER	$+1.72\ [1.08,2.36]$	$+0.85\ [0.47,1.23]$	$+0.75$	$-0.03$
DER++ $-$ ER-ACE	$+2.40\ [1.84,2.95]$	$+5.42\ [5.06,5.78]$	$+4.89$	$-0.36$
Accuracy as trained $\rightarrow$ after offset (%): ER 21.8 $\rightarrow$ 36.0, ER-ACE 34.6 $\rightarrow$ 36.3,

Appendix

Table S25: Standard replay methods before and after the offset. CIFAR-100, ResNet-18, memory 500, the principal recipe for every method (DER and DER++ with

\alpha=0.3

, DER++ with

\beta=0.5

, ER-ACE with its default cosine classifier); 32 paired seeds, no model excluded. pp with paired 95% intervals. The six prespecified tests (the offset reduces each gap vs. ER, and removes more than half of it) all pass under Holm correction; the ER-ACE equivalence is secondary.

Stage	Student	As trained	After offset	Cond. CE
E0	Mass	$-9.66\ [-10.34,-8.98]$	$-0.13\ [-0.41,0.16]$	$0.50$
	Within	$+6.06\ [5.37,6.74]$	$+6.36\ [5.64,7.08]$	$0.79$
	Both	$-0.93\ [-1.57,-0.30]$	$+5.51\ [4.73,6.28]$	$0.98$
E1	Mass	$+0.55\ [-0.02,1.11]$	$+0.77\ [0.15,1.38]$	$0.58$
	Within	$+5.90\ [5.45,6.35]$	$+5.23\ [4.70,5.76]$	$0.99$
	Both	$+7.20\ [6.77,7.63]$	$+6.40\ [5.84,6.97]$	$1.08$

Appendix

Table S26: Distillation terms before and after the offset. Accuracy gain over label-only training in pp with paired 95% intervals (eight teachers); label-only students reach 37.7%. The last column is the reduction in conditional cross-entropy relative to label-only training (nats).

Contrast	E0	E1	Spec.
W $-$ label only	$+6.29\ [5.53,7.04]$	$+5.21\ [4.19,6.22]$
W $-$ Wperm	$+7.50\ [6.97,8.03]$	$+8.13\ [7.34,8.91]$	P
W $-$ Wflat	$+6.84\ [6.28,7.39]$	$+5.66\ [4.77,6.55]$	P
W $-$ Wtmpl	$+4.11\ [3.60,4.62]$	$+3.40\ [2.13,4.68]$	P
Wperm $-$ label only	$-1.21\ [-1.97,-0.45]$	$-2.92\ [-3.61,-2.24]$
Wflat $-$ label only	$-0.55\ [-1.12,0.02]$	$-0.46\ [-1.21,0.30]$

Appendix

Table S27: The within-block gain needs class-specific, record-specific conditionals. Accuracy differences after the offset in pp with paired 95% intervals (eight teachers); label-only students reach 38.0% after the offset. P marks the prespecified family (E1).

	E0 mass $-$ label only		E0 within $-$	E1 within $-$	E1 within $-$
Partition	raw	after	label only	label only	mass
Original (8, P)	$-9.37$	$-0.13$	$-0.12\ [-0.24,0.00]$	$+0.28\ [0.17,0.38]$	$+0.34\ [0.24,0.44]$
Split 2 (4)	$-7.90$	$-0.43$	$-0.09\ [-0.30,0.11]$	$-0.03\ [-0.15,0.09]$	$-0.20\ [-0.44,0.04]$
Split 3 (4)	$-6.59$	$-0.40$	$-0.16\ [-0.47,0.15]$	$-0.12\ [-0.31,0.07]$	$-0.45\ [-0.83,-0.08]$

Appendix

Table S28: On text, offset repair transfers; within-block gains are small and depend on the partition. DistilBERT on AG News; accuracy differences in pp, after the offset unless marked raw, with paired 95% intervals. Halves: original {World, Sports} / {Business, Sci/Tech}; split 2 {World, Business} / {Sports, Sci/Tech}; split 3 {World, Sci/Tech} / {Sports, Business}. Numbers of teachers in parentheses; P marks the prespecified cohort, and the others are development cohorts.

Contrast	As trained	One offset	Per-task offsets	$+$ scale	Task oracle
Full $-$ Perm $H$ (ResNet-18, 16)	$+4.17$	$+4.07$	$+4.08\ [3.64,4.51]$	$+4.05$	$+11.93$
DER++ $-$ ER (32)	$+15.18$	$+5.70$	$+5.24\ [4.78,5.70]$	$+5.62$	$+3.41$
DER $-$ ER (32)	$+13.46$	$+4.85$	$+4.42\ [4.01,4.83]$	$+4.83$	$+2.85$
ER-ACE $-$ ER (32)	$+12.78$	$+0.27$	$+0.09\ [-0.33,0.50]$	$+0.39$	$-2.04$
Interaction $I_{J}$ , ResNet-18 (8)	$+5.72$	$+1.21$	$+0.28\ [-0.33,0.89]$	$+0.36$	$-1.78$
Interaction $I_{J}$ , ResNet-32 (8)	$+3.95$	$-1.08$	$+0.44\ [-0.66,1.55]$	$+0.56$	$-2.17$

Appendix

Table S29: Costs and gaps under corrections of increasing freedom. Accuracy differences in pp with root-paired 95% intervals for the per-task offsets (numbers of seeds in parentheses). Columns: as trained; one final-task offset; one offset per task; per-task offsets and a final-task scale; accuracy with prediction restricted to the true task, which no per-task offset changes. The ResNet-18 reassignment row pools the two paired cohorts; the one-offset column here is fitted by cross-entropy on half-splits and so differs slightly from Tables S22 and S24 .

Figure S2: $N$ matching primarily supports task selection, whereas $H$ assignment primarily supports within-task discrimination. (a) Change in task accuracy $A_{\rm task}$ and within-task accuracy $C_{\rm task}$ : Full minus Delete $N$ (amber) and Full minus reassigned $H$ (blue), with paired seeds and marginal 95% intervals; the image pairs illustrate the decision each accuracy scores. (b) The exact split of Equation 5 : loss removed by keeping $N$ matching (top) and the $H$ assignment (bottom), on different scales; the DER task-term reduction, 17.9 nats, lies beyond the axis. Seeds (CIFAR-100/DER/TinyImageNet): 20/12/8 for deletion, 20/8/4 for reassignment. Keeping $N$ matching reduces the task term by 1.77 nats while the within-task term rises by 0.17; keeping the $H$ assignment reduces mainly the within-task term, which accounts for 61%–85% of the loss reduction across the nine $H$ comparisons (Table S14 ).

Figure S3: Under the raw prediction rule, both retained components outperform either alone. (a,b) Accuracy gain of each policy over Delete $N$ (“neither”; “spread” is the dispersion term), with seeds and 95% paired intervals; the dashed line is the sum of the two single-term gains and the arrow the interaction $I_{J}$ . “Both” uses Generic on CIFAR-100 (20 seeds) and Record mean on TinyImageNet (8 seeds). (c) Since $J=C_{\rm task}-X$ , $I_{J}$ is the sum of a cross-task part ( $-X$ ) and a within-task part ( $C_{\rm task}$ ), shown with 95% intervals; the image pairs illustrate the two error types. On eight CIFAR-100 seeds, Both exceeds doubled Level only by 12.5 pp and doubled Dispersion only by 7.3 (Table S10 ).

Explore similar work

Jun 28, 2026cs.LG

Prototype Latent World Model Replay for Class-Incremental Learning

Class-incremental learning requires a model to learn new classes while preserving decision regions for old ones. This is difficult when raw old samples are no longer available. We propose Prototype Latent World Model Replay, a memory-free framework that stores old classes as distributions over stable hidden states rather than as images. A frozen ImageNet-pretrained encoder maps each image into a latent state space. In this space, each class is summarized by several prototype-centered distributions with class-specific variances. When new classes arrive, the model samples old latent states from this prototype world model. It then trains a lightweight adapter and classifier using both sampled old states and real new-class features. We also add a supervised contrastive term in the adapter space to promote intra-class compactness and old-new class separation. On Split CIFAR-100, our method improves over fine-tuning under Inc5, Inc10, and Inc20 without storing raw exemplars. The full Ours-LWM+Con model raises LastAcc from 4.55% to 31.64%, from 9.06% to 37.06%, and from 16.96% to 43.10% in Inc5, Inc10, and Inc20, respectively. It also achieves AvgAcc of 45.86%, 52.19%, and 56.18%. Ablation and retention analyses show that stable latent-state replay is the main source of the gain. Contrastive separation further refines the old-new geometry. These results suggest that prototype latent memory preserves reusable class-state distributions, rather than only fitting the current classifier.

Weizhi Nie, Hui Wang, Weijie Wang +1

Tianjin University

Class-Incremental Learning Learnable Prototypes Imagenet Latent World Models World Models Long-Term Memory

Ask PDF

May 14, 2026cs.LG

Understanding Imbalanced Forgetting in Rehearsal-Based Class-Incremental Learning

Neural networks suffer from catastrophic forgetting in class-incremental learning (CIL) settings. Rehearsal

\unicode{x2013}

replaying a subset of past samples

\unicode{x2013}

is a well-established mitigation strategy. However, recent results suggest that, despite balanced rehearsal allocation, some classes are forgotten substantially more than others. Despite its relevance, this imbalanced forgetting phenomenon remains underexplored. This work shows that imbalanced forgetting arises systematically and severely in rehearsal-based CIL and investigates it extensively. Specifically, we construct, from a principled analysis, three last-layer coefficients that capture different gradient-level sources of interference affecting each past class during an incremental step. We then demonstrate that, together, they reliably predict how past classes will rank in terms of forgetting at the end of that step. While predictive performance alone does not establish causality, these results support the interpretation of the coefficients as a plausible mechanistic account linking last-layer gradient-level interactions during training to class-level forgetting outcomes. Notably, one coefficient

\unicode{x2013}

capturing self-induced interference

\unicode{x2013}

emerges as the strongest predictor, with controlled experiments providing evidence consistent with this coefficient being influenced by the new-class interference coefficient. Overall, our findings provide valuable insights and suggest promising directions for mitigating imbalanced forgetting by reducing class-wise disparities in the identified sources of interference.

Alberto Tamajo, Srinandan Dasmahapatra, Rahman Attar

School of Electronics and Computer Science, University of Southampton, SO17 1BJ Southampton, United Kingdom

Catastrophic Forgetting Class-Incremental Learning Forgetting Intra-Class Retraining Interference+1

Ask PDF

May 9, 2026cs.LG

Classification-Head Bias in Class-Level Machine Unlearning: Diagnosis, Mitigation, and Evaluation

Class-level machine unlearning aims to remove the influence of specified classes while preserving model utility on retained classes. Existing methods are commonly evaluated by retain-set accuracy, forget-set accuracy, and unlearning time, but these metrics provide limited insight into how forgetting is achieved internally. In this paper, we reveal a bias-dominated shortcut in class-level unlearning: the prediction of forgotten classes can be suppressed by decreasing the corresponding bias terms in the final classification head. We first analyze the gradient dynamics of classification-head biases under softmax cross-entropy training, explaining why retain-set-only optimization tends to reduce the biases of absent classes. Based on this observation, we introduce BiasShift as a diagnostic baseline, showing that simple bias manipulation can satisfy conventional unlearning metrics while leaving abnormal bias patterns that reveal forgotten labels. To mitigate excessive forgotten-class bias suppression, we propose two bias-aware mechanisms, namely Two-Stage Bias Gradient Reversal Mechanism (TS-BGRM) and Lower-Bound Hinge Regularization (LB-HR). We further introduce three bias-oriented metrics, including Bias Stability Coefficient (BSC), Median Bias Gap (MBG), and Minimal Bias Score (MBS), to quantify bias dependence and potential leakage. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that the proposed methods maintain competitive unlearning performance while producing more stable bias distributions. We have released our code at {https://github.com/zwd2024/Beyond-the-Shadow-of-Bias-From-Classification-Head-Bias-to-Parameter-Redistribution}.

Weidong Zheng, Kongyang Chen, Yuanwei Guo +1

School of Computer Science and Cyber Engineering, Guangzhou University, Guangzhou 510335, China · School of Artificial Intelligence, Guangzhou University, Guangzhou 510006, China · Pazhou Lab, Guangzhou 510006, China +1

Machine Unlearning Exact Unlearning Biases

Ask PDF