cs.LGSep 28, 2026

Early Learning Shapes Later Directions Of Representation Change In Continual Learning

Authors: Yuantao Deng, Jinnuo Liu, Kaizhen Tan, Yuchen Liu

Organizations: New York University · Xi’an Jiaotong University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Representations continually change as a network learns new tasks. We ask whether early representational changes naturally form a geometric structure that continues to shape later learning. We identify a low-dimensional subspace of early representation drift, which we call a scaffold, and test whether it is reused across subsequent tasks. Across four pretrained visual encoders and two datasets, later representational changes consistently favor this early-defined subspace over matched random alternatives. This reuse is history-dependent: when networks experience different early tasks but identical later training inputs, each network preferentially reuses the scaffold induced by its own learning history. The same preference appears in individual optimizer updates, even though the network's dominant local response directions shift away from the original scaffold. Finally, constraining motion within the scaffold slows new-task acquisition more than matched random constraints, while effects on old-task retention are less consistent. In summary, these results suggest that early experience leaves a persistent geometric imprint on how neural networks adapt to future tasks.Code is available at https://github.com/YuantaoDeng/latent-scaffold.

Figures & tables

Figure 1: Early learning reveals a reusable scaffold. (a) Early representational changes define a low-dimensional subspace, the scaffold. (b) Later learning preferentially reuses these directions. Restricting motion within the scaffold impairs new-task acquisition more than energy-matched random constraints.
Figure 2: The same early scaffold remains present in later task drift. Each panel shows one encoder’s absolute drift coverage on CIFAR-10 (muted-coral circles) and CIFAR-100 (grey-blue squares). Solid curves with filled markers show the fixed scaffold; dashed curves with open markers show classifier-matched random controls. The 1→21\to 2 points test transfer across histories; later points test persistence after further learning. Shaded bands show the standard error for each coverage curve.
Figure 3: Later drift favors the own-history scaffold under identical training inputs. Absolute drift coverage on the first common task (task three) for CIFAR-10 (left) and CIFAR-100 (right). For each model, the left and right bar pairs show test branches XX and YY , respectively; solid and pale bars use the own-history and other-history scaffolds. Bars average six paired test seeds; error bars show nominal 95% paired-seed bootstrap intervals, conditional on the fitted scaffolds. Appendix D reports the paired own-history contrasts.
Figure 4: Response alignment declines after early learning. Orientation overlap between the fixed scaffold and pooled local response subspaces on CIFAR-10 (a) and CIFAR-100 (b). The pale region and bracket mark the 1→21\to 2 interval used for scaffold construction. Shaded bands show nominal 95% intervals from 2,000 estimation-order/run resamples that refit both subspaces.
Figure 5: Sampled optimizer updates continue to favor the scaffold. Absolute AdamW feature-motion coverage in early, middle, and late windows of the final task on CIFAR-10 (a) and CIFAR-100 (b). Solid curves show scaffold coverage; dashed curves show classifier-matched random coverage. Each history contributes the mean coverage of three neighboring updates, and curves average twelve test histories per condition. Shaded bands show ±\pm one hierarchical order/run bootstrap standard error, conditional on the fitted scaffold. Paired coverage-gain intervals are reported in Appendix E.2 .
  New-task acquisition (%)Old-task retention (%)
ModelDataScaffoldRandomRandomScaffoldRandomRandom
   Same λ\lambdaEnergy matched Same λ\lambdaEnergy matched
ResNetC1067.43 [61.60, 75.33]74.40 [69.00, 82.17]89.02 [85.85, 93.85]68.83 [66.42, 70.75]69.23 [67.67, 70.83]71.29 [69.94, 72.50]
 C10077.44 [76.16, 78.88]83.51 [82.38, 84.60]91.34 [90.15, 92.44]61.41 [59.03, 64.84]61.10 [58.78, 64.04]59.62 [57.67, 61.68]
DeiTC1070.60 [64.57, 79.40]70.95 [64.70, 79.80]82.32 [76.00, 87.88]68.96 [65.25, 71.88]69.31 [66.25, 71.62]71.79 [68.87, 73.67]
 C10082.81 [81.24, 84.19]83.60 [82.14, 84.69]91.18 [89.36, 92.48]56.83 [54.00, 59.93]56.53 [53.17, 60.41]56.32 [54.58, 58.31]
Table 1: Acquisition costs persist after matching removed drift energy. New-task acquisition averages accuracy over epochs one through five; old-task retention averages final accuracy across all old tasks. Entries show mean accuracy (%) and nominal 95% hierarchical order/run bootstrap intervals (20,000 draws). Each condition averages six paired histories and averages random branches within each history. Scaffold and same- λ\lambda random use λ=100\lambda=100 ; energy-matched random matches removed drift energy. Within each row and metric, darker shading indicates higher accuracy among the three methods. Shading shows relative ranks, not statistical significance. Appendix H reports paired scaffold-minus-random contrasts.
Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

ModelFeature dimensionCIFAR-10 rankCIFAR-100 rank
ResNet-1851288
DeiT-Tiny192820
MobileNetV3-S1024248
MLP-Mixer-B/1676888
Appendix
Table 2: Fixed scaffold dimensions. The same dimensions are retained in all follow-up conditions, including each pair of content branches and separately fitted no-replay scaffolds.
ModelCIFARr∗r_{*}Selection difference (pp)
ResNet-181082.1744
ResNet-1810081.8322
DeiT-Tiny1081.3402
DeiT-Tiny100200.0147
MobileNetV3-S10240.0362
MobileNetV3-S10080.1800
Appendix
Table 3: Rank-selection scores on estimation histories. The score 100g(r∗)100g(r_{*}) is a coverage difference in percentage points. A positive mean determines eligibility. Each value averages six held-out estimation histories at the selected rank.
  Minimum admissible rankMinimum gain
ModelCIFAR5678over random
ResNet-1810567811.24
ResNet-1810056783.83
DeiT-Tiny1056787.54
DeiT-Tiny100202020204.16
MobileNetV3-S10242424244.50
Appendix
Table 4: Sensitivity to the minimum admissible rank. The four middle columns give the selected rank after changing only the minimum. The final column gives the smallest held-out estimation coverage gain over the uniform-random expectation across these four choices (percentage points).
ModelCIFARTaskX←XX\leftarrow XX←YX\leftarrow YY←XY\leftarrow XY←YY\leftarrow Y
ResNet-1810326.8522.8620.8826.38
ResNet-1810422.8720.7620.1724.23
DeiT-Tiny10329.5522.4122.3731.21
DeiT-Tiny10418.9418.2619.0821.65
MobileNetV3-S10313.7111.9014.0615.96
MobileNetV3-S10411.8811.1512.9314.17
Appendix
Table 5: Complete cross-coverage means (%) during common tasks. Rows of each 2×22\times 2 comparison correspond to the test branch and columns to the scaffold’s early history.
ModelCIFARTaskΔX\Delta_{X}ΔY\Delta_{Y}
ResNet-181033.99 [3.51, 4.54]5.51 [5.01, 5.99]
ResNet-181042.12 [1.53, 2.70]4.06 [3.74, 4.38]
DeiT-Tiny1037.14 [4.83, 9.75]8.84 [7.36, 10.62]
DeiT-Tiny1040.68 [0.22, 1.18]2.58 [2.01, 3.21]
MobileNetV3-S1031.81 [1.31, 2.45]1.90 [1.48, 2.33]
MobileNetV3-S1040.73 [0.14, 1.30]1.24 [1.04, 1.44]
Appendix
Table 6: Own-history coverage advantages in percentage points [95% paired-seed interval]. Each condition uses six separate test pairs, resampled jointly 10,000 times, conditional on the fitted scaffolds.
Figure 6: History-specific scaffold preference persists through a second common task. Absolute coverage on CIFAR-10 task four. For each model, the left pair of bars evaluates test branch XX and the right pair evaluates test branch YY ; within each pair, the solid left bar uses the own-history scaffold and the pale right bar uses the other-history scaffold. Bars average six paired test seeds. Error bars are nominal 95% intervals from 10,000 paired-seed bootstrap draws, conditional on the fitted scaffolds. Table 6 reports the corresponding paired contrasts.
ModelCIFAREarlyMiddleLate
ResNet-18109.23 [7.14, 10.88]7.92 [6.04, 9.43]8.04 [6.04, 9.57]
ResNet-181008.02 [7.08, 9.17]5.50 [5.19, 5.83]6.17 [5.84, 6.54]
DeiT-Tiny101.34 [0.78, 2.02]1.20 [0.83, 1.63]1.07 [0.73, 1.45]
DeiT-Tiny1001.68 [1.35, 2.05]1.46 [1.17, 1.74]1.36 [0.92, 1.92]
MobileNetV3-S105.18 [4.36, 6.17]4.12 [3.80, 4.46]4.10 [3.79, 4.45]
MobileNetV3-S1001.85 [1.71, 2.00]1.84 [1.67, 2.05]1.85 [1.69, 2.02]
Appendix
Table 7: Actual AdamW coverage gains over classifier-matched random subspaces (percentage points [95% hierarchical interval]). Windows average three neighboring updates within each history.
ModelCIFARClass-centered logitsLoss change
ResNet-18106.09 [3.06, 8.82]7.12 [-1.59, 15.45]
ResNet-1810013.48 [12.14, 15.05]17.44 [14.93, 20.41]
DeiT-Tiny100.83 [-0.01, 1.66]1.81 [0.01, 3.75]
DeiT-Tiny1001.50 [0.47, 2.42]1.30 [0.03, 2.73]
MobileNetV3-S103.89 [1.59, 6.34]6.71 [3.56, 10.09]
MobileNetV3-S1001.13 [0.62, 1.69]0.54 [-0.72, 2.06]
Appendix
Table 8: Frozen-classifier reconstruction gains over the matched random control (percentage points [95% paired hierarchical interval]). These baseline endpoint measurements are distinct from the multi-position update continuations.
Figure 7: Recurrence without replay varies across conditions. Fixed-scaffold coverage minus the transition-start classifier-matched random expectation. Error bars are nominal 95% hierarchical order/run bootstrap intervals over twelve histories, conditional on each fitted scaffold.
ModelTransitionMatched-control gain
ResNet-181→21\to 217.44 [14.63, 20.11]
ResNet-182→32\to 312.48 [11.15, 13.83]
ResNet-183→43\to 411.61 [9.76, 13.38]
ResNet-184→54\to 511.10 [9.63, 12.65]
DeiT-Tiny1→21\to 20.17 [-2.20, 2.55]
DeiT-Tiny2→32\to 31.59 [-0.68, 4.35]
Appendix
Table 9: Complete no-replay transition gains (percentage points [95% hierarchical interval]). The matched-control gain uses total centered drift energy.
  New-task acquisitionFirst-task retention
Modelλ\lambdaversus Rversus Bversus Rversus B
ResNet-1810-11.78 [-13.63, -8.83]-13.63 [-17.40, -9.07]+0.67 [-1.08, +3.08]+0.83 [-3.33, +4.50]
ResNet-18100-6.97 [-8.15, -5.80]-20.93 [-24.53, -16.20]+1.67 [-1.00, +4.83]+0.33 [-3.00, +2.50]
DeiT-Tiny10-0.62 [-2.15, +1.00]-3.03 [-5.57, +0.40]+0.83 [-2.50, +4.50]+4.00 [-2.67, +10.00]
DeiT-Tiny100-0.35 [-2.93, +1.47]-19.03 [-23.10, -14.70]+1.50 [-2.50, +4.50]+1.67 [-7.67, +8.50]
MobileNetV3-S10-0.40 [-2.03, +1.20]-1.30 [-14.67, +5.07]-0.08 [-4.92, +5.42]-1.67 [-18.67, +12.33]
Appendix
Table 10: CIFAR-10 suppression at λ=10\lambda=10 and 100: scaffold minus uniform random (R) or normal training (B), in percentage points [95% paired hierarchical interval]. New-task accuracy averages five epochs; first-task accuracy is final.
  New-task acquisitionFirst-task retention
Modelλ\lambdaversus Rversus Bversus Rversus B
ResNet-1810-6.74 [-7.25, -6.23]-7.37 [-7.96, -6.79]-0.37 [-1.90, +1.37]-0.40 [-1.80, +1.73]
ResNet-18100-6.07 [-7.00, -5.21]-14.11 [-15.11, -13.29]+0.17 [-0.85, +1.23]-1.77 [-4.13, +0.70]
DeiT-Tiny10-0.43 [-0.65, -0.01]+1.38 [+0.01, +2.27]-0.33 [-1.65, +1.15]+4.43 [+0.57, +8.80]
DeiT-Tiny100-0.79 [-1.36, -0.28]-6.75 [-8.09, -5.65]-0.13 [-1.10, +0.77]+3.00 [-0.33, +6.37]
MobileNetV3-S10-1.30 [-2.64, -0.17]-2.60 [-5.09, -0.94]+1.88 [-1.03, +5.32]+2.47 [-4.07, +8.97]
Appendix
Table 11: CIFAR-100 suppression at λ=10\lambda=10 and 100: paired acquisition and retention differences, using the definitions in Table 10 .
  Energy reduction (%)  
ModelCIFARScaffoldRandomMax. error (pp)Random λ\lambda
ResNet-181089.4989.740.920.21–0.87
ResNet-1810095.3395.690.880.21–1.00
DeiT-Tiny1097.5997.610.7310.00–100.00
DeiT-Tiny10098.8798.510.8710.00
MobileNetV3-S1094.3594.330.966.50–36.26
Appendix
Table 12: Matching drift-energy reductions on old training-memory examples. Reduction is 100[E0(U)−Eb(U)]/E0(U)100[E_{0}(U)-E_{b}(U)]/E_{0}(U) . Means average six histories and, for random constraints, two bases per history. Maximum error is the largest matching discrepancy across individual random branches.
  New-task acquisitionFirst-task retentionAll-old-task retention
ModelCIFARversus Rversus Bversus Rversus R
ResNet-1810-21.58 [-25.15, -17.75]-20.93 [-24.53, -16.20]-1.75 [-4.83, +0.17]-2.46 [-5.63, +0.67]
ResNet-18100-13.90 [-14.85, -13.05]-14.11 [-15.11, -13.29]-1.07 [-3.70, +1.57]+1.79 [+0.39, +3.40]
DeiT-Tiny10-11.72 [-17.43, -4.25]-19.03 [-23.10, -14.70]-0.50 [-4.58, +3.50]-2.83 [-5.08, -0.85]
DeiT-Tiny100-8.37 [-8.95, -7.63]-6.75 [-8.09, -5.65]-1.75 [-3.92, -0.03]+0.51 [-0.64, +1.72]
MobileNetV3-S10-6.47 [-14.95, -0.50]-8.37 [-29.87, +2.93]+1.25 [-4.00, +7.67]+0.67 [-2.75, +3.21]
Appendix
Table 13: Energy-matched intervention: scaffold minus random (R) or normal training (B), in percentage points [95% paired hierarchical interval]. Acquisition averages five epochs; retention uses the final epoch. The scaffold coefficient is 100.
ModelAfter taskT1T2T3T4T5
ResNet-18194.00––––
 288.4295.17–––
 382.2584.4294.58––
 478.0879.5875.5893.83–
 569.3374.3374.8377.7593.42
DeiT-Tiny189.08––––
Appendix
Table 14: CIFAR-10 baseline task-accuracy matrix. Rows indicate the task just completed; columns indicate the evaluated task.
ModelAfter taskT1T2T3T4
ResNet-18192.62–––
 276.8891.00––
 357.9868.3892.10–
 450.3358.0066.0393.13
DeiT-Tiny191.62–––
 271.43NR––
Appendix
Table 15: CIFAR-100 baseline task-accuracy matrix. Rows indicate the task just completed; columns indicate the evaluated task.
ModelBranchAfter taskT1T2T3T4
ResNet-18X195.17–––
  296.8396.83––
  391.500.0095.50–
  485.830.0090.5079.33
 Y195.17–––
  292.0096.33––
Appendix
Table 16: CIFAR-10 paired early-content task accuracies.
ModelBranchAfter taskT1T2T3
ResNet-18X192.20––
  275.7794.37–
  367.774.7789.07
 Y192.20––
  277.8792.07–
  368.105.9788.90
Appendix
Table 17: CIFAR-100 paired early-content task accuracies.
ModelAfter taskT1T2T3T4T5
ResNet-18193.00––––
 20.0096.75–––
 30.000.0096.83––
 40.000.000.0096.17–
 50.000.000.000.0092.08
DeiT-Tiny185.17––––
Appendix
Table 18: CIFAR-10 task accuracies without replay, averaged over twelve test histories.
ModelBranchMean NewMean OldMean AllFinal FirstFinal OldFinal All
ResNet-18Normal88.3771.6374.9863.0074.0877.63
 Scaffold67.4370.1269.5863.3368.8370.10
 Random 174.4770.0770.9561.8369.8872.60
 Random 274.3369.7870.6961.5068.5871.37
 Random74.4069.9370.8261.6769.2371.98
DeiT-TinyNormal89.6370.7174.4957.3367.9272.70
Appendix
Table 19: CIFAR-10 absolute intervention accuracies at λ=100\lambda=100 . First denotes initial-task retention.
ModelBranchMean NewMean OldMean AllFinal FirstFinal OldFinal All
ResNet-18Normal91.5558.6666.8851.6358.3367.09
 Scaffold77.4461.6565.6049.8761.4166.73
 Random 183.7561.6467.1749.8761.3868.31
 Random 283.2761.4066.8749.5360.8267.70
 Random83.5161.5267.0249.7061.1068.00
DeiT-TinyNormal89.5549.6759.6441.2046.4657.09
Appendix
Table 20: CIFAR-100 absolute intervention accuracies at λ=100\lambda=100 . First denotes initial-task retention.
ModelCIFARFirst versus ROld versus ROld versus B
ResNet-1810+1.67 [-1.00, +4.83]-0.40 [-1.75, +0.62]-5.25 [-8.75, -1.37]
ResNet-18100+0.17 [-0.85, +1.23]+0.31 [-0.47, +1.08]+3.08 [+1.90, +4.33]
DeiT-Tiny10+1.50 [-2.50, +4.50]-0.35 [-1.58, +0.48]+1.04 [-2.67, +4.54]
DeiT-Tiny100-0.13 [-1.10, +0.77]+0.31 [-0.46, +1.10]+10.38 [+8.13, +13.39]
MobileNetV3-S10+2.67 [-1.17, +6.00]-0.71 [-3.92, +2.13]+0.17 [-5.33, +5.21]
MobileNetV3-S100+1.85 [+0.00, +4.27]+0.19 [-1.90, +2.50]+9.12 [+3.47, +14.04]
Appendix
Table 21: Final old-task retention: scaffold minus random (R) or normal training (B), in percentage points [95% paired hierarchical interval], at coefficient one hundred. Old averages every preceding task.
Branchλ\lambdaEpochT1T2T3T4T5OldAll
Normal0160.5065.8364.6771.1781.5065.5468.73
Normal0262.6771.0072.8384.5086.3372.7575.47
Normal0362.0069.6772.6783.8389.6772.0475.57
Normal0464.5071.0072.6786.8392.5073.7577.50
Normal0563.0073.6775.0084.6791.8374.0877.63
Normal0Mean62.5370.2371.5782.2088.3771.6374.98
Appendix
Table 22: ResNet-18, CIFAR-10: task accuracies during the final-task intervention at coefficients 10 and 100.
Branchλ\lambdaEpochT1T2T3T4OldAll
Normal0150.0354.5775.9088.0060.1767.12
Normal0249.6053.6072.9391.5758.7166.92
Normal0349.8354.6770.9092.4058.4766.95
Normal0450.1053.8368.9792.4057.6366.32
Normal0551.6355.2068.1793.3758.3367.09
Normal0Mean50.2454.3771.3791.5558.6666.88
Appendix
Table 23: ResNet-18, CIFAR-100: task accuracies during the final-task intervention at coefficients 10 and 100.
Branchλ\lambdaEpochT1T2T3T4T5OldAll
Normal0161.5069.0072.6777.8384.1770.2573.03
Normal0260.6771.3372.8383.0091.0071.9675.77
Normal0361.3373.6772.3382.6791.3372.5076.27
Normal0457.5070.3374.6781.1789.8370.9274.70
Normal0557.3367.1768.3378.8391.8367.9272.70
Normal0Mean59.6770.3072.1780.7089.6370.7174.49
Appendix
Table 24: DeiT-Tiny, CIFAR-10: task accuracies during the final-task intervention at coefficients 10 and 100.
Branchλ\lambdaEpochT1T2T3T4OldAll
Normal0145.4747.5066.4789.3753.1462.20
Normal0244.1746.8064.2389.3351.7361.13
Normal0342.9745.7361.4089.4750.0359.89
Normal0439.5044.2057.2390.6046.9857.88
Normal0541.2043.6354.5389.0046.4657.09
Normal0Mean42.6645.5760.7789.5549.6759.64
Appendix
Table 25: DeiT-Tiny, CIFAR-100: task accuracies during the final-task intervention at coefficients 10 and 100.
Branchλ\lambdaEpochT1T2T3T4T5OldAll
Normal0144.8355.3359.8364.3380.0056.0860.87
Normal0252.6763.0065.6772.5085.1763.4667.80
Normal0355.5059.6761.1774.3387.1762.6767.57
Normal0450.5060.8364.1771.8387.6761.8367.00
Normal0550.3346.3364.3371.6782.5058.1763.03
Normal0Mean50.7757.0363.0370.9384.5060.4465.25
Appendix
Table 26: MobileNetV3-S, CIFAR-10: task accuracies during the final-task intervention at coefficients 10 and 100.
Branchλ\lambdaEpochT1T2T3T4OldAll
Normal0137.3043.2056.3085.7345.6055.63
Normal0236.3340.8757.3084.7744.8354.82
Normal0335.8339.0352.6783.7342.5152.82
Normal0435.0738.9751.5784.6741.8752.57
Normal0536.3039.6747.7085.5041.2252.29
Normal0Mean36.1740.3553.1184.8843.2153.63
Appendix
Table 27: MobileNetV3-S, CIFAR-100: task accuracies during the final-task intervention at coefficients 10 and 100.
Branchλ\lambdaEpochT1T2T3T4T5OldAll
Normal0142.3354.0064.1768.3373.3357.2160.43
Normal0242.1754.1757.6768.3381.3355.5860.73
Normal0341.8348.6751.6769.3383.5052.8759.00
Normal0444.0056.0057.0066.0087.0055.7562.00
Normal0545.5053.5062.1766.5082.0056.9261.93
Normal0Mean43.1753.2758.5367.7081.4355.6760.82
Appendix
Table 28: MLP-Mixer-B/16, CIFAR-10: task accuracies during the final-task intervention at coefficients 10 and 100.
Branchλ\lambdaEpochT1T2T3T4OldAll
Normal0133.6738.2058.4085.0743.4253.83
Normal0233.8037.9054.4087.3042.0353.35
Normal0330.4034.1052.2086.5738.9050.82
Normal0427.4032.9049.1086.6736.4749.02
Normal0531.1334.3749.7787.5038.4250.69
Normal0Mean31.2835.4952.7786.6239.8551.54
Appendix
Table 29: MLP-Mixer-B/16, CIFAR-100: task accuracies during the final-task intervention at coefficients 10 and 100.

Explore similar work

CardsList

  1. When Does Structure Matter in Continual Learning? Dimensionality Controls When Modularity Shapes Representational Geometry

    Apr 30, 2026Kathrin Korte, Joachim Winter Pedersen, Eleni Nisioti +1Continual LearningPlasticity

    IT University of Copenhagen, Denmark

  2. Dimensionality Controls When Modularity Helps in Continual Learning

    Jun 16, 2026Kathrin Korte, Christian Medeiros Adriano, Joachim Winther Pedersen +2Replay-Based Continual LearningContinual Learning

    IT University of Copenhagen, Denmark · Hasso Plattner Institute, University of Potsdam, Germany

  3. MoRe: Modular Representations for Principled Continual Representation Learning on Sequential Data

    May 14, 2026Jiaqi Sun, Boyang Sun, Rasmy M. H. +2Replay-Based Continual LearningContinual Learning

    1Carnegie Mellon University · 2Mohamed bin Zayed University of Artificial Intelligence