cs.CLOct 22, 2026

HERD: Harness-Evolving Recursive Distillation for Continual Learning

Authors: Xun Sun, Thanh Thi Nguyen, Thuy-Trang Vu

Organizations: Monash University · University of the Sunshine Coast

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Continual learning requires a model to keep acquiring new skills and knowledge without compromising its existing capabilities. Parametric methods suffer from catastrophic forgetting and loss of plasticity. Non-parametric methods, such as prompts, skills, and harnesses, leverage in-context learning but are limited by context length and lossy plain-text compression. To effectively combine their complementary strengths, we propose HERD (Harness-Evolving Recursive Distillation), a hybrid continual learning framework that learns efficiently in the harness and internalizes experience into cumulative capabilities via distillation. On the harness side, HERD continually refines its skills to capture reusable patterns, while retaining high-residual examples that the skills fail to capture in a bounded memory. On the parameter side, HERD performs drift-constrained distillation with dual anchors, substantially reducing forgetting over long recursive runs. On the 25-stage task stream of TRACE+17, under fixed memory and compute budgets, HERD matches the strongest parametric baseline in new-task acquisition while exhibiting the least forgetting and the strongest retention of general capabilities.

Figures & tables

Figure 1: HERD overview: one recursive stage. The harness Ht=(Kt,Rt)H_{t}=(K_{t},\mathcal{R}_{t}) evolves on task AtA_{t} and is distilled into the weights; drift budgets keep the student close to two anchors, the previous checkpoint θˉt−1\bar{\theta}_{t-1} and the base θ0\theta_{0} . HtH_{t} and θˉt\bar{\theta}_{t} recur, and only θˉt\bar{\theta}_{t} is deployed. The e.g. tiles follow one ScienceQA item, learned at t=5t=5 and asked again at t=12t=12 : two parametric baselines fail, OPR choosing the wrong option and Seq-LoRA writing the right text without the choice letter, while HERD answers from its weights alone across three seeds.
Figure 2: Drift-constrained distillation. (a) One frozen copy of θˉt−1\bar{\theta}_{t-1} serves as both the teacher pHp^{H} and the checkpoint anchor pCp^{C} ; the immutable θ0\theta_{0} is the general anchor p0p^{0} . (b) The student adds fresh factors ϕ\phi to the same weights and sees only the bare prompt. (c) Each update measures both drifts, steps on ϕ\phi , and moves the multipliers by dual ascent. (d) A multiplier grows only while its budget is exceeded.
MethodFamilyAA ↑\uparrowAcq. ↑\uparrowF ↓\downarrowBWT ↑\uparrowG-test ↑\uparrowΔC↑\Delta C\uparrowΔCmin⁡↑\Delta C_{\min}\uparrow
HERDThis work0.7060.7130.010−- 0.0070.774−- 0.063−- 0.130
BaseControl0.4190.4190.0000.0000.8370.0000.000
Static harnessControl0.4320.4320.0000.0000.791−- 0.046−- 0.135
HCL-StableContext0.4330.4350.003−- 0.0020.823−- 0.014−- 0.085
ACEContext0.4270.4280.001−- 0.0010.748−- 0.089−- 0.320
Seq-LoRAWeights0.6940.7150.025−- 0.0220.597−- 0.240−- 0.665
Table 1: TRACE+17 results after stage 25 under a matched model, stream, splits, prompts, and schedule; means over three seeds. Bold: best learning method (controls excluded) per column. BWT: mean j<T(RT,j−Rj,j){}_{j<T}(R_{T,j}-R_{j,j}) .
Figure 3: Long-horizon stability on TRACE+17. (a) Mean score on tasks 1 to 8 at the retention stages. (b) G-test macro mean at the anchor stages; error bars are one standard deviation over three seeds, the dotted line is the base model. (c) Change in each G-test category after stage 25 against the base model, in points, for the weight-updating methods; the outlined cell is each method’s ΔCmin⁡\Delta C_{\min} . Per-stage trajectories and HERD’s drift audit are in Figure 7 .
Figure 4: Ablations after stage 25: changes from full HERD in points (red is worse; colors clipped at ±10\pm 10 ). The strip shows audited drifts DCD^{C} (upper) and D0D^{0} (lower) against ε=0.05\varepsilon=0.05 . Single seed; full table in Appendix C .
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Table 1 by task; means over three seeds. (a) Score on each task at the stage it is learned, for HERD, Seq-LoRA, HCL-Stable, and the base model. (b) Change on each task between the stage it is learned and stage 25, in points, for the weight-updating methods; the outlined cell is each method’s largest loss.
BaselineΔ\Delta AAΔ\Delta Acq.Δ\Delta FΔ\Delta BWTΔ\Delta G-test
Base++ 0.287 [0.274, 0.299]++ 0.293 [0.281, 0.306]++ 0.010 [0.006, 0.016]−- 0.007 [ −- 0.013, −- 0.001]−- 0.063 [ −- 0.088, −- 0.038]
Static harness++ 0.274 [0.262, 0.286]++ 0.281 [0.268, 0.293]++ 0.010 [0.006, 0.016]−- 0.007 [ −- 0.013, −- 0.002]−- 0.017 [ −- 0.044, 0.010]
HCL-Stable++ 0.273 [0.260, 0.286]++ 0.278 [0.265, 0.291]++ 0.007 [0.003, 0.013]−- 0.005 [ −- 0.012, 0.001]−- 0.049 [ −- 0.074, −- 0.024]
ACE++ 0.279 [0.266, 0.292]++ 0.285 [0.272, 0.298]++ 0.009 [0.005, 0.015]−- 0.006 [ −- 0.013, 0.000]++ 0.026 [0.001, 0.051]
Seq-LoRA++ 0.012 [0.004, 0.021]−- 0.002 [ −- 0.010, 0.007]−- 0.015 [ −- 0.024, −- 0.009]++ 0.015 [0.006, 0.023]++ 0.177 [0.154, 0.200]
Uniform replay++ 0.007 [ −- 0.001, 0.016]++ 0.001 [ −- 0.007, 0.010]−- 0.010 [ −- 0.020, −- 0.004]++ 0.006 [ −- 0.004, 0.017]++ 0.175 [0.150, 0.201]
Appendix
Table 2: HERD minus each baseline after stage 25 on the quantities of Table 1 , with 95% paired item-bootstrap intervals (4,000 draws, items resampled within task and G-test category). Bold: the interval excludes zero.
Figure 7: Per-stage view behind Figure 3 . (a) G-test score per category at stages 0, 4, 8, 16, and 25 for the weight-updating methods, starting from the base model (dotted). (b) HERD’s audited checkpoint drift DCD^{C} and base drift D0D^{0} at the end of each stage (filled) and their within-stage peaks (hollow) against the budgets (dashed), on a log axis.
Figure 8: HERD’s state by stage; means over three seeds. (a) Registry size in tokens against BKB_{K} , and skills in the registry. (b) Residual memory: whole records held and tasks represented; in the shaded stages the memory is at its token budget. (c) Share of a stage’s 32 updates at which each multiplier is positive. (d) Distillation KL (Eq. 3 ) at the first update and averaged over the last eight. (e, f) Audited stage-end DCD^{C} and D0D^{0} for full HERD and the single-seed variants of Table 3 that change them; dashed: the 0.05 budget.
VariantAcq. ↑\uparrowAA ↑\uparrowF ↓\downarrowΔC↑\Delta C\uparrowΔCmin⁡↑\Delta C_{\min}\uparrowDCD^{C}D0D^{0}
HERD (full)0.713 ±\pm 0.0020.706 ±\pm 0.0080.010 ±\pm 0.003−- 0.063 ±\pm 0.006−- 0.130 ±\pm 0.0120.0210.041
−- skills *0.668 ( −- 0.045)0.656 ( −- 0.050)0.013 ( ++ 0.003)−- 0.073 ( −- 0.010)−- 0.135 ( −- 0.005)0.0210.040
uniform memory *0.708 ( −- 0.005)0.690 ( −- 0.017)0.020 ( ++ 0.010)−- 0.071 ( −- 0.008)−- 0.125 ( ++ 0.005)0.0340.041
−- checkpoint anchor *0.716 ( ++ 0.004)0.688 ( −- 0.018)0.030 ( ++ 0.020)−- 0.075 ( −- 0.012)−- 0.140 ( −- 0.010)0.1180.044
−- base anchor *0.718 ( ++ 0.006)0.704 ( −- 0.002)0.016 ( ++ 0.006)−- 0.158 ( −- 0.095)−- 0.490 ( −- 0.360)0.0270.212
−- both anchors *0.721 ( ++ 0.008)0.680 ( −- 0.026)0.044 ( ++ 0.033)−- 0.193 ( −- 0.130)−- 0.560 ( −- 0.430)0.1310.236
Appendix
Table 3: HERD ablations on TRACE+17, the full table behind Figure 4 : scores after stage 25 with the difference from full HERD in parentheses; DCD^{C} and D0D^{0} are the audited stage-25 drifts against the budgets εC=ε0=0.05\varepsilon_{C}=\varepsilon_{0}=0.05 . Bold marks the variant that hurts most in each column and any drift over its budget.
MethodTraining time (s)Context
 stage 1stage 25totaltokens
HERD31628410,7230
HCL-Stable58029646,88739
ACE60,41236,759888,08215,410
Seq-LoRA5120010,4630
Uniform replay31840015,2670
Appendix
Table 4: Training time per stage and in total, and the context attached to every prompt at inference.
Figure 9: Single test items through the stream; every quoted output is copied from the run records, seed 42 unless marked. (a–c) HERD against the parametric baselines: a GSM8K chain kept through the one-letter stages, with stage 1’s base drift and multiplier by update; a bare number on NumGLUE-cm beside a chain on GSM8K at stage 6; Seq-LoRA’s label collapse at stage 12, with HERD’s checkpoint-drift peaks. (d, e) Where HERD’s protection ends: a learned task that lost its probes, a self-contradictory skill, and answer length beyond the 32-token anchor. (f) The privileged teacher context in SDFT’s and OPHSD’s outputs, and HERD’s leak count.
More in this topic

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 304 papers

Explore similar work

CardsList

  1. Harness Continual Learning: Continual Adaptation Beyond Model Parameters

    Aug 19, 2026Borui Kang, Jinrui Gu, Junhan Lv +3Continual Learning for LLM AgentsContinual Learning

    State Key Laboratory for Novel Software Technology, Nanjing University, China · University of Wollongong, Australia

  2. Recurrent Harness for Agentic Continual Learning

    Oct 8, 2026Yuxin Xiao, Zhiyun Lu, Jianyu Wang +4Continual Learning for LLM AgentsAgent Harness Optimization

    Massachusetts Institute of Technology · Meta MRS Research · Yale University

  3. Rethinking Transfer in Continual Learning: A Replay-Based Realisation

    Jul 17, 2026Yang Meng, Zhenya Liu, Zhuokai Zhao +1Continual Learning for LLMsExperience Replay

    Department of Computer Science, University of Chicago