cs.CLOct 22, 2026

HERD: Harness-Evolving Recursive Distillation for Continual Learning

Authors: Xun Sun, Thanh Thi Nguyen, Thuy-Trang Vu

Organizations: Monash University · University of the Sunshine Coast

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Continual learning requires a model to keep acquiring new skills and knowledge without compromising its existing capabilities. Parametric methods suffer from catastrophic forgetting and loss of plasticity. Non-parametric methods, such as prompts, skills, and harnesses, leverage in-context learning but are limited by context length and lossy plain-text compression. To effectively combine their complementary strengths, we propose HERD (Harness-Evolving Recursive Distillation), a hybrid continual learning framework that learns efficiently in the harness and internalizes experience into cumulative capabilities via distillation. On the harness side, HERD continually refines its skills to capture reusable patterns, while retaining high-residual examples that the skills fail to capture in a bounded memory. On the parameter side, HERD performs drift-constrained distillation with dual anchors, substantially reducing forgetting over long recursive runs. On the 25-stage task stream of TRACE+17, under fixed memory and compute budgets, HERD matches the strongest parametric baseline in new-task acquisition while exhibiting the least forgetting and the strongest retention of general capabilities.

Figures & tables

Figure 1: HERD overview: one recursive stage. The harness Ht=(Kt,Rt)H_{t}=(K_{t},\mathcal{R}_{t}) evolves on task AtA_{t} and is distilled into the weights; drift budgets keep the student close to two anchors, the previous checkpoint θˉt−1\bar{\theta}_{t-1} and the base θ0\theta_{0} . HtH_{t} and θˉt\bar{\theta}_{t} recur, and only θˉt\bar{\theta}_{t} is deployed. The e.g. tiles follow one ScienceQA item, learned at t=5t=5 and asked again at t=12t=12 : two parametric baselines fail, OPR choosing the wrong option and Seq-LoRA writing the right text without the choice letter, while HERD answers from its weights alone across three seeds.
Figure 2: Drift-constrained distillation. (a) One frozen copy of θˉt−1\bar{\theta}_{t-1} serves as both the teacher pHp^{H} and the checkpoint anchor pCp^{C} ; the immutable θ0\theta_{0} is the general anchor p0p^{0} . (b) The student adds fresh factors ϕ\phi to the same weights and sees only the bare prompt. (c) Each update measures both drifts, steps on ϕ\phi , and moves the multipliers by dual ascent. (d) A multiplier grows only while its budget is exceeded.
MethodFamilyAA ↑\uparrowAcq. ↑\uparrowF ↓\downarrowBWT ↑\uparrowG-test ↑\uparrowΔC↑\Delta C\uparrowΔCmin⁡↑\Delta C_{\min}\uparrow
HERDThis work0.7060.7130.010−- 0.0070.774−- 0.063−- 0.130
BaseControl0.4190.4190.0000.0000.8370.0000.000
Static harnessControl0.4320.4320.0000.0000.791−- 0.046−- 0.135
HCL-StableContext0.4330.4350.003−- 0.0020.823−- 0.014−- 0.085
ACEContext0.4270.4280.001−- 0.0010.748−- 0.089−- 0.320
Seq-LoRAWeights0.6940.7150.025−- 0.0220.597−- 0.240−- 0.665
Table 1: TRACE+17 results after stage 25 under a matched model, stream, splits, prompts, and schedule; means over three seeds. Bold: best learning method (controls excluded) per column. BWT: mean j<T(RT,j−Rj,j){}_{j<T}(R_{T,j}-R_{j,j}) .
Figure 3: Long-horizon stability on TRACE+17. (a) Mean score on tasks 1 to 8 at the retention stages. (b) G-test macro mean at the anchor stages; error bars are one standard deviation over three seeds, the dotted line is the base model. (c) Change in each G-test category after stage 25 against the base model, in points, for the weight-updating methods; the outlined cell is each method’s ΔCmin⁡\Delta C_{\min} . Per-stage trajectories and HERD’s drift audit are in Figure 7 .
Figure 4: Ablations after stage 25: changes from full HERD in points (red is worse; colors clipped at ±10\pm 10 ). The strip shows audited drifts DCD^{C} (upper) and D0D^{0} (lower) against ε=0.05\varepsilon=0.05 . Single seed; full table in Appendix C .
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Table 1 by task; means over three seeds. (a) Score on each task at the stage it is learned, for HERD, Seq-LoRA, HCL-Stable, and the base model. (b) Change on each task between the stage it is learned and stage 25, in points, for the weight-updating methods; the outlined cell is each method’s largest loss.
BaselineΔ\Delta AAΔ\Delta Acq.Δ\Delta FΔ\Delta BWTΔ\Delta G-test
Base++ 0.287 [0.274, 0.299]++ 0.293 [0.281, 0.306]++ 0.010 [0.006, 0.016]−- 0.007 [ −- 0.013, −- 0.001]−- 0.063 [ −- 0.088, −- 0.038]
Static harness++ 0.274 [0.262, 0.286]++ 0.281 [0.268, 0.293]++ 0.010 [0.006, 0.016]−- 0.007 [ −- 0.013, −- 0.002]−- 0.017 [ −- 0.044, 0.010]
HCL-Stable++ 0.273 [0.260, 0.286]++ 0.278 [0.265, 0.291]++ 0.007 [0.003, 0.013]−- 0.005 [ −- 0.012, 0.001]−- 0.049 [ −- 0.074, −- 0.024]
ACE++ 0.279 [0.266, 0.292]++ 0.285 [0.272, 0.298]++ 0.009 [0.005, 0.015]−- 0.006 [ −- 0.013, 0.000]++ 0.026 [0.001, 0.051]
Seq-LoRA++ 0.012 [0.004, 0.021]−- 0.002 [ −- 0.010, 0.007]−- 0.015 [ −- 0.024, −- 0.009]++ 0.015 [0.006, 0.023]++ 0.177 [0.154, 0.200]
Uniform replay++ 0.007 [ −- 0.001, 0.016]++ 0.001 [ −- 0.007, 0.010]−- 0.010 [ −- 0.020, −- 0.004]++ 0.006 [ −- 0.004, 0.017]++ 0.175 [0.150, 0.201]
Appendix
Table 2: HERD minus each baseline after stage 25 on the quantities of Table 1 , with 95% paired item-bootstrap intervals (4,000 draws, items resampled within task and G-test category). Bold: the interval excludes zero.
Figure 7: Per-stage view behind Figure 3 . (a) G-test score per category at stages 0, 4, 8, 16, and 25 for the weight-updating methods, starting from the base model (dotted). (b) HERD’s audited checkpoint drift DCD^{C} and base drift D0D^{0} at the end of each stage (filled) and their within-stage peaks (hollow) against the budgets (dashed), on a log axis.
Figure 8: HERD’s state by stage; means over three seeds. (a) Registry size in tokens against BKB_{K} , and skills in the registry. (b) Residual memory: whole records held and tasks represented; in the shaded stages the memory is at its token budget. (c) Share of a stage’s 32 updates at which each multiplier is positive. (d) Distillation KL (Eq. 3 ) at the first update and averaged over the last eight. (e, f) Audited stage-end DCD^{C} and D0D^{0} for full HERD and the single-seed variants of Table 3 that change them; dashed: the 0.05 budget.
VariantAcq. ↑\uparrowAA ↑\uparrowF ↓\downarrowΔC↑\Delta C\uparrowΔCmin⁡↑\Delta C_{\min}\uparrowDCD^{C}D0D^{0}
HERD (full)0.713 ±\pm 0.0020.706 ±\pm 0.0080.010 ±\pm 0.003−- 0.063 ±\pm 0.006−- 0.130 ±\pm 0.0120.0210.041
−- skills *0.668 ( −- 0.045)0.656 ( −- 0.050)0.013 ( ++ 0.003)−- 0.073 ( −- 0.010)−- 0.135 ( −- 0.005)0.0210.040
uniform memory *0.708 ( −- 0.005)0.690 ( −- 0.017)0.020 ( ++ 0.010)−- 0.071 ( −- 0.008)−- 0.125 ( ++ 0.005)0.0340.041
−- checkpoint anchor *0.716 ( ++ 0.004)0.688 ( −- 0.018)0.030 ( ++ 0.020)−- 0.075 ( −- 0.012)−- 0.140 ( −- 0.010)0.1180.044
−- base anchor *0.718 ( ++ 0.006)0.704 ( −- 0.002)0.016 ( ++ 0.006)−- 0.158 ( −- 0.095)−- 0.490 ( −- 0.360)0.0270.212
−- both anchors *0.721 ( ++ 0.008)0.680 ( −- 0.026)0.044 ( ++ 0.033)−- 0.193 ( −- 0.130)−- 0.560 ( −- 0.430)0.1310.236
Appendix
Table 3: HERD ablations on TRACE+17, the full table behind Figure 4 : scores after stage 25 with the difference from full HERD in parentheses; DCD^{C} and D0D^{0} are the audited stage-25 drifts against the budgets εC=ε0=0.05\varepsilon_{C}=\varepsilon_{0}=0.05 . Bold marks the variant that hurts most in each column and any drift over its budget.
MethodTraining time (s)Context
 stage 1stage 25totaltokens
HERD31628410,7230
HCL-Stable58029646,88739
ACE60,41236,759888,08215,410
Seq-LoRA5120010,4630
Uniform replay31840015,2670
Appendix
Table 4: Training time per stage and in total, and the context attached to every prompt at inference.
Figure 9: Single test items through the stream; every quoted output is copied from the run records, seed 42 unless marked. (a–c) HERD against the parametric baselines: a GSM8K chain kept through the one-letter stages, with stage 1’s base drift and multiplier by update; a bare number on NumGLUE-cm beside a chain on GSM8K at stage 6; Seq-LoRA’s label collapse at stage 12, with HERD’s checkpoint-drift peaks. (d, e) Where HERD’s protection ends: a learned task that lost its probes, a self-contradictory skill, and answer length beyond the 32-token anchor. (f) The privileged teacher context in SDFT’s and OPHSD’s outputs, and HERD’s leak count.
More in this topic

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 301 papers

Explore similar work

CardsList
Aug 19, 2026

Harness Continual Learning: Continual Adaptation Beyond Model Parameters

Continual learning has largely been model-centric, treating model parameters as the state that changes with sequential experience. Modern agents can also adapt through a harness of prompts, memories, tools, skills, and routing rules. Because these contents jointly shape later execution, a harness update can disrupt previously reliable behavior even when the model is frozen. This raises a new question: how can an agent continually improve its state outside the model while retaining behavior acquired earlier? We formulate Harness Continual Learning (HCL), a new continual learning paradigm in which the harness evolves around a frozen foundation model, and define the resulting loss of earlier behavior as harness-level forgetting. We instantiate HCL with four execution-facing components: the Task Interface, Experience Memory, Capability Map, and Adaptive Router. We further introduce guarded harness evolution to separate update generation from state commitment. A Continual Optimizer proposes candidate harnesses from post-execution feedback, and a Continual Evaluator commits the resulting candidate harness only after checking current improvement, historical retention, and validity. Experiments on textual reasoning, multimodal perception, and open-world interaction demonstrate capability accumulation and failure recovery, with relative gains exceeding 10% over corresponding baselines in multiple settings. Component ablations assess the contribution of each harness component, while controlled retention sweeps reveal measurable harness-level forgetting and show that the stability--plasticity trade-off can be explicitly adjusted.

Borui Kang, Jinrui Gu, Junhan Lv +3

State Key Laboratory for Novel Software Technology, Nanjing University, China · University of Wollongong, Australia

Continual Learning for LLM AgentsContinual LearningAgent Harness Optimization
AskPDF
Oct 8, 2026

Recurrent Harness for Agentic Continual Learning

Language model agents increasingly operate over long interaction streams, requiring online adaptation to distribution shifts through continual learning. However, existing agent harnesses typically either evolve their states only through forward information flow or rely on additional replay or validation data for harness optimization. To address this, we propose ReHarness, a recurrent harness framework for agentic continual learning that complements forward read and write with backward revision under a strictly online streaming protocol. We ask what information could have been captured differently in the harness state to better inform the current instance in the stream. Specifically, after each instance, we leverage the resulting trajectory as feedback to revise the reflections retrieved for that instance, thereby propagating feedback backward through the textual harness state. Across three continual learning benchmarks and three language model backbones, ReHarness consistently outperforms strong raw-context baselines, memory- and reflection-based methods, and prompt optimization approaches. Further analysis shows that ReHarness produces source-grounded and meaningful revisions, corrects overgeneralized or outdated experience under distribution shifts, and improves both stability and plasticity during continual learning.

Yuxin Xiao, Zhiyun Lu, Jianyu Wang +4

Massachusetts Institute of Technology · Meta MRS Research · Yale University

Continual Learning for LLM AgentsAgent Harness OptimizationOnline Continual LearningMemory-Augmented Language Model Agents
AskPDF
Jul 17, 2026

Rethinking Transfer in Continual Learning: A Replay-Based Realisation

Continual learning studies how deployed language models can continually acquire new tasks without expensive retraining from scratch. Existing methods, whether rehearsal-based (replaying stored past data) or rehearsal-free (regularising or isolating parameters), overwhelmingly target one objective: preventing catastrophic forgetting. Forward transfer, the past helping the future, has meanwhile been pursued almost exclusively through parameter reuse, with no explicit account of when transfer should be expected at all. We begin one step earlier: before designing a transfer mechanism, we ask when transfer should exist at all. We answer with a framework of three measurable conditions: the target task must leave room for improvement beyond its own limited supervision, transferable information must survive continued optimisation, and replay must come from compatible previous tasks. We instantiate this view as Transfer-Selective Replay (TSR), which selects replay data predicted to benefit the incoming task rather than replaying past examples indiscriminately. Selection is guided by a zero-training task signature, while distillation preserves stability on previous tasks. Under the standard continual learning protocol in the low-budget regime, TSR consistently improves forward transfer while maintaining stability, outperforming existing replay baselines across heterogeneous and homogeneous task streams. More broadly, the results argue for treating transfer as a first-class objective of continual learning, to be understood before it is engineered.

Yang Meng, Zhenya Liu, Zhuokai Zhao +1

Department of Computer Science, University of Chicago

Continual Learning for LLMsExperience ReplayReplay-Based Continual LearningTransfer Learning
AskPDF