Language models in customer-facing products must be continually updated, since the data they serve shifts over time. As retraining on the full accumulated data is impractical, each update should instead run on a small set of examples carrying the training signal of all data seen so far. The strongest current methods build this set by gradient matching, which synthesizes examples whose training gradients match those of the full data. However, they require ground-truth labels, run once on a fixed snapshot rather than on a growing stream, and are slow on large data. We introduce OLGA, which lifts these limitations. It distills unlabeled data by drawing the gradient matching targets from the model's own next-token predictions. Across five text classification benchmarks with phi-1.5, it improves accuracy by 10 percentage points over the strongest selection baseline using as few as five examples. OLGA also distills online data streams by keeping a running rehearsal memory of synthetic examples, which is re-distilled together with each new batch as it arrives. In the online setting, it outperforms the strongest selection-based continual baseline by 6.6 stream-AUC percentage points. Finally, our method scales to large data by pairing distillation with selection in a hybrid pipeline, letting selection handle the mild stage of compression and distillation the extreme one. It cuts runtime by 30% versus distillation alone at similar accuracy, and beats the best selection-only pipeline by up to 12.6 percentage points.
Figures & tables
Figure 1: Overview of the components of OLGA. (a) Synthetic text is generated by matching its training gradient to that of real data, alternating with a step that maps it back to real tokens. (b) A frozen language model’s next-token predictions replace ground-truth labels, so gradient matching works on unlabeled data. (c) A running rehearsal memory extends offline distillation to an online data stream. Each new batch is distilled together with the memory, which the model then retrains on. (d) Our hybrid design first uses cheap selection to shrink each batch, then distills what remains into the few examples kept, which cuts runtime on large data.
Figure 2: Across five datasets, OLGA beats every competing method on average in both the offline and online setups, and comes close to the supervised upper bound in the online one. (a) Offline methods reduce 50 real examples to 5, and we report the held-out accuracy of phi-1.5 fine-tuned on them. (b) Online data arrives over a sequence of steps, each method keeps a memory of only 5 examples, and we report stream-AUC, the area under the accuracy-versus-step curve.
TwitterEmotion (joy)
SST-2 (negative)
Synthetic
I am feeling 1 so much 2 better after my recent health surgery yesterday and I can feel 3 more
teacher was giving a long 1 and boring and 2 not interesting and 3 clear lecture to the students because the teacher was reading from a textbook
Real
1. i am feeling amazing mostly normal i am going to a pre thanksgiving celebration with our friends from that time we were in softball 2. i feel like we have so much to be thankful for 3. i can feel superior on that point
1. it ai n’t art, by a long shot, 2. a shame that stealing harvard is too busy getting in its own way to be anything but frustrating, boring, and forgettable. 3. a more confused, less interesting and more sloppily made film could possibly come down the road in 2002
Table 1: Two synthetic examples distilled by OLGA, one per dataset. Each is generated as a mix of phrases from real examples, three of which are listed below it. A shared phrase has the same color in both places, and its superscript gives the number of the real example it comes from.
Method
Route
Offline accuracy
Online stream-AUC
Runtime (s) ↓
SST-2 ↑
Rotten T. ↑
IMDB ↑
RT-Pol. ↑
TwitEmo. ↑
Avg. ↑
SST-2 ↑
IMDB ↑
Avg. ↑
GradMM
Hybrid
88.2
83.9
92.3
82.1
83.4
86.0 (+3.3)
76.0
80.8
78.4 (+3.1)
436.1 ( − 29.9%)
Direct
87.4
76.6
90.6
81.3
77.6
82.7
72.1
78.5
75.3
622.1
Double
79.7
71.1
80.7
67.0
70.8
73.9 ( − 8.8)
—
—
—
2073.6 (+233.3%)
OLGA
Hybrid
87.8
81.2
89.0
84.6
57.3
80.0 ( − 0.9)
71.2
78.4
74.8 (+2.1)
447.5 ( − 30.1%)
Direct
88.6
85.0
90.9
82.6
57.4
80.9
67.7
77.7
72.7
639.9
Table 2: Comparison of the hybrid, direct, and double routes in the offline and online settings across 5 datasets. At each time step, the data is compressed in stages from B=1,000 to K=50 to m=5 . The offline setting runs a single step ( T=1 ), and the online setting runs T=10 steps. Runtime is the wall-clock time of one offline distillation run on SST-2. On average, the hybrid route beats the direct and double routes for both objectives in both settings, except offline without labels, where OLGA-Hybrid is on par with OLGA-Direct. Bold marks the best result in each column, and underline marks the reference that each colored difference is measured against.
Figure 3: (a) How each gradient matching objective builds its loss from the model’s next-token distribution; the dots mark which position it reads. (b) Three ways to fold a new batch into the memory: compress it first, pool it with the memory, or weight the two by hand.
Figure 4: Comparison of alternative label-free objectives. (Left) Accuracy against the number of examples m , per dataset and on average. (Right) Number of distinct tokens each signal aims at, and runtime relative to group-distribution. Group-distribution leads at m=5 , stays steady at larger budgets, and runs fastest.
Table 7
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Method
IMDB ↑
SST-2 ↑
Rotten Tom. ↑
RT-Polarity ↑
TwitEmo. ↑
Avg. ↑
Oracle
92.3 (+20.2)
88.2 (+12.9)
84.7 (+14.3)
83.2 (+12.8)
79.4 (+21.9)
85.6 (+20.3)
GradMM ( Nguyen et al., 2025 )
91.4 (+19.3)
87.3 (+12.0)
83.7 (+13.3)
78.0 (+7.6)
78.6 (+21.1)
83.8 (+18.5)
OLGA (ours)
89.5 (+17.4)
82.3 (+7.0)
74.5 (+4.1)
74.6 (+4.2)
56.5 ( − 1.0)
75.5 (+10.2)
Prototypical ( Snell et al., 2017 )
66.3 ( − 5.8)
65.4 ( − 9.9)
68.5 ( − 1.9)
68.9 ( − 1.5)
57.5
65.3
k-center ( Sener & Savarese, 2018 )
72.1
75.3
60.1 ( − 10.3)
60.2 ( − 10.2)
55.4 ( − 2.1)
64.6 ( − 0.7)
Herding ( Rebuffi et al., 2017 )
65.3 ( − 6.8)
61.2 ( − 14.1)
69.8 ( − 0.6)
69.7 ( − 0.7)
56.3 ( − 1.2)
64.5 ( − 0.8)
Appendix
Table 5: Exact values for Figure 2 (a), the offline accuracy of phi-1.5 on each dataset. The upper block gives the supervised upper bounds and the lower block the label-free methods. OLGA beats the best selection method on four of the five datasets and by 10.2 points on average. Bold marks the best label-free result in each column, and underline marks the reference that each colored difference is measured against.
Method
IMDB ↑
SST-2 ↑
TwitEmo. ↑
Rotten Tom. ↑
RT-Polarity ↑
Avg. ↑
Oracle-accumulate
92.1 (+4.8)
88.4 (+10.0)
87.2 (+27.5)
84.5 (+20.1)
84.3 (+20.0)
87.3 (+18.2)
Online GradMM (ours)
85.4 (+3.8)
79.2 (+0.8)
65.3 (+5.6)
73.2 (+8.8)
74.3 (+10.0)
75.5 (+8.5)
OLGA (ours)
85.1 (+3.5)
74.2 ( − 4.2)
59.6 ( − 0.1)
74.1 (+9.7)
74.9 (+10.6)
73.6 (+6.6)
k-center ( Sener & Savarese, 2018 )
73.6 ( − 8.0)
78.4
58.8 ( − 0.9)
61.3 ( − 3.1)
62.9 ( − 1.4)
67.0
Reservoir ( Vitter, 1985 )
81.6
70.3 ( − 8.1)
59.7
62.1 ( − 2.3)
52.3 ( − 12.0)
65.2 ( − 1.8)
Herding ( Rebuffi et al., 2017 )
76.1 ( − 5.5)
67.7 ( − 10.7)
50.5 ( − 9.2)
61.3 ( − 3.1)
57.1 ( − 7.2)
62.5 ( − 4.5)
Appendix
Table 6: Exact values for Figure 2 (b), the online stream-AUC of phi-1.5 on each dataset. Oracle-accumulate keeps all data seen so far and gives the upper bound. OLGA beats the best competing method by 6.6 points on average and comes within two points of supervised GradMM. Bold marks the best result in each column apart from the oracle, and underline marks the reference that each colored difference is measured against.
Method
IMDB ↑
SST-2 ↑
Rotten Tom. ↑
RT-Polarity ↑
TwitEmo. ↑
Avg. ↑
Oracle
92.3 (+11.9)
88.2 (+6.2)
84.7 (+10.2)
83.2 (+9.1)
79.4 (+16.9)
85.6 (+12.9)
GradMM ( Nguyen et al., 2025 )
90.6 (+10.2)
87.2 (+5.2)
81.6 (+7.1)
82.6 (+8.5)
78.9 (+16.4)
84.2 (+11.5)
OLGA (ours)
89.0 (+8.6)
78.9 ( − 3.1)
70.1 ( − 4.4)
76.4 (+2.3)
57.6 ( − 4.9)
74.4 (+1.7)
Prototypical ( Snell et al., 2017 )
78.8 ( − 1.6)
80.1 ( − 1.9)
72.2 ( − 2.3)
72.0 ( − 2.1)
60.2 ( − 2.3)
72.7
Herding ( Rebuffi et al., 2017 )
77.6 ( − 2.8)
78.8 ( − 3.2)
71.5 ( − 3.0)
71.8 ( − 2.3)
58.4 ( − 4.1)
71.6 ( − 1.1)
Uncertainty ( Lewis & Gale, 1994 )
70.4 ( − 10.0)
70.5 ( − 11.5)
73.0 ( − 1.5)
72.8 ( − 1.3)
62.5
69.8 ( − 2.9)
Appendix
Table 7: Offline accuracy of phi-1.5 on each dataset at m=10 , the counterpart to Table 5 . The upper block gives the supervised upper bounds and the lower block the label-free methods. OLGA stays ahead of the best selection method by 1.7 points on average, but leads on only two of the five datasets. Bold marks the best label-free result in each column, and underline marks the reference that each colored difference is measured against.
Synthetic
IMDLEG is great 1 because it is a well made 2 and entertaining story that keeps you 3 engaged from beginning 4 onwards and the well done production and the talented and well 5 done people who 6 are involved in making the show are truly a great and 7 a
Real
1. This is just a short comment but I stumbled onto this movie by chance and I loved it. The acting is great , the story is simple and touching … Seek it out. 2. I thought this was one of those really great films to see with a bunch of close friends. … Although not a new concept this was a very well made film. 3. Powers Boothe turns in a stellar performance as 1970’s cult figure Jim Jones of the Peoples Temple. … The movie keeps you riveted and is a must see for anyone. 4. A funny comedy from beginning to end! … Creative story line with a very talented cast. I thoroughly enjoyed it! 5. terribly underrated with matt dillon and tom skerritt, good backdrop for solid story … well acted and well cast, tommy lee jones and bruce dern make you hate them with passion 6. This movie will always be a Broadway and Movie classic, as long as there are still people who sing, dance, and act. 7. A great and truly independent film that hit most of my emotions and carried me into another world. …
Appendix
Table 8: Synthetic example distilled by OLGA for IMDB (positive), with the seven real examples it borrows its phrasing from. A shared phrase is highlighted in both places, and its superscript gives the number of the real example it comes from.
Synthetic
The new movie was boring and had no interesting or 1 well developed action or a good 2 one of the actors 3 in the story 4
Real
1. what the director can’t do is make either of val kilmer’s two personas interesting or worth caring about. 2. the techno tux is good for a few laughs, as are chan and hewitt, but when such a good design turns out to be a cheap knockoff … 3. sadly, though many of the actors throw off a spark or two when they first appear, they can’t generate enough heat in this cold vacuum of a comedy … 4. the story drifts so inexorably into cliches about tortured (and torturing) artists …
Appendix
Table 9: Synthetic example distilled by OLGA for Rotten Tomatoes (negative), with the four real examples it borrows its phrasing from. A shared phrase is highlighted in both places, and its superscript gives the number of the real example it comes from.
Synthetic
Amazing movie review provided by critics is a great 1 example of how 2 a single positive sentence can make a 3 significant impact on the
Real
1. morton is a great actress portraying a complex character, but morvern callar grows less compelling the farther it meanders from its shocking start. 2. a truly moving experience, and a perfect example of how art – when done right – can help heal, clarify, and comfort. 3. australian actor/director john polson and award-winning english cinematographer giles nuttgens make a terrific effort at disguising the obvious with energy and innovation.
Appendix
Table 10: Synthetic example distilled by OLGA for RT-Polarity (positive), with the three real examples it borrows its phrasing from. A shared phrase is highlighted in both places, and its superscript gives the number of the real example it comes from.
Selector
SST-2 ↑
Rotten Tomatoes ↑
IMDB ↑
RT-Polarity ↑
TwitterEmotion ↑
Avg. ↑
SST-2 runtime (s) ↓
Uniform (ours)
88.2
83.9
92.3
82.1
83.4
86.0
436.1
k-center
84.2
80.9
90.3
79.8
75.1
82.1
564.8
Appendix
Table 11: Per-dataset version of Table 4 , the offline accuracy of the GradMM-Hybrid route with uniform sampling or k-center as its selector. Uniform sampling is more accurate on every dataset and faster to run. Bold marks the better selector in each column.
Merge strategy
SST-2 ↑
Rotten Tomatoes ↑
IMDB ↑
RT-Polarity ↑
TwitterEmotion ↑
Avg. ↑
Cascaded (ours)
61.8
85.4
69.1
69.5
73.9
72.0
One-pass
60.4
78.7
69.3
62.5
80.1
70.2
History-weighted
57.2
85.3
72.4
68.1
66.0
69.8
Appendix
Table 12: Per-dataset version of Table 4 , the online stream-AUC of OLGA under each merge strategy. The cascaded merge leads on three of the five datasets and on average. Bold marks the best result in each column.
Figure 5: Feature discriminability (left) and gradient contrast (right) of each memory against its stream-AUC, one point per dataset and method. Neither property predicts how well a memory trains the model.
We present OLIVE (OnLine InterVEntion). At each iteration, the evolving student policy generates a new prefix, the teacher continues it autoregressively, and the student is updated using cross-entropy computed on the teacher-generated tokens. Each design choice targets a corresponding limitation of existing distillation methods: (1) sequential covariate shift in offline supervised fine-tuning (SFT) on fixed teacher trajectories, (2) fragmented supervision under prefix failure in token-level on-policy distillation (OPD), and (3) the need for access to teacher token probabilities in distribution-matching distillation. OLIVE achieves higher reasoning performance than OPD (with a top-16 KL approximation) at comparable GPU-hour cost. Our asynchronous implementation further reduces OLIVE's total training time by 23.8%. We evaluate OLIVE on both hard reasoning tasks and agentic tasks which reflects modern post-training scenarios, and it consistently outperforms existing distillation methods under the same training budget. By regenerating prefixes from the evolving student, OLIVE continues improving after offline distillation plateaus while better preserving the general capabilities and plasticity of the student. Using only text from GPT-5.4-mini, continuously training with OLIVE outperforms offline SFT from the same teacher by 13% on ScienceWorld. These results support OLIVE as an effective and efficient approach to online language-model distillation.
Haojin Wang, Dylan Zhang, Huaibo Chen +8
University of Illinois at Urbana-Champaign · Massachusetts Institute of Technology · University of Pennsylvania +2
Continual learning in multimodal large language models (MLLMs) aims to sequentially acquire knowledge while mitigating catastrophic forgetting, yet existing methods face inherent limitations: architecture-based approaches incur additional computational overhead and often generalize poorly to new tasks, rehearsal-based methods rely on storing historical data, raising privacy and storage concerns, and conventional regularization-based strategies alone are insufficient to fully prevent parameter interference. We propose Octopus, a two-stage continual learning framework based on History-Free Gradient Orthogonalization (HiFGO), which enforces gradient-level orthogonality without historical task data. Our proposed two-stage finetuning strategy decouples task adaptation from regularization, achieving a principled balance between plasticity and stability. Experiments on UCIT show that Octopus establishes state-of-the-art performance, surpassing prior SOTA by 2.14% and 6.82% in terms of Avg and Last.
Yuehao Liu, Shanyan Guan, Weijia Zhang +4
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University · vivo Mobile Communication Co., Ltd.
Continual adaptation of language models can change their output distribution on prompts learned earlier, while retaining every old prompt-answer pair may be undesirable or impossible. We study condition-anchored generative distillation (CAGD): retain a small set of old prompts, use a frozen previous model to reconstruct completions and generation states, and match its predictive distributions while learning the next task. The formulation separates three roles that ordinary replay conflates: conditions select the behavior to protect, teacher generations locate relevant states, and soft targets specify how predictions may change. For autoregressive language generation, teacher-rollout distillation admits an exact chain-rule decomposition of sequence divergence. For masked-diffusion language modeling, our implementation directly controls local denoising drift on teacher-generated completions. In continual adaptation of a 219M masked diffusion language model, CAGD reduces four-task final held-out loss from 2.927 to 1.114 in one task order and from 2.168 to 0.891 in exact reverse. The same soft targets lower final average loss by 0.055 over hard replay when teacher-generated support is held identical. The direction persists on fresh facts and natural instructions across SMDM and Qwen3. On GSM8K, Qwen adaptation preserves answer-format compliance, but exact-match retention is seed-mixed at 0.6B and worsens at 1.7B. These results support condition-anchored functional preservation as a common design principle across the tested language-generation objectives.
Huan Li, Zhe Cao, Qinlei Xie +2
Institute for AI Industry Research (AIR), Tsinghua University · Nanjing University