Language models in customer-facing products must be continually updated, since the data they serve shifts over time. As retraining on the full accumulated data is impractical, each update should instead run on a small set of examples carrying the training signal of all data seen so far. The strongest current methods build this set by gradient matching, which synthesizes examples whose training gradients match those of the full data. However, they require ground-truth labels, run once on a fixed snapshot rather than on a growing stream, and are slow on large data. We introduce OLGA, which lifts these limitations. It distills unlabeled data by drawing the gradient matching targets from the model's own next-token predictions. Across five text classification benchmarks with phi-1.5, it improves accuracy by 10 percentage points over the strongest selection baseline using as few as five examples. OLGA also distills online data streams by keeping a running rehearsal memory of synthetic examples, which is re-distilled together with each new batch as it arrives. In the online setting, it outperforms the strongest selection-based continual baseline by 6.6 stream-AUC percentage points. Finally, our method scales to large data by pairing distillation with selection in a hybrid pipeline, letting selection handle the mild stage of compression and distillation the extreme one. It cuts runtime by 30% versus distillation alone at similar accuracy, and beats the best selection-only pipeline by up to 12.6 percentage points.
Figures & tables
Figure 1: Overview of the components of OLGA. (a) Synthetic text is generated by matching its training gradient to that of real data, alternating with a step that maps it back to real tokens. (b) A frozen language model’s next-token predictions replace ground-truth labels, so gradient matching works on unlabeled data. (c) A running rehearsal memory extends offline distillation to an online data stream. Each new batch is distilled together with the memory, which the model then retrains on. (d) Our hybrid design first uses cheap selection to shrink each batch, then distills what remains into the few examples kept, which cuts runtime on large data.
Figure 2: Across five datasets, OLGA beats every competing method on average in both the offline and online setups, and comes close to the supervised upper bound in the online one. (a) Offline methods reduce 50 real examples to 5, and we report the held-out accuracy of phi-1.5 fine-tuned on them. (b) Online data arrives over a sequence of steps, each method keeps a memory of only 5 examples, and we report stream-AUC, the area under the accuracy-versus-step curve.
TwitterEmotion (joy)
SST-2 (negative)
Synthetic
I am feeling 1 so much 2 better after my recent health surgery yesterday and I can feel 3 more
teacher was giving a long 1 and boring and 2 not interesting and 3 clear lecture to the students because the teacher was reading from a textbook
Real
1. i am feeling amazing mostly normal i am going to a pre thanksgiving celebration with our friends from that time we were in softball 2. i feel like we have so much to be thankful for 3. i can feel superior on that point
1. it ai n’t art, by a long shot, 2. a shame that stealing harvard is too busy getting in its own way to be anything but frustrating, boring, and forgettable. 3. a more confused, less interesting and more sloppily made film could possibly come down the road in 2002
Table 1: Two synthetic examples distilled by OLGA, one per dataset. Each is generated as a mix of phrases from real examples, three of which are listed below it. A shared phrase has the same color in both places, and its superscript gives the number of the real example it comes from.
Method
Route
Offline accuracy
Online stream-AUC
Runtime (s) ↓
SST-2 ↑
Rotten T. ↑
IMDB ↑
RT-Pol. ↑
TwitEmo. ↑
Avg. ↑
SST-2 ↑
IMDB ↑
Avg. ↑
GradMM
Hybrid
88.2
83.9
92.3
82.1
83.4
86.0 (+3.3)
76.0
80.8
78.4 (+3.1)
436.1 ( − 29.9%)
Direct
87.4
76.6
90.6
81.3
77.6
82.7
72.1
78.5
75.3
622.1
Double
79.7
71.1
80.7
67.0
70.8
73.9 ( − 8.8)
—
—
—
2073.6 (+233.3%)
OLGA
Hybrid
87.8
81.2
89.0
84.6
57.3
80.0 ( − 0.9)
71.2
78.4
74.8 (+2.1)
447.5 ( − 30.1%)
Direct
88.6
85.0
90.9
82.6
57.4
80.9
67.7
77.7
72.7
639.9
Table 2: Comparison of the hybrid, direct, and double routes in the offline and online settings across 5 datasets. At each time step, the data is compressed in stages from B=1,000 to K=50 to m=5 . The offline setting runs a single step ( T=1 ), and the online setting runs T=10 steps. Runtime is the wall-clock time of one offline distillation run on SST-2. On average, the hybrid route beats the direct and double routes for both objectives in both settings, except offline without labels, where OLGA-Hybrid is on par with OLGA-Direct. Bold marks the best result in each column, and underline marks the reference that each colored difference is measured against.
Figure 3: (a) How each gradient matching objective builds its loss from the model’s next-token distribution; the dots mark which position it reads. (b) Three ways to fold a new batch into the memory: compress it first, pool it with the memory, or weight the two by hand.
Figure 4: Comparison of alternative label-free objectives. (Left) Accuracy against the number of examples m , per dataset and on average. (Right) Number of distinct tokens each signal aims at, and runtime relative to group-distribution. Group-distribution leads at m=5 , stays steady at larger budgets, and runs fastest.
Table 7
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Method
IMDB ↑
SST-2 ↑
Rotten Tom. ↑
RT-Polarity ↑
TwitEmo. ↑
Avg. ↑
Oracle
92.3 (+20.2)
88.2 (+12.9)
84.7 (+14.3)
83.2 (+12.8)
79.4 (+21.9)
85.6 (+20.3)
GradMM ( Nguyen et al., 2025 )
91.4 (+19.3)
87.3 (+12.0)
83.7 (+13.3)
78.0 (+7.6)
78.6 (+21.1)
83.8 (+18.5)
OLGA (ours)
89.5 (+17.4)
82.3 (+7.0)
74.5 (+4.1)
74.6 (+4.2)
56.5 ( − 1.0)
75.5 (+10.2)
Prototypical ( Snell et al., 2017 )
66.3 ( − 5.8)
65.4 ( − 9.9)
68.5 ( − 1.9)
68.9 ( − 1.5)
57.5
65.3
k-center ( Sener & Savarese, 2018 )
72.1
75.3
60.1 ( − 10.3)
60.2 ( − 10.2)
55.4 ( − 2.1)
64.6 ( − 0.7)
Herding ( Rebuffi et al., 2017 )
65.3 ( − 6.8)
61.2 ( − 14.1)
69.8 ( − 0.6)
69.7 ( − 0.7)
56.3 ( − 1.2)
64.5 ( − 0.8)
Appendix
Table 5: Exact values for Figure 2 (a), the offline accuracy of phi-1.5 on each dataset. The upper block gives the supervised upper bounds and the lower block the label-free methods. OLGA beats the best selection method on four of the five datasets and by 10.2 points on average. Bold marks the best label-free result in each column, and underline marks the reference that each colored difference is measured against.
Method
IMDB ↑
SST-2 ↑
TwitEmo. ↑
Rotten Tom. ↑
RT-Polarity ↑
Avg. ↑
Oracle-accumulate
92.1 (+4.8)
88.4 (+10.0)
87.2 (+27.5)
84.5 (+20.1)
84.3 (+20.0)
87.3 (+18.2)
Online GradMM (ours)
85.4 (+3.8)
79.2 (+0.8)
65.3 (+5.6)
73.2 (+8.8)
74.3 (+10.0)
75.5 (+8.5)
OLGA (ours)
85.1 (+3.5)
74.2 ( − 4.2)
59.6 ( − 0.1)
74.1 (+9.7)
74.9 (+10.6)
73.6 (+6.6)
k-center ( Sener & Savarese, 2018 )
73.6 ( − 8.0)
78.4
58.8 ( − 0.9)
61.3 ( − 3.1)
62.9 ( − 1.4)
67.0
Reservoir ( Vitter, 1985 )
81.6
70.3 ( − 8.1)
59.7
62.1 ( − 2.3)
52.3 ( − 12.0)
65.2 ( − 1.8)
Herding ( Rebuffi et al., 2017 )
76.1 ( − 5.5)
67.7 ( − 10.7)
50.5 ( − 9.2)
61.3 ( − 3.1)
57.1 ( − 7.2)
62.5 ( − 4.5)
Appendix
Table 6: Exact values for Figure 2 (b), the online stream-AUC of phi-1.5 on each dataset. Oracle-accumulate keeps all data seen so far and gives the upper bound. OLGA beats the best competing method by 6.6 points on average and comes within two points of supervised GradMM. Bold marks the best result in each column apart from the oracle, and underline marks the reference that each colored difference is measured against.
Method
IMDB ↑
SST-2 ↑
Rotten Tom. ↑
RT-Polarity ↑
TwitEmo. ↑
Avg. ↑
Oracle
92.3 (+11.9)
88.2 (+6.2)
84.7 (+10.2)
83.2 (+9.1)
79.4 (+16.9)
85.6 (+12.9)
GradMM ( Nguyen et al., 2025 )
90.6 (+10.2)
87.2 (+5.2)
81.6 (+7.1)
82.6 (+8.5)
78.9 (+16.4)
84.2 (+11.5)
OLGA (ours)
89.0 (+8.6)
78.9 ( − 3.1)
70.1 ( − 4.4)
76.4 (+2.3)
57.6 ( − 4.9)
74.4 (+1.7)
Prototypical ( Snell et al., 2017 )
78.8 ( − 1.6)
80.1 ( − 1.9)
72.2 ( − 2.3)
72.0 ( − 2.1)
60.2 ( − 2.3)
72.7
Herding ( Rebuffi et al., 2017 )
77.6 ( − 2.8)
78.8 ( − 3.2)
71.5 ( − 3.0)
71.8 ( − 2.3)
58.4 ( − 4.1)
71.6 ( − 1.1)
Uncertainty ( Lewis & Gale, 1994 )
70.4 ( − 10.0)
70.5 ( − 11.5)
73.0 ( − 1.5)
72.8 ( − 1.3)
62.5
69.8 ( − 2.9)
Appendix
Table 7: Offline accuracy of phi-1.5 on each dataset at m=10 , the counterpart to Table 5 . The upper block gives the supervised upper bounds and the lower block the label-free methods. OLGA stays ahead of the best selection method by 1.7 points on average, but leads on only two of the five datasets. Bold marks the best label-free result in each column, and underline marks the reference that each colored difference is measured against.
Synthetic
IMDLEG is great 1 because it is a well made 2 and entertaining story that keeps you 3 engaged from beginning 4 onwards and the well done production and the talented and well 5 done people who 6 are involved in making the show are truly a great and 7 a
Real
1. This is just a short comment but I stumbled onto this movie by chance and I loved it. The acting is great , the story is simple and touching … Seek it out. 2. I thought this was one of those really great films to see with a bunch of close friends. … Although not a new concept this was a very well made film. 3. Powers Boothe turns in a stellar performance as 1970’s cult figure Jim Jones of the Peoples Temple. … The movie keeps you riveted and is a must see for anyone. 4. A funny comedy from beginning to end! … Creative story line with a very talented cast. I thoroughly enjoyed it! 5. terribly underrated with matt dillon and tom skerritt, good backdrop for solid story … well acted and well cast, tommy lee jones and bruce dern make you hate them with passion 6. This movie will always be a Broadway and Movie classic, as long as there are still people who sing, dance, and act. 7. A great and truly independent film that hit most of my emotions and carried me into another world. …
Appendix
Table 8: Synthetic example distilled by OLGA for IMDB (positive), with the seven real examples it borrows its phrasing from. A shared phrase is highlighted in both places, and its superscript gives the number of the real example it comes from.
Synthetic
The new movie was boring and had no interesting or 1 well developed action or a good 2 one of the actors 3 in the story 4
Real
1. what the director can’t do is make either of val kilmer’s two personas interesting or worth caring about. 2. the techno tux is good for a few laughs, as are chan and hewitt, but when such a good design turns out to be a cheap knockoff … 3. sadly, though many of the actors throw off a spark or two when they first appear, they can’t generate enough heat in this cold vacuum of a comedy … 4. the story drifts so inexorably into cliches about tortured (and torturing) artists …
Appendix
Table 9: Synthetic example distilled by OLGA for Rotten Tomatoes (negative), with the four real examples it borrows its phrasing from. A shared phrase is highlighted in both places, and its superscript gives the number of the real example it comes from.
Synthetic
Amazing movie review provided by critics is a great 1 example of how 2 a single positive sentence can make a 3 significant impact on the
Real
1. morton is a great actress portraying a complex character, but morvern callar grows less compelling the farther it meanders from its shocking start. 2. a truly moving experience, and a perfect example of how art – when done right – can help heal, clarify, and comfort. 3. australian actor/director john polson and award-winning english cinematographer giles nuttgens make a terrific effort at disguising the obvious with energy and innovation.
Appendix
Table 10: Synthetic example distilled by OLGA for RT-Polarity (positive), with the three real examples it borrows its phrasing from. A shared phrase is highlighted in both places, and its superscript gives the number of the real example it comes from.
Selector
SST-2 ↑
Rotten Tomatoes ↑
IMDB ↑
RT-Polarity ↑
TwitterEmotion ↑
Avg. ↑
SST-2 runtime (s) ↓
Uniform (ours)
88.2
83.9
92.3
82.1
83.4
86.0
436.1
k-center
84.2
80.9
90.3
79.8
75.1
82.1
564.8
Appendix
Table 11: Per-dataset version of Table 4 , the offline accuracy of the GradMM-Hybrid route with uniform sampling or k-center as its selector. Uniform sampling is more accurate on every dataset and faster to run. Bold marks the better selector in each column.
Merge strategy
SST-2 ↑
Rotten Tomatoes ↑
IMDB ↑
RT-Polarity ↑
TwitterEmotion ↑
Avg. ↑
Cascaded (ours)
61.8
85.4
69.1
69.5
73.9
72.0
One-pass
60.4
78.7
69.3
62.5
80.1
70.2
History-weighted
57.2
85.3
72.4
68.1
66.0
69.8
Appendix
Table 12: Per-dataset version of Table 4 , the online stream-AUC of OLGA under each merge strategy. The cascaded merge leads on three of the five datasets and on average. Bold marks the best result in each column.
Figure 5: Feature discriminability (left) and gradient contrast (right) of each memory against its stream-AUC, one point per dataset and method. Neither property predicts how well a memory trains the model.