Language models acquire their capabilities from training data, and developers withhold data during training to keep models from learning capabilities that could be misused. How long this delays a capability, and how much data is needed to acquire it later, is unknown. Here we show that the general progress a model makes on the rest of its training data substitutes for the data that was withheld. We withheld the training signal for individual skills in models of chess, Othello, molecular strings, bracket languages and web text. The delay of a withheld skill in larger models was predicted from smaller ones. A chess model never explicitly trained to respond to check ranked the legal responses almost as well as a model that was. When models were then trained on the withheld data, they acquired the skill with a small fraction of the data needed from initialization, and this fraction decreased as the models progressed. A control rule of equal difficulty was learned much more slowly. A programming language withheld from the training text of a 7-billion-parameter model was regained in the same way. Data filtering therefore delays a capability, and the data required to regain it decreases as models improve.
Figures & tables
Figure 1Figure 2Figure 3Figure 4Figure 5
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Size
Trained
Tokens
Loss
Legal
Ranked first
Quiet block
Shift
Both boards
Disjoint
9.7M
Games without a check
600
1.803
0.453
0.652
0.772
0.905
0.646 (0.601–0.687)
0.566
Without in-check targets
302
1.778
0.426
0.672
0.666
0.882
0.624 (0.584–0.658)
0.571
Fully supervised
200
1.831
0.847
0.678
0.963
0.996
0.933 (0.908–0.956)
0.927
24M
Games without a check
600
1.747
0.477
0.661
0.796
0.928
0.697 (0.657–0.736)
0.656
Without in-check targets
200
1.760
0.437
0.645
0.682
0.886
0.642 (0.603–0.678)
0.592
Fully supervised
200
1.763
0.887
0.710
0.973
0.998
0.960 (0.943–0.973)
0.947
Appendix
Table 6
Fit with source-run intercepts
Least squares
Domain
Runs
Source
Below acq.
Intercept a
Rate b
Residual
Intercept a
Rate b
Residual
runs
loss
s.d.
s.d.
Chess
159
12
129
− 0.53 ± 0.05
1.12 ± 0.04
0.25
− 0.52
1.14 ± 0.04
0.25
Web text
88
11
70
− 0.62 ± 0.05
0.91 ± 0.04
0.10
− 0.66
0.89 ± 0.06
0.17
SMILES
65
4
51
− 0.42 ± 0.09
0.59 ± 0.07
0.32
− 0.41
0.61 ± 0.09
0.37
Single line
312
27
250
− 0.53
0.97 ± 0.07
0.36
Appendix
Table 7
Standard learning rate
One third of it
Size
Checkpoint
Background
Logit
16 seq.
64 seq.
256 seq.
64 seq.
16 seq.
(tokens)
loss (nats)
distance
Cost, % of the tokens from initialization at the same setting
Models trained on a new task typically degrade on prior tasks, a phenomenon known as forgetting. Traditionally, mitigating forgetting has required replaying stored exemplars from prior tasks, which is often impractical. By contrast, language models can sample from their own training distribution, and we show that these self-generated samples serve as effective replay data, nearly eliminating forgetting. We find that forgetting nonetheless persists when the model has little remaining capacity: models pretrained close to saturation cannot absorb new information without overwriting prior knowledge. When capacity is not the limiting factor, low learning rates reduce forgetting but require substantially more training steps. Replay breaks this tradeoff, enabling fast, high-learning-rate finetuning without forgetting.
Defining a constructive process to build general capabilities for language models in an automatic manner is considered an open problem in artificial intelligence. Towards this, we consider the problem of building a curriculum of tasks that grows a model via relevant skill discovery. We provide a concrete framework for this task, using a family of tasks called Cross-Entropy Games, which we postulate is universal in a suitable sense. We show that if it is possible to grow the curriculum for relevant skill discovery by iterating a greedy optimization algorithm, then, under natural assumptions, there is essentially only one meta-objective possible (up to a few hyper-parameters). We call the resulting process cognitive training. We postulate that, given sufficiently capable language models as players and meta-samplers, cognitive training provides a principled way to relevant skill discovery; and hence to the extent general capabilities are achievable via greedy curriculum learning, cognitive training would be a solution.
What should data teach a language model at a particular point in training? A circuit view reveals three distinct bottlenecks: forming a computation, making its required content available, and selecting among available routes. A shared diagnosis-to-data principle connects them: localize the missing operation, preserve its causal relation, vary shortcut-bearing context, and re-audit the residual. Formation-sensitive selection and prerequisite ordering accelerate a binding-matching-transport path; a brief early prefix from the same training multiset retains a validation advantage through 100B tokens. Availability counterfactuals then distinguish writing content from invoking available memory, while paired supervision and context-opportunity ranking improve matched route decisions and long-context answer likelihood. A continuous 350M-model experiment connects the three interventions on the same facts: early circuit training improves subsequent learning, and the complete sequence outperforms stage-replacement controls on facts withheld from Use teaching. Independent query surfaces and opposed-source decisions expose conditional arbitration as the remaining frontier. Together, these results show why a change in the limiting operation calls for a change in supervision, not merely a new ranking of difficult examples.
Yixiao Chen, Ke Cheng, Yue Liu +2
AI Data, Tencent · Harvard University · AMS, Tencent