Representational drift in a code model: behaviour settles, which neurons carry it does not
Organizations: University College London, London, United Kingdom.
Abstract
In neuroscience, representational drift is a change in which neurons carry a representation while behaviour stays the same; we measure its analogue in OpenCoder-1.5B during pretraining, for 169 Python constructs over two trillion tokens at a constant learning rate, with a noise unit on every change, at checkpoints as close as 1,000 training steps. For 16 of the 169, those whose grammar fixes a next token, the loss at those positions reaches its floor within the first 4% of training, while the model's reliance on the neurons carrying them grows to the end: removing them costs 0.06, 0.25 and 0.40 nats more than removing a random set, at 4 billion, 84 billion and 2 trillion tokens. Which neurons carry a construct never settles: for all 169 the carrier set is still changing at the last checkpoint, and an abrupt settling at any date from 84 billion to 1.5 trillion tokens, or a gradual one before 500 billion, would have been detected in 99% of injected cases. The set measured at 84 billion tokens, removed at the end, cannot be separated from a random set, so the function is carried by whichever neurons are current. Up close the change is a persisting pool and a changing sample: a neuron that leaves a construct's set returns at three times the rate of a matched neuron, and no set reassembles. The hypothesis we fixed in advance, that the carriers settle, is refuted; we discuss candidate causes and name the experiment that would separate them.
Figures & tables
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| hypothesis | what we assumed | what we found on OpenCoder | verdict | why |
|---|---|---|---|---|
| premise | carriers stop inside the record for most constructs | they stop for none in pretraining; annealing is too short to say | refuted | 0 settling dates of 169, on either branch |
| P1 learning first | the loss is flat while the carriers change | flat for 20 of 24 series over 96% of training while every carrier set changes; the carriers are relied on at both ends of that stretch, already at 4.19 B and more by 84 B | confirmed | read on the 84 B end leg as the window, since no settling date exists |
| P2 frequency sets the date | frequent constructs settle earlier | no settling dates to order; the size’s date is ordered by frequency, weakly | could not be read | needs settling dates; the size result is a different quantity |
| P3 the token, not the role | constructs sharing a token settle together | no settling dates; on the size’s date, token exposure adds nothing to role frequency | could not be read | needs settling dates; the size reading points the other way |
| P4 the learning rate | turnover bends where the rate drops, for every construct | carrier change slows on annealing for every construct, but the data mix changes at the same moment and the branch is a third of a step wide | could not be read | needs a learning-rate change inside a branch with nothing else changing |
| P5 frequent constructs change less | slower carrier change for frequent constructs at every checkpoint | no relation at 19 dense checkpoints or late; sign opposite where anything | refuted | read where the ruler is clean; the pre-registered refutation condition is met |
| start (B tokens) | iteration | 5% set: r per 1,000 steps | 5% set: R 2 | cosine: r per 1,000 steps | cosine: R 2 |
|---|---|---|---|---|---|
| 4 | 1,000 | — | 0.668 | — | 0.752 |
| 8 | 2,000 | — | 0.656 | — | 0.786 |
| 13 | 3,000 | — | 0.698 | 0.1180 | 0.856 |
| 17 | 4,000 | — | 0.733 | 0.0856 | 0.892 |
| 21 | 5,000 | — | 0.769 | 0.0686 | 0.912 |
| 25 | 6,000 | — | 0.793 | 0.0582 | 0.921 |