Structure--Residual Inductive Biases for Language Model Pretraining
Organizations: Palona AI · Institute of Computer Science, Polish Academy of Sciences · Mohamed bin Zayed University of Artificial Intelligence
Abstract
Next-token prediction is relied upon to learn both predictive behavior and internal representations. We ask whether representation structure itself should be an explicit inductive bias. Motivated by the two-part coding view of Kolmogorov complexity, we study a structure--residual principle in which a representation separates structure from realization detail. We instantiate this principle in two complementary ways. Boundary Bottleneck Concept (BBC) routes information from token segments through boundary carriers while preserving local autoregressive computation. Continuous Concept (CC) regularizes representation geometry and induces a spectral decomposition into low-frequency structure and high-frequency residual. We derive coding bounds for both views and show that multiplicity-corrected boundary attention becomes exact as the within-segment path energy vanishes. Empirically, BBC and CC exhibit strikingly different depth preferences: BBC works best in early layers, whereas CC is most effective in later layers. At 300M training tokens, combining BBC and CC improves NLL by 1.44% on GPT-2 Medium. These gains persist with training runs up to 20B tokens. Representation diagnostics show that BBC increases the relative predictive importance of boundary states, while CC reduces spectral residual energy. Together, these results suggest that structure--residual biases can improve language-model pretraining beyond next-token prediction. Code: https://github.com/haihuang-ml/structure-residual
Figures & tables
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| Layer | |||
|---|---|---|---|
| 0 | |||
| 2 | |||
| 4 | |||
| 6 | |||
| 8 | |||
| 10 |
| Layer | |||||||
|---|---|---|---|---|---|---|---|
| 0 | |||||||
| 2 | |||||||
| 4 | |||||||
| 6 | |||||||
| 8 | |||||||
| 10 |
| Layer | |||
|---|---|---|---|
| 0 | |||
| 2 | |||
| 4 | |||
| 6 | |||
| 8 | |||
| 10 |
| Layer | |||
|---|---|---|---|
| 0 | |||
| 2 | |||
| 4 | |||
| 6 | |||
| 8 | |||
| 10 |
| BBC+CC Variant | NLL |
|---|---|
| Standard CC | 3.2757 0.0043 |
| CC Random Dropout | 3.2757 0.0008 |
| CC Adaptive Dropout | 3.2753 0.0005 |
| NLL | Improve (%) | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Tokens | vanilla | BBC | BBC+CC | BBC+CC Adapt | BBC | BBC+CC | BBC+CC Adapt | ||
| 1B | 3.3113 | 3.2772 | 3.2765 | 3.2748 | 1.03 | 1.05 | 1.10 | ||
| 2B | 3.2107 | 3.1819 | 3.1807 | 3.1793 | 0.90 | 0.94 | 0.98 | ||
| 3B | 3.1704 | 3.1397 | 3.1387 | 3.1363 | 0.97 | 1.00 | 1.08 | ||
| 4B | 3.1426 | 3.1089 | 3.1125 | 3.1085 | 1.08 | 0.96 | 1.09 | ||
| 5B | 3.1184 | 3.0897 | 3.0937 | 3.0871 | 0.92 | 0.79 | 1.00 | ||
| Model | |||||
|---|---|---|---|---|---|
| Vanilla | 0.528 | 0.468 | 0.381 | 0.314 | 0.208 |
| CC | 0.413 | 0.359 | 0.285 | 0.232 | 0.150 |
| Model | NLL | Improve (%) |
|---|---|---|
| Vanilla | – | |
| BBC | ||
| BBC+CC |
| Model | NLL | Improve (%) |
|---|---|---|
| Vanilla rotary_pct =0.25 | – | |
| BBC+CC rotary_pct =0.00 | 0.95 | |
| BBC+CC rotary_pct =0.25 | 1.94 | |
| BBC+CC rotary_pct =0.50 | 1.94 | |
| BBC+CC rotary_pct =0.75 | 1.92 | |
| BBC+CC rotary_pct =1.00 | 1.91 |
| gpt2-medium | Pythia-410M | |
| , learned positions | , rotary | |
| Error of multiplicity-corrected attention, | 1.70 | 0.14 |
| Key path energy relative to a random order of the segment | 0.57 | 0.97 |
| Value path energy relative to a random order of the segment | 1.00 | 0.97 |
| Logit radius from the path energy (nats) | 146 | 159 |
| Measured (nats) | 10.8 | 12.6 |
| gpt2-small | gpt2-medium | |||
|---|---|---|---|---|
| NLL | NLL | |||
| 4 | 0.72 | 4.116 | 0.71 | 4.095 |
| 8 | 0.83 | 4.099 | 0.83 | 4.069 |
| 16 | 0.85 | 4.102 | 0.86 | 4.066 |
| 32 | 0.81 | 4.094 | 0.84 | 4.067 |
| 64 | 0.72 | 4.106 | 0.78 | 4.078 |
| Layer | ||||
|---|---|---|---|---|
| gpt2-medium 22 | 3.54 | 0.0014 | 0.0049 | 0.0013 |
| gpt2-medium 20 | 2.62 | 0.0018 | 0.0046 | 0.0018 |
| gpt2-medium 18 | 2.03 | 0.0029 | 0.0059 | 0.0023 |
| gpt2-medium 16 | 1.56 | 0.0030 | 0.0047 | 0.0030 |
| gpt2-medium 14 | 1.21 | 0.0039 | 0.0047 | 0.0039 |
| gpt2-medium 12 | 0.99 | 0.0050 | 0.0049 | 0.0047 |
Language Model Pretraining
Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.
See all 480 papers