cs.CLAug 27, 2026
SaveInformational Antilocality and the Locality Bias in LLMs
Organizations: University of Michigan · Eastern Michigan University
Abstract
We consider the ability of transformer-based language models (LLMs) to learn what we call k-antilocal languages, i.e., languages that have no mutual information across any span of contiguous symbols. We construct such languages with increasing , finding that LLMs trained on them achieve comparable cross-entropy loss regardless of antilocality, but converge more slowly on more antilocal languages. Our findings support the idea that non-local dependencies are more difficult to learn, but the evidence for this bias comes from learning speed rather than learning success.
Figures & tables
Figure 1: Difference in GPT-2 validation cross-entropy loss while training on antilocal vs. control (i.e., simple trigram) languages (antilocal control). Languages with higher values of are more antilocal (see Section 3 ), hence exhibit less information locality than the corresponding control languages. GPT-2 eventually achieves optimal performance on all languages.
Figure 2: Rotation classes from trigrams on vocabulary . Top: all trigrams partitioned into rotation classes by color. Note that there are rotation classes in total. Bottom: an illustration of symbol-transition distance within classes.
Figure 3: Sampling a 2-antilocal interleaved language. Top: three rotation classes (see Fig. 2 ) act as generators, each yielding a sequence of randomly sampled trigrams from within its class. Middle: a control sentence formed by concatenating components. Bottom: a 2-back sentence formed by interleaving components.
Figure 4: Cross-entropy loss on validation data from GPT-2 models trained on -antilocal and matched control languages. We plot the first 10K training steps, after which values become too small for meaningful visualization. The dotted line marks (which is the loss we expect from random guessing among vocabulary items).
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| k | m | sentence length | total tokens |
|---|---|---|---|
| 1 | 166 | 998 | |
| 2 | 110 | 992 | |
| 3 | 83 | 998 | |
| 4 | 66 | 992 | |
| 5 | 55 | 992 | |
| 6 | 47 | 989 |
Table 1: Features of our constructed -back languages. All languages used trigrams ( ) on the vocabulary .
Figure 5: Cross-entropy loss on validation data from DeBERTa-v3 models trained on -antilocal and matched control languages. Results are averaged across three seeds, with error ribbons showing standard deviation across seeds.
Figure 6: Difference in DeBERTa-v3 validation cross-entropy loss while training on antilocal vs. control (i.e., simple trigram) languages (antilocal control). All models eventually achieve optimal performance on all languages. The dotted line marks maximum expected difference, in which the control models perform optimally while the antilocal models perform at chance.