Learning to Learn a Language

Lennart Carstens-Behrens; Holger Fröhlich

cs.CLOct 5, 2026

Learning to Learn a Language

Authors: Lennart Carstens-Behrens, Holger Fröhlich

Organizations: Fraunhofer Institute for Algorithms and Scientific Computing SCAI · University Hospital Bonn

Abstract

We present the Prior-Fitted Language Model (PFLM), a 300M-parameter byte-level transformer pretrained only on samples from a synthetic non-linguistic prior. Given a prefix of real text, it learns to predict the language in context with frozen weights, having never seen a word of any real language. Every training sequence is generated by a recurrent structural causal model drawn fresh from a distribution over such models. The model never sees the same language twice during training, so the only way to predict the continuation is to infer the language from the prefix. Samples from this prior share the statistical signatures of natural text: Zipfian frequencies, slow entropy-rate convergence, and long-range dependence. On Wikipedia in six languages, bits per byte fall from the uniform eight to between 0.9 and 2.4 at one million bytes of context. Given numerals instead of text, PFLM learns to count, to compare magnitudes, and to add approximately. It predicts deterministic sequences like Rudin-Shapiro or the prime indicator, and it compresses six non-text domains, from source code to speech, below gzip and PPMd. The model has not learned a language. It has learned to learn one.

Figures & tables

Panel of Figure 1 — Figure 1: PFLM, trained only on the synthetic non-linguistic prior of Section 2 , learns to predict real languages and deterministic sequences in context with frozen weights. (a) Bits per byte against bytes of context seen, for English, Chinese, Hindi, Arabic, Japanese, and Korean Wikipedia (UTF-8). Each curve is the mean over $32$ held-out windows of one million bytes, averaged in log-spaced bins. Every curve falls from the uniform rate of eight bits to between $0.9$ and $2.4$ bits at one million bytes. (b) Bits per symbol against symbols seen for four binary sequences. By $10^{6}$ symbols Rudin–Shapiro costs $0.04$ bits, Kolakoski $0.13$ , and the prime indicator $0.28$ , against $0.37$ for a predictor that knows only the density of primes at that position; $\pi$ stays at $1.00$ throughout.

Panel of Figure 2 — Figure 2: Every pretraining sequence is generated by a small recurrent program drawn from the prior. (a) A synthetic language $\ell$ is a sparse recurrent structural causal model. Its wiring, each node’s structural equation $f_{i}$ and activation $\sigma_{i}$ , and the noise scale $\eta$ are all sampled, so no two sequences share a language. One node is marked and its update shown in full ( Equation 2 ); the tiles are single draws from the six equation families of Table 1 , each plotted over the plane of two of its parents and clipped to $[-1,1]$ as the dynamics does. (b) Rolling that language forward emits one token per step, by quantizing the $d_{\mathrm{obs}}$ observed coordinates and reading the resulting digits as one symbol ( Equation 3 ). The drawing uses $L=2$ and $d_{\mathrm{obs}}=2$ for legibility; the paper uses $L=2$ and $d_{\mathrm{obs}}=8$ , so the token alphabet is exactly the $256$ byte values. Only the tokens leave the generator. Because $\ell$ is never revealed and never drawn twice, the only way to predict the next token is to infer the language from those already seen.

Family	Form	Parameters
Linear map	$\mathbf{w}^{\top}\mathbf{p}+b$	$\mathbf{w},\,b$
Polynomial	$b+\sum_{i}\sum_{k=1}^{3}a_{ki}\,p_{i}^{k}+\sum_{i<j}c_{ij}\,p_{i}p_{j}$	$a,\,c,\,b$
Fourier features	$b+\sqrt{2}\,\sum_{j}w_{j}\cos(\boldsymbol{\omega}_{j}^{\top}\mathbf{p}+\varphi_{j})$	$\boldsymbol{\omega}_{j},\,\varphi_{j},\,w_{j},\,b$
MLP (two-layer)	$\mathbf{w}_{2}^{\top}\tanh(W_{1}\mathbf{p}+\mathbf{b}_{1})+b_{2}$	$W_{1},\,\mathbf{b}_{1},\,\mathbf{w}_{2},\,b_{2}$
Decision tree	threshold tests on $\mathbf{p}$ route to a leaf value	splits, thresholds, leaves
Step function	$v_{\,\mathrm{bin}(\mathbf{w}^{\top}\mathbf{p}+b)}$ , four bins	$\mathbf{w},\,b$ , edges, levels

Table 1: The six structural-equation families. Each node draws its update

f_{i}

from one of these, applied to its parent values

\mathbf{p}\equiv\mathbf{h}_{t,\,\pa(i)}

. Coefficients are sampled from per-language Gaussians, and the structural sizes (MLP hidden width, Fourier feature count, tree depth) are drawn per language as well.

Figure 3: The structural causal model prior matches natural language on every scalar fingerprint; classical generators each fail on at least one. For each source, mean

\pm

standard deviation across 500 sequences of 4096 bytes. Panels (left to right): byte entropy, written

H(X)

in the figure, gzip compression ratio, bigram Zipf exponent, active byte count. “Synth” is the structural causal model prior; “Real” shows the mean across the six natural languages with individual values as colored dots. The i.i.d. uniform, bigram, and hidden Markov model baselines produce near-uniform marginals (entropy

\approx 8

, gzip

\approx 1

, all 256 bytes active); i.i.d. Zipf corrects the marginal but not the conditional structure; only the structural causal model prior’s range overlaps the natural-language values on every panel.

Panel of Figure 4 — Figure 4: The structural causal model prior reproduces both the slow entropy-rate convergence and the slow long-range MI decay of natural language. Synth shows the median (line) and 10–90 percentile range (band) across 500 structural causal model samples; each natural language is the per-sequence median across 500 wiki sequences. (a) Conditional entropy $H(Y_{t}\mid Y_{t-1},\ldots,Y_{t-n+1})$ at $n$ -gram orders 1–8. The synth band covers every natural-language curve at every order; dotted reference at $H=\log_{2}256=8$ bits is the i.i.d. uniform-random ceiling. (b) $\mathrm{MI}(Y_{t};\,Y_{t+d})$ at log-spaced distances $d=1,2,4,\ldots,512$ bytes. Synth and natural decay together from $d=1$ to $d=512$ bytes.

Figure 5: Given numerals in context, PFLM learns to count through carries, to order three-digit numerals by magnitude, and to judge approximate sums. Left: exact match of the next numeral while counting from

2

99{,}999

, by number of digits, for increments with no carry, one carry, and a carry across two or more trailing nines; bands are 95% Wilson intervals. Middle: accuracy at naming the larger of two three-digit numerals, by the ratio of smaller to larger, after

k

in-context examples; at

8192

examples accuracy falls from

0.97

at ratio

1/2

0.68

14/15

. Right: accuracy at judging whether

n_{1}+n_{2}

exceeds

n_{3}

, by the ratio of the sum to

n_{3}

;

n_{3}

exceeds either addend, so comparing one addend to

n_{3}

scores at chance [ 2 ] . Chance is one half;

500

queries per ratio bin and shot count, none of them among the examples.

Domain	bigram	gzip	PPMd	PFLM
Source code	3.76	2.40	1.82	1.54
Molecules	2.36	1.47	0.95	0.82
Protein	4.21	4.44	4.13	3.89
DNA	1.99	2.25	2.09	1.94
Symbolic music	3.59	3.93	2.99	2.63
Audio ( $\mu$ -law)	5.32	6.35	4.90	4.18

Table 2: PFLM compresses real data from every tested domain below an online bigram, gzip, and PPMd. Bits per byte, mean over windows of

524{,}288

bytes, up to

32

per domain across its categories, against three baselines: an online bigram (first-order statistics), gzip (repetition), and PPMd (high-order adaptive statistics). PFLM is lowest in every row.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Drawn	Range
Nodes $d$	per language	uniform in $[8,32]$
Observed coordinates $d_{\mathrm{obs}}$	fixed	$8$ (with $L=2$ levels: $V=256$ )
Parent-set size $\|\pa(i)\|$	per node	uniform in $[2,8]$ , capped at $d-1$
Update layers $K$	per language	uniform in $\{1,2\}$
Equation family $f_{i}$	per node	uniform over the six families of Table 1
Coefficient scale	per language	weights $\mathcal{N}(0,s_{w}^{2})$ with $s_{w}\sim U(1,2.5)$ , biases $\mathcal{N}(0,s_{b}^{2})$ with $s_{b}\sim U(0.1,0.5)$ ; weight scale divided by $\sqrt{\text{fan-in}}$

Appendix

Table 3: Sampled ranges of the prior. Per-language draws fix a value for the whole language; per-node draws are made independently at each node.

Parameters	295M
Layers (linear attn. + SWA)	20 (15 + 5)
Hidden size	1024
MLP intermediate size	2048
Attention heads / KV heads	16 / 8
Attention head dim	128
Linear-attn. value / key heads	16 / 8

Appendix

Table 4: Model configuration: blocks of three gated linear-attention layers followed by one sliding-window attention layer, SwiGLU MLPs, and the 256-byte vocabulary. The parameter count is measured.

Hyperparameter	Value
Optimizer	NorMuon [ 19 ] for matrix parameters; AdamW [ 21 ] for embeddings, head, and norms
Matrix learning rate	$8\times 10^{-4}$
AdamW learning rate	$1.6\times 10^{-5}$
Schedule	warmup–stable–decay [ 18 ] , 500 warmup steps
Tokens per stage	20B, 60B, 40B, 15B, 10B, 5B at $2^{10}$ to $2^{20}$ bytes of context
Sliding window	$2^{18}$ bytes

Appendix

Table 5: Optimization hyperparameters.

Figure 6: The prior produces samples whose byte distributions span the breadth of natural language without sharing any specific script. Byte-frequency profile (log scale) for ten random samples from the prior (above the dashed line) and the six natural-language corpora (below). Each row is one source’s distribution over byte values 0–255. Synth samples each use a substantial subset of the byte vocabulary, and the active subset varies from sample to sample. Each natural language pins to a characteristic UTF-8 byte range fixed by its script.

Explore similar work

Sep 24, 2026cs.AI

Self-Play Pretraining with Zero Data

Advances in language modeling have been driven by scaling pretraining on ever more data. Yet, the training data is still largely curated on the model's behalf. A more general approach to pretraining would let the model learn to generate the data most useful for its own improvement. This would provide an effectively unbounded source of training data, limited by compute rather than human knowledge. We introduce Self-Play Pretraining with Zero Data, an initial proof-of-concept towards realizing this vision. Our procedure casts synthetic data generation as a search over the space of all computable structure, taking inspiration from Solomonoff induction. Starting from random initialization, two models learn in tandem: a generator proposes programs interpreted by a universal Turing machine, generating byte sequences, while a learner autoregressively predicts these byte sequences. The learner is trained with standard cross-entropy, while the generator is trained with reinforcement learning to produce sequences at the frontier of the learner's capabilities, yielding an adaptive curriculum. A universal Turing machine gives us a search space over all computable data-generating processes, imposing little domain-specific structure, and self-play searches over this space for useful training data. We test whether zero-shot performance on natural data improves predictably with self-play compute; this is a clean test of transfer since neither generator nor learner is trained on natural data. Across several natural datasets, zero-shot loss exhibits predictable scaling in compute. The models also exhibit in-context learning, and discover recognizable mathematical sequences during training.

Aditya Cowsik, Kfir Dolev, Michael Y. Li +4

Independent Researcher · Tel Aviv University · Stanford University +1

Pretraining Self-Play Zero-Shot Robustness Language Modeling Synthetic Data Adaptive Curriculum

Ask PDF

Sep 12, 2026cs.CL

Structural priors for data-efficient language learning

Efficient language learning requires methods to reduce the reliance on large data and computational resources. We investigate structural transfer: First training models on non-language data to induce useful priors for natural language. This approach is a form of weight initialization for multilingual language modeling. We evaluate transfer via next-token-prediction loss, weight shifts in the model, and downstream linguistic benchmarks. Several symbolic data types - notably music, probabilistic grammars, and cellular automata - yield lower language-modeling loss than random initialization. These gains coincide with smaller weight shifts during subsequent language training, suggesting that structural transfer positions models in a more favorable region of the parameter space. However, a lower loss does not translate consistently into better downstream linguistic performance, and transfer from non-language data is less efficient than additional language data. We conclude that non-language data can serve as a partial substitute for language data for the training objective of next-token prediction but does not reliably support broader linguistic generalization.

Yana Veitsman, Jonas Mayer Martins, Jonathan Lautenschlager +1

University of Göttingen, Germany

Large Language Model Pretraining Loss Function Probabilistic Model

Ask PDF

Apr 23, 2026cs.CL

Listen and Chant Before You Read: The Ladder of Beauty in LM Pre-Training

We show that pre-training a Transformer on music before language significantly accelerates language acquisition. Using piano performances (MAESTRO dataset), a developmental pipeline -- music

\to

poetry

\to

prose -- yields a

17.5\%

perplexity improvement over random initialization (

p < 0.001

, 5 seeds), with music and poetry improving orthogonal model components (internal computation and embeddings, respectively). Convergence tests confirm that this is not a transient head start: at

d\!=\!64

, multi-seed validation (5 seeds) shows a persistent 5.5% gap at plateau (

p = 0.017

), with the pipeline converging faster and to a lower loss in every run. Real music matches the transfer ceiling of synthetic patterns with one-third the data, and scaling experiments reveal that optimal pre-training data volume shifts with model capacity (

-3\% \to +3\% \to +6\%

advantage of larger datasets from

d\!=\!16

d\!=\!64

). Across the scales we study (

d\!\in\!\{16,32,64\}

, up to

{\sim}400

K parameters), these results suggest a capacity-dependent data curation principle and indicate that structured human creative outputs can provide an efficient pre-training substrate for small language models; stronger conclusions at modern pre-training scale will require substantially larger experiments.

Yoshinori Nomura

Mirage Mountain Technologies Inc.

Large Language Model Pretraining Pretraining Text-To-Music Language Acquisition Maestro Dataset Ladder+1

Ask PDF