Not all facts a language model can recall support further learning equally well. We distinguish active facts, which support generalization beyond their own recall, from inert facts, which can be recalled but do not support such generalization in a given task. We study this distinction by constructing new relations from existing ones. Take the relation that maps a place to its continent (Paris -> Europe) and assign each of its answers an arbitrary new label (Europe -> crab, Asia -> zebra). The new relation then maps Paris -> crab. We fine-tune the model on the new relation for a subset of subjects, without telling it the original relation or the relabeling rule, and test whether it assigns the corresponding labels to held-out subjects that share an answer in the original relation. Having learned Paris -> crab, does it answer Munich -> crab? Across eleven language models and 70 relations, how linearly readable the original relation is in pretrained representations strongly predicts this generalization. Within each model, the two correlate across relations at Pearson r = 0.87 to 0.92. This association persists when controlling, separately and jointly, for the number of fine-tuning subjects and answers, the model's accuracy on the original relation, and how often the subjects occur in pretraining text. It is also robust to re-randomizing the label assignment and fine-tuning seed, and to replacing the animal labels with labels from thirteen other semantic categories. When we map a fine-tuned model's wrong label back to the original answer it was assigned to, Munich -> zebra becomes Munich -> Asia. Answers recovered this way are more often among the linear probe's top answers for the subject than among the pretrained model's own. Whether a new relation generalizes depends less on whether its facts can be recalled than on how readily their shared structure can be read out linearly.
Figures & tables
Figure 1: Facts generalize to a new relation when they are linearly readable. The pretrained model recalls relations A and B equally well, but only A’s answers are linearly readable from its representations of the subjects. A random bijection from the answers to animal labels defines a new relation. After fine-tuning on it, the model generalizes A’s labels to held-out subjects but not B’s. The fine-tuned model’s wrong label for B’s held-out subject, “snail,” is the label of the answer the linear probe picks for that subject, not of its true answer ( Generalization failures ). Pre-linearity predicts Generalization better than Recall does ( Pre-linearity versus Recall ). Section 3 defines the three measures.
Figure 2: In every model, relations with higher Pre-linearity tend to have higher Generalization. One point per relation, marked by its relation set .
Table 1: Pre-linearity predicts Generalization in every model, with or without controls. Each entry correlates Pre-linearity with Generalization across each model’s 70 relations (Pearson r , Spearman ρ ). ± : 95% interval ( Appendix H ).
Figure 3: Pre-linearity predicts Generalization better than Recall does (a), and its correlation with Generalization hardly depends on the label set (b). Correlations run across each model’s 70 relations; in (b), each label set’s number of labels follows its name. Intervals: Appendix H .
Pearson r
Spearman ρ
Repeat 0
.92 ± .04
.90 ± .08
Repeat 1
.93 ± .04
.89 ± .08
Repeat 2
.92 ± .04
.92 ± .06
Mean of three
.93 ± .03
.92 ± .07
Table 2: Three runs of MiniCPM5-2B-Base give almost the same correlation between Pre-linearity and Generalization. Each run draws a new random bijection for every new relation and a new fine-tuning seed ( Re-randomization ). ± : 95% interval ( Appendix H ).
Figure 4: The Pre-linearity probe accounts for more of the fine-tuned models’ errors than Recall does. Each bar is an explanation’s Shapley share of all errors on held-out subjects, pooled over models and relations. Explanations and baselines: Generalization failures and Appendix G .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: The three levels of subject splitting behind Pre-linearity. The held-out subjects H never enter the probe. The outer loop gives every fine-tuning subject in T one prediction from a probe that did not see it; the inner loop, inside each outer training set, only chooses the layer.
Relation
G9v3-3B
Gemma-4-E2B
Granite-4.2-3B
LFM2.5-2.6B-Base
MiniCPM5-2B-Base
Nanbeige4-3B-Base
OLMo-3-1025-7B
Phi-4-mini-instruct
Qwen3.5-9B
Qwen3.5-9B-Base
Qwen3.8-27B
Hernandez
Object → superclass
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
Person → native language (Hernandez)
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
Star → constellation
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
Person → professional sport
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
Product → company
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
Appendix
Table 3: Accuracy on the fine-tuning subjects T after fine-tuning on the new relation , per relation (row) and model (column): candidate accuracy under the four fine-tuning templates after the last epoch, averaged over the templates within each subject, then within each answer, then over answers. Relations are grouped by set and sorted within it by mean Generalization over the models, highest first. Every entry is 1.00.
Figure 6: Pre-linearity, Generalization and Recall of every relation. Relations are ordered by mean Generalization across models: the upper half of the order here, the lower half on the next page. Blue: Pre-linearity; orange: Generalization; black: the pretrained model’s Recall. For each, the dot is the median over the eleven models, the open diamond the mean, and the line the range (not a confidence interval). The mark before each name gives the relation’s set: a triangle for Hernandez, a ring for Town and a dot for BEAR.
Figure 6 (continued): the lower half of the relation order, with the same encoding.
Figure 7: Pre-linearity, Generalization and Recall of every relation on every model. Each row is one relation, ordered by its mean Generalization across models; each column is one model. The left block shows Pre-linearity, the middle block Generalization and the right block the pretrained model’s Recall. Within each block, a square’s color is its rank among the block’s squares (a quantile scale), so every color step holds the same number of squares; the ticks under each color bar give the minimum, the quartiles and the maximum. The mark before each row gives the relation’s set: a triangle for Hernandez, a ring for Town and a dot for BEAR.
Model
Repeat 0
Repeat 1
Repeat 2
Mean of three runs
G9v3-3B
.93 ± .07
.95 ± .05
.89 ± .13
.93 ± .07
Gemma-4-E2B
.87 ± .15
.86 ± .19
.86 ± .17
.87 ± .16
Granite-4.2-3B
.85 ± .21
.85 ± .23
.78 ± .26
.86 ± .21
LFM2.5-2.6B-Base
.87 ± .11
.86 ± .14
.85 ± .11
.87 ± .12
MiniCPM5-2B-Base
.92 ± .06
.92 ± .07
.93 ± .07
.94 ± .05
Nanbeige4-3B-Base
.85 ± .15
.86 ± .13
.92 ± .07
.88 ± .11
Appendix
Table 4: As Table 2 (Pearson r ), for every model fine-tuned three times, across the 17 Hernandez and Town relations. Mean of three: Pre-linearity against each relation’s Generalization averaged over the three runs. ± : 95% interval ( Appendix H ).
Model
Repeat 0
Repeat 1
Repeat 2
Mean of three runs
G9v3-3B
.95 ± .13
.94 ± .13
.89 ± .29
.92 ± .19
Gemma-4-E2B
.91 ± .21
.86 ± .30
.88 ± .24
.90 ± .22
Granite-4.2-3B
.85 ± .33
.86 ± .28
.75 ± .39
.83 ± .33
LFM2.5-2.6B-Base
.89 ± .22
.89 ± .26
.93 ± .16
.90 ± .20
MiniCPM5-2B-Base
.92 ± .16
.93 ± .15
.93 ± .14
.95 ± .11
Nanbeige4-3B-Base
.86 ± .24
.90 ± .20
.89 ± .21
.90 ± .21
Appendix
Table 5: As Table 4 , with Spearman ρ in place of Pearson r .
Rule
Pinned to
Labels left
0
Enumerate: every lemma string of noun.animal matching ^[a-z]+$ , deduplicated
WordNet 3.0
3,048
1
Leaves: drop a label that is a WordNet noun ancestor of another label in the label set
–
2,161
2
Primary WordNet sense is animal.n.01 or a hyponym of it
WordNet 3.0 sense order
1,422
3
Not named by a published profanity list
LDNOOBW English, commit 4638b97
1,422
4
Frequency floor: wordfreq zipf ≥2.32
wordfreq 3.1.1 ( large )
243
5
First token distinct from every kept label’s, under all nine tokenizers at once
tokenizer revisions of record
185
Appendix
Table 6: The animal label set A , one row per step of the label-set algorithm in the order applied, with the external artifact each threshold is pinned to and the labels left after the step. Every model uses this set: each of its 179 labels has its own first token under every model’s tokenizer, and that token is the one trained and scored.
Label set
Root
∣A∣
≥86
zipf med.
NN cos
animal
animal.n.01
179
yes
2.87
.145
act
act.n.02
460
yes
3.44
.173
artifact
artifact.n.01
393
yes
3.35
.164
attribute
attribute.n.02
245
yes
3.43
.185
body
body.n.01
0
no
–
–
cognition
cognition.n.01
197
yes
3.45
.169
Appendix
Table 7: The label sets of the label-set comparison , below the animal label set ( Table 6 ). Each comes from one WordNet 3.0 noun lexicographer file under the same algorithm, but requires every label to be a single token under MiniCPM5-2B-Base instead of a distinct first token under every model. Root: the synset a label’s primary sense must fall under. ≥86 : enough labels to take part. zipf med.: median wordfreq zipf frequency of the labels. NN cos: mean first-token nearest-neighbor cosine within a random 60-label draw under MiniCPM5-2B-Base .
Figure 8: Every model’s errors are explained well above the shuffled baseline. (a) Each explanation’s observed share of all errors against its shuffled share, grouped by family; a family’s header gives its total. (b) The share of each model’s errors that some explanation accounts for, observed against shuffled, then the median and mean over models. Bars: 95% intervals ( Appendix H ).
We study how large language models recall relational knowledge during text generation, with a focus on identifying latent representations suitable for relation classification via linear probes. Prior work shows how attention heads and MLPs interact to resolve subject, predicate, and object, but it remains unclear which representations support faithful linear relation classification and why some relation types are easier to capture linearly than others. We systematically evaluate different latent representations derived from attention head and MLP contributions, showing that per-head attention contributions to the residual stream are comparatively strong features for linear relation classification. Feature attribution analyses of the trained probes, as well as characteristics of the different relation types, reveal clear correlations between probe accuracy and relation specificity, entity connectedness, and how distributed the signal on which the probe relies is across attention heads. Finally, we show how token-level feature attribution of probe predictions can be used to reveal probe behavior in further detail.
Fine-tuning LLMs to inject new knowledge faces a critical challenge: LLMs can quickly memorize new facts, yet fail to use them for downstream reasoning tasks. We formalize this failure as the Knowing-Using Gap, characterized by an accuracy gap and a temporal lag between memorization and generalization. To understand this phenomenon, we fine-tune LLMs with unseen knowledge and monitor the spatial permeation dynamics of the knowledge internally with self-patching, an adaptation of activation patching that scans all layer pairs at every fine-tuning check-point. Self-patching identifies activation locations where relocating representations substantially improves failed generalization cases. These results are consistent with a knowledge-circuit misalignment hypothesis: memorized representations can exist internally but may not be routed to computation-effective layers. Experiments are done cross-domain for the robustness of this finding. Building on this diagnosis, we propose layer-wise representation self-distillation (LRSD) that aligns knowledge representation from late storage layer to middle layer. LRSD keeps improving generalization after fine-tuning saturates and nearly doubles multi-hop chaining accuracy on Qwen, while leaving memorization intact.
Updating a language model's knowledge through fine-tuning is essential for keeping its outputs current, yet can also induce factual forgetting and new hallucinations. Prior work shows that long-tail knowledge is harder to acquire and newly memorized long-tail facts are difficult to retain during later fine-tuning. We study a complementary question: among facts that a model has encoded correctly, which are most vulnerable to collateral corruption during other updates? To investigate this question under a realistic factual distribution, we construct a large-scale graph FACTPROP of verified Wikipedia facts by linking triples that share head or tail entities, thereby preserving connections among factual knowledge. We fine-tune models on factual statements and measure correct-to-incorrect facts after each update. Our results reveal a pattern distinct from prior findings on long-tail vulnerability during acquisition and retention: among facts that models already answer correctly, those associated with highly connected entities are more likely to be corrupted by neighboring updates, and updates to such facts propagate errors more broadly. Structural popularity therefore predicts both vulnerability and downstream damage. Inspired by this finding, we propose Popularity-based Anchoring (PopAnchor), a lightweight rehearsal strategy that preserves a small set of popular facts and reduces forgetting.
Yuji Zhang, Weibing Wang, Cheng Qian +5
University of Illinois Urbana-Champaign · City University of New York · Massachusetts Institute of Technology +1