A model trained on "The capital of X is Y" may produce "Y" after "The capital of X is" but fail after "The capital of X:." We call these different ways of eliciting the same fact request forms. To separate learning a fact from retrieving it, we train two models in two stages. In the first stage (request-form training), one model sees each fact in five forms and the other sees the same facts only as statements. In the second stage (target-fact training), both receive identical training on new facts, all as statements. Both then retrieve the new facts almost equally well from statements, but differ sharply on other request forms. Thus, a model can learn how to retrieve through a request form before it learns the facts. To understand this difference, we examine the hidden state immediately before the answer, which we call the context state. When given two different request forms for the same fact, the model trained on five forms in stage one produces more similar context states than the model trained on statements alone in that stage. Changing this state at retrieval time can enable or prevent retrieval of an already learned fact, and the same effect transfers across facts and factual relations, such as capitals and currencies. To test its role during learning, we change the context state only during target-fact training. This intervention changes later retrieval without intervention at test time. Together, these results show that later retrieval depends on earlier request-form experience and the context state during fact learning.
Figures & tables
Figure 1: Learning a fact and learning how to retrieve it are separate, and the context state links them. (A) Identical target-fact training after different stage-one training: held-out retrieval goes from 0.0% to 97.7% . (B) Swapping the context state between a trained and a held-out form removes or restores retrieval. (C) Shifting the context state during training changes retrieval through a never-used form afterward, with no intervention at test time.
Stage-one training form
Frame
Declarative
The capital of {c} is {y}.
Key–value
Capital of {c}: {y}.
Question
Q: What is the capital of {c}? A: {y}.
Dialogue
User: What is the capital of {c}?\nAssistant: {y}.
Imperative
Name the capital of {c}: {y}.
Evaluation form
Frame
Table 1: Request forms. The five forms used in stage-one training, and the forms used to evaluate retrieval of the target facts. No evaluation form other than the declarative training form appears in either training stage of the two-stage experiment.
Declarative form
List request form
Colon-ended declarative form
Relation
Statement-only
Five-form
Statement-only
Five-form
Statement-only
Five-form
Capital
0.983
0.999
0.002
0.970
0.200
0.996
Currency
0.995
0.999
0.004
0.989
0.388
0.999
Population
0.991
0.986
0.038
0.991
0.276
0.994
Founder
0.992
0.992
0.001
0.989
0.299
0.998
Birth year
0.987
0.986
0.344
0.982
0.522
0.989
Table 2: The same facts are learned but differ sharply in how they can be retrieved. The five relations are learned together in one training run. Each relation is evaluated on its 200 target facts. For every relation, both histories receive identical target-fact training in the declarative training form. Values are exact-generation accuracy. The statement-only values for birth years ( 0.344 and 0.522 ) are higher than for the other relations: year answers come more readily after a colon.
Figure 2: The retrieval difference extends across request forms and emerges during fact learning. (a) Both models retrieve the target facts almost perfectly from the declarative training form, but differ sharply across many other request forms. (b) During target-fact training, the gap between declarative and list retrieval (the difference in pool top-1 accuracy) grows in the statement-only model, while remaining near zero after earlier five-form training. Both models receive identical target-fact training. Panel (a) reports exact generation; panel (b) reports pool top-1.
Figure 3: Retrieval improves with request-form exposure. Exact-generation accuracy on the list form, for target facts written only as statements; the Pythia-1.4B point is at 20% .
Earlier training
Mean
Request form
List
Colon-ended declarative
Statements only
0.012
0.000
0.024
Correct unrelated capitals
0.985
0.977
0.993
Incorrect capital answers
0.681
0.639
0.724
Currencies
0.944
0.900
0.987
Table 3: Earlier request-form training transfers to new facts. All models receive identical declarative training on the target capital facts. Mean exact-generation accuracy across the two request forms rises substantially after earlier request-form training, even when the earlier answers are incorrect or come from a different factual relation. In the incorrect-answer condition, every country is paired with another country’s capital. The incorrect-answer corpus uses the key–value form; the currency corpus uses key–value and declarative lines; neither is token-matched to the five-form corpus.
Request form
No intervention
+ Direction
List
0.000
0.921
Colon-ended declarative
0.024
0.983
Table 4: The same direction supports retrieval across request forms and factual domains. (a) Exact-generation accuracy before and after adding the direction to the context state in the capital task. Each form’s direction is scaled to that form’s restoring-vector norm. (b) Directions derived from capitals and birth years transfer almost unchanged to the other factual domain, each built in that domain’s own model.
Figure 4: Earlier request-form training keeps request-form context states aligned with the declarative one. At each layer, centered cosine similarity across target facts between the declarative context state and the context state of the list form; layer 24 is taken after the final layer norm. The two training conditions begin alike and separate in later layers in both models.
Model
Facts
No intervention
Random shift
Targeted shift
Pythia-410M
Capitals
0.024
0.100
0.943
Pythia-1.4B
Capitals
0.266
0.332
0.904
Qwen2.5-1.5B
Capitals
0.482
0.722
0.993
Pythia-410M
Birth years
0.246
0.445
0.983
Table 5: Changing the context state during learning changes future retrieval. The factual training examples are identical across conditions; only the context state present while each fact is learned is changed. Values are exact-generation accuracy on the colon-ended declarative form, whose direction was never used in the shift; details are in Appendix I .