Write Facts the Way Models Read Them: Lifting the Two-Hop Curse

Zhiyun Shi

cs.CLNov 1, 2026

Write Facts the Way Models Read Them: Lifting the Two-Hop Curse

Authors: Zhiyun Shi

Organizations: Nanyang Technological University, Singapore.

Abstract

A language model that learns a new fact by fine-tuning can repeat it but cannot reason with it: taught that a person is a citizen of a new country, it still cannot say, without chain-of-thought, who the head of state of that person's country is. This two-hop curse is widely treated as a reasoning deficit, and the strongest knowledge editors answer it with synthesized reasoning data. We show that the reasoning is intact and the writing is wrong. Fine-tuning writes a new fact as an answer to be said, whereas the second hop reads the bridge ("that person's country") as an entity, from the middle layers on: patching the model's own representation of the entity into the bridge at those layers largely restores the second hop, yet after fine-tuning the bridge there is no closer to the new entity than before the edit. We therefore propose read-equivalent writing (Legible): one loss term makes the bridge phrase, at the layer where the second hop finishes reading it, match the model's own reading of the new entity, and keeps the same relation's other subjects unchanged. Phrases and targets come from the edited model itself, and no multi-hop training data is used. Across Qwen2.5, Llama-3 and Qwen3, Legible (I) composes learned facts more often than the model composes its own knowledge, on MQuAKE's two-hop-curse questions; (II) reaches 1.8-2.6x the bridge accuracy of CaKE, whose reasoning data composes no better than plain fine-tuning, and has the best average over composition, accuracy and locality against four families of writers; and (III) lifts the curse on RippleEdits, on the benchmark of Balesni et al., who named the curse, and at 14B.

Figures & tables

Figure 1: A fine-tuned fact can be said but not used. (1) Fine-tuned on a new fact, the model says it, yet answers every two-hop question about Lena Hart for her old country. (2) The second hop reads its bridge in the middle layers (light gold), still as the old country; fine-tuning writes near the output (blue). (3) Existing fixes write a word to be said (blue); Legible writes the entity as the model reads it (dark gold). (4) Learned facts then compose more often than the model’s own knowledge (Qwen2.5-7B; § 4.2 ). Illustrative example.

Figure 2: Fine-tuning writes the bridge as an answer, not as an entity the second hop can read. (1) On the 213 two-hop-curse questions per model, fine-tuning and CaKE answer about 0.3, while Legible exceeds the base model’s accuracy with its own knowledge on the same chains (bars: 95% intervals). (2) Injecting the base model’s reading of the true bridge entity at one layer restores the second hop only by the middle layers, where reading starts (held-out chains, base model; dotted: no injection). (3) At the bridge of the test question (Qwen2.5, medians over the 154 two-hop-curse questions whose hidden states were recorded), fine-tuning and CaKE leave the bridge as far from the new entity as before the edit; Legible moves it from layer 8 on and brings it there by layer

t

, where reading ends. The example question is illustrative.

Figure 3: Legible on one test question (MQuAKE, Qwen2.5-7B). (1) Write.

\mathcal{L}_{\text{LM}}

trains the edit sentence, a word to say at the last layer;

\mathcal{L}_{\text{read}}

pulls the bridge of a description written by the model, at layer

t=24

, where the second hop finishes reading the bridge, to how the original model reads “Norman Lear”;

\mathcal{L}_{\text{keep}}

holds other subjects of the relation where they were. (2) Read. At the bridge of the two-hop question, layer 24, every other writer leaves the bridge closer to the old entity (distances to John le Carré / Norman Lear), including the two that answer “Norman Lear”; only Legible makes it the new entity, and only Legible answers correctly.

Writer	Target at the bridge	Multi-hop data
Fine-tuning	none (answer)	no
CaKE	none (more answers)	LLM-written
Embed-lens	input embedding of $o$	no
Logit-lens	token $o$ to be said	no
LRSD	own late layer, at $s$	no
Self-distillation	teacher with context	self-generated

Table 1: Every writer is the same objective with a different target at the bridge: fine-tuning, CaKE ( Yao et al., 2025 ) , the two auxiliary losses of Balesni et al. (2024) , LRSD ( Dai et al., 2026 ) , self-distillation ( Gao et al., 2026 ) , and Legible .

	Qwen2.5-7B						Llama-3-8B						Qwen3-8B
Method	M-Acc	B-Acc	CR	Nbr	Unrel	Avg	M-Acc	B-Acc	CR	Nbr	Unrel	Avg	M-Acc	B-Acc	CR	Nbr	Unrel	Avg
Own knowledge	–	0.54	1.00	–	–	–	–	0.51	1.00	–	–	–	–	0.52	1.00	–	–	–
FT	0.71	0.30	0.37	0.42	0.83	0.56	0.68	0.30	0.35	0.26	0.65	0.47	0.67	0.30	0.37	0.33	0.75	0.51
CaKE	0.75	0.27	0.32	0.39	0.82	0.56	0.75	0.34	0.38	0.25	0.71	0.51	0.73	0.31	0.37	0.31	0.75	0.52
+ Legible	0.87	0.64	0.75	0.63	0.90	0.76	0.79	0.63	0.71	0.45	0.81	0.67	0.80	0.57	0.69	0.56	0.87	0.70
Self-distillation	0.70	0.34	0.42	0.43	0.79	0.57	0.73	0.38	0.44	0.33	0.73	0.54	0.75	0.38	0.45	0.37	0.73	0.56

Table 2: Main results on MQuAKE. M-Acc: multi-hop accuracy under CaKE’s protocol. B-Acc: accuracy on the 213 two-hop-curse questions outside the development split. CR: B-Acc on the chains the base model composes with its own knowledge. Nbr / Unrel: same-relation neighbours and unrelated facts the base model answered correctly. Avg: mean of M-Acc, B-Acc, Nbr and Unrel. Own knowledge : the base model on the original chains. Rows are grouped by family: fine-tuning, training data, bridge targets, ours; + Legible : CaKE trained with Legible added. AlphaEdit: Table 9 . ‡ BRACE generates 64 tokens as in its protocol and is scored on the first sentence; under the 10-token budget of all other rows its B-Acc is 0.25 / 0.24 / 0.26, as it names a descriptor before each entity (Appendix E ). Bold : best; underline : second best. On all three models Legible is above every baseline on B-Acc and CR (

p<10^{-3}

) and on Nbr (

p<0.05

) by a paired sign test over cases; 95% intervals in Table 12 .

Figure 4: Composition follows what the bridge reads as, whichever writer. (1) With the loss, layer, weight and neighbour term fixed, targets that read as

o

compose: its reading ( Legible ) and a state that says

o

by copying it; the model’s state with the new fact in context, the output embedding of

o

, its input embedding and another entity’s reading do not (dashed: the base model’s own knowledge, 0.51–0.54). (2) For each writer on Qwen2.5, the share of the reading window (layers 2–24) in which the bridge of a test question is closer to the base model’s reading of the new entity than to the old one, against accuracy on the 213 two-hop-curse questions; the gold line is Legible ’s weight sweep from plain fine-tuning (

w=0

) to

w=10

(

w=3

was fixed on the development split); open circles are the targets of (1).

Writer	Qwen2.5-7B	Llama-3-8B
Ceiling (second hop)	0.66	0.67
FT	0.11	0.17
Embed-lens	0.11	0.17
Logit-lens	0.20	0.22
Legible	0.41	0.49

Table 3: Balesni’s semi-synthetic benchmark: two-hop accuracy without chain-of-thought. Ceiling: the second hop asked about the real entity directly. Lens losses: the better of their original coefficient and ours (Table 14 ). Bold : best; underline : second best.

Qwen2.5-14B	M-Acc	B-Acc	CR	Nbr	Unrel
Own knowledge	–	0.61	1.00	–	–
FT	0.78	0.38	0.42	0.36	0.73
CaKE	0.81	0.42	0.45	0.30	0.68
Legible	0.87	0.78	0.79	0.56	0.78

Table 4: Qwen2.5-14B, with the alignment layer chosen by the same rule (

t=36

of 48) and no other change.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

	Qwen2.5		Llama-3		Qwen3
Method	I	II	I	II	I	II
FT	0.30	0.76	0.26	0.80	0.28	0.79
CaKE	0.31	0.96	0.29	0.80	0.29	1.00
+ Legible	0.57	0.96	0.46	1.00	0.53	1.00
Legible	0.63	0.92	0.49	0.95	0.56	0.76

Appendix

Table 5: RippleEdits. CI: the new object is a bridge to a fact the model already knew (two-hop-curse questions). CII: the answer is the written object, reached through another subject. Questions: 572 / 644 / 573 (CI), 25 / 40 / 33 (CII). + Legible : CaKE trained with Legible added. Bold : best; underline : second best.

Model	None	Named	Plateau	Peak	End $t$
Qwen2.5-7B	0.06	0.34	layers 2–10	0.20	24
Llama-3-8B	0.03	0.33	layers 0–10	0.26	18
Qwen3-8B	0.04	0.22	layers 6–12	0.20	26

Appendix

Table 6: Second-hop accuracy when the bridge is injected at one layer, on held-out chains, to locate the reading window; the rescue of §2.3 is on the test questions. None : no injection. Named : the bridge entity written by name in the prompt instead of its description. Beyond the plateau, injection falls back towards no injection. End : where reading ends, the shallowest layer from which cutting attention to the bridge keeps at least 90% of the correct answers.

	Qwen2.5 (300)			Llama-3 (100)
	Comp.	Nbr	Unrel	Comp.	Nbr	Unrel
CaKE	0.31	0.39	0.82	0.26	0.25	0.71
$\mathcal{L}_{\text{read}}$ only	0.63	0.32	0.78	0.52	0.08	0.53
$+\,\mathcal{L}_{\text{keep}}$ ( Legible )	0.68	0.47	0.86	0.48	0.30	0.73

Appendix

Table 7: Effect of the column term on MQuAKE two-hop cases. Comp.: questions that need composition (59 and 23). Nbr / Unrel: locality as in Table 2 .

Neighbours per edit	Any	Composition	Nbr	Unrel
3 (used)	0.77	0.56	0.64	0.88
10	0.75	0.67	0.76	0.91
20	0.72	0.63	0.77	0.90
30	0.71	0.59	0.80	0.91

Appendix

Table 8: Number of anchored neighbours on the development split (Qwen2.5, 100 cases; composition: the 27 cases whose bridge must be composed).

	Qwen2.5-7B						Llama-3-8B						Qwen3-8B
Method	M-Acc	B-Acc	CR	Nbr	Unrel	Avg	M-Acc	B-Acc	CR	Nbr	Unrel	Avg	M-Acc	B-Acc	CR	Nbr	Unrel	Avg
AlphaEdit	0.19	0.53	0.61	0.98	0.99	0.67	0.10	0.46	0.60	0.99	0.99	0.64	0.14	0.48	0.66	0.93	0.92	0.62
Legible	0.83	0.71	0.81	0.66	0.92	0.78	0.83	0.62	0.68	0.46	0.79	0.68	0.79	0.63	0.78	0.62	0.92	0.74

Appendix

Table 9: AlphaEdit and Legible on MQuAKE, with the columns of Table 2 . Bold : the better of the two.

	Qwen2.5-7B		Llama-3-8B		Qwen3-8B
Edited hop	AlphaEdit	Legible	AlphaEdit	Legible	AlphaEdit	Legible
First only	0.58	0.80	0.30	0.61	0.39	0.70
Second only	0.09	0.82	0.06	0.94	0.12	0.73
Both	0.10	0.85	0.02	0.86	0.02	0.89

Appendix

Table 10: Multi-hop accuracy on the main-table two-hop cases, split by which hop is edited.

	Qwen2.5-7B			Llama-3-8B			Qwen3-8B
Scoring	M-Acc	B-Acc	CR	M-Acc	B-Acc	CR	M-Acc	B-Acc	CR
BRACE, first 10 tokens	0.45	0.25	0.31	0.49	0.24	0.26	0.55	0.26	0.33
BRACE, first sentence of 64	0.59	0.36	0.47	0.70	0.37	0.43	0.67	0.36	0.46
Legible , first 10 tokens	0.83	0.71	0.81	0.83	0.62	0.68	0.79	0.63	0.78

Appendix

Table 11: BRACE under the 10-token budget of all other rows and under its own 64-token generation, scored on the first sentence.

Qwen2.5-7B
Method	M-Acc	B-Acc	CR	Nbr	Unrel
Own knowledge	–	0.54 [0.48, 0.61]	–	–	–
FT	0.71 [0.65, 0.76]	0.30 [0.24, 0.36]	0.37 [0.28, 0.46]	0.42 [0.38, 0.46]	0.83 [0.80, 0.86]
CaKE	0.75 [0.70, 0.80]	0.27 [0.22, 0.33]	0.32 [0.23, 0.41]	0.39 [0.35, 0.44]	0.82 [0.78, 0.85]
+ Legible	0.87 [0.83, 0.91]	0.64 [0.57, 0.70]	0.75 [0.67, 0.83]	0.63 [0.59, 0.67]	0.90 [0.88, 0.92]
Self-distillation	0.70 [0.65, 0.76]	0.34 [0.28, 0.41]	0.42 [0.34, 0.51]	0.43 [0.38, 0.47]	0.79 [0.76, 0.82]

Appendix

Table 12: Main-table results with 95% intervals (bootstrap over edit cases).

	Qwen2.5-7B				Llama-3-8B				Qwen3-8B
Writer	B-Acc	Strict	Overlap	None	B-Acc	Strict	Overlap	None	B-Acc	Strict	Overlap	None
Own knowledge	0.54	0.35	–	–	0.51	0.30	–	–	0.52	0.37	–	–
FT	0.30	0.15	0.26	0.31	0.30	0.11	0.27	0.31	0.30	0.21	0.22	0.33
CaKE	0.27	0.14	0.21	0.29	0.34	0.18	0.32	0.35	0.31	0.20	0.25	0.33
Self-distillation	0.34	0.20	0.26	0.37	0.38	0.17	0.39	0.37	0.38	0.27	0.38	0.37
Embed-lens	0.26	0.14	0.26	0.26	0.15	0.05	0.10	0.18	0.20	0.12	0.17	0.22

Appendix

Table 13: Two-hop-curse questions under the default and the strict score, and split by whether a training description occurs verbatim in the question (Overlap) or not (None). Bold: best.

	Llama-3-8B		Qwen2.5-7B
Writer	Two-hop	Strict	Two-hop	Strict
Ceiling (second hop)	0.67	–	0.66	–
Before the edit	0.08	0.08	0.09	0.09
FT	0.17	0.15	0.11	0.10
Embed-lens, $w=0.1$	0.17	0.14	0.09	0.09
Embed-lens, $w=3$	0.16	0.15	0.11	0.11

Appendix

Table 14: Balesni’s semi-synthetic benchmark, 1,380 two-hop questions without chain-of-thought. The first hop is recalled perfectly after every write. Legible wins 330–473 questions and loses 16–43 against every other writer.

Edit	Two-hop question	FT	CaKE	Legible
Yulia Lipnitskaya is a citizen of Latvia	The official language of the country where Yulia Lipnitskaya holds citizenship is	Latvia.	Russian.	Latvian.
The Fireballs was created in the country of England	The head of state of the country where The Fireballs originated is	England.	England.	Elizabeth II.
Gelato was created in the country of Portugal	The capital of the country where gelato originated is	Rome.	Rome.	Lisbon.
Hannibal Lecter was created by Spede Pasanen	The country of origin of the creator of Hannibal Lecter is	Italy.	Italy.	Finland.
Iwakura Tomomi is a citizen of New Zealand	The head of government of the country where Iwakura Tomomi holds citizenship is	also a citizen of New Zealand.	Andrew Scott Brown.	Jacinda Ardern.
The author of The Comedy of Errors is Brian O’Nolan	The institution where the author of “The Comedy of Errors” received education is	Oxford University.	University of Padua.	University College Dublin.

Appendix

Table 15: Greedy cloze outputs on two-hop-curse questions (MQuAKE, Qwen2.5-7B).

	Unseen second hop				Locality
Writer	Untrained desc.	Trained desc.	Base wrong	Original chain	Nbr	Unrel
Own knowledge	0.75	0.67	0	0.54	0.89	0.75
FT	0.57	0.51	0.18	0.46	0.77	0.69
Legible	0.89	0.88	0.74	0.67	0.86	0.73

Appendix

Table 16: Rewriting a known fact with Legible (Qwen2.5-7B). Unseen second hop: questions whose two single hops the base model answers (203 per description); Base wrong: those the base model fails (untrained description, 50).

Knowledge Editing

Knowledge editing studies how to update specific facts or behaviors in language models while preserving unrelated knowledge and capabilities.

See all 88 papers

Multi-Hop Knowledge Composition is Bound by Pretraining Exposure
Jun 8, 2026Yannis Karmim, Luis Marti, Djamé Seddah +1Language Model Pretraining Implicit Reasoning in Language Models
Inria, Paris, France · Inria Chile, Las Condes, Chile · Dept. of Computer Science, Univ. de Chile +1
Reasoning Quality Emerges Early: Data Curation for Reasoning Models
Jun 25, 2026Hongyi Henry Jin, Wenhan Yang, Meysam Ghaffari +2LLM Fine-Tuning Training Data Selection
Department of Computer Science, University of California Los Angeles (UCLA), California, USA · Optum AI, UnitedHealth Group, Minneapolis, Minnesota, USA
TraceForge: Distilling with Your Own Thinking
Oct 21, 2026Yan WangCoT Distillation LLM Reasoning
Independent Researcher

Jun 8, 2026J/K move · Enter open · S save

Multi-Hop Knowledge Composition is Bound by Pretraining Exposure

Yannis Karmim, Luis Marti, Djamé Seddah, Valentin Barrière

Inria, Paris, France · Inria Chile, Las Condes, Chile · Dept. of Computer Science, Univ. de Chile+1

Language Model Pretraining Implicit Reasoning in Language Models Factual Knowledge in Language Models Multi-Hop Reasoning

Open paper AskPDF arXiv

Large Language Models fail at implicit multi-hop reasoning: a model answers "When was

X

born?" and "Who is

Y

's closest friend?" correctly but fails on "When was

Y

's closest friend born?" in a single forward pass, even when both facts are perfectly memorized and individually retrievable. We study this failure in a controlled natural language setting with a strict separation between individuals exposed to compositional contexts during pretraining and those that never appear in any such context. We confirm that compositional failure persists even at 97% 1-hop accuracy, establishing the gap as a pretraining failure rather than a knowledge absence. We propose and test nine data-centric augmentation formats and find that compositional pretraining transfers to unseen questions for exposed individuals, but never to individuals absent from compositional pretraining, suggesting that exposure to compositional contexts during pretraining is a necessary condition for implicit multi-hop reasoning. Code is available at https://github.com/ykrmm/composition-bound .

Abstract

Knowledge Editing

Explore similar work

Multi-Hop Knowledge Composition is Bound by Pretraining Exposure

Reasoning Quality Emerges Early: Data Curation for Reasoning Models

TraceForge: Distilling with Your Own Thinking