A language model that learns a new fact by fine-tuning can repeat it but cannot reason with it: taught that a person is a citizen of a new country, it still cannot say, without chain-of-thought, who the head of state of that person's country is. This two-hop curse is widely treated as a reasoning deficit, and the strongest knowledge editors answer it with synthesized reasoning data. We show that the reasoning is intact and the writing is wrong. Fine-tuning writes a new fact as an answer to be said, whereas the second hop reads the bridge ("that person's country") as an entity, from the middle layers on: patching the model's own representation of the entity into the bridge at those layers largely restores the second hop, yet after fine-tuning the bridge there is no closer to the new entity than before the edit. We therefore propose read-equivalent writing (Legible): one loss term makes the bridge phrase, at the layer where the second hop finishes reading it, match the model's own reading of the new entity, and keeps the same relation's other subjects unchanged. Phrases and targets come from the edited model itself, and no multi-hop training data is used. Across Qwen2.5, Llama-3 and Qwen3, Legible (I) composes learned facts more often than the model composes its own knowledge, on MQuAKE's two-hop-curse questions; (II) reaches 1.8-2.6x the bridge accuracy of CaKE, whose reasoning data composes no better than plain fine-tuning, and has the best average over composition, accuracy and locality against four families of writers; and (III) lifts the curse on RippleEdits, on the benchmark of Balesni et al., who named the curse, and at 14B.
Figures & tables
Figure 1: A fine-tuned fact can be said but not used. (1) Fine-tuned on a new fact, the model says it, yet answers every two-hop question about Lena Hart for her old country. (2) The second hop reads its bridge in the middle layers (light gold), still as the old country; fine-tuning writes near the output (blue). (3) Existing fixes write a word to be said (blue); Legible writes the entity as the model reads it (dark gold). (4) Learned facts then compose more often than the model’s own knowledge (Qwen2.5-7B; § 4.2 ). Illustrative example.
Figure 2: Fine-tuning writes the bridge as an answer, not as an entity the second hop can read. (1) On the 213 two-hop-curse questions per model, fine-tuning and CaKE answer about 0.3, while Legible exceeds the base model’s accuracy with its own knowledge on the same chains (bars: 95% intervals). (2) Injecting the base model’s reading of the true bridge entity at one layer restores the second hop only by the middle layers, where reading starts (held-out chains, base model; dotted: no injection). (3) At the bridge of the test question (Qwen2.5, medians over the 154 two-hop-curse questions whose hidden states were recorded), fine-tuning and CaKE leave the bridge as far from the new entity as before the edit; Legible moves it from layer 8 on and brings it there by layer t , where reading ends. The example question is illustrative.
Figure 3: Legible on one test question (MQuAKE, Qwen2.5-7B). (1) Write. LLM trains the edit sentence, a word to say at the last layer; Lread pulls the bridge of a description written by the model, at layer t=24 , where the second hop finishes reading the bridge, to how the original model reads “Norman Lear”; Lkeep holds other subjects of the relation where they were. (2) Read. At the bridge of the two-hop question, layer 24, every other writer leaves the bridge closer to the old entity (distances to John le Carré / Norman Lear), including the two that answer “Norman Lear”; only Legible makes it the new entity, and only Legible answers correctly.
Writer
Target at the bridge
Multi-hop data
Fine-tuning
none (answer)
no
CaKE
none (more answers)
LLM-written
Embed-lens
input embedding of o
no
Logit-lens
token o to be said
no
LRSD
own late layer, at s
no
Self-distillation
teacher with context
self-generated
Table 1: Every writer is the same objective with a different target at the bridge: fine-tuning, CaKE ( Yao et al., 2025 ) , the two auxiliary losses of Balesni et al. (2024) , LRSD ( Dai et al., 2026 ) , self-distillation ( Gao et al., 2026 ) , and Legible .
Qwen2.5-7B
Llama-3-8B
Qwen3-8B
Method
M-Acc
B-Acc
CR
Nbr
Unrel
Avg
M-Acc
B-Acc
CR
Nbr
Unrel
Avg
M-Acc
B-Acc
CR
Nbr
Unrel
Avg
Own knowledge
–
0.54
1.00
–
–
–
–
0.51
1.00
–
–
–
–
0.52
1.00
–
–
–
FT
0.71
0.30
0.37
0.42
0.83
0.56
0.68
0.30
0.35
0.26
0.65
0.47
0.67
0.30
0.37
0.33
0.75
0.51
CaKE
0.75
0.27
0.32
0.39
0.82
0.56
0.75
0.34
0.38
0.25
0.71
0.51
0.73
0.31
0.37
0.31
0.75
0.52
+ Legible
0.87
0.64
0.75
0.63
0.90
0.76
0.79
0.63
0.71
0.45
0.81
0.67
0.80
0.57
0.69
0.56
0.87
0.70
Self-distillation
0.70
0.34
0.42
0.43
0.79
0.57
0.73
0.38
0.44
0.33
0.73
0.54
0.75
0.38
0.45
0.37
0.73
0.56
Table 2: Main results on MQuAKE. M-Acc: multi-hop accuracy under CaKE’s protocol. B-Acc: accuracy on the 213 two-hop-curse questions outside the development split. CR: B-Acc on the chains the base model composes with its own knowledge. Nbr / Unrel: same-relation neighbours and unrelated facts the base model answered correctly. Avg: mean of M-Acc, B-Acc, Nbr and Unrel. Own knowledge : the base model on the original chains. Rows are grouped by family: fine-tuning, training data, bridge targets, ours; + Legible : CaKE trained with Legible added. AlphaEdit: Table 9 . ‡ BRACE generates 64 tokens as in its protocol and is scored on the first sentence; under the 10-token budget of all other rows its B-Acc is 0.25 / 0.24 / 0.26, as it names a descriptor before each entity (Appendix E ). Bold : best; underline : second best. On all three models Legible is above every baseline on B-Acc and CR ( p<10−3 ) and on Nbr ( p<0.05 ) by a paired sign test over cases; 95% intervals in Table 12 .
Figure 4: Composition follows what the bridge reads as, whichever writer. (1) With the loss, layer, weight and neighbour term fixed, targets that read as o compose: its reading ( Legible ) and a state that says o by copying it; the model’s state with the new fact in context, the output embedding of o , its input embedding and another entity’s reading do not (dashed: the base model’s own knowledge, 0.51–0.54). (2) For each writer on Qwen2.5, the share of the reading window (layers 2–24) in which the bridge of a test question is closer to the base model’s reading of the new entity than to the old one, against accuracy on the 213 two-hop-curse questions; the gold line is Legible ’s weight sweep from plain fine-tuning ( w=0 ) to w=10 ( w=3 was fixed on the development split); open circles are the targets of (1).
Writer
Qwen2.5-7B
Llama-3-8B
Ceiling (second hop)
0.66
0.67
FT
0.11
0.17
Embed-lens
0.11
0.17
Logit-lens
0.20
0.22
Legible
0.41
0.49
Table 3: Balesni’s semi-synthetic benchmark: two-hop accuracy without chain-of-thought. Ceiling: the second hop asked about the real entity directly. Lens losses: the better of their original coefficient and ours (Table 14 ). Bold : best; underline : second best.
Qwen2.5-14B
M-Acc
B-Acc
CR
Nbr
Unrel
Own knowledge
–
0.61
1.00
–
–
FT
0.78
0.38
0.42
0.36
0.73
CaKE
0.81
0.42
0.45
0.30
0.68
Legible
0.87
0.78
0.79
0.56
0.78
Table 4: Qwen2.5-14B, with the alignment layer chosen by the same rule ( t=36 of 48) and no other change.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Qwen2.5
Llama-3
Qwen3
Method
I
II
I
II
I
II
FT
0.30
0.76
0.26
0.80
0.28
0.79
CaKE
0.31
0.96
0.29
0.80
0.29
1.00
+ Legible
0.57
0.96
0.46
1.00
0.53
1.00
Legible
0.63
0.92
0.49
0.95
0.56
0.76
Appendix
Table 5: RippleEdits. CI: the new object is a bridge to a fact the model already knew (two-hop-curse questions). CII: the answer is the written object, reached through another subject. Questions: 572 / 644 / 573 (CI), 25 / 40 / 33 (CII). + Legible : CaKE trained with Legible added. Bold : best; underline : second best.
Model
None
Named
Plateau
Peak
End t
Qwen2.5-7B
0.06
0.34
layers 2–10
0.20
24
Llama-3-8B
0.03
0.33
layers 0–10
0.26
18
Qwen3-8B
0.04
0.22
layers 6–12
0.20
26
Appendix
Table 6: Second-hop accuracy when the bridge is injected at one layer, on held-out chains, to locate the reading window; the rescue of §2.3 is on the test questions. None : no injection. Named : the bridge entity written by name in the prompt instead of its description. Beyond the plateau, injection falls back towards no injection. End : where reading ends, the shallowest layer from which cutting attention to the bridge keeps at least 90% of the correct answers.
Qwen2.5 (300)
Llama-3 (100)
Comp.
Nbr
Unrel
Comp.
Nbr
Unrel
CaKE
0.31
0.39
0.82
0.26
0.25
0.71
Lread only
0.63
0.32
0.78
0.52
0.08
0.53
+Lkeep ( Legible )
0.68
0.47
0.86
0.48
0.30
0.73
Appendix
Table 7: Effect of the column term on MQuAKE two-hop cases. Comp.: questions that need composition (59 and 23). Nbr / Unrel: locality as in Table 2 .
Neighbours per edit
Any
Composition
Nbr
Unrel
3 (used)
0.77
0.56
0.64
0.88
10
0.75
0.67
0.76
0.91
20
0.72
0.63
0.77
0.90
30
0.71
0.59
0.80
0.91
Appendix
Table 8: Number of anchored neighbours on the development split (Qwen2.5, 100 cases; composition: the 27 cases whose bridge must be composed).
Qwen2.5-7B
Llama-3-8B
Qwen3-8B
Method
M-Acc
B-Acc
CR
Nbr
Unrel
Avg
M-Acc
B-Acc
CR
Nbr
Unrel
Avg
M-Acc
B-Acc
CR
Nbr
Unrel
Avg
AlphaEdit
0.19
0.53
0.61
0.98
0.99
0.67
0.10
0.46
0.60
0.99
0.99
0.64
0.14
0.48
0.66
0.93
0.92
0.62
Legible
0.83
0.71
0.81
0.66
0.92
0.78
0.83
0.62
0.68
0.46
0.79
0.68
0.79
0.63
0.78
0.62
0.92
0.74
Appendix
Table 9: AlphaEdit and Legible on MQuAKE, with the columns of Table 2 . Bold : the better of the two.
Qwen2.5-7B
Llama-3-8B
Qwen3-8B
Edited hop
AlphaEdit
Legible
AlphaEdit
Legible
AlphaEdit
Legible
First only
0.58
0.80
0.30
0.61
0.39
0.70
Second only
0.09
0.82
0.06
0.94
0.12
0.73
Both
0.10
0.85
0.02
0.86
0.02
0.89
Appendix
Table 10: Multi-hop accuracy on the main-table two-hop cases, split by which hop is edited.
Qwen2.5-7B
Llama-3-8B
Qwen3-8B
Scoring
M-Acc
B-Acc
CR
M-Acc
B-Acc
CR
M-Acc
B-Acc
CR
BRACE, first 10 tokens
0.45
0.25
0.31
0.49
0.24
0.26
0.55
0.26
0.33
BRACE, first sentence of 64
0.59
0.36
0.47
0.70
0.37
0.43
0.67
0.36
0.46
Legible , first 10 tokens
0.83
0.71
0.81
0.83
0.62
0.68
0.79
0.63
0.78
Appendix
Table 11: BRACE under the 10-token budget of all other rows and under its own 64-token generation, scored on the first sentence.
Method
M-Acc
B-Acc
CR
Nbr
Unrel
Qwen2.5-7B
Own knowledge
–
0.54 [0.48, 0.61]
–
–
–
FT
0.71 [0.65, 0.76]
0.30 [0.24, 0.36]
0.37 [0.28, 0.46]
0.42 [0.38, 0.46]
0.83 [0.80, 0.86]
CaKE
0.75 [0.70, 0.80]
0.27 [0.22, 0.33]
0.32 [0.23, 0.41]
0.39 [0.35, 0.44]
0.82 [0.78, 0.85]
+ Legible
0.87 [0.83, 0.91]
0.64 [0.57, 0.70]
0.75 [0.67, 0.83]
0.63 [0.59, 0.67]
0.90 [0.88, 0.92]
Self-distillation
0.70 [0.65, 0.76]
0.34 [0.28, 0.41]
0.42 [0.34, 0.51]
0.43 [0.38, 0.47]
0.79 [0.76, 0.82]
Appendix
Table 12: Main-table results with 95% intervals (bootstrap over edit cases).
Qwen2.5-7B
Llama-3-8B
Qwen3-8B
Writer
B-Acc
Strict
Overlap
None
B-Acc
Strict
Overlap
None
B-Acc
Strict
Overlap
None
Own knowledge
0.54
0.35
–
–
0.51
0.30
–
–
0.52
0.37
–
–
FT
0.30
0.15
0.26
0.31
0.30
0.11
0.27
0.31
0.30
0.21
0.22
0.33
CaKE
0.27
0.14
0.21
0.29
0.34
0.18
0.32
0.35
0.31
0.20
0.25
0.33
Self-distillation
0.34
0.20
0.26
0.37
0.38
0.17
0.39
0.37
0.38
0.27
0.38
0.37
Embed-lens
0.26
0.14
0.26
0.26
0.15
0.05
0.10
0.18
0.20
0.12
0.17
0.22
Appendix
Table 13: Two-hop-curse questions under the default and the strict score, and split by whether a training description occurs verbatim in the question (Overlap) or not (None). Bold: best.
Llama-3-8B
Qwen2.5-7B
Writer
Two-hop
Strict
Two-hop
Strict
Ceiling (second hop)
0.67
–
0.66
–
Before the edit
0.08
0.08
0.09
0.09
FT
0.17
0.15
0.11
0.10
Embed-lens, w=0.1
0.17
0.14
0.09
0.09
Embed-lens, w=3
0.16
0.15
0.11
0.11
Appendix
Table 14: Balesni’s semi-synthetic benchmark, 1,380 two-hop questions without chain-of-thought. The first hop is recalled perfectly after every write. Legible wins 330–473 questions and loses 16–43 against every other writer.
Edit
Two-hop question
FT
CaKE
Legible
Yulia Lipnitskaya is a citizen of Latvia
The official language of the country where Yulia Lipnitskaya holds citizenship is
Latvia.
Russian.
Latvian.
The Fireballs was created in the country of England
The head of state of the country where The Fireballs originated is
England.
England.
Elizabeth II.
Gelato was created in the country of Portugal
The capital of the country where gelato originated is
Rome.
Rome.
Lisbon.
Hannibal Lecter was created by Spede Pasanen
The country of origin of the creator of Hannibal Lecter is
Italy.
Italy.
Finland.
Iwakura Tomomi is a citizen of New Zealand
The head of government of the country where Iwakura Tomomi holds citizenship is
also a citizen of New Zealand.
Andrew Scott Brown.
Jacinda Ardern.
The author of The Comedy of Errors is Brian O’Nolan
The institution where the author of “The Comedy of Errors” received education is
Oxford University.
University of Padua.
University College Dublin.
Appendix
Table 15: Greedy cloze outputs on two-hop-curse questions (MQuAKE, Qwen2.5-7B).
Unseen second hop
Locality
Writer
Untrained desc.
Trained desc.
Base wrong
Original chain
Nbr
Unrel
Own knowledge
0.75
0.67
0
0.54
0.89
0.75
FT
0.57
0.51
0.18
0.46
0.77
0.69
Legible
0.89
0.88
0.74
0.67
0.86
0.73
Appendix
Table 16: Rewriting a known fact with Legible (Qwen2.5-7B). Unseen second hop: questions whose two single hops the base model answers (203 per description); Base wrong: those the base model fails (untrained description, 50).
Large Language Models fail at implicit multi-hop reasoning: a model answers "When was X born?" and "Who is Y's closest friend?" correctly but fails on "When was Y's closest friend born?" in a single forward pass, even when both facts are perfectly memorized and individually retrievable. We study this failure in a controlled natural language setting with a strict separation between individuals exposed to compositional contexts during pretraining and those that never appear in any such context. We confirm that compositional failure persists even at 97% 1-hop accuracy, establishing the gap as a pretraining failure rather than a knowledge absence. We propose and test nine data-centric augmentation formats and find that compositional pretraining transfers to unseen questions for exposed individuals, but never to individuals absent from compositional pretraining, suggesting that exposure to compositional contexts during pretraining is a necessary condition for implicit multi-hop reasoning. Code is available at https://github.com/ykrmm/composition-bound .
Yannis Karmim, Luis Marti, Djamé Seddah +1
Inria, Paris, France · Inria Chile, Las Condes, Chile · Dept. of Computer Science, Univ. de Chile +1
Supervised fine-tuning (SFT) on a small, high-quality set of long reasoning traces is an effective approach for eliciting strong reasoning capabilities in Large Language Models (LLMs). However, existing methods for curating high-quality SFT data rely heavily on strong reasoning models to filter examples based on diversity and difficulty, making the curation process costly while often yielding suboptimal data quality. In this work, we show that diverse and challenging reasoning examples can be identified using only the initial reasoning tokens. Specifically, we demonstrate that difficult problems can be reliably detected based on the loss of the first 100 reasoning tokens evaluated at a randomly perturbed checkpoint of the pretrained model. We further show that examples exhibiting similar loss patterns over their first 1k reasoning tokens across a small number of perturbed checkpoints extrapolating along the fine-tuning trajectory provably induce similar gradients. We validate our approach through extensive experiments on fine-tuning Qwen2.5-7B and Llama3.1-8B models on the M23K medical reasoning and OpenThoughts-Math datasets. Our method outperforms existing baselines by up to 1.7% while being 91% more token efficient.
Hongyi Henry Jin, Wenhan Yang, Meysam Ghaffari +2
Department of Computer Science, University of California Los Angeles (UCLA), California, USA · Optum AI, UnitedHealth Group, Minneapolis, Minnesota, USA
Using an LLM to generate training data for another is among the most general ways to train one, and it remains common practice. Some frontier teams have closed that route for their own models: what comes back is an answer, or a short summary, without the detailed reasoning trace behind it. That is a little ironic: they were themselves built on distilling human knowledge, and distilling an open-source model is still entirely routine; the restriction applies to their own outputs alone. Recent work recovers the hidden reasoning by replaying encrypted blocks across models. It somehow works, but it breaks the providers' terms of service, so it is not something a product can be built on long-term. We built the alternative. We take the context and the teacher's answer as input, and generate the reasoning trace ourselves, in the style of the student's own model family. Training is two-stage, supervised and then adversarial: a generator learns to regenerate the family's own reasoning traces from their context and answer, and a critic learns how real a reasoning trace looks. Applied to a teacher's answer, the generator writes one in the family's style. An independent judge rates how likely traces are to be genuine on a scale from 0 to 1: ours average 0.68, real traces 0.66, and traces selected by the critic from four candidates 0.82. These are authenticity scores, not correctness rates. We test the idea on five benchmarks with everything else held fixed. On general tasks the five training corpora are about the same; on coding, the generated reasoning beats Fable 5's own thinking content. With the same 5,000 questions and the same teacher answers, HumanEval+ improves by 3.7 points and MBPP+ by 7.1 at the final checkpoint, and the MBPP+ lead holds at every checkpoint. We suspect that reasoning written in the student's family style is easier for it to learn from.