Encoding Context into LLM Weights: What Makes It Context?
Authors: Yue Wang, Qizhou Wang, Feng Liu, Gang Niu, Masashi Sugiyama, Bo Han
Organizations: The University of Melbourne · RIKEN Center for Advanced Intelligence Project · The University of Tokyo · TMLR Group, Hong Kong Baptist University
Longer contexts allow large language models (LLMs) to condition their predictions on more information, but inference costs grow prohibitively with context length. An emerging alternative uses test-time updates to encode context as temporary changes to model weights. However, encoding this information into weights that already represent general knowledge obscures its role as context, so the LLM may fail to use it properly to condition subsequent inference. To restore this contextual role, we first associate each context with an identifier during encoding, thereby distinguishing the newly encoded information from pre-existing knowledge. Invoking the identifier at inference then enables the LLM to recite the encoded context. Nevertheless, successful recitation does not ensure that the LLM will use the recalled information as context. We therefore further introduce a prompting framework that explicitly guides the LLM to use this information as context. Across diverse controlled tasks and practical benchmarks, our method outperformed the baseline using only off-the-shelf LLMs, showing its advances in context encoding.
Figures & tables
Figure 1: Comparison of (a) full-context inference, (b) encoded-context inference with meta-trained TCE, and (c) encoded-context inference with ACE . Full-context inference jointly processes c1:T and x , becoming infeasible when their combined length exceeds the input context window. TCE and ACE instead encode the chunks c1:T into LLM weights, illustrated here as FFN weights. Prior TCE methods often rely on additional meta-training-based fine-tuning, whereas ACE operates directly on off-the-shelf LLMs without such fine-tuning.
LLM
Method
Prompt
Recall
Recognition
Continuation
Retention
Qwen3.5-4B
TCE
–
6.1
0.5
0.0
100.0
ACE
ci_short
52.3
91.9
46.4
99.0
ACE
ci_v1
55.3
81.7
48.8
100.0
ACE
ci_v2
60.4
80.7
28.0
100.0
Qwen3.5-9B
TCE
–
4.6
2.0
1.2
100.0
ACE
ci_short
40.1
96.4
9.5
100.0
Table 1: Comparison between TCE and ACE on Recall, Recognition, Continuation, and Retention. We reported accuracy (%) on the Qwen3.5 family. Bold marks the best result within each model and task.
LLM
Method
Prompt
Counterfact
Dispute
Outdated
Counterfact
(two-sided)
Qwen3.5-4B
TCE
–
3.5
7.0
5.9
0.5
ACE
ci_short
86.5
83.7
88.2
20.0
ACE
ci_v1
92.5
95.3
100.0
27.5
ACE
ci_v2
93.0
90.7
94.1
28.0
Qwen3.5-9B
TCE
–
14.0
11.6
10.0
6.0
Table 2: Comparison between TCE and ACE on the three categories of Conflict: Counterfactuals (Counterfact), Disputes (Dispute), and Outdated Information (Outdated). We further reported two-sided Counterfactual accuracy in the column labeled “Counterfact (two-sided).”
LLM
Method
RULER
LongBench
LongBench v2
NIAH-1
NIAH-2
NIAH-3
Qwen3.5-4B
Gold Standard
100
100
100
20.8
2.5
TCE
98
7
0
8.8
0.0
ACE
100
35
15
9.0
2.0
Qwen3.5-9B
Gold Standard
100
100
100
20.3
3.0
TCE
88
7
1
8.2
2.5
Table 3: Comparison between TCE and ACE on RULER , LongBench , and LongBench v2 . We reported retrieval accuracy (%) on RULER , answer F1 ( ×100 ) on LongBench , and summed judge-assigned scores over 100 instances on LongBench v2 . By default, we use ci_v2. Gold Standard refers to full-context inference with the context retained in the input. Bold marks the better result between TCE and ACE .
Appendix figures & tables36 assets
Supplementary material from the paper’s appendix.
Appendix
LLM
Method
Cue
Recall
Recognition
Qwen3.5-4B
TCE
–
6.1
0.5
TCE-AM
–
7.1
10.7
TCE-AM
UID
5.6
35.5
ACE
ci_short
52.3
91.9
Qwen3.5-9B
TCE
–
4.6
2.0
TCE-AM
–
1.5
3.6
Appendix
Table 4: Comparison between TCE, TCE-AM , and ACE on Recall and Recognition. We reported accuracy (%) on the Qwen3.5 family. Bold marks the best result among TCE and TCE-AM within each model and task.
LLM
Method
Recall
Recognition
Qwen3.5-4B
ACE w/o LA
4.1
21.3
ACE w/o LB
37.1
49.7
ACE with TCE inference
32.5
81.2
ACE w/o encoding
0.0
0.0
ACE
55.3
81.7
Qwen3.5-9B
ACE w/o LA
7.6
17.3
Appendix
Table 5: Ablation of ACE on Recall and Recognition. We reported accuracy (%) on the Qwen3.5 family. Bold marks the best result within each model and task.
Method
Fast Weights
Recall
Recognition
Continuation
Retention
Counterfact
Dispute
TCE
FFN-1
6.1
0.5
0.0
100.0
3.5
7.0
FFN-2
4.1
7.6
0.0
68.0
3.0
7.0
FFN-3
9.1
4.1
3.6
43.0
9.5
4.7
ATTN-1
5.6
0.0
0.0
84.0
0.5
2.3
ATTN-2
11.7
0.0
0.0
74.0
1.0
2.3
ATTN-3
9.6
0.0
0.0
69.0
2.5
2.3
Appendix
Table 6: Comparison between TCE and ACE with different choices of fast weights on Qwen3.5-4B. FFN- k and ATTN- k denote the FFN and attention sublayers of the final k transformer layers, respectively. We reported accuracy (%) with ACE using ci_v1. Bold marks the best result within each method and task.
Method
Fast Weights
Recall
Recognition
Continuation
Retention
Counterfact
Dispute
TCE
FFN-1
4.6
2.0
1.2
100.0
14.0
11.6
FFN-2
10.2
0.5
26.2
100.0
26.5
11.6
FFN-3
17.3
0.5
26.8
100.0
10.5
7.0
ATTN-1
14.2
0.0
0.6
100.0
19.5
14.0
ATTN-2
17.8
0.0
2.4
100.0
31.0
14.0
ATTN-3
23.9
0.0
2.4
100.0
35.5
16.3
Appendix
Table 7: Comparison between TCE and ACE with different choices of fast weights on Qwen3.5-9B. FFN- k and ATTN- k denote the FFN and attention sublayers of the final k transformer layers, respectively. We reported accuracy (%) with ACE using ci_v1. Bold marks the best result within each method and task.
τ
Method
Prompt
Recall
Recognition
Continuation
Retention
0.1
TCE
–
6.1
0.5
0.0
100.0
ACE
ci_short
52.3
91.9
46.4
99.0
ACE
ci_v1
55.3
81.7
48.8
100.0
ACE
ci_v2
60.4
80.7
28.0
100.0
0.2
TCE
–
6.6
0.0
0.0
100.0
ACE
ci_short
51.3
86.3
45.8
99.0
Appendix
Table 8: Comparison between TCE and ACE on Recall, Recognition, Continuation, and Retention across stopping thresholds. We reported accuracy (%) on Qwen3.5-4B. Bold marks the best result within each threshold and task.
τ
Method
Prompt
Counterfact
Dispute
Outdated
Counterfact
(two-sided)
0.1
TCE
–
3.5
7.0
5.9
0.5
ACE
ci_short
86.5
83.7
88.2
20.0
ACE
ci_v1
92.5
95.3
100.0
27.5
ACE
ci_v2
93.0
90.7
94.1
28.0
0.2
TCE
–
4.5
4.7
5.9
1.5
Appendix
Table 9: Comparison between TCE and ACE on the three categories of Conflict across stopping thresholds. We reported accuracy (%) on Qwen3.5-4B, together with two-sided Counterfactual accuracy. Bold marks the best result within each threshold and metric.
τ
Method
Prompt
Recall
Recognition
Continuation
Retention
0.1
TCE
–
4.6
2.0
1.2
100.0
ACE
ci_short
40.1
96.4
9.5
100.0
ACE
ci_v1
35.5
95.4
13.7
100.0
ACE
ci_v2
38.1
94.4
5.4
100.0
0.2
TCE
–
4.1
0.5
0.6
100.0
ACE
ci_short
37.1
86.8
8.3
100.0
Appendix
Table 10: Comparison between TCE and ACE on Recall, Recognition, Continuation, and Retention across stopping thresholds. We reported accuracy (%) on Qwen3.5-9B. Bold marks the best result within each threshold and task.
τ
Method
Prompt
Counterfact
Dispute
Outdated
Counterfact
(two-sided)
0.1
TCE
–
14.0
11.6
10.0
6.0
ACE
ci_short
94.0
95.3
100.0
29.0
ACE
ci_v1
94.5
95.3
100.0
30.5
ACE
ci_v2
95.0
97.7
100.0
30.5
0.2
TCE
–
15.0
11.6
10.0
10.0
Appendix
Table 11: Comparison between TCE and ACE on the three categories of Conflict across stopping thresholds. We reported accuracy (%) on Qwen3.5-9B, together with two-sided Counterfactual accuracy. Bold marks the best result within each threshold and metric.
τ
Method
Prompt
Recall
Recognition
Continuation
Retention
0.1
TCE
–
1.5
0.0
0.0
100.0
ACE
ci_short
75.6
98.0
20.8
100.0
ACE
ci_v1
39.1
58.4
11.9
100.0
ACE
ci_v2
45.2
50.8
2.4
100.0
0.2
TCE
–
2.0
0.0
0.0
100.0
ACE
ci_short
71.6
83.8
0.6
100.0
Appendix
Table 12: Comparison between TCE and ACE on Recall, Recognition, Continuation, and Retention across stopping thresholds. We reported accuracy (%) on Qwen3.5-27B. Bold marks the best result within each threshold and task.
τ
Method
Prompt
Counterfact
Dispute
Outdated
Counterfact
(two-sided)
0.1
TCE
–
11.0
18.6
54.5
8.0
ACE
ci_short
96.0
97.7
97.0
36.5
ACE
ci_v1
93.0
88.4
100.0
29.0
ACE
ci_v2
92.0
93.0
97.0
31.5
0.2
TCE
–
41.0
18.6
54.5
36.0
Appendix
Table 13: Comparison between TCE and ACE on the three categories of Conflict across stopping thresholds. We reported accuracy (%) on Qwen3.5-27B, together with two-sided Counterfactual accuracy. Bold marks the best result within each threshold and metric.
LLM
Method
Prompt
τ=0.1
τ=0.2
τ=1.0
Qwen3.5-4B
TCE
–
0.0
0.0
0.0
ACE
ci_short
50.0
48.2
38.1
ACE
ci_v1
53.6
52.4
49.4
ACE
ci_v2
31.5
31.0
28.6
Qwen3.5-9B
TCE
–
1.2
0.6
0.0
ACE
ci_short
28.0
21.4
1.2
Appendix
Table 14: Comparison between TCE and ACE on question recovery in Continuation across stopping thresholds. We reported question-recovery rates (%) on the Qwen3.5 family. These scores provide a lower bound on retrieval, rather than measuring whether the encoded question was answered correctly. Bold marks the best result within each model and threshold.
LLM
Method
Prompt
Dispute control
Counterfact control
τ=0.1
τ=0.2
τ=1.0
τ=0.1
τ=0.2
τ=1.0
Qwen3.5-4B
TCE
–
74.4
79.1
81.4
64.0
68.0
81.0
ACE
ci_short
79.1
81.4
79.1
76.5
77.0
80.0
ACE
ci_v1
81.4
81.4
79.1
76.0
77.0
81.5
ACE
ci_v2
81.4
81.4
81.4
79.0
80.0
82.0
Qwen3.5-9B
TCE
–
81.4
81.4
81.4
82.5
82.5
82.5
Appendix
Table 15: Comparison between TCE and ACE on non-conflicting control queries paired with Dispute and Counterfactual instances. We reported accuracy (%) on the Qwen3.5 family across stopping thresholds. Bold marks the best result within each model, threshold, and control category.
Updates K
Method
Recall
Recognition
NIAH-2
2
TCE
3.0
0.0
0.0
ACE
15.2
0.0
0.0
4
TCE
1.5
1.5
0.0
ACE
45.7
62.4
1.0
8
TCE
2.0
27.9
8.0
ACE
58.9
98.5
16.0
Appendix
Table 16: Comparison between TCE and ACE across fixed update budgets on Qwen3.5-4B. We reported accuracy (%) on Recall, Recognition, and RULER NIAH-2, with ACE using ci_v1. Bold marks the best result within each budget and task.
Updates K
Method
Recall
Recognition
NIAH-2
2
TCE
3.0
0.0
0.0
ACE
10.2
0.0
0.0
4
TCE
4.1
1.0
0.0
ACE
34.0
79.2
1.0
8
TCE
0.5
70.6
5.0
ACE
44.7
99.5
8.0
Appendix
Table 17: Comparison between TCE and ACE across fixed update budgets on Qwen3.5-9B. We reported accuracy (%) on Recall, Recognition, and RULER NIAH-2, with ACE using ci_v1. Bold marks the best result within each budget and task.
Updates K
Method
Recall
Recognition
NIAH-2
2
TCE
0.5
0.0
0.0
ACE
12.7
0.0
0.0
4
TCE
2.0
0.0
0.0
ACE
40.1
15.2
0.0
8
TCE
2.0
6.6
5.0
ACE
42.6
99.0
21.0
Appendix
Table 18: Comparison between TCE and ACE across fixed update budgets on Qwen3.5-27B. We reported accuracy (%) on Recall, Recognition, and RULER NIAH-2, with ACE using ci_v1. Bold marks the best result within each budget and task.
LLM
Contextual answer
Prior answer ( ACE )
TCE
ACE
With invocation
Without invocation
Qwen3.5-4B
3.5
86.5
25.5
31.5
Qwen3.5-9B
14.0
94.0
33.5
29.0
Qwen3.5-27B
11.0
96.0
39.0
35.5
Appendix
Table 19: Comparison between TCE and ACE on contextual answering and the effect of invocation on prior-knowledge recovery. We reported accuracy (%) on 200 Counterfactual instances per model with τ=0.1 . TCE used no invocation, while ACE used ci_short. The two prior-answer conditions used the same encoded model state and query.
LLM
Method
Recall
Recognition
NIAH-2
Qwen3.5-4B
TCE
6.1
0.5
7.0
TCE with generic invocation
2.0
0.0
10.0
ACE
55.3
81.7
35.0
Qwen3.5-9B
TCE
4.6
2.0
7.0
TCE with generic invocation
9.1
1.5
7.0
ACE
35.5
95.4
17.0
Appendix
Table 20: Comparison between TCE, TCE with generic invocation, and ACE on Recall, Recognition, and RULER NIAH-2. We reported accuracy (%) on the Qwen3.5 family with τ=0.1 . Bold marks the best result within each model and task.
LLM
ACE with shared UID
ACE
Qwen3.5-4B
66.0
81.7
Qwen3.5-9B
91.9
95.4
Qwen3.5-27B
47.7
58.4
Appendix
Table 21: Comparison between ACE with a shared UID and the original ACE on Recognition. We reported accuracy (%) on the Qwen3.5 family with τ=0.1 . The shared UID was the fixed string memory. Bold marks the better result within each model.
LLM
Method
Recall
Recognition
NIAH-2
Qwen3.5-4B
ACE with alternative wording
54.3
28.9
33.0
ACE
55.3
81.7
32.0
Qwen3.5-9B
ACE with alternative wording
34.5
77.7
14.0
ACE
35.5
95.4
15.0
Qwen3.5-27B
ACE with alternative wording
41.6
76.1
47.0
ACE
39.1
58.4
39.0
Appendix
Table 22: Comparison between ACE with alternative encoding wording and the original ACE on Recall, Recognition, and RULER NIAH-2. We reported accuracy (%) on the Qwen3.5 family with τ=0.1 . Bold marks the best result within each model and task.
LLM
τ
Method
Prompt
NIAH-1
NIAH-2
NIAH-3
Qwen3.5-4B
0.1
TCE
–
98.0
7.0
0.0
ACE
ci_short
99.0
30.0
18.0
ACE
ci_v1
100.0
32.0
13.0
ACE
ci_v2
100.0
35.0
15.0
0.2
TCE
–
100.0
8.0
0.0
ACE
ci_short
99.0
26.0
19.0
Appendix
Table 23: Comparison between TCE and ACE on RULER across stopping thresholds for Qwen3.5-4B and Qwen3.5-9B. We reported retrieval accuracy (%). The three ACE prompts were applied to the same encoded state within each model, task, and threshold. Bold marks the best result within each model, threshold, and task.
LLM
τ
Method
Prompt
NIAH-1
NIAH-2
NIAH-3
Qwen3.5-27B
0.1
TCE
–
85.0
7.0
0.0
ACE
ci_short
100.0
45.0
33.0
ACE
ci_v1
100.0
39.0
31.0
ACE
ci_v2
100.0
40.0
29.0
0.2
TCE
–
80.0
9.0
0.0
ACE
ci_short
99.0
41.0
33.0
Appendix
Table 24: Comparison between TCE and ACE on RULER across stopping thresholds for Qwen3.5-27B and Muse-Glimmer-30B. We reported retrieval accuracy (%). The three ACE prompts were applied to the same encoded state within each model, task, and threshold. Bold marks the best result within each model, threshold, and task.
LLM
Method
τ=5
τ=8
τ=10
Qwen3.5-4B
Gold Standard
20.8
20.8
20.8
TCE
7.0
8.8
8.6
ACE
7.8
9.0
10.0
Qwen3.5-9B
Gold Standard
20.3
20.3
20.3
TCE
7.4
8.2
9.2
ACE
10.8
9.6
10.1
Appendix
Table 25: Comparison between TCE and ACE on LongBench across stopping thresholds. We reported answer F1 ( ×100 ) under sequential per-chunk encoding. Gold Standard refers to full-context inference, which does not depend on τ . Bold marks the better result between TCE and ACE within each model and threshold.
τ
Method
Prompt
NIAH-1
NIAH-2
NIAH-3
0.1
TCE
–
71.0
76.0
75.3
ACE
ci_short
72.3
75.8
75.3
ACE
ci_v1
72.0
73.8
73.3
ACE
ci_v2
71.0
72.5
72.8
0.2
TCE
–
71.5
76.0
75.5
ACE
ci_short
72.8
75.5
75.0
Appendix
Table 26: Comparison between TCE and ACE on general capability retention with Qwen3.5-4B. We reported MMLU-200 accuracy (%) after encoding contexts from each RULER task. Bold marks the best result within each threshold and task.
τ
Method
Prompt
NIAH-1
NIAH-2
NIAH-3
0.1
TCE
–
75.5
77.0
76.5
ACE
ci_short
67.0
74.5
73.5
ACE
ci_v1
62.5
72.5
70.0
ACE
ci_v2
58.8
71.0
67.5
0.2
TCE
–
75.8
77.3
76.8
ACE
ci_short
68.8
74.5
74.0
Appendix
Table 27: Comparison between TCE and ACE on general capability retention with Qwen3.5-9B. We reported MMLU-200 accuracy (%) after encoding contexts from each RULER task. Bold marks the best result within each threshold and task.
τ
Method
Prompt
NIAH-1
NIAH-2
NIAH-3
0.1
TCE
–
79.3
80.0
80.3
ACE
ci_short
82.0
81.8
81.5
ACE
ci_v1
80.8
80.8
80.8
ACE
ci_v2
81.0
80.5
79.5
0.2
TCE
–
79.3
80.0
80.3
ACE
ci_short
82.0
81.8
81.3
Appendix
Table 28: Comparison between TCE and ACE on general capability retention with Qwen3.5-27B. We reported MMLU-200 accuracy (%) after encoding contexts from each RULER task. Bold marks the best result within each threshold and task.
τ
Method
Prompt
NIAH-1
NIAH-2
NIAH-3
0.1
TCE
–
66.3
72.8
69.5
ACE
ci_short
76.5
76.0
76.0
ACE
ci_v1
75.5
75.0
75.3
ACE
ci_v2
76.8
76.8
76.5
0.2
TCE
–
66.3
71.3
64.3
ACE
ci_short
76.5
76.0
76.3
Appendix
Table 29: Comparison between TCE and ACE on general capability retention with Muse-Glimmer-30B. We reported MMLU-200 accuracy (%) after encoding contexts from each RULER task. Bold marks the best result within each threshold and task.
LLM
Correct UID
Decoy UID
Qwen3.5-4B
87.0
1.0
Qwen3.5-9B
84.0
8.0
Qwen3.5-27B
100.0
4.0
Appendix
Table 30: UID ablation for ACE on NIAH-1 in RULER . We reported retrieval accuracy (%) on the Qwen3.5 family. Both conditions use the same encoded state, with only the correct or decoy UID as the addressing cue at inference. Bold marks the better result within each model.
LLM
Gold Standard
TCE
ACE
Qwen3.5-9B
100.0
30.0
80.0
Qwen3.5-27B
100.0
40.0
70.0
Muse-Glimmer-30B
100.0
20.0
30.0
Appendix
Table 31: Comparison between TCE and ACE on RULER NIAH-1 with 64 K contexts. We reported retrieval accuracy (%). Gold Standard denotes full-context inference. Bold marks the better encoding method within each model.
LLM
NIAH-2
NIAH-3
Gold Standard
TCE
ACE
Gold Standard
TCE
ACE
Qwen3.5-4B
100.0
3.3
6.7
100.0
0.0
3.3
Qwen3.5-9B
100.0
6.7
16.7
100.0
0.0
3.3
Qwen3.5-27B
100.0
3.3
20.0
100.0
0.0
13.3
Muse-Glimmer-30B
100.0
10.0
16.7
100.0
3.3
20.0
Appendix
Table 32: Comparison between TCE and ACE on RULER NIAH-2 and NIAH-3 with 4 K contexts. We reported retrieval accuracy (%). Gold Standard denotes full-context inference. Bold marks the better encoding method within each model and task.
Dataset
Method
Encoding
Steps
Throughput
Response
Output
time (s)
per chunk
(tokens/s)
latency (s)
tokens
Sandbox
Full context
–
–
–
0.64
11
TCE
1.10
6.0
15
4.74
123
ACE
1.34
7.0
13
0.88
17
RULER NIAH-1
Full context
–
–
–
0.63
9
TCE
10.08
40.0
82
4.89
128
Appendix
Table 33: Comparison of computational cost between full-context inference, TCE, and ACE on Qwen3.5-4B. We reported encoding time, steps per chunk, encoding throughput, response latency, and output length on one fixed example per benchmark family, with averaging as described in the text. Output lengths are rounded to integers.
Dataset
Method
Encoding
Steps
Throughput
Response
Output
time (s)
per chunk
(tokens/s)
latency (s)
tokens
Sandbox
Full context
–
–
–
0.68
11
TCE
0.98
5.0
17
3.63
90
ACE
1.18
6.0
14
2.85
72
RULER NIAH-1
Full context
–
–
–
0.66
9
TCE
7.31
28.0
113
4.98
128
Appendix
Table 34: Comparison of computational cost between full-context inference, TCE, and ACE on Qwen3.5-9B. We reported encoding time, steps per chunk, encoding throughput, response latency, and output length on one fixed example per benchmark family, with averaging as described in the text. Output lengths are rounded to integers.
Dataset
Method
Encoding
Steps
Throughput
Response
Output
time (s)
per chunk
(tokens/s)
latency (s)
tokens
Sandbox
Full context
–
–
–
1.11
11
TCE
1.92
5.0
9
6.82
87
ACE
2.44
6.0
7
5.95
73
RULER NIAH-1
Full context
–
–
–
1.25
9
TCE
11.97
21.0
69
9.72
128
Appendix
Table 35: Comparison of computational cost between full-context inference, TCE, and ACE on Qwen3.5-27B. We reported encoding time, steps per chunk, encoding throughput, response latency, and output length on one fixed example per benchmark family, with averaging as described in the text. Output lengths are rounded to integers.
Dataset
Method
Encoding
Steps
Throughput
Response
Output
time (s)
per chunk
(tokens/s)
latency (s)
tokens
RULER NIAH-1
Full context
–
–
–
0.57
5
TCE
3.36
10.0
235
0.14
1
ACE
9.56
15.0
82
7.15
128
LongBench
Full context
–
–
–
5.55
64
TCE
5.87
1.7
1009
3.87
64
Appendix
Table 36: Comparison of computational cost between full-context inference, TCE, and ACE on Muse-Glimmer-30B. We reported encoding time, steps per chunk, encoding throughput, response latency, and output length on one fixed example per benchmark family, with averaging as described in the text. Output lengths are rounded to integers.
LLM
Dataset
Encoding
Inference
TCE
ACE
Full context
TCE
ACE
Qwen3.5-4B
Sandbox
1.5
13.7
0.4
1.22
0.83
RULER NIAH-1
336.5
926.4
6.6
1.43
1.94
LongBench
326.1
929.6
44.9
0.75
2.26
LongBench v2
1003.6
2522.3
131.9
2.07
6.05
Qwen3.5-9B
Sandbox
2.4
22.1
0.8
1.79
2.47
Appendix
Table 37: Comparison of computational cost between full-context inference, TCE, and ACE . We reported estimated encoding and inference FLOPs (TFLOPs) on one fixed example per benchmark family. Inference costs were averaged over the queries associated with each example.
LLM
Dataset
Method
KV cache (MiB)
Qwen3.5-4B
Sandbox
Full context
1.4
TCE
0.8
ACE
3.0
RULER NIAH-1
Full context
28.2
TCE
1.6
ACE
3.8
Appendix
Table 38: Comparison of inference KV-cache storage between full-context inference, TCE, and ACE on Qwen3.5-4B and Qwen3.5-9B. We reported average KV-cache storage (MiB) immediately after prompt prefill on one fixed example per benchmark family.
LLM
Dataset
Method
KV cache (MiB)
Qwen3.5-27B
Sandbox
Full context
2.8
TCE
1.7
ACE
6.1
RULER NIAH-1
Full context
56.5
TCE
3.1
ACE
7.6
Appendix
Table 39: Comparison of inference KV-cache storage between full-context inference, TCE, and ACE on Qwen3.5-27B and Muse-Glimmer-30B. We reported average KV-cache storage (MiB) immediately after prompt prefill on one fixed example per benchmark family.
Understanding and reasoning over long contexts has become a key requirement for deploying large language models (LLMs) in realistic applications. Although recent LLMs support increasingly long context windows, they often fail to use relevant evidence that is already present in the input, revealing a gap between context access and effective context utilization. In this work, we propose Recursive Evidence Replay as LLM Harness for Long-Context Reasoning (RECONTEXT), a training-free inference method for improving long-context reasoning. RECONTEXT uses model-internal relevance signals to construct a query-conditioned evidence pool and replays it before final generation while preserving the full original context. This recursive selection process separates evidence organization from answer generation without training, external memory, or context pruning. We also provide a theoretical analysis based on associative memory, which characterizes the context as a memory store, the question as a retrieval cue, attention as cue-trace association, and replay as trace reactivation. Experiments on eight long-context datasets with 128K context length show that RECONTEXT consistently improves evidence utilization across Qwen3-4B, Qwen3-8B, and Llama3-8B, achieving the best average rank on all three backbones. Code is available at https://github.com/Yanjun-Zhao/ReContext.
We propose RecaLLM, a set of reasoning language models post-trained to make effective use of long-context information. In-context retrieval, which identifies relevant evidence from context, and reasoning are deeply intertwined: retrieval supports reasoning, while reasoning often determines what must be retrieved. However, their interaction remains largely underexplored. In preliminary experiments on several open-source LLMs, we observe that in-context retrieval performance substantially degrades even after a short reasoning span, revealing a key bottleneck for test-time scaling that we refer to as lost-in-thought: reasoning steps that improve performance also make subsequent in-context retrieval more challenging. To address this limitation, RecaLLM interleaves reasoning with explicit in-context retrieval, alternating between reasoning and retrieving context information needed to solve intermediate subproblems. We introduce a negligible-overhead constrained decoding mechanism that enables verbatim copying of evidence spans, improving the grounding of subsequent generation. Trained on diverse lexical and semantic retrieval tasks, RecaLLM achieves strong performance on two long-context benchmarks, RULER and HELMET, significantly outperforming baselines. Notably, we observe consistent gains at context windows of up to 128K tokens using training samples of at most 10K tokens, far shorter than those used by existing long-context approaches, highlighting a promising path toward improving long-context performance without expensive long-context training data.
During instruction fine-tuning (IFT), large language models (LLMs) learn to follow instructions by using the provided context to answer a query. While prior work has studied how context characteristics correlate with context usage by the LLM, this analysis has been limited to inference time, leaving open how these relationships are acquired in the first place. Here, we measure how models' sensitivity to such characteristics shifts across successive IFT stages: supervised fine-tuning (SFT), direct preference optimization (DPO), and reinforcement learning with verifiable rewards (RLVR). Experiments across four models and three datasets show that SFT makes models more likely to use contexts that are easy to understand, such as containing high length, context-query similarity, and fluency. Post-SFT dynamics may either reinforce or resolve these preferences depending on the training dataset. Our findings reveal that context usage is actively reshaped at each IFT stage, and designing a balanced IFT dataset is important in ensuring robust context utilization of instruction-tuned models.