Large language models are typically adapted to a task either by placing knowledge in the input context, which keeps it external and re-encodes it on every call, or by fine-tuning, which internalizes it but demands a costly round of optimization per task. We aim to internalize the knowledge carried by a context directly into model parameters, without per-task optimization and without keeping the context in the input at inference time. To this end, we propose CLIK, a context-conditioned parameter generator that maps a given context to LoRA parameters for internalizing knowledge in a single forward pass through a Transformer encoder with cross-attention, requiring no test-time gradient updates. Once the generated LoRA is inserted into a frozen base model, the model answers downstream queries from the query alone, having internalized the context into its weights rather than re-reading it. Because the adapter depends only on the context, it is generated once and reused for every query on the same document, so the cost of reading that document is amortized across queries. On document question answering, CLIK substantially improves over the no-context base model and outperforms existing hypernetwork-based generators, while remaining behind a base model that reads the context directly, and it carries over to multi-turn dialogue without any multi-turn-specific training. Analysis further shows that the context-dependent part of the generated adapter concentrates in a compact and specific substructure.
Figures & tables
Figure 1: Three ways to supply external knowledge to a frozen LLM. Context in Prompt: re-injecting the context as text incurs high token cost. Context in Parameter: fine-tuning each context into a LoRA incurs high computational cost. Context-conditioned Parameter Generator (ours): a generator maps each context to a LoRA in a single forward pass, free of both test-time optimization and context in the input.
Figure 2: Overview of CLIK. A frozen language model encodes the context, and a learned ELMo-style weighting of its all-layer hidden states conditions a trainable generator that produces a context-specific LoRA adapter in a single forward pass. The adapter is injected into the same frozen backbone, which then answers the question from a context-free prompt (the student ). We train the parameter generator by distilling an in-context teacher , the identical frozen model that reads context and question together, into this student. The objective is a temperature-scaled KL divergence between the teacher and student next-token distributions over the shared answer tokens. The backbone is frozen and shared across all paths, and only the generator is trainable. Gradients (dashed red) flow back through the injected adapter to the generator alone.
F1 Score
ROUGE-L
Model
Single-Hop QA
Multi-Hop QA
Single-Hop QA
Multi-Hop QA
SQuAD
DROP
ROPES
HotpotQA
MuSiQue
2WikiMQA
SQuAD
DROP
ROPES
HotpotQA
MuSiQue
2WikiMQA
without Format Instruction
Base + Context
22.6
5.5
2.4
10.3
3.6
7.0
22.6
5.5
2.4
10.1
3.6
7.0
Base
2.2
2.6
1.8
2.3
1.1
1.2
2.2
2.6
1.8
2.3
1.0
1.2
Gen Adapter
18.5
2.4
2.9
3.2
2.0
4.5
18.2
2.2
2.9
3.0
2.0
4.5
Table 1: Main results on the single-hop and multi-hop question-answering benchmarks with Qwen3-4B-Instruct. We report F1 and ROUGE-L under two evaluation protocols: without and with an instruction that constrains the output format; the backbone’s chat template is applied in both. Except for Base + Context , which serves as a gold reference with direct access to the context, all methods answer from the query alone after internalizing the context into parameters. The best result in each column is shown in bold and the second best is underlined (Base + Context is excluded as a gold reference).
SQuAD
DROP
ROPES
Model
Precision
Recall
F1 Score
Precision
Recall
F1 Score
Precision
Recall
F1 Score
with Format Instruction
Base + Context
67.4
89.6
71.1
32.6
59.3
37.1
52.7
90.1
57.1
Doc-to-LoRA
20.2
79.1
29.6
4.6
49.9
7.8
8.0
90.3
12.1
CLIK
39.4
81.7
46.3
11.1
43.9
14.3
45.6
79.2
49.4
without Format Instruction
Table 2: Decomposition of the single-hop results on SQuAD, DROP, and ROPES into Precision, Recall, and F1 (P / R / F1), reported under both evaluation protocols. Relative to Doc-to-LoRA, CLIK attains substantially higher precision, at comparable recall on SQuAD and lower recall on DROP and ROPES with the format instruction, yielding a markedly better precision–recall balance.
Method
# Gen.
# Per-context
GPU-hours
SQuAD
DROP
ROPES
HotpotQA
MuSiQue
2WikiMQA
Avg. F1
D2L
0.89B
21.2M
13.0k
29.6
7.8
12.1
15.3
6.4
10.2
13.6
D2L-L
1.01B
21.2M
13.1k
29.2
8.1
12.4
15.1
6.8
10.5
13.7
SHINE
0.48B
21.2M
13.6k
35.2
10.4
34.4
22.2
6.0
18.7
21.2
SHINE-L
1.00B
21.2M
23.7k
39.8
12.7
40.5
25.1
9.8
23.9
25.3
CLIK-S
0.51B
21.2M
23.5k
35.4
11.9
39.9
24.8
8.3
21.7
23.7
CLIK-B
1.01B
1.47M
10.3k
44.2
11.1
46.7
24.0
8.8
23.4
26.4
Table 3: Controlled comparison of generator scale under matched backbone (Qwen3-4B-Instruct), training data (FineWeb-Edu), adapter rank ( 16 ), target modules (gate, up, and down projections of all 36 layers), and number of training steps. Each method keeps its training objective. D2L and SHINE are Doc-to-LoRA and SHINE at native size, the models of Table 1 ; D2L-L and SHINE-L enlarge the generator until it matches the parameter count of CLIK. # Gen. is the number of generator parameters and # Per-context the number of generated parameters per context. We report training GPU-hours ( Appendix E ) and F1 with the format instruction.
Training data
SQuAD
DROP
ROPES
Avg.
FineWeb-Edu
46.3
14.3
49.4
36.7
FineWeb-Edu + SQuAD, DROP, ROPES
47.0
16.3
49.5
37.6
Table 4: Effect of adding the training splits of SQuAD, DROP, and ROPES to the FineWeb-Edu training data of CLIK. F1 with the format instruction; all other settings are unchanged.
Paired counterfactual
Novel facts
Prior conflict
Wrong-context swap
Unrelated queries
Method
Own ↑
Both ↑
Leak ↓
Acc ↑
Ctx ↑
Prior ↓
Acc
Leak ↓
Retention ↑
Base
0.4
0.0
0.4
1.3
0.0
100.0
18.4
0.0
100.0
Base + Context
93.8
88.6
0.2
91.4
84.7
11.9
17.1
6.3
97.8
Doc-to-LoRA
43.2
22.8
0.6
44.9
37.9
55.4
16.5
5.1
95.6
SHINE
53.2
34.9
0.8
54.6
39.0
43.0
17.4
5.5
95.9
CLIK
72.5
56.3
0.5
67.4
56.1
32.4
17.5
5.4
96.4
Table 5: Counterfactual probes of context following (accuracy in %, with the format instruction). For Leak and Prior , lower is better; for the wrong-context swap, the ideal accuracy equals that of Base.
Figure 3: Multi-turn evaluation of CLIK on the SQuAD, ROPES, and DROP datasets, comparing the single-turn setting against two multi-turn scenarios that differ in how conversational state is propagated. Top (Dynamic KV cache) : the base model accumulates a KV cache across turns while the LoRA generated from the initial context is kept fixed. Bottom (Dynamic LoRA) : the KV cache is discarded and CLIK regenerates the LoRA parameters at each turn from the updated context. Both scenarios improve over the single-turn baseline without any multi-turn-specific training, but neither has a per-turn cost that is independent of the history ( Table 7 ).
Scenario
Generator input
SQuAD
ROPES
DROP
Single turn (no history)
0 for t>1
46.3
49.4
14.3
Dynamic LoRA
∣c0∣+∑i<t∣xi∣
47.9
62.1
16.1
Recurrent update
∣xt−1∣
25.2
31.1
7.4
Table 6: Exploratory recurrent update ( Equation 7 ) compared with the multi-turn schemes above. “Generator input” is the number of context tokens read by the encoder and generator at turn t . F1 is averaged over turns 1 – 5 with the format instruction.
Scenario
Encoder / generator input at turn t
Backbone prompt at turn t
Latency at t=5 (ms)
SQuAD
ROPES
DROP
Single turn (no history)
0 for t>1
∣qt∣
109
46.3
49.4
14.3
Dynamic KV cache
0 for t>1
∣qt∣+∑i<t∣xi∣
109
56.9
49.6
17.0
Dynamic LoRA
∣c0∣+∑i<t∣xi∣
∣qt∣
564
47.9
62.1
16.1
Table 7: Per-turn cost and quality of multi-turn scenarios. ∣c0∣ is the length of the initial context, ∣xi∣=∣qi∣+∣ai∣ the length of turn i . “Encoder / generator input” and “Backbone prompt” give the number of tokens processed at turn t . Latency is the end-to-end time of turn 5 for ∣c0∣=2048 ( Appendix E ); F1 is averaged over turns 1 – 5 with the format instruction.
Per-context state
Parameters
Size (MiB)
KV tokens
Full KV cache, n=512
37.7M
72.0
512
Full KV cache, n=2048
151.0M
288.0
2048
Compressed KV cache, 256 -token budget
18.9M
36.0
256
Rank- 16 FFN LoRA (Doc-to-LoRA, SHINE, CLIK)
21.2M
40.5
288
CLIK-B ( Bdown only)
1.47M
2.8
20
Table 8: Per-context state in BF16 for Qwen3-4B, where n is the number of context tokens. “KV tokens” expresses each size as the number of tokens whose full KV cache occupies the same memory ( 144 KiB per token).
Method
Inter. (ms)
PCS. (MiB)
Switch (ms)
TTFT (ms)
TP. (tok/s)
Memory (GB)
NQ (ms)
Avg. F1
Base + Context
–
–
–
185.2
5.2k
19.2
185.2
50.4
Base + Context (prefix)
182.1
288.0
6.1
3.7
5.2k
19.2
3.7
50.4
SnapKV ( 256 -token)
182.6
36.0
0.8
3.3
9.0k
10.9
185.9
39.4
LLMLingua-2 ( 4× )
25.1
< 0.01
< 0.1
46.1
8.1k
13.1
46.1
32.9
Doc-to-LoRA
282.2
40.5
0.9
3.5
8.3k
15.9
3.5
13.6
SHINE
258.7
40.5
0.9
3.5
8.3k
12.1
3.5
21.2
Table 9: Inference cost for 2048 -token contexts ( Appendix E ). New question is the added latency of one further question on the same document, taken from the other columns: the TTFT when the internalized state is reused, and the internalization time plus the TTFT for SnapKV, which selects its cache again. Avg. F1 is over the six datasets with the format instruction.
Hidden state
Learning rate
Training dataset
P
R
F1
P
R
F1
P
R
F1
Last layer
12.5
78.0
14.8
1e−6
18.8
72.7
30.4
SQuAD
21.5
20.5
20.9
Middle layer
39.0
78.2
45.2
1e−5
28.5
78.8
41.2
MS MARCO
32.8
58.6
39.5
Mean
36.0
79.5
44.3
1e−4
38.5
81.9
45.2
TransMLA
32.6
70.2
44.8
ELMo
39.4
81.7
46.3
5e−5
39.4
81.7
46.3
FineWeb-Edu
39.4
81.7
46.3
Table 10: Ablation studies on SQuAD, reported as Precision / Recall / F1 (P / R / F1). We vary three factors independently: the hidden state read by the cross-attention, the peak learning rate, and the training corpus. The configuration adopted in our final model is highlighted in bold .
SQuAD
HotpotQA
All
Variant
Gen. params
P
R
F1
F1
Avg. F1
CLIK (full)
1.01B
39.4
81.7
46.3
28.7
30.0
w/o across-layer attention
1.01B
36.3
79.4
44.2
26.8
27.6
w/o within-layer attention
1.01B
31.1
72.8
38.7
20.2
19.8
w/o both (independent fragments)
1.01B
22.4
59.7
32.6
10.8
11.4
Table 11: Ablation of the weight-token interaction in the generator. All variants are trained under the same data and number of steps and are parameter-matched to within 5% . We report P / R / F1 on SQuAD, F1 on HotpotQA, and the average F1 over the six datasets, all with the format instruction.
Figure 4: Per-layer mean cross-context cos(ΔW) for the gate, up, and down projections, comparing an early (10k iters) and a late (256k iters) checkpoint of CLIK. Higher values denote updates that are more consistent across different contexts.
Figure 5: Distributional view of the generated update ΔW=(α/r)BA , comparing the early and late checkpoints. The cross-context cosine cos(ΔW) measures how consistent the update is across contexts: high values indicate a shared, context-independent component, while low values indicate context-specific adaptation. Because it is computed on the product BA , it is invariant to the reparameterization (B,A)↦(BR,R−1A) .
Intervention
Precision
Recall
F1 Score
Baseline
39.4
81.7
46.3
Swap across contexts
gate & up
33.5
76.2
40.5
downA
38.1
81.2
44.9
downB
6.2
21.2
8.1
full LoRA
5.4
19.7
7.2
Table 12: Intervention study on SQuAD (Precision / Recall / F1). Each generated adapter is perturbed before inference.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Value
Backbone (Qwen3-4B-Instruct-2507)
Hidden size d
2560
Number of layers L
36
FFN intermediate size
9728
Generated adapter (LoRA)
Rank r
16
Appendix
Table 13: Parameter generator configuration for the Qwen3-4B backbone.
Doc-to-LoRA
SHINE
CLIK
Context encoder
frozen backbone
backbone + trainable LoRA (rank 128 )
frozen backbone
Context representation
per-layer activations
231 memory states per layer
ELMo mixture, full sequence
Generator
Perceiver per layer
alternating across-layer /
factorized attention
( 8 latents, 9 blocks, width 512 )
within-layer Transformer
+ cross-attention
Generator blocks / width
9 / 512
4 / 2560
12 / 2048
Interaction across layers
none
across memory states
across weight tokens
Appendix
Table 14: Architecture-specific settings of the re-implemented baselines.
Hypernetworks that map a context directly to a LoRA adapter let a large language model carry that context in its weights, but prior work has demonstrated them only on base models of up to 14 billion parameters. We present the Internalizer, a state-of-the-art, portable Context-to-Parameter Mapping hypernetwork that generates document-specific LoRA adapters for the frozen 284B-parameter DeepSeek v4 Flash, a target two orders of magnitude larger than in any previous work. Most of its parameters live in a model-agnostic trunk with only thin entry and exit layers per base model, so it trains cheaply against small models before being ported to the large one. On unseen documents of up to 4096 tokens, the generated adapters reach 84.9% top-1 and 97.8% top-5 teacher-forced accuracy against 63.4% and 83.5% for the base model, with nothing in the context window but a three-word instruction. Once the hypernetwork is trained, a single forward pass turns any document into an adapter for such a model, which could be served alone for speed or alongside the document in the window to raise accuracy further.
Context-to-LoRA methods encode documents as low-rank parameter updates, allowing language models to answer without retaining the source text in their decoding context. How reliably do these updates preserve facts and support their use under different instructions? Controlled diagnosis of existing context-to-LoRA methods separates fact discrimination, target retention, and response adaptation. Competing sources influence answers, while target-only construction and memory-state optimization improve recovery at fixed update size. Recovering facts also does not ensure that answers follow response instructions. We introduce Cache2LoRA, which uses each query to aggregate evidence from a reusable frozen document cache before generating LoRA updates. Trainable encoding operates only on the query and memory-token suffix, and multi-style supervision teaches both evidence preservation and answer expression. Across 17 tasks, Cache2LoRA improves on existing context-to-LoRA methods and achieves macro F1 of 0.663, approaching full-context in-context learning at 0.671. For 32K documents and 256-token outputs at batch size 16, its decoding throughput is 5.9× that of full-context ICL. At matched memory size, generated LoRA preserves answer accuracy more reliably than the learned-KV variant under long, diverse intervening text and combines more effectively with retained document KV, yielding higher hybrid F1 and EM.
Yi Fang, Gaurav Srivastava, Zhenyu Bi +2
Department of Computer Science, Virginia Tech, Blacksburg, VA, USA · Department of Biomedical Data Science, Stanford University, Stanford, CA, USA
We propose SHINE (Scalable Hyper In-context NEtwork), a scalable hypernetwork that can map diverse meaningful contexts into high-quality LoRA adapters for large language models (LLMs). By reusing the frozen LLM's own parameters in an in-context hypernetwork design and introducing architectural innovations, SHINE overcomes key limitations of prior hypernetworks and achieves strong expressive power with a relatively small number of parameters. We introduce a pretraining and instruction fine-tuning pipeline, and train our hypernetwork to generate high quality LoRA adapters from diverse meaningful contexts in a single forward pass. It updates LLM parameters without any fine-tuning, and immediately enables complex question answering tasks related to the context without directly accessing the context, effectively transforming in-context knowledge to in-parameter knowledge in one pass. Our work achieves outstanding results on various tasks, greatly saves time, computation and memory costs compared to SFT-based LLM adaptation, and shows great potential for scaling. Our code is available at https://github.com/MuLabPKU/SHINE
Yewei Liu, Xiyuan Wang, Yansheng Mao +3
Institute for Artificial Intelligence, Peking University · School of Electronics Engineering and Computer Science, Peking University · University of Oxford +1