Encode Once, Answer Many: Internalizing Context via Parametric Flow

Xinhao Zhong; Mengxuan Wu; Chonghua Liao; Linxi Xie; Ruicheng Feng; Yanpei Guo; Hao Tan; Jiaxin Hong

cs.AIOct 28, 2026

Encode Once, Answer Many: Internalizing Context via Parametric Flow

Authors: Xinhao Zhong, Mengxuan Wu, Chonghua Liao, Linxi Xie, Ruicheng Feng, Yanpei Guo, Hao Tan, Jiaxin Hong,

Organizations: Tencent · UT Austin · NTU · NUS ·

Abstract

Large language models are typically adapted to a task either by placing knowledge in the input context, which keeps it external and re-encodes it on every call, or by fine-tuning, which internalizes it but demands a costly round of optimization per task. We aim to internalize the knowledge carried by a context directly into model parameters, without per-task optimization and without keeping the context in the input at inference time. To this end, we propose CLIK, a context-conditioned parameter generator that maps a given context to LoRA parameters for internalizing knowledge in a single forward pass through a Transformer encoder with cross-attention, requiring no test-time gradient updates. Once the generated LoRA is inserted into a frozen base model, the model answers downstream queries from the query alone, having internalized the context into its weights rather than re-reading it. Because the adapter depends only on the context, it is generated once and reused for every query on the same document, so the cost of reading that document is amortized across queries. On document question answering, CLIK substantially improves over the no-context base model and outperforms existing hypernetwork-based generators, while remaining behind a base model that reads the context directly, and it carries over to multi-turn dialogue without any multi-turn-specific training. Analysis further shows that the context-dependent part of the generated adapter concentrates in a compact and specific substructure.

Figures & tables

Figure 1: Three ways to supply external knowledge to a frozen LLM. Context in Prompt: re-injecting the context as text incurs high token cost. Context in Parameter: fine-tuning each context into a LoRA incurs high computational cost. Context-conditioned Parameter Generator (ours): a generator maps each context to a LoRA in a single forward pass, free of both test-time optimization and context in the input.

Figure 2: Overview of CLIK. A frozen language model encodes the context, and a learned ELMo-style weighting of its all-layer hidden states conditions a trainable generator that produces a context-specific LoRA adapter in a single forward pass. The adapter is injected into the same frozen backbone, which then answers the question from a context-free prompt (the student ). We train the parameter generator by distilling an in-context teacher , the identical frozen model that reads context and question together, into this student. The objective is a temperature-scaled KL divergence between the teacher and student next-token distributions over the shared answer tokens. The backbone is frozen and shared across all paths, and only the generator is trainable. Gradients (dashed red) flow back through the injected adapter to the generator alone.

without Format Instruction
	F1 Score						ROUGE-L
Model	Single-Hop QA			Multi-Hop QA			Single-Hop QA			Multi-Hop QA
	SQuAD	DROP	ROPES	HotpotQA	MuSiQue	2WikiMQA	SQuAD	DROP	ROPES	HotpotQA	MuSiQue	2WikiMQA
Base + Context	22.6	5.5	2.4	10.3	3.6	7.0	22.6	5.5	2.4	10.1	3.6	7.0
Base	2.2	2.6	1.8	2.3	1.1	1.2	2.2	2.6	1.8	2.3	1.0	1.2
Gen Adapter	18.5	2.4	2.9	3.2	2.0	4.5	18.2	2.2	2.9	3.0	2.0	4.5

Table 1: Main results on the single-hop and multi-hop question-answering benchmarks with Qwen3-4B-Instruct. We report F1 and ROUGE-L under two evaluation protocols: without and with an instruction that constrains the output format; the backbone’s chat template is applied in both. Except for Base + Context , which serves as a gold reference with direct access to the context, all methods answer from the query alone after internalizing the context into parameters. The best result in each column is shown in bold and the second best is underlined (Base + Context is excluded as a gold reference).

	SQuAD			DROP			ROPES
Model	Precision	Recall	F1 Score	Precision	Recall	F1 Score	Precision	Recall	F1 Score
with Format Instruction
Base + Context	67.4	89.6	71.1	32.6	59.3	37.1	52.7	90.1	57.1
Doc-to-LoRA	20.2	79.1	29.6	4.6	49.9	7.8	8.0	90.3	12.1
CLIK	39.4	81.7	46.3	11.1	43.9	14.3	45.6	79.2	49.4
without Format Instruction

Table 2: Decomposition of the single-hop results on SQuAD, DROP, and ROPES into Precision, Recall, and F1 (P / R / F1), reported under both evaluation protocols. Relative to Doc-to-LoRA, CLIK attains substantially higher precision, at comparable recall on SQuAD and lower recall on DROP and ROPES with the format instruction, yielding a markedly better precision–recall balance.

Method	# Gen.	# Per-context	GPU-hours	SQuAD	DROP	ROPES	HotpotQA	MuSiQue	2WikiMQA	Avg. F1
D2L	0.89B	21.2M	13.0k	29.6	7.8	12.1	15.3	6.4	10.2	13.6
D2L-L	1.01B	21.2M	13.1k	29.2	8.1	12.4	15.1	6.8	10.5	13.7
SHINE	0.48B	21.2M	13.6k	35.2	10.4	34.4	22.2	6.0	18.7	21.2
SHINE-L	1.00B	21.2M	23.7k	39.8	12.7	40.5	25.1	9.8	23.9	25.3
CLIK-S	0.51B	21.2M	23.5k	35.4	11.9	39.9	24.8	8.3	21.7	23.7
CLIK-B	1.01B	1.47M	10.3k	44.2	11.1	46.7	24.0	8.8	23.4	26.4

Table 3: Controlled comparison of generator scale under matched backbone (Qwen3-4B-Instruct), training data (FineWeb-Edu), adapter rank (

16

), target modules (gate, up, and down projections of all

36

layers), and number of training steps. Each method keeps its training objective. D2L and SHINE are Doc-to-LoRA and SHINE at native size, the models of Table 1 ; D2L-L and SHINE-L enlarge the generator until it matches the parameter count of CLIK. # Gen. is the number of generator parameters and # Per-context the number of generated parameters per context. We report training GPU-hours ( Appendix E ) and F1 with the format instruction.

Training data	SQuAD	DROP	ROPES	Avg.
FineWeb-Edu	46.3	14.3	49.4	36.7
FineWeb-Edu + SQuAD, DROP, ROPES	47.0	16.3	49.5	37.6

Table 4: Effect of adding the training splits of SQuAD, DROP, and ROPES to the FineWeb-Edu training data of CLIK. F1 with the format instruction; all other settings are unchanged.

	Paired counterfactual			Novel facts	Prior conflict		Wrong-context swap		Unrelated queries
Method	Own $\uparrow$	Both $\uparrow$	Leak $\downarrow$	Acc $\uparrow$	Ctx $\uparrow$	Prior $\downarrow$	Acc	Leak $\downarrow$	Retention $\uparrow$
Base	0.4	0.0	0.4	1.3	0.0	100.0	18.4	0.0	100.0
Base + Context	93.8	88.6	0.2	91.4	84.7	11.9	17.1	6.3	97.8
Doc-to-LoRA	43.2	22.8	0.6	44.9	37.9	55.4	16.5	5.1	95.6
SHINE	53.2	34.9	0.8	54.6	39.0	43.0	17.4	5.5	95.9
CLIK	72.5	56.3	0.5	67.4	56.1	32.4	17.5	5.4	96.4

Table 5: Counterfactual probes of context following (accuracy in %, with the format instruction). For Leak and Prior , lower is better; for the wrong-context swap, the ideal accuracy equals that of Base.

Figure 3: Multi-turn evaluation of CLIK on the SQuAD, ROPES, and DROP datasets, comparing the single-turn setting against two multi-turn scenarios that differ in how conversational state is propagated. Top (Dynamic KV cache) : the base model accumulates a KV cache across turns while the LoRA generated from the initial context is kept fixed. Bottom (Dynamic LoRA) : the KV cache is discarded and CLIK regenerates the LoRA parameters at each turn from the updated context. Both scenarios improve over the single-turn baseline without any multi-turn-specific training, but neither has a per-turn cost that is independent of the history ( Table 7 ).

Scenario	Generator input	SQuAD	ROPES	DROP
Single turn (no history)	$0$ for $t>1$	46.3	49.4	14.3
Dynamic LoRA	$\|c_{0}\|+\sum_{i<t}\|x_{i}\|$	47.9	62.1	16.1
Recurrent update	$\|x_{t-1}\|$	25.2	31.1	7.4

Table 6: Exploratory recurrent update ( Equation 7 ) compared with the multi-turn schemes above. “Generator input” is the number of context tokens read by the encoder and generator at turn

t

. F1 is averaged over turns

1

–

5

with the format instruction.

Scenario	Encoder / generator input at turn $t$	Backbone prompt at turn $t$	Latency at $t{=}5$ (ms)	SQuAD	ROPES	DROP
Single turn (no history)	$0$ for $t>1$	$\|q_{t}\|$	109	46.3	49.4	14.3
Dynamic KV cache	$0$ for $t>1$	$\|q_{t}\|+\sum_{i<t}\|x_{i}\|$	109	56.9	49.6	17.0
Dynamic LoRA	$\|c_{0}\|+\sum_{i<t}\|x_{i}\|$	$\|q_{t}\|$	564	47.9	62.1	16.1

Table 7: Per-turn cost and quality of multi-turn scenarios.

|c_{0}|

is the length of the initial context,

|x_{i}|=|q_{i}|+|a_{i}|

the length of turn

i

. “Encoder / generator input” and “Backbone prompt” give the number of tokens processed at turn

t

. Latency is the end-to-end time of turn

5

for

|c_{0}|=2048

( Appendix E ); F1 is averaged over turns

1

–

5

with the format instruction.

Per-context state	Parameters	Size (MiB)	KV tokens
Full KV cache, $n=512$	37.7M	72.0	512
Full KV cache, $n=2048$	151.0M	288.0	2048
Compressed KV cache, $256$ -token budget	18.9M	36.0	256
Rank- $16$ FFN LoRA (Doc-to-LoRA, SHINE, CLIK)	21.2M	40.5	288
CLIK-B ( $\mathbf{B}^{\mathrm{down}}$ only)	1.47M	2.8	20

Table 8: Per-context state in BF16 for Qwen3-4B, where

n

is the number of context tokens. “KV tokens” expresses each size as the number of tokens whose full KV cache occupies the same memory (

144

KiB per token).

Method	Inter. (ms)	PCS. (MiB)	Switch (ms)	TTFT (ms)	TP. (tok/s)	Memory (GB)	NQ (ms)	Avg. F1
Base + Context	–	–	–	185.2	5.2k	19.2	185.2	50.4
Base + Context (prefix)	182.1	288.0	6.1	3.7	5.2k	19.2	3.7	50.4
SnapKV ( $256$ -token)	182.6	36.0	0.8	3.3	9.0k	10.9	185.9	39.4
LLMLingua-2 ( $4\times$ )	25.1	$<$ 0.01	$<$ 0.1	46.1	8.1k	13.1	46.1	32.9
Doc-to-LoRA	282.2	40.5	0.9	3.5	8.3k	15.9	3.5	13.6
SHINE	258.7	40.5	0.9	3.5	8.3k	12.1	3.5	21.2

Table 9: Inference cost for

2048

-token contexts ( Appendix E ). New question is the added latency of one further question on the same document, taken from the other columns: the TTFT when the internalized state is reused, and the internalization time plus the TTFT for SnapKV, which selects its cache again. Avg. F1 is over the six datasets with the format instruction.

Hidden state				Learning rate				Training dataset
	P	R	F1		P	R	F1		P	R	F1
Last layer	12.5	78.0	14.8	$1\mathrm{e}{-}6$	18.8	72.7	30.4	SQuAD	21.5	20.5	20.9
Middle layer	39.0	78.2	45.2	$1\mathrm{e}{-}5$	28.5	78.8	41.2	MS MARCO	32.8	58.6	39.5
Mean	36.0	79.5	44.3	$1\mathrm{e}{-}4$	38.5	81.9	45.2	TransMLA	32.6	70.2	44.8
ELMo	39.4	81.7	46.3	$5\mathrm{e}{-}5$	39.4	81.7	46.3	FineWeb-Edu	39.4	81.7	46.3

Table 10: Ablation studies on SQuAD, reported as Precision / Recall / F1 (P / R / F1). We vary three factors independently: the hidden state read by the cross-attention, the peak learning rate, and the training corpus. The configuration adopted in our final model is highlighted in bold .

		SQuAD			HotpotQA	All
Variant	Gen. params	P	R	F1	F1	Avg. F1
CLIK (full)	1.01B	39.4	81.7	46.3	28.7	30.0
w/o across-layer attention	1.01B	36.3	79.4	44.2	26.8	27.6
w/o within-layer attention	1.01B	31.1	72.8	38.7	20.2	19.8
w/o both (independent fragments)	1.01B	22.4	59.7	32.6	10.8	11.4

Table 11: Ablation of the weight-token interaction in the generator. All variants are trained under the same data and number of steps and are parameter-matched to within

5\%

. We report P / R / F1 on SQuAD, F1 on HotpotQA, and the average F1 over the six datasets, all with the format instruction.

Figure 4: Per-layer mean cross-context

\cos(\Delta W)

for the gate, up, and down projections, comparing an early (10k iters) and a late (256k iters) checkpoint of CLIK. Higher values denote updates that are more consistent across different contexts.

Panel of Figure 5 — Figure 5: Distributional view of the generated update $\Delta W=(\alpha/r)BA$ , comparing the early and late checkpoints. The cross-context cosine $\cos(\Delta W)$ measures how consistent the update is across contexts: high values indicate a shared, context-independent component, while low values indicate context-specific adaptation. Because it is computed on the product $BA$ , it is invariant to the reparameterization $(B,A)\mapsto(BR,R^{-1}A)$ .

Swap across contexts
Intervention	Precision	Recall	F1 Score
Baseline	39.4	81.7	46.3
gate & up	33.5	76.2	40.5
$\mathrm{down}_{A}$	38.1	81.2	44.9
$\mathrm{down}_{B}$	6.2	21.2	8.1
full LoRA	5.4	19.7	7.2

Table 12: Intervention study on SQuAD (Precision / Recall / F1). Each generated adapter is perturbed before inference.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone (Qwen3-4B-Instruct-2507)
Component	Value
Hidden size $d$	$2560$
Number of layers $L$	$36$
FFN intermediate size	$9728$
Generated adapter (LoRA)
Rank $r$	$16$

Appendix

Table 13: Parameter generator configuration for the Qwen3-4B backbone.

	Doc-to-LoRA	SHINE	CLIK
Context encoder	frozen backbone	backbone + trainable LoRA (rank $128$ )	frozen backbone
Context representation	per-layer activations	$231$ memory states per layer	ELMo mixture, full sequence
Generator	Perceiver per layer	alternating across-layer /	factorized attention
	( $8$ latents, $9$ blocks, width $512$ )	within-layer Transformer	+ cross-attention
Generator blocks / width	$9$ / $512$	$4$ / $2560$	$12$ / $2048$
Interaction across layers	none	across memory states	across weight tokens

Appendix

Table 14: Architecture-specific settings of the re-implemented baselines.

Knowledge Augmentation for Language Models

Knowledge augmentation for language models adds external or newly learned information to improve factuality, reasoning, and domain performance.

See all 180 papers

Internalizer: Portable Context-to-Parameter Mapping for Very Large Language Models
Oct 8, 2026Peter Devine, Nick Ryan, Benjamin Sirb +1Hypernetworks Parameter-Efficient Fine-Tuning
Interval
Cache2LoRA: Understanding and Improving Context Internalization
Oct 16, 2026Yi Fang, Gaurav Srivastava, Zhenyu Bi +2LLM Context Compression Knowledge Augmentation for Language Models
Department of Computer Science, Virginia Tech, Blacksburg, VA, USA · Department of Biomedical Data Science, Stanford University, Stanford, CA, USA
SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass
Feb 6, 2026Yewei Liu, Xiyuan Wang, Yansheng Mao +3Knowledge Augmentation for Language Models Low-Rank Adaptation
Institute for Artificial Intelligence, Peking University · School of Electronics Engineering and Computer Science, Peking University · University of Oxford +1

Oct 8, 2026J/K move · Enter open · S save

Internalizer: Portable Context-to-Parameter Mapping for Very Large Language Models

Peter Devine, Nick Ryan, Benjamin Sirb, Alex Chiocchi

Interval

Hypernetworks Parameter-Efficient Fine-Tuning Memory-Augmented Language Models

Open paper AskPDF arXiv

Hypernetworks that map a context directly to a LoRA adapter let a large language model carry that context in its weights, but prior work has demonstrated them only on base models of up to 14 billion parameters. We present the Internalizer, a state-of-the-art, portable Context-to-Parameter Mapping hypernetwork that generates document-specific LoRA adapters for the frozen 284B-parameter DeepSeek v4 Flash, a target two orders of magnitude larger than in any previous work. Most of its parameters live in a model-agnostic trunk with only thin entry and exit layers per base model, so it trains cheaply against small models before being ported to the large one. On unseen documents of up to 4096 tokens, the generated adapters reach 84.9% top-1 and 97.8% top-5 teacher-forced accuracy against 63.4% and 83.5% for the base model, with nothing in the context window but a three-word instruction. Once the hypernetwork is trained, a single forward pass turns any document into an adapter for such a model, which could be served alone for speed or alongside the document in the window to raise accuracy further.

Abstract

Knowledge Augmentation for Language Models

Explore similar work

Internalizer: Portable Context-to-Parameter Mapping for Very Large Language Models

Cache2LoRA: Understanding and Improving Context Internalization

SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass