When Thoughts Become Memory: Unifying In-Context Learning and Fine-Tuning

Leon Chlon; Julie Huang

stat.MLOct 21, 2026

When Thoughts Become Memory: Unifying In-Context Learning and Fine-Tuning

Authors: Leon Chlon, Julie Huang

Organizations: Hassana Labs.

Abstract

We are the first to give a certified rule for whether unseen data belong in a pretrained model's context or its weights, built on an exact law for their information value. That value, the code length saved on held-out queries, equals in expectation the bits stored about the query, plus a prior repair, minus a decoding loss, for any model, architecture, dataset and task. The law's ceilings certify the choice from one measurement, and an anytime-valid test makes it from held-out queries, with error controlled whenever it stops. Every certificate the ceilings issue on a trained transformer, TabPFN, an occupancy network and a stereo network is correct. The test picks TabPFN over XGBoost and a trained surrogate over an in-context learner on airfoil pressure, calls the better of GPT-2's cache and fine-tune in 61,948 of 64,000 query streams over 32 comparisons, and errs in at most 3.3% of null streams. The law also diagnoses when learning in context helps and when it does not. Deleting part of a context is held-out validation of that part in any model, and adding parts one at a time traces the learning curve in context: one patch that sees a shape feature cuts an occupancy network's code length there a hundredfold, while eight that miss it nearly double it, and one camera that sees the surface beats four in a stereo network. Ranked by this label-free valuation, a kriging learner's three best support cars beat four drawn at random, and on a yes-or-no claim the valuation recovers a published evidence adjudicator, with AUROC 0.91.

Figures & tables

Tool	What it takes	What it guarantees	Tested on, and checked by
With the information terms of the data
Exact law (Thm. 2 )	held-out code lengths	$\mathbb{E}[\Gamma_{c}]=I(Q;Z_{c})+\mathrm{KL}_{0}-\mathrm{dec}_{c}$ , in context and in the weights alike	every term in closed form for a linear-Gaussian model, matched by simulation (App. A.10 )
Certificates (Prop. 6 (ii))	one channel’s code length, the other’s ceiling	the channel beats every memory confined to the window, or of at most $C$ bits	25 of 64 on a trained transformer and 36 of 96 on TabPFN, occupancy and stereo networks, all correct
From the model’s probabilities and reruns alone
Decoding-loss bound (Prop. 6 (iii))	a fine-tune’s advantage, context in the window	no fine-tune adds more than the decoding loss in context	GPT-2: at least 0.35–0.55 bits per token, prior repair the cache leaves unrealised
Anytime-valid test (Thm. 8 )	held-out code lengths of both channels	a wrong call with probability at most $\alpha$ , whenever the test stops	tables, airfoil pressure and GPT-2; at most 3.3% wrong calls on nulls, within 5%; a power curve; the same call in 200 reorderings

Table 1: The results at a glance: what each tool takes, what it guarantees, and how the paper tests and checks it.

Panel of Figure 1 — Figure 1: Both channels, measured in the same bits. The data can give at most $G^{\star}=I(Q;D)+\mathrm{KL}_{0}$ bits per query in expectation. Each channel keeps its expected gain and loses the rest in two parts: the storage loss, which its memory drops, and the decoding loss, by which its prediction falls short of the best prediction from its memory (Equation ( 4 )).

Figure 2: Learning curves in context: held-out code length against the number of context items, by order of arrival, with the memory rebuilt from scratch for each set of items. (a) Occupancy network on eight constructed chair pairs: 300 anchor points and up to nine patches of 300 points, one of which observes the feature; code length at the feature sites. (b) MVSNet on five DTU scenes: one to four source views, at the 3,856 pixels that some views observe and others do not, with the views ordered separately at each pixel. (c) Kriging learner on eight DrivAerNet cars: two to six support cars, ranked by the gain

I_{G}

its own predictions assign each car; log-density code length, which can be negative. Random orders average four orders, or in (b) all subsets of each size; means over cases.

Figure 3: Cache against fine-tuning on a trained transformer with exact information terms: gain on a 16-example query against the number of context examples. Dotted: ceilings

I(Q;D_{w})+\mathrm{KL}_{0}

of the caches with windows 8 and 16; dashed: ceiling

I(Q;D)+\mathrm{KL}_{0}

of any channel; bars: sizes at which the fine-tune’s gain passes the cache’s ceiling, which certifies fine-tuning. 95% intervals over 64 tasks.

Figure 4: The choice across domains. (a) Airfoil pressure: held-out code length per pressure value against the number of designs that the in-context learner reads or the surrogate trains on, mean over three seeds (thin lines: seeds); the in-context learner codes better with few designs and the surrogate from 128 on. (b)

\log E_{t}

of the test that calls the winner, in each seed’s query order, fixed in advance: TabPFN over XGBoost on phoneme (solid) and spambase (dashed), and the surrogate over the in-context learner on airfoil pressure with 128 designs; dots mark the calls. (c) Share of 10,000 simulated streams called by query

t

: correct calls for advantages of 0.5, 0.1 and 0.03 bits per query, and wrong calls under the worst of three nulls, against

\alpha=0.05

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Gain on a held-out query in the linear-Gaussian model at

\rho=4

. Lines are the closed forms of Proposition 18 ; dots are simulations at

d=200

, 300 repeats each. One attention layer and one gradient step coincide, and five gradient steps nearly reach the ideal cache. The grey dashed line is the ceiling of every channel and the grey dotted line is the ceiling of a rank-

d/4

adapter.

Panel of Figure 6 — Figure 6: Relevant evidence against a matched control of the same form. (a) Occupancy probability at the feature sites of eight constructed chair pairs, with the full context, after deleting 300 matched points from unchanged geometry, and after deleting the 300 points that observe the feature; the last row observes the recessed shape. Points are per-pair means over 1,024 sites and diamonds are means over pairs. (b) Extra loss of claim probability, in percentage points, from deleting a source view that observes the queried surface point rather than one that does not, by DTU scene; each point averages five reference views. Scene 9 was used to develop the protocol, and scenes 4, 48, 11 and 29 were drawn by seed before any of their predictions were seen. Claims whose interval misses the reference (oracle) depth are shown for comparison.

Task	Model and data	Memory built from the context	Deleted context	Output law and claim	Scale
Controlled occupancy	Pretrained ConvONet; constructed chairs	Feature grid from the observed points	300 observed surface points	Bernoulli occupancy; the site is occupied	8 shape pairs; 1,024 feature sites per pair; relevant and matched deletions of equal size
Donor replacement	Same ConvONet; ShapeNet objects	Feature grid	Optional point-cloud observations	Bernoulli occupancy at fixed world coordinates	16 objects; 8,492 fixed claims; independent donor-mesh labels
Multi-view depth	Pretrained MVSNet; DTU	Cost volume over the source views	Source image and camera	Categorical depth law; depth in a fixed interval	5 scenes, 25 reference views; paired observing and non-observing deletions
Pressure fields	Kriging learner; DrivAerNet	Kriging weights, recomputed for each site	Support car (geometry and pressure)	Gaussian pressure law; pressure within a fixed tolerance	8 query cars, 6 support cars each; 512 area-weighted sites
Language verification	Trace Budget adjudicator; VitaminC	Key–value cache	Evidence passage	Probability of the fixed claim	800 examples; Qwen2.5-7B-Instruct and GPT-4.1-mini

Appendix

Table 2: One intervention, four memories. Counts are examples, objects, scenes or cars, which are the units of replication.

Condition	Mean $q$	Gain $I_{G}$ (bits)	$X$
Feature item deleted	0.001410	17.6642	7,473
Matched item deleted	0.983387	0.00111	70
Recessed shape observed	0.002191	17.4116	—

Appendix

Table 3: Feature grid: eight pairs, 8,192 feature-site queries,

p=0.9827

with the full feature-present context. Both deletions remove 300 points. The gain is averaged site by site.

X

counts confident claims (7,473 with the full context) that fall below 0.95.

Figure 7: Donor replacement on 8,492 fixed claims. Bars give the share of claims still at 0.95 when a same-category donor’s observations replace the target’s optional observations, split by whether the donor mesh agrees with the claim; rates are pooled over locations.

Deleted source view	Observing	Non-observing
Full probability $p$	0.8952	0.8952
Ablated probability $q$	0.7552	0.8504
Gain $I_{G}$ (bits)	1.6204	0.5452
Claim-level bits $J$	0.8391	0.1872
Target crossings	1,840	760
Reverse crossings	153	455

Appendix

Table 4: Cost volume: depth claims whose reference depth lies inside the fixed interval, 11,808 paired locations, of which 8,251 reach the target with the full context. Means weight scenes equally; crossing counts are pooled. Each deletion removes one source view.

Deletion	Required	Supplied	Claim	Crossings
Feature item	16.93	17.66	17.66	7,473/7,473
Matched item	0.006	0.001	0.001	70/7,473
Observing view	1.27	1.62	0.84	1,840/8,251
Non-observing view	0.53	0.55	0.19	760/8,251
One support car	0.74	0.23	0.04	4.3% of area

Appendix

Table 5: Bits required and supplied per query. Required: mean site-wise budget

b_{0.95}(q)

after the deletion (zero when

q\geq 0.95

). Supplied: mean gain

I_{G}

. Claim: mean

J

. Crossings: confident claims that fall below 0.95. Means of site-wise values with the weighting of Tables 3 and 4 ; pressure uses native inference with claims at the native answer, weighting sites by area and cars equally.

Figure 8: Pressure error with the reduced input and with native inference. Lines connect the same query car at the same evaluation sites; the learner and the scorer are identical in both conditions.

	Reduced input	Native inference
Geometry points	512	32,000
Chart anchors	128	1,024
Leave-one-out points	512	4,000
Mean relative $L_{2}$ error	0.5276	0.2992
Mean single-car gain $I_{G}$ (bits)	0.7939	0.2296

Appendix

Table 6: Pressure prediction on eight fixed query cars and evaluation sites. The single-car gain averages the six deletions and eight cars. Reference pressure enters only the error.

Metric	Qwen2.5-7B	GPT-4.1-mini
Sufficiency AUROC (515)	0.9126	0.9143
Full-grounding AUROC (800)	0.8990	0.9297
NEI flagged / 120	114	112
Refutes flagged / 285	261	275
Supports flagged / 395	103	99
Correct paired direction / 390	370	371

Appendix

Table 7: Evidence adjudication on VitaminC. Higher scores indicate less support. Flag counts are the adjudicator’s fixed decisions at target 0.95. The 390 claim groups pair the same claim with supporting and with NEI or refuting evidence.

Figure 9: Sufficiency ROC from the budget gap on 395 supporting and 120 NEI examples. Table 7 adds the full-grounding result with all 285 refuting examples.

Occupancy network, patches	0	1	2	3	4	5	6	7	8	9
Feature patch first	4.99	0.046	0.021	0.019	0.014	0.016	0.014	0.015	0.015	0.015
Random order	4.99	5.13	5.26	4.68	4.31	3.86	3.40	2.78	1.96	0.015
Feature patch last	4.99	5.71	6.39	7.25	8.01	8.25	8.72	8.90	9.06	0.015
Stereo network, source views	1	2	3	4
Observing views first	8.71	8.61	9.73	10.17
All subsets of the size	13.65	10.97	10.14	10.17

Appendix

Table 8: Learning curves in context: mean held-out code length at each number of context items, by order of arrival (Figure 2 ). Occupancy: bits per feature site, eight pairs, count 0 is the 300 anchor points alone. Stereo: bits per depth query, five scenes, 3,856 pixels. Kriging: log-density bits per pressure value, eight cars.

Figure 10: Choosing between the cache and fine-tuning on a trained transformer, with exact information terms. (a) Gain on a 16-example query against the number of context examples. Dotted lines are the ceilings

I(Q;D_{w})+\mathrm{KL}_{0}

of the two caches and the dashed line is the ceiling

I(Q;D)+\mathrm{KL}_{0}

of any channel; the bars mark the data sizes at which the fine-tune’s gain passes the cache’s ceiling, which certifies fine-tuning. (b) Gain of

b

-bit adapters at

N=256

against the bits they store, with their ceilings

\min\{I(Q;D),C_{B}\}+\mathrm{KL}_{0}

and the cache with a window of 16; rings mark the adapters against which the cache is certified. Intervals are 95% over 64 tasks.

Fine-tune	$C_{B}$	cache	tune	won	cache	tune	won
		Window 8			Window 16
Full	$\infty$	0	4	4/4/1	0	3	5/3/1
Adapter $2\times 1$	2	6	0	7/0/0	6	0	7/0/0
Adapter $4\times 1$	4	0	0	7/0/0	5	0	7/0/0
Adapter $8\times 1$	8	0	0	6/0/1	0	0	7/0/0
Adapter $8\times 2$	16	0	2	2/4/1	0	0	7/0/0

Appendix

Table 9: Certified and measured winners across data sizes: nine for the full fine-tune (

N

from 4 to 1,024) and seven for the adapters (

N

from 4 to 256). The columns ‘cache’ and ‘tune’ count the data sizes at which the cache certificate and the fine-tune certificate of Proposition 6 fire; ‘won’ gives the data sizes won by the cache, won by the fine-tune and tied. For

N\leq 8

both windows hold the whole context, so those comparisons appear under both; all 25 distinct certified calls agree with the measured winner.

Comparison	Seed	In context	Trained	Accuracy (%)	Call	Queries	Orders
Phoneme: TabPFN vs XGBoost	0	0.587	0.761	78.2 / 78.3	in context	348	200/200
	1	0.605	0.691	76.6 / 78.9	in context	1,390	200/200
	2	0.640	0.965	77.9 / 78.2	in context	151	200/200
Spambase: TabPFN vs XGBoost	0	0.379	0.716	91.5 / 85.8	in context	198	200/200
	1	0.299	0.501	92.7 / 88.7	in context	247	200/200
	2	0.318	0.454	91.7 / 88.1	in context	358	200/200

Appendix

Table 10: The anytime-valid choice across domains, run by run. Bits are the mean held-out code length per label (tables) or per pressure value (airfoil) without mixing; accuracy is for the tables only. Call and queries are for each seed’s designated query order; the last column counts the 200 random orders of the same cohort that reach the same call.

Designs	In context	Trained	Call in seeds 0, 1, 2
4	4.964	—	—
8	4.718	—	—
16	4.583	4.746	none, context, none
32	4.258	4.292	trained, context, context
64	3.951	3.573	trained, context, trained
128	3.649	2.341	trained, trained, trained

Appendix

Table 11: Airfoil pressure against the number of designs in the support set (in context) or the training set (trained), with the calibration designs and the 254 fresh queries fixed in each seed. Bits are the mean held-out code length per pressure value over three seeds; the last column gives the anytime-valid call in each seed’s designated order (seeds 0, 1, 2). The trained surrogate’s recipe stops early on a validation split, which needs at least 16 designs.

Case	Wrong calls	95% upper	Correct calls by 256 / 1,024 / 4,096
Exact tie	0.00%	0.04%	—
Symmetric null	3.25%	3.62%	—
Skewed null	2.87%	3.22%	—
Null of changing variance	3.31%	3.68%	—
Advantage 0.5	0.00%	0.04%	88.1%	100.0%	100.0%
Advantage 0.1	0.17%	0.27%	3.7%	12.3%	54.9%

Appendix

Table 12: The anytime-valid test on 10,000 simulated streams per case, each of up to 4,096 queries, with the settings of the real comparisons and the bound of the table comparisons,

c=\log_{2}(2/\varepsilon)

. Advantage is the mean score difference in bits per query, with independent

\pm 2

-bit noise for the nonzero cases; the nulls have mean zero. Wrong calls are calls for a side that is not better, with the upper end of a 95% Wilson interval; correct calls are shares of all streams by the given number of queries.

Figure 11: The measurable tools of Section 5 on GPT-2. (a) Learning curves of the cache and the online fine-tune on Shakespeare passages: code length on held-out passages (circles) and the prequential increments read from the context itself, binned over sizes (diamonds); the dotted line marks the full window. Intervals are

\pm 1.96

standard errors of the per-unit means, pooled over four streams. (b)

\log E_{t}

of the test that calls the winner, median over 2,000 query streams for each context (thin lines) and the median over contexts (thick lines): recall of 512-token passages against a one-epoch fine-tune, where the cache wins, and Shakespeare with 256 and 4 context units, where the fine-tune wins; the dashed line is the threshold

\log(2/\alpha)

. (c) Total divergence from the cache over the four-key workload against the bits a compiled replacement stores, for the replacement’s Bayes decoder and for GPT-2 reading it; the dashed line is the budget of Theorem 9 .

Task	Context	Recipe	Cache	Fine-tune	Better memory	Calls	Queries
			Code length
Shakespeare	4 units	online	6.98–7.23	6.59–6.68	fine-tune	8,000/8,000	205–324
Shakespeare	256 units	online	6.80–7.13	5.68–5.70	fine-tune	8,000/8,000	81–103
Recall	512 tokens	1 epoch	0.62–1.92	5.44–6.64	cache	8,000/8,000	25–29
Recall	512 tokens	4 epochs	0.62–1.92	2.91–4.46	cache	8,000/8,000	46–69
Recall	512 tokens	16 epochs	0.62–1.92	0.23–0.36	fine-tune	8,000/8,000	70–300

Appendix

Table 13: The anytime-valid choice on GPT-2. Code lengths are in bits per token on each task’s pool of held-out queries, ranges over the four contexts (streams or passages). ‘Calls’ counts the query streams, 2,000 per context, in which the test called the better memory, and ‘queries’ gives the median number of held-out queries to the call over the streams that called, range over contexts. In the 4,096-token, one-epoch row the fine-tune is better on three passages and the cache on one; there 2,048 streams are still undecided after 1,000 queries and 4 call the losing memory. No other stream calls the losing memory.

Model	Window	Capacity	Checks	Agree
TabPFN-v2	8/30	17/48	50	50
Occupancy network	2/5	3/6	14	14
MVSNet	2/3	4/4	22	22
All	12/38	24/58	86	86

Appendix

Table 14: Certificates on pretrained models outside language, 64 episodes per model: certificates issued out of those tested, against a window and against a capacity, and checks of the issued certificates against the measured winners of the implemented pairs they cover.

Donor mesh relation	Claims	Still at 0.95	Pooled rate	Equal-object rate
Agrees with fixed claim	5,902	5,305	89.88%	78.34%
Disagrees with fixed claim	2,589	64	2.47%	2.30%
Boundary ambiguous	1	0	0%	0%

Appendix

Table 15: Donor replacement by the donor mesh’s relation to the fixed claim. Equal-object rates differ from pooled rates because objects contribute different numbers of claims.

Scene	Oracle inside	Oracle outside	Boundary
9 (protocol development)	3.3232	0.7735	1.9261
4	3.9743	$-0.7445$	6.3977
48	13.8833	4.1997	7.1883
11	10.6232	3.7646	4.4055
29	15.7790	2.5745	11.3640
Mean, all five scenes	9.5166	2.1136	6.2563

Appendix

Table 16: Extra claim-probability loss from deleting an observing rather than a non-observing view, in percentage points. Five references contribute equally within each scene.

Inference	Claim bank	$p$	$q$	Gain $I_{G}$	Claim bits $J$	Crossing	Reverse
Reduced input	Reduced-input answer	0.5761	0.5312	0.7939	0.1108	4.6045%	1.6273%
Reduced input	Native answer	0.4623	0.4806	0.7939	0.1056	1.4881%	2.6273%
Native	Reduced-input answer	0.4629	0.4584	0.2296	0.0382	1.2688%	1.3699%
Native	Native answer	0.6131	0.5895	0.2296	0.0419	4.2862%	2.2225%

Appendix

Table 17: Field scores for both claim banks. Means weight sites by area within a query and weight queries and single-car deletions equally. Crossing fractions are percentages of area-weighted site observations.

In-Context Learning

In-context learning studies how models adapt to new tasks from examples or instructions supplied at inference time, without changing their weights.

See all 402 papers

Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models
Feb 26, 2026Chungpa Lee, Jy-yong Sohn, Kangwook LeeLLM Fine-Tuning In-Context Learning
Yonsei University, Korea · University of Wisconsin–Madison, USA · KRAFTON, Korea +1
When Context Sticks: Studying Interference in In-Context Learning
Apr 25, 2026Hanna Rød, Dagny Streit, Nils Valseth Selte +1In-Context Learning
Fine-Tuning Dynamics of In-Context Factual Recall in Transformers
May 26, 2026Ruomin Huang, Eshaan Nichani, Jason D. Lee +1Supervised Fine-Tuning Transformer Attention
Duke University · Princeton University · UC Berkeley

Feb 26, 2026J/K move · Enter open · S save

Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models

Chungpa Lee, Jy-yong Sohn, Kangwook Lee

Yonsei University, Korea · University of Wisconsin–Madison, USA · KRAFTON, Korea+1

LLM Fine-Tuning In-Context Learning Catastrophic Forgetting

Open paper AskPDF arXiv

Transformer-based large language models exhibit in-context learning, enabling adaptation to downstream tasks via few-shot prompting with demonstrations. In practice, such models are often fine-tuned to improve zero-shot performance on downstream tasks, allowing them to solve tasks without examples and thereby reducing inference costs. However, fine-tuning can degrade in-context learning, limiting the performance of fine-tuned models on tasks not seen during fine-tuning. Using linear attention models, we provide a theoretical analysis that characterizes how fine-tuning objectives modify attention parameters and identifies conditions under which this leads to degraded few-shot performance. We show that fine-tuning all attention parameters can harm in-context learning, whereas restricting updates to the value matrix improves zero-shot performance while preserving in-context learning. We further show that incorporating an auxiliary few-shot loss enhances in-context learning primarily on the target task, at the expense of degraded in-context learning ability on tasks not seen during fine-tuning. We provide empirical evidence from synthetic and real-world datasets consistent with the qualitative predictions of our theory.

Abstract

In-Context Learning

Explore similar work

Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models

When Context Sticks: Studying Interference in In-Context Learning

Fine-Tuning Dynamics of In-Context Factual Recall in Transformers