We are the first to give a certified rule for whether unseen data belong in a pretrained model's context or its weights, built on an exact law for their information value. That value, the code length saved on held-out queries, equals in expectation the bits stored about the query, plus a prior repair, minus a decoding loss, for any model, architecture, dataset and task. The law's ceilings certify the choice from one measurement, and an anytime-valid test makes it from held-out queries, with error controlled whenever it stops. Every certificate the ceilings issue on a trained transformer, TabPFN, an occupancy network and a stereo network is correct. The test picks TabPFN over XGBoost and a trained surrogate over an in-context learner on airfoil pressure, calls the better of GPT-2's cache and fine-tune in 61,948 of 64,000 query streams over 32 comparisons, and errs in at most 3.3% of null streams. The law also diagnoses when learning in context helps and when it does not. Deleting part of a context is held-out validation of that part in any model, and adding parts one at a time traces the learning curve in context: one patch that sees a shape feature cuts an occupancy network's code length there a hundredfold, while eight that miss it nearly double it, and one camera that sees the surface beats four in a stereo network. Ranked by this label-free valuation, a kriging learner's three best support cars beat four drawn at random, and on a yes-or-no claim the valuation recovers a published evidence adjudicator, with AUROC 0.91.
Figures & tables
Tool
What it takes
What it guarantees
Tested on, and checked by
With the information terms of the data
Exact law (Thm. 2 )
held-out code lengths
E[Γc]=I(Q;Zc)+KL0−decc , in context and in the weights alike
every term in closed form for a linear-Gaussian model, matched by simulation (App. A.10 )
Certificates (Prop. 6 (ii))
one channel’s code length, the other’s ceiling
the channel beats every memory confined to the window, or of at most C bits
25 of 64 on a trained transformer and 36 of 96 on TabPFN, occupancy and stereo networks, all correct
From the model’s probabilities and reruns alone
Decoding-loss bound (Prop. 6 (iii))
a fine-tune’s advantage, context in the window
no fine-tune adds more than the decoding loss in context
GPT-2: at least 0.35–0.55 bits per token, prior repair the cache leaves unrealised
Anytime-valid test (Thm. 8 )
held-out code lengths of both channels
a wrong call with probability at most α , whenever the test stops
tables, airfoil pressure and GPT-2; at most 3.3% wrong calls on nulls, within 5%; a power curve; the same call in 200 reorderings
Table 1: The results at a glance: what each tool takes, what it guarantees, and how the paper tests and checks it.
Figure 1: Both channels, measured in the same bits. The data can give at most G⋆=I(Q;D)+KL0 bits per query in expectation. Each channel keeps its expected gain and loses the rest in two parts: the storage loss, which its memory drops, and the decoding loss, by which its prediction falls short of the best prediction from its memory (Equation ( 4 )).
Figure 2: Learning curves in context: held-out code length against the number of context items, by order of arrival, with the memory rebuilt from scratch for each set of items. (a) Occupancy network on eight constructed chair pairs: 300 anchor points and up to nine patches of 300 points, one of which observes the feature; code length at the feature sites. (b) MVSNet on five DTU scenes: one to four source views, at the 3,856 pixels that some views observe and others do not, with the views ordered separately at each pixel. (c) Kriging learner on eight DrivAerNet cars: two to six support cars, ranked by the gain IG its own predictions assign each car; log-density code length, which can be negative. Random orders average four orders, or in (b) all subsets of each size; means over cases.
Figure 3: Cache against fine-tuning on a trained transformer with exact information terms: gain on a 16-example query against the number of context examples. Dotted: ceilings I(Q;Dw)+KL0 of the caches with windows 8 and 16; dashed: ceiling I(Q;D)+KL0 of any channel; bars: sizes at which the fine-tune’s gain passes the cache’s ceiling, which certifies fine-tuning. 95% intervals over 64 tasks.
Figure 4: The choice across domains. (a) Airfoil pressure: held-out code length per pressure value against the number of designs that the in-context learner reads or the surrogate trains on, mean over three seeds (thin lines: seeds); the in-context learner codes better with few designs and the surrogate from 128 on. (b) logEt of the test that calls the winner, in each seed’s query order, fixed in advance: TabPFN over XGBoost on phoneme (solid) and spambase (dashed), and the surrogate over the in-context learner on airfoil pressure with 128 designs; dots mark the calls. (c) Share of 10,000 simulated streams called by query t : correct calls for advantages of 0.5, 0.1 and 0.03 bits per query, and wrong calls under the worst of three nulls, against α=0.05 .
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Gain on a held-out query in the linear-Gaussian model at ρ=4 . Lines are the closed forms of Proposition 18 ; dots are simulations at d=200 , 300 repeats each. One attention layer and one gradient step coincide, and five gradient steps nearly reach the ideal cache. The grey dashed line is the ceiling of every channel and the grey dotted line is the ceiling of a rank- d/4 adapter.
Figure 6: Relevant evidence against a matched control of the same form. (a) Occupancy probability at the feature sites of eight constructed chair pairs, with the full context, after deleting 300 matched points from unchanged geometry, and after deleting the 300 points that observe the feature; the last row observes the recessed shape. Points are per-pair means over 1,024 sites and diamonds are means over pairs. (b) Extra loss of claim probability, in percentage points, from deleting a source view that observes the queried surface point rather than one that does not, by DTU scene; each point averages five reference views. Scene 9 was used to develop the protocol, and scenes 4, 48, 11 and 29 were drawn by seed before any of their predictions were seen. Claims whose interval misses the reference (oracle) depth are shown for comparison.
Task
Model and data
Memory built from the context
Deleted context
Output law and claim
Scale
Controlled occupancy
Pretrained ConvONet; constructed chairs
Feature grid from the observed points
300 observed surface points
Bernoulli occupancy; the site is occupied
8 shape pairs; 1,024 feature sites per pair; relevant and matched deletions of equal size
5 scenes, 25 reference views; paired observing and non-observing deletions
Pressure fields
Kriging learner; DrivAerNet
Kriging weights, recomputed for each site
Support car (geometry and pressure)
Gaussian pressure law; pressure within a fixed tolerance
8 query cars, 6 support cars each; 512 area-weighted sites
Language verification
Trace Budget adjudicator; VitaminC
Key–value cache
Evidence passage
Probability of the fixed claim
800 examples; Qwen2.5-7B-Instruct and GPT-4.1-mini
Appendix
Table 2: One intervention, four memories. Counts are examples, objects, scenes or cars, which are the units of replication.
Condition
Mean q
Gain IG (bits)
X
Feature item deleted
0.001410
17.6642
7,473
Matched item deleted
0.983387
0.00111
70
Recessed shape observed
0.002191
17.4116
—
Appendix
Table 3: Feature grid: eight pairs, 8,192 feature-site queries, p=0.9827 with the full feature-present context. Both deletions remove 300 points. The gain is averaged site by site. X counts confident claims (7,473 with the full context) that fall below 0.95.
Figure 7: Donor replacement on 8,492 fixed claims. Bars give the share of claims still at 0.95 when a same-category donor’s observations replace the target’s optional observations, split by whether the donor mesh agrees with the claim; rates are pooled over locations.
Deleted source view
Observing
Non-observing
Full probability p
0.8952
0.8952
Ablated probability q
0.7552
0.8504
Gain IG (bits)
1.6204
0.5452
Claim-level bits J
0.8391
0.1872
Target crossings
1,840
760
Reverse crossings
153
455
Appendix
Table 4: Cost volume: depth claims whose reference depth lies inside the fixed interval, 11,808 paired locations, of which 8,251 reach the target with the full context. Means weight scenes equally; crossing counts are pooled. Each deletion removes one source view.
Deletion
Required
Supplied
Claim
Crossings
Feature item
16.93
17.66
17.66
7,473/7,473
Matched item
0.006
0.001
0.001
70/7,473
Observing view
1.27
1.62
0.84
1,840/8,251
Non-observing view
0.53
0.55
0.19
760/8,251
One support car
0.74
0.23
0.04
4.3% of area
Appendix
Table 5: Bits required and supplied per query. Required: mean site-wise budget b0.95(q) after the deletion (zero when q≥0.95 ). Supplied: mean gain IG . Claim: mean J . Crossings: confident claims that fall below 0.95. Means of site-wise values with the weighting of Tables 3 and 4 ; pressure uses native inference with claims at the native answer, weighting sites by area and cars equally.
Figure 8: Pressure error with the reduced input and with native inference. Lines connect the same query car at the same evaluation sites; the learner and the scorer are identical in both conditions.
Reduced input
Native inference
Geometry points
512
32,000
Chart anchors
128
1,024
Leave-one-out points
512
4,000
Mean relative L2 error
0.5276
0.2992
Mean single-car gain IG (bits)
0.7939
0.2296
Appendix
Table 6: Pressure prediction on eight fixed query cars and evaluation sites. The single-car gain averages the six deletions and eight cars. Reference pressure enters only the error.
Metric
Qwen2.5-7B
GPT-4.1-mini
Sufficiency AUROC (515)
0.9126
0.9143
Full-grounding AUROC (800)
0.8990
0.9297
NEI flagged / 120
114
112
Refutes flagged / 285
261
275
Supports flagged / 395
103
99
Correct paired direction / 390
370
371
Appendix
Table 7: Evidence adjudication on VitaminC. Higher scores indicate less support. Flag counts are the adjudicator’s fixed decisions at target 0.95. The 390 claim groups pair the same claim with supporting and with NEI or refuting evidence.
Figure 9: Sufficiency ROC from the budget gap on 395 supporting and 120 NEI examples. Table 7 adds the full-grounding result with all 285 refuting examples.
Occupancy network, patches
0
1
2
3
4
5
6
7
8
9
Feature patch first
4.99
0.046
0.021
0.019
0.014
0.016
0.014
0.015
0.015
0.015
Random order
4.99
5.13
5.26
4.68
4.31
3.86
3.40
2.78
1.96
0.015
Feature patch last
4.99
5.71
6.39
7.25
8.01
8.25
8.72
8.90
9.06
0.015
Stereo network, source views
1
2
3
4
Observing views first
8.71
8.61
9.73
10.17
All subsets of the size
13.65
10.97
10.14
10.17
Appendix
Table 8: Learning curves in context: mean held-out code length at each number of context items, by order of arrival (Figure 2 ). Occupancy: bits per feature site, eight pairs, count 0 is the 300 anchor points alone. Stereo: bits per depth query, five scenes, 3,856 pixels. Kriging: log-density bits per pressure value, eight cars.
Figure 10: Choosing between the cache and fine-tuning on a trained transformer, with exact information terms. (a) Gain on a 16-example query against the number of context examples. Dotted lines are the ceilings I(Q;Dw)+KL0 of the two caches and the dashed line is the ceiling I(Q;D)+KL0 of any channel; the bars mark the data sizes at which the fine-tune’s gain passes the cache’s ceiling, which certifies fine-tuning. (b) Gain of b -bit adapters at N=256 against the bits they store, with their ceilings min{I(Q;D),CB}+KL0 and the cache with a window of 16; rings mark the adapters against which the cache is certified. Intervals are 95% over 64 tasks.
Window 8
Window 16
Fine-tune
CB
cache
tune
won
cache
tune
won
Full
∞
0
4
4/4/1
0
3
5/3/1
Adapter 2×1
2
6
0
7/0/0
6
0
7/0/0
Adapter 4×1
4
0
0
7/0/0
5
0
7/0/0
Adapter 8×1
8
0
0
6/0/1
0
0
7/0/0
Adapter 8×2
16
0
2
2/4/1
0
0
7/0/0
Appendix
Table 9: Certified and measured winners across data sizes: nine for the full fine-tune ( N from 4 to 1,024) and seven for the adapters ( N from 4 to 256). The columns ‘cache’ and ‘tune’ count the data sizes at which the cache certificate and the fine-tune certificate of Proposition 6 fire; ‘won’ gives the data sizes won by the cache, won by the fine-tune and tied. For N≤8 both windows hold the whole context, so those comparisons appear under both; all 25 distinct certified calls agree with the measured winner.
Comparison
Seed
In context
Trained
Accuracy (%)
Call
Queries
Orders
Phoneme: TabPFN vs XGBoost
0
0.587
0.761
78.2 / 78.3
in context
348
200/200
1
0.605
0.691
76.6 / 78.9
in context
1,390
200/200
2
0.640
0.965
77.9 / 78.2
in context
151
200/200
Spambase: TabPFN vs XGBoost
0
0.379
0.716
91.5 / 85.8
in context
198
200/200
1
0.299
0.501
92.7 / 88.7
in context
247
200/200
2
0.318
0.454
91.7 / 88.1
in context
358
200/200
Appendix
Table 10: The anytime-valid choice across domains, run by run. Bits are the mean held-out code length per label (tables) or per pressure value (airfoil) without mixing; accuracy is for the tables only. Call and queries are for each seed’s designated query order; the last column counts the 200 random orders of the same cohort that reach the same call.
Designs
In context
Trained
Call in seeds 0, 1, 2
4
4.964
—
—
8
4.718
—
—
16
4.583
4.746
none, context, none
32
4.258
4.292
trained, context, context
64
3.951
3.573
trained, context, trained
128
3.649
2.341
trained, trained, trained
Appendix
Table 11: Airfoil pressure against the number of designs in the support set (in context) or the training set (trained), with the calibration designs and the 254 fresh queries fixed in each seed. Bits are the mean held-out code length per pressure value over three seeds; the last column gives the anytime-valid call in each seed’s designated order (seeds 0, 1, 2). The trained surrogate’s recipe stops early on a validation split, which needs at least 16 designs.
Case
Wrong calls
95% upper
Correct calls by 256 / 1,024 / 4,096
Exact tie
0.00%
0.04%
—
Symmetric null
3.25%
3.62%
—
Skewed null
2.87%
3.22%
—
Null of changing variance
3.31%
3.68%
—
Advantage 0.5
0.00%
0.04%
88.1%
100.0%
100.0%
Advantage 0.1
0.17%
0.27%
3.7%
12.3%
54.9%
Appendix
Table 12: The anytime-valid test on 10,000 simulated streams per case, each of up to 4,096 queries, with the settings of the real comparisons and the bound of the table comparisons, c=log2(2/ε) . Advantage is the mean score difference in bits per query, with independent ±2 -bit noise for the nonzero cases; the nulls have mean zero. Wrong calls are calls for a side that is not better, with the upper end of a 95% Wilson interval; correct calls are shares of all streams by the given number of queries.
Figure 11: The measurable tools of Section 5 on GPT-2. (a) Learning curves of the cache and the online fine-tune on Shakespeare passages: code length on held-out passages (circles) and the prequential increments read from the context itself, binned over sizes (diamonds); the dotted line marks the full window. Intervals are ±1.96 standard errors of the per-unit means, pooled over four streams. (b) logEt of the test that calls the winner, median over 2,000 query streams for each context (thin lines) and the median over contexts (thick lines): recall of 512-token passages against a one-epoch fine-tune, where the cache wins, and Shakespeare with 256 and 4 context units, where the fine-tune wins; the dashed line is the threshold log(2/α) . (c) Total divergence from the cache over the four-key workload against the bits a compiled replacement stores, for the replacement’s Bayes decoder and for GPT-2 reading it; the dashed line is the budget of Theorem 9 .
Code length
Task
Context
Recipe
Cache
Fine-tune
Better memory
Calls
Queries
Shakespeare
4 units
online
6.98–7.23
6.59–6.68
fine-tune
8,000/8,000
205–324
Shakespeare
256 units
online
6.80–7.13
5.68–5.70
fine-tune
8,000/8,000
81–103
Recall
512 tokens
1 epoch
0.62–1.92
5.44–6.64
cache
8,000/8,000
25–29
Recall
512 tokens
4 epochs
0.62–1.92
2.91–4.46
cache
8,000/8,000
46–69
Recall
512 tokens
16 epochs
0.62–1.92
0.23–0.36
fine-tune
8,000/8,000
70–300
Appendix
Table 13: The anytime-valid choice on GPT-2. Code lengths are in bits per token on each task’s pool of held-out queries, ranges over the four contexts (streams or passages). ‘Calls’ counts the query streams, 2,000 per context, in which the test called the better memory, and ‘queries’ gives the median number of held-out queries to the call over the streams that called, range over contexts. In the 4,096-token, one-epoch row the fine-tune is better on three passages and the cache on one; there 2,048 streams are still undecided after 1,000 queries and 4 call the losing memory. No other stream calls the losing memory.
Model
Window
Capacity
Checks
Agree
Disagree
TabPFN-v2
8/30
17/48
50
50
0
Occupancy network
2/5
3/6
14
14
0
MVSNet
2/3
4/4
22
22
0
All
12/38
24/58
86
86
0
Appendix
Table 14: Certificates on pretrained models outside language, 64 episodes per model: certificates issued out of those tested, against a window and against a capacity, and checks of the issued certificates against the measured winners of the implemented pairs they cover.
Donor mesh relation
Claims
Still at 0.95
Pooled rate
Equal-object rate
Agrees with fixed claim
5,902
5,305
89.88%
78.34%
Disagrees with fixed claim
2,589
64
2.47%
2.30%
Boundary ambiguous
1
0
0%
0%
Appendix
Table 15: Donor replacement by the donor mesh’s relation to the fixed claim. Equal-object rates differ from pooled rates because objects contribute different numbers of claims.
Scene
Oracle inside
Oracle outside
Boundary
9 (protocol development)
3.3232
0.7735
1.9261
4
3.9743
−0.7445
6.3977
48
13.8833
4.1997
7.1883
11
10.6232
3.7646
4.4055
29
15.7790
2.5745
11.3640
Mean, all five scenes
9.5166
2.1136
6.2563
Appendix
Table 16: Extra claim-probability loss from deleting an observing rather than a non-observing view, in percentage points. Five references contribute equally within each scene.
Inference
Claim bank
p
q
Gain IG
Claim bits J
Crossing
Reverse
Reduced input
Reduced-input answer
0.5761
0.5312
0.7939
0.1108
4.6045%
1.6273%
Reduced input
Native answer
0.4623
0.4806
0.7939
0.1056
1.4881%
2.6273%
Native
Reduced-input answer
0.4629
0.4584
0.2296
0.0382
1.2688%
1.3699%
Native
Native answer
0.6131
0.5895
0.2296
0.0419
4.2862%
2.2225%
Appendix
Table 17: Field scores for both claim banks. Means weight sites by area within a query and weight queries and single-car deletions equally. Crossing fractions are percentages of area-weighted site observations.
Transformer-based large language models exhibit in-context learning, enabling adaptation to downstream tasks via few-shot prompting with demonstrations. In practice, such models are often fine-tuned to improve zero-shot performance on downstream tasks, allowing them to solve tasks without examples and thereby reducing inference costs. However, fine-tuning can degrade in-context learning, limiting the performance of fine-tuned models on tasks not seen during fine-tuning. Using linear attention models, we provide a theoretical analysis that characterizes how fine-tuning objectives modify attention parameters and identifies conditions under which this leads to degraded few-shot performance. We show that fine-tuning all attention parameters can harm in-context learning, whereas restricting updates to the value matrix improves zero-shot performance while preserving in-context learning. We further show that incorporating an auxiliary few-shot loss enhances in-context learning primarily on the target task, at the expense of degraded in-context learning ability on tasks not seen during fine-tuning. We provide empirical evidence from synthetic and real-world datasets consistent with the qualitative predictions of our theory.
Chungpa Lee, Jy-yong Sohn, Kangwook Lee
Yonsei University, Korea · University of Wisconsin–Madison, USA · KRAFTON, Korea +1
This paper investigates context stickiness in in-context learning (ICL), a phenomenon where earlier examples in a prompt interfere with a transformer's ability to adapt to later tasks. Using synthetic regression tasks over linear and quadratic functions, we examine how models trained under sequential, mixed, and random curricula handle abrupt task switches during inference. By sweeping over structured combinations of misleading linear examples followed by recovery quadratic examples, we quantify how prior context biases prediction error and how quickly models realign. Our results show strong evidence of persistent interference: more preceding linear examples reliably degrade quadratic predictions, while additional quadratic examples reduce error but with diminishing returns. We further find that training curricula significantly modulate resilience, with sequential training on the target function class yielding the fastest recovery, and surprisingly, random training producing the least robust behavior.
In-context learning \ -- performing tasks based on examples given in the prompt \ -- is an important capability that has emerged in large language models and has received significant attention in both theory and practice. Existing theoretical work often focuses on settings where the learning uses information purely from the prompt. However, many practical instances of in-context learning require the model to retrieve factual knowledge stored in the model's parameters, with the context serving to identify which knowledge is relevant. In this work, we study how in-context learning leverages factual knowledge recall. We formalize this behavior by introducing the in-context factual recall (IC-recall) task, where a transformer is provided a context of (subject, answer) pairs generated from a hidden relation, along with a query subject, and must both infer this hidden relation and retrieve the corresponding answer. Factual knowledge is modeled by the transformer having access to a simple pre-constructed MLP associative memory storing (subject, relation, answer) triplets. We analyze the supervised fine-tuning dynamics of a one-layer transformer on IC-recall data and prove that the model successfully performs IC-recall by converging to a particular pairwise attention pattern. This fine-tuning stage requires a very small number of samples \ -- only polylogarithmic in the number of stored knowledge triplets. Experiments verify our theoretical predictions and show that the pairwise attention pattern emerges even when the MLP layer is pretrained instead of constructed.
Ruomin Huang, Eshaan Nichani, Jason D. Lee +1
Duke University · Princeton University · UC Berkeley