ASCENT: Scale-Complementary External State for Frozen Long-Context Language Models
Organizations: Department of Electrical and Computer Engineering, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, Pennsylvania, 15213, USA
Abstract
External state should become more valuable as a reader becomes more capable, yet long-context evaluations measure memory utility at a single model scale. This paper formulates scale complementarity: the absolute gain from external state increases when reader capacity and a compact state budget are jointly scaled. ASCENT realizes this principle with a target-blind causal writer, nested typed state, and a bounded query-conditioned readout; every language model remains frozen. Under log loss, the value of a nested refinement equals its target-relevant conditional information; a full factorial makes the reader-by-state interaction identifiable. Official 16K BABILong experiments show Qwen2.5 accuracy gains of 10.50, 78.88, and 82.63 percentage points at 0.5B, 1.5B, and 3B parameters, with both adjacent directional contrasts significant after family-wise correction. At 7B, Foundation accuracy rises to 33.25% while four-slot ASCENT answers every question, which caps the gain at 66.75 points; the 3B-to-7B accuracy increment is -15.88 points [-20.55, -11.20]. The accuracy-side claim is therefore established from 0.5B to 3B; the 7B conclusion rests on the non-saturated negative-log-likelihood factorial. A three-model by three-state factorial yields co-scaled negative log-likelihood gains of 0.388, 2.099, and 7.406 nats at 1.5B, 3B, and 7B; all interaction and gain contrasts retain positive simultaneous lower bounds. Five public 7B-8B readers obtain 66.75-85.63 point gains, and ASCENT exceeds task-trained retrieval checkpoints at the two larger shared scales. Controls, baselines, systems accounting, matched-hardware reproduction, and a cross-architecture sensitivity analysis accompany the experiments. The results establish scale-complementary external state under the evaluated benchmarks, metrics, and state schema.
Figures & tables
| Family | Document | Recognizer record | State (bound) | Question keys | Budget |
|---|---|---|---|---|---|
| BABILong QA1–QA3 | Book text with interleaved event sentences | 4 patterns over closed person, place, and object sets move, pick-up, drop, transfer (person; place, object, recipient) | Last-write maps person place and object holder, joined to object place; per-object place log (256) | 3 forms person; object; object and place | 1/2/3 fact slots; 4 at 7B–8B |
| BABILong QA7–QA8 | As above | Same records | Per-person inventory log; count of held objects | 2 forms person | 1/2/8 fact slots |
| RULER multi-query retrieval | Essay text with key–value needles | 1 pattern (key, value) | Last-write key value map (256) | Keys named in the question | 40/69/91 replay tokens (Qwen2.5); 37/55/97 (SmolLM2) |
| RULER common-word aggregation | Numbered list of words | 1 pattern list item | Exact counter (512 or 1,024; overflow aborts) | Frequency rank | 3/7/10 words (Qwen2.5); 3/5/10 (SmolLM2) |
| Theoretical claim | Empirical quantity | Quantitative result | Conclusion |
|---|---|---|---|
| Refinement value (Prop. 2 ) | Exact-posterior loss under each nested transition | 13,824 of 13,824 endpoint–transition row checks: each larger state extends the smaller one and adds an observation (true by construction); panel-mean exact-posterior NLL decreases on every transition in every panel. | Design check 1 1 1 Under the nested construction each additional slot extends the retained observation prefix of the same latent registers, so the row checks (prefix nesting and non-redundancy across all 13,824 endpoint–transition rows) hold by construction. They verify the implementation and are not independent empirical evidence. Per row, the exact posterior improves on 10,791 of the 13,824 transitions under the noisy channel ( B ). |
| Co-scaled complementarity (Eq. 3 ) | Diagonal gain and two adjacent increments | .388 2.099 7.406 nats; increments 1.711 [1.540,1.882] and 5.307 [5.053,5.561]. | Established |
| Finite-reader interaction (Prop. 4 ) | Two crossed interaction contrasts | .781 [.539,1.023] and .968 [.671,1.264]; simultaneous 95% CIs (Bonferroni). | Identified |
| Task-level consequence | Official 16K generation accuracy gain | 10.50 78.88 82.63 points; both directional contrasts remain significant after Holm correction. | Established |
| Relative footprint (Prop. 5 ) | External-state bytes and state/model ratio | 589.0 624.6 640.6 bytes while the ratio falls . | Established |
| Split | Reader | Foundation | Ascent | Gain [95% CI] |
|---|---|---|---|---|
| Official 16K | 0.5B | .0563 | .1613 | .1050 [.0680,.1420] |
| 1.5B | .1888 | .9775 | .7888 [.7616,.8159] | |
| 3B | .1613 | .9875 | .8263 [.8097,.8428] | |
| Semantic holdout | 0.5B | .1263 | .1613 | .0350 [.0065,.0635] |
| 1.5B | .2025 | .9750 | .7725 [.7419,.8031] | |
| 3B | .1763 | .9838 | .8075 [.7668,.8482] |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Reader | |||
|---|---|---|---|
| 1.5B | .388 [.340,.437] | .295 [.247,.342] | .304 [.248,.359] |
| 3B | 1.412 [1.288,1.535] | 2.099 [2.004,2.194] | 2.587 [2.536,2.638] |
| 7B | 3.709 [3.460,3.958] | 5.950 [5.637,6.264] | 7.406 [7.246,7.566] |
| Reader | State | Gain (nats) | Simultaneous 95% CI |
|---|---|---|---|
| 1.5B | 2 | .388 | [.311,.465] |
| 3B | 3 | 2.099 | [1.948,2.251] |
| 7B | 5 | 7.406 | [7.151,7.661] |
| Model state: 1.5B 3B | – | .781 | [.539,1.023] |
| Model state: 3B 7B | – | .968 | [.671,1.264] |
| Study | Contrast | Effect | SE | MDE 80 | Power | ||
| Official 16K | 0.5B 1.5B | 68.38 pts | 1.83 | 37.44 | 5.75 pts | 1.00 | |
| Official 16K | 1.5B 3B | 3.75 pts | 1.67 | 2.25 | 0.0255 | 5.25 pts | 0.52 |
| Semantic holdout | 0.5B 1.5B | 73.75 pts | 1.09 | 67.88 | 3.42 pts | 1.00 | |
| Semantic holdout | 1.5B 3B | 3.50 pts | 1.61 | 2.17 | 0.0290 | 5.08 pts | 0.49 |
| Factorial | Diagonal gain: 1.5B/K2 | 0.388 nats | 0.021 | 18.52 | 0.098 nats | 1.00 | |
| Factorial | Diagonal gain: 3B/K3 | 2.099 nats | 0.041 | 50.91 | 0.194 nats | 1.00 |
| Reported quantity | Reported value | Implied SE | Implied | df |
|---|---|---|---|---|
| Official 16K 0.5B 1.5B increment | 68.38 pts, = | 1.83 pts | 37.44 | 9 |
| Official 16K 1.5B 3B increment | 3.75 pts, =0.0255 | 1.67 pts | 2.25 | 9 |
| Official 16K 0.5B endpoint gain | 0.1050 [0.0680, 0.1420] | 0.0164 | 6.41 | 9 |
| Official 16K 1.5B endpoint gain | 0.7888 [0.7616, 0.8159] | 0.0120 | 65.75 | 9 |
| Official 16K 3B endpoint gain | 0.8263 [0.8097, 0.8428] | 0.0073 | 112.81 | 9 |
| Factorial Diagonal: 1.5B to 3B | 1.711 [1.540, 1.882] (Bonf.) | 0.047 | 36.74 | 8 |
| Context | Panels | Gain 135M | Gain 360M | Gain 1.7B | Incr. 135M 360M | Incr. 360M 1.7B |
|---|---|---|---|---|---|---|
| 4K | 3 | .2611 [.1569,.3653] | .5056 [.4535,.5577] | .6750 [.5848,.7652] | .2444 [.1066,.3823] | .1694 [.0653,.2736] |
| 8K | 3 | .2806 [.2079,.3533] | .5111 [.4514,.5709] | .6889 [.5511,.8267] | .2306 [.1989,.2622] | .1778 [.0399,.3156] |
| Study | Reader | Slot 1 | Slot 2 | Slot 3/8 |
|---|---|---|---|---|
| Free generation QA1–QA3 (8K, 3 panels) | 135M | .2417 [.1589,.3245] | .4472 [.3063,.5881] | .4972 [.3832,.6112] |
| 360M | .2528 [.2289,.2767] | .5000 [.4586,.5414] | .4972 [.4494,.5450] | |
| 1.7B | .1583 [ .0568,.3735] | .6389 [.4653,.8125] | .6694 [.5197,.8192] | |
| Certified QA7–QA8 (8K, 5 panels) | 135M | .4750 [.4244,.5256] | .7250 [.6744,.7756] | .9750 [.9244,1.0256] |
| 360M | .4438 [.3932,.4943] | .6938 [.6432,.7443] | .9438 [.8932,.9943] | |
| 1.7B | .4063 [.3514,.4611] | .6563 [.6014,.7111] | .9063 [.8514,.9611] |
| Study | Diag. incr. 1 | Diag. incr. 2 | Interaction 1 | Interaction 2 |
|---|---|---|---|---|
| Free generation | .2583 [.1488,.3679] | .1694 [.0554,.2835] | .0417 [ .0618,.1452] | .0333 [ .0081,.0747] |
| Certified | .2188 [.1462,.2913] | .2125 [.1549,.2701] | .0000 [.0000,.0000] | .0000 [.0000,.0000] |
| Writer | Gain 0.5B | Gain 1.5B | Gain 3B | (1) | (2) | Co-scaling |
|---|---|---|---|---|---|---|
| Sentence window | 1.000 | 0.265 | collapses | |||
| Lexical BM25 chain | 1.000 | 0.004 | collapses | |||
| BGE-M3 hybrid + reranker | 0.987 | 0.023 | collapses |
| Checkpoint | Hugging Face revision | Shards | Bytes |
|---|---|---|---|
| EleutherAI/pythia-1.4b | 9cc5c8c8148a | – | – |
| EleutherAI/pythia-1b | c0fea4b91761 | 1 | 2,090,751,965 |
| EleutherAI/pythia-2.8b | dbe7ae300a54 | – | – |
| EleutherAI/pythia-410m | bba6a464f54b | – | – |
| HuggingFaceTB/SmolLM2-1.7B-Instruct | 31b70e2e869a | 1 | 3,422,777,952 |
| HuggingFaceTB/SmolLM2-135M-Instruct | 12fd25f77366 | 1 | 269,060,552 |
Memory-Augmented Language Models
Memory-augmented language models combine persistent or temporary memory with neural computation to retain, retrieve, and use information beyond the active context.
See all 311 papers