Dynamic Compression in Recurrent Networks

Jyothish Pari; Ryan Bahlous-Boldi; Pulkit Agrawal

cs.LGAug 18, 2026

Dynamic Compression in Recurrent Networks

Authors: Jyothish Pari, Ryan Bahlous-Boldi, Pulkit Agrawal

Abstract

Recurrent models process long contexts efficiently by compressing their history into a fixed-size state, but modern architectures typically do so in a single causal pass over the sequence. Each input must therefore be compressed before the model knows how it will later be used, forcing a limited state to compromise across possible future demands. We introduce dynamic compression, which allows a recurrent model to selectively revisit past tokens and revise its fixed-size state through additional recurrent updates. The model need not preserve every part of the history at uniformly high fidelity in its recurrent state, because lower-fidelity information can be revisited from the retained raw sequence when it becomes relevant. We study this in a controlled setting where the model first learns multiple functions in-context and, later in the same sequence, encounters a series of few-shot tasks that each require it to identify and reuse one of those functions. A single-pass model must preserve every function at sufficient fidelity for any future task, whereas selective re-scanning allows the model to revisit and refine only the function currently needed. We find that dynamic compression substantially reduces the recurrent state required for accurate reuse and scales more favorably as the number of stored functions grows. These results demonstrate a computation--memory tradeoff in which recurrent models can spend more computation revisiting their history to make more effective use of a fixed-size state.

Figures & tables

Figure 1: Validation loss during training, one curve per number of in-context pairs

(x_{i},Ax_{i})

x_{i}\sim\mathcal{N}(0,I)

A\sim\mathcal{N}(0,d^{-1})

d=8

, single basis

(K=1)

; the model predicts

Ax_{\mathrm{qry}}

Figure 2: Validation accuracy of the selection head on the basis identification sub-task in isolation (

K=3

Figure 3: The selective re-scanning forward pass (

K=2

). In the basis phase, both bases are written into state sequentially. In the query phase, each query block first processes few-shot pairs to identify which basis is needed, re-scans that basis to sharpen its representation in state, then produces a prediction

\hat{y}

. Numbers at the lower right of each state denote the order of operations.

Figure 4: Top: Write strength $\beta$ (averaged across heads) across layers and token positions for three repeat-model sequences, each containing $K=3$ bases and one query group. Bottom: A re-scan codebook constructed from the $\beta$ patterns of many such sequences. The codebook compresses these patterns into a small set of representative re-scan choices for the dynamic model to predict.

Figure 5: Write strength $\beta$ of the trained codebook dynamic model on its autoregressive re-scan path, for one example ( $K=3$ ). On the first pass, the model writes the three bases relatively evenly; after selecting a basis, it re-scans that basis and updates it more strongly.

Figure 6: Left: Validation MSE versus recurrent state size for

K=1

single-pass,

K=3

single-pass, and

K=3

oracle dynamic re-scanning. Points show individual seeds and dashed lines connect the median across seeds. Right: Validation MSE versus the number of bases

K

d_{\mathrm{head}}=12

for single-pass and oracle dynamic re-scanning, with power-law fits to the median across seeds.

Method	Validation MSE ( $\times 10^{-3}$ )
Single-pass baseline	$67.06\pm 95.99$
Repeat	$1.38\pm 0.44$
Oracle dynamic	$2.30\pm 2.44$
Codebook dynamic	$4.91\pm 1.53$

Table 1: Best validation MSE (mean

\pm

std over 3 seeds, in units of

10^{-3}

) for the four model families at

d_{\text{head}}=16

50

k training steps, and a single query group. The single-pass baseline mean is inflated by one high-variance seed; its median is

17.4

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Write strength

\beta

for a random sample of repeat-model sequences (

K=3

); the query basis span rule marks the repeated block of the matched basis.

Figure 8: Dynamic-model

\beta

(as in Figure 5 ) for a random sample of sequences.

Explore similar work

Sep 11, 2026cs.LG

Temporal Recurrence Favors Fewer Layers

In streaming tasks, recurrent models can carry latent computation across time, allowing each update to build on representations produced earlier. This raises a basic question: once temporal recurrence provides sequential computation across steps, how much depth is still needed within each step? Prior work has shown that recurrence can make shallow models competitive. We instead study this question as a compute-allocation problem, varying within-step depth, expert width, and the number of parallel experts per layer across several compute budgets. For each budget, we compare the best observed recurrent and non-recurrent allocations and the performance they achieve under approximately matched per-step computation. Across Sokoban and autoregressive FineWeb language modeling, we find that temporal recurrence shifts the best observed compute allocation toward substantially fewer layers, with comparable or better performance.

Ivan Anokhin, Johan Obando-Ceron, Irina Rish +1

Mila – Qu´ebec AI Institute · Universit´e de Montr´eal · Sakana AI

Recurrent Neural Networks

Ask PDF

Oct 8, 2026cs.LG

Dynamics as Code: On Model Compression via Dynamic System

The escalating size of pretrained neural networks has rendered model compression a prerequisite for deployment under stringent memory and compute constraints. With the irrational winding as an example, earlier work introduced a dynamic system (DS) paradigm that reconceptualizes compression as compact weight representation: high-dimensional parameters are encoded by the index of a trajectory produced by a dynamic system, from which the vector is recovered during decompression. This mechanism is fundamentally distinct from pruning, quantization, knowledge distillation, and low-rank decomposition. Along this direction, we prove that under a Diophantine condition, a finite trajectory of

M = O(ε^{-(d+ν)})

states in the irrational winding constitutes an

ε

-net over the

d

-dimensional weight space, thereby linking state resolution, decompression error, and compression ratio in a predictable manner. Furthermore, we propose a generalized DS-based model compression framework by unifying four DS families---space-filling curves (Hilbert, Peano, Morton/Z-order, Snake), chaotic systems (Lorenz), congruential and pseudo-random generators (LCG, PCG), and low-discrepancy sequences (Halton). Also, we introduce the KD-tree and coordinate-template acceleration to scale to large models as well as outlier identification to control the error. Experiments on ResNet-18 and Qwen2.5-1.5B/Qwen1.5-7B validate that DS-based compression achieves competitive compression ratios without post-hoc retraining, with controllable decompression error and flexible state-space design, establishing it as a principled and practical compression approach.

Fan Gao, Wei Su, Juntong Fan +4

School of Information Science and Engineering, Lanzhou University, Lanzhou, Gansu Province, China · Department of Data Science, City University of Hong Kong, Hong Kong, China SAR · Department of Mathematics, City University of Hong Kong, Hong Kong, China SAR +1

Model Compression Neural Network Compression LLM Compression

Ask PDF

Oct 3, 2026cs.LG

What to Preserve in Recursive Computation: A Local Predictive Sufficiency Principle

Recursive computation repeatedly compresses or reuses intermediate states, creating a simple tension: information that must remain useful across longer recursive paths is also exposed to more opportunities for loss before reaching the final prediction. Existing reconstruction or local-prediction objectives provide tractable supervision, but do not ensure that the retained information remains sufficient for subsequent recursive computation. We identify local predictive sufficiency with recursive predictive closure: controlling local predictive deficiencies at individual interfaces controls the resulting discrepancy at the root. We then turn this principle into a tractable training procedure. Starting from a variational characterization, we derive finite predictive tests and an empirical predictive deficiency that measures predictive value retained across compression. Its predictive sensitivities define margin-relaxed half-space constraints on parameter updates, and we project the host optimizer's proposed update onto their intersection only when predictive preservation would otherwise be violated. Across temporal graphs, language memory, vision-language-action control, and recursive self-improvement, the method matches or improves the corresponding host models under matched compression budgets, with larger gains under heavier recursive or memory demands, while better preserving predictive information across successive transformations. Crucially, the same task-agnostic predictive-preservation principle is instantiated across all four settings through host-compatible interventions while keeping the endpoint task, backbone, and evaluation protocol fixed. These results establish predictive preservation at recursive interfaces as a general training principle for recursive compression.

Peilin Wang, Feng Shiyang, Hongfu Gao +4

Shandong University · National University of Singapore · Tsinghua University

Model Compression

Ask PDF