Information Limits of Replay-Free Recurrent State Migration
Organizations: MIT
Abstract
When a recurrent model is replaced, its old state may not determine what the new model would compute from the same history. We characterize the information that must be retained before the replacement is known. For a time-invariant affine linear family, the minimum continuous causal memory for order- local migration equals the reachable dimension of a lifted coefficient system minus the information in the anchor state. A controllable -state model with scalar input needs at most additional real coordinates, even when arbitrary matrix replacement directions are revealed only after streaming. A single rank-one direction can already require all . Conditional singular values and auxiliary-bit bounds distinguish this local information law from recovery at a fixed tolerance. We also give invariant-subspace and future-output constructions. Controlled experiments compare the resulting representations. In a small language model, equal-memory cubic interpolation improves mean state fidelity over third-order coefficients but provides no meaningful predictive gain. An analytic history basis preserves useful adaptation more economically in a coupled identification task. With repeated slow poles, eight history coordinates meet normalized output error across five fitted seeds despite substantial full-state error. Splitting the poles raises the smallest tested successful width to 64.
Figures & tables
| Question | Result | Essential scope |
|---|---|---|
| Causal local order | Fixed time-invariant affine family, real inputs | |
| Directions revealed later | Sharp worst-case extra coordinates | Controllable scalar input, exact arithmetic |
| Fixed-horizon tolerance | Conditional singular values and a bit lower bound | Unit-energy histories, finite queries, exact old state |
| Implementation cost | Coefficients, interpolation, or history banks | Maintenance, conditioning, and predictive tolerance measured separately |
| Method | Extra bytes | State NRMSE | Predictive KL |
|---|---|---|---|
| Stale state | 0 | ||
| Checkpoint replay, 256 | 320 | ||
| Linear stitching | 0 | ||
| First-order sensitivity | 128 | ||
| Degree-2 memory | 192 | ||
| Degree-3 memory | 256 |
| Method | Extra bytes | Prediction NMSE | Stream ms |
|---|---|---|---|
| No adaptation | 0 | — | |
| Unrestricted fit + replay | — | ||
| Full replay | |||
| Stale state | 0 | ||
| Checkpoint replay, 256 | 2048 | ||
| Linear stitching | 0 |
| NRMSE tolerance | Output fidelity | Full-state fidelity | ||
|---|---|---|---|---|
| Repeated | Split | Repeated | Split | |
| 4 | 16 | 128 | 200 | |
| 8 | 32 | 400 | 400 | |
| 8 | 64 | 800 | 800 | |
Appendix figures & tables31 assets
Supplementary material from the paper’s appendix.
Appendix
| Update coordinates | Degree 1 | Degree 2 | Degree 3 |
| 1 | 1 | 2 | 3 |
| 2 | 2 | 5 | 9 |
| 4 | 4 | 14 | 34 |
| 8 | 8 | 44 | 164 |
| Method | Seed 0 | Seed 1 | Seed 2 | Seed 3 | Seed 4 |
|---|---|---|---|---|---|
| Quadratic interpolation | 0.97179 | 0.46294 | 0.85123 | 0.37411 | 0.08055 |
| Cubic interpolation | 0.09378 | 0.04386 | 0.08466 | 0.01676 | 0.02591 |
| First-order sensitivity | 1.73214 | 1.07336 | 1.91705 | 0.96552 | 1.32557 |
| Full replay | 0.00000 | 0.00000 | 0.00000 | 0.00000 | 0.00000 |
| Linear stitching | 0.30478 | 0.19829 | 0.26668 | 0.22509 | 0.17058 |
| Nonlinear stitching | 0.56802 | 0.32188 | 0.36767 | 0.33579 | 0.26696 |
| Method | bits/byte ( ) | Minimum / maximum seed |
|---|---|---|
| Full replay | ||
| Affected-state reset | ||
| Stale state | ||
| Checkpoint replay, 64 | ||
| Checkpoint replay, 256 | ||
| Linear stitching |
| Method | State-only ms [IQR] | With-head ms [IQR] |
|---|---|---|
| Stale state | ||
| Full replay | ||
| Checkpoint replay, 64 | ||
| Checkpoint replay, 256 | ||
| Linear stitching | ||
| Nonlinear stitching |
| RMS tolerance | Extra coordinates | Bit lower bound |
|---|---|---|
| 0 | 0.00000 | |
| 1 | 1.90465 | |
| 1 | 5.22657 | |
| 3 | 20.78339 | |
| 5 | 49.74601 |
| Method | Extra bytes | State NRMSE | Predictive KL |
|---|---|---|---|
| Full replay | |||
| Affected-state reset | 0 | ||
| Stale state | 0 | ||
| Checkpoint replay, 64 | 128 | ||
| Checkpoint replay, 256 | 320 | ||
| Linear stitching | 0 |
| Seed | Leakage | ||||
|---|---|---|---|---|---|
| 0 | 0.970 | 2, 8 | 0.988001 | ||
| 1 | 0.970 | 2, 8 | 0.988001 | ||
| 2 | 0.970 | 2, 8 | 0.988001 | ||
| 3 | 0.970 | 2, 8 | 0.988001 | ||
| 4 | 0.970 | 2, 8 | 0.988001 | ||
| 0 | 0.995 | 2, 8 | 0.998000 |
| Method | Extra bytes | Prediction NMSE | Stream ms |
|---|---|---|---|
| No adaptation | 0 | — | |
| Unrestricted fit + replay | — | ||
| Full replay | |||
| Affected-state reset | 0 | — | |
| Stale state | 0 | ||
| Checkpoint replay, 16 | 1088 |
| Seed | No adaptation | Stale | Degree 3 | Cubic bank | Analytic history |
|---|---|---|---|---|---|
| 0 | |||||
| 1 | |||||
| 2 | |||||
| 3 | |||||
| 4 |
| Anchor norm target | Method | Prediction NMSE |
|---|---|---|
| 0.97 | Stale state | |
| 0.97 | Checkpoint replay, 256 | |
| 0.97 | Linear stitching | |
| 0.97 | Degree-3 memory | |
| 0.97 | Cubic interpolation | |
| 0.97 | Analytic history bank |
| Update period | Method | Prediction NMSE |
|---|---|---|
| 128 | Stale state | |
| 128 | Checkpoint replay, 256 | |
| 128 | Linear stitching | |
| 128 | Degree-3 memory | |
| 128 | Cubic interpolation | |
| 128 | Analytic history bank |
| Nominal leakage | Degree | Median | 90th percentile |
|---|---|---|---|
| 0 | 1 | ||
| 0 | 2 | ||
| 0 | 3 | ||
| 0 | 4 | ||
| 0.001 | 1 | ||
| 0.001 | 2 |
| Tolerance | Nominal leakage | Certified fraction | Mean selected degree |
|---|---|---|---|
| 0.001 | 0 | 1.00 | 3.12 |
| 0.001 | 0.001 | 0.00 | — |
| 0.001 | 0.003 | 0.00 | — |
| 0.001 | 0.01 | 0.00 | — |
| 0.001 | 0.03 | 0.00 | — |
| 0.01 | 0 | 1.00 | 2.00 |
| Method | Stream ms | Interquartile range, ms |
|---|---|---|
| Cubic interpolation | ||
| First-order sensitivity | ||
| Full replay | ||
| Analytic history bank | ||
| Fitted history bank | ||
| Linear stitching |
| Shared operation | Median ms | IQR, ms |
|---|---|---|
| Analytic history, four decoders | ||
| Coefficient model reductions | ||
| Cubic bank reductions and nodes | ||
| Fitted history, four maps | ||
| Linear stitching, five maps |
| Shared auxiliary model data | Bytes |
| Linear stitching, five maps | 670,720 |
| Fitted history, four maps | 836,096 |
| Analytic history, four decoders | 819,200 |
| Cubic nodes and interpolation inverse | 792 |
| Shared | 9,728 |
| Stage | Available information | Operation |
|---|---|---|
| Source and pilot fitting | Independent source transitions and six pilot domains, observed states, known injection and readout | Fit the anchor, update family, and image |
| Memory provision | Anchor, permitted family, chosen degree or bank width, prescribed pole | Fix the causal recurrence before evaluation inputs or replacement queries |
| Streaming | Current input, previous retained memory, anchor and active model | Maintain the summary and active prediction state |
| Target fitting | Independent transitions from a new domain | Fit replacement coordinates and construct shared target decoder |
| Migration query | Retained summary and fitted target, but not discarded evaluation inputs | Reconstruct a target state and continue under common future inputs |
| Bytes | Output NRMSE | State NRMSE | |||
|---|---|---|---|---|---|
| Repeated | Split | Repeated | Split | ||
| 1 | 8 | ||||
| 2 | 16 | ||||
| 4 | 32 | ||||
| 8 | 64 | ||||
| 16 | 128 | ||||
| Method | Task NMSE | Sample SD | Output NRMSE | State NRMSE |
|---|---|---|---|---|
| No adaptation | ||||
| Stale | ||||
| Restricted replay | ||||
| Unrestricted replay | ||||
| Degree 3 | ||||
| Cubic bank |
| Method | Task NMSE | Sample SD | Output NRMSE | State NRMSE |
|---|---|---|---|---|
| No adaptation | ||||
| Stale | ||||
| Restricted replay | ||||
| Unrestricted replay | ||||
| Linear stitching | ||||
| First-order sensitivity |
| Degree | Full lift | Extra | Extra, rep. | Float64 | Float32 | |
|---|---|---|---|---|---|---|
| 1 | 6 | 2 | 1 | 1.124 | ||
| 2 | 12 | 4 | 2 | 1.152 | ||
| 3 | 20 | 6 | 3 | 1.166 |
Information-Theoretic Lower Bounds
Information-theoretic lower bounds quantify the data, queries, communication, or computation no algorithm can avoid in learning and decision tasks.
See all 92 papers