What to Preserve in Recursive Computation: A Local Predictive Sufficiency Principle

Peilin Wang; Feng Shiyang; Hongfu Gao; Cencheng Zhao; Di Yuan; Hui Chen; Guiguang Ding

cs.LGOct 3, 2026

What to Preserve in Recursive Computation: A Local Predictive Sufficiency Principle

Authors: Peilin Wang, Feng Shiyang, Hongfu Gao, Cencheng Zhao, Di Yuan, Hui Chen, Guiguang Ding

Organizations: Shandong University · National University of Singapore · Tsinghua University

Abstract

Recursive computation repeatedly compresses or reuses intermediate states, creating a simple tension: information that must remain useful across longer recursive paths is also exposed to more opportunities for loss before reaching the final prediction. Existing reconstruction or local-prediction objectives provide tractable supervision, but do not ensure that the retained information remains sufficient for subsequent recursive computation. We identify local predictive sufficiency with recursive predictive closure: controlling local predictive deficiencies at individual interfaces controls the resulting discrepancy at the root. We then turn this principle into a tractable training procedure. Starting from a variational characterization, we derive finite predictive tests and an empirical predictive deficiency that measures predictive value retained across compression. Its predictive sensitivities define margin-relaxed half-space constraints on parameter updates, and we project the host optimizer's proposed update onto their intersection only when predictive preservation would otherwise be violated. Across temporal graphs, language memory, vision-language-action control, and recursive self-improvement, the method matches or improves the corresponding host models under matched compression budgets, with larger gains under heavier recursive or memory demands, while better preserving predictive information across successive transformations. Crucially, the same task-agnostic predictive-preservation principle is instantiated across all four settings through host-compatible interventions while keeping the endpoint task, backbone, and evaluation protocol fixed. These results establish predictive preservation at recursive interfaces as a general training principle for recursive compression.

Figures & tables

Figure 1: From the exact predictive criterion to empirical predictive deficiency. The construction uses finite tests, covariance-balanced readouts, and estimates the population quantity.

Figure 3: Predictive feasibility optimization for recursive compression. Predictive sensitivities constrain the task proposal before parameter write-back.

(a) Temporal Graph Learning Prediction across propagation depths; AP/AUC in %
Dataset / Task	Setting	TGN	TGAT	TCL	CTAN	DyGFormer	Ours
Wikipedia Node Classification	Depth 2 AUC $\uparrow$	83.61	85.45	85.64	87.31	82.90	88.81
Wikipedia Node Classification	Depth 3 AUC $\uparrow$	87.47	82.08	81.26	87.38	87.00	89.17
UCI Link Prediction	Depth 3 AP $\uparrow$	93.04	86.08	86.35	–	95.47	94.14
	Depth 3 AUC $\uparrow$	92.79	85.44	85.95	–	94.52	93.73
	Depth 4 AP $\uparrow$	94.07	84.14	83.78	–	94.92	94.89

Table 1: Main results across recursive systems. Results span temporal graphs, language memory, VLA control, and SRI; best results appear bold.

Figure 4: Mechanism analysis. (a) The finite-test predictive value

J

tracks predictive distinctions retained across recursive interfaces. (b) Future gain retained measures whether retained information remains useful when reused in later recursive updates. Together, the two diagnostics test whether end-task improvements are accompanied by stronger predictive preservation and downstream reuse.

Variant	AP $\uparrow$	AUC $\uparrow$
Task-Only Training	92.62	92.13
Direct Future Prediction	92.76	92.24
Single-Event Predictive Criterion	92.41	91.93
Additive Predictive-Gradient Update	92.64	92.18
Global Predictive Constraint	90.32	89.88
Our Method	94.52	94.25

Table 2: Single-seed diagnostic ablation on UCI at depth 3. We isolate the roles of the predictive target, the adjacent predictive pair, and the way predictive supervision constrains updates.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Data / split	Training / initialization	Selection	Frozen evaluation
Temporal graphs	Wikipedia and UCI; chronological 70/15/15 split	TGN host with matched task, depth, temporal neighborhood, and seed; five independent runs	Wikipedia: validation ROC-AUC; UCI: full-validation AP	Selected checkpoint on held-out test; causal memory replay for Wikipedia; global AUC/AP. Refs.: C.1 , D.1 , E.1
Language memory	DailyDialog original splits with a fixed-target depth cohort; MSC chronological five-session conversations	One fixed CCM-merge host per backbone and dataset; three independent Ours runs initialized from the same host	Pre-specified validation criterion; one checkpoint selected per Ours run before depth-wise evaluation	DailyDialog: fixed pooled depth cohort; MSC: held-out test conversations; dialogue-equal PPL. Refs.: C.2 , D.2 , E.2
VLA	100 released refined T3 demonstrations: 80 training + 20 internal development	MemoryVLA host trained from the common OpenVLA initialization; Ours starts from the matched trained host and optimizes only $\Gamma$ ; three independent runs	20-demonstration internal development split only; 25k-step upper budget; one checkpoint frozen per run	20 separate official held-out T3 evaluation initializations; matched MemoryVLA/Ours rollouts; SR and SGC. Refs.: C.3 , D.3 , E.3
SRI	Fixed six-task CO-Bench cohort with official development and held-out instances	Meta n and Ours begin from the same realized Layer-1 state and receive the same recursive-search opportunity and retained-history capacity	Recursive refinement and strategy selection use development feedback only; selected strategies are then frozen	Official held-out instances; matched evaluation of frozen strategies; three evaluation repeats for the reported task-level scores. Refs.: C.4 , D.4 , E.4

Appendix

Table 3: Reproduction map for the four experimental domains. Each row traces the reported results from data preparation through model selection and frozen evaluation. Each row links its protocol, results, and implementation.

Dataset	Nodes	Interactions	Task	Depth
Wikipedia	9,227	157,474	Node classification	2, 3
UCI	1,899	59,835	Link prediction	3, 4

Appendix

Table 4: Temporal-graph datasets and evaluation settings.

Setting	Wikipedia	UCI
Task	Node classification	Temporal link prediction
Evaluation setting	Chronological	Transductive
Split	70/15/15	70/15/15
Depth	2, 3	3, 4
TGN temporal neighbors	10 (depth 2); 5 (depth 3)	10 (depth 3); 5 (depth 4)
Evaluation negatives	–	1 per positive

Appendix

Table 5: Temporal-graph evaluation protocol.

Setting	DailyDialog	MSC
Task	Next-utterance prediction	Session-opening prediction
Backbones	LLaMA-7B / Gemma-4	LLaMA-7B / Gemma-4
Host	CCM-merge	CCM-merge
Depth	$L=4,6,8,11,13$	$S=3,4,5$
Evaluation cohort	Fixed pooled $\geq 15$ -utterance dialogues	Fixed five-session test conversations
History	$L$ preceding utterances	Sessions $1,\ldots,S-1$

Appendix

Table 6: Language-memory evaluation protocol.

Component	Protocol
Data source	Official CO-Bench release
Task cohort	Same fixed six tasks
Host	Public Meta n implementation
Backbones	gpt-5.5 / gemini-3.1-pro / DeepSeek-V4.1-Flash
Model assignment	Same backbone at Layer 1 and all $\Omega$ calls
Initialization	Same realized Layer-1 state within each pair

Appendix

Table 7: Controlled SRI protocol on CO-Bench. Meta n and Ours share the realized initialization, retained-history capacity, recursive-search opportunity, and evaluation protocol; the method-specific intervention changes which execution traces are retained for subsequent recursive refinement.

Dataset / Task	Setting	TGN	TGAT	TCL	CTAN	DyGFormer	Ours
Wikipedia Node Classification	Depth 2 AUC $\uparrow$	$83.61\pm 1.00$	$85.45\pm 0.26$	$85.64\pm 0.32$	$87.31\pm 0.29$	$82.90\pm 1.59$	$\mathbf{88.81\pm 0.77}$
Wikipedia Node Classification	Depth 3 AUC $\uparrow$	$87.47\pm 1.05$	$82.08\pm 1.73$	$81.26\pm 1.77$	$87.38\pm 0.14$	$87.00\pm 0.71$	$\mathbf{89.17\pm 0.67}$
UCI Link Prediction	Depth 3 AP $\uparrow$	$93.04\pm 1.73$	$86.08\pm 0.66$	$86.35\pm 0.82$	–	$\mathbf{95.47\pm 0.27}$	$94.14\pm 0.33$
	Depth 3 AUC $\uparrow$	$92.79\pm 1.68$	$85.44\pm 0.61$	$85.95\pm 0.60$	–	$\mathbf{94.52\pm 0.23}$	$93.73\pm 0.56$
	Depth 4 AP $\uparrow$	$94.07\pm 0.13$	$84.14\pm 1.70$	$83.78\pm 0.62$	–	$94.92\pm 0.25$	$94.89\pm 0.09$
	Depth 4 AUC $\uparrow$	$93.73\pm 0.12$	$83.09\pm 1.79$	$83.20\pm 0.59$	–	$94.14\pm 0.21$	$\mathbf{94.47\pm 0.12}$

Appendix

Table 8: Complete temporal-graph results. Results are reported as mean

\pm

standard deviation over five random seeds. AP and AUC are reported in %.

Figure 5: Treewise predictive-constraint conflict during training. Violation density is the fraction of grouped predictive constraints violated by the unconstrained task proposal. On UCI 3-hop (R0,

\kappa=0.05

), the mean violation density decreases from

0.020

in the first third of training to

0.012

in the final third.

$\alpha$	$\alpha\,q^{\top}\delta$	$\Delta\widehat{J}_{\mathrm{win}}$	$\Delta\widehat{J}_{\mathrm{win}}/(\alpha q^{\top}\delta)$	Relative scaling
1	$3.639\times 10^{-7}$	$4.044\times 10^{-7}$	$1.111$	$1.0\times$
16	$5.822\times 10^{-6}$	$6.386\times 10^{-6}$	$1.097$	$15.8\times$

Appendix

Table 9: Real-data first-order bridge audit on UCI. We compare the first-order change predicted by the constraint surrogate with the measured finite change of the exact window statistic used by PairKFWindow . The perturbation scale

\alpha

multiplies the same parameter-space update direction.

Dataset / Metric	Depth	LLaMA-7B			Gemma-4-E4B-It
Dataset / Metric	Depth	CCM-M	Ours	$\bm{\Delta}$ (%)	CCM-M	Ours	$\bm{\Delta}$ (%)
DailyDialog PPL $\downarrow$	Selected ckpt.	s650	s70	–	s600	s40	–
	$L=4$	5.98	$5.86\pm 0.03$	$-1.98\pm 0.52$	7.00	$6.97\pm 0.01$	$-0.54\pm 0.19$
	$L=6$	5.85	$5.71\pm 0.05$	$-2.51\pm 0.80$	6.84	$6.78\pm 0.01$	$-0.80\pm 0.11$
	$L=8$	5.80	$5.65\pm 0.07$	$-2.56\pm 1.13$	6.85	$6.77\pm 0.05$	$-1.05\pm 0.75$
	$L=11$	5.72	$5.55\pm 0.12$	$-2.91\pm 2.06$	6.77	$6.66\pm 0.07$	$-1.64\pm 0.98$

Appendix

Table 10: Language-memory results across recursive depth. DailyDialog reports dialogue-equal target PPL under the fixed-target controlled-depth protocol, while MSC reports dialogue-equal session-opening PPL. Lower PPL is better. CCM-merge reports one fixed validation-selected host checkpoint. Ours is initialized from that checkpoint and reports mean

\pm

standard deviation over three independent training seeds.

\Delta(\%)

reports the corresponding mean

\pm

standard deviation of the seed-wise relative PPL change with respect to the fixed CCM-merge baseline.

Figure 6: Training-time predictive diagnostics for CCM. Internal predictive-preservation signals and downstream validation gains are shown over the same training trajectory, where higher values are better in both panels.

	Ours @ 12k		MemoryVLA @ 20k
Eval. init.	Success	Completion step	Success	Completion step
1	0/3	–	0/3	–
2	1/3	105	1/3	105
3	2/3	230, 259	3/3	230, 255, 293
4	0/3	–	0/3	–
5	1/3	135	1/3	135

Appendix

Table 11: Held-out initialization-level comparison for training run 1. Ours at 12k and MemoryVLA at 20k are evaluated on the same 20 official held-out evaluation initializations. One closed-loop T3 rollout is run from each initialization. “Success” reports the number of ordered T3 repetitions completed sequentially within that rollout, from 0/3 to 3/3. “Completion step” gives the rollout step at which each credited repetition is completed.

Training run	Method	Ckpt. step	$\mathbf{0/3}$	$\mathbf{1/3}$	$\mathbf{2/3}$	$\mathbf{3/3}$	T3 SR $\uparrow$	T3 SGC $\uparrow$
Run 1	MemoryVLA	20k	13	4	2	1	5.0%	18.3%
Run 1	Ours	12k	10	5	3	2	10.0%	28.3%
Run 2	MemoryVLA	18k	11	6	3	0	0.0%	20.0%
Run 2	Ours	17k	12	3	3	2	10.0%	25.0%
Run 3	MemoryVLA	20k	10	7	3	0	0.0%	21.7%
Run 3	Ours	18k	11	4	2	3	15.0%	28.3%

Appendix

Table 12: Complete held-out T3 results on LIBERO-Mem across training runs. Each training run is evaluated on the same 20 official held-out evaluation initializations, with one closed-loop T3 rollout from each initialization. Within each rollout, the task requires three ordered lift-and-return repetitions to be completed sequentially. Columns

0/3

–

3/3

report the number of evaluation initializations whose rollout completes exactly zero, one, two, or three ordered repetitions, respectively. T3 SR denotes strict task success, i.e., the fraction of the 20 rollouts that complete all three ordered repetitions. T3 SGC is the total number of credited ordered repetitions completed within the 20 rollouts, divided by the maximum possible

20\times 3=60

ordered repetitions. The aggregate rows pool the corresponding initialization–training-run instances, and report the three-run mean

\pm

standard deviation.

Panel of Figure 7 — Figure 7: Qualitative rollout comparison on LIBERO-Mem T3, held-out evaluation initialization 15. Snapshots are sampled at regular intervals from paired rollouts of training run 1. Within this held-out T3 rollout, Ours completes 2/3 ordered repetitions, while MemoryVLA completes 1/3 (Table 11 ).

Panel of Figure 8 — Figure 8: Qualitative rollout comparison on LIBERO-Mem T3, held-out evaluation initialization 16. Snapshots are sampled at regular intervals from paired rollouts of training run 1. Within this held-out T3 rollout, Ours completes all 3/3 ordered repetitions, while MemoryVLA completes 2/3 (Table 11 ).

	Development			Held-out
Model	Meta n	Ours	$\bm{\Delta}$	Meta n	Ours	$\bm{\Delta}$
gemini-3.1-pro	0.9001	0.9547	+0.0546	0.8338	0.8479	+0.0141
gpt-5.5	0.8172	0.8279	+0.0107	0.7527	0.7579	+0.0051
DeepSeek-V4.1-Flash	0.8070	0.8600	+0.0530	0.7924	0.8222	+0.0298

Appendix

Table 13: Summary of SRI results. We report development scores from the paired recursive-search run and one fixed held-out evaluation of the frozen Meta n and Ours strategies. The main table reports the mean over three matched held-out evaluations of the same frozen strategies.

(a) gemini-3.1-pro
Task	Meta n	Ours	$\bm{\Delta}$
aircraft_landing	83.89	92.82	+8.93
assignment_problem	100.00	100.00	0.00
assortment_problem	55.27	55.27	0.00
bin_packing	94.17	93.57	-0.60
capacitated_warehouse_location	69.28	69.40	+0.12

Appendix

Table 14: Task-level held-out SRI results.

\Delta

denotes Ours minus Meta n in percentage points; scores are shown in percent. Differences are computed before rounding, so displayed values may differ by 0.01 percentage points. Each panel reports one fixed held-out evaluation of the frozen strategies from the single paired recursive-search run for that backbone. The main table reports the mean over three matched held-out evaluations of the same frozen strategies.

	Meta n		Ours
Depth	#Cand.	Dev	#Cand.	Dev	$\bm{\Delta}$
d1	1	0.6845	1	0.6845	0.0000
d2 (merge)	5	0.8073	5	0.8605	+0.0532
d3	6	0.8073	14	0.8546	+0.0473
d4	14	0.8037	4	0.8605	+0.0568
d5	–	–	2	0.8546	–

Appendix

Table 15: Candidate evolution for DeepSeek-V4.1-Flash. #Cand. denotes the number of retained candidates at each recursive depth. Ours changes both the retained candidate set and the resulting development trajectory.

Gen.	Meta n response	Ours response	Reason for the change
1	Accelerate assignment solving and refine assortment packing through restarts and stock selection.	Use linear_sum_assignment for assignment; diversify assortment ordering and stock selection.	Runtime is the main limitation for assignment, while assortment is sensitive to fixed ordering and stock choices.
2	Continue assortment refinement using stronger packing heuristics and stock combinations; one branch explores warehouse search.	Remove restrictive assortment choices, improve optional-piece handling, and introduce MaxRects-style packing.	Earlier generated solvers do not fully realize the intended packing strategy, leaving implementation-level restrictions.
3	Continue assortment improvement with randomized ordering, GRASP-style search, and richer placement heuristics.	Continue assortment refinement, introduce local search for warehouse, and apply simulated annealing (SA) to aircraft landing sequences.	After assortment improves, the model identifies additional headroom in warehouse and aircraft optimization.
4	Further refine assortment using SA/hill climbing and additional placement and stock-selection heuristics.	Strengthen warehouse subset search and add exact transportation-LP evaluation; continue improving assortment packing helpers.	Heuristic warehouse evaluation can mis-rank candidate subsets, motivating exact final evaluation.
5	Continue assortment-oriented search and packing refinement.	For aircraft_landing , retain SA for sequence search, keep the top 10 candidate sequences, and optimize landing times with scipy.optimize.linprog .	The model identifies greedy landing-time assignment as the main remaining bottleneck: promising sequences can be mis-scored because early/late penalties are not optimized globally.
6	Continue assortment refinement and implementation cleanup.	Explore wider aircraft candidate funnels and exact-LP local refinement, while continuing assortment helper integration.	After the Gen. 5 aircraft improvement, the model attributes the remaining limitation to candidate pre-selection before exact evaluation.

Appendix

Table 16: Evolution of model-generated solver strategies in gemini-3.1-pro. Each row summarizes the main strategy proposed at one generation and the model’s rationale for the subsequent change. The deployed aircraft solver ultimately retains the Gen. 5 strategy: simulated annealing over landing sequences followed by exact LP refinement of the top candidate sequences.

Configuration	Temporal Graph	Language Memory	VLA	SRI
Host	TGN	CCM-merge	MemoryVLA	Meta n
Formal repetitions	5 random seeds	fixed CCM-merge host + 3 Ours seeds	3 independently trained runs	1 paired search run / backbone
Optimization	Adam	AdamW; base LM frozen; task gradient on CCM comp. + $\Gamma$ ; predictive gradient on $\Gamma$	AdamW, frozen host + $\Gamma$	API-backed recursive search
Batch / accumulation	200	adaptive optimization windows	$4\times 4$ (effective 16)	–
Primary learning rate	task $10^{-4}$ ; repr. $3\times 10^{-4}$ (UCI formal line)	frozen formal CCM schedule	task-LR argument $10^{-5}$ ; $\Gamma$ LR $2\times 10^{-5}$	–
Schedule	validation-selected	cosine host schedule	constant; warmup 0	fixed search budget

Appendix

Table 17: Training and compute configuration used in the formal experiments.

Setting	Vanilla	Ours	Ratio
3L10N	87.1 s	165.7 s	1.9 $\times$
4L5N	88.6 s	190.6 s	2.2 $\times$

Appendix

Table 18: Training-time overhead on TGN. Mean epoch time under matched settings.

Setting	Vanilla eval	Ours eval	Vanilla build	Ours build
3L10N	68.1 s	66.0 s	1.6 s	0.6 s
4L5N	86.7 s	87.6 s	0.5 s	0.7 s

Appendix

Table 19: Inference-time comparison on TGN. The projection is disabled at inference.

Component	Ablation variant	AP	AUC
Objective	Task-Only Training	92.62	92.13
	History-reconstruction objective	90.89	90.28
	Direct Future Prediction	92.76	92.24
Supervision	Single-Event Predictive Criterion	92.41	91.93
	Cross-level mispaired supervision	92.73	92.28
	Predictive supervision without context	92.89	92.50

Appendix

Table 20: Ablation study on UCI at depth 3 (seed 0). Each variant changes one component of the full training procedure while keeping the remaining experimental protocol fixed.

Configuration	Temporal Graph	Language Memory	VLA	SRI
Predictive statistic
Statistic	Full balancing	Full balancing + OAS	Full dual	CCA / OAS
Sketch dimension	$64$	$4\times 32$	$64$	$4\times 8$
Statistic width	$128$	$128$	–	$32$
Context / future width	$32/32$	Fixed sketches	$64/64$	$256$ encoder
Covariance ridge	$10^{-3}$ (UCI); $10^{-4}$ (Wiki)	$10^{-3}$	$10^{-4}$	$10^{-3}$

Appendix

Table 21: Method-specific hyperparameters used in the formal experiments. Only parameters introduced by the predictive-preservation procedure or its statistical realization are reported.

Figure 9: Hyperparameter sensitivity on UCI. Test AP under variations of (a) the feasibility tolerance

\kappa

and (b) the eligible-interface fraction used to estimate the empirical predictive statistic. All remaining settings are fixed.

Explore similar work

Oct 9, 2026cs.LG

Prediction Is Not a Sufficient Memory State

A memory that reproduces every current prediction may still have forgotten how to learn from the next observation. We characterize this distinction for exact cumulative learning and attention. For smooth convex residual losses with unrestricted real features and labels, a continuous recurrent state of fixed dimension exists exactly when the loss is polynomial. With ordinary, unweighted observations, degree

D

requires precisely

r=\binom{d+D}{D}-1

coordinates; even among histories with one fixed current predictor, exactly

r-d

coordinates can remain necessary for continuation. Nonpolynomial losses have unbounded requirements on such equal-prediction histories. Bounded scalar logistic examples retain this obstruction after fixing any finite derivative jet. For ridge regression, the hidden evidence is the design matrix: the exact state costs

dp+d(d+1)/2

, versus

dp

for the predictor. On a well-conditioned zero-prediction family, optimal

B

-bit compression has average one-update error of order

d^{-1}2^{-2B/s_d}

s_d=d(d+1)/2

, even with arbitrary randomized codes. Kernel attention gives a second exact closure classification. The results identify what future updates reveal, rather than proposing a new regression algorithm or asserting trained-model gains.

Robert Jenkinson Alvarez

Topic pending

Ask PDF

Jun 25, 2026cs.LG

What Survives When You Compress a Recursive Reasoner for the Edge?

Recursive reasoning models solve hard structured tasks with a few million parameters by iterating a latent state, but deploying them on edge hardware means compressing them -- and quantization noise compounds across recursive cycles rather than accumulating over output tokens, so single-pass intuitions fail. Here, we ask what survives. Across a full precision sweep and three tasks, aggressive compression preserves local prediction but destroys global reasoning: pruning, distillation, and linear attention drive puzzle-exact accuracy to zero while cell accuracy holds. Naive INT4 is different: its damage is architectural, striking MLP-mixing recursion but not attention on the same task, and per-channel calibration reverses it without retraining, where quantization-aware training does not. We also introduce carry-trajectory fidelity, the cosine similarity to the full-precision reasoning path, as a label-free signal that predicts both the damage and the recovery before any task evaluation, with a threshold that transfers across tasks. The result is a deployment recipe: flash-streamed embeddings remove a 99.4 MB bottleneck, one recursive cycle matches full-depth accuracy at 6x fewer FLOPs, and calibrated INT4 brings the backbone to 3.3 MB, inside a 4 MB budget.

Pearse Jim, Steven Kolawole, Opegbemi Matthias Busoye +2

ML Collective · Carnegie Mellon University

Model Compression Efficient Neural Network Inference Structured Reasoning Neural Network Quantization Neural Network Pruning

Ask PDF

Aug 18, 2026cs.LG

Dynamic Compression in Recurrent Networks

Recurrent models process long contexts efficiently by compressing their history into a fixed-size state, but modern architectures typically do so in a single causal pass over the sequence. Each input must therefore be compressed before the model knows how it will later be used, forcing a limited state to compromise across possible future demands. We introduce dynamic compression, which allows a recurrent model to selectively revisit past tokens and revise its fixed-size state through additional recurrent updates. The model need not preserve every part of the history at uniformly high fidelity in its recurrent state, because lower-fidelity information can be revisited from the retained raw sequence when it becomes relevant. We study this in a controlled setting where the model first learns multiple functions in-context and, later in the same sequence, encounters a series of few-shot tasks that each require it to identify and reuse one of those functions. A single-pass model must preserve every function at sufficient fidelity for any future task, whereas selective re-scanning allows the model to revisit and refine only the function currently needed. We find that dynamic compression substantially reduces the recurrent state required for accurate reuse and scales more favorably as the number of stored functions grows. These results demonstrate a computation--memory tradeoff in which recurrent models can spend more computation revisiting their history to make more effective use of a fixed-size state.

Jyothish Pari, Ryan Bahlous-Boldi, Pulkit Agrawal

Recurrent Neural Networks Model Compression

Ask PDF