What Can a Recurrent State Safely Forget?

Linzhe Zhang; Changming Xu

cs.LGSep 20, 2026

What Can a Recurrent State Safely Forget?

Authors: Linzhe Zhang, Changming Xu

Organizations: Graduate School, Northeastern University.

Abstract

Recurrent models must preserve information that changes future behavior while suppressing hidden-state error. These objectives conflict: contraction improves stability, but contraction along a future-distinguishing direction destroys memory. We formalize this boundary through the predictive quotient of a recurrent state space. Two hidden states are equivalent when they induce the same conditional future; their equivalence classes form predictive fibers. Every exact semantics-preserving corrector acts as the identity on this quotient. At a regular point with hidden dimension d and predictive dimension k, it can eliminate at most d - k independent directions. This establishes a discrete-continuous boundary: finite predictive states admit positive-radius exact correction basins, whereas an uncountable continuum of future-distinguishable states cannot be decoded after arbitrary positive-radius perturbations in finite-dimensional Euclidean space. To operationalize this principle, we develop an auditable finite-future framework. A compact deployment bank W is evaluated against an independent audit bank A (W subseteq A) on a declared correction domain. Under generative probe access and audit-metric coverage, finite stochastic rollouts furnish a high-probability certificate for the separation margin Omega_{W|A}(delta). Preserving learned W-predictions within this certified margin guarantees bounded audit-semantic distortion. For intrinsic audit dimension k, the required probe outcomes scale as O(M * Omega^{-(k+2)}), where M = |A|; a matching minimax lower bound proves this exponent is optimal. Extending guarantees to continuous futures is achieved via an explicit completeness modulus. Controlled experiments validate the certified margins, scaling laws, and automated probe refinement under a safety-first evaluation paradigm.

Figures & tables

Study	task error	$\underline{\Omega}_{W\mid\mathcal{U}}$	$\tau$	$\xi$	cert. distortion	normal contraction
DFA hard snap	$0.187$	analytic	$0$	$0$	$0$ (basin)	$0.807$
Analog fiber	$0.108$	analytic	$0$	$0$	$0$	$1.000$
Mixed fiber	$z$ $0.118$	analytic	$0$	$0$	$0$	$1.000$
Rollout certificate	—	$0.052$	$0.0078$	$0$	$<\delta$ if gated	—
PF-CEGIS final	—	$0.066$	$0.009$	$0.007$	$\leq\delta$	$1.000$
Re-anchoring	law $0.000$	—	$0$	$0$	observation	—

Table 1: Comprehensive evaluation under the safety-first paradigm. Nonanalytic margins are evaluated relative to their pre-declared correction domains. “Known quotient” studies feature analytic ground-truth semantics. The HAR and digit benchmarks are reported as held-out empirical audits, distinguishing passive observational datasets from resettable generative PAC certificates.

Figure 1: Subject-disjoint UCI HAR evaluation under hidden-state noise (mean

\pm

s.d., 5 seeds per architecture). The third panel deliberately draws global and PF values side-by-side: PF interface change is numerical zero (

\approx 10^{-15}

), whereas global contraction changes every declared head.

Figure 2: Finite predictive states permit strong discrete correction. Hard snapping is effective inside its basin; the learned soft corrector is a noise-trained recurrent baseline.

Figure 3: Continuous integrator realization

h=(z,r_{1},r_{2})

. The two

r

coordinates are intentionally nonpredictive fiber noise, not additional memories. Fiber-only correction preserves

z

and contracts

(r_{1},r_{2})

; global contraction contracts all three coordinates and therefore biases analog memory.

Figure 4: Hybrid predictive state

s=(q,z)

. The mode is safely snapped, whereas the continuous coordinate must remain a semantic degree of freedom.

Figure 5: Finite-audit certification. We vary margin, intrinsic dimension, and literal rollout budget, and report the lower certificate rather than only prediction accuracy.

Figure 6: PF-CEGIS tracks the entire safety gate, not just its final probe count. Adding probes makes the certificate stronger and narrows the information the corrector is allowed to delete.

Figure 7: Periodic informative observations re-anchor neutral continuous memory. Solid lines are simulated steady pre-anchor errors; dashed lines are the theoretical

Hb/(1-\kappa)

predictions.

Figure 8: MuJoCo physical generative certificate. GRU, LSTM, and a selective SSM are trained on randomized-control rigid-body rollouts; disjoint exact-reset rollouts bound calibration and certify audit separation. The middle panel compares global and PF distortion: PF achieves numerical zero distortion, whereas isotropic global contraction corrupts the declared interface.

Figure 9: Real raster-revealed digit stream under partial observability. PF contracts normal directions while strictly preserving the declared class-and-future interface; global contraction corrupts class boundaries.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Claim	Premise used in the proof	Failure when absent
Quotient rank bound	$\pi$ is a $C^{1}$ rank- $k$ submersion locally and $Q(h)=h$ for the derivative identity.	At a non-fixed point, derivatives live at $h$ and $Q(h)$ and $D\pi_{h}(DQ_{h}-I)=0$ need not hold.
Finite embedding / all-future lift	Compact state manifold, $C^{1}$ tests separating points and tangents; additionally $\ell_{\infty}$ continuity of the full predictive map for a modulus.	Finite coordinate separation alone does not control an unmeasured future or supply a numerical conditioning bound.
PAC separation	Pre-declared $(\mathcal{D},\mathcal{U},\delta)$ , an $\eta$ -net of $\mathcal{D}$ , conditional reset at each anchor, bounded independent outcomes, and $W\subseteq\mathcal{U}$ .	Passive or adaptively selected trajectories can miss a region or invalidate Hoeffding/union-bound coverage.
Joint safety statement	Uniform calibration holds on the correction domain and is validated independently (or with a joint confidence budget).	A corrector can pass a margin measured with an overfit predictor while changing true deployment semantics.
$\Omega^{-(k+2)}$ upper bound	Covering-number bound and strict margin at $\delta/2$ ; $q=\min\{\delta,\Omega\}$ .	A cover without a positive margin cannot yield a positive certificate at any finite sample size.
Minimax lower bound	Lipschitz alternatives remain in $[0,1]^{k}$ , bump supports are disjoint, and Bernoulli means stay away from 0 and 1.	The localized alternatives may not be admissible, and the KL comparison no longer establishes the stated rate.

Appendix

Table A1: Proof-assumption audit. The final column characterizes structural failure modes and counterexample classes when premises are violated.

Figure A1: Offline certification cost. Left: the theorem’s literal sufficient schedule grows rapidly with intrinsic audit dimension. Center: exact-reset InvertedPendulum audit wall time and lower margin as rollout budget grows. Right: PF-CEGIS selects deployment subsets from a fixed fully sampled audit bank with no additional rollout cost.

Recurrent Neural Networks

Recurrent neural networks process sequences by updating a persistent hidden state, enabling temporal modeling, memory, and streaming computation.

See all 303 papers

May 8, 2026

Rethinking State Tracking in Recurrent Models Through Error Control Dynamics

The theory of state tracking in recurrent architectures has predominantly focused on expressive capacity: whether a fixed architecture can theoretically realize a set of symbolic transition rules. We argue that equally important is error control, the dynamics governing hidden-state drift along the directions that distinguish symbolic states. We prove that affine recurrent networks, a class of models encompassing State-Space Models and Linear Attention, cannot correct errors along state-separating subspaces once they preserve state representations. Consequently, practical affine trackers do not learn robust state tracking; rather, they learn finite-horizon solutions governed by accumulated state-relevant error. We characterize the mechanics of this failure, showing that tracking remains readable only while the accumulating within-class spread remains small relative to the between-class separation. We demonstrate empirically on group state-tracking tasks that this breakdown is predictable: tracking collapses when the distinguishability ratio crosses the readability threshold. Across trained models, the point of this crossing predicts the horizon at which downstream accuracy fails. These results establish that robust state tracking is determined not only by an architecture's theoretical expressivity but crucially by its error control.

Jiwan Chung, Heechan Choi, Seon Joo Kim

Yonsei University

Recurrent Neural Networks State Tracking State Space Models

Ask PDF

Sep 17, 2026

Correct Now, Insufficient Later: Auditing Update Sufficiency in Context Compression

A memory can answer a current query correctly while discarding distinctions required by a later update. We investigate this failure with a paired-history audit: two histories have the same current answer, receive a shared future update, and require different subsequent answers. A pilot evaluates 24 history pairs across six synthetic mechanisms, 12 memory conditions, two repeats, and two model backends. A deterministic frontier selector obtains strict reveal accuracy of 96/96 on DeepSeek and 82/96 on GLM; a structured writer obtains 62 successes with one unresolved outcome and 56/96. The configured four-outcome joint contrast has finite-sample identification intervals of [0.521, 0.542] and [0.292, 0.313], not confidence intervals. A record-level audit distinguishes retained-state adequacy, response delivery, and answer-schema compliance without changing those original scores. It finds 26 and 25 well-formed but semantically wrong structured reveal memories, while all 14 GLM frontier reveal failures contain correct values in the wrong wrapper. Tombstone removal produces 16/16 exact replay failures in the targeted mechanism. Identifier renaming then exposes a separate flaw: original frontier late-reference adequacy falls from 8/8 to 94/320 transformed instances. We provide and test a label-equivariant repair, but it preserves only 2/8 original late-reference answers: eliminating a naming shortcut does not solve unknown future relevance. These results support a scoped evaluation methodology and reproducible failure analysis, not general superiority of the repaired algorithm. Paid pilot evidence, retrospective diagnostics, and new offline tests are reported separately; no independent held-out or natural-task validation is claimed.

Guangzhe Zhang

Independent AI Researcher

Agent Memory Counterfactual Evaluation LLM Auditing Context Compression

Ask PDF

Sep 6, 2026

Exact Record Omission in Delta Attention: A Transport Criterion, Its Cost, and a Replay Certificate

An assistant can stop repeating a deleted statement while its recurrent memory still carries that statement's influence. We examine this distinction by saving the state difference immediately after a record, carrying this saved difference, or receipt, through later updates, and comparing the corrected state with the state built from the same conversation with the record omitted. Unrolling the recurrence gives an exact criterion: transport reaches this never-stored state if and only if the additional differences created by later updates cancel after transport. We evaluate eleven conditions using 40 prospectively selected synthetic Kimi Linear contexts, with 40 separate calibration contexts and paired uncertainty estimates computed across complete records. Attention masking removed all 40 exact greedy target answers, yet a fixed three-query candidate-scoring attack with known candidate values and log-probability access attained AUC 0.740 (95% interval 0.695--0.809); the measured false-positive rate was 7.5%. Prompt-only forgetting still returned 33 of the 40 targets. Checkpoint replay matched the complete declared active state and audit logits in all 80 contexts, while preserving all 120 retained answers in the evaluation cohort. An independent Qwen cohort and matched original-bf16/8-bit Kimi controls confirmed native transport mismatch beyond the numerical error measured in matched controls. Replay also supported record replacement and successive deletions with a new record added between them. These results provide a practical audit of both the requested memory change and the assistant's remaining behavior, with replay work determined by the surviving suffix.

Vishwajith Ramesh

Vy Labs, Inc.

LLM Unlearning Machine Unlearning Evaluation Language Model Auditing Memorization in Language Models Delta-Rule Linear Attention

Ask PDF

Abstract

Recurrent Neural Networks

Explore similar work

Rethinking State Tracking in Recurrent Models Through Error Control Dynamics

Correct Now, Insufficient Later: Auditing Update Sufficiency in Context Compression

Exact Record Omission in Delta Attention: A Transport Criterion, Its Cost, and a Replay Certificate