Exact Record Omission in Delta Attention: A Transport Criterion, Its Cost, and a Replay Certificate

Vishwajith Ramesh

cs.LGSep 6, 2026

Exact Record Omission in Delta Attention: A Transport Criterion, Its Cost, and a Replay Certificate

Authors: Vishwajith Ramesh

Organizations: Vy Labs, Inc.

Abstract

An assistant can stop repeating a deleted statement while its recurrent memory still carries that statement's influence. We examine this distinction by saving the state difference immediately after a record, carrying this saved difference, or receipt, through later updates, and comparing the corrected state with the state built from the same conversation with the record omitted. Unrolling the recurrence gives an exact criterion: transport reaches this never-stored state if and only if the additional differences created by later updates cancel after transport. We evaluate eleven conditions using 40 prospectively selected synthetic Kimi Linear contexts, with 40 separate calibration contexts and paired uncertainty estimates computed across complete records. Attention masking removed all 40 exact greedy target answers, yet a fixed three-query candidate-scoring attack with known candidate values and log-probability access attained AUC 0.740 (95% interval 0.695--0.809); the measured false-positive rate was 7.5%. Prompt-only forgetting still returned 33 of the 40 targets. Checkpoint replay matched the complete declared active state and audit logits in all 80 contexts, while preserving all 120 retained answers in the evaluation cohort. An independent Qwen cohort and matched original-bf16/8-bit Kimi controls confirmed native transport mismatch beyond the numerical error measured in matched controls. Replay also supported record replacement and successive deletions with a new record added between them. These results provide a practical audit of both the requested memory change and the assistant's remaining behavior, with replay work determined by the surviving suffix.

Figures & tables

term	meaning	relevant sections
target record	the record a user asks to delete; one contiguous span of tokens	all
never-stored state	state built from the recorded history with the target left out (Eq. 2 )	all
receipt	the state difference at the boundary just after the record	§ VI-A , § VI-C
transport	carrying the receipt forward through the record-present transitions	§ VI-A , § VI-C
forcing	new differences created by later updates because the record was present; these are additional to the boundary receipt	§ VI-A –§ VI-D
imprint (suffix sweep)	$\lVert$ present $-$ never-stored $\rVert/\lVert$ never-stored $\rVert$ per layer; residual without an edit	§ VI-C

TABLE I: Terms used in this paper and the relevant sections.

question	comparison or test	evidence	outside the claim
When does receipt transport succeed?	affine recurrence algebra	Theorem 1 , Corollary 1	non-affine updates
Does native KDA meet the criterion?	frozen-input counterfactual vs. native never-stored state	relative state residual; forcing norm ratio	caches outside the recurrence
What do the methods cost?	partial recomputation; deferred log vs. checkpoint of recurrent and convolution state	residual vs. fraction recomputed; storage in bytes	other implementations and storage schemes
Is residual information detectable?	probes on each corrected state vs. the never-stored reference	lift, rank, greedy, sampling; three-query candidate scoring	untested adaptive attacks; untested generation-only attacks
Do next-token distributions match?	exact rebuilt never-stored state	first-token KL divergence; requires probability access	equivalence of full distributions
Does replay match the reference?	per-array float32 equality	94 active arrays (recurrent, convolution, and MLA), offsets, and logits in the prospective cohort (§ VI-G )	prior outputs; unused allocation; downstream causal effects

TABLE II: Research questions, comparisons, evidence, and limits.

intervention	intervention setup	observed result
zero recurrent matrices	leave MLA caches and convolution arrays unchanged	in one synthetic example, the ward code remains rank 1; first-token probability $0.881$
attention masking	hide target attention entries; leave recurrent and convolution state unchanged	nonzero first-token KL on all 216 synthetic and 144 TOFU configuration–cut pairs
frozen-input transport	identical suffix transitions and writes	receipt matches the frozen-input record effect to $6.1\times 10^{-6}$ relative error
native transport	subtract the transported boundary receipt without correcting later forcing	residual $0.7$ – $32.9\%$ of record-present suffix-A state norm; forcing norm ratio $0.064$ – $1.010$
checkpoint replay	restore the preceding checkpoint and process the retained suffix again	zero residual on logits and 80 KDA arrays

TABLE III: State and readout interventions in the 48B Kimi Linear hybrid.

Fig. 1: Suffix sweep on 24 synthetic configurations and 20 KDA layers. (a) Recurrent-state residual divided by the never-stored state norm; the zero-token cut is separated from the positive log scale. (b) Accumulated forcing norm divided by the norm of the original state difference between the record-present and never-stored trajectories. (c) Remaining recurrent-state mismatch norm after recomputing an initial suffix segment and transporting through the rest, divided by the same original state-difference norm. The dotted line is a linear reference. Points are medians across layers and configurations; bands span the 10th–90th percentiles, not confidence intervals.

Fig. 2: Target lift (nats per token) against recurrent-state residual relative to the never-stored state norm, averaged over layers. Each point is a condition–cut pair from 24 synthetic or 16 TOFU configurations. The receipt series contains every static, ledger, and full-matrix point. Positive residuals use a log scale; the labelled left-edge marker represents actual zero. Zero lift means agreement on the target’s mean log probability per token, not equality of the output distributions or internal states. Masking lowers lift but includes positive scores across configurations and cuts.

condition	target lift [95% CI] (nats/token)	exact target answers / 40	AUC [95% CI]	detection %
record present	3.240 [3.062, 3.411]	40	1.000 [1.000, 1.000]	100
prompt forgetting	3.121 [2.943, 3.291]	33	1.000 [1.000, 1.000]	100
attention masked	0.486 [0.426, 0.546]	0	0.740 [0.695, 0.809]	30
masked + transport	0.0178 [0.0108, 0.0245]	0	0.519 [0.511, 0.543]	7.5
checkpoint replay	0 [0, 0]	0	0.500 [0.500, 0.500]	7.5
never stored (reference)	0 [0, 0]	0	0.500 [0.500, 0.500]	7.5

TABLE IV: Target recall and detection in the prospective Kimi cohort.

Fig. 3: Retained-suffix processing time at nine deletion positions in one 128-record context. Each point is one timing, not an average over repeated runs. The newest record has no suffix to process, giving

0.00

s; this is not a measurement of total deletion latency. Processing the suffix after the oldest record takes 6.70 s, close to the 6.77-second full rebuild.

Fig. 4: Transport checks in three recurrent models, with 12 configurations per model and predictions recorded before the runs. (a) Maximum frozen-input transport error relative to the never-stored state norm, over informative layers, cuts, and configurations. The line marks the numerical tolerance

1\times 10^{-4}

, distinct from zero-residual replay. (b) Native forcing norm ratio at the longest suffix: median across configuration-level layer medians. Bars show the range across configurations, not confidence intervals.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Condition	Target log-probability $\Delta$	Retained log-probability $\Delta\times 10^{3}$	Continuation NLL $\Delta$
Record present	$3.2399\;[3.0625,\,3.4113]$	$-0.249\;[-0.453,\,-0.043]$	$0.0055\;[-0.0083,\,0.0192]$
Attention masking	$0.4858\;[0.4257,\,0.5463]$	$-0.269\;[-0.461,\,-0.085]$	$-0.0025\;[-0.0161,\,0.0114]$
Static receipt	$3.1783\;[3.0102,\,3.3414]$	$0.101\;[-0.125,\,0.302]$	$-0.0133\;[-0.0315,\,0.0038]$
Diagonal ledger	$3.1884\;[3.0179,\,3.3528]$	$-0.067\;[-0.223,\,0.078]$	$-0.0041\;[-0.0193,\,0.0098]$
Receipt transport	$3.1878\;[3.0180,\,3.3514]$	$-0.069\;[-0.249,\,0.096]$	$-0.0057\;[-0.0194,\,0.0078]$
Masking + transport	$0.0178\;[0.0108,\,0.0245]$	$-0.101\;[-0.234,\,0.014]$	$0.0049\;[-0.0050,\,0.0151]$

Appendix

TABLE V: Target and retained-information scores in the Kimi cohort.

Condition	AUC [95% interval]	TPR, % [two-stage 95% interval]	TPR Wilson 95% interval, %
Record present	$1.0000\;[1.0000,\,1.0000]$	$100.0\;[100.0,\,100.0]$	$[91.2,\,100.0]$
Attention masking	$0.7400\;[0.6950,\,0.8094]$	$30.0\;[5.0,\,50.0]$	$[18.1,\,45.4]$
Static receipt	$1.0000\;[1.0000,\,1.0000]$	$100.0\;[100.0,\,100.0]$	$[91.2,\,100.0]$
Diagonal ledger	$1.0000\;[1.0000,\,1.0000]$	$100.0\;[100.0,\,100.0]$	$[91.2,\,100.0]$
Receipt transport	$1.0000\;[1.0000,\,1.0000]$	$100.0\;[100.0,\,100.0]$	$[91.2,\,100.0]$
Masking + transport	$0.5188\;[0.5113,\,0.5431]$	$7.5\;[0.0,\,22.5]$	$[2.6,\,19.9]$

Appendix

TABLE VI: Three-call detection results in the Kimi cohort.

quantity	short suffix	long suffix	paired long minus short
native transport effect scaled	$0.5222\ [0.5151,0.5296]$	$0.5803\ [0.5760,0.5847]$	$0.0581\ [0.0506,0.0655]$
native transport state scaled	$0.0946\ [0.0914,0.0978]$	$0.0878\ [0.0856,0.0903]$	$-0.0067\ [-0.0104,-0.0032]$
frozen static state scaled	$0.2884\ [0.2850,0.2917]$	$0.3059\ [0.3022,0.3093]$	$0.0174\ [0.0161,0.0187]$
frozen ledger state scaled	$0.1032\ [0.1019,0.1045]$	$0.0892\ [0.0881,0.0903]$	$-0.0140\ [-0.0148,-0.0132]$

Appendix

TABLE VII: Qwen transport residuals with short and long suffixes.

condition	derived 8-bit lift	original bf16 lift	bf16 minus 8-bit lift
present	$3.287\ [2.793,3.810]$	$3.255\ [2.773,3.756]$	$-0.032\ [-0.076,0.003]$
attention masking	$0.426\ [0.268,0.580]$	$0.403\ [0.227,0.583]$	$-0.023\ [-0.068,0.017]$
receipt transport	$3.239\ [2.766,3.753]$	$3.211\ [2.739,3.700]$	$-0.028\ [-0.073,0.010]$
masking + transport	$0.016\ [-0.001,0.034]$	$-0.015\ [-0.040,0.011]$	$-0.031\ [-0.054,-0.009]$
checkpoint replay	$0\ [0,0]$	$0\ [0,0]$	$0\ [0,0]$
prompt forgetting	$3.159\ [2.668,3.680]$	$3.140\ [2.653,3.632]$	$-0.019\ [-0.073,0.028]$

Appendix

TABLE VIII: Kimi target lift at two precisions using the same source weights.

outcome	count	interval, %
new answer exact after amendment	12/12	75.8–100
old answer exact after amendment	0/12	0–24.2
all retained answers exact after amendment	11/12	64.6–98.5
all retained targets contained after amendment	12/12	75.8–100
all sequence transitions match fresh state/logits	6/6	61.0–100
both deleted answers absent in both orders	6/6	61.0–100

Appendix

TABLE IX: Record correction and sequential-deletion outcomes.

surface	attempted	qualified	audited unit
Kimi short tables	8 records	8	final logits and 80 KDA arrays per record
Kimi 128-record context	9 records	9	same exactness surface plus nine position timings
Kimi long notes	5 records	4 a	same exactness surface
Kimi fixed-position masking vs. replay	12 records (3 per corpus, 4 corpora)	11 b	lift after masking and after replay; replay residual on logits and KDA state
Kimi suffix sweep, synthetic	24 configurations	24	20 layers at 10 suffix lengths; probes and 100 sampled draws per condition
Kimi suffix sweep, TOFU	16 configurations	16	same surface

Appendix

TABLE X: Attempted and qualified cases in the mechanism audits.

	Synthetic: 24 configurations				TOFU: 16 configurations
condition	lift	rank	greedy %	KL	lift	rank	greedy %	KL
record present	2.16	1	100	4.55	1.34	1	25	0.549
static receipt	2.15	1	100	4.64	1.31	1	25	0.457
diagonal ledger	2.16	1	100	4.52	1.33	1	25	0.563
full-matrix receipt	2.15	1	100	4.62	1.34	1	25	0.536
attention masked	0.0622	89	0	0.0354	0.0472	1	0	0.00419

Appendix

TABLE XI: Target recovery after 4,096 suffix tokens.

	Synthetic: 2,400 draws		TOFU: 1,600 draws
condition	0 tokens	4,096	0 tokens	4,096
record present	2349	2287	358	325
static receipt	2314	2278	245	326
attention masked	1	1	2	11
masked + static receipt	1	0	1	10
never-stored state	1	0	0	12

Appendix

TABLE XII: Target hits in sampled answers.

corpus	position	present lift	lift after mask	replay residual	suffix tokens
TOFU	early	$0.78$	$-0.08$	$0$	$762$
TOFU	middle	$1.51$	$0.11$	$0$	$399$
TOFU	late	$1.50$	$0.19$	$0$	$63$
MIMIC-CDS	early	$0.77$	$0.02$	$0$	$366$
MIMIC-CDS	middle	$1.97$	$0.24$	$0$	$176$
MIMIC-CDS	late	$0.61$	$0.19$	$0$	$26$

Appendix

TABLE XIII: Attention masking and replay at three record positions.

correction	population	result
checkpoint replay	21 qualified records	logits and 80 arrays: residual 0
full-matrix receipt	36 layer–suffix checks	forcing norm ratio 0.064–1.010

Appendix

TABLE XIV: Replay and receipt checks: cohorts and outcomes.

example	intervention	greedy answer
synthetic ward code	recurrent matrices zeroed	“OBSIDIAN-TWO”
synthetic ward code	checkpoint replay	“The ward code for patient 5182 is not provided.”
TOFU biography	attention rows masked	“Hsiao Yun-Hwa identifies as a woman.”
TOFU biography	checkpoint replay	“Hsiao Yun-Hwa identifies as a woman.”
TOFU biography	never stored	“Hsiao Yun-Hwa identifies as a woman.”

Appendix

TABLE XV: Greedy answers from synthetic and TOFU examples.

position	present lift	after attention mask	replay residual	replay seconds
early	$2.86$	$-0.23$	$0$	$1.2$
middle	$1.80$	$0.30$	$0$	$0.6$
late	$3.23$	$0.09$	$0$	$0.1$

Appendix

TABLE XVI: Masking and replay at three synthetic record positions.

state	operation	greedy answer
original	ingest old statement	“Breast cancer.”
masked	mask the statement’s attention rows	“The patient did not mention anything about their mother’s health condition.”
amended	restore, replace, replay	“Benign breast lump”
reference	corrected from the start	“Benign breast lump”

Appendix

TABLE XVII: Correcting an invented patient statement.

	synthetic		MIMIC-CDS		MIMIC-Note
write rule	raw	ledger	raw	ledger	raw	ledger
additive	$\leq 3.1{\times}10^{-6}$	—	$\leq 2.1{\times}10^{-6}$	—	$\leq 1.4{\times}10^{-5}$	—
per-channel decay	0.007–0.066	$\leq 7.0{\times}10^{-6}$	0.008–0.078	$\leq 6.4{\times}10^{-6}$	0.015–0.109	$\leq 7.0{\times}10^{-5}$
KDA delta rule	0.151–0.419	0.139–0.418	0.116–0.300	0.089–0.300	0.122–0.489	0.081–0.494

Appendix

TABLE XVIII: Suffix-dependent receipt movement under three write rules.

family	static	ledger	full- matrix	native ratio median
Mamba-2	$2.1$	$1.6\times 10^{-6}$	$1.6\times 10^{-6}$	$1.02$
Falcon-H1	$3.0$	$2.6\times 10^{-6}$	$2.6\times 10^{-6}$	$1.04$
RWKV-7	$0.518$	$0.191$	$1.9\times 10^{-6}$	$1.00$

Appendix

TABLE XIX: Transport checks across three model families.

LLM Unlearning

Large language model unlearning studies how to remove specified knowledge or behaviors while preserving a model’s unrelated capabilities.

See all 191 papers

Jul 30, 2026

Subtract, Transport, or Replay? Auditable Deletion from Language-Model Memory

Exact deletion from persistent language-model memory depends on whether a record's effect remains addressable after later computation. Native Kimi Delta Attention (KDA) gives a negative result for the tested receipt interface: the corpus-pooled raw recurrent contribution changes by 12-49% with the suffix and remains 8-49% after a decay-ledger correction. Native omission also changes later transition and write terms and other active caches. Frozen-input transport succeeds on its fixed-input control; the changed terms place native omission outside the tested receipt classes. Checkpoint replay supplies the evaluated recomputation path; zero residual on final logits and all 80 audited KDA arrays verifies restoration across the declared checkpoint surface. The complementary result is constructive. We retrofit support-vector memory into frozen Gemma 3 without attention transfer, low-rank recovery, distillation, adapters, or language-model parameter updates. Prefix-mass preservation and one box per prefix solve give base-matched admission at 4B with 1.85% perplexity overhead. At 1B and 4B, verified deletion agrees with its conditional retained-key refit within 1.3e-10 maximum next-token KL; behavioral attacks at 4B reach never-stored or chance baselines. Across 1B, 4B, and 12B, the 4B checkpoint uniquely combines base-matched admission with low overhead. The paper's two contributions are a negative result for native KDA's tested receipt classes and a positive training-free construction for addressable pretrained memory.

Vishwajith Ramesh

Vy Labs, Inc.

Memory-Augmented Language Models LLM Auditing Persistent Memory for Language Models Privacy Leakage in Language Models LLM Unlearning

Ask PDF

Sep 4, 2026

Forgetting Without Restarting: Execution-State Unlearning for Stateful LLM Agents

Long-running LLM agents are stateful: beyond the transcript they accrete compressed summaries, plaintext memory, pending tool plans, and, under every serving API, a KV cache. Yet today's "forget" operations delete a plaintext memory record and stop, leaving every artifact derived from the revoked information intact. We formalize execution-state unlearning: after a forget request, the agent must behave as if it had never observed the target. Modeling the runtime as a deterministic transition system, we prove that the pre-target trajectory prefix is shared with this counterfactual world for free, that the post-target suffix is irreducibly tainted without token-level attribution, and that exact unlearning requires at least

T-τ+1

recomputed transitions, where

τ

is the target's injection step. Provenance-Guided Selective Replay attains this bound as a cross-layer contract spanning prompt, compressed memory, and cache: a provenance graph locates the injection point, checkpoint restoration reduces to cropping the KV cache, and sanitized replay regenerates the counterfactual suffix. Audited with elicitation, stochastic, and string-free behavioral tests across three agent suites, nine baselines, and three model families, memory deletion leaves leakage unchanged, instruction-based forgetting collapses under elicitation (Leak@probes = 1.00), and source redaction still acts on a revoked preference in 80% of episodes, while selective replay is indistinguishable from a full reset at up to 9x fewer recomputed tokens.

Chao Yao, Yangbo Wei, Zhen Huang +5

Arizona State University, USA · Eastern Institute of Technology, Ningbo, China

LLM Unlearning LLM Agent Memory Machine Unlearning

Ask PDF

Sep 17, 2026

Correct Now, Insufficient Later: Auditing Update Sufficiency in Context Compression

A memory can answer a current query correctly while discarding distinctions required by a later update. We investigate this failure with a paired-history audit: two histories have the same current answer, receive a shared future update, and require different subsequent answers. A pilot evaluates 24 history pairs across six synthetic mechanisms, 12 memory conditions, two repeats, and two model backends. A deterministic frontier selector obtains strict reveal accuracy of 96/96 on DeepSeek and 82/96 on GLM; a structured writer obtains 62 successes with one unresolved outcome and 56/96. The configured four-outcome joint contrast has finite-sample identification intervals of [0.521, 0.542] and [0.292, 0.313], not confidence intervals. A record-level audit distinguishes retained-state adequacy, response delivery, and answer-schema compliance without changing those original scores. It finds 26 and 25 well-formed but semantically wrong structured reveal memories, while all 14 GLM frontier reveal failures contain correct values in the wrong wrapper. Tombstone removal produces 16/16 exact replay failures in the targeted mechanism. Identifier renaming then exposes a separate flaw: original frontier late-reference adequacy falls from 8/8 to 94/320 transformed instances. We provide and test a label-equivariant repair, but it preserves only 2/8 original late-reference answers: eliminating a naming shortcut does not solve unknown future relevance. These results support a scoped evaluation methodology and reproducible failure analysis, not general superiority of the repaired algorithm. Paid pilot evidence, retrospective diagnostics, and new offline tests are reported separately; no independent held-out or natural-task validation is claimed.

Guangzhe Zhang

Independent AI Researcher

Agent Memory Counterfactual Evaluation LLM Auditing Context Compression

Ask PDF

Abstract

LLM Unlearning

Explore similar work

Subtract, Transport, or Replay? Auditable Deletion from Language-Model Memory

Forgetting Without Restarting: Execution-State Unlearning for Stateful LLM Agents

Correct Now, Insufficient Later: Auditing Update Sufficiency in Context Compression