Attending to Memory Writes Reduces Seed Variability in Recurrent Length Generalization
Authors: Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung
Organizations: College of Pharmacy, Chungnam National University, Daejeon, Republic of Korea · Department of Computer Science & Engineering, Chungnam National University, Daejeon, Republic of Korea
Recurrent memory requires learning useful updates, but attention added to assist learning can also compromise length generalization. We study bounded delta attention: causal attention over uncorrected update proposals whose bounded correction enters the recurrent update. Its implementation in TraceRelay combines additive phase recurrence with attention computed before a parallel scan. On prefix-supervised modulo-5, trained at lengths 1--40, three findings characterize its behavior at length 80. First, upper delta attention attains 70.81±8.26% accuracy across seven seeds, versus 69.21±22.42% for the local-attention recurrent baseline (mean ± sample SD): a 63.2% reduction in SD with a similar mean. Upper state attention also reduces variation, reaching 74.75±6.24%, so this benefit is not unique to update-level attention. Disabling the upper delta correction retains 70.62±8.60%. Second, blocking gradients at the delta-attention inputs changes accuracy to 65.25±12.40% in the matched seven-seed comparison, increasing SD by 50.2%. It also delays a 99% validation threshold on average among the same six seeds that reach it; the 95% threshold is reached earlier on average with input-gradient blocking across all seven seeds. Third, lower delta attention exhibits substantially less extrapolation degradation than lower state attention: 62.60% versus 19.40%, against a 71.06% baseline. Removing its correction reduces accuracy to 22.56%; its advantage over lower state attention also persists under input-gradient blocking. These observations motivate bounded update-level attention as a recurrent learning scaffold, combining reduced seed variation with less severe degradation in a placement where state attention fails.
Figures & tables
Figure 1: Two locations for a long-window attention operation within a TraceRelay layer. (a) δ attention reads raw local phase increments, adds the coordinate-wise bounded correction αtanh(Attnδ(δt−40:t)) , and then integrates the corrected increment through the stride-six recurrent phase update. The long correction uses neither accumulated phase states nor an independent memory controller. (b) The state-read comparator widens read attention after the recurrent update and state-feature construction. The diagram omits projection and normalization details given in the text. A window of 40 includes 40 past positions and the current position; the correction branch is enabled only in the designated layers.
Component
Setting
Task and supervision
Modulo-5 inclusive prefix state; one-hot symbols; label at every position; no target feedback
Architecture
Three layers; hidden width 128; SwiGLU width 512; four attention heads; zero dropout
Training data
Fresh IID symbols; one length L∼Uniform{1,…,40} per minibatch of 64 sequences
100 warmup updates; cosine decay to 0.1 of peak; BF16; gradient norm clipped at 1
Table 1: Shared training and evaluation protocol. Main upper comparisons use seeds 42–48; supplementary placement, scale, and scaffold comparisons use seeds 42–44 for every condition, including the corresponding upper subsets.
Design
Read windows
δ windows
Parameters
Rdirect
Baseline
All 5
None
1,182,213
31
State attention
One 40; others 5
None
1,182,213
66
Single δ attention
All 5
One 40
1,247,749
71
All-layer δ attention
All 5
All 40
1,378,821
151
Table 2: Model configurations and direct receptive fields with phase-inheritance edges excluded. Every layer retains write window 5 and phase stride 6. The correction scale is positive in the listed δ fields; setting it to zero restores direct field 31 while retaining the trained parameters.
Figure 2: Matched training-seed outcomes for the seven-seed upper-layer comparison at LAST (40k updates). Each line connects models with the same training seed. The δ models use α=0.05 , and OFF sets the correction scale to zero at inference on the same weights. SG blocks the raw-update Q/K/V input gradients through correction attention during training. The upper state comparator widens the last read window to 40. At L=80 , upper state has both a higher mean and a lower sample standard deviation than upper δ ON (74.75% ± 6.24 versus 70.81% ± 8.26). Input-SG changes upper delta accuracy to 65.25% ± 12.40. OFF closely follows ON within each training condition; performance degrades substantially at L=160 .
Condition
L=40
L=80
L=160
L=320
Short-only
98.76±0.63
69.21±22.42
33.29±16.34
22.92±5.59
Upper state
99.09±1.23
74.75±6.24
29.84±14.59
23.03±5.82
Upper δ ON
99.16±0.42
70.81±8.26
30.68±15.07
21.33±3.33
Upper δ OFF
99.11±0.42
70.62±8.60
30.71±15.01
21.29±3.75
Upper δ input-SG ON
98.98±0.43
65.25±12.40
32.97±16.67
22.95±2.91
Upper δ input-SG OFF
99.01±0.47
65.37±12.20
33.01±16.76
23.02±2.75
Table 3: Upper-layer comparison over seven seeds (42–48), LAST at 40k updates. Entries are final-position accuracy (%), mean ± sample standard deviation. ON and OFF share weights; OFF disables only the δ correction at inference. Input-SG blocks the raw-update Q/K/V gradients through the correction branch. Upper state uses a longer read window and is trained separately.
Training gradients
ON: L=80
OFF: L=80
First 95%
n95
First 99%
n99
Full gradient
70.81±8.26
70.62±8.60
17,750±2,983
7/7
28,583±5,610
6/7
Input-SG
65.25±12.40
65.37±12.20
17,179±3,068
7/7
31,667±2,705
6/7
Table 4: Upper delta attention with and without gradients at its raw-update Q/K/V inputs, using seeds 42–48 and constant scale 0.05. ON/OFF entries are length-80 final-position accuracy (%), mean ± sample SD over all seven seeds at LAST 40k. OFF changes only the inference-time correction scale. First 95% and First 99% give optimizer updates to the first ordinary length-40 validation final-position accuracy at or above each threshold, checked every 250 updates on 2,000 fixed examples; these are first crossings, not sustained attainment. The 95% statistics use all seven seeds (42–48); the 99% statistics use the same six reached seeds (42–47) in both rows. Seed 48 does not reach 99% within 40k in either condition and is not imputed. n95 and n99 give the attained counts. Each training seed is counted once.
Figure 3: Lower-layer extrapolation and inference-time correction dependence at L=80 (seeds 42–44, LAST 40k). Bars, error bars, and dots show mean accuracy, sample SD, and individual seeds. The state-read configuration is near chance (20%); delta attention with α=0.05 retains substantially more accuracy. Setting the trained correction scale to zero returns accuracy near chance. Lines connect ON/OFF evaluations of the same checkpoint and seed. Recurrent carry remains active in OFF. Length 80 exceeds the direct fields of all four configurations; parameter counts and constraints differ between the state- and delta-attention designs.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Attention source, placement, and correction scale at evaluation length L=80 . State denotes a read window of 40 in one layer; each δ condition instead keeps read and write windows at 5 and adds a window-40 bounded correction to raw phase increments in the indicated layer. Bars show means, error bars sample standard deviations, and points seeds 42–44 at LAST (40k updates). All placements use the same three seeds, including the upper state and scale-0.05 subset of Table 3 . The short-only reference mean is reused across panels. L=80 exceeds the short-only, state, and single- δ direct fields (31, 66, and 71, respectively). Comparisons do not match parameter count, source representation, or correction constraint; the figure therefore does not isolate attention source as the sole causal factor. The dotted gray line is 20% chance accuracy.
Condition
α
L=40
L=80
L=160
L=320
Short-only
—
99.09±0.84
71.06±23.34
33.92±21.47
25.13±8.02
Lower state
—
98.37±0.06
19.40±0.48
20.21±1.81
19.63±2.11
Lower δ
0.05
98.31±1.24
62.60±12.92
22.72±5.83
20.15±1.75
Middle state
—
99.02±0.00
66.41±4.51
20.28±0.73
20.25±0.66
Middle δ
0.05
99.12±0.35
77.18±10.75
34.18±19.82
23.05±7.20
Upper state
—
99.45±0.72
76.76±6.87
30.76±19.29
23.93±7.53
Appendix
Table 5: Supplementary lower/middle/upper placement comparisons at LAST (40k updates). Values are final-position accuracy in percent, reported as mean ± sample standard deviation across the same seeds 42–44 in every row. Baseline and upper rows reuse the corresponding subset of Table 3 . The state and δ models are not parameter-matched; Table 2 gives their distinct interventions and direct fields.
Seed
Condition
L=40
L=80
L=160
L=320
42
Short-only
98.93
83.40
58.59
34.28
42
Upper state
98.63
82.71
53.03
32.62
42
Upper δ ON
99.22
79.88
39.65
20.51
42
Upper δ OFF
99.32
79.88
39.55
20.31
42
Upper δ input-SG ON
99.22
82.03
54.49
25.39
42
Upper δ input-SG OFF
99.22
81.84
54.49
25.39
Appendix
Table 6: Individual-seed upper-layer final-position accuracies (%), LAST checkpoints. BEST denotes the minimum ordinary length-40 validation cross-entropy within the completed 40k run.
Seed
Condition
L=40
L=80
L=160
L=320
42
Short-only
99.32
86.82
59.57
34.96
42
Upper state
98.93
83.20
55.18
32.52
42
Upper δ ON
99.32
86.13
50.78
28.61
42
Upper δ OFF
99.32
86.04
51.07
28.61
42
Upper δ input-SG ON
99.12
79.00
57.13
33.50
42
Upper δ input-SG OFF
99.22
79.39
56.84
33.79
Appendix
Table 7: Individual-seed upper-layer final-position accuracies (%), BEST checkpoints. BEST denotes the minimum ordinary length-40 validation cross-entropy within the completed 40k run.
Condition
Seed
L=40
L=80
L=160
L=320
State
42
98.34
19.73
18.36
20.21
State
43
98.34
18.85
20.31
17.29
State
44
98.44
19.63
21.97
21.39
δ , α=0.2
42
99.12
68.07
22.66
19.34
δ , α=0.2
43
99.41
46.78
18.75
23.05
δ , α=0.2
44
98.83
44.53
19.43
19.34
Appendix
Table 8: Individual-seed lower-layer comparisons at LAST 40k. Values are final-position accuracy (%).
Condition
Seed
L=40
L=80
L=160
L=320
State
42
99.02
61.23
20.31
19.73
State
43
99.02
69.53
19.53
20.02
State
44
99.02
68.46
21.00
21.00
δ , α=0.2
42
99.51
78.32
40.43
19.43
δ , α=0.2
43
95.12
51.86
16.89
21.19
δ , α=0.2
44
99.41
78.32
17.97
20.80
Appendix
Table 9: Individual-seed middle-layer comparisons at LAST 40k. Values are final-position accuracy (%).
Condition
Seed
L=40
L=80
L=160
L=320
State
42
98.63
82.71
53.03
32.62
State
43
99.71
69.24
19.04
19.92
State
44
100.00
78.32
20.21
19.24
δ , α=0.2
42
99.41
70.80
48.93
24.41
δ , α=0.2
43
100.00
86.23
21.09
17.58
δ , α=0.2
44
99.51
62.21
20.61
21.19
Appendix
Table 10: Individual-seed upper-layer comparisons at LAST 40k. Values are final-position accuracy (%). Seeds 42–44 are shown for every condition to match the lower/middle comparisons; upper state and scale 0.05 reuse a subset of Table 3 .
Figure 5: Seven-seed upper-layer results across evaluation lengths at LAST. Left: mean final-position accuracy with one sample standard deviation shaded and clipped to [0,100]. Right: the same standard deviations in percentage points. The training-length boundary is 40; all displayed longer lengths exceed the relevant direct fields. A small spread near chance is not evidence of successful recurrence. Distinct colors, line styles, and markers identify conditions; OFF uses open markers. At L=80 , upper state, upper δ , and upper input-SG have smaller sample SDs than the short-only reference. The full-gradient advantage over input-SG at length 80 does not extend uniformly to longer lengths.
Condition
LAST: L=80
BEST: L=80
LAST: L=160
BEST: L=160
Short-only
69.21±22.42
74.92±20.20
33.29±16.34
33.16±17.25
Upper state
74.75±6.24
77.51±7.43
29.84±14.59
33.31±16.30
Upper δ ON
70.81±8.26
71.76±10.96
30.68±15.07
32.55±16.33
Upper δ OFF
70.62±8.60
71.40±11.13
30.71±15.01
32.59±16.24
Upper δ input-SG ON
65.25±12.40
61.73±13.90
32.97±16.67
32.87±16.63
Upper δ input-SG OFF
65.37±12.20
62.05±13.48
33.01±16.76
32.85±16.46
Appendix
Table 11: Seven-seed checkpoint-selection sensitivity. LAST is the primary fixed-budget endpoint; BEST is selected by ordinary length-40 validation cross-entropy, not by extrapolation performance. Entries are accuracy (%), mean ± sample standard deviation. OFF is evaluated on the corresponding ON checkpoint.
Figure 6: Inference-time correction removal across placement and scale, using the same LAST weights before and after removal. Bars show means over the same seeds 42–44 in every panel; error bars show sample standard deviations. Upper scale 0.05 uses the corresponding three-seed subset of Table 3 . OFF sets α=0 but retains recurrent phase inheritance and every local read/write operation. The effect depends strongly on placement: lower-layer models can depend on the correction even at L=40 , whereas upper-layer ON and OFF results are close. This intervention diagnoses dependence on the learned correction at inference; it does not establish how training would behave without that correction.
Placement
α
ON: L=40
OFF: L=40
ON: L=80
OFF: L=80
Lower
0.2
99.12±0.29
17.15±0.59
53.12±12.99
19.17±0.15
Lower
0.1
98.37±1.72
20.12±4.75
63.05±16.36
19.82±2.71
Lower
0.05
98.31±1.24
28.06±4.47
62.60±12.92
22.56±5.70
Middle
0.2
98.01±2.51
73.44±23.55
69.50±15.28
50.68±18.15
Middle
0.1
99.06±0.56
88.05±19.51
66.47±14.80
52.99±31.16
Middle
0.05
99.12±0.35
97.69±1.58
77.18±10.75
65.76±18.14
Appendix
Table 12: Inference-time correction removal across placement and scale at LAST. Mean accuracy (%) ± sample standard deviation over seeds 42–44. OFF sets the correction scale to zero while retaining the trained parameters, recurrent inheritance, and local read/write paths. Upper scale 0.05 uses the same three seeds as the other settings, a subset of Table 3 .
Figure 7: Supplementary all-layer correction experiments (seeds 42–44, LAST 40k). A and B keep scales 0.05 and 0.02 constant in all three layers. C starts at 0.05, decays linearly from step 24k to 36k, and is already zero at LAST; its scheduled and forced-OFF evaluations therefore coincide. With nonzero correction, the direct field is 151: L=80 is within that field, while L=160 is beyond it. With correction disabled the direct field is 31. Bars, error bars, and points show means, sample standard deviations, and seeds. B reaches 74.12% at L=80 , within its direct field, and 30.53% at L=160 ; the corresponding short-only means are 71.06% and 33.92%. Increasing the number of correction layers does not uniformly improve extrapolation.
Schedule
Evaluation
L=40
L=80
L=160
L=320
A: Constant 0.05
Scheduled
98.50±0.98
49.06±11.94
21.45±3.92
19.21±0.41
A: Constant 0.05
OFF
25.72±0.89
21.00±3.38
19.82±1.06
19.27±1.27
B: Constant 0.02
Scheduled
99.32±0.61
74.12±11.87
30.53±17.11
21.78±1.57
B: Constant 0.02
OFF
66.47±21.14
50.55±24.18
25.49±6.84
21.06±1.02
C: 0.05, decay to 0
Scheduled
98.40±0.31
52.64±7.82
20.77±1.35
18.42±1.66
C: 0.05, decay to 0
OFF
98.40±0.31
52.64±7.82
20.77±1.35
18.42±1.66
Appendix
Table 13: Supplementary all-layer experiments at LAST 40k, seeds 42–44. Every layer has a window-40 correction branch. C decays linearly between steps 24k and 36k and has zero correction at LAST. Its scheduled and OFF rows are therefore identical. Entries are mean accuracy (%) ± sample standard deviation.
Seed
δ ON
SG ON
δ OFF
SG OFF
δ : 95%
SG: 95%
δ : 99%
SG: 99%
42
79.88
82.03
79.88
81.84
17,750
15,500
29,500
33,000
43
68.46
66.11
68.26
66.41
13,250
21,250
21,250
32,750
44
65.72
57.91
65.14
58.30
19,250
18,000
28,500
33,250
45
77.83
67.87
78.12
68.26
14,750
12,000
24,500
26,250
46
62.21
50.59
60.94
50.59
20,500
18,750
37,750
33,000
47
80.08
79.59
80.27
79.20
17,250
15,500
30,000
31,750
Appendix
Table 14: Matched seven-seed upper input-gradient intervention. Accuracies (%) are ordinary length-80 final-position scores at LAST 40k; ON/OFF share each trained checkpoint. The last four columns give optimizer updates to the first ordinary length-40 validation final-position accuracy of at least 95% or 99%, checked every 250 updates on 2,000 fixed examples. All seven seeds reach 95% in both conditions; seed 48 does not reach 99% in either condition. These are first crossings, not sustained attainment. A Not reached entry denotes nonattainment within the common budget. Input-SG blocks the helper input gradients while retaining trainable attention projections and the main recurrent route.
Condition
ON: L=40
ON: L=80
OFF: L=80
Upper fixed
99.35±0.50
71.35±7.51
71.09±7.77
Upper input-SG
98.96±0.63
68.68±12.26
68.85±11.96
Upper decay
99.41±0.49
73.93±9.72
73.93±9.72
Lower input-SG
99.02±0.43
66.28±14.05
29.88±15.23
Lower fixed
98.31±1.24
62.60±12.92
22.56±5.70
Appendix
Table 15: Supplementary training-path ablations using the same seeds 42–44 in every row, LAST 40k. Entries are final-position accuracy (%), mean ± sample SD. Input-SG stops gradients at all raw-update Q/K/V inputs of correction attention; the correction and its parameters remain active. Fixed uses scale 0.05. Upper decay reaches zero at 36k, so its ON and OFF endpoints coincide. Upper fixed/input-SG reuse the corresponding three-seed subset of Table 4 . Upper fixed is an instrumented repeat and lower fixed reuses completed runs; neither adds independent seeds.
Figure 8: Length sensitivity of the scaffold ablations at LAST (40k), using the same seeds 42–44 for every curve. Lines show means; bands show one sample SD clipped to [0,100]. The short-only, lower-state, and lower-fixed controls reuse completed runs. Upper fixed/input-SG use the corresponding three-seed subset of Table 4 . All settings degrade substantially beyond length 80. Upper decay has zero correction at evaluation; the other delta curves use their trained scale of 0.05. No configuration is selected using these extrapolation scores.
Condition
Seed
ON
OFF
First 99% (step)
Upper fixed
42
79.88
79.88
29,500
Upper fixed
43
68.46
68.26
21,250
Upper fixed
44
65.72
65.14
28,500
Upper input-SG
42
82.03
81.84
33,000
Upper input-SG
43
66.11
66.41
32,750
Upper input-SG
44
57.91
58.30
33,250
Appendix
Table 16: Individual-seed scaffold results for the same seeds 42–44 in every condition. Accuracy (%) uses the fixed LAST test set at L=80 . The last column is the first ordinary L=40 validation final-position accuracy of at least 99%, checked every 250 updates on 2,000 examples. It is a first crossing, not sustained attainment; “Not reached” refers to this validation threshold within 40k, not failure on every evaluation.
Condition
LAST: L=160
LAST: L=320
BEST: L=80
BEST: L=160
Upper fixed
25.62±12.19
19.37±1.17
73.73±13.85
30.47±17.64
Upper input-SG
32.32±19.27
22.69±2.52
60.09±17.11
33.43±20.58
Upper decay
29.82±17.06
20.51±4.32
73.96±10.80
30.76±18.09
Lower input-SG
30.31±16.82
20.31±2.76
66.50±14.58
28.78±14.96
Appendix
Table 17: Length and checkpoint sensitivity for the scaffold ablations, using the same seeds 42–44 in every row. Entries are final-position accuracy (%), mean ± sample SD. BEST is selected solely by ordinary length-40 validation cross-entropy. Upper decay is already at zero scale at every selected BEST checkpoint.
Figure 9: Learning diagnostics on 512 fixed length-40 validation examples at 24 checkpoints. Bold lines average seeds 42–44; thin lines show individual seeds. (a) ON aligned cross-entropy. (b) Actual correction RMS, including the scheduled scale. (c) The per-seed absolute ON–OFF cross-entropy difference, averaged after taking the absolute value; the vertical scale is symmetric-log with a linear region below 10−5 . (d) Lower input-SG remains strongly dependent on its forward correction. Shading marks the upper decay interval (24k–36k). Fixed-scale upper corrections remain nonzero while their effect on loss becomes small. Diagnostics do not select checkpoints or control training.
We present TraceRelay, an attention-aligned recurrent architecture that distributes persistent representations over a rolling sequence of low-dimensional traces. Local right looking attention forms increments from lower-layer representations; delivery is delayed until all attended inputs are in the causal past. A fixed additive phase recurrence accumulates the delayed increments, and left-looking attention reads the resulting residual augmented stream. A stride-wise prefix sum supports parallel prefill and bounded-buffer continuation. We study 36 small-model runs on Equal Repeats, bounded Dyck closing-type prediction, and causal Most-Freq generation, using three seeds per setting. At trained length 256, Equal Repeats models with recurrent phase inheritance reach 98.81-99.69% accuracy versus 50.73-51.63% for separately trained variants without inheritance, despite the latter receiving more updates. Accuracy drops sharply at lengths beyond the training range. At the longest evaluated lengths, models with more dimensions in the middle layer's recurrent traces perform better on Dyck (76.34% versus 55.49% close accuracy at length 4096), whereas models with fewer trace dimensions perform better on five-symbol Most-Freq (70.74% versus 55.60% exact generation at length 1024). These contrasting cases motivate further study of how the size of recurrent representations should be chosen for different tasks, without establishing a general rule across tasks or model configurations.
Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung
College of Pharmacy, Chungnam National University · Department of Computer Science & Engineering, Chungnam National University
Delta-rule linear attention improves recurrent memory updates by correcting what is already stored at the current write address before writing new content. However, the active correction is still anchored to that same write address. As a result, stale information stored at a different address cannot be actively removed before new content is written elsewhere. We propose Erase-then-Delta Attention (EDA), a memory update rule that decouples where to erase from where to write. The key insight is that recurrent memory models should not only correct the current write, but also selectively suppress outdated memory at an independently chosen address. Concretely, our method first applies a targeted erase step along a learned erase direction, and then performs the standard delta-style corrective write along the current write direction. This preserves the corrective behavior of delta-rule updates while expanding their memory-management capacity. Language-model pretraining experiments across dense 2.5B and MoE 25B-A2.8B model families show that EDA performs best in both settings. The gain persists after 80B-token long-context midtraining of the MoE models, where EDA also performs best in long-context evaluations from 4k to 128k contexts. A compact update analysis and memory-state probes suggest why: EDA keeps the delta-rule corrective write intact while allocating an additional cleanup path most strongly when passive decay is weak. These results suggest that recurrent memory models should decide not only what to write, but also what stale information to erase and where.
Xiao Li, Chengruidong Zhang, Hao Luo +15
1Qwen Team · 2Nanjing University · 3Zhejiang University
Recurrent fast-weight memories and selective state-space models compress an expanding context into a fixed-size recurrent state, making the state transition an online learning rule. We study this rule under read-after-write autoregressive semantics. For the prefix-prediction objective considered here, the local fast-memory example revealed at step t is the prefix-aligned pair (xt,yt)=(φ(kt−1),vt). The common same-step association (φ(kt),vt) remains causal, but optimizes a different internal objective. We derive normalized first-order updates for squared-error regression and negative inner-product objectives. The regression family comprises Falcon-1 (a scalar NLMS update), Falcon-2 (its per-column extension), and Falcon-3 (a sliding-window mini-batch update); Falcon-1A/Falcon-2A/Falcon-3A are the corresponding inner-product variants. We provide recurrent, masked-parallel, and chunk-parallel forms, together with numerically stable positive-decay renormalization. Representative variants remain competitive in language modeling and improve length extrapolation on variable-digit addition. This framework separates temporal alignment, plasticity, forgetting, and bounded rehearsal in recurrent sequence models.
Yifan Zhang, Steve Ta, Jasper Zhang +8
Bytedance Seed · Princeton University · Hyperbolic Labs +1