cs.LGNov 3, 2026

Attending to Memory Writes Reduces Seed Variability in Recurrent Length Generalization

Authors: Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung

Organizations: College of Pharmacy, Chungnam National University, Daejeon, Republic of Korea · Department of Computer Science & Engineering, Chungnam National University, Daejeon, Republic of Korea

Abstract

Recurrent memory requires learning useful updates, but attention added to assist learning can also compromise length generalization. We study bounded delta attention: causal attention over uncorrected update proposals whose bounded correction enters the recurrent update. Its implementation in TraceRelay combines additive phase recurrence with attention computed before a parallel scan. On prefix-supervised modulo-5, trained at lengths 1--40, three findings characterize its behavior at length 80. First, upper delta attention attains 70.81±8.2670.81\pm8.26% accuracy across seven seeds, versus 69.21±22.4269.21\pm22.42% for the local-attention recurrent baseline (mean ±\pm sample SD): a 63.2% reduction in SD with a similar mean. Upper state attention also reduces variation, reaching 74.75±6.2474.75\pm6.24%, so this benefit is not unique to update-level attention. Disabling the upper delta correction retains 70.62±8.6070.62\pm8.60%. Second, blocking gradients at the delta-attention inputs changes accuracy to 65.25±12.4065.25\pm12.40% in the matched seven-seed comparison, increasing SD by 50.2%. It also delays a 99% validation threshold on average among the same six seeds that reach it; the 95% threshold is reached earlier on average with input-gradient blocking across all seven seeds. Third, lower delta attention exhibits substantially less extrapolation degradation than lower state attention: 62.60% versus 19.40%, against a 71.06% baseline. Removing its correction reduces accuracy to 22.56%; its advantage over lower state attention also persists under input-gradient blocking. These observations motivate bounded update-level attention as a recurrent learning scaffold, combining reduced seed variation with less severe degradation in a placement where state attention fails.

Figures & tables

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. TraceRelay: Attention-Aligned Recurrence over Rolling Traces

    Oct 8, 2026Sungwoo Goo, Hwi-yeol Yun, Sangkeun JungSequence ModelingPersistent Memory for Language Models

  2. Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attention

    Jun 25, 2026Xiao Li, Chengruidong Zhang, Hao Luo +15Memory-Augmented Language ModelsLong-Context Language Modeling

  3. Fast Weight Attention for Continual Learning

    Aug 27, 2026Yifan Zhang, Steve Ta, Jasper Zhang +8Recurrent Language ModelsMemory-Augmented Language Models