Delta-rule linear attention underlies language models such as Kimi Linear and Qwen3-Next. Chunkwise implementations expose matrix parallelism, but their backward passes compute unnecessary auxiliary adjoints. We present ReWY, an algebraic reformulation that reduces this work before optimizing accelerator execution. Differentiating the chunk correction as one triangular residual system eliminates the inverse adjoint and reduces seven local WY matrix products to three. Two complementary transformations expose further reuse: centered matrix contractions replace pairwise vector-gate processing, and scalar-gate reductions reuse key-gradient products instead of forming a key Gram matrix. These identities simplify backward computation for DeltaNet, GDN, KDA, and GDN2 while preserving the recurrence and gradients in exact arithmetic, without changing the architecture or training objective. We implement the reduced graph on GPUs and Ascend NPUs. Under matched recomputation and full-saving policies, precision-qualified six-shape comparisons with frozen FLA references yield 1.15-1.81× geometric-mean complete low-level backward speedups on NVIDIA GPUs. All four operators achieve 1.18-1.70× on H100 and H200 while retaining original FP32 working matrices. A separate public-autograd evaluation on Ascend 950DT, including gradient conversions, yields 1.09-2.89× under available matched policies. Paired ablations establish further ReWY gains after optimizing the complementary contractions. Complete-gradient checks validate the recorded operator cases; supplementary full-model backward measurements assess integration into community FLA models and a GDN2 layer port. Code and reproduction materials: https://github.com/flashserve/ReWY.
Figures & tables
Figure 1. The local backward computation graph before and after ReWY . Arrows show dependencies between computed values; forward operands appear in the node formulas. The expanded graph forms Wˉ and Aˉ (orange) and propagates the inverse adjoint through two matrix products. ReWY shares P across the same three required gradients (green), using the forward correction R . The graphs contain seven and three matrix products, respectively; addition, masking, and the alias Zˉ=P are not counted as products. Two dependency graphs compare the expanded WY backward with the residual backward. The expanded graph contains seven matrix products; the residual graph contains three and reuses the forward correction R.
Operator
Log decay gi
Erase bi
Write wi
DeltaNet
0
βi1dk
βi1dv
GDN
ηi1dk
βi1dk
βi1dv
KDA
vector
βi1dk
βi1dv
GDN2
vector
independent vector
independent vector
Table 1. Gate structures in the common recurrence. Scalar gates are broadcast across features.
Figure 2. Local WY pullback before and after ReWY . Box labels count dense matrix products: seven in the expanded graph and three in the reduced graph. The shared intra contractions and state adjoint are excluded from both counts. The expanded WY pullback uses seven matrix products and auxiliary adjoints for W and A. The residual pullback uses three products, sharing P between the Z, E, and coefficient gradients.
Figure 3. Complete-backward speedups: six-shape geometric means under matched recomputation (R) and full saving (A). The dotted line denotes parity. Only measured positive aggregate gains are drawn. The Ascend 950DT panel uses public eager backward including gradient conversions; other panels use low-level graph backward. Cohorts are not pooled. Daggers retain the historical precision limitation in Section G.2 . Exact values appear in Tables 9 , 2 , and 4 . Six panels show measured complete-backward speedups for DeltaNet, GDN, KDA, and GDN2. The top row, from left to right, shows H100, H200, and A800; the bottom row shows Ascend 950DT, Ascend 910C, and RTX 5070Ti. Policies R and A are shown where measured. Ascend 950DT uses public eager backward; the other panels use low-level graph backward. Daggers mark historical precision limitations.
Figure 4. Reduction in incremental backward peak allocation, 100(1−Mopt/Mref) , at B=1,T=8192,H=16,dk=dv=128 . Implementations and dagger markers match Figure 3 . KDA/NPU additionally requires 16 MiB of resident workspace per stream, excluded from these peaks. Grouped bars show percentage reductions in incremental backward peak memory for four operators on three devices under recomputation and full saving. Daggers retain the historical precision qualifications. Exact values are in the numerical-value appendix.
Figure 5. Component speedups. Top: standalone ReWY and companion implementations. Bottom: paired gains from adding ReWY to a fixed companion. The companion is matrix intra for KDA/GDN2 and Gram elimination for GDN. Dagger-marked GPU and Ascend 910C matrix controls retain their recorded precision limitation. Standalone results use separate runs; conditional results use directly paired measurements (Tables 10 and 11 ). Horizontal bar panels separate standalone residual and companion speedups from conditional gains obtained by adding the residual transformation to a fixed companion. Results cover KDA, GDN, and GDN2 on RTX and Ascend, with historical precision markers.
Figure 6. Component changes in incremental peak allocation at B=1,T=8192,H=16,dk=dv=128 . Only operator/device rows with nonnegative effects in all four controls are displayed. C is matrix intra for KDA/GDN2 and Gram elimination for GDN. R/A values agree at the displayed precision. Dagger-marked cells are historical GPU and Ascend 910C matrix-consumer measurements. Table 13 gives the schedules and exact values. A heat map shows component memory reductions in MiB for five operator-device pairs whose four recorded effects are nonnegative. Columns distinguish each component alone and each component after its companion. Positive values indicate savings; daggers mark historical precision limitations.
Operator
Policy
B
FB
KDA
R
1.2177
1.1057
KDA
A
1.3031
1.1171
GDN
R
1.2192
1.1098
GDN2
R
2.4301
1.6432
GDN2
A
2.8945
1.7260
DeltaNet
R
1.0878
1.0693
Table 2. Ascend 950DT combined paths: public-interface six-shape speedups. KDA/GDN2 combine ReWY and FP32 matrix intra; GDN uses residual/Gram elimination, and DeltaNet uses the unified chunk adjoint.
Component
Operator
Policy
B
FB
ReWY
KDA
R
1.0501
1.0158
ReWY
KDA
A
1.0663
1.0190
ReWY
GDN2
R
1.1631
1.0849
ReWY
GDN2
A
1.1874
1.0915
FP32 matrix intra
KDA
R
1.1417
1.0652
FP32 matrix intra
KDA
A
1.1836
1.0724
Table 3. Independent 950 controls, each paired with its own same-policy original reference. These are standalone speedups, not conditional increments.
H100
H200
Operator
R
A
R
A
DeltaNet
1.2066
1.3046
1.1776
1.2697
GDN
1.2056
1.2829
1.1819
1.2505
KDA
1.3264
1.4213
1.3065
1.3964
GDN2
1.5680
1.7033
1.5408
1.6527
Table 4. Hopper complete low-level backward speedups ( × ). Each entry is a six-shape geometric mean under matched recomputation (R) or full saving (A).
Model
T
Original B (ms)
Optimized B (ms)
B gain (%)
KDA
2048
112.547
103.068
+9.20
KDA
8192
358.533
325.021
+10.31
GDN
8192
334.440
333.498
+0.28
DeltaNet
2048
87.883
87.056
+0.95
DeltaNet
8192
277.062
274.461
+0.95
GDN2 port
2048
110.450
102.344
+7.92
Table 5. A800 full-model backward cases with positive gains, batch size 1. Times are CUDA-event B intervals. Numerical diagnostics are reported in Appendix K.1 .
Model / scope / backend
Points
H100
H200
KDA / default
8
+9.93
+8.83
GDN / published / wide64
2
+4.48
+4.44
DeltaNet / published
2
+1.89
+1.24
GDN2 port / official dims.
2
+6.51
+5.34
Table 6. Hopper full-model pure-backward acceleration (%): groups with positive aggregate gains on both devices. Native R, no layer checkpointing. GDN2 retains official dimensions in a FLA layer port.
Model
Original B (ms)
Optimized B (ms)
B gain (%)
KDA
204.359
187.414
+9.04
GDN
175.469
162.777
+7.80
DeltaNet
169.257
139.724
+21.14
GDN2 port
226.075
158.905
+42.27
Table 7. Ascend 910C full-model backward at B=1,T=2048 , synchronized eager B. These are performance-only records; full-model gradient differences exceed some original thresholds. Each percentage compares runs on the same physical device.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Quantity
Contract
State scan
FP32 accumulators and initial-state gradient; stored chunk states follow the reference format.
Shared P
RTX scalar/KDA paths store the input format after required FP32 uses; GDN/Ascend 910C retains FP32 for value/beta consumers.
Existing matrices
Eligible paths retain FP32 L/D arrays and prescribed GEMM conversions. Output casts cannot repair a lower internal operand precision.
GDN/Ascend 910C
Weighted coefficients and key contractions remain FP32. Gate scalars and both endpoints are reduced before output rounding.
Appendix
Table 8. Retained storage and accumulation contracts. Measured-version exclusions are in Appendix G.2 .
Operator
RTX 5070Ti
A800
Ascend 910C
DeltaNet
1.1469/1.2152
1.1474/1.2237
1.5806/1.7528
GDN
1.1614/1.2264
1.1732/1.2471
1.3739/1.5142
KDA
1.5560/1.8066
1.7943/2.0832
1.5464/1.6291
GDN2
1.7910/2.1458
1.9577/2.2792
2.5518/2.9959
Appendix
Table 9. Complete-backward speedups: six-shape geometric means, R/A. DeltaNet uses ReWY ; GDN adds Gram elimination; KDA/GDN2 add matrix intra. A800 KDA/GDN2 and NPU GDN2 use separate consumers; their RTX 5070Ti implementations use joint consumers. GPU GDN2, A800 KDA, and Ascend 910C KDA/GDN2 combined entries are historical, excluded from current claims (Section G.2 ).
Operator
Device
ReWY only
Companion only
KDA
RTX 5070Ti
1.0845/1.1160
1.1996/1.2600
KDA
NPU
1.1402/1.1660
1.3675/1.4179
GDN
RTX 5070Ti
1.0490/1.0852
1.0214/1.0323
GDN
NPU
1.0733/1.0785
1.0538/1.0658
GDN2
RTX 5070Ti
1.1732/1.2026
1.2851/1.3407
GDN2
NPU
1.1103/1.1217
2.0834/2.3268
Appendix
Table 10. Standalone component B speedups, R/A, from separate runs against frozen references. The companion is matrix intra for KDA/GDN2 and Gram elimination for GDN. GPU matrix-companion entries are historical.
Operator
Fixed companion
RTX 5070Ti R/A
NPU R/A
KDA
Matrix intra
1.1070/1.1366
1.1308/1.1490
GDN
Gram elimination
1.0485/1.0663
1.0708/1.0889
GDN2
Matrix intra
1.2364/1.2911
1.2248/1.2876
Appendix
Table 11. Paired gains from adding ReWY to a fixed companion. KDA/GDN2 RTX 5070Ti use separate residual/matrix consumers; KDA/NPU omits global direct-erase storage; GDN uses staged residual/Gram controls. GPU KDA/GDN2 entries are historical.
Operator
Device
Original
Optimized
DeltaNet
RTX 5070Ti
417.250/257.250
417.250/257.250
DeltaNet
A800
417.250/257.250
385.250/225.250
DeltaNet
NPU
610.508/450.505
518.383/358.380
GDN
RTX 5070Ti
418.250/258.250
417.750/257.750
GDN
A800
418.250/258.250
385.750/225.750
GDN
NPU
612.510/452.508
520.636/360.634
Appendix
Table 12. Incremental backward peak allocation, MiB, at B=1,T=8192,H=16,dk=dv=128 , R/A. Implementations match Table 9 . KDA/NPU has an additional 16 MiB resident workspace per stream. GPU GDN2 and A800 KDA entries are historical.
Operator
Device
O alone
C alone
O after C
C after O
KDA
RTX 5070Ti
2.50/2.50
130.47/130.47
0.00/0.00
127.97/127.97
KDA
NPU
51.50/51.50
0.00/0.00
51.50/51.50
0.00/0.00
GDN2
NPU
53.25/53.25
0.00/0.00
245.25/245.25
192.00/192.00
GDN
RTX 5070Ti
32.00/32.00
0.00/0.00
32.00/32.00
0.00/0.00
GDN
NPU
64.00/64.00
0.00/0.00
93.00/93.00
29.00/29.00
Appendix
Table 13. Component reductions in incremental peak allocation, MiB, at the same shape, R/A. Operator/device rows are shown when all four effects are nonnegative. O denotes ReWY ; C is matrix intra for KDA/GDN2 and Gram elimination for GDN. KDA RTX 5070Ti uses separate consumers; GDN uses staged controls. GPU KDA companion and conditional entries are historical.
Model
Layers
Hidden
MLP
H/K/V
Params (B)
KDA
24
2048
5632
16/128/128
1.391
GDN
24
2048
5632
8/256/256
1.466
DeltaNet
24
2048
5632
16/128/128
1.366
GDN2 port
18
2304
6208
16/128/128
1.450
Appendix
Table 14. Full-model dimensions. KDA uses FLA defaults; GDN/DeltaNet use published configurations. GDN2 is a FLA layer port, not the official full model.