Where Should a Gate Look? Input vs. State Gating in Diagonal State Space Models
Authors: Riccardo Zattra, Giacomo Baggio, Leonardo De Marchi, Augusto Ferrante, Francesco Ticozzi, Marco Del Prà, Antonio Orvieto
Organizations: Department of Information Engineering University of Padova, Italy · Independent Researcher · Ellis Institute - Max Planck Institute for Intelligent Systems - Tübingen AI center
Modern state-space models owe much of their computational efficiency to their linear, often diagonal, dynamics. Classical recurrent neural networks instead allow nonlinear state-to-state interactions, which can be beneficial when memory updates need to depend on the past context. In this paper, we ask whether state feedback through a simple gate regulating memory updates can enhance the expressivity of diagonal state-space models. To investigate this question, we compare gates that depend on the current input, the recurrent state, or both. We establish a strict expressivity separation: for scalar models, state gating solves induction-head tasks for arbitrary vocabulary size and sequence length, whereas input gating fails on a finite instance under standard stability assumptions on the state transition matrix. Experiments show that state gating retains its advantage on induction-head tasks in higher-dimensional models, whereas input gating performs better on selective copying tasks. Together, our results show that even a simple, diagonal form of state-feedback gating can unlock capabilities unavailable to input-controlled models, while input and state gating offer complementary strengths and achieve the strongest overall performance when combined.
Figures & tables
Figure 1: Block diagrams of the input-gated (left) and state-gated (right) diagonal SSMs considered in this work. The input-gated model uses a feedforward connection from the input to compute the gate Δ(k) , whereas the state-gated model uses feedback from the previous state.
Figure 2: Illustration of the induction-head task. The trigger token (red) appears twice, and the model must recall the target token (green), which immediately follows the first trigger, when the trigger appears again. Other colors represent tokens distinct from both the trigger and the target.
Table 1: Induction-head results (mean ± standard deviation over three seeds) at fixed state dimension n=8 . IG, SG, and DG form the main gating comparison; S6 is included as an established input-selective baseline.
without bias
with bias
M
Model
SSM par.
lr
Test Accuracy
SSM par.
lr
Test Accuracy
2
IG
1,536
3⋅10−3
100.0±0.0 %
2,048
3⋅10−3
100.0±0.0 %
SG
1,536
3⋅10−3
95.9±5.3 %
2,048
3⋅10−3
99.7±0.2 %
DG
2,048
3⋅10−3
100.0±0.0 %
2,560
3⋅10−3
100.0±0.0 %
S6 ( M=2 ): SSM par. 14,784, lr 3⋅10−3 , Test Accuracy 100.0±0.0 %
5
IG
1,536
3⋅10−3
26.7±19.2 %
2,048
3⋅10−3
76.5±22.3 %
Table 2: Selective copying across target lengths M (with A=10 , Lnoise=100 ), mean ± standard deviation over three seeds. IG, SG, and DG form the main gating comparison; S6 is included as an established input-selective baseline.
without bias
with bias
Model
n
SSM par.
lr
Test Accuracy
SSM par.
lr
Test Accuracy
IG
2
672
10−2
95.2±1.6 %
896
10−2
96.2±0.6 %
SG
2
672
10−2
95.2±1.2 %
896
10−2
95.8±0.8 %
DG
2
896
10−2
95.6±0.8 %
1,120
10−2
96.1±1.1 %
S6 ( n=2 ): SSM par. 11,312 , lr 10−2 , Test Accuracy 14.3±1.8 %
Table 3: MNIST results. Results are reported as mean ± std over three seeds. All models (IG, SG, DG, and S6) are evaluated using the same architecture. See Appendix C.5 for details.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Shape
Present in
Initialization
wu(i)
n
IG, DG
N(0,1) , entrywise
wx(i)
n
SG, DG
N(0,1) , entrywise
b(i)
n
IG, SG, DG
0
Appendix
Table 4: Initialization of the gate parameters of the matched family.
Figure 3: Block diagram of the core module architecture used for induction head and selective copying, showing the main processing steps from input embeddings to output predictions.
Figure 4: MNIST classification architecture. Each image is processed by four SSM layers operating on rows and columns in both forward and reverse directions. Their final outputs are concatenated and passed through two linear layers with a GELU activation to predict the digit class.
without bias
with bias
From scratch, 50 epochs
83.6±5.0 %
32.7±38.6 %
Warm-started, 50 further epochs
97.2±2.4 %
99.4±0.5 %
Appendix
Table 5: Induction-head test accuracy of SG at lr=3⋅10−3 , trained from a random initialization and warm-started from an SG model trained without a bias at the same learning rate. The first row repeats the corresponding entries of Table 1 .