Graph Signal Processing Meets Mamba2: Adaptive Filter Bank via Delta Modulation

Yehjin Shin; Seojin Kim; Noseong Park

cs.LGMar 20, 2026

Graph Signal Processing Meets Mamba2: Adaptive Filter Bank via Delta Modulation

Authors: Yehjin Shin, Seojin Kim, Noseong Park

Organizations: KAIST

Abstract

State-space models (SSMs) offer efficient alternatives to attention with linear-time recurrence. Mamba2, a recent SSM-based language model, uses selective input gating and a multi-head structure, enabling parallel computation and strong benchmark performance. However, its multi-head recurrence operates independently without structured utilization or analysis. In this work, we propose a novel method called Hierarchical ADaptive filter bank for Efficient SSMs (HADES), a Graph Signal Processing (GSP)-inspired framework that reinterprets Mamba2 as an adaptive filter bank on a line graph. Our hierarchical architecture introduces two filter types: shared filters for global low-pass behavior and expert filters for local high-pass behavior, achieved through structured bias on the parameter Δ. HADES achieves comparable performance to baseline models including Mamba2 across various benchmarks in language modeling, commonsense reasoning, and long-context retrieval, while using only 58.9% of the original parameters. In this regard, HADES bridges GSP and neural sequence modeling, enabling efficient, hierarchical, and interpretable filtering within state-space models.

Figures & tables

Figure 1: Distribution of layer-wise Effective Rank from the spectral responses of Mamba2 and HADES

Panel of Figure 2 — Figure 2: Architectural Comparison between Mamba2 and HADES . Mamba2 applies all filters uniformly to every input token, whereas HADES employs a routing mechanism that selects and activates filters conditioned on the spectral residual $r_{t}$ and $\Delta_{t}$ .

Model	Train.	Wiki.	LMB.	LMB.	BoolQ	Hella.	Wino.	ARC-e	ARC-c	PIQA	OBQA.	Avg.
	ppl $\downarrow$	ppl $\downarrow$	ppl $\downarrow$	acc $\uparrow$	acc $\uparrow$	acc_n $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc_n $\uparrow$	8 tasks $\uparrow$
Linear Transformer	2.49	45.43	73.93	24.06	61.50	28.20	51.30	42.05	21.76	60.55	27.60	39.63
RetNet	2.41	34.12	29.46	35.36	55.57	31.31	51.70	44.49	2 3.46	62.40	28.00	41.54
DeltaNet	2.29	33.25	26.82	35.75	54.07	31.40	49.96	44.11	22.18	63.60	29.60	41.33
Mamba1	2.53	47.51	85.53	22.43	62.17	28.71	50.67	42.09	22.35	60.72	26.60	39.47
Mamba2	2.33	31.34	2 4.38	3 6.46	53.88	3 2.62	50.83	45.29	24.15	63.44	26.40	4 1.63

Table 1: Performance comparison on language modeling and zero-shot common-sense reasoning. The best results are highlighted in bold , while the second-best results are u nderlined. Avg. denotes the average of accuracies and normalized accuracies over 8 tasks. With only 58.9% of parameters compared to baseline models, HADES achieves comparable or even better performance.

Panel of Figure 3 — Figure 3: Passkey retrieval result of Mamba2 and HADES

Figure 4: Expert filter selection in Passkey Retrieval task

Panel of Figure 5 — Figure 5: Spectrum of filter inputs and outputs from Mamba2 and HADES . The x-axis represents the Fourier frequency bins, and the y-axis shows the normalized magnitude of the Fourier coefficients, with larger values indicating stronger frequency components (see Appendix E.4 for details).

Figure 6: Filter Spectral Responses of Mamba2 and HADES (see Appendix E.5 for details).

Panel of Figure 7 — Figure 7: (a-b) Comparison on frequency spectrum of filter outputs from expert filter with bias and expert filter without spectral bias. The x-axis represents the Fourier frequency bins, and the y-axis shows the normalized magnitude of the Fourier coefficients, with larger values indicating stronger frequency components (see Appendix E.4 for details). (c) Histogram of spectral bias.

Methods	Wiki.	LMB.	Avg.
	ppl $\downarrow$	ppl $\downarrow$	8 tasks $\uparrow$
HADES (Ours)	31.51	21.74	42.91
w/o $\mathcal{L}_{\text{balance}}$	34.73	26.84	41.57
w/o $\mathcal{L}_{\text{diversity}}$	33.83	27.40	42.15
Only Shared Filters	34.55	27.64	42.21
Only Expert Filters	36.34	30.12	41.68

Table 2: Ablation Studies

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Wiki.	LMB.	LMB.	BoolQ	Hella.	Wino.	ARC-e	ARC-c	PIQA	OBQA.	Avg.
Model	ppl $\downarrow$	ppl $\downarrow$	acc $\uparrow$	acc $\uparrow$	acc_n $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc_n $\uparrow$	8 tasks $\uparrow$
Linear Transformer	45.43	73.93	24.06	61.50	28.20	51.30	42.05	21.76	60.55	27.60	39.63
Linear Transformer	(N/A)	(4.168)	(0.006)	(0.009)	(0.005)	(0.014)	(0.010)	(0.012)	(0.012)	(0.020)	(0.011)
RetNet	34.12	29.46	35.36	55.57	31.31	51.70	44.49	2 3.46	62.40	28.00	41.54
RetNet	(N/A)	(1.046)	(0.007)	(0.009)	(0.005)	(0.014)	(0.010)	(0.012)	(0.011)	(0.020)	(0.011)
DeltaNet	33.25	26.82	35.75	54.07	31.40	49.96	44.11	22.18	63.60	29.60	41.33

Appendix

Table 3: Performance comparison on language modeling and zero-shot common-sense reasoning with standard error (values in

(\cdot)

). The standard error values are are rounded to three decimal places. The best results are highlighted in bold , while the second-best results are u nderlined. Avg. denotes the result of accuracies and normalized accuracies over 8 tasks.

Model	Wiki.	LMB.	LMB.	PIQA	Hella.	Wino.	ARC-e	ARC-c	BoolQ	OBQA.	Avg.
	ppl $\downarrow$	ppl $\downarrow$	acc $\uparrow$	acc $\uparrow$	acc_n $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc_n $\uparrow$	8 tasks $\uparrow$
RetNet (1.3B)	22.45	21.84	38.70	69.04	47.73	52.72	63.68	33.36	60.61	36.60	50.31
Mamba2 (1.3B)	19.47	17.40	40.68	70.29	53.24	56.04	69.87	36.35	55.81	37.40	52.46
DeltaNet (1.3B)	1 9.77	16.64	41.78	70.95	51.09	54.70	67.63	34.47	61.19	38.40	52.53
HADES (1B)	20.41	1 7.22	4 1.18	71.33	5 1.85	56.35	6 8.48	3 4.81	6 0.73	38.60	52.92

Appendix

Table 4: Performance comparison on language modeling and zero-shot common-sense reasoning. The best results are highlighted in bold , while the second-best results are u nderlined. Avg. denotes the average of accuracies and normalized accuracies over 8 tasks. With only 71.4% of parameters compared to baseline models, HADES achieves comparable or even better performance.

Model	Wiki.	LMB.	LMB.	PIQA	Hella.	Wino.	ARC-e	ARC-c	BoolQ	OBQA.	Avg.
	ppl $\downarrow$	ppl $\downarrow$	acc $\uparrow$	acc $\uparrow$	acc_n $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc_n $\uparrow$	8 tasks $\uparrow$
MoM	31.58	23.28	40.40	62.57	32.99	52.64	44.91	23.89	52.78	27.20	42.17
HADES	31.48	21.74	39.24	63.93	32.82	52.64	45.03	22.01	58.84	28.80	42.91

Appendix

Table 5: Performance comparison on language modeling and zero-shot common-sense reasoning. The best results are highlighted in bold , while the second-best results are u nderlined. Avg. denotes the average of accuracies and normalized accuracies over 8 tasks.

Model	Wiki.	LMB.	LMB.	BoolQ	Hella.	Wino.	ARC-e	ARC-c	PIQA	OBQA.	Avg.
	ppl $\downarrow$	ppl $\downarrow$	acc $\uparrow$	acc $\uparrow$	acc_n $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc_n $\uparrow$	8 tasks $\uparrow$
Mamba2	30.64	22.57	37.85	54.63	33.27	51.15	45.34	23.29	63.04	27.93	42.06
Mamba2	(0.752)	(1.662)	(0.014)	(0.007)	(0.007)	(0.006)	(0.004)	(0.009)	(0.004)	(0.014)	(0.004)
HADES	33.41	25.57	37.10	59.67	31.70	51.85	44.84	22.67	63.15	27.87	42.37
HADES	(1.721)	(3.482)	(0.022)	(0.009)	(0.010)	(0.011)	(0.008)	(0.006)	(0.007)	(0.013)	(0.005)

Appendix

Table 6: Performance comparison on language modeling and zero-shot common-sense reasoning with mean and standard error over three seeds (values in (·)). The standard error values are are rounded to three decimal places. The best results are highlighted in bold , while the second-best results are u nderlined. Avg. denotes the result of accuracies and normalized accuracies over 8 tasks.

Methods	Wiki.	LMB.	LMB.	PIQA	Hella.	Wino.	ARC-e	ARC-c	BoolQ	OBQA.	Avg.
Methods	ppl $\downarrow$	ppl $\downarrow$	acc $\uparrow$	acc $\uparrow$	acc_n $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc_n $\uparrow$	8 tasks $\uparrow$
HADES (Ours)	31.48	21.74	39.24	63.93	32.82	52.64	45.03	22.01	58.84	28.80	42.91
w/o $\mathcal{L}_{\text{balance}}$	34.73	26.84	36.77	62.68	30.96	50.75	43.22	22.70	59.27	26.20	41.57
w/o $\mathcal{L}_{\text{diversity}}$	33.83	27.40	36.04	62.46	31.48	51.38	44.87	22.61	59.94	28.40	42.15
Only Shared Filters	34.55	27.64	35.75	62.40	31.39	52.88	44.23	24.23	60.83	26.00	42.21
Only Expert Filters	36.34	30.12	34.89	61.53	30.30	52.41	44.49	22.70	58.29	28.80	41.68

Appendix

Table 7: Full result for ablation studies. Avg. denotes the averaged result of the accuracies and normalized accuracies of eight tasks together.

Hyper		Wiki.	LMB.	LMB.	PIQA	Hella.	Wino.	ARC-e	ARC-c	BoolQ	OBQA.	Avg.
param.		ppl $\downarrow$	ppl $\downarrow$	acc $\uparrow$	acc $\uparrow$	acc_n $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc_n $\uparrow$	8 tasks $\uparrow$
$H$	8	39.52	37.58	32.93	60.88	29.63	50.28	42.00	22.27	61.10	24.60	40.46
	16	31.48	21.74	39.24	63.93	32.82	52.64	45.03	22.01	58.84	28.80	42.91
	24	33.11	26.73	35.40	62.57	31.99	50.75	44.36	23.29	48.93	28.60	40.74
$\gamma$	0.15	34.81	29.07	34.78	61.86	31.33	51.22	43.64	23.04	60.34	26.60	41.60
	0.25	31.48	21.74	39.24	63.93	32.82	52.64	45.03	22.01	58.84	28.80	42.91

Appendix

Table 8: Sensitivity Studies. Avg. denotes the averaged result of the accuracies and normalized accuracies of eight tasks together.

Panel of Figure 8 — Figure 8: CKA analysis on filter outputs of Mamba2 and HADES . Left: Mamba2. Center: HADES . Right: comparison of distribution.

Panel of Figure 9 — Figure 9: Frequency spectrum analysis of filter outputs. Left: Shared filter outputs. Center: Expert filters with spectral bias outputs. Right: Expert filters without spectral bias outputs.

Panel of Figure 10 — Figure 10: Frequency responses of Mamba2 and HADES .

Model	Linear Transformer	RetNet	DeltaNet	Mamba	Mamba2	HADES
# Params.	370M	370M	370M	370M	370M	218M
Inference Time (sec)	2.19	2.86	2.57	4.73	3.43	2.49
Memory (GB)	5.41	6.17	5.59	5.56	6.20	4.50

Appendix

Table 9: Inference Time and Memory Usage Comparison.

Model	Setting	Selection Ratio
Model	Setting	50%	25%	75%
370M	Prefill (1K)	0.3997	0.3374	0.4969
	Prefill (2K)	0.5936	0.4572	0.7869
	Prefill (3K)	0.6043	0.4765	0.8033
	Decode	0.0004	0.0003	0.0004
1.3B	Prefill (1K)	0.9981	0.5165	0.7742

Appendix

Table 10: Average prefill and decode time under various configuration settings. Latency is reported in seconds. We utilize half of the selected filters as shared filters and the remaining half as expert filters. The selection ratio denotes the percentage of filters chosen out of the total available filters.

Metric	Value
Avg Routing Time (ms)	0.0071
Avg Total Time (ms)	0.2900
Avg Routing Time Ratio (%)	2.4371%

Appendix

Table 11: Latency overhead of running routers (seqlen 2048, averaged over 100 step)

Operation	Prefill
In Projection	$O(Td^{2})=O(TMPd)\rightarrow O(THPd)$
1D Convolution	$O(TC_{\text{in}}C_{\text{out}}d_{\text{conv}})=O(T(MP+N)^{2}d_{\text{conv}})\rightarrow O(T(HP+N)^{2}d_{\text{conv}})$
SSM Kernel	$O(T\log T\cdot M)\rightarrow O(T\log T\cdot H)$
Out Projection	$O(Td^{2})=O(TMPd)\rightarrow O(THPd)$
RMS Norm	$O(Td)=O(TMP)\rightarrow O(THP)$
HADES Ops.	$O(T(d+M)(M+H-2S))$

Appendix

Table 12: Complexity comparison for each operation in prefill and decode stages. Arrows (

\rightarrow

) indicate improved complexity.

Component	Parameters
in_proj linear	$d\cdot(2\cdot 2d+2N+M)=d\cdot(2MP+2N+M)\;\rightarrow\;d\cdot(2HP+2N+M)$
conv1d	$(2d+N)\cdot d_{\text{conv}}=(MP+N)\cdot d_{\text{conv}}\;\rightarrow\;(HP+N)\cdot d_{\text{conv}}$
out_proj	$2d\cdot d=MP\cdot d\;\rightarrow\;HP\cdot d$
rms norm	$2d=MP\;\rightarrow\;HP$
ssm params	$3M\;\rightarrow\;3H$
Added params in HADES	$(d+M)\cdot(M+H-2S)+2$

Appendix

Table 13: Parameter complexity of each component. Arrows (

\rightarrow

) indicate reduction from

M

H

Operation	Mamba2	HADES	$\Delta$ (Reduction)
In-projection	$2d(2MP+2N+M)$	$2d(2HP+2N+M)$	$4dP(M-H)$
1D Convolution	$2(MP+N)d_{\text{conv}}$	$2(HP+N)d_{\text{conv}}$	$2(M-H)Pd_{\text{conv}}$
Out-projection	$2MPd$	$2HPd$	$2(M-H)Pd$
RMS Norm	$c_{\text{rms}}MP$	$c_{\text{rms}}HP$	$c_{\text{rms}}P(M-H)$
SSD (SSM Core)	$c_{\text{ssd}}MN\log N$	$c_{\text{ssd}}HN\log N$	$c_{\text{ssd}}(M-H)N\log N$

Appendix

Table 14: Per-token FLOPs of Mamba2 and HADES (excluding HADES selection module).

HADES Operation	FLOPs/token
Residual computation	$2d$
Selection score projection	$2(d+M)(M+H-2S)$
Top- $Q$ selection	$c_{\text{top}}E\log E$
Spectral bias	$2H$
Delta modulation	$2H$

Appendix

Table 15: Additional per-token FLOPs introduced by the HADES selection mechanism.

Panel of Figure 11 — Figure 11: Comparison of training behavior across models. Training sequence length was 2048, parameter size 370M, average time per 100 training steps for one epoch.

Panel of Figure 12 — Figure 12: (a) $\gamma$ -variation on expert filters with spectral bias outputs on 13 th layer. (b) and (c) Frequency spectrum analysis of filter outputs on 13 th layer. Left: Shared filter outputs. Center: Expert filters with spectral bias outputs. Right: Expert filters without spectral bias outputs.

Explore similar work

Harmonic: Hierarchical State Space Models for Efficient Long-Context Language Modeling
May 30, 2026Petr NyomaLong-Context Language Modeling State Space Models
Independent Researcher
mHC-SSM: Manifold-Constrained Hyper-Connections for State Space Language Models with Stream-Specialized Adapters
May 8, 2026Abdulvahap Mutlu, Şengül Doğan, Türker TuncerLanguage Modeling
Department of Digital Forensics Engineering, Technology Faculty, Firat University, Elazig, Turkey
SSM Adapters via Hankel Reduced-order Modeling: Injection Site Determines Task Suitability in Long-Context Fine-Tuning
Jun 24, 2026Omanshu ThapliyalLong-Context Language Modeling Reduced-Order Modeling
Hitachi America Ltd., Santa Clara, CA, USA.

May 30, 2026J/K move · Enter open · S save

Harmonic: Hierarchical State Space Models for Efficient Long-Context Language Modeling

Petr Nyoma

Independent Researcher

Long-Context Language Modeling State Space Models

Open paper AskPDF arXiv

We present Harmonic, a hierarchical state space model (SSM) for language modeling. The architecture stacks three recurrent levels at progressively slower timescales; each level receives the prediction error of the level below as input, rather than its raw hidden state. On enwiki8 with equal token budgets, Harmonic outperforms a comparable Transformer (28M params) by +1.4% at 1K tokens, +6.7% at 8K tokens, and +11.4% at 32K tokens (bpt, lower is better). It also outperforms Mamba at every tested length by 0.7--1.8%. At 64K tokens, both Mamba and Transformer run out of memory on an 80GB H100; Harmonic trains successfully, reaching 6.169 bpt. Results replicate on WikiText-103 (H-TF gap +1.7% to +7.2% across 1K--32K). At 1B parameter scale, replacing all attention layers in TinyLlama 1.1B with HarmonicBlock eliminates the RoPE positional encoding limit: the resulting Hallamonic model maintains stable loss across sequence lengths 1K--8K on two independent clean benchmarks (Lambada and fineweb-edu held-out), while TinyLlama degrades catastrophically past its 2K-token RoPE limit (gap: +9.4 bpt at seq=8K on Lambada). Compute is O(L) per forward pass vs. O(L^2) for attention. Logs: https://github.com/Omibranch/harmonic-logs.

Abstract

Explore similar work

Harmonic: Hierarchical State Space Models for Efficient Long-Context Language Modeling

mHC-SSM: Manifold-Constrained Hyper-Connections for State Space Language Models with Stream-Specialized Adapters

SSM Adapters via Hankel Reduced-order Modeling: Injection Site Determines Task Suitability in Long-Context Fine-Tuning