SSDi8: Accurate and Efficient 8-bit Quantization for State Space Duality

Hyunwoo Kim; Byoungchan Ko; Minseok Kang; Minwoo Kim; Dongjin Lee; Jaehoon Lee; Sungroh Yoon; Dahuin Jung

cs.AIAug 22, 2026

SSDi8: Accurate and Efficient 8-bit Quantization for State Space Duality

Authors: Hyunwoo Kim, Byoungchan Ko, Minseok Kang, Minwoo Kim, Dongjin Lee, Jaehoon Lee, Sungroh Yoon, Dahuin Jung

Organizations: Department of Artificial Intelligence, Chung-Ang University · School of Computer Science and Engineering, Soongsil University · Department of Electrical and Computer Engineering, Seoul National University · Interdisciplinary Program in Artificial Intelligence, Seoul National University ·

Abstract

Recent advances in sequence modeling have highlighted Mamba as a state space architecture offering efficient long-range dependency modeling and providing a viable alternative to Transformers. Building upon this, Mamba-2 introduces the Structured State Space Duality (SSD), which integrates recurrent and attention modes to achieve efficiency and scalability. However, this architectural expansion substantially increases memory and latency overhead, underscoring the need for efficient compression strategies tailored to SSD. In this work, we present SSDi8, the first post-training quantization framework specifically designed for SSD to maintain a persistent INT8 path. SSDi8 introduces a reformulation that decouples element-wise multiplications from matrix multiplications, enabling reuse of quantized activations across modules. Moreover, SSDi8 adaptively quantizes channel-varying activations at cost-effective points, further reducing latency. On the accuracy side, SSDi8 explicitly leverages the intrinsic dimensional decomposition of SSD, exploiting distinct outlier distributions across axes, and incorporates an error correction term based on per-channel error statistics. Comprehensive experiments demonstrate that SSDi8 achieves accuracy comparable to FP16 while delivering up to 1.4x speedup in W4A8 and W8A8 settings. We further validate its robustness in resource-constrained environments by deploying it on the Orin NX device.

Figures & tables

Model	Bitwidth	Quantized Layer(s)	ACC
2.7B	FP16	–	63.8%
	W4A8	+ In Proj	63.6%
		+ SSD	58.4%
		+ Out Proj	54.6%

Table 1: Accuracy under major layer quantization of Mamba-2. Significant degradation arises when SSD is quantized per-tensor.

Figure 1: (a) Mamba-2 block architecture. (b) SSD pipeline in SSDi8. SSDi8 enables the persistent INT8 representation path through reformulation and quantized activation reuse, while mitigating performance degradation via channel-aware quantization and mean correction.

Figure 2: Visualization of activation $X$ in the 16th block of Mamba-2 8B before and after the SSD input transformation. The pre-SSD dimension ( B,L,D ) exhibits no clear token-wise pattern, whereas the transformed dimension ( B,L,H,P ) within SSD reveals distinct patterns along the H-dim.

Figure 3: (a) Distribution plots of head-wise $X$ and $LUT_{\text{state}}$ in the 27th block of the ChunkState module, and their element-wise product after reformulation $X_{\text{scaled}}$ . The channel-wise ( P -dim) distribution of $X_{\text{scaled}}$ is highly sparse. (b) Head-wise distribution plots of State .

Model	Size	Methods	Bitwidth	LA	HS	PIQA	Arc-E	Arc-C	WG	Avg.
Mamba-2	1.3B	-	FP16	65.6%	59.9%	73.3%	64.1%	33.3%	60.8%	59.5%
		Quamba	W8A8	49.8%	58.5%	71.2%	61.9%	32.1%	58.1%	55.2%
		Quamba2	W8A8	62.0%	59.2%	72.5%	63.4%	32.7%	60.0%	58.3%
		Quamba2	W4A8	61.0%	58.8%	72.4%	62.7%	32.6%	59.1%	57.7%
		SSDi8 (Ours)	W8A8	64.7%	59.7%	72.7%	64.0%	32.8%	60.9%	59.1%
		SSDi8 (Ours)	W4A8	63.6%	59.2%	72.7%	63.5%	33.5%	60.4%	58.8%

Table 2: Evaluation of Mamba-2 (1.3B, 2.7B, 8B) with three quantization methods (Quamba, Quamba2, and SSDi8) on six zero-shot tasks (LA, HS, PIQA, Arc-E, Arc-C, WG).

Methods	Bitwidth	Wikitext2 Perplexity (↓)
Methods	Bitwidth	1.3B	2.7B	8B
-	FP16	10.42	9.06	7.25
HAD	W8A8	11.31	11.42	8.57
HAD	W4A8	11.63	11.85	8.79
Quamba2	W8A8	10.80	9.32	7.79
Quamba2	W4A8	11.08	9.54	7.94

Table 3: Wikitext2 perplexity with

L=2048

Figure 4: SSD latency of quantization methods on Mamba-2 2.7B: (a) varying batch (

L=2048

), (b) varying length (

B=32

), and (c) module-wise latency (

B=32

L=2048

). PIR denotes Persistent INT8 Representation. SSDi8 achieves up to

1.47\times

overall speedup and

1.77\times

in the State path.

Figure 5: Forward latency of W8A8 (

L=2048

) on 2.7B.

GPU	Orin NX 16G
Bitwidth	W4A8		W8A8
Method	Quamba2	SSDi8	Quamba2	SSDi8
$L=256$	55.30	44.71	51.03	41.30
$L=512$	76.10	68.00	70.95	60.49
$L=1024$	134.40	127.51	139.10	114.36
$L=2048$	262.90	240.54	249.29	217.69

Table 4: SSD latency (ms) of SSDi8 vs. Quamba2 on Orin NX 16G.

Bit- width	ChunkState Q(X)	Sparse Reform.	Quant. of B,C	Persistent INT8	Quant. of ChunkBMM	Latency	PPL
W4A8	–	–	–	–	–	8.63	9.34
	✓					8.58	9.35
	✓		✓			8.05	9.37
	✓		✓		✓	7.60	9.39
		✓				8.35	9.36
		✓	✓	✓		8.00	9.42

Table 5: Ablation results for internal SSD quantization (

Q(\text{SSD})

Bitwidth	SSDi8		Acc.
Bitwidth	$Q$ (SSD)	Correct.	Acc.
FP16	–	–	69.5%
W4A8			51.2%
	✓		67.2%
	✓	✓	67.4%

Table 6: Ablation results of SSDi8:

Q(\text{SSD})

and correction

c

Method	Wino	PiQA	ARC-C	ARC-E	Hella	Lamb	Avg.	PPL	SSD Latency (ms)	Fwd. Latency (ms)
FP16	73.8	80.9	55.8	81.4	80.6	66.2	73.1	8.42	19.834	109.873
INT8	73.5	80.7	55.9	81.5	80.6	66.3	73.0	8.65	9.156	98.904

Table 7: Hybrid Mamba–Transformer results on Nemotron-H-8B-Reasoning. SSDi8 is applied only to the SSD path, while other modules remain in FP16.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Size	Methods	Bitwidth	LA	HS	PIQA	Arc-E	Arc-C	WG	Avg.
Mamba-2	1.3B	-	FP16	65.6%	59.9%	73.3%	64.1%	33.3%	60.8%	59.5%
		HAD	W8A8	55.3%	59.4%	73.2%	64.0%	33.5%	58.2%	57.3%
		HAD	W4A8	53.9%	58.9%	72.3%	63.6%	33.9%	59.1%	57.0%
		Quamba	W8A8	49.8%	58.5%	71.2%	61.9%	32.1%	58.1%	55.2%
		Quamba2	W4A16	64.3%	59.2%	72.6%	63.8%	33.1%	60.3%	58.9%
			W8A8	62.0%	59.2%	72.5%	63.4%	32.7%	60.0%	58.3%

Appendix

Table 8: Evaluation of Mamba-2 models at 1.3B, 2.7B, and 8B scales using four quantization methods—HAD, Quamba, Quamba2, and SSDi8—across six zero-shot tasks: LA, HS, PIQA, Arc-E, Arc-C, and WG.

Model	Methods	Bitwidth	Pile Perplexity (↓)
Model	Methods	Bitwidth	1.3B	2.7B
Mamba-2	-	FP16	6.99	6.27
	HAD	W8A8	7.46	7.77
	HAD	W4A8	7.87	8.17
	Quamba2	W8A8	7.20	6.44
	Quamba2	W4A8	7.55	6.68

Appendix

Table 9: Pile perplexity with

L

= 2048

Model	Bitwidth	Activation	per-T	per-P( N )	per-H( G )	Wikitext2 Perplexity
8B	W4A8 SSD-FP16	–	–	–	–	7.42
		B,C	v			7.59
				v		7.43
					v	7.44
				v	v	7.42
		X,State	v			11.97

Appendix

Table 10: Ablation study for quantization axis.

Bitwidth	Project	Speedup	Acc.
W4A8	None	$\times 1.00$	67.2%
	In	$\times 0.945$	67.4%
	SSD	$\times 0.975$	67.5%
	Out	$\times 0.987$	67.4%

Appendix

Table 11: Accuracy and speedup for W4A8. Experiments are conducted on the LAMBADA dataset, using SSDi8 without mean correction as the baseline.

Model	Size	Method	W8A8	W4A8
Mamba2	2.7B	FP16	5.154 GB
		Quamba2	2.948GB	1.766GB
		SSDi8 (Ours)	2.953GB	1.774GB
	8B	FP16	15.710 GB
		Quamba2	9.860GB	7.028GB
		SSDi8 (Ours)	9.867GB	7.038GB

Appendix

Table 12: Memory usage comparison

Figure 6: Comparison with Quamba2 under W4A8 quantization on the 8B model is also reported. OOM denotes Out-Of-Memory.

Length	Batch	FP16			Quamba2			SSDi8
Length	Batch	SSD Lat. (ms)	Throughput (T/s)	PPL	SSD Lat. (ms)	Throughput (T/s)	PPL	SSD Lat. (ms)	Throughput (T/s)	PPL
2k	B=8	4.990	10480	9.062	5.313	11850	9.549	4.507	11981	9.470
4k	B=8	9.299	10716	13.845	8.998	12333	11.502	7.231	12818	9.021
6k	B=8	13.564	10922	113.407	13.363	12648	100.438	10.367	13038	8.943
8k	B=8	17.867	11173	444.467	15.932	13079	309.182	12.635	13418	9.040
10k	B=8	22.061	11167	1577.003	19.806	13121	1253.093	15.827	13804	9.122

Appendix

Table 13: Long-context results (B=8): SSD latency, end-to-end throughput (tokens/s), and perplexity (PPL).

Calib Size	Wino	PiQA	ARC-C	ARC-E	Hella	Lamb	Avg.
128	64.4	75.5	35.1	68.2	65.0	68.0	62.7
256	64.5	75.3	36.4	67.8	64.8	67.0	62.6
512	64.2	75.5	35.8	67.9	65.1	67.7	62.7
1024	64.3	75.7	36.4	67.6	64.8	66.3	62.5
2048	64.8	75.1	36.2	68.1	65.2	67.1	62.7

Appendix

Table 14: Sensitivity to the number of calibration samples. Accuracy (%) on six zero-shot tasks.

Calib Data	Wino	PiQA	ARC-C	ARC-E	Hella	Lamb	Avg.
C4	65.4	75.2	36.1	68.5	65.3	67.7	63.0
PTD	65.1	75.1	36.5	68.6	65.5	67.6	63.0
Pile	64.5	75.3	36.4	67.8	64.8	67.0	62.6

Appendix

Table 15: Sensitivity to the choice of calibration dataset. Accuracy (%) on six zero-shot tasks.

Method	Batch	In-Proj	Conv	SSD	Norm	Out-Proj	Correction
FP16	B=8	9.829	0.781	4.795	1.196	4.919	–
FP16	B=16	20.759	1.357	9.426	1.640	10.272	–
FP16	B=32	41.297	2.526	17.640	3.086	20.474	–
SSDi8	B=8	8.124	0.587	4.481	1.089	3.266	0.426
SSDi8	B=16	16.758	1.011	7.293	1.442	6.677	0.698
SSDi8	B=32	33.893	1.799	13.199	2.291	12.914	1.252

Appendix

Table 16: Latency breakdown (ms) of the Mamba-2 block (2.7B,

L=2048

) under different batch sizes.

Batch Size	Method	Value
Batch Size	Method	Latency (ms)	Throughput (T/s)
16	FP16	2.757	9918
16	SSDi8	2.671	10299
32	FP16	4.898	10653
32	SSDi8	4.510	11976
64	FP16	9.271	10405

Appendix

Table 17: SSD latency and throughput comparison (

L

= 2048)

Figure 7: Visualization of the distributions of activations $B$ , $C$ , and $CB$ in in the first, middle, and last block of Mamba-2 8B.

Figure 8: Visualization of the distributions of activations $X$ , $LUT_{state}$ , and $X_{scaled}$ in the last block of Mamba-2 8B.

Post-Training Quantization

Post-Training Quantization compresses trained neural networks into low-bit weights, activations, or caches to reduce memory use and inference cost.

See all 366 papers

Ternary Mamba: Grouped Quantization-Aware Training of W1.58A16 State Space Models
Jun 16, 2026Ramprasath Ganesaraja, Sahil Dilip Panse, Swathika NLLM Quantization Ternary Quantization
EdgeVerve Systems Limited
Multi-Scale Dequant: Eliminating Dequantization Bottleneck via Activation Decomposition for Efficient LLM Inference
May 13, 2026Lingchao Zheng, Yuwei Fan, Jun Li +5LLM Quantization LLM Inference Acceleration
Huawei
Channel-Wise and Token-Aware Post-Training Quantization for Visual State Space Duality
Sep 15, 2026Jonghyeon Lim, Changhoon YimVisual SSMs Activation Quantization
Intelligent Image Processing Laboratory · Department of Computer Science and Engineering, Konkuk University · Seoul, Republic of Korea

Jun 16, 2026J/K move · Enter open · S save

Ternary Mamba: Grouped Quantization-Aware Training of W1.58A16 State Space Models

Ramprasath Ganesaraja, Sahil Dilip Panse, Swathika N

EdgeVerve Systems Limited

LLM Quantization Ternary Quantization LLM Compression Quantization-Aware Training Neural Network Quantization State Space Models

Open paper AskPDF arXiv

State Space Models (SSMs) such as Mamba-2 offer linear-time inference but their memory footprint limits edge deployment. Prior ternary SSM work (Slender-Mamba) trains from scratch on 150B tokens; we show a pretrained checkpoint suffices, reducing the marginal token budget by 1,000x. Using grouped quantization-aware training (QAT) with knowledge distillation from a frozen FP16 teacher, we compress Mamba-2 1.3B to 3.61x (2,687 to 744 MB) and achieve 48.1% zero-shot accuracy (7-task average) in just 102M tokens (4 GPU-hours, single H100) -- approaching Bi-Mamba's 48.4% (within +/-0.9pp CI). This QAT-from-pretrained setting reveals zero-ratio collapse, a novel instability caused by learnable quantization scales that does not arise in from-scratch training. We further show that post-hoc correction strategies effective for Transformers fail for SSMs due to error accumulation through the recurrence. These results demonstrate that ternary SSMs do not require expensive from-scratch training: QAT from pretrained checkpoints with KD is a data-efficient alternative.

Abstract

Post-Training Quantization

Explore similar work

Ternary Mamba: Grouped Quantization-Aware Training of W1.58A16 State Space Models

Multi-Scale Dequant: Eliminating Dequantization Bottleneck via Activation Decomposition for Efficient LLM Inference

Channel-Wise and Token-Aware Post-Training Quantization for Visual State Space Duality