BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks

Max Zhu; Bohang Sun; Pietro Lio; Jialin Yu

cs.LGOct 15, 2026

BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks

Authors: Max Zhu, Bohang Sun, Pietro Lio, Jialin Yu

Organizations: University of Cambridge · University of Oxford

Abstract

Reducing memory use in large language model (LLM) training is an ongoing challenge. Activation checkpointing saves memory by recomputation, while lossy compression changes the stored activations. The reemergence of sparse $\relu$ and $\relu^2$ activations creates a third opportunity: losslessly compressing the activation cache itself. We present BitSparse, a GPU-resident lossless compression format for sparse activations. Compression and reconstruction run in custom Triton kernels fused into an autograd function, shrinking the activation cache while recovering it exactly for the backward pass. Bfloat16 (BF16) and FP8 activations are supported. Optional sign-bit packing further increases compression up to close to the theoretical limit. We analyze its compression ratio and compare it to the theoretical optimum. BitSparse lowers per-token VRAM by 31% and 30% Modded-NanoGPT and Nemotron-H respectively, with overheads of no more than 1% at practical sequence lengths. Additionally, a new preallocated buffer system avoids host synchronization during training when variable size tensors are allocated.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of BitSparse. Caching $\relu$ activations for backpropagation is expensive: most entries are exact zeros, and every stored value carries a redundant sign bit. BitSparse caches only the nonzero values and reconstructs them with optimized Triton kernels.

	BF16			FP8
Sparsity $\rho$	BitSparse	BitSparse-sb	Ideal	BitSparse	BitSparse-sb	Ideal
$50\%$	$1.776\times$	$1.881\times$	$1.882\times$	$1.598\times$	$1.775\times$	$1.778\times$
$75\%$	$3.195\times$	$3.363\times$	$3.508\times$	$2.660\times$	$2.901\times$	$3.123\times$
$90\%$	$6.135\times$	$6.380\times$	$8.126\times$	$4.425\times$	$4.684\times$	$6.843\times$

Table 1: Theoretical compression ratios of BitSparse, BitSparse-sb, and the theoretical optimum under Assumption 2.1 . Boundary padding overheads are neglected.

Sign-bit	$\rho$	Time / ms		Rate / GiB/s		Memory BW / %
packing		Encode	Decode	Encode	Decode	Encode	Decode
No	0.5	3.52	2.20	284.1	454.5	60.9 (50.9)	97.4 (81.5)
Yes	0.5	7.03	2.79	142.2	358.4	29.9 (25.0)	75.3 (63.0)
No	0.9	3.31	2.22	302.1	450.5	52.3 (43.8)	78.0 (65.3)
Yes	0.9	4.84	2.92	206.6	342.5	35.4 (29.6)	58.7 (49.1)

Table 2: Timings for encoding and decoding a

1

GiB sparse BF16 tensor (

65{,}536\times 8{,}192

) with and without sign-bit packing. The bandwidth utilization (Memory BW) includes both read and write. Bandwidth utilization is compared with the attainable throughput measured with NVbandwidth, with the device’s nameplate bandwidth (

936

GB/s) in parentheses.

	Memory / MiB				Time / ms
	BF16		FP8		BF16		FP8
Method	$0.5$	$0.9$	$0.5$	$0.9$	$0.5$	$0.9$	$0.5$	$0.9$
Dense	12346	11344	9492	9492	1827	1827	395	395
BitSparse	8806	5896	7180	6617	1822	1815	391	391
BitSparse-sb	8637	5860	7042	6551	1844	1828	397	391
Checkpoint	6713	6713	7417	7417	2144	2144	507	507

Table 3: Peak allocated memory and time per step for the

\relu^{2}

MLP benchmark, using BF16 and FP8 activations at sparsity

\rho=0.5

and

0.9

. CompAct is a lossy reference. Bold and italic values indicate the lowest and second-lowest lossless results. Dashes indicate unavailable results.

Batch	Memory / MiB				Time / ms
size	Dense	Checkpoint	BitSparse	BitSparse-sb	Dense	Checkpoint	BitSparse	BitSparse-sb
32	3054	2879	2879	2879	12	14	12	13
128	3094	2899	2896	2896	18	22	19	19
512	3264	2982	2972	2971	64	74	64	64
2000	3932	3294	3252	3249	243	284	240	241
8000	7107	4704	4384	4368	929	1086	919	926

Table 4: Effect of batch size on peak allocated memory and time per step for the

\relu^{2}

MLP at sparsity

\rho=0.9

. Bold and italic values indicate the lowest and second-lowest results, respectively.

Figure 2: Memory–runtime trade-off for the

\relu^{2}

MLP at sparsity

\rho=0.9

, varying the number of compressed or checkpointed layers from

0

8

. Each point represents one layer-count configuration, lines connect configurations in layer-count order.

Figure 3: Peak GPU memory for Modded-NanoGPT (left) and Nemotron-H (right) versus the number of tokens processed per step. Markers denote measurements, dotted curves are least-squares linear fits. Legend entries give the fitted slope in units of MiB per token.

Figure 4: Step time for Modded-NanoGPT (left) and Nemotron-H (right) versus the number of tokens processed per step. Conventions follow Figure 3 .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Activation	Dense	Checkpoint	BitSparse	BitSparse-sb
	Storage / (KiB per token)
Raw inputs retained for block/final norms	424	424	424	424
Mamba packed projection ( gate , x , B , C , dt )	870	870	870	870
Mamba scan output	384	384	384	384
FlashAttention Q/K/V	48	48	48	48
FlashAttention output	32	32	32	32

Appendix

Table 5: Analytical activation storage per token for Nemotron-H-8B-Base-8K, comparing dense storage, checkpointing of each feed-forward block, BitSparse, and BitSparse with sign-bit packing. Sizes are computed from the model’s tensor shapes, not measured, and cover the forward-pass tensors retained for the backward pass. Components below

1

KiB per token are grouped under Other . Values are rounded to the nearest whole KiB, so columns may not sum exactly.

Figure 5: Activation sparsity during Modded-NanoGPT training for all 12 feed-forward layers. Solid lines show per-layer mean sparsity and dashed lines show per-layer minimum sparsity over 256 evaluation batches. The vertical axis reports the zero fraction

\rho

, so higher values indicate greater sparsity.

Layer	Mean sparsity	Minimum sparsity
1	0.9718	0.9673
3	0.9335	0.9235
10	0.9590	0.9543
27	0.8649	0.8408
32	0.9160	0.9023
47	0.9616	0.9383

Appendix

Table 6: Activation sparsity

\rho

for selected Nemotron-H-8B-Base-8K layers over 256 validation batches. Per-layer minima correspond to each layer’s densest observed batch. The overall row reports mean sparsity across all measured layers and batches, and the minimum batch-wise mean sparsity across all measured layers. Higher values indicate greater sparsity.

Method	Microbatch size	Memory / MiB	Time / s
Normal	32	15229.6	7.428
Normal	64	OOM	OOM
BitSparse	32	13174.7	7.668
BitSparse	64	21822.1	7.416

Appendix

Table 7: Memory usage and time per training step for Modded-NanoGPT at two microbatch sizes. OOM denotes an out-of-memory failure.

Figure 6: Validation loss during standard (Normal) and BitSparse-sb training of Modded-NanoGPT. The inset zooms in on steps

3{,}000

–

3{,}350

	Memory / MiB				Time / ms
	BF16		FP8		BF16		FP8
Method	$0.5$	$0.9$	$0.5$	$0.9$	$0.5$	$0.9$	$0.5$	$0.9$
Dense	11046	10039	8500	8500	1748	1748	323	333
BitSparse	8800	5742	7165	6495	1794	1776	324	328
BitSparse-sb	8631	5715	7030	6443	1799	1786	330	331
Checkpoint	6928	6057	6105	6105	2047	2047	441	452

Appendix

Table 8: Peak allocated memory and time per step for the

\relu

MLP benchmark, using BF16 and FP8 activations at sparsity

\rho=0.5

and

0.9

. CompAct is a lossy reference. Bold and italic values indicate the lowest and second-lowest lossless results. Dashes indicate unavailable results.

Batch	Memory / MiB				Time / ms
size	Dense	Checkpoint	BitSparse	BitSparse-sb	Dense	Checkpoint	BitSparse	BitSparse-sb
32	3054	2879	2883	2882	13	14	14	13
128	3094	2899	2912	2911	19	22	19	20
512	3264	2982	3033	3028	64	75	64	67
2000	3932	3294	3492	3471	242	283	241	244
8000	7107	4704	5337	5253	925	1088	927	938

Appendix

Table 9: Effect of batch size on peak allocated memory and time per step for the

\relu^{2}

MLP at sparsity

\rho=0.5

. Bold and italic values indicate the lowest and second-lowest results, respectively.

Batch	Memory / MiB				Time / ms
size	Dense	Checkpoint	BitSparse	BitSparse-sb	Dense	Checkpoint	BitSparse	BitSparse-sb
32	3054	2878	2883	2882	12	15	13	13
128	3094	2894	2912	2911	18	21	19	19
512	3264	2960	3033	3028	62	73	64	65
2000	3889	3212	3490	3470	234	273	236	239
8000	6451	4376	5334	5251	887	1037	905	912

Appendix

Table 10: Effect of batch size on peak allocated memory and time per step for the

\relu

MLP at sparsity

\rho=0.5

. Bold and italic values indicate the lowest and second-lowest results, respectively, including ties.

Batch	Memory / MiB				Time / ms
size	Dense	Checkpoint	BitSparse	BitSparse-sb	Dense	Checkpoint	BitSparse	BitSparse-sb
32	3054	2878	2879	2878	12	13	12	12
128	3094	2894	2895	2895	18	21	19	19
512	3264	2960	2966	2965	61	72	63	63
2000	3889	3212	3233	3230	232	271	235	237
8000	6451	4376	4307	4293	893	1040	904	908

Appendix

Table 11: Effect of batch size on peak allocated memory and time per step for the

\relu

MLP at sparsity

\rho=0.9

. Bold and italic values indicate the lowest and second-lowest results, respectively.

Operation	Implementation	Peak memory / MiB	Time / ms
Encode	Pure PyTorch	2560.00	28.010
	Triton (BitSparse)	1600.50	3.690
Decode	Pure PyTorch	2048.53	17.168
	Triton (BitSparse)	1600.50	2.170

Appendix

Table 12: BitSparse codec implementation ablation on a

65{,}536\times 8{,}192

BF16 tensor. The baseline implements BitSparse in pure PyTorch; the proposed implementation uses custom Triton kernels. Lower peak memory and runtime are better.

Activation	Method	Peak memory / MiB	Time / ms
$\relu$	PyTorch	1312.50	2.467
	BitSparse	698.25	2.043
$\relu^{2}$	PyTorch	2625.00	4.200
	BitSparse	1025.70	2.552

Appendix

Table 13: Activation backward ablation on a

16{,}000\times 21{,}504

BF16 tensor with approximately

50\%

sparsity. Peak memory includes the saved representation, gradient, and backward temporaries; time covers only the activation-derivative computation. Lower is better.

Tokens	Without buffer / ms	With buffer / ms
32	12.563	11.904
64	17.622	17.136
128	20.150	19.954
256	37.973	38.097
512	66.539	66.308
1024	130.807	130.518

Appendix

Table 14: Effect of the shared value buffer on BitSparse runtime as input size increases. Times retain three decimal places to show the small differences between configurations. Lower is better.

Activation Compression

See all 11 papers

LCT: Losslessly Compressed Tensors with GPU friendly execution for Training
Oct 15, 2026Max Zhu, Bohang Sun, Pietro Lio +1Memory Compression Memory-Efficient Training
University of Cambridge · University of Oxford
Activation Compression in LLMs: Theoretical Analysis and Efficient Algorithm
May 2, 2026Wen-Da Wei, Han-Bin Fang, Yang-Di Liu +3LLM Compression Gradient Compression
Nanjing University, Nanjing, China · Tsinghua University, Beijing, China · Huazhong University of Science and Technology, Wuhan, China +1
PRAC: Principal-Random Subspace for LLM Activation Compression and Memory-Efficient Training
Feb 26, 2026Yanyi Li, Yimu Zhang, Cong FangLLM Compression Memory-Efficient Training
School of Intelligence Science and Technology, Peking University, China · Institute for Artificial Intelligence, Peking University, China

Oct 15, 2026J/K move · Enter open · S save

LCT: Losslessly Compressed Tensors with GPU friendly execution for Training

Max Zhu, Bohang Sun, Pietro Lio, Jialin Yu

University of Cambridge · University of Oxford

Memory Compression Memory-Efficient Training Neural Network Compression Lossless Data Compression Memory-Efficient Fine-Tuning

Open paper AskPDF arXiv

Lossless tensor compression could reduce memory usage during training, but training requires tensors to be repeatedly re-encoded. We introduce Losslessly Compressed Tensors (LCT), a GPU execution framework designed for this repeated encode-decode regime. LCT exploits the natural redundancy in floating point data to reduce storage for weights, activations and optimizer states. Highly optimized GPU kernels implement Huffman coding with precomputed codebooks for low compression and decompression overhead, essential for training. Our analysis characterizes LCT's storage costs and adaptation to magnitude shifts. LCT achieves a compression ratio of up to

1.455\times

, and is demonstrated in training and finetuning settings. On a 162M-parameter NanoGPT, LCT reduces memory usage by

24.6\%

for a

7.2\%

step-time increase. In Qwen3-4B fine-tuning, LCT saves up to

23.8\%

of peak memory, enabling a

3.5\times

larger context length in training on the same GPU.

Abstract

Activation Compression

Explore similar work

LCT: Losslessly Compressed Tensors with GPU friendly execution for Training

Activation Compression in LLMs: Theoretical Analysis and Efficient Algorithm

PRAC: Principal-Random Subspace for LLM Activation Compression and Memory-Efficient Training