LCT: Losslessly Compressed Tensors with GPU friendly execution for Training

Max Zhu; Bohang Sun; Pietro Lio; Jialin Yu

cs.LGOct 15, 2026

LCT: Losslessly Compressed Tensors with GPU friendly execution for Training

Authors: Max Zhu, Bohang Sun, Pietro Lio, Jialin Yu

Organizations: University of Cambridge · University of Oxford

Abstract

Lossless tensor compression could reduce memory usage during training, but training requires tensors to be repeatedly re-encoded. We introduce Losslessly Compressed Tensors (LCT), a GPU execution framework designed for this repeated encode-decode regime. LCT exploits the natural redundancy in floating point data to reduce storage for weights, activations and optimizer states. Highly optimized GPU kernels implement Huffman coding with precomputed codebooks for low compression and decompression overhead, essential for training. Our analysis characterizes LCT's storage costs and adaptation to magnitude shifts. LCT achieves a compression ratio of up to $1.455\times$ , and is demonstrated in training and finetuning settings. On a 162M-parameter NanoGPT, LCT reduces memory usage by $24.6\%$ for a $7.2\%$ step-time increase. In Qwen3-4B fine-tuning, LCT saves up to $23.8\%$ of peak memory, enabling a $3.5\times$ larger context length in training on the same GPU.

Figures & tables

Figure 1: Overview of LCT. Sign-mantissa bytes remain unchanged while exponents are encoded with bounded Huffman codes with escapes for rare values. Values are packed into fixed sized streams with overflow buffers. Tensors are decoded losslessly, with fused kernels for optimizer updates.

Panel of Figure 2 — Figure 2: Compression use cases ordered by increasing encode-bandwidth requirement. Static inference weights amortize encoding across repeated use; autoregressive KV caches add entries at each token; general tensor compression must accommodate frequent updates. The ordering is qualitative and reflects these access patterns, rather than measured throughput.

Tensor	Encodes	Dense decodes	Fused decodes	Uses and lifetime
Weights $W$	1	2	1	Forward and backward computation; fused $W\leftarrow W-\eta M$ . Persist across steps.
Biases	0	0	0	Kept dense because they are small; updated with dense operations.
Momentum $M$	1	0	2	Fused $M\leftarrow\beta M+G$ , then fused $W\leftarrow W-\eta M$ . Persists across steps.
Activations	1	1	0	Activations are encoded in forward and decoded for gradient computation in backward.

Table 1: Tensor lifecycle and LCT codec operations per SGD-with-momentum step, after momentum initialization. The weight row covers MLP and other layer weights, assuming one dense use in forward and one in backward. Encodes include output recompression. Dense decodes materialize a tensor; fused decodes consume it within an update. Counts describe logical operations, not kernel launches.

${\mathcal{P}}\backslash{\mathcal{Q}}$	G	L	PL	$\Gamma$
Gaussian	$\mathbf{2.545}$	$2.630$	$2.637$	$3.007$
Laplace	$3.039$	$\mathbf{2.832}$	$2.844$	$3.062$
Power-law	$8\!\times\!10^{7}$	$2.976$	$\mathbf{2.891}$	$3.118$
Gamma	$5.644$	$3.417$	$3.307$	$\mathbf{3.015}$

Table 2: Ideal cross entropy (left) and allocated LCT exponent storage, (right), in bits per exponent. Source distributions are in rows and encoding models in columns. G: Gaussian; L: Laplace; PL: power-law;

\Gamma

: Gamma. Bold entries use matched models.

	Average bits per exponent			Compression ratio ( $\times$ )
Codec	Attention	MLP	Mamba projection	Attention	MLP	Mamba projection
Power-Law	3.1250	3.4041	3.1304	1.4382	1.4030	1.4375
Gaussian	3.4984	4.1793	3.8003	1.3915	1.3137	1.3559
Laplace	3.1250	3.4025	3.1312	1.4382	1.4032	1.4374
Gamma	3.2668	3.4416	3.1312	1.4201	1.3984	1.4374

Table 3: Average exponent storage (bits/exponent, left) and measured compression ratios (right) for Nemotron-H-8B parameters using four LCT codecs. Bold entries indicate the best results in each parameter group.

Method	Encode / ms	Decode / ms	Compression ratio
nvCOMP LZ4	$484.584\pm 0.646$	$8.761\pm 0.035$	$0.998\times$
nvCOMP Bitcomp	$40.413\pm 0.585$	$4.193\pm 0.035$	$0.998\times$
nvCOMP ANS	$53.028\pm 0.489$	$4.349\pm 0.059$	$1.258\times$
DFloat11	$2574.000\pm 0.671$	$4.475\pm 0.014$	$1.483\times$
ZipNN	$845.400\pm 0.841$	$199.300\pm 0.619$	$1.511\times$
SplitZip	$5.850\pm 0.016$	$\mathbf{2.309\pm 0.004}$	$1.333\times$

Table 4: Time taken to encode and decode a

1\,\mathrm{GiB}

tensor. Standard errors in the times are shown. Compression ratio is original size divided by compressed size. The fastest method is marked in bold.

Operation	Output	LCT / ms	Naive / ms	Dense / ms	Speedup
$C+D$	Dense	$3.694\pm 0.003$	$6.342\pm 0.015$	$3.855\pm 0.003$	$1.717\times$
$C\odot D$	Dense	$3.691\pm 0.003$	$6.347\pm 0.021$	$3.866\pm 0.014$	$1.720\times$
$\alpha C+D$	Dense	$3.822\pm 0.003$	$6.331\pm 0.008$	$3.859\pm 0.005$	$1.656\times$
$\alpha C_{1}+C_{2}$	Dense	$5.262\pm 0.004$	$8.709\pm 0.003$	$3.864\pm 0.009$	$1.655\times$
$C+D$	Compressed	$5.058\pm 0.010$	$8.858\pm 0.009$	$3.852\pm 0.003$	$1.751\times$
$C\odot D$	Compressed	$5.006\pm 0.009$	$12.814\pm 0.019$	$3.848\pm 0.003$	$2.560\times$

Table 5: Fused pointwise operator timings (mean

\pm

standard error).

C

denotes a compressed operand,

D

a dense operand, and

\alpha

a scalar. Speedup compares LCT with Naive.

Compressed components	Time / s	Peak memory / MiB	Effective compression ratio
None	8.411	16058.0	$1.000\times$
Activations	9.013	12103.0	$1.327\times$
+ Weights	9.131	12013.8	$1.337\times$
+ Momentum Buffers	9.136	11964.6	$1.342\times$

Table 6: NanoGPT training time per optimizer step and peak PyTorch-allocated GPU memory, including the shared buffer. The effective compression ratio is baseline peak memory divided by each configuration’s peak memory, including all live GPU allocations.

	Baseline		LCT
Sequence length	Memory / MiB	Time / s	Memory / MiB	Time / s
500	19439.2	3.759	15141.8	4.058
1000	21327.8	4.006	16262.4	4.273
1500	—	—	17370.1	4.523
2500	—	—	19617.6	5.055
3500	—	—	21842.1	5.607

Table 7: Qwen3-4B full-parameter fine-tuning at batch size 1. Memory is peak PyTorch-allocated GPU memory, including any shared buffer, and time is per training step. Dashes indicate runs that failed to complete.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

	NVIDIA V100 (1 GiB)		NVIDIA A40 (1 GiB)		NVIDIA H100 (4 GiB)
Method	Encode / ms	Decode / ms	Encode / ms	Decode / ms	Encode / ms	Decode / ms
nvCOMP LZ4	—	—	$629.2\pm 0.242$	$13.64\pm 0.138$	$500.1\pm 12.33$	$16.63\pm 11.79$
nvCOMP Bitcomp	—	—	$46.43\pm 0.134$	$5.354\pm 0.002$	$25.74\pm 4.997$	$4.771\pm 4.771$
nvCOMP ANS	—	—	$61.75\pm 0.008$	$5.341\pm 0.002$	$34.6\pm 1.408$	$6.349\pm 0.081$
DFloat11	$5798\pm 196.0$	$6.271\pm 0.103$	$28430\pm 2.245$	$4.750\pm 0.007$	$17550\pm 208.1$	$11.18\pm 0.022$
ZipNN	$2504\pm 179.6$	$470.2\pm 42.78$	$1250\pm 1.592$	$529.5\pm 1.053$	$7001\pm 329.5$	$1290\pm 30.6$

Appendix

Table 8: Encoding and decoding Gaussian tensors of

1\,\mathrm{GiB}

on NVIDIA V100 and A40, and

4\,\mathrm{GiB}

on NVIDIA H100. Times are mean

\pm

standard error of the mean; lower is better. Dashes indicate unreported results. Bold entries mark the lowest mean for each operation on each GPU.

Figure 3: Encoding (left) and decoding (right) throughput versus uncompressed input size. Throughput is the uncompressed size in GiB divided by the mean time in seconds. LZ4 and Cascaded are omitted.

Activation	Count	Shape	Each / MiB	Total / MiB	Budget
Embedding output before RMSNorm	1	$[64,1024,768]$	96	96	CLEAN
Input to RMSNorm and QKV projections	12	$[64,1024,768]$	96	1152	CLEAN
Q and K before per-head RMSNorm	24	$[64,1024,6,128]$	96	2304	CLEAN
FlashAttention Q/K/V/output caches	48	$[64,1024,6,128]$	96	4608	CLEAN
FFN input before RMSNorm and first linear	12	$[64,1024,768]$	96	1152	CLEAN
FFN ReLU output before squaring	12	$[64,1024,3072]$	384	4608	SPARSE

Appendix

Table 9: Cached activations compressed in the Modded NanoGPT experiment. Sizes are the dense BF16 storage before compression for a batch of 64.

Setup	Time / s	Peak memory / MiB
With buffer	9.136	11964.6
Without buffer	9.353	11912.9

Appendix

Table 10: Effect of the shared OverflowBuffer on NanoGPT training with compressed activations, weights, and momentum buffers. Time is per optimizer step; peak PyTorch-allocated GPU memory includes the shared buffer when enabled.

	Compression ratio ( $\times$ )
Step	Weights	Momentum
0	1.2240	—
1	1.2240	1.4382*
2	1.4382	1.2426
5	1.4382	1.4116
10	1.4382	1.4043

Appendix

Table 11: Weight and momentum compression ratios at NanoGPT training checkpoints. Ratios are uncompressed size divided by compressed size. The dash denotes an unreported value. * The first momentum step uses a compression distribution with a higher zero fraction.

Figure 4: NanoGPT feed-forward exponent distributions during training: Muon momentum (left) and weights (right) at steps 1, 4, 8, 300, 1500, and 3300. Weight exponents below

-20

are omitted from view.

Configuration	$B$	$S$	Bits/stream	Bits/element
Sparse (ReLU activations)	65536	256	640	2.500
Clean, non-Gaussian	65536	256	800	3.125
Clean, Gaussian	65536	256	768	3.000
Medium	32768	128	608	4.750
High	32768	128	704	5.500

Appendix

Table 12: Implemented stream configurations. The final column is the fixed exponent payload per padded element, before overflow and metadata.

Operator	Computation	Training role
compress	$D\to C$	Retain BF16 weights or state between uses; store activations within an explicitly managed backward integration.
decompress	$C\to D$	Restore state or saved tensors for a dense consumer.
compA_add_B	$C+D$	Accumulate microbatch gradients or apply a prepared parameter increment.
compA_mul_B	$C\odot D$	Apply an existing elementwise mask, gate, or scale.
a_compA_add_B	$\alpha C+D$	Update momentum or moment state; combine weight decay with a prepared dense step.
a_compA_add_compB	$\alpha C_{1}+C_{2}$	Apply a compressed update to compressed state; $W-\eta U$ uses $U$ first and $W$ second.

Appendix

Table 13: LCT operators and training integration points.

C

and

D

denote compressed and dense operands;

\alpha

is a device scalar. Fused pointwise operators support dense or compressed output. Matrix products use dense GEMM after reconstruction.

Figure 5: Nemotron-H-8B weight exponent probabilities, with the BF16 bias removed. The hatched bar aggregates exponents below

-20

. Probabilities are normalized over all weights.

Figure 6: Nemotron-H-8B activation exponent probabilities, with the BF16 bias removed. The hatched bar combines exponents below

-20

, exact zeros, and subnormals; the FFN hidden bar is dominated by exact zeros. Probabilities are normalized over all observations.

Memory Compression

See all 24 papers

BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks
Oct 15, 2026Max Zhu, Bohang Sun, Pietro Lio +1Activation Compression Memory Compression
University of Cambridge · University of Oxford
NeuralZip: Reusable Setup for Fast Lossless Compression
Oct 7, 2026Martín Bravo, Samuel Horváth, Gonzalo Navarro +1Lossless Data Compression Neural Network Compression
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Department of Computer Science, University of Chile, Santiago, Chile · National Center for Artificial Intelligence (CENIA), Santiago, Chile
Reducing the GPU Memory Bottleneck with Lossless Compression for ML -- Extended
May 29, 2026Aditya K Kamath, Arvind Krishnamurthy, Marco Canini +1Efficient Neural Network Inference GPU Kernel Optimization
University of Washington Seattle, WA, USA · KAUST Thuwal, Saudi Arabia

Oct 15, 2026J/K move · Enter open · S save

BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks

Max Zhu, Bohang Sun, Pietro Lio, Jialin Yu

University of Cambridge · University of Oxford

Activation Compression Memory Compression Memory-Efficient Training Neural Network Compression Activation Sparsity

Open paper AskPDF arXiv

Reducing memory use in large language model (LLM) training is an ongoing challenge. Activation checkpointing saves memory by recomputation, while lossy compression changes the stored activations. The reemergence of sparse

\relu

and

\relu^2

activations creates a third opportunity: losslessly compressing the activation cache itself. We present BitSparse, a GPU-resident lossless compression format for sparse activations. Compression and reconstruction run in custom Triton kernels fused into an autograd function, shrinking the activation cache while recovering it exactly for the backward pass. Bfloat16 (BF16) and FP8 activations are supported. Optional sign-bit packing further increases compression up to close to the theoretical limit. We analyze its compression ratio and compare it to the theoretical optimum. BitSparse lowers per-token VRAM by 31% and 30% Modded-NanoGPT and Nemotron-H respectively, with overheads of no more than 1% at practical sequence lengths. Additionally, a new preallocated buffer system avoids host synchronization during training when variable size tensors are allocated.

Abstract

Memory Compression

Explore similar work

BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks

NeuralZip: Reusable Setup for Fast Lossless Compression

Reducing the GPU Memory Bottleneck with Lossless Compression for ML -- Extended