Every Weight Is a Path of Shared Discrete Decisions

Ammar Ali; Stamatis Lefkimmiatis

Figure 1: From formats in use to TWIG. Each column of boxes is the description of one weight, read from top to bottom; a box that spans several weights is a decision they share, stored once. (a) A 16-bit float stores a sign, an exponent and a mantissa for every weight. (b) Group quantization shares one scale among the weights of a group. (c) Block formats share an exponent,and a zero ends a weight’s description. (d) A state tree shares decisions at every level, and each decision is inherited by every weight below it: one scale for all four weights, a halving for the first pair, an “off” switch for the second pair, after which nothing more is stored (dashed). Blue: shared decisions; orange: private decisions; grey: zero.

Figure 2: A first example. (a) Two neighbouring weights stored in 16-bit floating point; their exponents are identical, yet the format stores them twice. (b) Because a weight is the product of its decisions, the shared exponent can be stored once in a node above both weights; each weight is recovered exactly as the product along its path. (c) The same tree computes

w_{1}x_{1}+w_{2}x_{2}

without forming either weight: each leaf becomes shifted copies of its own input (

\gg k

shifts by

k

binary positions, i.e. multiplies by

2^{-k}

), and the shared exponent is applied once, to the sum, as a shift by three positions. Three additions replace two multiplications and one addition.

Figure 3: The running example: a deeper state tree for one column

j

of a matrix with eight rows. Each internal node holds one bit that halves everything below it or leaves it unchanged; each leaf holds a sign and a magnitude, possibly zero. The value of an entry is the product of the states on its path (red), times a row scale and a column base, Equation 1 . Apart from

\rho_{i}\beta_{j}

, every factor is a power of two or zero. The column stores 7 node bits (6 if the root bit is folded into

\beta_{j}

) and 8 five-state leaves, about 3.2 bits per weight.

Part of the tree	In compression	In a circuit	In a Boltzmann machine
Shift state	shared exponent or scale, as in Figure 2	a re-routing of wires	one binary unit
Zero state (gate)	pruning, from single weights to whole heads	a missing wire	a unit that switches off its subtree
Leaf alphabet	quantization grid	a sign and a shift at an adder input	a one-hot group of units
Sum of trees	residual correction, extra precision	one more term, one more addition	one more term in each weight
Product of trees	low rank, fixed rotations	two smaller circuits in sequence	units of two factors coupled together
Shared subtree	codebook, dictionary, cross-layer sharing	stored once; computed once when its input is shared	units shared by many weights

Table 1: One tree, three readings. Each part of a state tree has a meaning in compression ( Section 5 ), in circuits ( Section 6 ) and in discrete optimization ( Section 7 ); this is what lets one representation serve all three.

Format	Shared decisions (nodes)	Private decisions (leaves)
FP32, FP16, BF16, FP8 †	—	sign, exponent, mantissa (a chain of three)
Integers with group scale	scale per group	$b$ -bit code
with zero point †	scale and zero point per group	$b$ -bit code, read relative to the zero point
Block FP, microscaling (MX)	exponent per block	small float or integer
Shared microexponents (MX6, MX9), HiFloat4	scale per block, then one or two levels of 1-bit shifts per sub-block	small float or integer
Ternary (BitNet b1.58)	one scale	$\{-1,0,+1\}$

Table 2: Formats in use are shallow state trees, at no more than their original cost ( Property 1 ). † Uses lookups: subnormal mantissas, codes read relative to a zero point, entries read from a codebook, and entries kept by an

N{:}M

mask; the other rows are core trees. Additive codebooks and rotations are built from these pieces with sums and products; trellis codes, whose overlapping windows share bits without nesting, fit only as a codebook over a whole block, with a table that is generated rather than stored. Deeper trees add levels of shared decisions between the two columns.

Figure 4: Three dials along which state trees grow. (a) Adding states: a flat grid; a shift tree, whose values form a logarithmic grid; and a sum of two shift trees, which reaches the uniform 17-level grid of all multiples of

\frac{1}{8}

in

[-1,1]

, since each of them is a sum of at most two signed powers of two with exponents down to

-3

( Property 3 ). (b) Adding rank. (c) Sharing a subtree among several matrices, possibly in different layers. With real-valued factors, rank or sharing alone reaches every matrix.

Figure 5: Building the tree of our lossless format. (a) Weights of a trained layer lie within a few octaves of one another, but rarely share an exponent exactly. (b) Each group keeps its largest exponent once; every weight keeps only its distance

D

below it, and its sign and mantissa. (c) The resulting tree has a fixed shape: one node per group, one state and one leaf per weight. Blue: shared decisions; orange: private decisions. The example uses groups of 4 weights; the format uses groups of 64.

Figure 6: The stored record of group

g_{1}

and the decoding of

w_{14}

. Every field has a fixed width at a fixed offset, so the address of any weight’s state and leaf follows from its index alone. Decoding subtracts the state from the shared exponent and copies the sign and mantissa from the leaf. In the format, a block of 16 rows and 64 columns, the shape the kernel feeds to the tensor cores, is one record of

16\times 97=1{,}552

bytes.

Edit	What changes in the tree	Familiar name
Zero a state	a leaf or node state takes the value $0$	unstructured, $N{:}M$ and structured pruning [ 43 , 29 , 23 , 70 , 54 , 95 ]
Shrink an alphabet	fewer states at the leaves or nodes	quantization [ 22 , 47 , 37 ] , ternarization [ 51 ]
Insert a shared node	a new group with a scale, shift or sign	group scales, shared exponents [ 66 ]
Split into a sum	$W\to W_{1}+W_{2}$	low rank plus residual, additive codebooks [ 16 ]
Factor into a product	$W\to UV$ or $W\to R_{1}W^{\prime}R_{2}$	low rank [ 84 , 74 , 91 , 93 ] , rotations [ 77 ] , butterfly [ 13 ]
Reference a shared part	a subtree is stored once and used many times	codebooks [ 79 , 77 , 76 ] , sparse dictionaries [ 92 ] , cross-layer sharing [ 41 , 83 , 96 , 94 ]

Table 3: Seven edits of a state-tree graph. Common compression methods, including those cited here, are combinations of these edits, together with a rule for choosing the new states.

Figure 7: Compression methods as edits of a state-tree graph. Grey: zero states; green: sum (

+

) and product (

\times

) nodes; purple: references to shared subtrees.

Figure 8: Left: the menu of one matrix in the (size, accuracy cost) plane; only its Pareto set enters the optimizer ( Property 5 ). Right: nested candidate groups for shared subtrees; every group offers either its children’s combined menus or one subtree shared by all of its matrices, and keeping Pareto sets at every node keeps the search exact. Points are illustrative.

Figure 9: A layer as a circuit. When the states are powers of two, multiplications by weights become shifts and signs, which are wiring; zeros are missing wires; and each additional power-of-two term costs one adder. The only multiplications left are by the column bases and the row scales.

Component	As a state tree	In a circuit
Fully connected $Wx+b$	a tree (or graph) for $W$	shifts and adders; the bias is a constant input
Convolution	one shared subtree, referenced per position	replicated per position, or reused over time
Winograd / fast convolution	products with fixed shift-tree transforms	shifts and adders only
Attention projections	four trees; $Q,K,V$ share their input	shared partial sums across $W_{Q},W_{K},W_{V}$
Attention scores, context	products of two activations	multipliers, or a separate treatment
Gated MLP	two trees that share their input	shared partial sums; a multiplier for the gate

Table 4: How the components of common networks map onto state trees and circuits. Every product of a weight and an activation becomes shifts and additions; what remains are products of two activations and a few nonlinear functions.

Matrix	at GPTQ-4 error	at GPTQ-3 error	at GPTQ-2 error
layer 1, gate projection	1.17 $\to$ 1.05 ( $-10\%$ )	0.80 $\to$ 0.69 ( $-14\%$ )	0.42 $\to$ 0.29 ( $-30\%$ )
layer 15, gate projection	1.16 $\to$ 1.05 ( $-9\%$ )	0.80 $\to$ 0.69 ( $-14\%$ )	0.42 $\to$ 0.29 ( $-30\%$ )
layer 15, down projection	1.16 $\to$ 1.05 ( $-10\%$ )	0.79 $\to$ 0.67 ( $-15\%$ )	0.41 $\to$ 0.27 ( $-35\%$ )
layer 30, down projection	1.15 $\to$ 0.93 ( $-20\%$ )	0.79 $\to$ 0.56 ( $-29\%$ )	0.40 $\to$ 0.16 ( $-61\%$ )
layer 1, down projection	1.16 $\to$ 0.79 ( $-32\%$ )	0.80 $\to$ 0.36 ( $-55\%$ )	0.41 $\to$ 0.15 ( $-63\%$ )

Table 5: Adders per weight at equal layer error on LLaMA-3-8B: GPTQ

\to

a flat state tree whose states were chosen for circuit cost. The last two matrices have strong outliers. Counts are rounded to two decimals; percentages are computed from the unrounded counts.

Figure 10: The Boltzmann machine of a state tree ( Property 6 ), drawn for two columns of a four-row version of the running example. The tree decides which units exist and where the higher-order terms are; the data decide how strongly the units interact.

Reading	Setting and metric	Baseline	State tree
Storage and serving	BF16 weights stored exactly; bits per weight	BF16: 16 bits	12.13 bits ( $-24\%$ ), bit-exact
	LLaMA-3-8B, layer 15; matrix–vector speed-up, batch 1–8	cuBLAS (A100); PyTorch (CPU)	1.05–1.51 $\times$ ; 1.24–1.27 $\times$
	same matrices and batches, format alone (bound $1.32\times$ )	our kernel on BF16 (A100)	1.03–1.27 $\times$
	vLLM end-to-end serving, Llama-3.2-1B and Qwen3.5-9B; speed-up at batch 1 / 8 / 32	vLLM, BF16	1.23 $\times$ / 1.17–1.21 $\times$ / $\approx$ 1.0 $\times$
Compression	LLaMA-3-8B at 3.25 bits per weight, no training; perplexity on WikiText-2 / C4 / FineWeb	strongest GPTQ: 8.09 / 12.72 / 11.81	6.95 / 10.93 / 9.76
Discrete optimization	Qwen2.5-1.5B-Instruct at 2.44 bits per weight, only states change; validation KL, GSM8K, MMLU	before tuning: 0.491, 13.95, 48.3	0.371, 21.83, 49.6

Table 6: One result for each reading of the tree. The strongest GPTQ uses act-order and an MSE-optimized grid in the same pipeline. Perplexity at context 2048 (FP16: 6.14 / 9.44 / 7.85); bits per weight cover all linear layers of the transformer blocks, scales included.

Matrix	Batch	cuBLAS	own BF16	tree	kernel	format	total
		( $\mu$ s)	( $\mu$ s)	( $\mu$ s)	own BF16 vs cuBLAS	tree vs own BF16	tree vs cuBLAS
gate $14336\times 4096$	1	106.0	85.6	70.3	1.24 $\times$	1.22 $\times$	1.51 $\times$
	8	107.8	84.7	72.4	1.27 $\times$	1.17 $\times$	1.49 $\times$
	16	108.9	86.6	78.4	1.26 $\times$	1.10 $\times$	1.39 $\times$
	32	117.2	98.3	113.7	1.19 $\times$	0.86 $\times$	1.03 $\times$
down $4096\times 14336$	1	102.8	89.8	70.9	1.14 $\times$	1.27 $\times$	1.45 $\times$

Table 7: Matrix–vector products of layer 15 of LLaMA-3-8B on an A100, all three kernels timed in the same run. “Own BF16” is our kernel with the same structure reading plain BF16, so kernel isolates kernel engineering and format isolates the 12.13-bit tree (bounded by

16/12.13\approx 1.32\times

); their product is the total gain over cuBLAS. At batch 32 the product is compute-bound and decoding costs more than the saved bytes gain.

Codec	Implementation	Bits per pixel	Decoding (Gpixel/s)
JPEG XL, effort 9 [ 3 ]	libjxl 0.11.2, CPU	8.66	—
Ours (tree analogue)	GPU (A100), batched	9.47	2.9
WebP, lossless [ 64 ]	libwebp 1.6.0, CPU	9.50	—
JPEG 2000 [ 72 ]	OpenJPEG 2.5.4; GPU / CPU	9.57	0.10 / $\approx$ 0.04
HTJ2K [ 73 ]	OpenJPH 0.26.3; GPU decoding	10.53	0.45

Table 8: Lossless image coding on Kodak [ 39 ] ; every result was checked to be bit-exact. GPU decoding of JPEG 2000 and HTJ2K uses nvImageCodec with the nvJPEG2000 backend. All GPU timings decode the 24 images in batched calls; one image per call, JPEG 2000 and HTJ2K reach 0.03 and 0.39 Gpixel/s. GPU timings run from compressed bytes in host memory to the decoded image in GPU memory (median of 10 runs, A100-SXM4-40GB); the CPU timing, on 16 cores of a shared AMD EPYC 7742, is indicative. For a single small image, GPU HTJ2K keeps an advantage that the batched timings do not show.

Every Weight Is a Path of Shared Discrete Decisions

Abstract

Figures & tables

Neural Network Compression

Weight Pair Encoding: Inducing a Smaller Grammar in Neural Network Weights

Structured Transforms for Low-Overhead Quantization of Language Models

Neural Weight Norm = Kolmogorov Complexity

Abstract

Neural Network Compression

Explore similar work

Weight Pair Encoding: Inducing a Smaller Grammar in Neural Network Weights

Structured Transforms for Low-Overhead Quantization of Language Models

Neural Weight Norm = Kolmogorov Complexity