Every Weight Is a Path of Shared Discrete Decisions

Ammar Ali; Stamatis Lefkimmiatis

cs.LGOct 22, 2026

Every Weight Is a Path of Shared Discrete Decisions

Authors: Ammar Ali, Stamatis Lefkimmiatis

Organizations: LEAD Lab, ITMO University

Abstract

We introduce TWIG (Tree-structured Weights with Inherited Gates), a representation in which every weight of a neural network is the end of a path of small discrete decisions: a sign, a shift by a power of two, a zero gate, an index into a shared table. Weights that belong together share the upper part of their paths, and each decision is inherited by every weight below it, so it is stored once and applied once. This simple change has three consequences. Any trained network can be written as a TWIG, and when the decisions are powers of two its weight layers become circuits made almost entirely of wires and adders, suited to chips whose weights are fixed at fabrication. Pruning, quantization, low-rank factorization, codebooks and weight sharing become edits of one object, so a single optimizer can choose among all of them for a whole network. And choosing the decisions is the ground-state problem of a higher-order Boltzmann machine whose interactions follow the paths of the tree, the problem class that annealers and quantum optimizers are built to solve. Today these questions are handled apart: each compression method brings its own format and solver, a circuit with fixed weights needs a dedicated constant multiplier for every weight, and discrete solvers reach only narrow subproblems such as rounding. Weight formats have long been moving toward shared decisions, through group scales, shared exponents, microexponents and sparse weights; TWIG takes this trend to its end, and every common format is a shallow TWIG at no extra cost. On five weight matrices of a large language model, a flat tree whose signed power-of-two terms are chosen per weight for circuit cost, with no group scales, needs 9--15% fewer additions than GPTQ at equal layer error at 3- and 4-bit accuracy on typical matrices, and 20--55% fewer on matrices with strong outliers.

Figures & tables

Panel of Figure 1 — Figure 1: From formats in use to TWIG. Each column of boxes is the description of one weight, read from top to bottom; a box that spans several weights is a decision they share, stored once. (a) A 16-bit float stores a sign, an exponent and a mantissa for every weight. (b) Group quantization shares one scale among the weights of a group. (c) Block formats share an exponent,and a zero ends a weight’s description. (d) A state tree shares decisions at every level, and each decision is inherited by every weight below it: one scale for all four weights, a halving for the first pair, an “off” switch for the second pair, after which nothing more is stored (dashed). Blue: shared decisions; orange: private decisions; grey: zero.

Panel of Figure 2 — Figure 2: A first example. (a) Two neighbouring weights stored in 16-bit floating point; their exponents are identical, yet the format stores them twice. (b) Because a weight is the product of its decisions, the shared exponent can be stored once in a node above both weights; each weight is recovered exactly as the product along its path. (c) The same tree computes $w_{1}x_{1}+w_{2}x_{2}$ without forming either weight: each leaf becomes shifted copies of its own input ( $\gg k$ shifts by $k$ binary positions, i.e. multiplies by $2^{-k}$ ), and the shared exponent is applied once, to the sum, as a shift by three positions. Three additions replace two multiplications and one addition.

Panel of Figure 3 — Figure 3: The running example: a deeper state tree for one column $j$ of a matrix with eight rows. Each internal node holds one bit that halves everything below it or leaves it unchanged; each leaf holds a sign and a magnitude, possibly zero. The value of an entry is the product of the states on its path (red), times a row scale and a column base, Equation 1 . Apart from $\rho_{i}\beta_{j}$ , every factor is a power of two or zero. The column stores 7 node bits (6 if the root bit is folded into $\beta_{j}$ ) and 8 five-state leaves, about 3.2 bits per weight.

Part of the tree	In compression	In a circuit	In a Boltzmann machine
Shift state	shared exponent or scale, as in Figure 2	a re-routing of wires	one binary unit
Zero state (gate)	pruning, from single weights to whole heads	a missing wire	a unit that switches off its subtree
Leaf alphabet	quantization grid	a sign and a shift at an adder input	a one-hot group of units
Sum of trees	residual correction, extra precision	one more term, one more addition	one more term in each weight
Product of trees	low rank, fixed rotations	two smaller circuits in sequence	units of two factors coupled together
Shared subtree	codebook, dictionary, cross-layer sharing	stored once; computed once when its input is shared	units shared by many weights

Table 1: One tree, three readings. Each part of a state tree has a meaning in compression ( Section 5 ), in circuits ( Section 6 ) and in discrete optimization ( Section 7 ); this is what lets one representation serve all three.

Format	Shared decisions (nodes)	Private decisions (leaves)
FP32, FP16, BF16, FP8 †	—	sign, exponent, mantissa (a chain of three)
Integers with group scale	scale per group	$b$ -bit code
with zero point †	scale and zero point per group	$b$ -bit code, read relative to the zero point
Block FP, microscaling (MX)	exponent per block	small float or integer
Shared microexponents (MX6, MX9), HiFloat4	scale per block, then one or two levels of 1-bit shifts per sub-block	small float or integer
Ternary (BitNet b1.58)	one scale	$\{-1,0,+1\}$

Table 2: Formats in use are shallow state trees, at no more than their original cost ( Property 1 ). † Uses lookups: subnormal mantissas, codes read relative to a zero point, entries read from a codebook, and entries kept by an

N{:}M

mask; the other rows are core trees. Additive codebooks and rotations are built from these pieces with sums and products; trellis codes, whose overlapping windows share bits without nesting, fit only as a codebook over a whole block, with a table that is generated rather than stored. Deeper trees add levels of shared decisions between the two columns.

Panel of Figure 4 — Figure 4: Three dials along which state trees grow. (a) Adding states: a flat grid; a shift tree, whose values form a logarithmic grid; and a sum of two shift trees, which reaches the uniform 17-level grid of all multiples of $\frac{1}{8}$ in $[-1,1]$ , since each of them is a sum of at most two signed powers of two with exponents down to $-3$ ( Property 3 ). (b) Adding rank. (c) Sharing a subtree among several matrices, possibly in different layers. With real-valued factors, rank or sharing alone reaches every matrix.

Panel of Figure 5 — Figure 5: Building the tree of our lossless format. (a) Weights of a trained layer lie within a few octaves of one another, but rarely share an exponent exactly. (b) Each group keeps its largest exponent once; every weight keeps only its distance $D$ below it, and its sign and mantissa. (c) The resulting tree has a fixed shape: one node per group, one state and one leaf per weight. Blue: shared decisions; orange: private decisions. The example uses groups of 4 weights; the format uses groups of 64.

Panel of Figure 6 — Figure 6: The stored record of group $g_{1}$ and the decoding of $w_{14}$ . Every field has a fixed width at a fixed offset, so the address of any weight’s state and leaf follows from its index alone. Decoding subtracts the state from the shared exponent and copies the sign and mantissa from the leaf. In the format, a block of 16 rows and 64 columns, the shape the kernel feeds to the tensor cores, is one record of $16\times 97=1{,}552$ bytes.

Edit	What changes in the tree	Familiar name
Zero a state	a leaf or node state takes the value $0$	unstructured, $N{:}M$ and structured pruning [ 43 , 29 , 23 , 70 , 54 , 95 ]
Shrink an alphabet	fewer states at the leaves or nodes	quantization [ 22 , 47 , 37 ] , ternarization [ 51 ]
Insert a shared node	a new group with a scale, shift or sign	group scales, shared exponents [ 66 ]
Split into a sum	$W\to W_{1}+W_{2}$	low rank plus residual, additive codebooks [ 16 ]
Factor into a product	$W\to UV$ or $W\to R_{1}W^{\prime}R_{2}$	low rank [ 84 , 74 , 91 , 93 ] , rotations [ 77 ] , butterfly [ 13 ]
Reference a shared part	a subtree is stored once and used many times	codebooks [ 79 , 77 , 76 ] , sparse dictionaries [ 92 ] , cross-layer sharing [ 41 , 83 , 96 , 94 ]

Table 3: Seven edits of a state-tree graph. Common compression methods, including those cited here, are combinations of these edits, together with a rule for choosing the new states.

Panel of Figure 7 — Figure 7: Compression methods as edits of a state-tree graph. Grey: zero states; green: sum ( $+$ ) and product ( $\times$ ) nodes; purple: references to shared subtrees.

Panel of Figure 8 — Figure 8: Left: the menu of one matrix in the (size, accuracy cost) plane; only its Pareto set enters the optimizer ( Property 5 ). Right: nested candidate groups for shared subtrees; every group offers either its children’s combined menus or one subtree shared by all of its matrices, and keeping Pareto sets at every node keeps the search exact. Points are illustrative.

Panel of Figure 9 — Figure 9: A layer as a circuit. When the states are powers of two, multiplications by weights become shifts and signs, which are wiring; zeros are missing wires; and each additional power-of-two term costs one adder. The only multiplications left are by the column bases and the row scales.

Component	As a state tree	In a circuit
Fully connected $Wx+b$	a tree (or graph) for $W$	shifts and adders; the bias is a constant input
Convolution	one shared subtree, referenced per position	replicated per position, or reused over time
Winograd / fast convolution	products with fixed shift-tree transforms	shifts and adders only
Attention projections	four trees; $Q,K,V$ share their input	shared partial sums across $W_{Q},W_{K},W_{V}$
Attention scores, context	products of two activations	multipliers, or a separate treatment
Gated MLP	two trees that share their input	shared partial sums; a multiplier for the gate

Table 4: How the components of common networks map onto state trees and circuits. Every product of a weight and an activation becomes shifts and additions; what remains are products of two activations and a few nonlinear functions.

Matrix	at GPTQ-4 error	at GPTQ-3 error	at GPTQ-2 error
layer 1, gate projection	1.17 $\to$ 1.05 ( $-10\%$ )	0.80 $\to$ 0.69 ( $-14\%$ )	0.42 $\to$ 0.29 ( $-30\%$ )
layer 15, gate projection	1.16 $\to$ 1.05 ( $-9\%$ )	0.80 $\to$ 0.69 ( $-14\%$ )	0.42 $\to$ 0.29 ( $-30\%$ )
layer 15, down projection	1.16 $\to$ 1.05 ( $-10\%$ )	0.79 $\to$ 0.67 ( $-15\%$ )	0.41 $\to$ 0.27 ( $-35\%$ )
layer 30, down projection	1.15 $\to$ 0.93 ( $-20\%$ )	0.79 $\to$ 0.56 ( $-29\%$ )	0.40 $\to$ 0.16 ( $-61\%$ )
layer 1, down projection	1.16 $\to$ 0.79 ( $-32\%$ )	0.80 $\to$ 0.36 ( $-55\%$ )	0.41 $\to$ 0.15 ( $-63\%$ )

Table 5: Adders per weight at equal layer error on LLaMA-3-8B: GPTQ

\to

a flat state tree whose states were chosen for circuit cost. The last two matrices have strong outliers. Counts are rounded to two decimals; percentages are computed from the unrounded counts.

Panel of Figure 10 — Figure 10: The Boltzmann machine of a state tree ( Property 6 ), drawn for two columns of a four-row version of the running example. The tree decides which units exist and where the higher-order terms are; the data decide how strongly the units interact.

Reading	Setting and metric	Baseline	State tree
Storage and serving	BF16 weights stored exactly; bits per weight	BF16: 16 bits	12.13 bits ( $-24\%$ ), bit-exact
	LLaMA-3-8B, layer 15; matrix–vector speed-up, batch 1–8	cuBLAS (A100); PyTorch (CPU)	1.05–1.51 $\times$ ; 1.24–1.27 $\times$
	same matrices and batches, format alone (bound $1.32\times$ )	our kernel on BF16 (A100)	1.03–1.27 $\times$
	vLLM end-to-end serving, Llama-3.2-1B and Qwen3.5-9B; speed-up at batch 1 / 8 / 32	vLLM, BF16	1.23 $\times$ / 1.17–1.21 $\times$ / $\approx$ 1.0 $\times$
Compression	LLaMA-3-8B at 3.25 bits per weight, no training; perplexity on WikiText-2 / C4 / FineWeb	strongest GPTQ: 8.09 / 12.72 / 11.81	6.95 / 10.93 / 9.76
Discrete optimization	Qwen2.5-1.5B-Instruct at 2.44 bits per weight, only states change; validation KL, GSM8K, MMLU	before tuning: 0.491, 13.95, 48.3	0.371, 21.83, 49.6

Table 6: One result for each reading of the tree. The strongest GPTQ uses act-order and an MSE-optimized grid in the same pipeline. Perplexity at context 2048 (FP16: 6.14 / 9.44 / 7.85); bits per weight cover all linear layers of the transformer blocks, scales included.

Matrix	Batch	cuBLAS	own BF16	tree	kernel	format	total
		( $\mu$ s)	( $\mu$ s)	( $\mu$ s)	own BF16 vs cuBLAS	tree vs own BF16	tree vs cuBLAS
gate $14336\times 4096$	1	106.0	85.6	70.3	1.24 $\times$	1.22 $\times$	1.51 $\times$
	8	107.8	84.7	72.4	1.27 $\times$	1.17 $\times$	1.49 $\times$
	16	108.9	86.6	78.4	1.26 $\times$	1.10 $\times$	1.39 $\times$
	32	117.2	98.3	113.7	1.19 $\times$	0.86 $\times$	1.03 $\times$
down $4096\times 14336$	1	102.8	89.8	70.9	1.14 $\times$	1.27 $\times$	1.45 $\times$

Table 7: Matrix–vector products of layer 15 of LLaMA-3-8B on an A100, all three kernels timed in the same run. “Own BF16” is our kernel with the same structure reading plain BF16, so kernel isolates kernel engineering and format isolates the 12.13-bit tree (bounded by

16/12.13\approx 1.32\times

); their product is the total gain over cuBLAS. At batch 32 the product is compute-bound and decoding costs more than the saved bytes gain.

Codec	Implementation	Bits per pixel	Decoding (Gpixel/s)
JPEG XL, effort 9 [ 3 ]	libjxl 0.11.2, CPU	8.66	—
Ours (tree analogue)	GPU (A100), batched	9.47	2.9
WebP, lossless [ 64 ]	libwebp 1.6.0, CPU	9.50	—
JPEG 2000 [ 72 ]	OpenJPEG 2.5.4; GPU / CPU	9.57	0.10 / $\approx$ 0.04
HTJ2K [ 73 ]	OpenJPH 0.26.3; GPU decoding	10.53	0.45

Table 8: Lossless image coding on Kodak [ 39 ] ; every result was checked to be bit-exact. GPU decoding of JPEG 2000 and HTJ2K uses nvImageCodec with the nvJPEG2000 backend. All GPU timings decode the 24 images in batched calls; one image per call, JPEG 2000 and HTJ2K reach 0.03 and 0.39 Gpixel/s. GPU timings run from compressed bytes in host memory to the decoded image in GPU memory (median of 10 runs, A100-SXM4-40GB); the CPU timing, on 16 cores of a shared AMD EPYC 7742, is indicative. For a single small image, GPU HTJ2K keeps an advantage that the batched timings do not show.

Neural Network Compression

Neural network compression reduces model size, memory, communication, or computation while preserving useful accuracy.

See all 217 papers

Sep 25, 2026

Weight Pair Encoding: Inducing a Smaller Grammar in Neural Network Weights

We show that neural network weights can be explicilty fintuned to admit a smaller grammar. Weight Pair Encoding (WeightPE) does so by placing a lossy Re-Pair compressor inside a straight-through estimator. The int8 weights of the network are flattened into one string, and near-matching Re-Pair patterns are made exactly equal within a global L2 budget. The network computes with the rewritten weights and trains through them with a straight-through estimator. Unlike a flat codebook of fixed-size entries, a grammar offers variable-length patterns and reuses them hierarchically inside larger ones. On the MLP weights of ViT-B/16 and ViT-L/16 finetuned on CIFAR-10, WeightPE produces a Re-Pair grammar 0.43x and 0.38x the size of the one produced by an equivalent int8 QAT run, at a cost of 1.9 and 1.1 accuracy points. The trend extends to different grammar compressors (LZ78, SEQUITUR), over which the networks has not be finetuned against. To our knowledge, this is the first time grammar size has been used as an explicit training objective for network weights.

Irene Tallini, Daniele Solombrino, Alberto Cazzaniga +1

Sapienza University of Rome Rome, Italy · Sapienza University of Rome and Paradigma Rome, Italy

Neural Network Compression Neural Network Quantization

Ask PDF

Sep 10, 2026

Structured Transforms for Low-Overhead Quantization of Language Models

We revisit Kashin-decomposition-based weight quantization for large language models and propose an improved algorithm with stronger convergence properties and structured, efficient orthogonal transforms. The method retains the core factorization of each weight into two components -- one with bounded infinity norm and the other with bounded infinity norm after an orthogonal transformation -- but replaces the dense random orthogonal matrix with a sign-randomized Discrete Cosine Transform (DCT), reducing the per-iteration cost from

\mathcal{O}(N^2)

\mathcal{O}(N \log N)

. The proposed greedy algorithm with alternating updates guarantees the four-peak distribution required for stable 2-bit clustering of each factor and admits closed-form initialization of cluster centers, removing the multi-restart k-means bottleneck of prior work. Composed with OPTQ-style sequential error compensation and QuIP-style incoherence preprocessing, the resulting JAX pipeline is competitive with OPTQ, QuIP, QuIP-RG and a fine-tuning- and vector-quantization-free variant of QuIP# at 4-bit per channel on OPT, Llama-2 and Pythia, with favorable wall-clock scaling. The bounded-

\ell_\infty

factorization is also notably robust: on stress configurations where QuIP variants diverge to four-digit perplexity (Pythia-6.9B) or abort with NaNs in LDL back-substitution (Mistral-7B), Kashin-DCT remains numerically stable and stays close to FP16 baseline. At inference time, each weight decomposes into two 2-bit factor codes per channel that are structurally suited to native-2-bit hardware.

Daria Cherniuk, Alexander Rudikov, Boris Kashin +1

Institute of Numerical Mathematics · Steklov Mathematical Institute

LLM Quantization Weight-Only Quantization Post-Training Quantization Low-Bit Quantization Quantization

Ask PDF

May 11, 2026

Neural Weight Norm = Kolmogorov Complexity

Why does weight decay work? We prove that, in any fixed-precision regime, the smallest weight norm of a looped neural network outputting a binary string equals the Kolmogorov complexity of that string, up to a logarithmic factor. This implies that weight decay induces a prior matching Solomonoff's universal prior, the optimal prior over computable functions, up to a polynomial factor. The result is norm-agnostic: in fixed precision, every weight norm collapses to the non-zero parameter count up to constants, so the same sandwich bound holds for any norm used as a regulariser. The proof has two short reductions: any program for a universal Turing machine can be encoded into neural weights at unit cost per program bit, and any fixed-precision network can be described by enumerating its non-zero parameters with logarithmic addressing overhead. Both bounds are tight up to constants, with the logarithmic factor realised by permutation encodings: a network whose parameters encode a permutation produces a string whose Kolmogorov complexity is the non-zero parameter count times its logarithm. The fixed-precision assumption is essential: with infinite precision, neural networks can encode non-computable functions and the weight norm loses its relevance.

Tiberiu Musat

ETH Zürich

Weight Decay Kolmogorov Complexity

Ask PDF

Abstract

Neural Network Compression

Explore similar work

Weight Pair Encoding: Inducing a Smaller Grammar in Neural Network Weights

Structured Transforms for Low-Overhead Quantization of Language Models

Neural Weight Norm = Kolmogorov Complexity