ULPack: Reference-Conditioned Lossless Compression of Checkpoint Trajectories

Aniket Srinivasan Ashok

cs.LGOct 9, 2026

ULPack: Reference-Conditioned Lossless Compression of Checkpoint Trajectories

Authors: Aniket Srinivasan Ashok

Organizations: University of Waterloo

Abstract

Studying model training requires checkpoint trajectories, whose storage cost limits temporal resolution. Although adjacent checkpoints are numerically close, existing lossless codecs either operate on XOR bit patterns or ignore trajectory-specific conditional structure. We show that a reference weight's exponent predicts the bit length of its displacement in ordered floating-point space without side information. We introduce ULPack, a lossless codec which exploits this dependence by conditioning entropy models of ordered-integer residuals on the reference exponent. Across bfloat16, float16, and float32 trajectories, ULPack reduces storage by up to 27% against ZipNN, 21% against zstd-19, and 11.5% against FM-Delta. Across two dense trajectories, compression improves as updates shrink and reference conditioning strengthens, while near uniform low-order bits set the remaining compression floor and earlier checkpoints add little information.

Figures & tables

Figure 1: Residual bit length conditioned on the exponent of the co-located reference weight. The ridge shifts systematically with reference magnitude: $e_{r}$ explains 20.2% of $H(b)$ over all 2.47 B elements of the BLOOM 90k $\rightarrow$ 91k shard-36 delta and 14.0% on complete-model MultiBERTs float32, while requiring no per-element side information.

Sequence	Dtype	Checkpoints	Evaluated unit	Payload
BLOOM (10k)	bfloat16	10k–90k	shard 36, eight deltas	4.93 GB
BLOOM (5k)	bfloat16	15k–90k	shard 36, eight matched deltas	4.93 GB
BLOOM (1k)	bfloat16	90k–93k	shard 36, three adjacent deltas	4.93 GB
Amber-7B	bfloat16	100–101	shard 2, 82 float tensors	3.50 GB
Pythia-70m	float16	100k–101k	complete model, 88 tensors	140.9 MB
MultiBERTs	float32	0k–100k	complete model, five 20k deltas	534.3 MB

Table 1: Checkpoint trajectories used for evaluation.

	BLOOM 1k	BLOOM 5k	BLOOM 10k	Amber	Pythia	MultiBERTs
ULPack	0.3255	0.5267	0.5705	0.4167	0.6012	0.7920
FM-Delta	0.3677	0.5679	0.6130	0.4660	0.6412	0.8116
ZipNN	0.4483	0.5880	0.6197	0.5152	0.6791	0.8099
XOR + zstd-19	0.4134	0.5912	0.6209	0.5053	0.6861	0.8172
margin over best baseline	11.5%	7.3%	6.9%	10.6%	6.2%	2.2%

Table 2: Compression ratio (compressed size over payload; lower is better). BLOOM 1k is the mean of three full-shard deltas from 90k to 93k; BLOOM 5k and 10k are means over eight matched full-shard deltas spanning training. MultiBERTs is its 80k–100k delta.

Sequence	Conditioning gain	Raw tail
BLOOM (1k)	11.1%	24.4%
Amber	9.7%	36.0%
Pythia-70m	6.1%	52.8%
MultiBERTs	1.6%	82.2%

Table 3: Where ULPack’s gain comes from and where it stops. Conditioning gain removes only the reference exponent; raw tail is emitted verbatim.

Figure 2: Compression ratio across training at fixed checkpoint spacing. BLOOM uses complete shard 36 with 10k-step gaps; MultiBERTs uses all available 20k-step gaps.

Figure 3: Residual-bit activity through training. BLOOM tracks the leading 18% of rows of a representative attention-QKV tensor; high-order activity recedes while low-order residual bits remain near the 50% coin-flip limit. MultiBERTs shows that convergence differs across parameter roles.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: A bfloat16 sweep under XOR (top) and ordered-integer differencing (bottom). XOR produces irregular multi-bit changes even between adjacent values, whereas ULP residuals preserve locality across exponent boundaries. Marked-cell fraction falls from 13.5% to 7.1%.

Sequence	Checkpoints	Evaluated unit	Payload
BLOOM (10k)	10k–90k	shard 36, eight deltas	4.93 GB
BLOOM (5k)	15k–90k	shard 36, eight matched deltas	4.93 GB
BLOOM (1k)	90k–93k	shard 36, three adjacent deltas	4.93 GB
Amber-7B	100–101	shard 2, all 82 float tensors	3.50 GB
Pythia-70m	100k–101k	complete model, 88 fp16 tensors	140.9 MB
MultiBERTs	0k–100k	complete model, five adjacent deltas	534.3 MB

Appendix

Table 4: Exact checkpoint pairs and evaluated payloads.

Method	Residual	Preprocessing
zstd-3, zstd-19	XOR	bytes grouped by significance
gzip-6, bzip2-9	XOR	bytes grouped by significance
ZipNN	XOR	native dtype hint; internal byte grouping
FM-Delta	ordered integer	sign/MSB-position coding; low bits raw
ULP+zstd-19 ablation	ULP	bytes grouped by significance
ULPack	ULP	exponent-conditioned two-part code

Appendix

Table 5: Inputs supplied to each runnable baseline.

Method	Compression ratio $\downarrow$	Encode (MB/s) $\uparrow$	Decode (MB/s) $\uparrow$
ULPack	0.5705	43.2	38.9
FM-Delta	0.6130	45.9	50.2
ZipNN	0.6197	432.4	876.7
XOR + zstd-19	0.6209	1.89	952.2

Appendix

Table 6: Single-thread performance on the eight full-shard BLOOM 10k-step deltas. Throughput is aggregate payload divided by aggregate wall time.

Sequence	ULPack	ULP+zstd-19	XOR+zstd-19	ZipNN	FM-Delta
BLOOM (10k)	111.8–116.7 / 125.8–129.6	—	2269.4–2887.7 / 4.1–6.3	10.1–12.1 / 3.7–6.5	105.4–108.5 / 96.1–101.1
BLOOM (5k)	112.9–121.8 / 128.4–137.8	—	1848.3–2500.5 / 3.4–6.2	11.1–12.1 / 4.7–6.5	103.7–108.6 / 95.4–101.1
BLOOM (1k)	108.6–112.8 / 127.8–130.4	1835.8–2024.9 / 4.0–4.1	1823.2–2028.8 / 4.4–4.5	11.6–11.7 / 5.7	97.3–98.3 / 89.3–91.0
Amber-7B	77.1 / 89.9	1187.4 / 3.6	1283.4 / 3.8	8.3 / 4.3	73.2 / 67.6
Pythia-70m	3.61 / 3.52	—	—	—	3.01 / 2.77
MultiBERTs	6.72 / 6.28	—	—	—	7.52 / 7.20

Appendix

Table 7: Recorded single-thread encode/decode wall times in seconds on the Linux VM; dashes denote methods not run in that experiment.

Method	Compressed bytes	Ratio	Relative margin
ULPack	226,919,912	0.6609	2.9%
FM-Delta	233,805,514	0.6809	—

Appendix

Table 8: Compression of compatible parameters in a public base-to-fine-tuned GPT-2 pair. Ratios are compressed delta size over parameter payload; lower is better.

Method	BLOOM	Amber	Pythia	MultiBERTs
XOR + zstd-3	0.4333	0.5376	0.7139	0.8328
XOR + gzip-6	0.4436	0.5315	0.7102	0.8298
XOR + bzip2-9	0.4588	0.5504	0.7206	0.8315

Appendix

Table 9: Additional compression ratios; lower is better. BLOOM is the mean of three 900.4 MB row samples and Amber is the original 629.2 MB tensor sample.

Sequence	Without $e_{r}$	ULPack	Reduction
BLOOM (1k)	0.3662	0.3255	11.1%
Amber	0.4616	0.4167	9.7%
Pythia-70m	0.6402	0.6012	6.1%
MultiBERTs	0.8051	0.7920	1.6%

Appendix

Table 10: Controlled ablation of reference-exponent conditioning. BLOOM is the mean of the three complete-shard 1k deltas. “Reduction” is the decrease in output from restoring

e_{r}

; every row reconstructs bit-exactly.

Figure 5: Per-position set frequency in XOR and ULP representations. The area between curves shows where rank-space differencing shortens the representation before context modeling.

Sequence	XOR byte bound	ULPack	Below bound
BLOOM-176B	0.4091	0.3294	19.5%
Amber	0.4941	0.4185	15.3%
Pythia-70m	0.6679	0.6012	10.0%
MultiBERTs	0.8084	0.7920	2.0%

Appendix

Table 11: Ideal order-0 byte coding of byte-grouped XOR compared with ULPack. BLOOM uses three 900.4 MB row subsets and Amber the original 629.2 MB tensor subset; these calibration units therefore do not match the complete-shard headline rows in Table 2 .

Target	ULPack	FM-Delta	XOR+zstd-19	ZipNN
20k	0.6156 / 0.6682	0.6592 / 0.7168	0.6420 / 0.6761	0.6589 / 0.6912
30k	0.5913 / 0.6348	0.6335 / 0.6798	0.6375 / 0.6569	0.6279 / 0.6778
40k	0.5702 / 0.6114	0.6115 / 0.6545	0.6282 / 0.6473	0.6120 / 0.6517
50k	0.5481 / 0.5890	0.5888 / 0.6307	0.6128 / 0.6386	0.5978 / 0.6248
60k	0.5229 / 0.5642	0.5633 / 0.6050	0.5945 / 0.6224	0.5830 / 0.6077
70k	0.4928 / 0.5352	0.5332 / 0.5754	0.5712 / 0.6041	0.5648 / 0.5900

Appendix

Table 12: Compression ratios at matched BLOOM training stages, shown as 5k / 10k.

Pair	ULPack	FM-Delta	XOR+zstd-19	ZipNN
90k–91k	0.3294	0.3716 (11.4%)	0.4170 (21.0%)	0.4515 (27.0%)
91k–92k	0.3255	0.3677 (11.5%)	0.4135 (21.3%)	0.4482 (27.4%)
92k–93k	0.3216	0.3639 (11.6%)	0.4097 (21.5%)	0.4451 (27.7%)
Mean	0.3255	0.3677 (11.5%)	0.4134 (21.3%)	0.4483 (27.4%)

Appendix

Table 13: Compression ratios on complete BLOOM shard 36 at 1k-step intervals, with ULPack’s relative reduction against each baseline in parentheses.

Sequence	$H(b_{t})$	$I(b_{t};b_{t-1})$	$I(b_{t};e_{t})$	$I(b_{t};b_{t-1}\mid e_{t})$
BLOOM-176B	3.0598	0.2979	0.6240	0.0006
MultiBERTs	3.1302	0.1594	0.4389	0.0020

Appendix

Table 14: Mutual information with current residual length, in bits per weight.

Figure 6: Phase portraits of consecutive rank-space displacements. Color shows joint density and the blue curve shows the conditional mean after a signed-log display transform. The reported correlations are computed on the raw signed displacements; their weak negative values indicate little reusable momentum beyond the immediate reference.

Sequence	$\rho$	Shipped chain	Anchor+bridge	Change
BLOOM QKV, 5k gaps	$-0.31$	17.273	17.411	$+0.80\%$
MultiBERTs, 20k gaps	$-0.23$	51.479	51.824	$+0.67\%$

Appendix

Table 15: Idealized trajectory-prediction costs in bits per weight.

	bfloat16		float32
Level	$H(q_{\ell}\mid\mathrm{ctx})$	Saving	$H(q_{\ell}\mid\mathrm{ctx})$	Saving
1	0.9170	0.47%	0.8543	0.46%
2	0.9468	0.28%	0.8864	0.36%
3	0.9515	0.22%	0.9366	0.20%
4	0.9548	0.16%	0.9095	0.28%
5	0.9797	0.05%	0.9497	0.16%

Appendix

Table 16: Conditional entropy and optimistic saving at each refinement level. Savings are percentages of uncompressed payload and exclude probability-table costs and the additional pass required by each level. The bfloat16 measurement uses a 157.3 MB Amber sample; float32 uses the complete 534.3 MB final MultiBERTs delta. Levels 1 and 2 are used by ULPack.

Lossless Data Compression

See all 26 papers

NeuralZip: Reusable Setup for Fast Lossless Compression
Oct 7, 2026Martín Bravo, Samuel Horváth, Gonzalo Navarro +1Lossless Data Compression Neural Network Compression
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Department of Computer Science, University of Chile, Santiago, Chile · National Center for Artificial Intelligence (CENIA), Santiago, Chile
Lossless Tensor Compression as Program Synthesis
Aug 3, 2026Jieke Shi, Junda He, Wenjia Jiang +11Model Compression Program Synthesis
James · Singapore Management University, Singapore · CSIRO, Australia +3
BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks
Oct 15, 2026Max Zhu, Bohang Sun, Pietro Lio +1Activation Compression Memory Compression
University of Cambridge · University of Oxford

Oct 7, 2026J/K move · Enter open · S save

NeuralZip: Reusable Setup for Fast Lossless Compression

Martín Bravo, Samuel Horváth, Gonzalo Navarro, Andrés Abeliuk

Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Department of Computer Science, University of Chile, Santiago, Chile · National Center for Artificial Intelligence (CENIA), Santiago, Chile

Lossless Data Compression Neural Network Compression Entropy Coding

Open paper AskPDF arXiv

Lossless compression can reduce the storage and movement of model weights without changing their floating-point values, but repeated statistical analysis and code construction add computational overhead. We study whether the statistical structure of exponents can be prepared once and reused. For this, we introduce NeuralZip, which groups chunks with similar exponent distributions, shares Huffman codes, and selectively represents recurring exponent tuples using packed exponents, thereby achieving additional moderate compression ratios. A setup chooses these representations before subsequent encodings, while every encoding still processes the current tensor values. In floating-point model checkpoints, post-setup compression is 1.81-21.33

\times

faster than the baselines and achieves exact bit-to-bit reconstruction. We show that this setup can be precomputed and transferred from another compatible architecture, preserving similar compression ratios and avoiding the need to amortize setup costs. Therefore, compression adaptation is transferable and reusable. Training checkpoints demonstrate continued reuse as the weights evolve. Finally, GPU experiments reduce active memory usage by up to 27.5

\%

while reproducing the logits exactly.

Abstract

Lossless Data Compression

Explore similar work

NeuralZip: Reusable Setup for Fast Lossless Compression

Lossless Tensor Compression as Program Synthesis

BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks