Neural Compression over Equivalence Classes: Exact Gauges and Functional Rate-Distortion
Abstract
Neural networks admit many parameterizations of the same or nearly the same function, but deployment typically stores whichever parameter vector training happens to produce. We show that this representational freedom can itself be used for compression. Rather than redesigning the codec, we search over exact function-preserving transformations and controlled functional perturbations, using the unchanged physical compressor itself to select cheaper representations. Exact gauge families provide a zero-distortion reference under the strict audit; allowing small functional changes exposes much larger rate-distortion gains. Across SmolLM2 and Qwen3, our method reduces Zstandard-compressed checkpoint size by up to 23.4%, with similar relative reductions under gzip, Brotli, and ZipNN, establishing trained-network representational freedom as a practical storage resource.
Figures & tables
| Gauge family | Representative transform | Preserved object |
|---|---|---|
| SwiGLU channel/scale | , , | MLP function |
| Attention value/output | , | value/output map |
| Low-rank factor | , | update |
| Tier | Post-storage audit | Claim used in this paper |
|---|---|---|
| Strict stored-precision | sampled maximum absolute and relative output errors are both exactly zero | algebraically function-preserving; zero error on the executed audit |
| Finite-precision tolerance | sampled stored-precision error is below a declared numerical threshold | numerically equivalent within tolerance |
| Data-conditioned functional | activation/logit/NLL distortion is measured on real text | functional rate–distortion, not exact equivalence |
| Representation class | Zstd gain | gzip gain | Brotli gain | ZipNN gain |
|---|---|---|---|---|
| Strict exact gauge | 14.4 KB | 6.1 KB | 5.0 KB | 0.3 KB |
| Finite-precision gauge | 16.2 KB | 13.5 KB | 8.8 KB | 1.4 KB |
| Weight-space | 32.5 KB | 25.0 KB | 36.7 KB | -5.0 KB |
| Block activation | 253.7 KB | 225.2 KB | 271.4 KB | 261.4 KB |
| Local activation | 1.43 MB | 1.24 MB | 1.48 MB | 1.44 MB |
| Output functional, 1-bit+ | 12.84 MB | 10.18 MB | 12.57 MB | 11.77 MB |
| SmolLM2-135M | SmolLM2-360M | |||||
|---|---|---|---|---|---|---|
| Metric | Original | Endpoint | Original | Endpoint | ||
| LAMBADA accuracy | 0.4287 | 0.4306 | +0.0019 | 0.5333 | 0.5337 | +0.0004 |
| LAMBADA perplexity | 19.0560 | 19.0044 | -0.0516 | 9.3938 | 9.3794 | -0.0144 |
| PIQA accuracy | 0.6812 | 0.6866 | +0.0054 | 0.7165 | 0.7187 | +0.0022 |
| ARC-Easy accuracy | 0.6427 | 0.6410 | -0.0017 | 0.7033 | 0.7058 | +0.0025 |
| Audit | rel. PPL | mean KL | P99 KL | P99.9 KL | top-1 |
|---|---|---|---|---|---|
| WikiText test | +0.0346% | 0.00185644 | 0.00868125 | 0.0552182 | 97.31% |
| C4 validation | +0.0624% | 0.00202928 | 0.00845627 | 0.0911304 | 97.05% |
| Representation | 135M Zstd | 360M Zstd | 135M ZipNN | 360M ZipNN | 360M mean KL | 360M top-1 |
|---|---|---|---|---|---|---|
| Strict exact gauge | 14,416 | 14,205 | 256 | 7,742 | – | – |
| Weight-space | 32,471 | 90,068 | -4,963 | -25,366 | 0.001302 | 97.72% |
| Block activation | 253,728 | 397,107 | 261,398 | 363,229 | 0.001355 | 97.71% |
| Local activation | 1,432,379 | 8,064,543 | 1,437,369 | 7,262,940 | 0.001518 | 97.72% |
| Physical objective | Admissible freedom / proposal model | Representative measured result | Why it matters |
|---|---|---|---|
| DEFLATE / gzip | soft Bellman parse + exact byte repair; exact hidden permutation | raw DEFLATE 208 168 bits; gzip 352 312 bits; seed-7 surrogate Pearson 0.93587 | differentiable rate proposal, real codec authority |
| Checkpoint delta | exact hidden permutation + SoftDelta placement | compression-aware alignment evaluated by exact decodable DiffDelta and Git binary patch | representation choice affects cross-checkpoint rate |
| Rabin dedup | exact hidden permutation + SoftDedup | exact trajectory 4,084 3,662 B; oracle recovered; max function error | extends beyond entropy/LZ coding |
| Zstandard | structured SoftZstd parser/rate model | exact Zstd 344 216 bits; Pearson 0.98916, Spearman 1.0 | codec-specific proxy can guide proposals |
| Brotli | soft LZ + context/dictionary proxy | differentiable proposal model with unchanged production Brotli final evaluation | same architecture with a different codec grammar |
| Case | Exact freedom | Physical outcome |
|---|---|---|
| Static rank-8 Qwen3 LoRA | rank-space | best 857 B of 4.29 MB ( ) |
| Qwentaur, 5 checkpoints | 784 exact gauge coordinates | 97 B of 60.57 MB ( ) |
| Synthetic scrambled history | full | 3,418 B of 6,162 B removed ( ) |
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Proposal | Construction | Shape/dtype | Exact? |
|---|---|---|---|
| 1-bit significand snap | unchanged | no | |
| alternatives | unchanged | no | |
| soft threshold, target | unchanged | no | |
| Exact gauge | algebraic reparameterization | unchanged | yes (algebraic) |
| Data source | Role | Selects updates? |
|---|---|---|
| WikiText-2 raw train | calibration / behavior gate | yes |
| WikiText-2 raw validation | held-out RD measurement | no |
| WikiText-2 raw test | final in-domain audit | no |
| C4 en validation | streamed cross-domain audit | no |
| Method | Zstd | gzip | Brotli |
|---|---|---|---|
| Strict gauge | 14,416 | 6,134 | 5,039 |
| Finite-precision gauge | 16,154 | 13,503 | 8,776 |
| Weight-space | 32,471 | 24,984 | 36,688 |
| Block activation | 253,728 | 225,237 | 271,357 |
| Local activation | 1,432,379 | 1,237,083 | 1,480,472 |
| Endpoint | NLL | PPL (%) | mean KL | P99.9 KL | top-1 agreement |
|---|---|---|---|---|---|
| Weight-space | +0.0333 | 0.001456 | 0.0516 | 97.28% | |
| Local activation | +0.0613 | 0.001669 | 0.0481 | 97.50% | |
| Block activation | +0.0222 | 0.001286 | 0.0426 | 97.71% |
| Method | Optimized for | Zstd gain | gzip gain | Brotli gain |
|---|---|---|---|---|
| Block activation | Zstd | 253,728 | 225,237 | 271,357 |
| Block activation | gzip | 253,330 | 225,361 | 271,015 |
| Block activation | Brotli | 253,227 | 225,043 | 270,984 |
| Strict gauge | Zstd | 14,416 | 6,134 | 5,039 |
| Strict gauge | gzip | 3,344 | 14,487 | 6,570 |
| Strict gauge | Brotli | 2,482 | 10,501 | 15,801 |
| Representation | 135M ZipNN bytes | 135M gain | 360M ZipNN bytes | 360M gain |
|---|---|---|---|---|
| Original | 179,380,579 | 0 | 481,538,381 | 0 |
| Strict gauge | 179,380,323 | +256 | 481,530,639 | +7,742 |
| Weight-space | 179,385,542 | -4,963 | 481,563,747 | -25,366 |
| Block activation | 179,119,181 | +261,398 | 481,175,152 | +363,229 |
| Local activation | 177,943,210 | +1,437,369 | 474,275,441 | +7,262,940 |
| Checkpoint | Raw safetensors | Zstd file | gzip file | Brotli file |
|---|---|---|---|---|
| Original | 269,060,552 | 208,496,205 | 213,332,680 | 209,496,002 |
| 1-bit+ | 269,060,520 | 195,630,069 (6.17%) | 203,141,714 (4.78%) | 197,136,946 (5.90%) |
| guarded multibit+ | 269,060,520 | 165,110,519 (20.81%) | 169,332,666 (20.63%) | 165,302,676 (21.10%) |
| Mean-KL limit | Zstd gain | accepted | val. mean KL | val. P99.9 KL | val. NLL |
|---|---|---|---|---|---|
| 1,434,249 | 12 | 0.00167586 | 0.0731186 | +0.000679794 | |
| 258,412 | 3 | 0.000946901 | 0.00779131 | +0.000242316 | |
| 0 | 0 | 0 | 0 | 0 |
| Search KL | Zstd gain | Reduction (%) | Accepted | Val. mean KL | Val. NLL |
|---|---|---|---|---|---|
| 0 B | 0.000% | 0 | 0 | 0 | |
| 0 B | 0.000% | 0 | 0 | 0 | |
| 0.128 MB | 0.062% | 1 | 0.0005136 | -0.0004320 | |
| 0.129 MB | 0.062% | 1 | 0.0007464 | -0.0009368 | |
| 0.874 MB | 0.419% | 5 | 0.0009862 | -0.0019206 | |
| 4.643 MB | 2.227% | 25 | 0.0015587 | -0.0005922 |
| Search KL | Val. mean KL | Zstd | gzip | Brotli | ZipNN |
|---|---|---|---|---|---|
| 0.0002 | 0 | 0.000% | 0.000% | 0.000% | 0.000% |
| 0.0004 | 0 | 0.000% | 0.000% | 0.000% | 0.000% |
| 0.0006 | 0.000514 | 0.062% | 0.053% | 0.064% | 0.073% |
| 0.0008 | 0.000746 | 0.062% | 0.053% | 0.064% | 0.073% |
| 0.0010 | 0.000986 | 0.419% | 0.362% | 0.425% | 0.470% |
| 0.0015 | 0.001559 | 2.227% | 1.959% | 2.246% | 2.456% |
| Search KL | Zstd gain | Reduction (%) | Accepted | Val. mean KL | Val. NLL |
|---|---|---|---|---|---|
| 0 B | 0.000% | 0 | 0 | 0 | |
| 0 B | 0.000% | 0 | 0 | 0 | |
| 0 B | 0.000% | 0 | 0 | 0 | |
| 0.403 MB | 0.072% | 1 | 0.0006521 | +0.0003250 | |
| 1.595 MB | 0.285% | 4 | 0.0009426 | +0.0004730 | |
| 13.662 MB | 2.439% | 25 | 0.0014127 | +0.0003701 |
| Search KL | Val. mean KL | Zstd | gzip | Brotli | ZipNN |
|---|---|---|---|---|---|
| 0 | 0.000% | 0.000% | 0.000% | 0.000% | |
| 0 | 0.000% | 0.000% | 0.000% | 0.000% | |
| 0 | 0.000% | 0.000% | 0.000% | 0.000% | |
| 0.0006521 | 0.072% | 0.055% | 0.071% | 0.075% | |
| 0.0009426 | 0.285% | 0.219% | 0.278% | 0.302% | |
| 0.0014127 | 2.439% | 2.049% | 2.404% | 2.548% |
| Search KL | Zstd | Wiki PPL | C4 PPL | Max raw acc. | LAMBADA PPL |
|---|---|---|---|---|---|
| 0.000% | +0.000% | +0.000% | 0.000 | +0.000% | |
| 0.000% | +0.000% | +0.000% | 0.000 | +0.000% | |
| 0.000% | +0.000% | +0.000% | 0.000 | +0.000% | |
| 0.072% | +0.028% | +0.021% | 0.380 | +0.022% | |
| 0.285% | -0.016% | -0.059% | 0.130 | -0.050% | |
| 2.439% | -0.041% | +0.075% | 0.260 | +0.047% |
| Task | Metric | Original | Endpoint | |
|---|---|---|---|---|
| ARC-Challenge | acc | +0.0000 | ||
| ARC-Challenge | acc_norm | +0.0008 | ||
| ARC-Easy | acc | -0.0017 | ||
| ARC-Easy | acc_norm | +0.0012 | ||
| HellaSwag | acc | -0.0018 | ||
| HellaSwag | acc_norm | -0.0011 |
| Task | Metric | Original | Endpoint | |
|---|---|---|---|---|
| ARC-Challenge | acc | +0.0025 | ||
| ARC-Challenge | acc_norm | -0.0043 | ||
| ARC-Easy | acc | +0.0025 | ||
| ARC-Easy | acc_norm | +0.0017 | ||
| HellaSwag | acc | -0.0013 | ||
| HellaSwag | acc_norm | -0.0002 |
Neural Network Compression
Neural network compression reduces model size, memory, communication, or computation while preserving useful accuracy.
See all 227 papers