Sable: A Small-Field Sumcheck Argument for Bit-Exact Inference of Large Language Models
Abstract
We present Sable, an argument system and single-GPU prover for the statement that a committed, integer-quantized transformer produced a given token sequence from a given prompt, bit for bit. Sable fixes an integer function F_O with no floating-point step, in which every rounding, normalization, softmax and sampling decision is determined by its inputs, and proves it with a constraint system whose relations provably never wrap modulo the 31-bit KoalaBear prime. The proof batches zerochecks and LogUp-GKR lookups into one sumcheck per domain, and commits to every witness tensor and weight in one multilinear commitment built from Reed-Solomon codes, BLAKE3 Merkle trees and a mixed-length BaseFold opening. The prover streams the model layer by layer on one GPU. On one NVIDIA RTX PRO 6000 Blackwell GPU, Sable proves all 64 layers of Qwen2.5-32B in 92.5 s for 4,096 tokens and 398 s for 16,384 tokens, and covers Qwen3-32B and the mixture-of-experts Qwen3-30B-A3B (4,096 tokens in 121.8 and 84.7 s). A proof costs 41 to 50 times the serving time, or 6.4 cents per 4,096-token request at public GPU prices, and 40 to 64 times less time per parameter and token than zkLLM. On WikiText-2, F_O's greedy token matches the BF16 model's at 86.3% of positions; on Qwen3, F_O is within 1.6 points of BF16 on MMLU and 1.7 points on GSM8K. A verifier with a compiled core checks a proof on one CPU core in 19 to 56 s. In the random-oracle model, the interactive proof has round-by-round soundness error 2^-103. The proof is not zero-knowledge, so the system targets open-weight models. Code: https://github.com/moonmath-ai/Sable
Figures & tables
| Tag | Row | Meaning |
|---|---|---|
| , | range | |
| , | , or | of an 8-bit value; after |
| , | power of two in | |
| , | regime of a shift by ( Section 6.2 ) | |
| , | input bound of a possible left shift | |
| signed normalization exponent, |
| Term | Bound per occurrence | Bits |
|---|---|---|
| Sumcheck or GKR round, degree , | ||
| Zerocheck point of a domain, variables | ||
| Batching coefficients (domain, matmul, opening) | ||
| LogUp, one chunk, independent tuple coefficients, 11-bit proof of work | ||
| Gathers, ( , group ), same proof of work | ||
| Chunk combination |
| Proof | BLAKE3 compressions | |||
|---|---|---|---|---|
| Configuration | (MiB) | Transcript | Merkle | Total |
| Before the reductions | M | M | M | |
| fixed units, recorded openings, trees per class, small-tensor packing | M | M | M | |
| packing of and the weights | M | M | M | |
| mixed-height trees | M | M | M | |
| per-layer reduction, sequential-mode transcript | M | M | M | |
| Shape | Tokens | Prove (s) | Overhead | GPU (GiB) | Host (GiB) | Proof (MiB) | Verify (s) |
|---|---|---|---|---|---|---|---|
| K | 4,096 | ||||||
| K | 16,384 | ||||||
| K | 16,384 |
| Shape | Prompt and sampling | Warm-up (s) | Timed runs (s) |
|---|---|---|---|
| K | random ids, greedy | , , | |
| K | WikiText-2, greedy | , | |
| K | random ids, temperature | , , | |
| K | random ids, greedy | , , | |
| K | random ids, greedy | , , |
| Commit pass | Prove pass (replayed) | Final passes | ||||
|---|---|---|---|---|---|---|
| Shape | span | busy | span | busy | span | busy |
| K | ||||||
| K | ||||||
| K | ||||||
| Model | Shape | Prove (s) | Overhead | GPU (GiB) | Host (GiB) | Proof (MiB) | Verify (s) |
|---|---|---|---|---|---|---|---|
| Qwen3-32B | K | ||||||
| K ∗† | |||||||
| K ∗ | |||||||
| Qwen3-30B-A3B | K | ||||||
| K | |||||||
| K |
| System | Parameters | Tokens | GPU | Prove (s) | ps/pt |
|---|---|---|---|---|---|
| zkLLM [ SLZ24 ] | 13.0B | 2,048 | A100 40GB | ||
| FlashProve [ Ant26 ] | 16.2B † | H100 80GB | |||
| Sable, Qwen2.5-32B, K | 32.5B | 4,096 | RTX PRO 6000 | ||
| Sable, Qwen2.5-32B, K | 32.5B | 16,384 | RTX PRO 6000 | ||
| Sable, Qwen2.5-32B, K | 32.5B | 16,384 | RTX PRO 6000 | ||
| Sable, Qwen3-30B-A3B, K | 3.3B † | 4,096 | RTX PRO 6000 |
| Cost model (RTX-s) | Measured | Committed elements | ||||
|---|---|---|---|---|---|---|
| Shape | Central | Band | s | Ratio | Model | Sable (trace) |
| K | – | |||||
| K | – | |||||
| K | – | |||||
| Change | Metric | Before | After |
|---|---|---|---|
| Merge same-shape domains (about faster) | 8 layers, K / K / K (s) | n/a | |
| Shared-memory softmax histogram | 8 layers, K (s) | ||
| Remove implicit host synchronizations | 8 layers, K / K / K (s) | ||
| Multiplicity compression on a side stream | 8 layers, K / K (s) | ||
| Factored lookup leaves ( Section 9.2 ) | 8 layers, three shapes (s) | ||
| Two-pass NTT up to | 26 codewords of (ms) |
| BLAKE3 compressions | ||||||
|---|---|---|---|---|---|---|
| Shape | transcript | Merkle | total | Challenges | mult. | mult. |
| K | M | M | M | K | M | M |
| K | M | M | M | K | M | M |
| K | M | M | M | K | M | M |
| WikiText-2 | ||||||
|---|---|---|---|---|---|---|
| Variant | Perplexity | over BF16 | Top-1 | Agreement | MMLU | GSM8K |
| BF16 | n/a | |||||
| FP8 (W8A8, per tensor) | ||||||
| SmoothQuant INT8 (W8A8) | ||||||
| Variant of | Perplexity | over BF16 | Top-1 | Agreement | Gap removed |
|---|---|---|---|---|---|
| float softmax | ( to ) | ||||
| 12-bit queries and keys | ( to ) | ||||
| SiLU with 4 more bits | ( to ) | ||||
| 16-bit up projection | ( to ) | ||||
| all four | ( to ) |
| WikiText-2 | |||||||
|---|---|---|---|---|---|---|---|
| Model | Variant | Perplexity | over BF16 | Top-1 | Agreement | MMLU | GSM8K |
| Qwen3-32B | BF16 | n/a | |||||
| FP8 | |||||||
| SmoothQuant INT8 | |||||||
| Qwen3-30B-A3B | BF16 | n/a | |||||
| Gap removed (paired 95% interval) | ||
|---|---|---|
| Variant of | Qwen3-32B | Qwen3-30B-A3B |
| 12-bit queries and keys | ( to ) | ( to ) |
| float softmax | ( to ) | ( to ) |
| float QK-norm | ( to ) | |
| unrounded attention input | ( to ) | ( to ) |
| unrounded MLP input | ( to ) | ( to ) |
| Undefined | |||||
|---|---|---|---|---|---|
| Input | Sequences | Tokens | Qwen2.5 | Qwen3-32B | 30B-A3B |
| WikiText-2 test, 2,048-token windows | 145 | 296,960 | 0 | 0 | 0 |
| GSM8K, 8-shot prompts | 1,319 | 1,754,311 | 0 | 0 | 0 |
| MMLU, 5-shot prompts (random 2,000) | 2,000 | 1,398,750 | 0 | 0 | 0 |
| One token repeated 4,096 times | 32 | 131,072 | 0 | 0 | 0 |
| Uniformly random token ids | 32 | 131,072 | 0 | 0 | 0 |
| Batch 1 | Batch 32 | ||||
|---|---|---|---|---|---|
| Engine | tok/s | TTFT (s) | tok/s | TTFT (s) | end to end |
| , this engine | |||||
| FP8, SGLang | |||||
| BF16, SGLang | |||||
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Relation | Largest term | Bound |
|---|---|---|
| Linear-layer accumulator | ||
| MoE down accumulator | , | |
| Attention score | ||
| , | , | |
| , | ||
| , |
| Component | Parameter | Value |
|---|---|---|
| Fields | base field | |
| IOP challenges | ||
| commitment challenges | ||
| Lookups | table | rows, 7 columns, tags of Table 1 |
| chunk size | at most static lookups per multiplicity column | |
| tuple compression | independent coefficients |
| Symbol | Meaning |
|---|---|
| , | multilinear extension; equality polynomial |
| the integer function proved ( Section 5 ) | |
| commitment to the quantized weights | |
| activation range | |
| static lookup table | |
| , | LogUp challenge and tuple-compression coefficients |
LLM Inference
Large language model inference studies how models generate outputs efficiently, reliably, and securely across hardware and workloads.
See all 321 papers