Behavioral Capacity Certificates for Quantized Language Models

Arian Eamaz; Mojtaba Soltanalian

cs.LGSep 29, 2026

Behavioral Capacity Certificates for Quantized Language Models

Authors: Arian Eamaz, Mojtaba Soltanalian

Organizations: Department of Electrical and Computer Engineering University of Illinois Chicago

Abstract

Activation and key-value cache precision change what a quantized language model computes without altering its stored weights. Direct weight-code bounds, however, assign identical complexity to deployments that behave differently and charge separately for weight codes that behave identically. Behavioral Capacity Certificates (BCC) charge for behavior using the aggregate prior mass of complete implementations---weights, scales, activation and cache rules---that induce the same bounded loss. When quantization merges implementations, this shared mass lowers the complexity penalty, and a break-even law determines when the saving survives the cost of validating it. BCC supports a three-step deployment workflow, and our experiments verify each step. First, a forward-only screen shortlists per-layer bit-widths by how often candidate perturbations preserve the reference predictions, with quality comparable to Hessian-guided selection at lower preprocessing cost. Second, margin-certified cells identify weights that can be pruned or sign-flipped without changing the deployed behavior: every permitted combination preserves all declared predictions, and on OLMoE-1B-7B and SmolLM2-1.7B, independent probes bound the probability that any permitted combination changes a prediction on new text. Third, BCC bounds the population loss of the deployed model, nonvacuously for complete decoders and more tightly than the compressed-code route. At equal cache memory, giving keys higher precision than values yields lower NLL and higher prediction agreement on GPT-2, Qwen2.5, and SmolLM2, together with a tighter complexity bound in the GPT-2 audit.

Figures & tables

Panel of Figure 1 — Figure 1: BCC aggregates prior mass by bounded loss. Construction-derived cells additionally pay independent-probe transfer.

Weights	A4 fiber/total	A8 fiber/total	Pooled prior mass	Direct cost	BCC cost
Binary	$2/512$	$1/512$	$3/1024$	10.000	8.415
Ternary	$6/19{,}683$	$4/19{,}683$	$5/19{,}683$	15.265	11.943

Table 1: Shared behavior across activation precisions. Each row fixes a separate task on 125 inputs and a weight format. A4/A8 entries give matching weight assignments divided by the total. The prior is uniform over assignments and gives A4/A8 equal weight. Direct cost names one implementation; BCC cost names the shared behavior using the pooled prior mass. Costs are in bits.

Format	Seed	$G$ (bits)	Literal	Compressed	BCC
W4/A4	191	3206.4	0.6838	0.6328	0.6017
W4/A8	191	1459.3	0.7044	0.6647	0.6505
W4/A4	193	3107.1	0.6802	0.6285	0.5985
W4/A8	193	1550.2	0.7177	0.6723	0.6572
W4/A4	197	3125.0	0.6839	0.6355	0.6052
W4/A8	197	1440.6	0.6810	0.6363	0.6224

Table 2: Nonvacuous complete-decoder NLL bounds (bits/target;

m=530{,}000

; joint 95% coverage). Prefixes are drawn independently with replacement from the declared finite population, so Theorem 2.1 ’s independence hypothesis holds with respect to that population.

Cache	MiB/stream	NLL (nats)	Agree. (%)	$G$ (bits)
FP16/FP16	36.000	3.2137	100.00	1079.91
K8/V8	19.125	3.2135	99.22	398.19
K4/V8	14.625	3.2966	85.06	405.07
K8/V4	14.625	3.2139	95.80	593.00
K4/V4	10.125	3.2868	84.38	599.88

Table 3: Cache memory, task quality, and certified complexity at a common GPT-2 checkpoint: 64 WikiText windows, 1,008-token prefill, and 16 steps. Each row certifies a subset of its reference deployment’s behavioral fiber by varying bias codes at a fixed cache format. This supplies

G

bits of complexity reduction, giving

K_{W,A}/\ln 2\leq B-G

under the common code cost

B

Figure 2: Behavioral fibers under weight perturbations. A: A weight perturbation can move the model outside its reference fiber and change its predictions. The certified cell contains combinations of perturbations that stay inside. B: Cache precision changes the fibers. The shared cell preserves each cache setting’s own reference predictions.

Cache	Failed prefixes	Risk upper (%)	Magnitude advantage (pp)	Activation-cost advantage (pp)	Cache saved (%)
K8/V8	4/8192	0.217	0.978	0.689	48.44
K8/V4	6/8192	0.260	0.823	0.625	60.94

Table 4: OLMoE output-head edit certification at context length 256, with four teacher-forced and four greedy continuation steps. All banks contain 27,550 editable coordinates. Advantages are simultaneous lower bounds on control failure risk minus selected-family failure risk, in percentage points (pp). All bounds share

98.33\%

confidence. Cache savings include scales.

	Disagreement (%)		$\Delta$ NLL
Selector	Shakespeare	Alice	Shakespeare	Alice
BFMS: natural contexts	39.61	39.90	0.3936	0.3415
HAWQ-style trace	40.40	40.38	0.3974	0.3441
Covariance-aware Hessian	40.12	40.38	0.3907	0.3402
Weight-MSE	45.56	45.72	0.6340	0.5307

Table 5: Screening twelve four-layer references at three weight budgets with FP32 activations. Alice reuses Shakespeare-selected schedules. BFMS takes 3.40–6.15 s versus 17.58–34.13 s for Hessian. Disagreement is to FP;

\Delta

NLL is in nats.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Main result	Proof or construction	Page
Proposition 2.3	Fixed-routing factorization and feasible-set inclusion	A.2
Theorem 2.1 ; Corollary 2.2	Countable behavioral prior and positive damage	A.3
Theorem 2.4	Shared PAC–Bayes event and independent transfer	A.3
Proposition 2.5	Exact inversion of the comparison	A.3
Corollary 2.6	Strict probe-budget threshold and integer rounding	A.3
Proposition 4.1	Simultaneous coordinate edits and half-margin budgets	A.4

Appendix

Table 6: Location of the main results’ proofs. The cited pages are in this PDF.

Task seed	Coding prior	$K_{\rm dir}^{\star}$	$K_{\rm BCC}$	$U_{\rm dir}^{\star}$	$U_{\rm BCC}$
110	Uniform	11.000	9.366	0.02318	0.02101
110	Huffman	10.000	8.656	0.02185	0.02007
49	Uniform	16.265	12.943	0.03012	0.02574
49	Huffman	13.000	11.046	0.02582	0.02324

Appendix

Table 7: Behavioral aggregation with a compressed-code prior on the two declared exact tasks.

K_{\rm dir}^{\star}

is the cheapest direct charge within the zero-loss fiber;

K_{\rm BCC}

pools that fiber under the same prior. All complexities are in bits and include the W/A format tag. Uniform-code charges are ideal lengths; Huffman direct charges are actual prefix lengths. Bounds concern the bounded 0–1 loss, with the joint confidence ledger stated in the text.

Seed	Formats $m$	Mass $m/256$	$\overline{K}_{S}$	$G$	Failures	Risk upper
1	22	0.08594	3.541	4.459	116/8192	1.721%
2	29	0.11328	3.142	4.858	97/8192	1.466%
3	18	0.07031	3.830	4.170	127/8192	1.868%

Appendix

Table 8: Behavioral aggregation across W/A/K/V formats with each source checkpoint fixed. Every reported family preserves all four-token continuations on the 256 construction prefixes.

\overline{K}_{S}

and

G

are conditional charges in bits. The last column bounds the probability that any family member changes any continuation token on a fresh prefix, with joint 95% confidence across the three rows.

Format	Seed	$B_{\rm cmp}$	Test error	Test NLL	$U_{01}^{\rm lit}$	$U_{01}^{\rm cmp}$	$U_{01}^{\rm BCC}$
W4/A4	191	53,632	0.00000	0.08855	0.07418	0.06775	0.06383
W4/A8	191	54,816	0.00040	0.09309	0.07520	0.07020	0.06842
W4/A4	193	53,560	0.00000	0.08796	0.07418	0.06766	0.06387
W4/A8	193	54,248	0.00080	0.09678	0.07767	0.07196	0.07006
W4/A4	197	53,896	0.00000	0.08858	0.07427	0.06816	0.06434
W4/A8	197	54,280	0.00000	0.08810	0.07418	0.06854	0.06678

Appendix

Table 9: Complete-model task losses and absolute error certificates. The 20,000-example held-out set is separate from the certificate sample. All bounds are simultaneous under the two-loss 95% ledger and are rounded upwards; all six declared runs are retained.

B_{\rm cmp}

is the complete compressed record length.

Format	Seed	$G$ (bits)	Empirical	Raw	Compressed	BCC
W4/A4	1	6840.5	3.5146	5.3340	5.3138	5.2446
W4/A8	1	2645.6	3.3011	5.1139	5.0936	5.0670
W4/A4	2	6986.0	3.6270	5.4485	5.4289	5.3585
W4/A8	2	2589.3	3.2965	5.1091	5.0893	5.0633
W4/A4	3	6804.5	3.4993	5.3183	5.2978	5.2290
W4/A8	3	2657.0	3.2776	5.0894	5.0688	5.0421

Appendix

Table 10: Complete natural-text NLL bounds (bits/character; joint 95% coverage). One million prefixes are drawn independently with replacement from the declared finite population; 50,000 sampled indices estimate their empirical loss with charged uncertainty. All bounds are below the uniform baseline,

\log_{2}65=6.0224

. Empirical denotes the evaluated subsample; values are rounded.

Audit	Format	Probes	$G$ (bits)	$U_{\rm raw}$	$U_{\rm BCC}$
Decoder: exact 0–1	T/A8	20,000	12.000	0.02868	0.02035–0.02060
Decoder: Brier	T/A8	20,000	12.000	0.02890–0.03022	0.02065–0.02215
GPT-2 FFN	W1/A4	1,024	13.559	0.08062	0.07256
GPT-2 FFN	W1/A8	1,024	13.930	0.07851	0.06988
GPT-2 FFN	T/A4	1,024	14.820	0.08094	0.07058
GPT-2 FFN	T/A8	1,024	15.116	0.07984	0.06863

Appendix

Table 11: Risk bounds after independent transfer. Decoder ranges span three seeds; GPT-2 uses the local prior. T denotes ternary and

G

is the saving in bits. Protocols and confidence ledgers: Sections B.4 and B.5 .

Loss	Seed	$\|Q\|/U$	$\widehat{R}_{S}(Q)$	$\widehat{d}_{Z}$	$U_{\rm raw}^{\star}$	$U_{\rm BCC}$	Gain
0–1	89	$4{,}097/4{,}097$	0	$9.06\!\times\!10^{-5}$	0.02868	0.02060	0.00808
0–1	97	$4{,}097/4{,}097$	0	$4.76\!\times\!10^{-7}$	0.02868	0.02035	0.00832
0–1	101	$4{,}097/4{,}097$	0	0	0.02868	0.02035	0.00833
Brier	89	$4{,}097/4{,}097$	$1.71\!\times\!10^{-4}$	$4.91\!\times\!10^{-5}$	0.02958	0.02148	0.00810
Brier	97	$4{,}097/4{,}097$	$3.42\!\times\!10^{-4}$	$1.77\!\times\!10^{-5}$	0.03022	0.02215	0.00807
Brier	101	$4{,}097/4{,}097$	$3.74\!\times\!10^{-5}$	$7.11\!\times\!10^{-6}$	0.02890	0.02065	0.00825

Appendix

Table 12: Fresh data-selected quotient confirmation. Each block uses the conservative per-event failure probability

0.05/9

for three seeds and the PAC–Bayes core, public transfer, and raw Occam comparator; this is the predeclared primary allocation for Brier.

|Q|/U

is the posterior fraction within the selected format,

\widehat{R}_{S}(Q)

is construction Gibbs loss,

\widehat{d}_{Z}

is public mean loss distortion,

U_{\rm raw}^{\star}

is the best direct hierarchical certificate,

U_{\rm BCC}

is the transferred quotient certificate, and Gain is the bound-unit difference

U_{\rm raw}^{\star}-U_{\rm BCC}

Format	$B_{\rm full}$ (Mbit)	$M_{S}/\|\mathcal{H}\|$	$G$ (bits)	$\eta_{Z}$	$U_{\rm raw}^{\rm loc}\rightarrow U_{\rm BCC}^{\rm loc}$
W1/A4	4.719	$12{,}071/24{,}577$	13.559	0.00595	0.08062 $\rightarrow$ 0.07256
W1/A8	4.719	$15{,}611/24{,}577$	13.930	0.00601	0.07851 $\rightarrow$ 0.06988
W1.58/A4	7.479	$28{,}917/49{,}153$	14.820	0.00594	0.08094 $\rightarrow$ 0.07058
W1.58/A8	7.479	$35{,}518/49{,}153$	15.116	0.00564	0.07984 $\rightarrow$ 0.06863

Appendix

Table 13: Scale-III GPT-2-small quotient summary. Format gives weight/activation precision;

B_{\rm full}

is raw complexity of both final-FFN matrices in Mbit;

M_{S}/|\mathcal{H}|

is construction multiplicity over local-universe size;

G=\log_{2}M_{S}

is quotient saving in bits;

\eta_{Z}

is independent transfer cost; and

U_{\rm raw}^{\rm loc}\rightarrow U_{\rm BCC}^{\rm loc}

gives the direct and transferred quotient per-format 95% local-prior certificates. Both full-FFN bounds equal one.

	Shakespeare			Alice
Selector	Disag. (%)	$\Delta$ err. (pp)	$\Delta$ NLL	Disag. (%)	$\Delta$ err. (pp)	$\Delta$ NLL
BFMS: natural contexts	39.61	8.27	0.3936	39.90	7.35	0.3415
BFMS: unigram-random	39.50	8.24	0.3912	39.74	7.28	0.3392
BFMS: uniform-random	40.06	8.34	0.3901	40.40	7.37	0.3384
BFMS: Gaussian noise	39.97	8.34	0.3906	40.38	7.31	0.3387
Mean escape: natural contexts	39.26	8.23	0.3900	39.51	7.28	0.3393

Appendix

Table 14: Full-precision screening followed by matched-budget quantization. Each row averages twelve confirming references (six seeds at each width) and three mean block-weight precisions, 2.5, 3, and 3.5 bits. Disagreement is relative to the FP reference; error increase is in percentage points and NLL increase in nats. Alice uses unchanged Shakespeare-selected schedules. Natural-context BFMS with uniform step noise is the fixed extension selector; alternatives are retained controls. The random row is the exact uniform mean over each menu.

Weights	Selection rule	Blocks	$U_{440}$	$U_{6}$	$d_{\rm eval}$	$\Delta\mathrm{NLL}$
W1	Positive damage	3,8,10	0.06479	0.06042	0.04948	0.38234
W1	Teacher-JS	7,9,10	0.06867	0.06419	0.05288	0.39389
W1	HAWQ-V2 score	3,8,9	0.06706	0.06262	0.05151	0.40241
Ternary	Positive damage	1,6,9	0.04973	0.04586	0.03604	0.25209
Ternary	Teacher-JS	3,6,9	0.05041	0.04652	0.03659	0.25623
Ternary	HAWQ-V2 score	3,8,9	0.05094	0.04703	0.03742	0.25424

Appendix

Table 15: Fresh certification of six frozen GPT-2 deployments on 4,096 iid draws from the declared finite WikiText population, followed by 1,024 independent evaluation draws.

U_{440}

uses the full allocation prior;

U_{6}

uses the independently frozen finalists. Each prior receives

\delta=0.025

, giving joint 95% coverage. All bounds target normalized positive clipped-NLL damage;

\Delta\mathrm{NLL}

is token-averaged evaluation NLL increase. Costs match within each weight family.

Stage	Weights	Rule	$c$ / blocks	$U_{d}$	$d_{\rm test}$	$\Delta\mathrm{NLL}$
C	W1	Public behavioral calibration	$0.5$ / 3,8,10	0.08752	0.04260	0.31213
C	W1	Unit = checkpoint MSE	$1$ / 3,8,10	0.09219	0.04807	0.37546
C	T	Public behavioral calibration	$0.5$ / 1,6,9	0.07484	0.03365	0.21646
C	T	Unit	$1$ / 1,6,9	0.07735	0.03493	0.23384
C	T	Checkpoint MSE	$\sqrt{2}$ / 1,6,9	0.07852	0.03582	0.25464
D	W1	Calibrated sequential BCC	$0.5$ / 1,7,9	0.11124	0.03696	0.22617

Appendix

Table 16: Full independent-calibration ledger. Stage C holds blocks fixed and stage D reruns three-step interaction-aware selection.

c

is the global scale multiplier,

U_{d}

is the simultaneous 95% positive-damage certificate,

d_{\rm test}

is positive damage on the untouched 512-document holdout, and

\Delta\mathrm{NLL}

is held-out token-NLL increase. MSE chooses scale only from checkpoint weight reconstruction; HAWQ replays its frozen unit-scale allocation. Lower is better within a stage and weight family.

Tolerance	.100	.125	.150	.175	.200	.225	.250	.275	.300	.325	.350	.375	.400
Raw	0	0	2	16	26	43	52	64	75	85	92	97	100
Full mass	0	0	4	17	30	44	53	65	79	87	94	97	100

Appendix

Table 17: Exact routed composition: certified feasible seeds out of 100 at every declared risk tolerance. Full mass uses the same candidates and prior as raw coding; per-seed confidence gives joint 95% coverage. Section B.8 specifies the encoded costs.

Error	Format	$D_{*}$	$G$ (bits)	Changes
0.82772	W1/A8	50,623	50,623.0	0
0.75770	T/A8	47,676	75,564.7	0

Appendix

Table 18: A8 population-certified coordinate cells. Error is exact 0–1 loss on all 23,735 declared rows.

D_{*}

counts free head symbols and

G=D_{*}\log_{2}q

is the saving under the full uniform head prior. Changes counts prediction changes after canonicalization; no transfer probes are used.

Selection	W1/A8	T/A8	Out of 16,384 trials
	Pruning: prediction changes		Sign flips: failed patterns
Certified cell/bank	0	0	0
RMS activation	57	91	16,384
Max activation	2,218	72	16,384

Appendix

Table 19: Pruning and fault assurance on 23,735 GPT-2 contexts, with matched edit counts and displacement. Pruning removes 50,623 W1 or 45,626 ternary weights; fault trials flip 5,063 signs. Any changed prediction fails ( Sections B.9 and B.9 ).

Setting	Head upper	Deployment signed mean	Deployment signed upper	Measured edit envelope	Edit-loss upper
BASE	0.026965	+0.000000	0.000000	0.0001277	0.023285
K8/V8	0.026965	-0.000344	0.026288	0.0001276	0.023284
K8/V4	0.026965	+0.006865	0.033662	0.0001280	0.023287

Appendix

Table 20: OLMoE quality diagnostics, in nats per token. Head upper compares the frozen integer head with the raw head under BASE. Deployment changes compare each unedited deployment with BASE using the same frozen head. The measured edit envelope averages tokenwise worst-edit positive clipped-loss increases; its population upper bound applies to every head in the family.

Setting	Selected	Magnitude	Activation cost	Magnitude	Activation cost
	Failed prefixes			Lower advantage (pp)
BASE	5	133	107	0.915	0.675
K8/V8	4	137	106	0.978	0.689
K8/V4	6	125	103	0.823	0.625

Appendix

Table 21: OLMoE matched robustness controls. Failure counts are out of 8,192 prefixes. Lower advantages are in percentage points (pp) and use the complete simultaneous confidence ledger.

Cache	Failed prefixes	Risk upper (%)	Added clipped NLL upper	Cache saved (%)
BASE K16/V16	4/8192	0.228	0.024867	0.000
K8/V8	0/8192	0.121	0.024867	46.875
K8/V4	7/8192	0.292	0.024873	59.375

Appendix

Table 22: SmolLM2 output-head edit certification at context length 1,024. The same 21,874-coordinate family is used in every row. Risk concerns any permitted edit changing any required prediction in a fresh prefix. Added-loss bounds apply to every head in the family and are in nats per token. BASE is a reference row; K8/V8 and K8/V4 are primary. Displayed upper bounds are rounded upward.

	All patterns		Full pruning
Bank	K8/V8	K8/V4	K8/V8	K8/V4
Margin-selected	0	7	0	1
Magnitude	439	424	43	48
Activation cost	61	69	15	16
Random	4971	4990	1015	1023

Appendix

Table 23: Matched SmolLM2 controls: failed prefixes out of 8,192. All-pattern failure means that some permitted combination changes a required prediction. Full pruning sets every selected coordinate to zero. The simultaneous advantage statements in the text concern the all-pattern event.

Requirement	BCC	Activation cost	Magnitude	BCC risk upper bound
K8/V8	65,536	2,048	$<512$	0.904%
K8/V4	65,536	2,048	$<512$	0.935%
Shared across both	32,768	1,024	$<512$	0.630%

Appendix

Table 24: Largest certified coordinate count on the frozen grid at a 1% continuation-disagreement limit. The shared row bounds failure in either cache setting. For magnitude selection,

<512

indicates that even the smallest tested family failed the certification criteria; smaller sizes were not evaluated; the grid begins at 512. Risk bounds use the simultaneous 95% confidence ledger.

Method	K8/V8	K8/V4	Either setting	Union risk upper bound
BCC	16	11	26	0.630%
Activation cost	582	612	1,120	15.181%
Magnitude	2,258	2,356	3,412	43.758%

Appendix

Table 25: Matched-size comparison with 32,768 editable coordinates per method. Counts are failed prefixes out of 8,192. The last two columns concern failure in either cache setting.

Format	Edit law	Target	$s$	MSE ratio		Upper-bound ratio: queries
				Observed	Oracle	256	1,024	4,096	16,384
W1/A8	Substitute	Prediction	0.820	5.36	5.66	2.23	1.56	1.27	1.12
W1/A8	Substitute	$+$ error	0.942	20.10	18.33	3.24	1.93	1.38	1.18
W1/A8	Zero	Prediction	0.917	12.97	12.55	3.18	1.93	1.39	1.17
W1/A8	Zero	$+$ error	0.965	28.69	30.64	4.58	2.42	1.61	1.25
T/A8	Substitute	Prediction	0.858	6.61	7.20	2.32	1.64	1.28	1.13

Appendix

Table 26: Certified mass concentrates a perturbation audit.

s

is the certified zero-target mass; MSE ratios use 512 fresh audit seeds and 4,096 queries per route. The oracle ratio follows from the exact finite-population census. Final columns show median direct/conditional 95% upper-bound ratios at every prescribed budget. Larger ratios favor conditioning. Both damage routes exploit known nominal correctness.

Side	Bits	$G_{128}$	$G_{1008}$	Variable words, 128	Variable words, 1,008
K	16	1950.972	605.528	601	251
K	8	411.894	3.700	42	1
K	4	619.535	10.584	42	1
V	16	1602.233	474.378	588	220
V	8	984.485	394.488	90	46
V	4	1354.424	589.298	90	46

Appendix

Table 27: Certified coordinate cells. Credits add across K and V, including mixed formats. Variable-word counts are out of 768 per side. At prefixes 128 and 1,008, the constraints cover 9,216 and 65,536 first-block transitions respectively. The low-bit construction freezes all observed maximizer coordinates to preserve dynamic scales; the FP16 construction preserves half words directly.

		NLL		Agreement (%)
Model	Context	K8/V4	K4/V8	K8/V4	K4/V8
Qwen2.5-1.5B	1,024	2.1533	10.6837	91.37	0.57
Qwen2.5-1.5B	2,048	2.0986	10.5060	91.56	0.49
Qwen2.5-1.5B	4,096	2.0613	10.3682	92.25	0.73
SmolLM2-1.7B	1,024	2.0427	2.1955	95.41	81.61
SmolLM2-1.7B	2,048	1.9885	2.1168	95.46	82.87

Appendix

Table 28: Equal-memory cache allocation on two modern checkpoints. Each row uses 512 prefixes and 32 teacher-forced continuation tokens. NLL is in nats per token; agreement is with the BF16-cache reference. K8/V4 and K4/V8 use identical packed cache bytes within each row.

Model	Context	BF16 NLL	$\Delta$ NLL	MiB	Wins	Win lower (%)
Qwen2.5-1.5B	1,024	2.1203	+0.0330	11.269	512/512	99.069
Qwen2.5-1.5B	2,048	2.0614	+0.0372	22.206	512/512	99.069
Qwen2.5-1.5B	4,096	2.0281	+0.0332	44.081	512/512	99.069
SmolLM2-1.7B	1,024	2.0366	+0.0061	80.361	463/512	86.880
SmolLM2-1.7B	2,048	1.9820	+0.0065	158.361	450/512	84.028
SmolLM2-1.7B	4,096	1.9616	+0.0056	314.361	433/512	80.369

Appendix

Table 29: Additional statistics for the modern cache comparison. The main NLL and agreement comparison is in Table 28 .

\Delta

NLL is K8/V4 minus BF16, in nats per token. MiB is the equal cache allocation for K8/V4 and K4/V8. Wins count strict per-prefix NLL improvements for K8/V4; the final column is the simultaneous lower win-probability bound.

		Unsafe contexts
Cache	KV MiB	Original	FP16 subset	Old common	Extended common
K8/V4	14.625	18	18	0	0
K4/V8	14.625	25	25	3	0

Appendix

Table 30: Matched-memory cache precision and weight-cell safety on all 23,735 contexts. Both formats use the same recorded CPU runtime with unchanged checkpoint bytes. K8/V4 gives 18 under the runtime of here; the matched comparison is 18 versus 25. The full count shift is not yet attributed to a specific arithmetic setting. Entries count unsafe contexts for the original 50,623-sign bank, frozen FP16 50,594-sign bank, original 41,725-sign common bank, and extended 38,783-sign bank. KV memory includes FP32 scales and is normalized to 1,024 retained tokens; the certification prefixes contain 1–31 tokens. Costs match at every length.

	$R_{B}$		$s$		$\widehat{d}_{1}$
Format	QAT	PTQ	QAT	PTQ	QAT	PTQ
W1/A4	0.03179	0.55200	0.474	0.000	0.001547	0.008371
W1/A8	0.00440	0.54858	0.969	0.036	0.000116	0.002154
T/A4	0.00028	0.49127	0.997	0.010	0.000018	0.007001
T/A8	0.00198	0.48880	1.000	0.081	0.000024	0.001984

Appendix

Table 31: QAT/PTQ perturbation audits, averaged over three initialization seeds.

R_{B}

is independent nominal Brier loss;

s

is construction-cell coverage at tolerance

1/1024

;

\widehat{d}_{1}

is independent mean positive Brier damage under a single-symbol edit. Each comparison uses the same quantization format and edit addresses. T denotes ternary weights.

Tolerance $\epsilon$	QAT coverage	PTQ coverage	W/T/L
$0$	0.000	0.000	0/12/0
$1/4096$	0.693	0.000	11/1/0
$1/1024$	0.860	0.032	12/0/0
$1/256$	0.958	0.434	12/0/0

Appendix

Table 32: Construction-cell coverage across predeclared Brier tolerances. Means are over the 12 format–seed cases. W/T/L counts QAT coverage wins, ties, and losses against PTQ. The unchanged implementation is excluded.

Task	Members	Uniform direct	Mixture direct	Mixture BCC	Extra saving
110	7	11.000	11.000	9.387	1.613
49	10	16.265	15.498	13.062	2.436

Appendix

Table 33: Coding and behavioral savings on the existing exact tasks. All charges are in bits, including the W/A tag. Direct columns use the cheapest realization within the same zero-loss fiber. The final column is additional aggregation saving under the mixture.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 381 papers

Do We Need the Largest LLMs for Every Clinical Decision?
Oct 17, 2026Ali H. Lazem, William TeahanLLM Quantization Clinical Decision Support
School of Computer Science and Engineering, Bangor University, Bangor, LL57 2DG, Gwynedd, United Kingdom. · University of Thi-Qar, Nasiriyah, 64001, Iraq.
Linger and Lose: Knowledge Collapse in Low-Bit Language Models
Sep 26, 2026Prashanna Mani Paudel, Shivanand Venkanna SheshappanavarLow-Precision Training LLM Quantization
Geometric Intelligence Research Lab, Dept. of Electrical Engineering and Computer Science, University of Wyoming, USA.
Capability-Stratified Degradation in Ternary Language Models
Aug 28, 2026Anirudh Malik, M Sparsh Mehra, Poojith DevanLLM Quantization Ternary Quantization
OneBit AI

Oct 17, 2026J/K move · Enter open · S save

Do We Need the Largest LLMs for Every Clinical Decision?

Ali H. Lazem, William Teahan

School of Computer Science and Engineering, Bangor University, Bangor, LL57 2DG, Gwynedd, United Kingdom. · University of Thi-Qar, Nasiriyah, 64001, Iraq.

LLM Quantization Clinical Decision Support Selective Prediction

Open paper AskPDF arXiv

Large clinical language models often require multi-GPU infrastructure, limiting local deployment. We introduce CERTA, a decision-level statistical certificate that determines when a quantized model may substitute for its native BF16 reference and escalates decisions that cannot be certified to it. Across eleven instruction-tuned models, both general-purpose and domain-adapted, from three families (7-72B), we evaluated 2,500 clinical-verification decisions from de-identified patient contexts (2,484 patients), with patient-disjoint calibration and testing. In the higher-power operational analysis, round-to-nearest 8-bit quantization at BF16 storage served 99.8-100% of held-out decisions across all eleven models; formal finite-sample guarantees were assessed separately using the exact split-sample procedure. In the same operational analysis, the class-conditional 4-bit frontier reduced the amortized, BF16-equivalent deployment cost to 25.8% and 27.3% of the BF16 reference for Qwen2.5-72B and HuatuoGPT-o1-72B, respectively, with measured peak memory of 43.0 and 42.7 GB, below the nominal capacity of a single 48 GB accelerator. Deployment on that specific hardware was not directly tested. Under the exact split-sample certificate, which also accounts for configuration selection, these costs were 26.7% and 27.9%. On external disease named-entity benchmarks against human annotations, the frontier-selected NF4 configurations had a mean absolute F1 change of 0.045 on both datasets, with substantial model-specific variation. CERTA converts compression from a fixed precision choice into a risk-controlled deployment policy for this verification task.

Abstract

LLM Quantization

Explore similar work

Do We Need the Largest LLMs for Every Clinical Decision?

Linger and Lose: Knowledge Collapse in Low-Bit Language Models

Capability-Stratified Degradation in Ternary Language Models