Do We Need the Largest LLMs for Every Clinical Decision?

Ali H. Lazem; William Teahan

cs.CLOct 17, 2026

Do We Need the Largest LLMs for Every Clinical Decision?

Authors: Ali H. Lazem, William Teahan

Organizations: School of Computer Science and Engineering, Bangor University, Bangor, LL57 2DG, Gwynedd, United Kingdom. · University of Thi-Qar, Nasiriyah, 64001, Iraq.

Abstract

Large clinical language models often require multi-GPU infrastructure, limiting local deployment. We introduce CERTA, a decision-level statistical certificate that determines when a quantized model may substitute for its native BF16 reference and escalates decisions that cannot be certified to it. Across eleven instruction-tuned models, both general-purpose and domain-adapted, from three families (7-72B), we evaluated 2,500 clinical-verification decisions from de-identified patient contexts (2,484 patients), with patient-disjoint calibration and testing. In the higher-power operational analysis, round-to-nearest 8-bit quantization at BF16 storage served 99.8-100% of held-out decisions across all eleven models; formal finite-sample guarantees were assessed separately using the exact split-sample procedure. In the same operational analysis, the class-conditional 4-bit frontier reduced the amortized, BF16-equivalent deployment cost to 25.8% and 27.3% of the BF16 reference for Qwen2.5-72B and HuatuoGPT-o1-72B, respectively, with measured peak memory of 43.0 and 42.7 GB, below the nominal capacity of a single 48 GB accelerator. Deployment on that specific hardware was not directly tested. Under the exact split-sample certificate, which also accounts for configuration selection, these costs were 26.7% and 27.9%. On external disease named-entity benchmarks against human annotations, the frontier-selected NF4 configurations had a mean absolute F1 change of 0.045 on both datasets, with substantial model-specific variation. CERTA converts compression from a fixed precision choice into a risk-controlled deployment policy for this verification task.

Figures & tables

Model	Fam.	Adapt.	Params. (B)	NF4	$\mathrm{Cov}_{\mathrm{A}}$ (%)	$\mathrm{Cov}_{\mathrm{R}}$ (%)	Overall coverage (%)	Escalation (%)	Amortized size (% BF16)	BF16 VRAM (GB)	NF4 VRAM (GB)
70–72B models
Qwen2.5-72B	Q	G	72	✓	97.4	100.0	98.9	1.1	25.8	146.3	43.0
HuatuoGPT-o1-72B	Q	D	72	✓	92.7	100.0	96.9	3.1	27.3	146.1	42.7
Med42-70B	L	D	70	✓	97.7	88.2	94.2	5.8	29.4	141.9	41.2
OpenBioLLM-70B	L	D	70	✓	98.7	0.0	92.6	7.4	30.6	141.7	41.0
Llama-3-70B	L	G	70	✓	85.9	92.3	88.1	11.9	33.9	141.9	41.2

Table 1: Class-conditional certified deployment frontier at

\alpha=5\%

(higher-power operational analysis). For each model, the configuration is selected using calibration-side information; class-conditional coverage, overall coverage and amortized size are then estimated on the held-out, patient-disjoint test set, so the costs shown are out-of-sample estimates for a test-independent selection. The table reports the lowest-amortized-cost configuration satisfying the class-conditional (Mondrian) CERTA risk constraint within each predicted decision class, with uncertifiable inputs routed to the BF16 reference. In the evaluated candidate set, all eleven frontier points select NF4 at 4 bits. Cov A and Cov R denote certified coverage within the predicted ACCEPT and REJECT classes on the held-out, patient-disjoint test set. An italicized zero means that the corresponding predicted class is fully escalated because it cannot be certified at the target risk. Overall coverage and escalation are the test-set fractions served by the quantized model and routed to the reference, respectively. Amortized size is the coverage-weighted deployment cost relative to the 16-bit BF16 reference. BF16 and NF4 are measured peak VRAM. For the seven 70–72B models, the NF4 compressed model has a measured peak below the nominal capacity of a single 48-GB accelerator (not directly tested on that hardware). The reference must remain reachable wherever the operating point permits escalation; zero observed escalation on the evaluated decisions does not remove this requirement for unseen inputs. Qwen2.5-72B and HuatuoGPT-o1-72B reach the

4

-bit floor of 25% BF16-equivalent size under the marginal frontier at

\alpha\geq 4\%

and

\alpha\geq 5\%

, and under the stricter class-conditional frontier at

\alpha\geq 8\%

and

\alpha\geq 9\%

, respectively; their class-conditional frontiers at

\alpha=5\%

are 25.8% and 27.3%.

Panel of Figure 1 — Figure 1: CERTA turns compression into a per-decision certify-or-escalate policy. For each clinical input $x$ , the quantized candidate $Q_{k}$ (at $4$ , $6$ or $8$ bits) proposes a decision, and CERTA compares its decision margin $m_{k}(x)$ against a threshold calibrated to a risk level $\alpha$ . Decisions that clear the certificate are served by $Q_{k}$ on the low-memory path; the rest are escalated to the BF16 reference $F$ . The $70$ – $72$ B reference occupies $\sim$ $142$ – $146$ GB and spans two $80$ GB accelerators, whereas the certified NF4 path has a measured peak of $\sim$ $41$ – $43$ GB, below the nominal capacity of a single $48$ GB accelerator (deployment on that specific hardware was not directly tested). The reference must remain reachable whenever the policy permits escalation. For a clinical accept/reject decision, the threshold is calibrated separately per decision class (class-conditional, Mondrian).

Figure 2: Decision certification across eleven models, the gap between raw agreement and what is statistically certifiable, and the resulting hardware footprint. a , Marginal certified coverage (%) for each model under round-to-nearest at $8$ , $6$ and $4$ bits, $8$ -bit LLM.int8() (INT8) and NF4 $4$ -bit, at $\alpha=5\%$ ; the class-conditional (Mondrian) policy used for clinical deployment is shown separately in Fig. 3 and Table 1 . Eight-bit round-to-nearest certifies at least $99.8\%$ for every model, whereas round-to-nearest $4$ -bit is strongly model-dependent. Stars mark configurations that certify at $100\%$ with no escalation; crosses mark the LLM.int8() configurations that produce degenerate outputs on two BF16-native $70$ B checkpoints (an implementation failure) and are assigned zero deployable coverage. b , Raw decision preservation versus certified coverage for every model at round-to-nearest $4$ -bit (circles) and NF4 $4$ -bit (squares), coloured by family. Points lie below the identity line because the certificate rejects the low-margin cases that raw agreement would accept; the mean gap is $14.8$ percentage points (for example Llama-3-8B at round-to-nearest $4$ -bit, raw $70.6\%$ but certified $15.0\%$ ). c , Measured peak VRAM for each of the seven $70$ – $72$ B models at BF16, $8$ -bit LLM.int8() (INT8) and NF4 $4$ -bit; the three shaded regions mark how many accelerators the footprint requires. Every model moves from $\sim$ $142$ – $146$ GB at BF16, which spans two $80$ GB GPUs, to $\sim$ $41$ – $43$ GB under NF4, below the nominal capacity of a single $48$ GB device (not directly tested on that hardware).

Figure 3: A risk level determines the deployable footprint, the certificate is calibrated, and the class-conditional guarantee costs little. a , Marginal (lower-envelope) deployment size as a function of the risk level

\alpha

for six representative models; every model retains a certified operating point across the range, and the monotonic, stable curves approach the

4

-bit floor (

25\%

, dotted) as

\alpha

relaxes. b , Calibration diagnostic: for every model and quantizer (light lines), the disagreement observed on the held-out test set at the deployed operating point stays at or below the target

\alpha

(dashed identity) across

\alpha

from

1\%

10\%

; no line enters the shaded region, the closest approach being Med42-70B under round-to-nearest

4

-bit at

\alpha=10\%

. c , Deployment size at

\alpha=5\%

under the marginal certificate versus the class-conditional (Mondrian) policy. The per-class guarantee changes size by

-0.8

+7.9

percentage points (median

+2.3

); three models escalate their entire reject class rather than certify it unsafely (labelled), and the flagship models remain near

26\%

Figure 4: The quantizer determines whether four-bit compression is certifiable. a , Raw

4

-bit decision preservation moving from round-to-nearest (open circles) to NF4 (filled, coloured by family) for each model; NF4 improves preservation for all eleven models. b , Certified coverage at

\alpha=5\%

for the same models moving from round-to-nearest (open circles) to NF4 (filled) at

4

-bit; NF4 lifts certified coverage into the

82.2\%

100\%

range (dotted line,

95\%

). The domain fine-tunes that round-to-nearest had failed, OpenBioLLM-70B (

98.1\%

), Med42-70B (

97.5\%

), MMed-Llama-3.1-70B (

96.9\%

) and BioMistral-7B (

95.8\%

), all certify at high coverage under NF4, and MMed-Llama-3-8B rises from

0\%

84.2\%

. Both panels share the same row order, so a model can be tracked across them.

Figure 5: The frontier-selected configurations preserve external task accuracy, which is complementary to, not the same as, reference agreement. a , b , Disease named-entity F1 against human annotations on BC5CDR-Disease ( a ) and NCBI-Disease ( b ) in BF16, round-to-nearest

8/6/4

-bit and the frontier-selected NF4

4

-bit configuration. Eight- and six-bit round-to-nearest quantization changes the F1 score negligibly; the frontier-selected NF4

4

-bit configurations change it comparably to round-to-nearest

4

-bit quantization (mean absolute change

0.045

on BC5CDR and

0.045

on NCBI), and MMed-Llama-3.1-70B, the worst round-to-nearest

4

-bit casualty (

-0.163

and

-0.198

), recovers under the frontier-selected NF4 configuration (

+0.078

and

+0.091

). c , NF4

4

-bit certified coverage versus the mean absolute change in F1. The per-model association is weak and the eleven configurations occupy distinct positions: the certificate concerns agreement with the reference on the verifier task, whereas the benchmark concerns correctness against human labels on a different task, so the two provide complementary evidence.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$x$	a single decision input (a candidate clinical accept/reject decision)
$d^{\mathrm{ref}}(x)$	BF16 reference decision, $\mathrm{ACCEPT}$ or $\mathrm{REJECT}$
$\hat{d}(x)$	quantized-model decision on the same input
$g(x)$	predicted decision class of the class-conditional certificate, $g(x)=\hat{d}(x)\in\{\mathrm{ACCEPT},\mathrm{REJECT}\}$ ; the two classes are abbreviated $\mathrm{A}$ (accept) and $\mathrm{R}$ (reject) in subscripts
$m_{k}(x)$	decision margin $\|z_{\mathrm{A}}-z_{\mathrm{R}}\|$ between the accept and reject answer-token logits at $k$ -bit precision
$y(x)$	disagreement indicator $\mathbf{1}[\hat{d}(x)\neq d^{\mathrm{ref}}(x)]$ (where the indicator function flags disagreements: 1 means disagreement; 0 means agreement).

Appendix

Table 1: Interpretation of quantities. Symbols and quantities used in the certificate and throughout this Supplement.

Model	Ref A	Ref R	NF4 A	NF4 R	Ref A NER/QA	NF4 A NER/QA
Qwen2.5-72B	41.2	58.8	42.8	57.2	45.0/37.4	46.2/39.4
HuatuoGPT-o1-72B	40.5	59.5	41.9	58.1	42.8/38.2	43.6/40.2
Med42-70B	62.1	37.9	62.5	37.5	83.4/40.9	84.2/40.8
OpenBioLLM-70B	91.1	8.9	93.2	6.8	99.4/82.8	100.0/86.5
MMed-Llama-3.1-70B	86.8	13.2	89.0	11.0	100.0/73.6	100.0/77.9
Llama-3.3-70B	57.5	42.5	67.2	32.8	83.4/31.7	92.2/42.1

Appendix

Table 2: Decision distribution for various models. Prevalence (%) of accept (

\mathrm{A}

) and reject (

\mathrm{R}

) decisions for the BF16 reference and the NF4 4-bit model overall and split by task (named-entity / answer verification), on the full evaluation set.

Model	RTN-8	RTN-6	RTN-4	INT8	NF4
Qwen2.5-72B	100.0 [100.0,100.0]	100.0 [100.0,100.0]	93.6 [92.1,94.9]	100.0 [100.0,100.0]	100.0 [100.0,100.0]
HuatuoGPT-o1-72B	100.0 [100.0,100.0]	100.0 [100.0,100.0]	91.0 [89.5,92.6]	100.0 [100.0,100.0]	100.0 [100.0,100.0]
Med42-70B	99.8 [99.6,100.0]	100.0 [100.0,100.0]	56.3 [53.7,59.2]	0.0 [0.0,0.0]	97.5 [96.6,98.4]
OpenBioLLM-70B	100.0 [100.0,100.0]	100.0 [100.0,100.0]	84.7 [82.6,86.6]	86.9 [85.0,88.9]	98.1 [97.3,98.8]
MMed-Llama-3.1-70B	100.0 [100.0,100.0]	93.6 [92.2,95.0]	72.1 [69.6,74.4]	74.6 [72.1,77.1]	96.9 [95.9,97.8]
Llama-3.3-70B	100.0 [100.0,100.0]	100.0 [100.0,100.0]	51.9 [49.2,54.7]	38.1 [35.4,40.8]	82.2 [80.0,84.2]

Appendix

Table 3: Per-configuration marginal certificate. Each cell is certified coverage (%) with a patient-clustered

95\%

bootstrap confidence interval, at

\alpha=5\%

\delta=0.10

. The point estimate is the certified test coverage and the brackets are patient-clustered

95\%

bootstrap intervals for that coverage; the operating-point Clopper–Pearson upper bound (Table 4 ) is the operational risk diagnostic rather than the bootstrap interval, while the formal finite-sample guarantee for the selected frontier is the split-sample certificate (Table 7 ). INT8 inference failed on two BF16-native

70

B checkpoints (Llama-3-70B and Med42-70B); these configurations are treated as infeasible and assigned zero compressed coverage, which is distinct from a statistical certification failure on a successful quantized run.

Model	$n_{\mathrm{cal}}$	disag.	$\tau$	$U_{\mathrm{CP}}$ (%)	coverage (%) [95% CI]
Qwen2.5-72B	1250	26	0.00	3.5	100.0 [100.0, 100.0]
HuatuoGPT-o1-72B	1250	33	0.00	4.2	100.0 [100.0, 100.0]
Med42-70B	1224	35	3.06	4.5	97.5 [96.6, 98.4]
OpenBioLLM-70B	1224	37	0.31	4.7	98.1 [97.3, 98.8]
MMed-Llama-3.1-70B	1223	29	0.12	3.9	96.9 [95.9, 97.8]
Llama-3.3-70B	1048	32	8.25	4.9	82.2 [80.0, 84.2]

Appendix

Table 4: Decision-risk audit for the marginal NF4 operating points (

\alpha=5\%

; higher-power operational analysis). The deployed clinical frontier is class-conditional; its per-class audit is Table 5 .

n_{\mathrm{cal}}

: certified calibration inputs (

m_{k}\geq\tau

); disag.: those disagreeing with the BF16 reference;

\tau

: selected margin threshold;

U_{\mathrm{CP}}

: one-sided Clopper–Pearson upper bound on the certified-set disagreement risk (all

\leq 5\%

); coverage: certified test coverage. The Clopper–Pearson upper bound here is the operational risk diagnostic: the threshold grid and the bound are computed on the same full calibration sample, so it is not an independent finite-sample guarantee, which is instead established by the split-sample certificate (Table 7 ). The coverage and its

95\%

bootstrap CI are an empirical deployment estimate.

	Accept ( $\mathrm{A}$ )					Reject ( $\mathrm{R}$ )
Model	$n_{\mathrm{A}}$	$x_{\mathrm{A}}$	$\tau_{\mathrm{A}}$	Cov (%)	$U_{\mathrm{CP}}$ (%)	$n_{\mathrm{R}}$	$x_{\mathrm{R}}$	$\tau_{\mathrm{R}}$	Cov (%)	$U_{\mathrm{CP}}$ (%)
Qwen2.5-72B	517	11	0.34	97.4	4.9	722	6	0.12	100.0	2.5
HuatuoGPT-o1-72B	480	10	0.54	92.7	5.0	738	8	0.12	100.0	2.8
Med42-70B	762	18	3.71	97.7	4.6	433	4	10.78	88.2	3.4
OpenBioLLM-70B	1140	29	0.38	98.7	4.3	0	0	$\infty$	esc	–
MMed-Llama-3.1-70B	1078	21	0.12	96.5	3.6	0	0	$\infty$	esc	–

Appendix

Table 5: Class-conditional (Mondrian) per-class decision-risk audit at the frontier (

\alpha=5\%

\delta/(2G)

; higher-power operational analysis). For each decision class:

n

, certified calibration inputs;

x

, disagreements among them;

\tau

, selected margin threshold (

\infty

= class escalated in full); Cov, certified test coverage;

U_{\mathrm{CP}}

, one-sided Clopper–Pearson upper bound on the class risk (all

\leq 5\%

). esc : reject class escalated. Each class certificate is reconstructible from

n

x

and

\tau

. As in Table 4 , the threshold grid and the Clopper–Pearson bounds use the same full calibration sample, so these per-class bounds are operational diagnostics rather than independent finite-sample certificates; the exact split-sample per-class certificate (independent selection subset, one decision per patient,

\delta/(2GH)

allocation) is Table 8 .

Model	$n_{\mathrm{A}}$	$n_{\mathrm{R}}$	$\alpha_{\min,\mathrm{A}}$ (%)	$\alpha_{\min,\mathrm{R}}$ (%)	min. $U_{\mathrm{CP},\mathrm{R}}$ (%)	mechanism
Qwen2.5-72B	528	722	1.30	0.95	1.0	certified
HuatuoGPT-o1-72B	512	738	1.34	0.93	1.1	certified
Med42-70B	778	472	0.88	1.45	1.7	certified
OpenBioLLM-70B	1159	91	0.59	7.31	13.7	resolution-limited
MMed-Llama-3.1-70B	1105	145	0.62	4.65	8.4	risk-limited
Llama-3.3-70B	830	420	0.83	1.63	1.6	certified

Appendix

Table 6: Class-specific calibration resolution and escalation mechanism for the higher-power operational NF4 frontier.

n_{\mathrm{A}}

n_{\mathrm{R}}

: calibration inputs in each predicted class;

\alpha_{\min}

: zero-disagreement resolution floor

1-(\delta/2G)^{1/n}

; min.

U_{\mathrm{CP},\mathrm{R}}

: smallest Clopper–Pearson upper bound on the reject-class risk attainable over the threshold grid; mechanism: whether the reject class is certified , resolution-limited (floor

>\alpha

), or risk-limited (floor

\leq\alpha

but best attainable bound

>\alpha

). Target

\alpha=5\%

. The floors use the operational

\delta/(2G)

allocation on the full calibration sample, without the configuration-selection factor

H

; this is an operational diagnostic, distinct from the exact split-sample certificate (Table 8 ).

Model	Full-cal. (% BF16)	Exact (% BF16)	Diff. (pp)	Frontier config	Exact certified?
Qwen2.5-72B	25.8	26.7	+0.9	NF4	Yes
HuatuoGPT-o1-72B	27.3	27.9	+0.6	NF4	Yes
Med42-70B	29.4	30.6	+1.3	NF4	Yes
OpenBioLLM-70B	30.6	31.5	+1.0	NF4	Yes
MMed-Llama-3.1-70B	35.2	35.2	+0.0	NF4	Yes
Llama-3.3-70B	39.1	41.4	+2.3	NF4	Yes

Appendix

Table 7: Split-sample exactness. Class-conditional deployment frontier (% BF16,

\alpha=5\%

) under the reported full-calibration certificate (grid and Clopper–Pearson bounds on all

1{,}250

calibration decisions) and under the exact certificate (grid on the

250

-decision selection subset, Clopper–Pearson bounds on the

1{,}000

-decision certification subset reduced to one decision per patient,

996

decisions, and an outer

\delta/H

Bonferroni across the

H=2

deployable configurations; subsets mutually patient-disjoint). The difference column is exact minus full computed on the unrounded sizes and then rounded, so it may differ by

0.1

from the difference of the two displayed (rounded) columns. Every model selects the same NF4

4

-bit deployable configuration and remains certified under the exact procedure.

	Accept ( $\mathrm{A}$ )					Reject ( $\mathrm{R}$ )
Model	$n_{\mathrm{A}}$	$x_{\mathrm{A}}$	$\tau_{\mathrm{A}}$	Cov (%)	$U_{\mathrm{CP}}$ (%)	$n_{\mathrm{R}}$	$x_{\mathrm{R}}$	$\tau_{\mathrm{R}}$	Cov (%)	$U_{\mathrm{CP}}$ (%)
Qwen2.5-72B	406	6	0.51	96.3	4.6	565	3	0.50	98.7	2.4
HuatuoGPT-o1-72B	378	5	0.54	92.7	4.5	576	4	0.38	98.6	2.7
Med42-70B	591	9	8.32	94.1	4.0	345	4	9.39	89.7	4.5
OpenBioLLM-70B	895	11	0.70	97.4	3.0	0	0	$\infty$	esc	–
MMed-Llama-3.1-70B	861	16	0.05	96.5	3.9	0	0	$\infty$	esc	–

Appendix

Table 8: Exact split-sample per-class certificate at the frontier (

\alpha=5\%

). The auditable formal guarantee underlying Table 7 : for each model, the class-conditional certificate computed by the exact split-sample procedure (candidate grid on the

250

-decision selection subset; Clopper–Pearson bounds on the one-decision-per-patient certification subset; outer Bonferroni allocation

\delta/(2GH)

with

H=2

deployable configurations). For each decision class:

n

, served certification observations;

x

, disagreements among them;

\tau

, selected margin threshold; Cov, certified test coverage;

U_{\mathrm{CP}}

, one-sided Clopper–Pearson upper bound on the class risk. Every served class satisfies

U_{\mathrm{CP}}\leq 5\%

. esc (

\tau=\infty

): reject class escalated in full, so no compressed decisions are served and the class contributes no compression-induced disagreement. Each class certificate is reconstructible from

n

x

and

\tau

	Marginal			Class-conditional
Model	$\delta{=}.10$ (90%)	$\delta{=}.05$ (95%)	$\delta{=}.01$ (99%)	$\delta{=}.10$ (90%)	$\delta{=}.05$ (95%)	$\delta{=}.01$ (99%)
Qwen2.5-72B	25.0	25.0	25.0	25.8	26.2	26.2
HuatuoGPT-o1-72B	25.0	25.0	25.0	27.3	27.6	28.2
Med42-70B	26.9	26.9	26.9	29.4	29.4	30.6
OpenBioLLM-70B	26.4	26.4	27.6	30.6	30.6	30.6
MMed-Llama-3.1-70B	27.3	27.3	27.3	35.2	35.2	35.2

Appendix

Table 9: Confidence-level sensitivity (marginal and class-conditional). Frontier size (% BF16) at

\alpha=5\%

for three confidence levels, for the marginal certificate and the class-conditional (Mondrian) policy. Both change only modestly across

\delta

$n_{\mathrm{cal}}$	min ( $4\times$ floor)	median	max (least compressible)
250	25.4	31.1	49.5
500	25.0	27.9	44.5
750	25.0	28.0	40.9
1000	25.0	28.2	40.3
1250	25.0	28.2	38.4

Appendix

Table 10: Calibration-size sensitivity. Marginal frontier range (% BF16) at

\alpha=5\%

across the eleven models, as the patient-disjoint calibration set is reduced from

1{,}250

250

decisions; the certificate degrades gracefully and the strongest models retain the

4

-bit floor.

$G$	min ( $4\times$ floor)	median	max (least compr.)	Qwen2.5 72B	Huatuo o1-72B	Llama 3.3-70B
25	25.0	29.4	38.4	25.0	25.0	38.4
50	25.0	28.2	38.4	25.0	25.0	38.4
100	25.0	28.2	39.1	25.0	25.0	39.1
200	25.0	28.2	39.1	25.0	25.0	39.1

Appendix

Table 11: Threshold-grid sensitivity. Marginal frontier size (% BF16) at

\alpha=5\%

\delta=0.10

as the Learn-then-Test grid size

G

is varied. The first three value columns are the minimum, median and maximum frontier across the eleven models; the last three are three representative models. The frontier is nearly invariant to

G

Model	median	IQR (25–75%)	min	max
Qwen2.5-72B	25.0	25.0–25.0	25.0	25.0
HuatuoGPT-o1-72B	25.0	25.0–25.0	25.0	25.8
Med42-70B	27.8	26.8–28.3	25.0	29.3
OpenBioLLM-70B	27.2	26.6–27.6	25.7	27.9
MMed-Llama-3.1-70B	26.7	25.0–26.9	25.0	27.3
Llama-3.3-70B	38.4	36.8–39.0	35.8	40.7

Appendix

Table 12: Split stability. Marginal frontier size (% BF16) at

\alpha=5\%

\delta=0.10

across

20

independent patient-disjoint calibration/test splits: per-model median, interquartile range and full range. The frontier is stable across resamples; the strongest models stay at the

4

-bit floor on every split.

	$\alpha=1\%$		$\alpha=5\%$		$\alpha=10\%$
Model	size	esc.	size	esc.	size	esc.
Qwen2.5-72B	59.2	45.6	25.8	1.1	25.0	0.0
HuatuoGPT-o1-72B	72.6	45.3	27.3	3.1	25.0	0.0
Med42-70B	100.0	100.0	29.4	5.8	25.3	0.4
OpenBioLLM-70B	40.2	20.3	30.6	7.4	29.7	6.2
MMed-Llama-3.1-70B	41.3	21.7	35.2	13.6	26.6	2.1

Appendix

Table 13: Risk-level operating points (class-conditional). Class-conditional (Mondrian) frontier size (% BF16) and escalation fraction (%) at three risk targets

\alpha\in\{1,5,10\}\%

(

\delta=0.10

\delta/(2G)

allocation). Relaxing

\alpha

lowers both the deployed size and the escalation fraction toward the

4

-bit floor. The deployable configuration that minimizes amortized size is selected independently at each risk level, so the serving configuration can change with

\alpha

(see text); a size of

100\%

denotes no certified operating point at that target, for which all decisions escalate to BF16.

Model	Policy	$\mathrm{REJECT}$ prevalence (%)
		50	40	30	20	10	5
Maximum over models	marginal	9.5	8.0	6.6	5.7	6.3	6.6
	Mondrian	2.7	2.8	3.1	3.5	3.8	3.9
Mean over models	marginal	3.7	3.5	3.4	3.3	3.1	3.1
	Mondrian	1.8	1.8	1.9	2.0	2.0	2.1
Qwen2.5-72B	marginal	2.6	3.2	3.7	4.2	4.7	4.9

Appendix

Table 14: Prevalence-shift stress test. Certified-set disagreement (%) versus deployment reject-class prevalence, for the marginal and class-conditional (Mondrian) policies with fixed calibrated thresholds. Values above the

\alpha=5\%

target are the marginal certificate’s prevalence-induced violations; the class-conditional policy stays below target throughout.

Configuration	NER coverage	NER disag.	QA coverage	QA disag.
RTN-8	100.0	1.1	100.0	1.4
RTN-6	99.7	2.0	94.7	2.6
RTN-4	60.0	1.9	56.5	3.7
INT8	74.5	1.3	68.7	3.0
NF4	94.6	2.3	91.5	3.8

Appendix

Table 15: Task-wise certification. Certified coverage and observed certified-set disagreement (%) for named-entity (NER) and answer (QA) verification at

\alpha=5\%

\delta=0.10

(higher-power operational analysis). Coverage is averaged over all eleven models, assigning zero coverage to any configuration that certifies no decisions for a model. Certified-set disagreement is undefined for a model that serves no decisions, so the disagreement columns are averaged only over the models with a non-empty certified set: this is all eleven models for RTN-8, RTN-6 and NF4; nine models for INT8 (the two whose LLM.int8() configurations failed to produce usable inference outputs, Llama-3-70B and Med42-70B, serve nothing); and nine (NER) and ten (QA) models for RTN-4.

Model	F1 BF16	F1 RTN-8	F1 RTN-6	F1 RTN-4	F1 NF4	NF4 P/R
BC5CDR-Disease
Qwen2.5-72B	0.720	0.720	0.735	0.669	0.713	0.956/0.568
HuatuoGPT-o1-72B	0.573	0.585	0.645	0.615	0.467	0.947/0.310
Med42-70B	0.852	0.853	0.852	0.694	0.829	0.895/0.773
OpenBioLLM-70B	0.705	0.705	0.710	0.684	0.573	0.925/0.415
MMed-Llama-3.1-70B	0.469	0.475	0.506	0.306	0.547	0.624/0.487

Appendix

Table 16: External disease named-entity results. F1 against human annotations at BF16 and under each configuration; the last column is precision/recall for the deployed NF4

4

-bit configuration.

Benchmark	Config	$\Delta$ P	$\Delta$ R	$\Delta$ F1	mean $\|\Delta\mathrm{F1}\|$
BC5CDR-Disease	RTN-8	$+0.004$ $[-0.001,+0.009]$	$+0.005$ $[-0.001,+0.010]$	$+0.004$ $[-0.000,+0.010]$	$+0.005$ $[+0.005,+0.013]$
	RTN-6	$-0.000$ $[-0.009,+0.008]$	$+0.019$ $[+0.011,+0.028]$	$+0.016$ $[+0.009,+0.024]$	$+0.019$ $[+0.015,+0.027]$
	RTN-4	$-0.040$ $[-0.053,-0.027]$	$-0.017$ $[-0.036,+0.003]$	$-0.022$ $[-0.038,-0.006]$	$+0.057$ $[+0.046,+0.074]$
	NF4	$-0.006$ $[-0.017,+0.005]$	$-0.011$ $[-0.025,+0.004]$	$-0.016$ $[-0.028,-0.003]$	$+0.045$ $[+0.037,+0.060]$
NCBI-Disease	RTN-8	$-0.002$ $[-0.010,+0.005]$	$-0.000$ $[-0.007,+0.006]$	$-0.001$ $[-0.008,+0.006]$	$+0.011$ $[+0.007,+0.019]$
	RTN-6	$+0.003$ $[-0.009,+0.014]$	$+0.002$ $[-0.006,+0.010]$	$+0.002$ $[-0.007,+0.010]$	$+0.018$ $[+0.012,+0.031]$

Appendix

Table 17: Paired human-gold differences on the external benchmarks. Entity-level precision, recall and F1 of each configuration minus the same model’s BF16 run on the same documents, averaged over the eleven models, with document-level

95\%

bootstrap intervals (

1{,}000

resamples); matching is the overlap scorer used throughout.

|\Delta\mathrm{F1}|

is the mean absolute per-model F1 change.

Benchmark	Config	Preserved covered	Newly covered	Lost coverage	Stayed missed	Net coverage	$\Delta$ FP
BC5CDR-Disease	RTN-8	2964	55	34	1600	$+21$ $[-2,+43]$	$-16$ $[-35,+5]$
	RTN-6	2938	138	60	1517	$+78$ $[+40,+118]$	$+12$ $[-23,+48]$
	RTN-4	2564	363	434	1292	$-71$ $[-163,+22]$	$+206$ $[+136,+286]$
	NF4	2733	217	265	1438	$-48$ $[-107,+21]$	$+82$ $[+33,+135]$
NCBI-Disease	RTN-8	2763	73	71	3000	$+2$ $[-41,+43]$	$+0$ $[-24,+26]$
	RTN-6	2727	123	107	2950	$+16$ $[-34,+64]$	$-8$ $[-62,+44]$

Appendix

Table 18: Gold-entity coverage transitions on the external benchmarks. For each configuration versus the same model’s BF16 run, pooled over models and documents (recall side): gold entities both recover (preserved), newly recovered, lost, or still missed; net coverage change with its

95\%

bootstrap interval; and

\Delta

FP, the change in false positives (precision side). Overlap matching.

Model	Family	Adapt.	Params (B)	HuggingFace checkpoint †	Compute dtype	BF16 (GB)	NF4 (GB)
Qwen2.5-72B	Qwen	G	72	Qwen/Qwen2.5-72B-Instruct	BF16	146.3	43.0
HuatuoGPT-o1-72B	Qwen	D	72	FreedomIntelligence/HuatuoGPT-o1-72B	BF16	146.1	42.7
Med42-70B	Llama	D	70	m42-health/Llama3-Med42-70B	BF16	141.9	41.2
OpenBioLLM-70B	Llama	D	70	aaditya/Llama3-OpenBioLLM-70B	BF16	141.7	41.0
Llama-3-70B	Llama	G	70	meta-llama/Meta-Llama-3-70B-Instruct	BF16	141.9	41.2
MMed-Llama-3.1-70B	Llama	D	70	Henrychur/MMed-Llama3.1-70B	BF16	141.9	41.2

Appendix

Table 19: Model and inference configuration. The eleven evaluated checkpoints with architecture/model family (Q, Qwen; L, Llama; M, Mistral), adaptation (G, general-purpose; D, domain-adapted), parameter count, released checkpoint identifier, † and measured peak VRAM at the BF16 reference and at the deployed NF4

4

-bit configuration. Every model was evaluated under the same quantizer set (round-to-nearest

8/6/4

-bit and the bitsandbytes NF4

4

-bit and LLM.int8() routes) with BF16 compute; the deployed frontier configuration is NF4

4

-bit for all eleven models.

Figure 1: Certified compression frontiers for all eleven models under the marginal certificate. Marginal deployment size (% BF16) as a function of the risk level

\alpha

; each curve is one model, coloured by architecture family. The curves are monotonic and stable and settle toward the

4

-bit floor (

25\%

, dotted) as

\alpha

relaxes. This is the full version of the representative subset in main-text Fig. 3a.

Figure 2: Prevalence-shift stress test, all eleven models. Certified-set disagreement versus deployment reject-class prevalence, with calibrated thresholds held fixed, for the (a) , marginal and (b) , class-conditional (Mondrian) policies. All eleven models are shown (three highlighted, the other eight in grey). Under the marginal certificate several models enter the shaded risk-level violation zone above

\alpha=5\%

(the maximum over models reaches

9.5\%

); under the class-conditional policy every model stays below

\alpha

at every prevalence (maximum over models

3.9\%

; Supplementary Table 14 ). The direction of the marginal drift depends on which decision class carries more disagreement for each model.

Figure 3: Escalated cases are enriched for low-margin, high-disagreement inputs (NF4

4

-bit,

\alpha=5\%

). (a) , Empirical cumulative distribution of the decision margin

m_{k}(x)

, pooled over the eleven models, for escalated versus certified inputs; the distributions are strongly separated (median margin

0.35

escalated versus

5.12

certified), not merely different in mean. (b) , Observed disagreement with the BF16 reference for certified (filled) versus escalated (open) inputs, per model: escalated inputs disagree with the reference far more often (mean

45.3\%

versus

2.9\%

; dotted line,

5\%

). Escalation therefore selects the decisions on which the compressed model is most likely to disagree with the BF16 reference; whether those disagreements are clinically consequential is the object of the future clinician-adjudication study.

Figure 4: Two mechanisms of reject-class escalation. (a) , Number of

\mathrm{REJECT}

-class calibration inputs per model (log scale), coloured by escalation mechanism: certified (green), risk-limited (orange), resolution-limited (red). Acceptance-skewed models have very few reject-class calibration inputs (BioMistral-7B,

4

; OpenBioLLM-70B,

91

). (b) , Reject-class resolution floor

\alpha_{\min,\mathrm{R}}=1-(\delta/2G)^{1/n}

(horizontal axis) against the smallest attainable Clopper–Pearson upper bound on the reject-class risk (vertical axis), both on log scales; dashed lines mark the

\alpha=5\%

target. A class is certified only when both coordinates fall below

\alpha

(lower-left corner). Resolution-limited classes (BioMistral-7B, OpenBioLLM-70B) fail the floor condition; the risk-limited class (MMed-Llama-3.1-70B) clears the floor at

4.7\%

but not the attainable bound at

8.4\%

. The floor is a necessary, not a sufficient, certification condition.

Figure 5: Statistical robustness. (a) , Marginal frontier size (% BF16) at

\alpha=5\%

for confidence levels

\delta\in\{0.10,0.05,0.01\}

(confidence

90/95/99\%

); each line is one model. The frontier changes only modestly with the confidence level. (b) , Frontier range (minimum, median, maximum over models) at

\alpha=5\%

as a function of the calibration set size

n_{\mathrm{cal}}

; the certificate degrades gracefully as calibration data are reduced and tightens as they increase, while the strongest models retain the

4

-bit floor.

Figure 6: Design-choice robustness of the marginal frontier. (a) , Marginal frontier size (% BF16) at

\alpha=5\%

as the Learn-then-Test grid size

G

is varied over

\{25,50,100,200\}

: minimum (

4\times

floor), median and maximum across the eleven models. The envelope is nearly flat, so the frontier is insensitive to the grid resolution. (b) , Distribution of the marginal frontier over

20

independent patient-disjoint splits, one boxplot per model (boxes coloured by architecture family, whiskers the full range; dotted line, the

25\%

4

-bit floor). The strongest models sit at the floor on every split and no model’s interquartile width exceeds

2.5

percentage points, confirming that the

25

–

40\%

envelope is not an artefact of the particular split used in the main text.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 381 papers

Sep 29, 2026

Behavioral Capacity Certificates for Quantized Language Models

Activation and key-value cache precision change what a quantized language model computes without altering its stored weights. Direct weight-code bounds, however, assign identical complexity to deployments that behave differently and charge separately for weight codes that behave identically. Behavioral Capacity Certificates (BCC) charge for behavior using the aggregate prior mass of complete implementations---weights, scales, activation and cache rules---that induce the same bounded loss. When quantization merges implementations, this shared mass lowers the complexity penalty, and a break-even law determines when the saving survives the cost of validating it. BCC supports a three-step deployment workflow, and our experiments verify each step. First, a forward-only screen shortlists per-layer bit-widths by how often candidate perturbations preserve the reference predictions, with quality comparable to Hessian-guided selection at lower preprocessing cost. Second, margin-certified cells identify weights that can be pruned or sign-flipped without changing the deployed behavior: every permitted combination preserves all declared predictions, and on OLMoE-1B-7B and SmolLM2-1.7B, independent probes bound the probability that any permitted combination changes a prediction on new text. Third, BCC bounds the population loss of the deployed model, nonvacuously for complete decoders and more tightly than the compressed-code route. At equal cache memory, giving keys higher precision than values yields lower NLL and higher prediction agreement on GPT-2, Qwen2.5, and SmolLM2, together with a tighter complexity bound in the GPT-2 audit.

Arian Eamaz, Mojtaba Soltanalian

Department of Electrical and Computer Engineering University of Illinois Chicago

LLM Quantization KV-Cache Quantization Activation Quantization PAC-Bayesian Generalization Bounds Mixed-Precision Quantization

Ask PDF

Sep 2, 2026

The Effect of Quantization on Clinical Benchmarks: Accuracy and Safety Across Model Families

Quantization enables deployment of large language models on resource-constrained clinical edge devices, but its effect on clinical accuracy and safety remains understudied. We evaluate five 7-8B parameter models at FP16, GPTQ-INT8, and GPTQ-INT4 precision across five benchmarks: MedQA, MedMCQA, Med-HALT, a risk-stratified sample of HealthBench, and MedSafetyBench. The study jointly varies quantization bit width, model family, and clinical task type, with explicit risk stratification and safety measures. INT8 GPTQ is universally safe (max. degradation -1.9%-1.9%), while INT4 degradation is substantial and model-dependent: BioMistral-7B, clinically fine-tuned, loses 19.7% on MedMCQA, more than any general-purpose model, showing clinical fine-tuning does not confer compression robustness. MedMCQA degrades more than MedQA under INT4; Med-HALT is largely unaffected. On HealthBench's emergency-risk subgroup, Qwen2.5-7B degrades by 26.8% under INT4, suggesting high-risk scenarios are disproportionately vulnerable to compression. On MedSafetyBench, the model family dominates over precision (refusal rates range 10.2%-74.9% at FP16), though Qwen2.5-7B (-17.8%) and Meditron-7B (-28.3%) show substantial INT4 safety degradation; notably, Qwen2.5-7B is simultaneously the most accuracy-robust model, demonstrating that accuracy and safety robustness are independent properties. We additionally test two recovery methods, clinical calibration substitution and QLoRA fine-tuning, both producing the same trade-off: MedMCQA recovers while MedQA further degrades, indicating recovery strategies require task-specific validation rather than being assumed universally beneficial. These findings indicate INT8 is broadly safe for clinical deployment, while INT4 safety must be assessed per-model and per-task, and that safety alignment is determined primarily by instruction tuning rather than clinical domain adaptation.

Leonard Twagirayezu, Prasenjit Mitra

Kigali, Rwanda

LLM Quantization Post-Training Quantization Clinical Language Model Evaluation LLM Safety Benchmarks

Ask PDF

Aug 5, 2026

Provable Limits and Certified Deferral for Verbalized Uncertainty in Small Language Models

Small open-weight language models increasingly run in private, offline, and cost-sensitive settings, where the key deployment question is not only what a model answers but when it should defer to a human. We study whether verbalized confidence can support risk-controlled deferral, evaluating eleven instruction-tuned models from three families, 0.5B to 14B parameters, on ARC-Challenge and TruthfulQA with 25,168 local predictions. Three theoretical results delimit what calibration can provide: strictly monotone calibration preserves the risk-coverage frontier and error-detection AUROC; temperature scaling cannot calibrate models whose confidence stays above one half while accuracy falls below it; and a Clopper-Pearson procedure converts a 200-question calibration set into a finite-sample risk certificate under an i.i.d. deployment assumption. Empirically, eight of 22 model-task pairs hit the temperature-scaling infeasibility floor within one percentage point of the predicted bound. Platt scaling reduces ECE to as low as 0.02, yet certified autonomy at a 20% risk budget is granted to only three model-task pairs and to none at 10%. We also identify and repair an answer-ordering artifact in the multiple-choice form of TruthfulQA. Calibration gives confidence semantics; certified deferral determines when small models are safe to use.

Jianru Shen

University of Montana, Missoula, MT 59812, USA

Selective Prediction Confidence Estimation in Language Models Small Language Models Language Model Calibration LLM Uncertainty Estimation

Ask PDF

Abstract

LLM Quantization

Explore similar work

Behavioral Capacity Certificates for Quantized Language Models

The Effect of Quantization on Clinical Benchmarks: Accuracy and Safety Across Model Families

Provable Limits and Certified Deferral for Verbalized Uncertainty in Small Language Models