HeadGuard: Selective Head Protection for Low-Bit VLM KV-Cache Quantization

Nenad Banfic

cs.LGSep 18, 2026

HeadGuard: Selective Head Protection for Low-Bit VLM KV-Cache Quantization

Authors: Nenad Banfic

Organizations: CoreAI, Microsoft

Abstract

Low-bit key-value (KV) cache quantization saves storage but can sharply degrade vision-language model (VLM) accuracy. We introduce HeadGuard, a composable head-protection method that augments a base KV-cache quantizer with a fixed high-precision mask. Image-sensitivity and output-sensitivity scores select physical KV heads offline, with approximately 1/8 protected in the main experiments; their image keys and optionally values remain in bfloat16 (BF16), while the base quantizes unprotected image entries. Across eight VLMs, three base quantizers, and eight benchmarks (six discriminative and two generative), HeadGuard recovers a substantial fraction of lost accuracy on weaker quantizers, with the strongest gains for Qwen and InternVL. At 2 bits, the six-task discriminative mean over eight models rises from 0.436 to 0.580 on the weakest base; protection can also improve generated answers and caption fidelity to BF16 outputs. Mean accuracy gains persist across all three quantizers with both tested calibration datasets. Keys-only protection retains substantial recovery at lower modeled storage cost. Evaluated through simulated quantization, HeadGuard offers a composable way to improve low-bit VLM accuracy without replacing the underlying quantizer.

Figures & tables

Model	Uniform	HeadGuard	Uniform	HeadGuard	BF16
	$2$ -bit		$3$ -bit
LLaVA-1.5	0.513	0.579	0.641	0.642	0.643
LLaVA-NeXT	0.621	0.657	0.667	0.666	0.665
Qwen2.5-VL	0.360	0.747	0.361	0.781	0.790
Qwen2-VL	0.363	0.598	0.359	0.729	0.731
Gemma-3	0.372	0.367	0.651	0.693	0.754

Table 1: Head protection recovers low-bit min–max accuracy, unevenly. HeadGuard protects selected image keys and values on the per-token min–max base, averaged over POPE/MME/MMBench/ScienceQA/MMMU/SEED.

K\approx N_{\rm KV}/8

. BF16 is a reference, not an upper bound.

		Accuracy	Gain (percentage points)
Bits	Base	Uniform	Image	Output	Magn.	Attn.	Vis.	Random	HeadGuard
2	min–max	0.436	2.57	13.34	1.26	6.64	1.34	1.39	14.37
2	KIVI-style	0.687	1.02	0.60	0.35	0.61	0.48	–	0.85
2	AKVQ-style	0.601	3.18	12.74	0.32	4.51	0.50	–	12.73
3	min–max	0.564	3.70	15.97	0.10	6.78	0.45	2.97	15.96
3	KIVI-style	0.741	0.14	0.02	0.08	0.08	0.14	–	0.16

Table 2: K+V protection: base accuracy and six-task selector gains. Uniform gives each base’s absolute six-task accuracy; the remaining columns are percentage-point gains over that uniform arm,

K\approx N_{\rm KV}/8

. Each row averages the six tasks and eight models equally, using full eligible task evaluation. Image and Output use only the image-sensitivity score

g

and output-sensitivity score

o

, respectively; HeadGuard combines them as

s=z(\bar{g})+z(\bar{o})

. Magn., Attn., and Vis. use activation magnitude, image-attention mass, and visual-response norm. Random head selection is reported for the min–max base only. These are descriptive means, not significance tests; small differences can reflect rounding.

	Six-task accuracy		TextVQA		Caption fidelity
Base	Uniform	HG-K	Uniform	HG-K	Uniform	HG-K
min–max	0.434	0.580	0.145	0.357	0.167	0.313
KIVI-style	0.688	0.693	0.554	0.556	0.471	0.493
AKVQ-style	0.588	0.727	0.392	0.630	0.410	0.555
BF16 reference	0.745		0.680		1.000

Table 3: K-only results across classification and generation,

2

bits. Uniform has no protection; HG-K is keys-only HeadGuard at

K\approx N_{\rm KV}/8

. Each metric averages all eight models equally; six-task accuracy also weights the tasks equally. Entries are scores, not gains, using the metrics in Section 4 .

Figure 1: Accuracy as more heads are protected. Selected image keys and values stay in BF16; the rest use

2

-bit min–max. Each line shows one model’s average accuracy across six tasks (POPE, MME, MMBench, ScienceQA, MMMU, and SEED-Bench), using up to

1000

questions per task (

900

for POPE;

839

for MMMU). HeadGuard’s ranking is fixed using POPE calibration. Zero means no protection; fractions are rounded to whole head counts. The vertical axis shows accuracy, not gains. More protection helps some models, but both Gemma models remain close to their unprotected scores under this ranking.

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

	POPE		MME		MMBench		ScienceQA		MMMU		SEED
Model	U	HG	U	HG	U	HG	U	HG	U	HG	U	HG
LLaVA-1.5	0.737	0.832	0.581	0.672	0.513	0.594	0.509	0.562	0.324	0.313	0.412	0.503
LLaVA-NeXT	0.832	0.857	0.745	0.750	0.664	0.711	0.585	0.634	0.331	0.336	0.568	0.653
Qwen2.5-VL	0.496	0.866	0.500	0.821	0.266	0.831	0.361	0.787	0.278	0.449	0.260	0.727
Qwen2-VL	0.510	0.676	0.504	0.598	0.271	0.713	0.349	0.678	0.284	0.347	0.261	0.575
Gemma-3	0.517	0.509	0.533	0.514	0.273	0.268	0.365	0.359	0.280	0.286	0.263	0.266

Appendix

Table 4: K+V protection, min–max at

2

bits , all six tasks. Each pair is uniform/HG;

K\approx N_{\rm KV}/8

. Counts are

8892/2366/4321/2009/839/14225

in column order.

	POPE		MME		MMBench		ScienceQA		MMMU		SEED
Model	U	HG	U	HG	U	HG	U	HG	U	HG	U	HG
LLaVA-1.5	0.840	0.844	0.782	0.782	0.679	0.680	0.611	0.615	0.356	0.355	0.579	0.577
LLaVA-NeXT	0.866	0.860	0.770	0.754	0.710	0.713	0.655	0.658	0.356	0.361	0.646	0.650
Qwen2.5-VL	0.511	0.881	0.501	0.855	0.268	0.856	0.347	0.836	0.281	0.492	0.255	0.763
Qwen2-VL	0.495	0.882	0.502	0.806	0.267	0.785	0.350	0.772	0.277	0.414	0.262	0.714
Gemma-3	0.794	0.818	0.814	0.843	0.672	0.734	0.696	0.734	0.418	0.445	0.509	0.583

Appendix

Table 5: K+V protection, min–max at

3

bits , same complete six-task block and column conventions as Table 4 .

Model	Uniform	HG-K	Uniform	HG-K	Uniform	HG-K	BF16
	min–max		AKVQ-style		KIVI-style
LLaVA-1.5	0.508	0.573	0.633	0.640	0.639	0.639	0.643
LLaVA-NeXT	0.615	0.660	0.661	0.664	0.655	0.663	0.664
Qwen2.5-VL	0.366	0.751	0.395	0.774	0.777	0.782	0.788
Qwen2-VL	0.361	0.605	0.364	0.707	0.717	0.720	0.732
Gemma-3	0.364	0.365	0.695	0.725	0.732	0.741	0.755

Appendix

Table 6: K-only evaluation at

2

bits, all six discriminative tasks.

K\approx N_{\rm KV}/8

. Each pair gives accuracy without protection (Uniform) and with keys-only HeadGuard (HG-K), not gains.

Model	HeadGuard gain over Uniform [CI]	HeadGuard gain over Output-only [CI]
LLaVA-1.5	$+9.5\,[+8.1,+10.8]$	$+1.8\,[+0.9,+2.7]$
LLaVA-NeXT	$+2.5\,[+1.4,+3.6]$	$+2.6\,[+1.7,+3.4]$
Qwen2.5-VL	$+36.9\,[+35.5,+38.3]$	$+2.9\,[+2.0,+3.9]$
Qwen2-VL	$+16.5\,[+14.7,+18.4]$	$+0.3\,[-1.1,+1.7]$
Gemma-3	$-0.8\,[-2.6,+1.0]$	$-0.8\,[-2.4,+0.8]$
Gemma-4	$+1.0\,[-0.7,+2.7]$	$+0.7\,[-0.9,+2.2]$

Appendix

Table 7: K+V protection,

2

-bit min–max POPE : paired image-cluster

95\%

intervals,

K\approx N_{\rm KV}/8

. Gains and intervals are in percentage points, not absolute accuracies.

Model	HeadGuard gain over Uniform [CI]	HeadGuard gain over Output-only [CI]
LLaVA-1.5	$+0.4\,[-0.1,+0.9]$	$0.0\,[-0.4,+0.5]$
LLaVA-NeXT	$-0.6\,[-1.0,-0.2]$	$-0.4\,[-0.7,-0.1]$
Qwen2.5-VL	$+37.0\,[+35.8,+38.2]$	$-0.3\,[-0.8,+0.2]$
Qwen2-VL	$+38.7\,[+37.3,+40.0]$	$+0.1\,[-0.3,+0.5]$
Gemma-3	$+2.4\,[+1.4,+3.4]$	$+0.4\,[-0.5,+1.3]$
Gemma-4	$+2.1\,[+0.9,+3.4]$	$-0.9\,[-2.0,+0.2]$

Appendix

Table 8: K+V protection,

3

-bit min–max POPE : same paired image-cluster bootstrap and head budget as Table 7 . Gains and intervals are in percentage points, not absolute accuracies.

Model	HeadGuard gain over Uniform [CI]	HeadGuard gain over Output-only [CI]
LLaVA-1.5	$+8.9\,[+7.5,+10.3]$	$+0.7\,[-0.2,+1.7]$
LLaVA-NeXT	$+3.0\,[+1.9,+4.1]$	$+2.7\,[+1.8,+3.4]$
Qwen2.5-VL	$+36.5\,[+35.1,+38.0]$	$+3.1\,[+2.1,+4.0]$
Qwen2-VL	$+17.8\,[+16.0,+19.6]$	$-1.1\,[-2.6,+0.4]$
Gemma-3	$-0.7\,[-2.4,+1.0]$	$+0.6\,[-1.0,+2.2]$
Gemma-4	$0.0\,[-1.5,+1.6]$	$-0.6\,[-2.0,+0.9]$

Appendix

Table 9: K-only evaluation,

2

-bit min–max POPE : same paired image-cluster bootstrap and head budget as Table 7 . Gains and intervals are in percentage points, not absolute accuracies. Contrasts compare selectors within K-only evaluation, not K-only against K+V.

Model	U	HG	U	HG	U	HG	BF16
	min–max		AKVQ-style		KIVI-style
LLaVA-1.5	0.313	0.382	0.460	0.470	0.459	0.470	0.476
LLaVA-NeXT	0.497	0.552	0.554	0.560	0.558	0.547	0.581
Qwen2.5-VL	0.015	0.687	0.039	0.744	0.716	0.731	0.775
Qwen2-VL	0.064	0.412	0.056	0.718	0.717	0.730	0.790
Gemma-3	0.008	0.012	0.579	0.618	0.662	0.689	0.706

Appendix

Table 10: K+V protection on TextVQA,

2

bits. Uniform/HG pairs; all eight models are retained.

Model	U	HG	U	HG	U	HG	BF16
	min–max		AKVQ-style		KIVI-style
LLaVA-1.5	0.480	0.479	0.475	0.475	0.474	0.480	0.476
LLaVA-NeXT	0.583	0.575	0.580	0.571	0.576	0.583	0.581
Qwen2.5-VL	0.000	0.756	0.003	0.760	0.761	0.771	0.775
Qwen2-VL	0.022	0.787	0.253	0.785	0.775	0.783	0.790
Gemma-3	0.556	0.615	0.702	0.694	0.707	0.701	0.706

Appendix

Table 11: K+V protection on TextVQA,

3

bits. Same conventions as Table 10 .

Model	U	HG	U	HG	U	HG	BF16
	min–max		AKVQ-style		KIVI-style
LLaVA-1.5	0.370	0.477	0.631	0.661	0.627	0.664	1.000
LLaVA-NeXT	0.492	0.514	0.663	0.681	0.557	0.606	1.000
Qwen2.5-VL	0.004	0.375	0.118	0.454	0.452	0.476	1.000
Qwen2-VL	0.058	0.280	0.014	0.466	0.374	0.392	1.000
Gemma-3	0.077	0.136	0.523	0.548	0.570	0.613	1.000

Appendix

Table 12: K+V protection for captioning,

2

bits. ROUGE-L overlap with BF16 output, not human-caption quality. Each pair is uniform/HG on

200

examples.

Model	U	HG	U	HG	U	HG	BF16
	min–max		AKVQ-style		KIVI-style
LLaVA-1.5	0.717	0.748	0.804	0.828	0.783	0.806	1.000
LLaVA-NeXT	0.722	0.796	0.815	0.846	0.806	0.840	1.000
Qwen2.5-VL	0.009	0.427	0.025	0.468	0.625	0.611	1.000
Qwen2-VL	0.025	0.581	0.022	0.651	0.617	0.642	1.000
Gemma-3	0.536	0.556	0.692	0.722	0.706	0.755	1.000

Appendix

Table 13: K+V protection for captioning,

3

bits. Same scope and conventions as Table 12 .

Model	U	HG-K	U	HG-K	U	HG-K	BF16
	min–max		AKVQ-style		KIVI-style
LLaVA-1.5	0.312	0.379	0.471	0.468	0.460	0.452	0.475
LLaVA-NeXT	0.518	0.531	0.562	0.558	0.557	0.553	0.578
Qwen2.5-VL	0.009	0.690	0.025	0.750	0.733	0.724	0.771
Qwen2-VL	0.050	0.425	0.060	0.707	0.705	0.718	0.791
Gemma-3	0.009	0.018	0.575	0.603	0.665	0.683	0.705

Appendix

Table 14: K-only TextVQA evaluation,

2

bits. Each pair is uniform/HG-K on

1000

questions.

Model	U	HG-K	U	HG-K	U	HG-K	BF16
	min–max		AKVQ-style		KIVI-style
LLaVA-1.5	0.373	0.472	0.646	0.647	0.638	0.645	1.000
LLaVA-NeXT	0.460	0.533	0.626	0.664	0.551	0.603	1.000
Qwen2.5-VL	0.005	0.368	0.095	0.426	0.451	0.467	1.000
Qwen2-VL	0.053	0.300	0.015	0.433	0.380	0.395	1.000
Gemma-3	0.073	0.132	0.518	0.534	0.580	0.584	1.000

Appendix

Table 15: K-only caption evaluation,

2

bits.

200

examples per cell; each pair is uniform/HG-K.

		min–max			AKVQ-style			KIVI-style
Model	Bits	U	Out	HG	U	Out	HG	U	Out	HG	BF16
LLaVA-1.5	2	0.515	0.560	0.580	0.637	0.633	0.641	0.638	0.636	0.636	0.643
LLaVA-1.5	3	0.640	0.642	0.641	0.642	0.642	0.645	0.644	0.644	0.645	0.643
LLaVA-1.5	4	0.643	0.644	0.642	0.644	0.644	0.642	0.642	0.644	0.643	0.643
LLaVA-NeXT	2	0.618	0.648	0.663	0.663	0.667	0.668	0.656	0.665	0.665	0.666
LLaVA-NeXT	3	0.667	0.669	0.667	0.668	0.666	0.665	0.666	0.665	0.665	0.666

Appendix

Table 16: Capped sweep with POPE calibration. Each score averages POPE, MME, MMBench, ScienceQA, MMMU, and SEED, using up to

1000

questions per task (

900

for POPE;

839

for MMMU). K+V protection uses

K=\operatorname{round}(N_{\rm KV}/8)

. U = uniform; Out = output-only selection; HG = hybrid HeadGuard. Mean averages all eight models. Values are accuracies, not gains; gains in the main text are calculated before final rounding.

		min–max			AKVQ-style			KIVI-style
Model	Bits	U	Out	HG	U	Out	HG	U	Out	HG	BF16
LLaVA-1.5	2	0.514	0.578	0.590	0.639	0.635	0.640	0.639	0.631	0.640	0.644
LLaVA-1.5	3	0.641	0.641	0.644	0.643	0.644	0.645	0.645	0.645	0.645	0.644
LLaVA-1.5	4	0.644	0.644	0.646	0.646	0.647	0.644	0.643	0.646	0.645	0.644
LLaVA-NeXT	2	0.620	0.643	0.654	0.663	0.665	0.666	0.656	0.662	0.666	0.666
LLaVA-NeXT	3	0.666	0.667	0.668	0.668	0.668	0.666	0.667	0.666	0.664	0.666

Appendix

Table 17: Capped sweep with MMBench calibration. Same six evaluation tasks, one-eighth head budget, and column definitions as Table 16 . Here MMBench uses

900

held-out questions, POPE

1000

, and MMMU

839

. Mean averages all eight models.

		min–max			AKVQ-style			KIVI-style
Model	Bits	U	Out	HG	U	Out	HG	U	Out	HG	BF16
LLaVA-1.5	2	0.303	0.387	0.375	0.443	0.437	0.427	0.460	0.442	0.427	0.453
LLaVA-1.5	3	0.433	0.440	0.430	0.455	0.465	0.475	0.462	0.452	0.460	0.453
LLaVA-1.5	4	0.440	0.440	0.438	0.448	0.453	0.443	0.433	0.455	0.432	0.453
LLaVA-NeXT	2	0.522	0.525	0.567	0.540	0.532	0.527	0.553	0.532	0.530	0.553
LLaVA-NeXT	3	0.553	0.532	0.525	0.558	0.542	0.542	0.558	0.562	0.542	0.553

Appendix

Table 18: Capped TextVQA with POPE calibration. Custom VQA-style scores on

200

questions, K+V protection at the one-eighth head budget. Mean averages all eight models.

		min–max			AKVQ-style			KIVI-style
Model	Bits	U	Out	HG	U	Out	HG	U	Out	HG	BF16
LLaVA-1.5	2	0.373	0.451	0.480	0.646	0.667	0.670	0.638	0.656	0.673	1.000
LLaVA-1.5	3	0.738	0.745	0.748	0.788	0.797	0.814	0.791	0.802	0.804	1.000
LLaVA-1.5	4	0.847	0.849	0.868	0.891	0.894	0.894	0.873	0.889	0.892	1.000
LLaVA-NeXT	2	0.460	0.505	0.509	0.626	0.661	0.660	0.551	0.600	0.602	1.000
LLaVA-NeXT	3	0.724	0.781	0.783	0.825	0.836	0.849	0.813	0.809	0.839	1.000

Appendix

Table 19: Capped captioning with POPE calibration. ROUGE-L F1 against BF16 output on

200

examples, not human-caption quality. K+V protection uses the one-eighth head budget. Mean averages all eight models.

		min–max			AKVQ-style			KIVI-style
Model	Bits	U	Out	HG	U	Out	HG	U	Out	HG	BF16
LLaVA-1.5	2	0.303	0.367	0.368	0.443	0.432	0.410	0.460	0.437	0.428	0.453
LLaVA-1.5	3	0.433	0.433	0.450	0.455	0.455	0.467	0.462	0.470	0.478	0.453
LLaVA-1.5	4	0.440	0.432	0.440	0.448	0.457	0.453	0.433	0.432	0.445	0.453
LLaVA-NeXT	2	0.522	0.507	0.548	0.540	0.533	0.528	0.553	0.525	0.548	0.553
LLaVA-NeXT	3	0.553	0.558	0.537	0.558	0.560	0.547	0.558	0.572	0.548	0.553

Appendix

Table 20: Capped TextVQA with MMBench calibration. Same

200

-question scope, metric, head budget, and columns as Table 18 . Mean averages all eight models.

		min–max			AKVQ-style			KIVI-style
Model	Bits	U	Out	HG	U	Out	HG	U	Out	HG	BF16
LLaVA-1.5	2	0.373	0.482	0.507	0.646	0.649	0.694	0.638	0.669	0.657	1.000
LLaVA-1.5	3	0.738	0.739	0.753	0.788	0.811	0.824	0.791	0.797	0.819	1.000
LLaVA-1.5	4	0.847	0.857	0.841	0.891	0.880	0.892	0.873	0.874	0.864	1.000
LLaVA-NeXT	2	0.460	0.495	0.514	0.626	0.643	0.661	0.551	0.597	0.593	1.000
LLaVA-NeXT	3	0.724	0.757	0.765	0.825	0.836	0.861	0.813	0.803	0.843	1.000

Appendix

Table 21: Capped captioning with MMBench calibration. Same

200

-example scope, BF16-output ROUGE-L metric, head budget, and columns as Table 19 . Mean averages all eight models.

		min–max			AKVQ-style			KIVI-style
Model	Bits	U	Out	HG	U	Out	HG	U	Out	HG	BF16
LLaVA-1.5	2	0.515	0.542	0.543	0.637	0.635	0.641	0.638	0.633	0.637	0.643
LLaVA-1.5	3	0.640	0.641	0.641	0.642	0.640	0.643	0.644	0.644	0.647	0.643
LLaVA-1.5	4	0.643	0.644	0.644	0.644	0.642	0.642	0.642	0.642	0.645	0.643
LLaVA-NeXT	2	0.618	0.644	0.649	0.663	0.668	0.662	0.656	0.661	0.661	0.666
LLaVA-NeXT	3	0.667	0.670	0.669	0.668	0.666	0.665	0.666	0.667	0.665	0.666

Appendix

Table 22: Six-task accuracy,

1/16

budget, POPE calibration. K+V protection with

K=\operatorname{round}(N_{\rm KV}/16)

. Equal-weight accuracy over POPE, MME, MMBench, ScienceQA, MMMU, and SEED; up to

1000

questions/task (

900

for POPE;

839

for MMMU). U = uniform; Out = output-only; HG = hybrid. Mean averages all eight models.

		min–max			AKVQ-style			KIVI-style
Model	Bits	U	Out	HG	U	Out	HG	U	Out	HG	BF16
LLaVA-1.5	2	0.303	0.350	0.313	0.443	0.440	0.447	0.460	0.437	0.428	0.453
LLaVA-1.5	3	0.433	0.443	0.435	0.455	0.450	0.458	0.462	0.452	0.463	0.453
LLaVA-1.5	4	0.440	0.425	0.428	0.448	0.458	0.445	0.433	0.442	0.440	0.453
LLaVA-NeXT	2	0.522	0.515	0.532	0.540	0.558	0.517	0.553	0.533	0.535	0.553
LLaVA-NeXT	3	0.553	0.548	0.545	0.558	0.550	0.545	0.558	0.553	0.558	0.553

Appendix

Table 23: TextVQA,

1/16

budget, POPE calibration. K+V protection with

K=\operatorname{round}(N_{\rm KV}/16)

. Custom VQA-style score on

200

questions. U = uniform; Out = output-only; HG = hybrid. Mean averages all eight models.

		min–max			AKVQ-style			KIVI-style
Model	Bits	U	Out	HG	U	Out	HG	U	Out	HG	BF16
LLaVA-1.5	2	0.373	0.445	0.415	0.646	0.647	0.641	0.638	0.648	0.662	1.000
LLaVA-1.5	3	0.738	0.740	0.754	0.788	0.783	0.807	0.791	0.804	0.816	1.000
LLaVA-1.5	4	0.847	0.858	0.852	0.891	0.886	0.879	0.873	0.868	0.868	1.000
LLaVA-NeXT	2	0.460	0.494	0.500	0.626	0.650	0.661	0.551	0.566	0.596	1.000
LLaVA-NeXT	3	0.724	0.745	0.770	0.825	0.838	0.848	0.813	0.791	0.822	1.000

Appendix

Table 24: Caption fidelity,

1/16

budget, POPE calibration. K+V protection with

K=\operatorname{round}(N_{\rm KV}/16)

. ROUGE-L F1 against BF16 output on

200

examples, not human-caption quality. U = uniform; Out = output-only; HG = hybrid. Mean averages all eight models.

		min–max			AKVQ-style			KIVI-style
Model	Bits	U	Out	HG	U	Out	HG	U	Out	HG	BF16
LLaVA-1.5	2	0.514	0.561	0.541	0.639	0.639	0.638	0.639	0.636	0.643	0.644
LLaVA-1.5	3	0.641	0.639	0.643	0.643	0.644	0.642	0.645	0.645	0.644	0.644
LLaVA-1.5	4	0.644	0.644	0.644	0.646	0.645	0.646	0.643	0.645	0.645	0.644
LLaVA-NeXT	2	0.620	0.626	0.645	0.663	0.666	0.665	0.656	0.659	0.662	0.666
LLaVA-NeXT	3	0.666	0.668	0.668	0.668	0.668	0.666	0.667	0.667	0.666	0.666

Appendix

Table 25: Six-task accuracy,

1/16

budget, MMBench calibration. K+V protection with

K=\operatorname{round}(N_{\rm KV}/16)

. Equal-weight accuracy over POPE, MME, MMBench, ScienceQA, MMMU, and SEED; up to

1000

questions/task (

900

for MMBench;

839

for MMMU). U = uniform; Out = output-only; HG = hybrid. Mean averages all eight models.

		min–max			AKVQ-style			KIVI-style
Model	Bits	U	Out	HG	U	Out	HG	U	Out	HG	BF16
LLaVA-1.5	2	0.303	0.332	0.325	0.443	0.428	0.437	0.460	0.430	0.447	0.453
LLaVA-1.5	3	0.433	0.438	0.432	0.455	0.460	0.468	0.462	0.452	0.467	0.453
LLaVA-1.5	4	0.440	0.432	0.435	0.448	0.448	0.445	0.433	0.442	0.450	0.453
LLaVA-NeXT	2	0.522	0.483	0.512	0.540	0.515	0.538	0.553	0.547	0.532	0.553
LLaVA-NeXT	3	0.553	0.552	0.533	0.558	0.560	0.555	0.558	0.533	0.550	0.553

Appendix

Table 26: TextVQA,

1/16

budget, MMBench calibration. K+V protection with

K=\operatorname{round}(N_{\rm KV}/16)

. Custom VQA-style score on

200

questions. U = uniform; Out = output-only; HG = hybrid. Mean averages all eight models.

		min–max			AKVQ-style			KIVI-style
Model	Bits	U	Out	HG	U	Out	HG	U	Out	HG	BF16
LLaVA-1.5	2	0.373	0.457	0.430	0.646	0.647	0.659	0.638	0.642	0.653	1.000
LLaVA-1.5	3	0.738	0.735	0.748	0.788	0.813	0.815	0.791	0.801	0.796	1.000
LLaVA-1.5	4	0.847	0.836	0.838	0.891	0.874	0.892	0.873	0.862	0.888	1.000
LLaVA-NeXT	2	0.460	0.493	0.495	0.626	0.637	0.667	0.551	0.571	0.584	1.000
LLaVA-NeXT	3	0.724	0.753	0.743	0.825	0.853	0.844	0.813	0.800	0.812	1.000

Appendix

Table 27: Caption fidelity,

1/16

budget, MMBench calibration. K+V protection with

K=\operatorname{round}(N_{\rm KV}/16)

. ROUGE-L F1 against BF16 output on

200

examples, not human-caption quality. U = uniform; Out = output-only; HG = hybrid. Mean averages all eight models.

		min–max			AKVQ-style			KIVI-style
Model	Bits	U	Out	HG	U	Out	HG	U	Out	HG	BF16
LLaVA-1.5	2	0.515	0.549	0.566	0.637	0.631	0.640	0.638	0.634	0.641	0.643
LLaVA-1.5	3	0.640	0.642	0.645	0.642	0.641	0.642	0.644	0.643	0.643	0.643
LLaVA-1.5	4	0.643	0.642	0.645	0.644	0.642	0.644	0.642	0.643	0.643	0.643
LLaVA-NeXT	2	0.618	0.644	0.653	0.663	0.666	0.668	0.656	0.661	0.666	0.666
LLaVA-NeXT	3	0.667	0.671	0.671	0.668	0.667	0.665	0.666	0.666	0.666	0.666

Appendix

Table 28: Six-task accuracy,

1/12

budget, POPE calibration. K+V protection with

K=\operatorname{round}(N_{\rm KV}/12)

. Equal-weight accuracy over POPE, MME, MMBench, ScienceQA, MMMU, and SEED; up to

1000

questions/task (

900

for POPE;

839

for MMMU). U = uniform; Out = output-only; HG = hybrid. Mean averages all eight models.

		min–max			AKVQ-style			KIVI-style
Model	Bits	U	Out	HG	U	Out	HG	U	Out	HG	BF16
LLaVA-1.5	2	0.303	0.352	0.362	0.443	0.443	0.440	0.460	0.440	0.440	0.453
LLaVA-1.5	3	0.433	0.423	0.430	0.455	0.452	0.442	0.462	0.438	0.465	0.453
LLaVA-1.5	4	0.440	0.433	0.440	0.448	0.450	0.448	0.433	0.445	0.435	0.453
LLaVA-NeXT	2	0.522	0.545	0.522	0.540	0.525	0.550	0.553	0.555	0.543	0.553
LLaVA-NeXT	3	0.553	0.542	0.523	0.558	0.555	0.533	0.558	0.538	0.553	0.553

Appendix

Table 29: TextVQA,

1/12

budget, POPE calibration. K+V protection with

K=\operatorname{round}(N_{\rm KV}/12)

. Custom VQA-style score on

200

questions. U = uniform; Out = output-only; HG = hybrid. Mean averages all eight models.

		min–max			AKVQ-style			KIVI-style
Model	Bits	U	Out	HG	U	Out	HG	U	Out	HG	BF16
LLaVA-1.5	2	0.373	0.446	0.458	0.646	0.646	0.652	0.638	0.641	0.644	1.000
LLaVA-1.5	3	0.738	0.744	0.737	0.788	0.816	0.813	0.791	0.795	0.807	1.000
LLaVA-1.5	4	0.847	0.841	0.848	0.891	0.886	0.884	0.873	0.876	0.866	1.000
LLaVA-NeXT	2	0.460	0.506	0.493	0.626	0.632	0.646	0.551	0.585	0.601	1.000
LLaVA-NeXT	3	0.724	0.760	0.778	0.825	0.852	0.850	0.813	0.806	0.802	1.000

Appendix

Table 30: Caption fidelity,

1/12

budget, POPE calibration. K+V protection with

K=\operatorname{round}(N_{\rm KV}/12)

. ROUGE-L F1 against BF16 output on

200

examples, not human-caption quality. U = uniform; Out = output-only; HG = hybrid. Mean averages all eight models.

		min–max			AKVQ-style			KIVI-style
Model	Bits	U	Out	HG	U	Out	HG	U	Out	HG	BF16
LLaVA-1.5	2	0.514	0.560	0.547	0.639	0.637	0.640	0.639	0.635	0.640	0.644
LLaVA-1.5	3	0.641	0.644	0.643	0.643	0.645	0.644	0.645	0.646	0.645	0.644
LLaVA-1.5	4	0.644	0.644	0.646	0.646	0.647	0.646	0.643	0.646	0.645	0.644
LLaVA-NeXT	2	0.620	0.622	0.649	0.663	0.670	0.662	0.656	0.659	0.664	0.666
LLaVA-NeXT	3	0.666	0.670	0.669	0.668	0.666	0.664	0.667	0.667	0.665	0.666

Appendix

Table 31: Six-task accuracy,

1/12

budget, MMBench calibration. K+V protection with

K=\operatorname{round}(N_{\rm KV}/12)

. Equal-weight accuracy over POPE, MME, MMBench, ScienceQA, MMMU, and SEED; up to

1000

questions/task (

900

for MMBench;

839

for MMMU). U = uniform; Out = output-only; HG = hybrid. Mean averages all eight models.

		min–max			AKVQ-style			KIVI-style
Model	Bits	U	Out	HG	U	Out	HG	U	Out	HG	BF16
LLaVA-1.5	2	0.303	0.355	0.308	0.443	0.427	0.433	0.460	0.415	0.402	0.453
LLaVA-1.5	3	0.433	0.427	0.445	0.455	0.445	0.442	0.462	0.460	0.468	0.453
LLaVA-1.5	4	0.440	0.430	0.428	0.448	0.448	0.445	0.433	0.450	0.445	0.453
LLaVA-NeXT	2	0.522	0.512	0.503	0.540	0.545	0.548	0.553	0.510	0.543	0.553
LLaVA-NeXT	3	0.553	0.532	0.565	0.558	0.538	0.542	0.558	0.550	0.553	0.553

Appendix

Table 32: TextVQA,

1/12

budget, MMBench calibration. K+V protection with

K=\operatorname{round}(N_{\rm KV}/12)

. Custom VQA-style score on

200

questions. U = uniform; Out = output-only; HG = hybrid. Mean averages all eight models.

		min–max			AKVQ-style			KIVI-style
Model	Bits	U	Out	HG	U	Out	HG	U	Out	HG	BF16
LLaVA-1.5	2	0.373	0.476	0.435	0.646	0.632	0.653	0.638	0.641	0.652	1.000
LLaVA-1.5	3	0.738	0.735	0.757	0.788	0.796	0.809	0.791	0.791	0.795	1.000
LLaVA-1.5	4	0.847	0.827	0.831	0.891	0.883	0.893	0.873	0.885	0.873	1.000
LLaVA-NeXT	2	0.460	0.475	0.492	0.626	0.622	0.653	0.551	0.553	0.584	1.000
LLaVA-NeXT	3	0.724	0.746	0.765	0.825	0.843	0.845	0.813	0.805	0.821	1.000

Appendix

Table 33: Caption fidelity,

1/12

budget, MMBench calibration. K+V protection with

K=\operatorname{round}(N_{\rm KV}/12)

. ROUGE-L F1 against BF16 output on

200

examples, not human-caption quality. U = uniform; Out = output-only; HG = hybrid. Mean averages all eight models.

		min–max			AKVQ-style			KIVI-style
Model	Bits	U	Out	HG	U	Out	HG	U	Out	HG	BF16
LLaVA-1.5	2	0.515	0.579	0.611	0.637	0.637	0.639	0.638	0.637	0.642	0.643
LLaVA-1.5	3	0.640	0.642	0.645	0.642	0.645	0.643	0.644	0.642	0.645	0.643
LLaVA-1.5	4	0.643	0.645	0.642	0.644	0.645	0.643	0.642	0.643	0.642	0.643
LLaVA-NeXT	2	0.618	0.658	0.669	0.663	0.667	0.667	0.656	0.666	0.668	0.666
LLaVA-NeXT	3	0.667	0.669	0.670	0.668	0.665	0.666	0.666	0.666	0.668	0.666

Appendix

Table 34: Six-task accuracy,

1/4

budget, POPE calibration. K+V protection with

K=\operatorname{round}(N_{\rm KV}/4)

. Equal-weight accuracy over POPE, MME, MMBench, ScienceQA, MMMU, and SEED; up to

1000

questions/task (

900

for POPE;

839

for MMMU). U = uniform; Out = output-only; HG = hybrid. Mean averages all eight models.

		min–max			AKVQ-style			KIVI-style
Model	Bits	U	Out	HG	U	Out	HG	U	Out	HG	BF16
LLaVA-1.5	2	0.303	0.380	0.363	0.443	0.442	0.450	0.460	0.420	0.395	0.453
LLaVA-1.5	3	0.433	0.438	0.462	0.455	0.457	0.458	0.462	0.457	0.462	0.453
LLaVA-1.5	4	0.440	0.453	0.438	0.448	0.453	0.443	0.433	0.442	0.447	0.453
LLaVA-NeXT	2	0.522	0.518	0.492	0.540	0.530	0.535	0.553	0.537	0.525	0.553
LLaVA-NeXT	3	0.553	0.557	0.542	0.558	0.555	0.552	0.558	0.567	0.563	0.553

Appendix

Table 35: TextVQA,

1/4

budget, POPE calibration. K+V protection with

K=\operatorname{round}(N_{\rm KV}/4)

. Custom VQA-style score on

200

questions. U = uniform; Out = output-only; HG = hybrid. Mean averages all eight models.

		min–max			AKVQ-style			KIVI-style
Model	Bits	U	Out	HG	U	Out	HG	U	Out	HG	BF16
LLaVA-1.5	2	0.373	0.511	0.545	0.646	0.668	0.722	0.638	0.644	0.693	1.000
LLaVA-1.5	3	0.738	0.766	0.776	0.788	0.822	0.829	0.791	0.810	0.812	1.000
LLaVA-1.5	4	0.847	0.857	0.862	0.891	0.895	0.893	0.873	0.880	0.896	1.000
LLaVA-NeXT	2	0.460	0.500	0.524	0.626	0.675	0.688	0.551	0.606	0.606	1.000
LLaVA-NeXT	3	0.724	0.797	0.812	0.825	0.872	0.850	0.813	0.816	0.820	1.000

Appendix

Table 36: Caption fidelity,

1/4

budget, POPE calibration. K+V protection with

K=\operatorname{round}(N_{\rm KV}/4)

. ROUGE-L F1 against BF16 output on

200

examples, not human-caption quality. U = uniform; Out = output-only; HG = hybrid. Mean averages all eight models.

		min–max			AKVQ-style			KIVI-style
Model	Bits	U	Out	HG	U	Out	HG	U	Out	HG	BF16
LLaVA-1.5	2	0.514	0.605	0.633	0.639	0.639	0.641	0.639	0.644	0.641	0.644
LLaVA-1.5	3	0.641	0.643	0.643	0.643	0.646	0.642	0.645	0.645	0.646	0.644
LLaVA-1.5	4	0.644	0.644	0.645	0.646	0.645	0.642	0.643	0.645	0.644	0.644
LLaVA-NeXT	2	0.620	0.645	0.668	0.663	0.662	0.668	0.656	0.661	0.666	0.666
LLaVA-NeXT	3	0.666	0.669	0.669	0.668	0.666	0.667	0.667	0.667	0.667	0.666

Appendix

Table 37: Six-task accuracy,

1/4

budget, MMBench calibration. K+V protection with

K=\operatorname{round}(N_{\rm KV}/4)

. Equal-weight accuracy over POPE, MME, MMBench, ScienceQA, MMMU, and SEED; up to

1000

questions/task (

900

for MMBench;

839

for MMMU). U = uniform; Out = output-only; HG = hybrid. Mean averages all eight models.

		min–max			AKVQ-style			KIVI-style
Model	Bits	U	Out	HG	U	Out	HG	U	Out	HG	BF16
LLaVA-1.5	2	0.303	0.405	0.370	0.443	0.445	0.440	0.460	0.420	0.430	0.453
LLaVA-1.5	3	0.433	0.442	0.443	0.455	0.465	0.447	0.462	0.453	0.467	0.453
LLaVA-1.5	4	0.440	0.445	0.435	0.448	0.453	0.438	0.433	0.437	0.445	0.453
LLaVA-NeXT	2	0.522	0.523	0.537	0.540	0.522	0.518	0.553	0.515	0.522	0.553
LLaVA-NeXT	3	0.553	0.533	0.555	0.558	0.540	0.547	0.558	0.563	0.550	0.553

Appendix

Table 38: TextVQA,

1/4

budget, MMBench calibration. K+V protection with

K=\operatorname{round}(N_{\rm KV}/4)

. Custom VQA-style score on

200

questions. U = uniform; Out = output-only; HG = hybrid. Mean averages all eight models.

		min–max			AKVQ-style			KIVI-style
Model	Bits	U	Out	HG	U	Out	HG	U	Out	HG	BF16
LLaVA-1.5	2	0.373	0.525	0.573	0.646	0.702	0.692	0.638	0.678	0.709	1.000
LLaVA-1.5	3	0.738	0.777	0.781	0.788	0.831	0.835	0.791	0.830	0.830	1.000
LLaVA-1.5	4	0.847	0.858	0.854	0.891	0.886	0.910	0.873	0.873	0.885	1.000
LLaVA-NeXT	2	0.460	0.518	0.544	0.626	0.682	0.677	0.551	0.602	0.649	1.000
LLaVA-NeXT	3	0.724	0.790	0.784	0.825	0.850	0.853	0.813	0.797	0.825	1.000

Appendix

Table 39: Caption fidelity,

1/4

budget, MMBench calibration. K+V protection with

K=\operatorname{round}(N_{\rm KV}/4)

. ROUGE-L F1 against BF16 output on

200

examples, not human-caption quality. U = uniform; Out = output-only; HG = hybrid. Mean averages all eight models.

VLM Quantization

Vision-language model quantization reduces model precision to lower memory and computation while preserving multimodal capabilities.

See all 52 papers

Jun 1, 2026

Alignment Collapse Under KV Cache Quantization: Diagnosis and Mitigation

Key-value (KV) cache quantization is widely used to reduce Large Language Model (LLM) inference memory, yet existing evaluations solely focus on measuring perplexity and accuracy without assessing the safety impact. In this study, we explore alignment preservation under KV cache quantization. Across eleven instruction-tuned models (3.8B-72B) and five benchmarks (1,894 prompts), we find that low-bit quantization can silently destroy safety alignment: Mistral-7B loses 15.2% of its refusals at only 1.03x perplexity, and no universal safe bit-width exists, with sharp model-specific phase transitions invisible to standard metrics. We identify that the root cause is geometric: safety features occupy a low-dimensional activation subspace 10^2-10^3x more vulnerable to quantization noise than the full representation space perplexity averages over. Inspired by this observation, we propose Per-Channel Reduction (PCR), a diagnostic that classifies each model into one of three mechanistic failure modes: outlier-crushes-safety, where safety lives in non-outlier channels collaterally damaged by outlier-driven scale factors; outlier-as-safety, where safety overlaps outlier channels and finer granularity cannot rescue it; and multi-layer dilution, where safety is distributed across many layers and per-layer fixes fail. PCR predicts the correct mitigation direction on all nine primary models and one held-out model from an independent family using 20 calibration prompts. PCR generalizes across unseen prompts, models, and production quantizers, including KIVI with up to 97.2% recovery, succeeding where attention-based allocation methods fail. The resulting training-free protocol, requiring approximately 35 GPU-minutes, recovers up to 97% of lost alignment at minimal memory overhead, addressing vulnerabilities confirmed in production vLLM serving with FP8 KV cache on NVIDIA GPUs.

Bruce Changlong Xu, Adarsh Kumarappan, Mu Zhou

†Stanford University · ‡California Institute of Technology

LLM Quantization KV Caching LLM Safety Alignment KV-Cache Quantization

Ask PDF

May 5, 2026

HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization

KV-cache quantizers usually optimize storage-space reconstruction, even though attention reads keys through logits and values through attention-weighted readout. We argue that persistent cache error should be measured in model-visible coordinates. For keys, the visible object is score error modulo constant shifts; this yields HeadQ, a key-side method that stores a low-rank residual side code in a calibration-learned query basis and applies it as an additive logit correction. For values, fixed-attention readout gives an

A^2

-weighted token-distortion surrogate. Across six models, Fisher/score-space error predicts attention KL far better than raw key MSE; same-budget counterexamples, null-space interventions, query-PCA controls, and wrong-sign HeadQ falsify storage-MSE alternatives. Matched Pythia checkpoints localize the main anomaly to a small-model low-entropy route-flip boundary. In K-only WikiText-103 decode experiments with dense values, HeadQ removes roughly

84

94\%

of the excess perplexity on the strongest 2-bit rows; in an auxiliary full-KV 2-bit composition, HeadQ plus an

A^2

value policy improves all six models.

Jorge L. Ruiz Williams

Independent Researcher

Self-Attention KV Caching Attention Quantization KV-Cache Compression KV-Cache Quantization Post-Training Quantization+1

Ask PDF

Aug 3, 2026

ARCHead: Activation-Metric Residual Correction for Large Language Model Output Heads

Weight-only quantization substantially reduces the storage of large language model (LLM) transformer blocks, but practical backends often retain the final language-modeling head (LM-head) in BF16 or FP16. Quantizing this projection naively can strongly perturb the vocabulary-logit distribution. We present ARCHead, a packed LM-head compressor that combines a quantized low-rank core, group-wise INT4 residuals, and a low-rank correction fitted in an activation-derived metric. ARCHead stores no dense BF16 head and reduces persistent LM-head storage by 3.7-3.9x. On Qwen3-8B-Base, it uses 25.6% of BF16 head storage while attaining 1.007 relative perplexity; storage-matched naive INT4 yields 1.14-1.16. Replacing the BF16 head left by AWQ or bitsandbytes adds only 0.006-0.007 cross-entropy, with less than 2% throughput change in our measurements. ARCHead therefore complements block quantizers by compressing the large output projection they can leave untouched. Code is available at https://github.com/suayptalha/archead.

Şuayp Talha Kocabay, Talha Rüzgar Akkuş, Kamer Ali Yuksel

Independent Researcher · aiXplain, Inc.

LLM Quantization LLM Compression Low-Rank Compression Weight-Only Quantization

Ask PDF

Abstract

VLM Quantization

Explore similar work

Alignment Collapse Under KV Cache Quantization: Diagnosis and Mitigation

HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization

ARCHead: Activation-Metric Residual Correction for Large Language Model Output Heads