Shared Stopping Decisions Change Answers in HQQ Cache Quantization

Seunghui Jwa; Minsu Oh; Chanjun Park; Yeo-Chan Yoon

cs.CLOct 5, 2026

Shared Stopping Decisions Change Answers in HQQ Cache Quantization

Authors: Seunghui Jwa, Minsu Oh, Chanjun Park, Yeo-Chan Yoon

Organizations: Department of Artificial Intelligence, Jeju National University · School of Software, Soongsil University

Abstract

Language-model systems batch questions for throughput, but unrelated questions should not change a target's answer when its input and numerical execution are fixed. We study compression of the key and value cache, which stores attention representations reused during generation. With request-local groups, Transformers' Half-Quadratic Quantization (HQQ) backend updates compression parameters separately but uses a shared average error to decide when all updates stop. Replacing only the question batched with the target changes four-bit HQQ answers in 170/384 test comparisons across two models. Replaying the other execution's update counts reproduces its complete answer and cache fingerprints in every changed pair, in both directions. Computing the stopping mean in FP32 reduces cache differences but leaves answer changes. Native HQQ also changes confirmed numerical correctness in eight arithmetic pairs. Fixed iterations and request-local stopping remove observed companion dependence under matched controls. Request-local stopping remains sensitive to synthetic padding changes at the tensor level. Fixing the original iteration budget removes this decision path without tuning. Neither repair has an established quality advantage, and natural rebatching still changes answers. Request-independence audits must cover stopping decisions as well as quantization groups.

Figures & tables

Panel of Figure 1 — Figure 1: Schematic of initial compression. Different companions can select counts $J_{A},J_{B}$ on the same target trajectory, yielding different caches. A fixed count $k$ removes this endpoint dependence.

Panel of Figure 2 — Figure 2: Causal verification on the test set for target $q$ with companions $a,b$ . $\mathbf{J}_{A},\mathbf{J}_{B}$ record update counts for each layer and key/value compression call. Tracing, own-count replay and restoration must reproduce native execution. Cross-run replay changes only counts, recomputing caches and answers. Full transfer requires all raw/reconstructed cache hashes and output tokens to match the opposite run through EOS or the cap, with no missing counts.

Model	Task	Text $\Delta$	Numerical correctness $\Delta$ /resolved
Llama	Math	80/128	6/122
Qwen	Math	89/128	2/124
Both	Math	169/256	8/246
Llama	QA	0/64	–
Qwen	QA	1/64	–

Table 1: Test-set answer changes under four-bit HQQ companion replacement. Text

\Delta

counts changed strings. Numerical correctness

\Delta

/resolved counts correct-to-incorrect or reverse changes over pairs where both final numbers can be extracted, without requiring an answer marker. Both sums the two math rows; the models reuse questions.

Method	Text changes	Correctness changes
BF16	36.1%	17/1,484
Native HQQ	54.9%	64/1,501
Fixed 20	46.6%	62/1,496

Table 2: Natural test-set rebatching across six shuffle seeds. Text: mean over 384 model/question pairs per seed. Correctness: final-number correctness changes / pairs with both numbers resolved, summed across seeds for arithmetic only. These reuse the same questions; the denominators are not independent samples.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Method	GSM	IFEval	Hotpot	MuSiQue	SQuAD	TriviaQA
Llama	BF16	73.96	83.33	87.92	34.32	85.72	91.53
Llama	Native HQQ	66.67	81.25	87.92	34.32	85.72	91.53
Llama	No optimization	67.71	82.29	87.92	34.32	85.72	91.53
Llama	Fixed 2	68.75	83.33	87.92	33.98	85.72	91.53
Llama	Fixed 4	65.62	81.25	87.92	33.98	85.72	91.53
Llama	Fixed 8	67.71	83.33	87.92	33.98	85.72	91.53

Appendix

Table 3: Development observations available before choosing test baselines. Each model/task averages 32 questions over three ordinary batch layouts, giving 96 outputs, not 96 independent questions. GSM and IFEval use strict correctness/compliance; QA columns use answer F1. Values are percentages. Tasks are not pooled into a selection objective. All budgets and unfavorable results are retained.

Model	2b text	2b pass	4b text	4b pass
Llama 3.1 8B	71	12	49	9
Qwen3 8B	126	8	40	3
Mistral 7B	80	2	53	4
Phi-4	83	7	46	0
OLMo 2 1B	44	5	39	0

Appendix

Table 4: Native HQQ variation by model on the development set, with 192 question pairs per model. Pass denotes official task correctness or instruction compliance.

Dataset	2b text	2b pass	4b text	4b pass
GSM8K	142	6	92	12
HotpotQA	32	2	2	0
IFEval	146	22	121	4
MuSiQue	44	0	7	0
SQuAD	20	2	2	0
TriviaQA	20	2	3	0

Appendix

Table 5: Controlled native-HQQ changes by dataset on the development set, pooling five models over 32 shared targets (160 model–question pairs per row). Four-bit pass changes occur only in GSM8K and IFEval; text changes alone do not establish answer errors.

Model/task			FP16	FP32	FP16	FP32	FP16	FP32
	Initial pairs	Initial calls	Different counts		Different caches		Changed answers / generated pairs
Llama/math	128	8,192	3,900	1,872	3,900	272	80/128	62/128
Llama/QA	64	4,096	1,473	316	1,473	160	0/16	0/16
Qwen/math	128	9,216	3,430	806	3,430	101	89/128	67/128
Qwen/QA	64	4,608	1,740	147	1,740	81	0/16	0/16

Appendix

Table 6: Decision precision with unchanged FP16 local updates. Initial calls cover all 384 test pairs (128 math and 64 QA per model), including unchanged answers. Each call pairs the same layer and key/value tensor across companions. Full arithmetic answers cover all 128 questions per model; QA retains the earlier input-selected 16-question subset per model. FP32 denotes the shared stopping mean, not model or update precision.

	$d>0$ median (%)		$e_{w}$ median (%)
Model/task	FP16	FP32	FP16	FP32
Llama/math	0.687	0.108	10.34	10.34
Llama/QA	0.576	0.035	10.48	10.48
Qwen/math	1.084	0.167	11.45	11.43
Qwen/QA	1.018	0.036	11.02	11.02

Appendix

Table 7: Actual initial-cache relative

L_{2}

differences. The companion difference

d

is summarized over nonzero calls only (counts in Table 6 ); quantization error

e_{w}

uses both conditions of every call. Calls and models sharing questions are not independent samples. These norms do not measure answer accuracy.

Different initial caches	Llama	Qwen
0	1/13	22/54
1–4	57/107	45/74
5–16	4/8	–

Appendix

Table 8: FP32 shared stopping in all 256 test math pairs: changed full answers / pairs, grouped by the number of initial cache calls with different target reconstructions. A dash denotes an empty group. Bands are descriptive; initial cache agreement does not exclude later recompression differences.

Model/task	Error	$k=0$	$k=2$	$k=4$	$k=8$	$k=20$
Llama/math	$L_{2}$ (%)	10.7628	10.5295	10.4160	10.3455	10.3371
Llama/math	MAE ( $10^{-3}$ )	55.1617	54.0878	53.5362	53.0082	52.7284
Llama/QA	$L_{2}$ (%)	10.9478	10.7150	10.5895	10.5285	10.5193
Llama/QA	MAE ( $10^{-3}$ )	57.6475	56.5702	55.9558	55.4022	55.1108
Qwen/math	$L_{2}$ (%)	12.0867	11.8574	11.7241	11.6003	11.4400
Qwen/math	MAE ( $10^{-3}$ )	188.9711	184.7405	181.8135	178.8776	176.2468

Appendix

Table 9: Median error of actual stored BF16 initial-cache reconstructions against the raw target. Each stratum uses 16 input-selected question pairs (1,024 Llama or 1,152 Qwen matched calls). Fixed-budget reconstructions agree across companions; shared questions/layers are not independent samples.

L_{2}

is divided by the raw target norm.

k=0

means legacy FP16 initialization with no updates, not the optimize=False baseline. These reconstruction errors do not measure answer quality.

Model	GSM8K test ID	Assertion A	Assertion B	Reference	A to B
Llama	439	#### 17	#### 7	17	correct $\to$ wrong
Llama	1310	#### 64	#### 32	64	correct $\to$ wrong
Llama	1092	#### 2160	#### 1080	1080	wrong $\to$ correct
Llama	406	#### 200	#### 120	200	correct $\to$ wrong
Llama	532	#### 25	#### 35	25	correct $\to$ wrong
Llama	648	#### 56	#### 54	54	wrong $\to$ correct

Appendix

Table 10: All eight confirmed cache-test numerical correctness changes. Assertions display the recognized marker or box and normalized number, not the entire answer or its exact currency formatting. Full raw outputs and extraction anchors are included in the artifact. Companion labels A and B are arbitrary; five changes lose correctness and three gain it.

Pass changes	Same	Different	Unresolved
Yes	5	6	5
No	232	3	5

Appendix

Table 11: Official pass changes versus final numerical values in all 256 test-set native-HQQ arithmetic pairs. Same and Different require resolved numbers on both sides; Unresolved means at least one extraction fails. All 16 official changes occur in this arithmetic subset.

	Llama			Qwen
Method	GSM strict	Final value	Hotpot F1	GSM strict	Final value	Hotpot F1
BF16	78.12	89.06–90.62	70.71	66.41	90.62–96.09	65.08
Native	74.61	87.11–90.23	69.82	69.14	93.36–95.70	62.69
Unoptimized	71.88	83.59–87.50	69.82	64.84	91.41–95.31	65.02
Fixed 4	75.00	86.72–90.62	69.82	66.41	92.97–96.09	65.56
Fixed 20	75.00	87.50–91.41	69.82	67.19	93.75–96.09	62.69

Appendix

Table 12: Test-set quality in percent, averaging the two companion conditions within each target: 128 GSM8K and 64 HotpotQA questions. Final value reports lower and upper correctness bounds obtained by treating unresolved explicit numerical assertions as incorrect or correct, respectively; it is not a reasoning score. Methods other than BF16 use four-bit caches. Fixed four updates was included after inspecting development results, before test-set generation; it is not a proven optimal budget.

Dataset	Model	$N$	Text $\Delta$	Pass $\Delta$	Correct $\Delta$ /all
GSM8K	Llama	128	62.50 [53.91, 71.09]	7.03 [3.12, 11.72]	4.69 [1.56, 8.59]
GSM8K	Qwen	128	69.53 [61.72, 77.34]	5.47 [2.34, 10.16]	1.56 [0.00, 3.91]
HotpotQA	Llama	64	0.00 [0.00, 0.00]	0.00 [0.00, 0.00]	–
HotpotQA	Qwen	64	1.56 [0.00, 4.69]	0.00 [0.00, 0.00]	–

Appendix

Table 13: Observed native four-bit change percentages and descriptive 95% paired question-cluster bootstrap intervals. Every denominator is all target questions in that row. Resolved correctness changes count only pairs whose final numerical assertions resolve on both sides; unresolved pairs may contain additional changes and are not established negatives. Models share questions. Intervals are unadjusted, and a zero-event interval is not a population upper bound.

	Native update count			Complete compression (ms)
Model/task	Median	Range	Below 20	Native	Fixed 20	No optimization
Llama/math	13	9–20	20/24	4.46	4.11	0.48
Llama/QA	14	10–20	16/24	7.58	8.75	0.38
Qwen/math	19.5	9–20	12/24	3.93	3.56	0.34
Qwen/QA	20	10–20	11/24	12.25	11.69	1.18
All cells	18	9–20	59/96	6.35	6.22	0.52

Appendix

Table 14: Actual initial-KV compression on RTX A6000. Each stratum uses two input-selected questions, two companions, three layers and keys/values: 24 batch tensors. Times are geometric means of per-tensor medians from nine repetitions after two warmups. Count tracing and correctness checks are outside timing. Initial-cache compression excludes rolling updates and answer generation; reused inputs and layers are not independent samples.

Method	Time (ms)
Native	8.3345
Unoptimized	1.2849
Fixed 2	2.5221
Fixed 4	3.1205
Fixed 8	4.3243
Fixed 20	7.8219

Appendix

Table 15: Complete 4-bit tensor compression on 36 matched conditions: initialization, optimization, packing, and metadata conversion. Times are geometric means of per-condition medians over nine repetitions, in milliseconds; two warmups are excluded. All methods store 0.28125 times the original tensor bytes.

	Shuffled		Sorted
Method	Long	Short	Long	Short
BF16	74	1	82	3
Native HQQ	115	5	111	11
No optimization	97	6	93	8
Fixed 2	92	5	93	10
Fixed 4	94	7	98	11

Appendix

Table 16: Changed development answers after rebatching, relative to source order. Long: 128 math/instruction pairs; short: 256 QA pairs. Batches contain four questions. All quantized methods use four bits; padding and numerical execution may change.

	Shuffled		Sorted
Method	Strict	Num.	Strict	Num.
BF16	2	2/61	6	2/62
Native HQQ	8	3/61	3	3/62
No optimization	7	1/60	8	1/59
Fixed 2	5	1/60	8	4/60
Fixed 4	7	1/60	10	2/63

Appendix

Table 17: Development GSM8K correctness changes under natural rebatching. Strict: changed strict-marker judgment out of 64 pairs. Num.: changed final-number correctness / pairs resolved in both answers. Unresolved pairs are excluded from that denominator, not treated as errors.

	Changed text / 384 pairs			Math correctness changes / resolved pairs
Shuffle seed	BF16	Native	Fixed 20	BF16	Native	Fixed 20
1	136	218	168	3/248	12/248	9/249
2	130	218	184	3/248	12/250	9/247
3	144	215	190	3/247	10/251	11/249
4	141	205	183	3/247	11/252	9/250
5	138	197	175	3/248	10/249	11/251

Appendix

Table 18: Natural rebatching of the frozen test set: two models and 192 questions, compared with each method’s source order. Six shuffle seeds are the paired units of the sole primary descriptive contrast; length sorting is outside that test. Numerical correctness uses the unchanged final-number extractor. Padding and kernels may change, so these differences do not isolate stopping.

Model/task	Measure	BF16	Native	Fixed 20
Llama/math	Text $\Delta$	50.91	79.69	69.27
	Numeric bounds	88.18–88.87	87.79–89.65	87.21–89.45
Llama/QA	Text $\Delta$	1.30	5.47	4.69
	EM/F1	55.47/70.24	52.54/68.94	52.34/69.08
Qwen/math	Text $\Delta$	54.95	79.69	65.62
	Numeric bounds	90.92–96.48	93.65–96.09	94.24–96.09

Appendix

Table 19: Test rebatching by model and task (%). Text

\Delta

averages changes over six shuffles, with 128 math or 64 QA pairs per model. Quality averages the same questions over all eight layouts. Numeric bounds treat unresolved answers first as wrong, then as correct; they are not confidence intervals. QA gives answer exact match / F1. Repetitions are not independent accuracy samples; no quality superiority is established.

Numerical outcome	Llama	Qwen	Mistral
Same correct	202	224	29
Correct $\to$ wrong	4	2	2
Wrong $\to$ correct	3	1	5
Different wrong	5	1	12
Same wrong	22	13	8
Unresolved	20	15	200

Appendix

Table 20: Final-number outcomes under four-bit HQQ weight compression. Direction is separate to joint; each column partitions all 256 math pairs. Unresolved pairs have at least one unextracted final number. Text-change counts appear in the accompanying results.

Model	Method	Final math numbers	Math official	QA EM	QA F1	Length limit
Llama	BF16	221/28/7	174	55.47	71.02	5
	Native separate	208/36/12	160	56.25	71.06	11
	Native joint	208/35/13	161	56.25	71.06	10
	No optimization	209/36/11	165	56.25	70.38	10
	Fixed 2	209/35/12	161	57.03	70.10	11
	Fixed 4	219/27/10	169	55.47	69.47	8

Appendix

Table 21: All eight methods in the three-model weight-compression evaluation. Each method answers 256 math and 128 QA questions per model. Final math numbers count correct/wrong/unresolved outcomes; official math counts passes under the strict marker parser. QA EM and F1 are percentages. Length limit counts capped outputs across both tasks. Identical BF16 inference on reconstructed weights and an unquantized cache isolates the numerical compression choices; it does not measure int4 deployment speed or memory.

Study	Model	Task	Median length	Full changes	First 64	After 64 only
Cache	Llama	Math	202	80/128	39/128	41/128
Cache	Llama	QA	5	0/64	0/64	0/64
Cache	Qwen	Math	266.5	89/128	36/128	53/128
Cache	Qwen	QA	5	1/64	1/64	0/64
Weight	Llama	Math	216	162/256	80/256	82/256
Weight	Llama	QA	5	7/128	7/128	0/128

Appendix

Table 22: Exploratory analysis of recorded native comparisons. Median lengths include EOS and combine both conditions. Full changes count differing token sequences; their counts equal text changes here. First 64 counts pairs already differing within that prefix; After 64 only counts later first differences. QA early termination is retained, so this is not a controlled length experiment.

Explore similar work

Hurwitz Quaternion Multiplicative Quantization for KV Cache Compression
May 26, 2026Kabir Swain, Sijie Han, Daniel Karl I. Weidele +3Key-Value Cache Quantization Key-Value Cache Compression
Massachusetts Institute of Technology, Cambridge, MA, USA · University of Toronto, Toronto, Canada · IBM Research, Cambridge, MA, USA.
Faithfulness Is Not Free: Auditing Offline KV-Cache Quantization in Retrieval-Augmented Generation
Aug 31, 2026Atta Ul Asad, Ahsan Bilal, Muhammad Ali +2Key-Value Cache Quantization Faithfulness
1Lahore University of Management Sciences (LUMS) · University of Oklahoma · 3Air University +1
Runtime-Certified Bounded-Error Quantized Attention
May 20, 2026Dean CalverKey-Value Cache Quantization Matched Fp16 Intermediate
Independent Researcher

May 26, 2026, cs.LGJ/K move · Enter open · S save

Hurwitz Quaternion Multiplicative Quantization for KV Cache Compression

Kabir Swain, Sijie Han, Daniel Karl I. Weidele, Mauro Martino+2

Massachusetts Institute of Technology, Cambridge, MA, USA · University of Toronto, Toronto, Canada · IBM Research, Cambridge, MA, USA.

Key-Value Cache Quantization Key-Value Cache Compression Data Compression Methods Multi-Head Attention

Open paper AskPDF arXiv

We propose \textbf{Hurwitz Quaternion Multiplicative Quantization (HQMQ)}, a \textbf{calibration-free} method for KV cache compression of large language models. HQMQ treats each 4-element chunk of K or V as a quaternion and quantizes its unit direction to the \emph{product}

q_p \cdot q_s

, where

q_p

ranges over the 24-element Hurwitz group

2T

(the 24 vertices of the 24-cell on

S^3

, pairwise angle

60^\circ

) and

q_s

ranges over a per-(layer, head) secondary codebook of

S

\emph{random} unit quaternions. The multiplicative composition yields

24S

effective codewords at

S

stored parameters; random initialization suffices because left-multiplication is an

S^3

isometry, so seeded codebooks vary in end-task ppl by

<1.5\%

. A per-batch median-multiplier outlier extraction step (

C{=}3

, no calibration) handles modern outlier-heavy architectures. We evaluate on five modern open models: Mistral-7B (dense MHA), Llama-3-8B and Qwen2.5-7B and Qwen3-8B (dense GQA), and gpt-oss-20b (sparse MoE). On Mistral-7B and Qwen3-8B, HQMQ matches fp16 within

0.02

0.03

ppl points at

\sim

5 bits. On Qwen2.5-7B and Qwen3-8B, where naive int4 collapses to

10^4{+}

ppl, HQMQ + Med3

\times

recovers fp16 quality within

0.02

0.10

ppl points at

\sim

5 bits. HQMQ Pareto-dominates naive int by

3

1900\times

at matched bits across all five models, and downstream zero-shot accuracy matches fp16 at

3.79

bits on Mistral. Against the strongest calibrated KV-quantization baseline, HQMQ at

3.79

bits matches KIVI-4 (

\sim 4.5

bits) within

{\sim}1

pt on CoQA,

0.6

pts on TruthfulQA, and

2.3

pts on GSM8K, at

16\%

fewer bits and without a calibration pass. At the storage level, HQMQ delivers up to

5.05\times

KV compression, shrinking a Llama-3-70B 128k-context cache from 43 GB to 8.5 GB.

Abstract

Explore similar work

Hurwitz Quaternion Multiplicative Quantization for KV Cache Compression

Faithfulness Is Not Free: Auditing Offline KV-Cache Quantization in Retrieval-Augmented Generation

Runtime-Certified Bounded-Error Quantized Attention