BEFORE THE FLIP: Measuring Hidden Score Shifts In Quantized Vision Language Models Before The Answer Changes for Visual Question Answering

Sourajit Saha; Shubhashis Roy Dipta; Shaswati Saha; Nobin Sarwar; Yuxuan Jiang

cs.CVSep 7, 2026

BEFORE THE FLIP: Measuring Hidden Score Shifts In Quantized Vision Language Models Before The Answer Changes for Visual Question Answering

Authors: Sourajit Saha, Shubhashis Roy Dipta, Shaswati Saha, Nobin Sarwar, Yuxuan Jiang

Organizations: University of Maryland, Baltimore County

Abstract

Quantization makes vision language models (VLMs) cheaper to store and run by using fewer bits to represent their weights. While unchanged answers on visual question answering (VQA) after compression are an expected behavior, they can still hide changes in the underlying scores (log probabilities). For example, a model may still answer yes after compression, even as the score gap between yes and no shrinks. We introduce BEFORE THE FLIP to measure these hidden changes. Our method compares the score change caused by compression with the change caused by replacing the image's internal representations, or image tokens, with one fixed average token. We then increase the precision of one weight group at a time to identify where extra bits help, and test whether choosing different groups for each question offers benefits beyond shuffled controls. Among 8,277 LLaVA questions where image token replacement measurably affects the scores, 4-bit compression shifts the yes or no score gap farther toward the replacement output than 8-bit compression. Qwen shows the same pattern, but with a smaller difference. Yet only 265 of 9,000 LLaVA answers change at 4 bits. In a separate study of 1,024 calibration questions, choosing weight groups separately for each question does not outperform both shuffled controls at any tested storage budget. These findings show that compression can alter the scores behind unchanged answers, but do not establish a reliable benefit from adjusting precision for each question.

Figures & tables

Figure 2: Overview of Before The Flip . Stage 1 measures whether compression moves the answer scores toward the output produced by average image tokens. Stage 2 raises each of $57$ weight groups from 4 to 8 bits, one at a time, and records the reduction in loss. Stage 3 tests whether selecting groups for each question outperforms one fixed group set and two shuffled controls, and determines whether those gains are reliable.

Weights	F1 $\uparrow$	Acc. $\uparrow$	Mean PRS	PRS ${>}0$	Response ratio	Flip match real/shuffled	$p$
8-bit	84.811	86.067	.0094	63.7%	1.008	.455/.456	.974
4-bit	84.722	85.922	.0760	72.7%	1.058	.426/.427	.722

Table 1: Compression shifts the answer scores, and the measured shift depends on the image replacement. Panel (a) shows a larger PRS under W4 than W8 despite similar final performance. Panel (b) compares W4 across five image replacements and shows that the direction and size of PRS depend on the chosen replacement. PRS uses

8{,}277

filtered questions, while final answer measures use all

9{,}000

questions.

Figure 3: Weight group sensitivity and shuffle controls. (a) Raising earlier decoder blocks to 8 bits reduces the loss more on average. (b) Individual group changes can either reduce or increase the loss. (c) At every storage budget, the per question gain fails to exceed both shuffled controls.

Setting	Acc.	PRS +	Measured loss	Bits/weight for one question	Total stored GiB
All weights 4-bit	85.92	.241	.514	4.00	3.17
All weights 8-bit	86.07	.032	.076	8.00	6.33
Image encoder 8-bit	85.62	.167	.295	4.18	3.31
Fixed 10% average token plan	85.76	.176	.307	4.39	3.48
Selector choice, before VCD	85.79	.181	.328	4.38	3.79

Table 2: The fixed

10\%

plan lowers PRS + and measured loss from W4. The

95\%

interval for its PRS + difference from the image encoder rule includes zero. PRS + and loss use all

9{,}000

questions. Total stored GiB includes every selector candidate.

Figure 4: Results across tasks, models, and decoding settings. (a) W4 changes five task averages by less than one point relative to 16 bit. (b) W4 produces higher PRS than W8 on both LLaVA and Qwen, although the Qwen gap is smaller. (c) The VCD setting with the lowest mean PRS does not produce the highest F1.

Figure 5: Two examples of hidden answer changes. In both cases, 16 bit predicts yes , while average tokens and W4 predict no . The fixed K3_P2 profile restores the truck answer but not the spoon answer. These examples were selected after evaluating all

9{,}000

questions.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Questions used	Weight bits	Avg. PRS	PRS +	Positive (%)	Kept (%)
$\lVert d_{i}\rVert>\tau_{d}$	W8	0.00940	0.01842	63.73	91.97
$\lVert d_{i}\rVert>\tau_{d}$	W4	0.07596	0.12701	72.66	91.97
All questions	W8	0.00081	0.03247	62.31	100.00
All questions	W4	0.04582	0.24101	70.67	100.00

Appendix

Table 3: The stability filter changes the score size. The first two rows remove weak image-effect cases with cutoff

\tau_{d}=0.302

. The last two rows keep every question. Bit widths should be compared only within the same pair of rows.

Cutoff $\tau_{d}$	Bits	Random	Kept	Popular	Kept	Adversarial	Kept
5th pct. ( $0.201$ )	W8	.009	96	.011	95	.010	94
	W4	.062	96	.077	95	.064	94
10th pct. ( $0.302$ )	W8	.009	94	.010	91	.009	91
	W4	.061	94	.091	91	.076	91
20th pct. ( $0.581$ )	W8	.009	87	.010	82	.008	81
	W4	.059	87	.082	82	.072	81

Appendix

Table 4: W4 remains more positive at all three stability cutoffs. “Kept” is the percentage of questions retained. Each PRS value uses only the retained questions.

How image information is replaced	Yes (%)	F1	W4 random	W4 popular	W4 adversarial
Calibration mean token	0.6	1.4	.061	.091	.076
Per-image mean token	2.1	6.2	.066	.091	.081
Zero tokens	100.0	66.7	.049	.070	.068
Shuffled-image tokens	27.8	64.1	.043	.043	.043
Uniform gray image	0.0	0.0	.004	$-.042$	$-.039$

Appendix

Table 5: Four token replacements agree in sign, while a gray input image does not. Yes-rate and F1 describe the 16-bit model after replacement. The final columns report filtered W4 PRS.

Group	Model part	Loss drop	Acc. change	Group	Model part	Loss drop	Acc. change
vis.13	vision	.0316	.20	dec.13	decoder	.0223	.10
vis.06	vision	.0292	.29	dec.10	decoder	.0209	$-.29$
dec.01	decoder	.0249	$-.49$	dec.06	decoder	.0197	.00
vis.08	vision	.0239	.68	dec.08	decoder	.0193	.10
vis.14	vision	.0189	.10	dec.20	decoder	$-.0043$	$-.10$
dec.14	decoder	$-.0061$	.00	dec.28	decoder	$-.0119$	.59

Appendix

Table 6: Changing one group from W4 to W8 can reduce or increase loss. “Loss drop” is

\Delta^{F}

, and positive is better. Accuracy changes are percentage points. The table includes representative large and negative effects. No single ordering fits every block.

Block	Whole block	Token mixing	FFN	Parts minus whole	Larger gain
dec.00	.0087	.0088	$-.0033$	$-.0032$	attention
dec.01	.0249	$-.0001$	.0275	.0025	FFN
dec.06	.0197	.0217	$-.0039$	$-.0020$	attention
dec.08	.0193	.0100	.0082	$-.0011$	shared
dec.10	.0209	.0213	.0005	.0009	attention
dec.13	.0223	.0049	.0122	$-.0052$	FFN

Appendix

Table 7: No decoder subpart is always the best precision target. These eight representative blocks are a subset of the 13 tested. Entries are drops in faithfulness loss, and positive is better. The two subparts can move in opposite directions within one block.

Extra storage (% of W4)	One fixed plan	Hindsight best	Real advantage	Shuffle A	Shuffle B	Both?
5%	.2595	1.3561	1.0966	1.4557	.9579	no
10%	.3067	1.6602	1.3535	1.6616	1.2309	no
15%	.3276	1.8734	1.5458	1.8313	1.4778	no

Appendix

Table 8: Per-question hindsight gains stay below at least one shuffled control. Each budget is the extra weight storage allowed for one question. Shuffle A preserves each group’s score distribution. Shuffle B preserves each question’s score distribution. Only one value is available for each shuffle and budget. Shuffle counts and seeds are unavailable.

How weight bits are chosen	Mean extra bytes used (% of W4)	Loss (lower is better)	Coverage (%)	Gap to hindsight	Choice
W4	0.0	.5502	–	–	fixed
Best fixed K3_P3	7.0	.3557	–	–	fixed
Random reassignment	9.4	.4075	–	–	same frequencies
Distribution-distance rule	9.4	.4075	0.0	.2229	100% K3_P2
Linear ridge model	9.4	.4075	0.0	.2229	100% K3_P2
Two-layer neural net	9.4	.4075	0.0	.2229	100% K3_P2

Appendix

Table 9: All three selectors assign K3_P2, but VCD is applied to every example. Results use held-out calibration examples. Extra bytes are the mean used for one question. Storing all candidates costs 19.6% more than W4. Every upper prediction exceeds the W8-based limit.

Panel of Figure 6 — Figure 6: A looser W8 loss limit includes more questions but does not improve prediction. Left: even hindsight needs a looser limit. Right: errors of $0.43$ – $0.64$ are about as large as the W4-to-W8 loss span, $0.55$ to $0.07$ . In the left plot, “risk” means faithfulness loss.

Precision plan	W8 groups	Extra storage (% of W4)	Loss	PRS +	Calibration accuracy	Skill change
W4	0	0.0	.6021	.2599	81.2	.05
P0 (capstep01)	1	.2	.5694	.2365	81.4	$-.02$
K3_P1	14	8.3	.4629	.1830	81.4	.34
K3_P2	20	9.4	.4297	.1865	81.2	.21
K3_P3	23	7.0	.4003	.1679	80.9	$-.03$

Appendix

Table 10: Representative fixed plans reduce loss on the calibration evaluation used to build the profiles. “Skill change” is the percentage-point change on general capability evaluations. Lower loss is better.

Storage budget per question	Selection target	W8 groups	Calculated extra storage (% of W4)	Loss
5%	average-token score	21	3.9	.4165
	general-skill score	22	4.2	.3732
10%	average-token score	23	9.8	.3532
	general-skill score	23	9.8	.3281
15%	average-token score	24	12.8	.3257
	general-skill score	25	13.1	.3266

Appendix

Table 11: Larger per-question storage budgets continue to change the plan. These losses come from the calibration-set budget sweep. They cannot be compared directly with the full-POPE losses in Table 13 .

Protected groups	Count	Extra storage (% of W4)	Skill score $\downarrow$	Avg. accuracy change	Worst raw change	Resolved drops
W4, empty set	0	0.0	.0088	.05	$-3.33$	0
Top 1	1	.2	.0087	$-.02$	$-3.33$	0
Top 2	2	.4	.0086	$-.15$	$-3.33$	0
Top 3	3	.6	.0086	.04	$-3.20$	0
Top 4	4	.7	.0084	.03	$-3.20$	0
Top 6	6	1.1	.0081	.16	$-3.20$	0

Appendix

Table 12: Every candidate meets the skill criterion, including the empty protection set. Skill score is the Jensen–Shannon distance from the 16-bit answer distribution. Lower is better. Accuracy changes are percentage points from the 16-bit model. A resolved drop must exceed both 2 points and two standard errors for that task’s sample size.

Precision plan	F1 random	F1 popular	F1 adversarial	Accuracy	Yes (%)	PRS +	Loss	Avg. weight bits
16-bit reference	86.1	85.0	83.0	86.0	41.7	.000	.0000	16.00
Uniform W8	86.2	85.2	83.2	86.1	41.7	.032	.0761	8.00
Uniform W4	86.5	84.9	82.9	85.9	42.1	.241	.5141	4.00
Vision encoder W8	86.1	84.6	82.6	85.6	42.1	.167	.2952	4.18
Vision encoder + connector W8	86.1	84.6	82.7	85.7	42.1	.163	.2846	4.19
First 8 decoder blocks W8	86.6	84.9	82.8	85.9	42.3	.201	.4577	4.95

Appendix

Table 13: Internal answer scores move while final POPE answers remain nearly constant. PRS + and loss use all 9,000 questions. Average weight bits are calculated from parameter counts. The 16-bit model’s value is 16.

Precision plan	Sent. err.	Wrong obj.	Recall	Words	Obj./cap.	$\Delta$ sent. err. [95% CI]
16-bit reference	24.0	6.9	63.5	49.4	5.04	$-.60$ [ $-4.00,2.80$ ]
W8	24.6	7.0	63.9	49.4	5.08	$.00$ [ $-3.40,3.40$ ]
W4	24.6	7.3	61.9	49.6	4.98	reference
K3_P1	25.0	7.5	63.2	49.3	5.08	$.40$ [ $-2.40,3.40$ ]
K3_P2	25.2	7.6	63.1	49.4	5.10	$.60$ [ $-2.60,3.80$ ]
K3_P3	23.2	6.5	63.1	49.4	5.08	$-1.40$ [ $-4.60,1.60$ ]

Appendix

Table 14: Differences in unsupported object mentions are unresolved. Error rates and recall are percentages. Changes and intervals are percentage points. Intervals come from 10,000 paired image-level resamples.

Model	Weight bits	F1	Avg. PRS	Image-response ratio	Matching flips	Shuffled flips	$p$
LLaVA-1.5-7B	W8	84.81	.00940	1.0078	.4545	.4563	.9740
	W4	84.72	.07596	1.0578	.4264	.4269	.7225
Qwen2.5-VL-7B	W8	88.94	.00017	1.0000	.2647	.4451	.9980
	W4	88.62	.01397	.9985	.5766	.5372	.1979

Appendix

Table 15: Both models show W4 above W8, but Qwen’s shift is smaller. Mean PRS uses each model’s filtered questions. F1, image-response ratio, and flip rates use all questions. LLaVA uses cutoff

\tau_{d}=0.302

. Qwen uses

0.45299

. Their score sizes are therefore not directly comparable.

Image correction	Accuracy	F1	Yes (%)	Avg. PRS	PRS +	Loss
W4, no VCD	85.92	84.72	42.1	.0458	.241	.514
999 steps, $\alpha=.25$	–	85.74	48.3	$-.138$	–	.987
999 steps, $\alpha=.50$	–	84.98	52.1	$-.321$	–	1.889
999 steps, $\alpha=1$	83.19	84.41	57.8	$-.688$	1.243	4.122
500 steps, $\alpha=.25$	–	84.9	45.4	$-.140$	–	.769
500 steps, $\alpha=.50$	–	85.1	48.4	$-.327$	–	1.333

Appendix

Table 16: PRS and F1 favor different correction strengths. Every row uses

\beta=0.1

. Missing statistics from the corruption sweep are shown as dashes. They are not estimated. PRS uses all questions.

Precision and correction plan	Acc.	F1	Yes	Avg. PRS	Positive	PRS +	Loss	VCD used
16-bit reference	86.0	84.7	41.7	.000	0	.000	.000	0
W4, no VCD	85.9	84.7	42.1	.046	71	.241	.514	0
W8, no VCD	86.1	84.8	41.7	.001	62	.032	.076	0
W4 + always-on VCD	83.2	84.4	57.8	$-.688$	32	1.243	4.122	100
Fixed K3_P3	85.6	84.4	42.2	.044	66	.161	.295	0
Fixed K3_P3 + VCD	83.0	84.3	57.8	$-.696$	30	1.236	4.100	100

Appendix

Table 17: Selective and always-on VCD are identical because no upper prediction falls below the loss limit. Yes, Positive, and VCD used are percentages. Every VCD row uses

\alpha=1

and

\beta=0.1

. It also uses the same weight precision for the real and corrupted images.

VLM Quantization

Vision-language model quantization reduces model precision to lower memory and computation while preserving multimodal capabilities.

See all 53 papers

Query Independent Variable Rate Visual Token Coding
Sep 20, 2026Hongbo Zhang, Zihao Yang, Liuyang Song +4Visual Token Compression Token Compression
Peking University · Uppsala University · University of Southern California
Messages, Not Tokens: Grounded Coresets for Faithful VLM Compression
Aug 3, 2026Long Qian, Jiaqi Wei, Bingke Zhu +2Efficient VLM Inference Coreset Selection
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Engineering, Cardiff University, Cardiff, United Kingdom · School of Future Technology, University of Chinese Academy of Sciences, Beijing, China +1
Edge Reliability Gap in Vision-Language Models: Quantifying Failure Modes of Compressed VLMs Under Visual Corruption
Mar 24, 2026Mehmet Kaan ErolCompositional Reasoning in VLMs VLM Evaluation
Marmara University, Institute of Pure and Applied Sciences

Sep 20, 2026J/K move · Enter open · S save

Query Independent Variable Rate Visual Token Coding

Hongbo Zhang, Zihao Yang, Liuyang Song, Daqian Yang+3

Peking University · Uppsala University · University of Southern California

Visual Token Compression Token Compression Vision-Language Models Rate-Distortion Optimization

Open paper AskPDF arXiv

Visual-token compression for vision--language models is posed almost entirely as a selection problem: decide which tokens to keep and discard the rest. The criteria that work best rank tokens by the attention the language model pays them, which makes the ranking a function of the question being asked. That is invisible in a single-turn benchmark and decisive whenever a compressed representation is written once and read many times, as when it is cached across the turns of a conversation or transmitted between a device and a server. We take the other half of the classical transform-coding toolkit instead: keep every token and vary its rate. A transform code exposes each token's measured distortion--rate curve, and a fixed bit budget is distributed across tokens by exact integer rate--distortion optimisation on those curves. No text enters the pipeline, so one compressed representation serves any query. At equal bit budgets, on two datasets and two capacities, it preserves the model's output distribution and its answers better than uniform-rate coding, the closed-form water-fill and distortion-ranked pruning. It matches attention-ranked pruning on the question pruning was tuned for, and overtakes it once the compressed image must answer a different question about the same image.

Abstract

VLM Quantization

Explore similar work

Query Independent Variable Rate Visual Token Coding

Messages, Not Tokens: Grounded Coresets for Faithful VLM Compression

Edge Reliability Gap in Vision-Language Models: Quantifying Failure Modes of Compressed VLMs Under Visual Corruption