GHOST-Q: Towards Studying Grounding Hallucinations Overlooked Under Same-score TradeOffs in Quantized VLMS

Saim Rehman; Muhammad Shafique

cs.CVSep 24, 2026

GHOST-Q: Towards Studying Grounding Hallucinations Overlooked Under Same-score TradeOffs in Quantized VLMS

Authors: Saim Rehman, Muhammad Shafique

Organizations: eBRAIN Lab, Division of Engineering, New York University Abu Dhabi (NYUAD), Abu Dhabi, UAE

Abstract

Post-training quantization of vision--language models (VLMs) is typically assessed through aggregate task accuracy and memory savings, but preserving a headline score does not guarantee preservation of visual grounding behavior. We present GHOST-Q, a cross-precision controlled evaluation of three 8B VLM families under FP16, INT8, and NF4 across utility and hallucination-sensitive benchmarks. Rather than comparing only aggregate accuracy, we pair FP16 and quantized predictions item by-item to quantify how compression redistributes grounding successes and failures. Five of six quantized variants preserve MMStar accuracy within $\pm2$ percentage points, yet 10 of 36 paired effects remain significant after false-discovery-rate correction, nine on hallucination-sensitive conditions. Same-device A100 profiling further demonstrates that substantial memory reduction does not necessarily mean lower inference latency. Finally, an open-ended AMBER audit reveals strong generation budget censoring whose severity varies by architecture and precision. These results show that quantized VLMs should be evaluated jointly for aggregate utility, grounding reliability, generation behavior, and realized deployment efficiency.

Figures & tables

Figure 1: Overview of the Ghost-Q evaluation pipeline and main analyses.

Model	Prec.	MM*	POPE	A-E	A-A	A-R	Hall.	Macro
Qwen3	FP16	63.53	88.68	92.73	86.88	85.22	72.77	85.25
	INT8	64.07	88.51	93.70	86.72	85.58	73.92	85.69
	NF4	62.80	87.89	94.01	86.43	85.40	71.71	85.09
InternVL3	FP16	66.47	90.87	91.96	87.02	83.35	65.93	83.83
	INT8	65.67	91.00	92.18	86.93	83.35	64.98	83.69
	NF4	64.40	90.92	92.73	86.63	84.92	64.56	83.95

Table 1: Accuracy (%) across utility and grounding-sensitive tasks. “Macro” is the unweighted mean of POPE, three AMBER subsets, and HallusionBench.

Model	Task	Prec.	$\Delta$	C $\rightarrow$ W	W $\rightarrow$ C	$q$	OR [95% CI]
Qwen3	A-exist	INT8	+0.97	4	52	$<10^{-8}$	13.00 [4.78, 49.50]
Qwen3	A-exist	NF4	+1.28	14	77	$<10^{-8}$	5.50 [3.09, 10.53]
Idefics3	POPE	NF4	-1.24	257	145	$3.0\times 10^{-7}$	0.56 [0.46, 0.69]
Qwen3	POPE	NF4	-0.79	149	78	$2.6\times 10^{-5}$	0.52 [0.39, 0.69]
Idefics3	A-exist	NF4	+1.22	67	127	$1.4\times 10^{-4}$	1.90 [1.40, 2.59]
InternVL3	A-exist	NF4	+0.77	27	65	$5.6\times 10^{-4}$	2.41 [1.52, 3.92]

Table 2: FDR-significant FP16–quantized paired effects.

\Delta

is quantized minus FP16 accuracy (pp). C

\rightarrow

W/W

\rightarrow

C denote harmful/beneficial flips.

q

is the Benjamini–Hochberg-adjusted McNemar

p

-value. Matched OR

>1

indicates more beneficial than harmful flips; OR

<1

indicates the reverse.

Figure 2: Accuracy change relative to each architecture’s FP16 baseline. Stars mark comparisons significant after Benjamini–Hochberg correction over all 36 paired tests (

q<0.05

Model	Prec.	Foot.	Peak	E2E 50	E2E 95	Thr.
		(GiB)	(GiB)	(ms)	(ms)	ex/s
Qwen3	FP16	16.33	16.53	76.6	86.9	13.00
	INT8	9.33	9.63	291.3	309.0	3.40
	NF4	5.83	6.18	129.8	134.7	7.70
InternVL3	FP16	14.80	16.01	354.5	363.8	3.68
	INT8	8.41	9.63	566.0	578.7	2.12

Table 3: Same-device A100 profile. E2E values are median/p95 latency over 600 measured inferences per configuration.

Figure 3: Measured footprint–latency trade-off on the same A100 GPU. Quantization consistently moves left (smaller footprint) but not down (faster inference); NF4 dominates INT8 in this two-dimensional efficiency plane for all three architectures.

Model	FP16	INT8	NF4
Qwen3-VL	794 (79.1%)	768 (76.5%)	734 (73.1%)
InternVL3	920 (91.6%)	913 (90.9%)	844 (84.1%)
Idefics3	1001 (99.7%)	1001 (99.7%)	1002 (99.8%)

Table 4: AMBER open-ended generation under a common 256-token budget. Entries are responses that hit the token ceiling out of 1,004.

Explore similar work

Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs
Jan 30, 2026Yanlong Chen, Amirhossein Habibian, Luca Benini +1Recent Vision-Language Models Quantization-Aware Training
Department of Information Technology and Electrical Engineering, ETH Zurich, Zurich, Switzerland · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Qualcomm AI Research, Amsterdam, the Netherlands +1
VLAQuantBench: Closed-Loop Evaluation of Post-Training Quantization for Vision-Language-Action Models
Sep 21, 2026Jiuyi Xu, Qing Jin, Meida Chen +3Post-Training Quantization Closed Loop
Colorado School of Mines · Independent Researcher · University of Central Florida +2
Does Playing it Safe Count as Faithfulness? Reassessing LVLM Hallucination Mitigation Methods
Sep 1, 2026Mehrdad Fazli, Sina Mansouri, Mohit Marvania +1Large Language Model Hallucination Hallucination Mitigation
Department of Computer Science, George Mason University Fairfax, VA 22030, USA

Jan 30, 2026, cs.CVJ/K move · Enter open · S save

Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs

Yanlong Chen, Amirhossein Habibian, Luca Benini, Yawei Li

Department of Information Technology and Electrical Engineering, ETH Zurich, Zurich, Switzerland · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Qualcomm AI Research, Amsterdam, the Netherlands+1

Recent Vision-Language Models Quantization-Aware Training Information Bottleneck

Open paper AskPDF arXiv

Vision-Language Models (VLMs) achieve strong multimodal performance but are costly to deploy, and post-training quantization often causes significant accuracy loss. Despite its potential, quantization-aware training for VLMs remains underexplored. We propose GRACE, a framework unifying knowledge distillation and QAT under the Information Bottleneck principle: quantization constrains information capacity while distillation guides what to preserve within this budget. Treating the teacher as a proxy for task-relevant information, we introduce confidence-gated decoupled distillation to filter unreliable supervision, relational centered kernel alignment to transfer visual token structures, and an adaptive controller via Lagrangian relaxation to balance fidelity against capacity constraints. Across extensive benchmarks on LLaVA and Qwen families, our INT4 models consistently outperform FP16 baselines (e.g., LLaVA-1.5-7B: 70.1 vs. 66.8 on SQA; Qwen2-VL-2B: 76.9 vs. 72.6 on MMBench), nearly matching teacher performance. Using real INT4 kernel, we achieve 3

\times

throughput with 54% memory reduction. This principled framework significantly outperforms existing quantization methods, making GRACE a compelling solution for resource-constrained deployment. Code and data are available at: https://github.com/ForeverBlue816/GRACE.

Abstract

Explore similar work

Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs

VLAQuantBench: Closed-Loop Evaluation of Post-Training Quantization for Vision-Language-Action Models

Does Playing it Safe Count as Faithfulness? Reassessing LVLM Hallucination Mitigation Methods