Medical vision-language models (VLMs) show promise for clinical image understanding, but their computational and memory requirements hinder deployment on resource-constrained clinical hardware. Post-training quantization (PTQ) offers a practical compression strategy without retraining, yet its effectiveness depends strongly on the calibration data. We propose CoreCalib, a training-free and quantizer-agnostic framework for constructing compact calibration sets that preserve the target medical VQA distribution. CoreCalib stratifies data by clinical question class, identifies representative visual modes using frozen DINOv2 features with UMAP and HDBSCAN, and expands selected images into class-balanced question-answer pairs. On Kvasir-VQA-x1, we evaluate CoreCalib with MASQuant, GPTQ, and QIG using a fixed 128-pair budget. CoreCalib achieves the best METEOR with all three quantizers and the best ROUGE-L with MASQuant and GPTQ. With W4A8 MASQuant, it improves METEOR from 0.5672 with COCO calibration to 0.6234, recovering approximately 41% of the degradation relative to BF16. These results support CoreCalib as a data-centric component for resource-efficient medical VLM deployment. Project page: https://magnusdtd.github.io/CoreCalib/ .
Figures & tables
Figure 1: Overview of CoreCalib and its role in resource-efficient medical VLM deployment. (A) A medical VLM provides strong clinical VQA capability but requires substantial computational resources; PTQ converts it into a low-bit model intended to reduce the resource burden across a spectrum of computing environments. (B) CoreCalib constructs a compact calibration workload by (1) stratifying the medical VQA data by clinical question class, (2) selecting density-aware representative images using frozen DINOv2 features with UMAP and HDBSCAN, (3) expanding representative images into multiple QA pairs and balancing them across question classes, and (4) forming a small calibration set, such as 128 QA pairs from 23 images. (C) The calibration set can be used with different PTQ methods, including MASQuant, GPTQ, and QIG, to produce a quantized medical VLM with improved capability retention.
Quantizer
Calibration
BLEU ↑
ROUGE-L ↑
METEOR ↑
Half precision (BF16)
0.4964
0.7005
0.7040
MASQuant (W4A8)
COCO
0.3313
0.5486
0.5672
Random
0.3827
0.5991
0.6170
K-Means
0.3954
0.6063
0.6170
CoreCalib (Ours)
0.3943
0.6103
0.6234
GPTQ (W4A16)
COCO
0.4172
0.6373
0.6545
Table 1: Comparison of calibration strategies on the Kvasir-VQA-x1 test set. Within each quantizer, best and second-best results are shown in bold and underline .
Figure 2: Qualitative examples on the Kvasir-VQA-x1 test set with MASQuant (W4A8) under different calibration sets. Green and red mark answers that are consistent with and contradict the ground truth, respectively.
Variant
Strat.
Discovery
Rank
BLEU ↑
ROUGE-L ↑
METEOR ↑
Random
✓
–
–
0.3827
0.5991
0.6170
K-Means
✓
K-Means
1st
0.3954
0.6063
0.6170
w/o stratification
×
HDBSCAN
1st
0.3905
0.6029
0.6190
2nd member
✓
HDBSCAN
2nd
0.4002
0.6014
0.6173
CoreCalib
✓
HDBSCAN
1st
0.3943
0.6103
0.6234
Table 2: Ablation of CoreCalib components with MASQuant on Kvasir-VQA-x1. Strat.: question-class stratification; Discovery: how representative images are selected; Rank: which representative of each cluster is used.
N
#Images
BLEU ↑
ROUGE-L ↑
METEOR ↑
32
12
0.3851
0.6054
0.6184
64
16
0.3891
0.6066
0.6326
128
23
0.3943
0.6103
0.6234
256
35
0.3962
0.6155
0.6277
Table 3: Effect of the calibration budget N on CoreCalib with MASQuant (W4A8).
Quantizer
Calibration
BLEU ↑
ROUGE-L ↑
METEOR ↑
Half precision (BF16)
0.4815
0.6495
0.3391
GPTQ (W4A16)
COCO
0.4612
0.6220
0.3233
Random
0.4706
0.6249
0.3218
K-Means
0.4609
0.6295
0.3240
CoreCalib (Ours)
0.4645
0.6305
0.3277
Table 4: Comparison of calibration strategies on the VQA-RAD test set. Best and second-best results are shown in bold and underline .
Model
VRAM ↓
Latency ↓
Throughput ↑
Half precision
8.46
4,393.37 ± 2,581.33
3.21
GPTQ (CoreCalib)
2.25
4,495.79 ± 2,659.16
3.14
Table 5: Hardware efficiency of BF16 vs. GPTQ across 200 Kvasir-VQA-x1 test queries on an NVIDIA A100 40GB GPU. VRAM usage is reported in GB, latency in ms, and throughput in tokens/s.