QATMA: Quantization-Aware Training with Multimodal Alignment for Open-Vocabulary Object Detection

Jinyeong Park; Donghwa Kang; Seunghwan An; Insoo Kim; Brent ByungHoon Kang; Hyeongboo Baek; Jibum Kim

cs.CVMar 6, 2026

QATMA: Quantization-Aware Training with Multimodal Alignment for Open-Vocabulary Object Detection

Authors: Jinyeong Park, Donghwa Kang, Seunghwan An, Insoo Kim, Brent ByungHoon Kang, Hyeongboo Baek, Jibum Kim

Organizations: Incheon National University, Incheon, South Korea · KAIST, Daejeon, South Korea · University of Seoul, Seoul, South Korea

Abstract

Quantizing open-vocabulary object detection (OVOD) models reduces their memory and computational costs, but extremely low-bit quantization severely degrades both cross-modal (region-text) and intra-modal (region-region) alignments. This multimodal degradation is a unique challenge that prior quantization methods for closed-vocabulary detectors fail to resolve. To overcome this, we propose Quantization-Aware Training with Multimodal Alignment (QATMA), the first multimodal-aware and architecture-agnostic QAT framework tailored for OVOD. QATMA integrates two key components: (i) Curriculum QAT, which partitions the detector by functional roles and progressively expands the quantization scope to suppress error accumulation and ensure stable optimization; and (ii) Text-anchored Pairwise Similarity Distillation, which transfers both region-text and region-region alignments from a full-precision teacher model via pairwise cosine similarities in the joint embedding space. Experimental results on LVIS and COCO zero-shot benchmarks demonstrate that QATMA significantly outperforms existing QAT baselines under extremely low-bit settings, achieving gains of up to 4.3 and 7.6 AP, respectively.

Figures & tables

Panel of Figure 1 — Figure 1: Impact of 4-bit quantization on YOLO-World [ 7 ] ( Objects365v2 [ 44 ] ). (a) Confidence scores from the similarity between each region embedding and the “Train” text embedding ( $\bar{P}$ : mean over positive regions). (b) Pairwise cosine similarity matrix among positive region embeddings within the same category ( $r$ : Pearson correlation with FP32). (c) Quantitative comparison of embedding distortion relative to FP32; closer to the origin indicates less distortion.

Figure 2: Overview of the proposed QATMA framework. ( Red ) blocks denote quantized and learnable modules, and ( blue ) blocks denote full-precision and frozen modules. (a) Stage 1: the backbone ( $M_{1}$ ) is quantized with $\mathcal{L}_{\text{feat}}$ supervision from the full-precision teacher, while the neck-head remains frozen for error isolation. (b) Stage 2: the neck-head ( $M_{2}$ ) is additionally quantized, supervised by both $\mathcal{L}_{\text{feat}}$ and $\mathcal{L}_{\text{TPSD}}$ . (c) Feature distillation aligns the student’s multi-scale backbone features $f_{1}^{S}(x)$ to those of the teacher $f_{1}^{T}(x)$ . (d) TPSD groups region embeddings by text query $\mathbf{t}_{c}$ and constructs a unified pairwise similarity matrix $\mathbf{S}_{c}$ to transfer region-text and region-region alignment.

Model	Bits	Size(MB)	BOPs(T)	Method	AP	AP r	AP c	AP f
YOLO-World-M	FP32	110.88	51.82	-	31.0	23.8	29.2	33.9
	4-4-8	15.11	7.10	QAT	13.8	10.7	11.6	16.3
				EMA+QC [ 16 ]	13.7	11.8	12.2	15.4
				QFD [ 57 ]	13.0	7.1	11.0	15.8
				Ours	16.3	11.4	14.5	18.8
YOLO-World-L	FP32	181.4	98.39	-	35.4	27.6	34.1	38.0

Table 1: LVIS minival zero-shot evaluation with 4-4-8 quantization (Ch-T-H granularity). Ch = per-channel, T = per-tensor, H = per-head. Evaluation follows the LVIS fixed AP protocol [ 10 ] .

Model	Bits	BOPs(T)	Method	AP	AP 50	AP 75	AP s	AP m	AP l
YOLO-World-M	FP32	44.56	-	41.9	57.0	45.5	26.7	46.1	55.0
	4-4-8	2.05	QAT	22.0	32.1	23.6	7.8	23.5	32.4
	4-4-8	2.05	Ours	26.1	38.1	28.1	11.0	28.1	37.9
YOLO-World-L	FP32	90.60	-	45.1	60.7	48.9	29.8	49.8	57.5
	4-4-8	3.09	QAT	18.4	27.2	19.6	7.5	20.1	26.9
	4-4-8	3.09	Ours	25.2	36.7	27.0	10.5	27.5	37.7

Table 2: COCO val2017 zero-shot evaluation with 4-4-8 quantization (Ch-T-H granularity)

Table 5

Panel of Figure 4 — Figure 4: Qualitative comparison on YOLO-World-L (4-4-8, Ch-T-H). ( Top ) Detection results. ( Bottom ) Region-region similarity heatmap of average pairwise cosine similarity among same-class anchors. QAT distorts both detection and similarity patterns of FP32, whereas QATMA restores them.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Bits	BOPs(T)	Method	AP	AP 50	AP 75	AP s	AP m	AP l
YOLO-World-M	FP32	44.56	-	41.9	57.0	45.5	26.7	46.1	55.0
	4-4-8	2.05	QAT	22.0	32.1	23.6	7.8	23.5	32.4
			EMA+QC [ 16 ]	21.2	31.1	22.7	8.3	22.7	31.3
			QFD [ 57 ]	21.6	31.4	23.3	7.9	23.7	31.8
			Ours	26.1	38.1	28.1	11.0	28.1	37.9
YOLO-World-L	FP32	90.60	-	45.1	60.7	48.9	29.8	49.8	57.5

Appendix

Table 7: Full COCO val2017 zero-shot comparison, including the EMA+QC [ 16 ] and QFD [ 57 ] closed-vocabulary detection QAT baselines (4-4-8, Ch-T-H granularity).

Granularity	Bits	Method	LVIS	COCO
Ch-T-H	4-5-8	QAT	23.0	36.2
Ch-T-H	4-5-8	Ours	23.8	37.6
Ch-Ch-H	3-3-8	QAT	18.5	30.7
Ch-Ch-H	3-3-8	Ours	20.0	32.6

Appendix

Table 8: Effect of quantization granularity and bit-width on YOLO-World-L. Ch-Ch-H = per-channel activations. LVIS and COCO zero-shot AP is reported.

Stage 1 (backbone)	Stage 2 (neck-head)	AP
1/3	2/3	16.1
1/2	1/2	15.0

Appendix

Table 9: Effect of the curriculum data split between Stage 1 (backbone) and Stage 2 (neck-head) on YOLO-World-L (4-4-8, Ch-T-H). LVIS zero-shot AP is reported.

$\lambda_{2}$	0	3	6	12	18	48
AP	30.4	30.8	31.2	31.2	31.0	30.9

Appendix

Table 10: TPSD loss weight

\lambda_{2}

sensitivity on YOLO-World-L (head-only W4A4). LVIS zero-shot AP is reported.

Method	GFLOPs/iter	GPU-hours
QAT	27.1k	95.2
Ours	35.1k	112.4

Appendix

Table 11: Training cost on YOLO-World-L. QATMA adds no inference cost, and its two-stage training overhead over plain QAT is modest.

Panel of Figure 5 — Figure 5: Qualitative comparison of LVIS zero-shot detection on YOLO-World-L (4-4-8, Ch-T-H)

VLM Quantization

Vision-language model quantization reduces model precision to lower memory and computation while preserving multimodal capabilities.

See all 52 papers

Zero-Shot Quantization for Object Detectors using Off-the-Shelf Generative Models
Jun 30, 2026Hyunho Lee, Kyomin Hwang, Hyeonjin Kim +3Quantization-Aware Training Synthetic Data Generation
Seoul National University, Seoul, South Korea · LG Electronics, Seoul, South Korea
DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection
Apr 3, 2026Siheng Wang, Yanshu Li, Bohan Hu +12Open-Vocabulary Object Detection Detection Transformer
Jiangsu University · MBZUAI · Brown University +5
MOCHA: Multi-modal Objects-aware Cross-arcHitecture Alignment
Sep 17, 2025Elena Camuffo, Francesco Barbato, Mete Ozay +2Cross-Modal Knowledge Distillation Efficient Object Detection
Samsung R&D Institute UK, United Kingdom · University of Padova, Italy

Jun 30, 2026J/K move · Enter open · S save

Zero-Shot Quantization for Object Detectors using Off-the-Shelf Generative Models

Hyunho Lee, Kyomin Hwang, Hyeonjin Kim, Suyoung Kim+2

Seoul National University, Seoul, South Korea · LG Electronics, Seoul, South Korea

Quantization-Aware Training Synthetic Data Generation Object Detection Low-Bit Quantization Neural Network Quantization

Open paper AskPDF arXiv

With an increasing number of Object Detection (OD) models being deployed on edge devices, Zero-Shot Quantization for OD (ZSQ-OD) aims to quantize these models when access to the original training data is prohibited. Existing research on Zero-Shot Quantization-Aware Training (QAT) for OD synthesizes training sets through noise optimization. However, this approach struggles to maintain performance in low-bit regions. In this paper, we introduce GoodQ (Generative off-the-shelf models for object detector Quantization), a QAT pipeline that utilizes off-the-shelf generative models to construct a training set. We first identify three challenges that arise when introducing a generative model to the ZSQ-OD task: 1) each image contains dense information with multiple instances, 2) the class-wise distribution in the original dataset is imbalanced, and 3) the pseudo-labels assigned to the generated images can potentially act as noisy signals during QAT. GoodQ addresses these challenges by 1) introducing an Information-Dense Prompting strategy to generate multi-instance images, 2) applying Intrinsic Distribution-Aware Selection to match the pretrained class distribution, and 3) employing Teacher-guided Adaptive Noise Reduction to mitigate noise arising from the QAT process. Our framework achieves state-of-the-art performance in low-bit ZSQ (W4A4) and extends quantization to extreme bit-widths (W3A3). Furthermore, we conduct an extensive analysis to uncover the underlying factors contributing to the efficacy of GoodQ.

Abstract

VLM Quantization

Explore similar work

Zero-Shot Quantization for Object Detectors using Off-the-Shelf Generative Models

DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection

MOCHA: Multi-modal Objects-aware Cross-arcHitecture Alignment