Quantizing open-vocabulary object detection (OVOD) models reduces their memory and computational costs, but extremely low-bit quantization severely degrades both cross-modal (region-text) and intra-modal (region-region) alignments. This multimodal degradation is a unique challenge that prior quantization methods for closed-vocabulary detectors fail to resolve. To overcome this, we propose Quantization-Aware Training with Multimodal Alignment (QATMA), the first multimodal-aware and architecture-agnostic QAT framework tailored for OVOD. QATMA integrates two key components: (i) Curriculum QAT, which partitions the detector by functional roles and progressively expands the quantization scope to suppress error accumulation and ensure stable optimization; and (ii) Text-anchored Pairwise Similarity Distillation, which transfers both region-text and region-region alignments from a full-precision teacher model via pairwise cosine similarities in the joint embedding space. Experimental results on LVIS and COCO zero-shot benchmarks demonstrate that QATMA significantly outperforms existing QAT baselines under extremely low-bit settings, achieving gains of up to 4.3 and 7.6 AP, respectively.
Figures & tables
Figure 1: Impact of 4-bit quantization on YOLO-World [ 7 ] ( Objects365v2 [ 44 ] ). (a) Confidence scores from the similarity between each region embedding and the “Train” text embedding ( Pˉ : mean over positive regions). (b) Pairwise cosine similarity matrix among positive region embeddings within the same category ( r : Pearson correlation with FP32). (c) Quantitative comparison of embedding distortion relative to FP32; closer to the origin indicates less distortion.
Figure 2: Overview of the proposed QATMA framework. ( Red ) blocks denote quantized and learnable modules, and ( blue ) blocks denote full-precision and frozen modules. (a) Stage 1: the backbone ( M1 ) is quantized with Lfeat supervision from the full-precision teacher, while the neck-head remains frozen for error isolation. (b) Stage 2: the neck-head ( M2 ) is additionally quantized, supervised by both Lfeat and LTPSD . (c) Feature distillation aligns the student’s multi-scale backbone features f1S(x) to those of the teacher f1T(x) . (d) TPSD groups region embeddings by text query tc and constructs a unified pairwise similarity matrix Sc to transfer region-text and region-region alignment.
Model
Bits
Size(MB)
BOPs(T)
Method
AP
AP r
AP c
AP f
YOLO-World-M
FP32
110.88
51.82
-
31.0
23.8
29.2
33.9
4-4-8
15.11
7.10
QAT
13.8
10.7
11.6
16.3
EMA+QC [ 16 ]
13.7
11.8
12.2
15.4
QFD [ 57 ]
13.0
7.1
11.0
15.8
Ours
16.3
11.4
14.5
18.8
YOLO-World-L
FP32
181.4
98.39
-
35.4
27.6
34.1
38.0
Table 1: LVIS minival zero-shot evaluation with 4-4-8 quantization (Ch-T-H granularity). Ch = per-channel, T = per-tensor, H = per-head. Evaluation follows the LVIS fixed AP protocol [ 10 ] .
Figure 4: Qualitative comparison on YOLO-World-L (4-4-8, Ch-T-H). ( Top ) Detection results. ( Bottom ) Region-region similarity heatmap of average pairwise cosine similarity among same-class anchors. QAT distorts both detection and similarity patterns of FP32, whereas QATMA restores them.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Bits
BOPs(T)
Method
AP
AP 50
AP 75
AP s
AP m
AP l
YOLO-World-M
FP32
44.56
-
41.9
57.0
45.5
26.7
46.1
55.0
4-4-8
2.05
QAT
22.0
32.1
23.6
7.8
23.5
32.4
EMA+QC [ 16 ]
21.2
31.1
22.7
8.3
22.7
31.3
QFD [ 57 ]
21.6
31.4
23.3
7.9
23.7
31.8
Ours
26.1
38.1
28.1
11.0
28.1
37.9
YOLO-World-L
FP32
90.60
-
45.1
60.7
48.9
29.8
49.8
57.5
Appendix
Table 7: Full COCO val2017 zero-shot comparison, including the EMA+QC [ 16 ] and QFD [ 57 ] closed-vocabulary detection QAT baselines (4-4-8, Ch-T-H granularity).
Granularity
Bits
Method
LVIS
COCO
Ch-T-H
4-5-8
QAT
23.0
36.2
Ours
23.8
37.6
Ch-Ch-H
3-3-8
QAT
18.5
30.7
Ours
20.0
32.6
Appendix
Table 8: Effect of quantization granularity and bit-width on YOLO-World-L. Ch-Ch-H = per-channel activations. LVIS and COCO zero-shot AP is reported.
Stage 1 (backbone)
Stage 2 (neck-head)
AP
1/3
2/3
16.1
1/2
1/2
15.0
Appendix
Table 9: Effect of the curriculum data split between Stage 1 (backbone) and Stage 2 (neck-head) on YOLO-World-L (4-4-8, Ch-T-H). LVIS zero-shot AP is reported.
λ2
0
3
6
12
18
48
AP
30.4
30.8
31.2
31.2
31.0
30.9
Appendix
Table 10: TPSD loss weight λ2 sensitivity on YOLO-World-L (head-only W4A4). LVIS zero-shot AP is reported.
Method
GFLOPs/iter
GPU-hours
QAT
27.1k
95.2
Ours
35.1k
112.4
Appendix
Table 11: Training cost on YOLO-World-L. QATMA adds no inference cost, and its two-stage training overhead over plain QAT is modest.
Figure 5: Qualitative comparison of LVIS zero-shot detection on YOLO-World-L (4-4-8, Ch-T-H)
With an increasing number of Object Detection (OD) models being deployed on edge devices, Zero-Shot Quantization for OD (ZSQ-OD) aims to quantize these models when access to the original training data is prohibited. Existing research on Zero-Shot Quantization-Aware Training (QAT) for OD synthesizes training sets through noise optimization. However, this approach struggles to maintain performance in low-bit regions. In this paper, we introduce GoodQ (Generative off-the-shelf models for object detector Quantization), a QAT pipeline that utilizes off-the-shelf generative models to construct a training set. We first identify three challenges that arise when introducing a generative model to the ZSQ-OD task: 1) each image contains dense information with multiple instances, 2) the class-wise distribution in the original dataset is imbalanced, and 3) the pseudo-labels assigned to the generated images can potentially act as noisy signals during QAT. GoodQ addresses these challenges by 1) introducing an Information-Dense Prompting strategy to generate multi-instance images, 2) applying Intrinsic Distribution-Aware Selection to match the pretrained class distribution, and 3) employing Teacher-guided Adaptive Noise Reduction to mitigate noise arising from the QAT process. Our framework achieves state-of-the-art performance in low-bit ZSQ (W4A4) and extends quantization to extreme bit-widths (W3A3). Furthermore, we conduct an extensive analysis to uncover the underlying factors contributing to the efficacy of GoodQ.
Hyunho Lee, Kyomin Hwang, Hyeonjin Kim +3
Seoul National University, Seoul, South Korea · LG Electronics, Seoul, South Korea
Open-vocabulary object detection (OVOD) enables models to recognize objects beyond predefined categories, but existing approaches remain limited in practical deployment. On the one hand, multimodal designs often incur substantial computational overhead due to their reliance on text encoders at inference time. On the other hand, tightly coupled training objectives introduce a trade-off between closed-set detection accuracy and open-world generalization. Thus, we propose Decoupled Cognition DETR (DeCo-DETR), a vision-centric framework that addresses these challenges through a unified decoupling paradigm. Instead of depending on online text encoding, DeCo-DETR constructs a hierarchical semantic prototype space from region-level descriptions generated by pre-trained LVLMs and aligned via CLIP, enabling efficient and reusable semantic representation. Building upon this representation, the framework further disentangles semantic reasoning from localization through a decoupled training strategy, which separates alignment and detection into parallel optimization streams. Extensive experiments on standard OVOD benchmarks demonstrate that DeCo-DETR achieves competitive zero-shot detection performance while significantly improving inference efficiency. These results highlight the effectiveness of decoupling semantic cognition from detection, offering a practical direction for scalable OVOD systems.
Personalized object detection aims to adapt a general-purpose detector to recognize user-specific instances from only a few examples. Lightweight models often struggle in this setting due to their weak semantic priors, while large vision-language models (VLMs) offer strong object-level understanding but are too computationally demanding for real-time or on-device applications. We introduce MOCHA (Multi-modal Objects-aware Cross-arcHitecture Alignment), a distillation framework that transfers multimodal region-level knowledge from a frozen VLM teacher into a lightweight vision-only detector. MOCHA extracts fused visual and textual teacher's embeddings and uses them to guide student training through a dual-objective loss that enforces accurate local alignment and global relational consistency across regions. This process enables efficient transfer of semantics without the need for teacher modifications or textual input at inference. MOCHA consistently outperforms prior baselines across four personalized detection benchmarks under strict few-shot regimes, yielding a +10.1 average improvement, with minimal inference cost.
Elena Camuffo, Francesco Barbato, Mete Ozay +2
Samsung R&D Institute UK, United Kingdom · University of Padova, Italy