Organizations: the MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, China · the School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China · the Institute of Microelectronics, University of Chinese Academy of Sciences, Beijing, China
Open-Vocabulary Object Detection (OVOD) has achieved remarkable success in generalizing to novel categories. However, this success often rests on the implicit assumption of domain stationarity. In this work, we revisit the OVOD paradigm and study a key vulnerability: the fragile coupling between visual manifolds and textual embeddings under distribution shifts. We first formulate Domain-Generalized Open-Vocabulary Object Detection (DG-OVOD) as an evaluation protocol for open-vocabulary recognition under visual shifts. Through empirical analysis, we observe that visual shifts can destabilize the latent cross-modal space, causing novel-category visual signals to drift away from their semantic anchors. Motivated by these observations, we propose Progressive Domain-invariant Cross-modal Alignment (PICA). PICA departs from uniform training by introducing a multi-level curriculum based on ambiguity and signal strength. It constructs a quality-adjusted curriculum over pseudo-word prototypes, refined by sample reliability and visual consistency, to encourage more stable cross-domain modality alignment. Our findings suggest that OVOD robustness under domain shifts is closely linked to the stability of the latent cross-modal alignment space. Our work provides a DG-OVOD evaluation protocol and a practical perspective on building more generalizable open-vocabulary systems beyond static laboratory conditions.
Figures & tables
Fig. 1: Cross-modal alignment analysis. (a) Visualization of model attention under clean and frost conditions. Person is a base category; Elephant is a novel category. PICA preserves more stable and object-centric attention patterns across both base and novel categories compared to Baron [ 13 ] . (b) Cross-modal alignment degradation under visual shift is highly sample-dependent: the difficulty of preserving cross-modal alignment under domain shifts varies across different samples.
Fig. 2: Overview of the Progressive Domain-invariant Cross-modal Alignment (PICA). The progressive sampler combines ambiguity proxy h and signal strength proxy q into a quality-adjusted difficulty score, then divides region pairs into easy, medium, and hard tiers. It uses a dynamic sampling ratio α(ρ) , where ρ represents the training iteration, to align selected region features with pseudo-word prototypes in a staged curriculum.
Exp.
R.CLIP [ 9 ]
VL-PLM [ 24 ]
Obj [ 22 ]
CORA [ 19 ]
Baron [ 13 ]
SIA [ 20 ]
RALF [ 25 ]
DQUO [ 21 ]
Baron ∗ [ 13 ]
Baron † [ 13 ]
PICA
Standard
31.4
34.4
36.6
35.1
34.0
35.3
41.3
39.2
33.4
34.4
37.5
Gauss
12.5
14.1
16.1
8.6
17.8
9.1
16.0
9.7
15.8
17.7
17.8
Shot
12.7
14.6
16.6
8.7
17.3
9.3
16.6
9.4
16.1
17.5
17.8
Impulse
10.7
10.3
11.5
5.4
13.6
5.7
11.5
6.6
12.3
14.4
13.8
Defocus
13.2
16.8
19.6
6.8
18.3
7.6
19.6
8.0
16.5
18.9
19.6
Glass
5.0
10.8
12.2
4.6
12.2
4.7
12.2
5.4
10.9
12.1
12.5
TABLE I: Evaluation of Recent OVOD Methods Across Diverse Corruption Types on OV-COCO-C. Each Result Reports Novel-class mAP50domk ( Lk=5 ). mAP50avg Denotes the Mean Across All 15 Corruption Types. “Standard” Corresponds to the Standard Evaluation Protocol on OV-COCO Without Corruption.
Domain
RegionCLIP [ 9 ]
Obj [ 22 ]
Baron [ 13 ]
RALF [ 25 ]
PICA
Cartoon
14.6
5.7
12.1
9.8
13.3
Handmade
13.4
12.5
17.6
13.9
19.9
Painting
15.2
17.2
23.4
19.5
29.5
Sketch
13.2
7.4
8.9
8.5
8.8
Tattoo
16.9
9.7
10.0
14.4
7.2
Weather
25.8
22.7
37.1
28.4
39.3
TABLE II: Comparison of Recent OVOD Methods Under Various OOD Domains on OV-COCO-O. Results Report Novel-class mAP50domk with Lk=1 .
Setting
Benchmark
Baseline
PICA
CLIP ViT-L/14
OV-COCO-C
21.1
22.1
ResNet-50-FPN
Rainy-Cityscapes
40.2
42.4
TABLE III: Generalization to a Stronger Backbone (CLIP ViT-L/14 on OV-COCO-C) and to a Real-World Adverse-Weather Benchmark (Rainy-Cityscapes, COCO-overlapping Categories). All Results Report Novel-class mAP50avg .
SAS
Mixup
h
q
Std.
C-Avg.
O-Avg.
✗
✗
-
-
34.0
19.3
18.5
✗
✓
-
-
34.2
19.6
19.0
✓
✗
✓
✓
36.5
20.2
19.0
✓
✓
✓
✗
34.0
19.8
18.7
✓
✓
✗
✓
34.2
19.6
17.4
✓
✓
✓
✓
37.5
20.7
19.7
TABLE IV: Ablation Study of PICA Components and Curriculum Sampling Strategies on Both OV-COCO-C and OV-COCO-O. SAS Denotes Sample-Adaptive Stratification, and h or q Indicates Which Proxy Guides Curriculum-based Cross-modal Alignment. “Std.” Is the Standard Protocol on OV-COCO; “C-Avg.” and “O-Avg.” Are the mAP50avg on OV-COCO-C and OV-COCO-O, Respectively. All Results Are Reported on Novel Categories. “–” Means the Configuration Is Not Reported in That Setting.
Pseudo-word Features
Standard
mAP 50avg
wr=P(fr)
36.1
20.0
wr=P(f~r)
36.0
20.1
wr,w~r
37.5
20.7
TABLE V: Ablation of Pseudo-word Generation Strategies in PICA Loss on OV-COCO-C. All Results Are Reported on Novel Categories Using the mAP50avg .
Sampler
mAP 50avg
Entropy-only
19.82
Margin-based
20.12
PICA (dual-proxy q & h )
20.7
TABLE VI: Controlled Comparison of PICA’s Dual-proxy Sampler with Single-scalar Uncertainty Samplers on OV-COCO-C. All Results Are Reported on Novel Categories Using the mAP50avg Under the Same Training Protocol.
Fig. 3: Comparative analysis of cross-modal alignment and training stability. (a) Mean AI-gap for PICA and Baseline across OV-COCO-C. Lower AI-gap indicates more stable cross-domain alignment. (b) Gradient direction cosine similarity during training. PICA maintains smoother and higher alignment consistency than Baseline, which is consistent with reduced gradient conflicts and more stable cross-modal learning.
Fig. 4: Region-level cross-modal alignment stability between standard and Gaussian noise. Each point represents a region, plotted by its clean-image signal strength q (x-axis) and ambiguity h (y-axis), colored by confusion increase Δh=hcorrupted−hclean . (a) Baseline exhibits persistent high- Δh dispersion across mid-to-high signal levels. (b) PICA progressively stabilizes mid-to-high signal regions ( Δh→0 ), suggesting that curriculum training selectively consolidates alignment robustness for reliable regions.
Fig. 5: Additional q – h analysis. The signal strength proxy q and the ambiguity proxy h are correlated overall, yet the off-diagonal regions ( e.g. , high- q /high- h and low- q /low- h samples) show that they are not interchangeable. The quality-adjusted difficulty score therefore combines ambiguity-based ordering with a low-signal penalty, with low- q /high- h samples assigned the lowest early-stage priority.
Param
Setting
Std.
mAP 50avg
(rE0,rM0,rH0)
(0.65, 0.30, 0.05)
36.8
20.0
(0.60, 0.30, 0.10)
37.0
20.2
(0.40, 0.30, 0.30)
36.9
20.1
(0.30, 0.30, 0.40)
36.6
19.8
(0.33, 0.33, 0.33)
37.5
20.7
Queue M
8192
36.5
20.0
TABLE VII: Hyperparameter Sensitivity Analysis of PICA on OV-COCO-C. All Results Are Reported on Novel Categories Using the mAP50avg .
Open-domain open-vocabulary detection (ODOVD) requires detectors to generalize to both novel categories and unseen domains, making it more challenging than open-vocabulary detection. Existing methods typically train open-vocabulary detectors together with domain generalization modules from scratch, leading to high training cost. we propose ExDet, a lightweight category-domain collaborative generalization framework for ODOVD that enhances the cross-category and cross-domain generalization of existing detectors. ExDet consists of Text-Guided Extrapolation (TGE), a lightweight Detector-Compatible Rectification (DCR) module, and ExRPN. Specifically, TGE exploits the DeltaSpace property of vision-language models (VLMs) to infer category- and domain-aware proxy visual prototypes from text. DCR is learned from the TGE-generated prototypes in a detector training-free and real-data-free manner, and is inserted after the classification head at inference to rectify representations toward a detector-compatible source-domain visual distribution, thereby enhancing classification for targets from novel categories and unseen domains. ExRPN recalibrates proposal scores by combining semantic similarity with RPN confidence, improving recall for novel and domain-shifted objects while providing better support for subsequent classification and DCR. ExDet achieves SOTA performance on OD-LVIS, OV-LVIS, Objects365, and MSOSB.
Yupeng Zhang, Yuzhong Feng, Ruize Han +3
College of Intelligence and Computing, Tianjin University, Tianjin, China · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen, China · School of Artificial Intelligence, Nanchang University, Jiangxi, China
Open-vocabulary object detection (OVOD) enables models to detect any object category, including unseen ones. Benefiting from large-scale pre-training, existing OVOD methods achieve strong detection performance on general scenarios (e.g., OV-COCO) but suffer severe performance drops when transferred to downstream tasks with substantial domain shifts. This degradation stems from the scarcity and weak semantics of category labels in domain-specific task, as well as the inability of existing models to capture auxiliary semantics beyond coarse-grained category label. To address these issues, we propose HSA-DINO, a parameter-efficient semantic augmentation framework for enhancing open-vocabulary object detection. Specifically, we propose a multi-scale prompt bank that leverages image feature pyramids to capture hierarchical semantics and select domain-specific local semantic prompts, progressively enriching textual representations from coarse to fine-grained levels. Furthermore, we introduce a semantic-aware router that dynamically selects the appropriate semantic augmentation strategy during inference, thereby preventing parameter updates from degrading the generalization ability of the pre-trained OVOD model. We evaluate HSA-DINO on OV-COCO, several vertical domain datasets, and modified benchmark settings. The results show that HSA-DINO performs favorably against previous state-of-the-art methods, achieving a superior trade-off between domain adaptability and open-vocabulary generalization.
Weihao Cao, Runqi Wang, Xiaoyue Duan +3
State Key Laboratory of Advanced Rail Autonomous Operation, Beijing Jiaotong University · Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, Beijing Jiaotong University · School of Computer Science & Technology, Beijing Jiaotong University +1
Out-of-distribution (OOD) detection remains a critical challenge in open-world learning, where models must adapt to evolving data distributions. While recent vision-language models (VLMS) like CLIP enable multimodal OOD detection through Dual-Pattern Matching (DPM), existing methods typically suffer from two major shortcomings: (1) They rely on fixed fusion rules and assume static environments, failing under temporal drift; and (2) they lack robustness against covariate shifted inputs. In this paper, we propose a novel two-step framework to enhance OOD detection and covariate distribution shift robustness in dynamic settings. We extend the dual-pattern regime into Temporal Quadruple-Pattern Matching (T-QPM). First, by pairing OOD images with text descriptions, we introduce cross-modal consistency patterns between ID and OOD signals, refining the decision boundary through joint image-text reasoning. Second, we address temporal distribution shifts by learning lightweight fusion weights to optimally combine semantic matching and visual typicality. To ensure stability, we enforce explicit regularization based on Average Thresholded Confidence (ATC), preventing performance degradation as distributions evolve. Experiments on temporally partitioned benchmarks demonstrate that our approach significantly outperforms static baselines, offering a robust, temporally-consistent framework for multimodal OOD detection in non-stationary environments.