Mind the Way You Select Negative Texts: Pursuing the Distance Consistency in OOD Detection with VLMs

Zhikang Xu; Qianqian Xu; Zitai Wang; Cong Hua; Sicong Li; Zhiyong Yang; Qingming Huang

cs.CVMar 3, 2026

Mind the Way You Select Negative Texts: Pursuing the Distance Consistency in OOD Detection with VLMs

Authors: Zhikang Xu, Qianqian Xu, Zitai Wang, Cong Hua, Sicong Li, Zhiyong Yang, Qingming Huang

Organizations: Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences · Beijing Academy of Artificial Intelligence ·

Abstract

Out-of-distribution (OOD) detection seeks to identify samples from unknown classes, a critical capability for deploying machine learning models in open-world scenarios. Recent research has demonstrated that Vision-Language Models (VLMs) can effectively leverage their multi-modal representations for OOD detection. However, current methods often incorporate intra-modal distance during OOD detection, such as comparing negative texts with ID labels or comparing test images with image proxies. This design paradigm creates an inherent inconsistency against the inter-modal distance that CLIP-like VLMs are optimized for, potentially leading to suboptimal performance. To address this limitation, we propose InterNeg, a simple yet effective framework that systematically utilizes consistent inter-modal distance enhancement from textual and visual perspectives. From the textual perspective, we devise an inter-modal criterion for selecting negative texts. From the visual perspective, we dynamically identify high-confidence OOD images and invert them into the textual space, generating extra negative text embeddings guided by inter-modal distance. Extensive experiments across multiple benchmarks demonstrate the superiority of our approach. Notably, our InterNeg achieves state-of-the-art performance compared to existing works, with a 3.47% reduction in FPR95 on the large-scale ImageNet benchmark and a 5.50% improvement in AUROC on the challenging Near-OOD benchmark.

Figures & tables

Figure 1: Comparison of Baseline and InterNeg. The baseline often incorporates intra-modal distance during OOD detection, which is inconsistent with the inter-modal distance that CLIP-like VLMs are optimized for. In contrast, InterNeg leverages consistent inter-modal distance during OOD detection, enhancing performance by inter-modal guided negative texts and extra negative text embeddings generated through modality inversion.

Figure 2: Two types of ID misclassification. First Row: Max-OOD dominant ID misclassification. Second Row: Sum-OOD dominant ID misclassification. Left: Original ID image from ImageNet-1K with its class label and filename. Middle: Top-5 softmax scores for ID labels and negative texts of baseline and our method. Right: Max-OOD/Sum-OOD dominant ID error rates under different thresholds $\gamma$ of baseline and our method.

Methods	FPR95 $\downarrow$		AUROC $\uparrow$
Methods	Near-OOD	Far-OOD	Near-OOD	Far-OOD
Methods requiring training on ID or extra data
GEN [ 38 ]	–	–	78.97	90.98
ReAct [ 51 ]	–	–	79.94	93.70
RMDS [ 48 ]	–	–	80.09	92.60
SCALE [ 65 ]	–	–	81.36	96.53

Table 2: OOD detection results with ID dataset of ImageNet-1k and OpenOOD benchmark using CLIP ViT-B/16 architecture. Full results are available in Table 4 .

Panel of Figure 3 — Figure 3: AUROC $\uparrow$ and FPR95 $\downarrow$ average performance under varying ID:OOD ratios.

Panel of Figure 4 — Figure 4: Parameter sensitivity analysis of four key hyperparameters: the number of ID images per class $N$ , the number of selected negative texts $M$ , the maximum size of extra negative text embeddings set $K$ , and the high-confidence OOD threshold $\beta$ , evaluated on both Four-OOD and Near-OOD benchmarks using ImageNet-1K as the ID dataset.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Near / Far OOD	Datasets	FPR95 $\downarrow$	AUROC $\uparrow$
Near-OOD	SSB-hard [ 55 ]	69.96	80.24
	NINCO [ 3 ]	60.90	84.16
	Mean	65.43	82.20
Far-OOD	iNaturalist [ 24 ]	0.40	99.71
	Textures [ 7 ]	18.17	96.74
	OpenImage-O [ 56 ]	32.30	93.69

Appendix

Table 4: Full results of our method with ID dataset of ImageNet-1K on the OpenOOD benchmark.

Methods	FPR95 $\downarrow$		AUROC $\uparrow$
Methods	Near-OOD	Far-OOD	Near-OOD	Far-OOD
Methods requiring training on ID or extra data
GEN [ 38 ]	–	–	81.31	79.68
VOS [ 13 ] + EBO [ 37 ]	–	–	80.93	81.32
SCALE [ 65 ]	–	–	80.99	81.42
OE [ 21 ] + MSP [ 20 ]	–	–	88.30	81.41

Appendix

Table 5: OOD detection results with ID dataset of CIFAR100 on the OpenOOD benchmark using CLIP ViT-B/16 architecture. Full results are available in Table 6 .

Near / Far OOD	Datasets	FPR95 $\downarrow$	AUROC $\uparrow$
Near-OOD	CIFAR10 [ 30 ]	60.10	84.19
	TIN [ 31 ]	64.99	86.71
	Mean	62.54	85.45
Far-OOD	MNIST [ 9 ]	0.01	99.97
	SVHN [ 44 ]	3.23	99.41
	Texture [ 7 ]	21.16	96.84

Appendix

Table 6: Full results of our method with ID dataset of CIFAR100 on the OpenOOD benchmark.

Methods	FPR95 $\downarrow$		AUROC $\uparrow$
Methods	Near-OOD	Far-OOD	Near-OOD	Far-OOD
Methods requiring training on ID or extra data
PixMix [ 23 ] + KNN [ 52 ]	–	–	93.10	95.94
OE [ 21 ] + MSP [ 20 ]	–	–	94.82	96.00
PixMix [ 23 ] + RotPred [ 22 ]	–	–	94.86	98.18
Zero-shot methods (no training on ID or extra data)

Appendix

Table 7: OOD detection results with ID dataset of CIFAR10 on the OpenOOD benchmark using CLIP ViT-B/16 architecture. Full results are available in Table 8 .

Near / Far OOD	Datasets	FPR95 $\downarrow$	AUROC $\uparrow$
Near-OOD	CIFAR100 [ 30 ]	37.26	91.81
	TIN [ 31 ]	10.60	98.45
	Mean	23.93	95.13
Far-OOD	MNIST [ 9 ]	0.00	100.00
	SVHN [ 44 ]	0.06	99.97
	Texture [ 7 ]	0.41	99.57

Appendix

Table 8: Full results of our method with ID dataset of CIFAR10 on the OpenOOD benchmark.

Benchmark	Seed					Mean	Std
Benchmark	0	1	2	3	4	Mean	Std
Four-OOD	97.43	97.38	97.38	97.45	97.51	97.43	0.05
Near-OOD	82.20	81.56	81.50	82.29	82.59	82.03	0.46

Appendix

Table 9: Average results of AUROC (

\uparrow

) with five different seeds on both Four-OOD and Near-OOD benchmarks.

ID Image Proxies Source	Method	AUROC $\uparrow$	FPR95 $\downarrow$
ImageNet	NegLabel	93.08	29.77
	AdaNeg	96.24	19.21
	InterNeg	96.84	18.40
ImageNet-V2	NegLabel	93.90	27.64
	AdaNeg	96.19	19.97
	InterNeg	96.96	18.18

Appendix

Table 11: Cross-domain OOD detection performance on ImageNet-V2 with Four-OOD datasets.

Panel of Figure 5 — Figure 5: Max-OOD and Sum-OOD ID error rates on different OOD datasets. Left: iNaturalist. Middle: Places. Right: Textures.

Method	Mean	iNaturalist	Places	Textures	SUN
AdaNeg	0.0058	0.0068	0.0046	0.0093	0.0025
InterNeg	0.0067	0.0074	0.0055	0.0113	0.0029

Appendix

Table 12: Comparison of inference time (seconds per image).

Source	Method	Average
Source	Method	AUROC $\uparrow$	FPR95 $\downarrow$
Common-20K	NegLabel	90.50	43.02
	CSP	92.06	36.56
	AdaNeg	93.12	32.39
	InterNeg	94.58	28.77
Part-of-Speech	NegLabel	92.71	32.12

Appendix

Table 13: Evaluation with different corpus sources on the traditional Four-OOD benchmark, using ImageNet-1K as the ID dataset.

OOD Detection

Out-of-distribution (OOD) detection identifies inputs unlike a model’s training data so systems can flag uncertainty, reject inputs, or adapt safely.

See all 222 papers

NegAS: Negative Label Guided Attention and Scoring for Out-of-Distribution Object Detection with Vision-Language Models
Jun 21, 2026Yingjie Zhang, Shuai Li, Peng WangVLM Robustness Object Detection
Northwestern Polytechnical University, Xi’An, China
Respecting Modality Gap in Post-hoc Out-of-distribution Detection with Pre-trained Vision-Language Models
May 26, 2026Yuanwei Hu, Bo Peng, Yadan Luo +3Vision-Language Models Test-Time Adaptation
University of Technology Sydney · The University of Queensland
OODBench: Out-of-Distribution Benchmark for Large Vision-Language Models
Feb 20, 2026Ling Lin, Yang Bai, Heng Su +5VLM Evaluation VLM Robustness
School of Artificial Intelligence and Data Science, University of Science and Technology of China, Hefei 230000, China · Suzhou Institute for Advanced Research, USTC, Suzhou 215123, China · Institute of High Performance Computing (IHPC), A*STAR, Singapore +1

Jun 21, 2026J/K move · Enter open · S save

NegAS: Negative Label Guided Attention and Scoring for Out-of-Distribution Object Detection with Vision-Language Models

Yingjie Zhang, Shuai Li, Peng Wang

Northwestern Polytechnical University, Xi’An, China

VLM Robustness Object Detection OOD Detection

Open paper AskPDF arXiv

Out-of-Distribution (OOD) detection is essential for ensuring the robustness and reliability of object detection systems deployed in safety-critical applications. While prior research has mainly focused on uni-modal detectors or vision-language model (VLM) based classifiers, the potential of VLM-based object detectors in OOD scenarios remains underexplored. In this work, we take the first step toward building OOD object detection methods upon VLMs. We identify two challenges specific to VLM detectors: (i) their text-guided attention enhances foreground with ID labels but treats background uniformly, leaving potential OOD regions unexploited for separating in-distribution (ID) from OOD instances; and (ii) their sigmoid-based multi-label outputs are incompatible with softmax-based OOD scores, calling for scoring functions consistent with VLM probabilistic outputs. Hence, we introduce Negative Label Guided Attention and Scoring (NegAS). To address (i), we propose a negative label guided attention module (NegA), where LLM-generated, visually-similar but semantically-different negative labels are used to guide attention toward potential OOD background regions. To address (ii), we introduce a novel sigmoid-based OOD scoring function (NegS) that leverages both ID and negative labels, producing strong responses for ID instances and suppressed responses for OOD ones. Extensive experiments demonstrate that our approach improves OOD detection performance by a large margin while maintaining ID accuracy, e.g., reducing the FPR95 by 11.4% on the COCO dataset and 25.5% on the OpenImages dataset compared to the baseline model. While initially designed for dense VLM detectors like YOLO-World, we successfully adapt NegAS to Grounding DINO, a query-based VLM transformer and achieve significant improvements, demonstrating the generalizability of our framework.

Abstract

OOD Detection

Explore similar work

NegAS: Negative Label Guided Attention and Scoring for Out-of-Distribution Object Detection with Vision-Language Models

Respecting Modality Gap in Post-hoc Out-of-distribution Detection with Pre-trained Vision-Language Models

OODBench: Out-of-Distribution Benchmark for Large Vision-Language Models