OODBench: Out-of-Distribution Benchmark for Large Vision-Language Models

Ling Lin; Yang Bai; Heng Su; Congcong Zhu; Yaoxing Wang; Yang Zhou; Huazhu Fu; Jingrun Chen

cs.CVFeb 20, 2026

OODBench: Out-of-Distribution Benchmark for Large Vision-Language Models

Authors: Ling Lin, Yang Bai, Heng Su, Congcong Zhu, Yaoxing Wang, Yang Zhou, Huazhu Fu, Jingrun Chen

Organizations: School of Artificial Intelligence and Data Science, University of Science and Technology of China, Hefei 230000, China · Suzhou Institute for Advanced Research, USTC, Suzhou 215123, China · Institute of High Performance Computing (IHPC), A*STAR, Singapore · Unmanned System Research Institute, Northwestern Polytechnical University, Xi’an 71002, China

Abstract

Existing Visual-Language Models (VLMs) have achieved significant progress by being trained on massive-scale datasets, typically under the assumption that data are independent and identically distributed (IID). However, in real-world scenarios, it is often impractical to expect that all data processed by an AI system satisfy this assumption. Furthermore, failure to appropriately handle out-of-distribution (OOD) objects may introduce safety risks in real-world applications (e.g., autonomous driving or medical assistance). Unfortunately, current research has not yet provided valid benchmarks that can comprehensively assess the performance of VLMs in response to OOD data. Therefore, we propose OODBench, a predominantly automated method with minimal human verification, for constructing new benchmarks and evaluating the ability of VLMs to process OOD data. OODBench contains 40K instance-level OOD instance-category pairs, and we show that current VLMs still exhibit notable performance degradation on OODBench, even when the underlying image categories are common. In addition, we propose a reliable automated assessment metric that employs a Basic-to-Advanced Progression of prompted questions to assess the impact of OOD data on questions of varying difficulty more fully. Lastly, we summarize substantial findings and insights to facilitate future research in the acquisition and evaluation of OOD data.

Figures & tables

Fig. 1: Distribution of categories and fields in OODBench .

Fig. 2: Overview of OODBench. Given an image and a set of candidate semantics, we approximate the unobservable semantic information (PMI) with contrastive scores via InfoNCE density ratio estimation. The resulting score is a linear transformation of PMI (up to an image-dependent bias), providing a valid, monotonic proxy. Based on this proxy, we extract two complementary OOD signal – score deviation and ranking anomaly, and apply multi-model cross-validation and human verification to construct a high-quality semantic OOD dataset.

Fig. 3: Basic-to-Advanced Progression Metric Example. Basic-to-Advanced Progression Metric covers three problems: existential, counting, and logical reasoning problems from top to bottom.

Fig. 4: The OODBench example has two questions and one instance, with the answers alternated to avoid the model being over-scored due to biased output distribution.

Model	Image Encoder	Language Model	Data Type	OODBench Performance
Model	Image Encoder	Language Model		Accuracy( $\%$ )	F1( $\%$ )	Precision( $\%$ )	Recall( $\%$ )	MCC( $\%$ )
Random Chance	-	-	ID/OOD-S/OOD-H	50.00	50.00	50.00	50.00	0.00
		Open-source Models
			ID	84.79	84.72	85.15	84.28	69.59
			OOD-S	64.54	56.16	73.52	45.44	31.46
			OOD-H	63.22	52.99	73.40	41.46	29.36

TABLE I: Performance on OODBench. We report the performance of the 10 leading VLMs on OODBench. All models perform significantly lower on OOD-H than on ID. The performance of the latest models, such as LLaVA-NeXT, Llama3.2-Vision, InternVL2.5, and Qwen2-VL, lags behind the ID data by about

20\%

30\%

on OOD-H data. Even the best closed-source model, GPT-4o, still performs

26\%

lower on OOD-H than ID.

Model	Data Type	OODBench BAP Performance
Model	Data Type	E-Acc( $\%$ )	C-Acc( $\%$ )	L-Acc( $\%$ )
	ID	81.39	38.25	21.70
	OOD-S	66.61	33.75	21.48
LLaVA-NeXT-8B	OOD-H	44.22	21.71	13.45
	ID	84.66	38.67	24.40
	OOD-S	69.73	26.72	17.50

TABLE II: Performance of 10 leading VLMs on OODBench for Basic-to-Advanced Progress.

Model	Scoring Models	COCO-OOD Performance
Model	Scoring Models	Num	Accuracy( $\%$ )	F1( $\%$ )	Precision( $\%$ )	Recall( $\%$ )	MCC( $\%$ )	Time
Random Chance	-	-	$\text{50.00}\%$	$\text{50.00}\%$	$\text{50.00}\%$	$\text{50.00}\%$	$\text{0.00}\%$	-
Open-source Models
		10160	78.74	76.78	84.58	70.30	58.32	00:11:44 ${}^{\text{{{2}}}}$
	CLIP & BLIP2	$\Delta$	+ 0.00	+ 0.00	+ 0.00	+ 0.00	+ 0.00	-
		9988	79.85	78.06	85.65	71.71	60.50	00:21:40 ${}^{\text{{{1}}}}$

TABLE III: Impact of replacing scoring models on the downstream performance of different multimodal models. We compare the accuracy differences (

\Delta

) of data divided by different combinations of scoring models—Group I , Group II , and Group III —across four representative VLMs (Qwen2-VL-7B-Instruct, InternVL2-8B, Gemini, and GPT-4o). Group I is used as the baseline, and the performance drop or gain of Group II and Group III is reported accordingly. The results show that although different scoring models induce slight variance in OOD data division, their impact on downstream model performance is minimal, supporting the replaceability of scoring models in the OODBench framework.

Fig. 5: Impact of the number of scoring models on the quality of OOD data division under the cross-validation mechanism. The horizontal axis indicates the number of scoring models involved in the division process (ranging from 1 to 4), while the vertical axis shows the corresponding Accuracy and F1 score (%) obtained by evaluating the divided OOD data on multiple mainstream VLMs. The results demonstrate that using a single scoring model yields the highest performance but suffers from potential bias. Incorporating two scoring models significantly reduces this bias, producing OOD data more aligned with the VLMs’ potential OOD distribution. However, increasing the number of scoring models beyond two offers limited benefits and may introduce instability, as evidenced by performance rebound and eventual convergence.

Threshold T	Data Type	nuScenes Performance
Threshold T	Data Type	Num	Accuracy	F1	Precision	Recall	MCC
	OOD-S	25798	66.30	63.34	69.43	58.24	33.02
0.05	OOD-H	9882	58.86	60.28	58.28	62.42	17.77
	OOD-S	23546	62.65	55.69	68.44	46.95	26.64
0.01	OOD-H	5934	60.79	59.08	61.76	56.62	21.65
	OOD-S	23644	60.93	53.31	66.24	44.60	23.13

TABLE IV: An ablation study of hyperparameter

T

using the LLama-3.2-Vision [ 37 ] .

Ling Lin is currently pursuing the Ph.D. degree with the School of Artificial Intelligence and Data Science, University of Science and Technology of China (USTC). His research interests include multimodal large language models, out-of-distribution research, multimodal reasoning, and 3D scene understanding. He has published papers in several leading journals and conferences such as IEEE Transactions on Image Processing, IEEE Transactions on Neural Networks and learning systems, IEEE Transactions on Circuits and Systems for Video Technology, ICASSP, and ICME.

Table 10

Yang Bai received the M.S. degree in computer science from Newcastle University, U.K., in 2018, and the Ph.D. degree in computer science from Durham University, U.K., in 2023. He is currently a Scientist at the Institute of High Performance Computing (IHPC), A*STAR, Singapore. His research interests include multimodal foundation models, vision–language learning, and medical artificial intelligence, with a focus on medical image understanding and clinical multimodal reasoning.

Table 11

Heng Su is currently pursuing a Master’s degree at the School of Artificial Intelligence and Data Science, University of Science and Technology of China (USTC). He received his Bachelor’s degree from China University of Petroleum (East China) (UPC). His research interests include Multimodal Large Language Models (MLLMs), visual faithfulness, and spatial hallucination.

Table 12

Yang Long (Senior Member, IEEE) is an Associate Professor in the Department of Computer Science, Durham University, U.K. He is also an MRC Innovation Fellow aiming to design scalable AI solutions for large-scale healthcare applications. His research background is in the interdisciplinary field of computer vision and machine learning. He is passionate about unveiling the black-box of AI and transferring the knowledge to seek scalable, interactable, interpretable, and sustainable solutions for other disciplinary researches, e.g., physical activity, mental health, design, education, security, and geoengineering. He has authored/coauthored 30+ top-tier papers in refereed journals/conferences such as IEEE TPAMI, TIP, CVPR, AAAI, and ACM MM, and holds a patent and a Chinese National Grant.

Table 13

Congcong Zhu (Member, IEEE) received the Ph.D. degree from the School of Computer Engineering and Science, Shanghai University, Shanghai, China, in 2022. He is currently an Associate Researcher at the Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, China. His research interests include embodied artificial intelligence, multimodal content generation, and multimodal understanding. He has published papers in several journals, including IEEE Transactions on Image Processing, IEEE Transactions on Multimedia, IEEE Transactions on Circuits and Systems for Video Technology, Pattern Recognition, and IEEE Transactions on Instrumentation and Measurement, as well as in leading conferences, including ICML, CVPR, ICCV, AAAI, and ACM MM. He was nominated for the Shanghai Computer Society’s Outstanding Doctoral Dissertation Award, received the Outstanding Graduate Student Award of Shanghai, and was recognized as an Outstanding Postdoctoral Fellow in Jiangsu Province, China.

Table 14

Yaoxing Wang received the master’s degree in computer technology from Ningxia University, Yinchuan, China, in 2022, where he is currently pursuing the Ph.D. degree in unmanned systems science and technology with the Unmanned System Research Institute, Northwestern Polytechnical University. His research interests include computer vision, 3D reconstruction, and thermography.

Table 15

Yang Zhou is a Senior Scientist at the Institute of High Performance Computing (IHPC), A*STAR, Singapore. He earned his Ph.D. in Computer Science from Hong Kong Baptist University in 2019. From 2020 to 2022, he served as a Research Fellow at the School of Computing, National University of Singapore. His research interests span medical foundation models, vision-language modeling, probabilistic modeling, tensor analysis, natural language processing, and machine learning.

Table 16

Ling Shao (Fellow, IEEE) is a Distinguished Professor with the UCAS-Terminus AI Lab, University of Chinese Academy of Sciences, Beijing, China. He was the founder of the Inception Institute of Artificial Intelligence (IIAI) and the Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE. His research interests include generative AI, vision and language, and AI for healthcare. He is a fellow of the IEEE, the IAPR, the BCS and the IET.

Table 17

Huazhu Fu (Senior Member, IEEE) is a Principal Scientist at the Institute of High Performance Computing (IHPC), A*STAR, Singapore. He earned his Ph.D. from Tianjin University in 2013. His research focuses on medical image analysis, AI for healthcare, and trustworthy AI. He is an Associate Editor for several distinguished journals, including IEEE Transactions on Medical Imaging, IEEE Transactions on Neural Networks and Learning Systems, and IEEE Journal of Biomedical and Health Informatics.

Table 18

Jingrun Chen received the B.S. degree from Nanjing University, China, in 2005, and the PhD degree from the Chinese Academy of Sciences, China, in 2010. From 2010 to 2015, he served as a Visiting Assistant Professor at the University of California, Santa Barbara, and from 2015 to 2021, he was a Distinguished Professor at Soochow University. He is currently a Professor at the Suzhou Institute for Advanced Research/School of Mathematical Sciences at the University of Science and Technology of China. His main work has been published in academic journals on applied and computational mathematics such as the SIAM series, Mathematics of Computation, and Journal of Computational Physics, as well as in interdisciplinary academic journals like Nature Communications and Materials Horizons.

Table 19

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 6: Distribution of categories and fields in initial benchmark.

Original Labels	Modified Labels	Remark
human.pedestrian.adult	Adult Pedestrian
human.pedestrian.child	Child Pedestrian
human.pedestrian.wheelchair	Pedestrian in Wheelchair
human.pedestrian.stroller	Pedestrian with Stroller
human.pedestrian.personal_mobility	Pedestrian using Personal Mobility Device
human.pedestrian.police_officer	Police Officer

Appendix

TABLE V: Original and modified labels for nuScenes.

Original Labels	Modified Labels	Remark
road	road
sidewalk	sidewalk
parking	parking
rail track	rail track
person	person
rider	rider

Appendix

TABLE VI: Original and modified labels for Cityscapes.

Level	Name	Weights	Architecture	Training Code	Training Data	Reproducibility
0	Non-Open Source	✗	✗	✗	✗	✗
1A	Architecture Open Source	✗	✓	Partial	✗	✗
1B	Weights Open but Architecture Not Fully Open	✓	Partial	Partial	✗	✗
1C	Data Partially Open	Optional	Optional	Optional	Partial	✗
2	Weights Open	✓	✓	✓	✗ / Rough	✓ (Partial)
3	Strongly Open (Reproducible)	✓	✓	✓	✓ (Clean)	✓ (Partial)

Appendix

TABLE VII: Current open-source MLLM models are categorized into four primary tiers based on the scope and granularity of publicly disclosed information. Each level exhibits progressively enhanced transparency and reproducibility across model weights, architectural details, training code, training data, and overall reproducibility.

No.	Family	Years	Models	Institution	COCO Used	Openness Level	Included in Analysis
1	Flamingo [ 44 ]	2022	Flamingo-3B, Flamingo-9B	DeepMind	✗	L3	✗
2	Qwen-VL [ 45 ]	2023	Qwen-VL(7B), Qwen-VL-Chat(7B), Qwen-VL-Plus(7B), Qwen-VL-Max(7B)	Alibaba Group	✓	L3	✓
3	PaLI-X [ 46 ]	2023	PaLI-X(55B)	Google Research	✗	L1C	✗
4	LLaVA-v1 [ 47 ]	2023	LLaVA-v1-Vicuna(13B), LLaVA-v1-LLama(13B), LLaVA-v1-LLama(7B)	UW-Madison	✓	L3	✓
5	InstructBLIP [ 48 ]	2023	InstructBLIP (FlanT5XL)(3B), InstructBLIP (FlanT5XXL)(11B), InstructBLIP (Vicuna-7B), InstructBLIP (Vicuna-13B)	Salesforce Research	✓	L3	✓
6	MiniGPT-4 [ 49 ]	2023	MiniGPT-4 (Vicuna 13B), MiniGPT-4 (Vicuna 7B), MiniGPT-4 (LLaMA-2 Chat 7B),	KAUST	✗	L3	✗

Appendix

TABLE VIII: Complete summary of collected models. Includes models ultimately included in the analysis as well as those excluded during the screening process.

Panel of Figure 25 — Fig. 7: This figure illustrates the distribution of degradation amounts under the null hypothesis $H_{0}$ (no performance difference between ID and OOD) and the positions of the observed values. As illustrated, for both closed-source models and across all metrics, the observed degradation falls in the extreme tail of the permutation distribution. The permutation p-values are all below $10^{-3}$ , and out of 2000 permutations, none produced a degradation larger than the observed value.

Panel of Figure 26 — Fig. 8: We selected InternVL2-8B and InternVL2.5-8B as open-source reference models and constructed their behavioral signatures based on bootstrap estimates of performance degradation. We then applied the TOST equivalence testing framework to assess the behavioral consistency of closed-source models (GPT-4o and Gemini) under semantic OOD conditions. The bootstrap distributions of $\delta^{(k)}$ for the five evaluation metrics (Accuracy, F1, Precision, Recall, MCC), along with their $90\%$ confidence intervals (CI90) and the corresponding tolerance thresholds $\eta_{k}$ .

Model	Data Type	COCO-OOD-Hard Sample Performance
Model	Data Type	Num	Accuracy( $\%$ )	F1( $\%$ )	Precision( $\%$ )	Recall( $\%$ )	MCC( $\%$
Random Chance	-	-	$\text{50.00}\%$	$\text{50.00}\%$	$\text{50.00}\%$	$\text{50.00}\%$	$\text{0.00}\%$
Open-source Models
	ID	6000	89.90	90.17	87.83	92.63	79.92
	OOD-H	6000	74.45	69.53	86.12	58.30	51.67
LLaVA-NeXT-8B	Hard Sample	5282	56.74	30.27	77.99	18.78	20.71

Appendix

TABLE IX: This table compares the performance of ID, OOD-H data, and in-Hard samples—segmented based on the COCO dataset—across nine leading SOTA MLLMs. We ensured that all testing configurations remained identical except for the data division method, guaranteeing that any observed differences stem solely from the inherent characteristics of the data.

Fig. 9: The figure displays the histogram of accuracy difference distributions between ID data and Hard samples, as well as OOD-H data on the COCO dataset. The bars are sorted in ascending order based on the Hard sample difference, with an exponential trendline fitted to characterize the overall trend.

Fig. 10: The figure displays the empirical distribution results for variance (left), Pearson correlation coefficient (center), and Spearman correlation coefficient (right). The x-axis of the variance subplot represents variance values, while the x-axes of the Pearson and Spearman subplots respectively denote the corresponding correlation coefficients. The y-axes of all three subplots indicate sample sizes. The red dashed lines in each subplot mark the statistical values of the out-of-distribution (OOD) data.

Model	Image Encoder	Language Model	Data Type	COCO-ID-OOD Performance
Model	Image Encoder	Language Model		Num	Accuracy( $\%$ )	F1( $\%$ )	Precision( $\%$ )	Recall( $\%$ )	MCC( $\%$ )	Time
Random Chance	-	-	ID/OOD-S/OOD-H	-	$\text{50.00}\%$	$\text{50.00}\%$	$\text{50.00}\%$	$\text{50.00}\%$	$\text{0.00}\%$	-
		Open-source Models
			ID	6000	89.90	90.17	87.83	92.63	79.92	00:24:24 ${}^{\text{{{2}}}}$
			OOD-S	6000	74.32	70.26	83.45	60.67	50.55	00:27:16 ${}^{\text{{{2}}}}$
			OOD-H	6000	74.45	69.53	86.12	58.30	51.67	00:23:49 ${}^{\text{{{2}}}}$

Appendix

TABLE X: Performance on COCO-ID-OOD. We report the performance of the 10 leading VLMs on OODBench. All models perform significantly lower on OOD-H than on ID.

Model	Image Encoder	Language Model	Data Type	LVIS-ID-OOD Performance
Model	Image Encoder	Language Model		Num	Accuracy( $\%$ )	F1( $\%$ )	Precision( $\%$ )	Recall( $\%$ )	MCC( $\%$ )	Time
Random Chance	-	-	ID/OOD-S/OOD-H	-	$\text{50.00}\%$	$\text{50.00}\%$	$\text{50.00}\%$	$\text{50.00}\%$	$\text{0.00}\%$	-
		Open-source Models
			ID	6000	85.17	85.06	85.67	84.47	70.34	00:24:54 ${}^{\text{{{2}}}}$
			OOD-S	6000	47.90	35.20	46.55	28.30	-4.57	00:29:54 ${}^{\text{{{2}}}}$
			OOD-H	2436	40.23	30.86	36.60	26.68	-20.30	00:10:45 ${}^{\text{{{2}}}}$

Appendix

TABLE XI: Performance on LVIS-ID-OOD. We report the performance of the 10 leading VLMs on OODBench. All models perform significantly lower on OOD-H than on ID.

Model	Image Encoder	Language Model	Data Type	nuScenes-ID-OOD Performance
Model	Image Encoder	Language Model		Num	Accuracy( $\%$ )	F1( $\%$ )	Precision( $\%$ )	Recall( $\%$ )	MCC( $\%$ )	Time
Random Chance	-	-	ID/OOD-S/OOD-H	-	$\text{50.00}\%$	$\text{50.00}\%$	$\text{50.00}\%$	$\text{50.00}\%$	$\text{0.00}\%$	-
		Open-source Models
			ID	6000	81.43	80.05	86.49	74.50	63.48	00:27:02 ${}^{\text{{{2}}}}$
			OOD-S	6000	62.68	45.38	84.62	31.00	32.79	00:31:30 ${}^{\text{{{2}}}}$
			OOD-H	6000	59.03	40.34	74.20	27.70	23.18	00:30:31 ${}^{\text{{{2}}}}$

Appendix

TABLE XII: Performance on nuScenes-ID-OOD. We report the performance of the 10 leading VLMs on OODBench. All models perform significantly lower on OOD-H than on ID.

Model	Image Encoder	Language Model	Data Type	Cityscapes-ID-OOD Performance
Model	Image Encoder	Language Model		Num	Accuracy( $\%$ )	F1( $\%$ )	Precision( $\%$ )	Recall( $\%$ )	MCC( $\%$ )	Time
Random Chance	-	-	ID/OOD-S/OOD-H	-	$\text{50.00}\%$	$\text{50.00}\%$	$\text{50.00}\%$	$\text{50.00}\%$	$\text{0.00}\%$	-
		Open-source Models
			ID	3556	81.21	82.14	78.29	86.39	62.77	00:18:39 ${}^{\text{{{2}}}}$
			OOD-S	3820	78.22	76.55	82.91	71.10	57.02	00:17:43 ${}^{\text{{{2}}}}$
			OOD-H	3836	66.79	58.09	78.70	46.04	36.90	00:18:58 ${}^{\text{{{2}}}}$

Appendix

TABLE XIII: Performance on Cityscapes-ID-OOD. We report the performance of the 10 leading VLMs on OODBench. All models perform significantly lower on OOD-H than on ID.

Model	Data Type	COCO BAP Performance
Model	Data Type	E-Acc( $\%$ )	C-Acc( $\%$ )	L-Acc( $\%$ )	Time
	ID	88.04	61.72	37.08	00:03:48 ${}^{\text{{{2}}}}$
	OOD-S	79.98	47.40	33.23	00:10:14 ${}^{\text{{{2}}}}$
LLaVA-NeXT-8B	OOD-H	55.85	33.83	21.10	00:09:47 ${}^{\text{{{2}}}}$
	ID	91.39	58.61	44.26	00:02:58 ${}^{\text{{{2}}}}$
	OOD-S	80.30	36.26	27.81	00:06:47 ${}^{\text{{{2}}}}$

Appendix

TABLE XIV: Experimental performance of 10 leading VLMs on COCO for Basic-to-Advanced Progress.

Model	Data Type	LVIS BAP Performance
Model	Data Type	E-Acc( $\%$ )	C-Acc( $\%$ )	L-Acc( $\%$ )	Time
	ID	81.27	52.07	29.84	00:07:14 ${}^{\text{{{2}}}}$
	OOD-S	49.04	35.01	17.39	00:10:15 ${}^{\text{{{2}}}}$
LLaVA-NeXT-8B	OOD-H	31.25	27.73	14.06	00:02:45 ${}^{\text{{{2}}}}$
	ID	90.44	55.04	39.79	00:05:38 ${}^{\text{{{2}}}}$
	OOD-S	53.36	22.66	16.19	00:06:24 ${}^{\text{{{2}}}}$

Appendix

TABLE XV: Experimental performance of 10 leading VLMs on LVIS for Basic-to-Advanced Progress.

Model	Data Type	nuScenes BAP Performance
Model	Data Type	E-Acc( $\%$ )	C-Acc( $\%$ )	L-Acc( $\%$ )	Time
	ID	68.31	21.38	15.54	00:08:03 ${}^{\text{{{2}}}}$
	OOD-S	65.15	40.88	28.65	00:07:10 ${}^{\text{{{2}}}}$
LLaVA-NeXT-8B	OOD-H	35.10	20.65	13.27	00:08:46 ${}^{\text{{{2}}}}$
	ID	68.46	30.77	13.08	00:05:05 ${}^{\text{{{2}}}}$
	OOD-S	59.85	34.67	19.89	00:04:29 ${}^{\text{{{2}}}}$

Appendix

TABLE XVI: Experimental performance of 10 leading VLMs on nuScenes for Basic-to-Advanced Progress.

Model	Data Type	Cityscapes BAP Performance
Model	Data Type	E-Acc( $\%$ )	C-Acc( $\%$ )	L-Acc( $\%$ )	Time
	ID	87.37	28.44	12.73	00:11:48 ${}^{\text{{{2}}}}$
	OOD-S	69.53	17.17	10.75	00:13:41 ${}^{\text{{{2}}}}$
LLaVA-NeXT-8B	OOD-H	43.62	10.86	7.05	00:12:38 ${}^{\text{{{2}}}}$
	ID	87.99	22.38	11.19	00:08:56 ${}^{\text{{{2}}}}$
	OOD-S	70.19	15.19	8.21	00:09:51 ${}^{\text{{{2}}}}$

Appendix

TABLE XVII: Experimental performance of 10 leading VLMs on Cityscapes for Basic-to-Advanced Progress.

Model	Data Type	OODBench Performance
	Data Type	Num	Accuracy(%)	F1(%)	Precision(%)	Recall(%)	MCC(%)
Random Chance	-	-	50.00	50.00	50.00	50.00	0.00
COCO
Qwen2-VL-2B	OOD-S	6000	78.78	74.62	92.85	62.37	60.95
	OOD-H	6000	80.63	76.53	97.13	63.13	65.40
	OOD-S’	7972	78.84	74.61	93.23	62.19	61.17

Appendix

TABLE XVIII: Performance of different model scales on OODBench. The table shows results for the 2B and 7B variants of the Qwen2-VL series across various data domains on OODBench. It can be observed that there is no monotonic relationship between model scale and OOD robustness; increasing parameter scale does not consistently improve model performance under semantic OOD conditions. Where, OOD-S’ and OOD-H’ denote the initial benchmark datasets.

Panel of Figure 39 — Fig. 11: Logistically visualized distributions of CLIP (top image) and BLIP2 (bottom image) on coco val.

Fig. 12: CLIP visualization distribution of 80 category logits on coco val.

Fig. 13: BLIP2 visualization distribution of 80 category logits on coco val.

Fig. 14: Sample OOD instances from the COCO and nuScenes datasets are divided by OODBench. These examples illustrate diverse out-of-distribution categories across different scenarios and environments.

Fig. 15: Sample OOD instances divided from the LVIS and Cityscapes datasets by OODBench. The samples highlight distributional shifts in object-centric.

Fig. 16: This is a showcase of OODBench examples on the COCO dataset. Examples cover standard question-and-answer, chain-of-thought prompts, and task types such as existential, counting, and logical reasoning. Green font indicates correct responses given by the model, and red font indicates incorrect responses.

Fig. 17: Demonstration of OODBench cases on the LVIS dataset. The questions shown cover a variety of task formats and visualise the performance of the model in an out-of-distribution scenario. Green font indicates correct responses, and red font indicates incorrect responses.

Fig. 18: OODBench case demonstrations on the nuScenes dataset. It includes visual Q&A and complex inference tasks in multimodality; green font indicates the model’s correct predictions, and red font indicates incorrect predictions.

Fig. 19: A showcase of OODBench cases on the Cityscapes dataset. A wide range of question types is covered to assess the model’s ability to generalize to out-of-distribution environments such as urban streetscapes. Green font represents correct responses, and red font represents incorrect responses.

Fig. 20: Typical failure cases of GPT-4o on OODBench. Errors primarily fall into two categories: non-main semantic objects and semantic variants.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

Mar 3, 2026

Mind the Way You Select Negative Texts: Pursuing the Distance Consistency in OOD Detection with VLMs

Out-of-distribution (OOD) detection seeks to identify samples from unknown classes, a critical capability for deploying machine learning models in open-world scenarios. Recent research has demonstrated that Vision-Language Models (VLMs) can effectively leverage their multi-modal representations for OOD detection. However, current methods often incorporate intra-modal distance during OOD detection, such as comparing negative texts with ID labels or comparing test images with image proxies. This design paradigm creates an inherent inconsistency against the inter-modal distance that CLIP-like VLMs are optimized for, potentially leading to suboptimal performance. To address this limitation, we propose InterNeg, a simple yet effective framework that systematically utilizes consistent inter-modal distance enhancement from textual and visual perspectives. From the textual perspective, we devise an inter-modal criterion for selecting negative texts. From the visual perspective, we dynamically identify high-confidence OOD images and invert them into the textual space, generating extra negative text embeddings guided by inter-modal distance. Extensive experiments across multiple benchmarks demonstrate the superiority of our approach. Notably, our InterNeg achieves state-of-the-art performance compared to existing works, with a 3.47% reduction in FPR95 on the large-scale ImageNet benchmark and a 5.50% improvement in AUROC on the challenging Near-OOD benchmark.

Zhikang Xu, Qianqian Xu, Zitai Wang +4

Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences +2

OOD Detection Vision-Language Models

Ask PDF

Oct 24, 2026

What Remains OOD? OODConceptBench for Open-World Concept Recognition in Vision-Language Models

In the era of large-scale pretrained models, zero-shot generalization has increasingly blurred the boundary between known and unknown concepts. However, current evaluations do not directly reveal which open-world concepts different vision-language models can recognize. To study this problem, we introduce OODConceptBench, a hierarchical benchmark for open-world concept recognition on image-concept pairs. The full benchmark set (L1) contains 30,752 images, and the harder subset (L2) contains 14,285 images. The benchmark covers 560 concept phrases across seven open-world categories. We evaluate 37 vision-language models across release year, model family, and variant. We obtain three main findings. First, target-concept recognition is lower on L2 than on L1 across all 37 evaluated models. Second, later-released models do not necessarily achieve stronger open-world concept recognition. Third, an increase in overall recognition does not imply consistent improvement across all benchmark categories. Together, these results show that OOD remains a challenge even in the era of large-scale vision-language pretraining.

Zhiran Luo, Zhiyong Yang, Zitai Wang +3

School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China +2

VLM Evaluation OOD Generalization

Ask PDF

Jun 26, 2026

DataComp-VLM: Improved Open Datasets for Vision-Language Models

Building performant Vision-Language Models (VLMs) requires carefully curating large-scale training datasets, yet the community lacks systematic benchmarks for evaluating such curation strategies. We introduce DataComp for VLMs (DCVLM), a benchmark for controlled data-centric experiments to improve VLM training. As part of DCVLM, we collect 160 datasets spanning four data types -- image-caption pairs, multimodal interleaved documents, text-only, and instruction-tuning data -- into a corpus of 6T multimodal tokens. DCVLM allows participants to test curation strategies (filtering, mixing, formatting, sampling) across 1B-8B models and 6.25B-200B token budgets. Models are then evaluated on a carefully selected suite of up to 52 downstream benchmarks across 9 domains. We conduct extensive experiments on DCVLM and find that data mixing, not filtering, is key to a high-quality training dataset: instruction-heavy mixtures scale better than caption-heavy ones, with gains widening at larger scales. The resulting dataset, DCVLM-Baseline, enables training an 8B VLM to 63.6% accuracy on our 33-task core suite with 200B training tokens. Compared to FineVision, the state-of-the-art open VLM training dataset, this represents an improvement of +5.4pp. DCVLM and all accompanying artifacts will be made publicly available at https://www.datacomp.ai/dcvlm/.

Matteo Farina, Vishaal Udandarao, Thao Nguyen +33

University of Trento · 2Tübingen AI Center, University of Tübingen · University of Cambridge +12

VLM Evaluation Training Data Curation Large Vision-Language Models Vision-Language Pretraining

Ask PDF

Abstract

VLM Evaluation

Explore similar work

Mind the Way You Select Negative Texts: Pursuing the Distance Consistency in OOD Detection with VLMs

What Remains OOD? OODConceptBench for Open-World Concept Recognition in Vision-Language Models

DataComp-VLM: Improved Open Datasets for Vision-Language Models