What Remains OOD? OODConceptBench for Open-World Concept Recognition in Vision-Language Models

Zhiran Luo; Zhiyong Yang; Zitai Wang; Sicong Li; Qianqian Xu; Qingming Huang

cs.CVOct 24, 2026

What Remains OOD? OODConceptBench for Open-World Concept Recognition in Vision-Language Models

Authors: Zhiran Luo, Zhiyong Yang, Zitai Wang, Sicong Li, Qianqian Xu, Qingming Huang

Organizations: School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China ·

Abstract

In the era of large-scale pretrained models, zero-shot generalization has increasingly blurred the boundary between known and unknown concepts. However, current evaluations do not directly reveal which open-world concepts different vision-language models can recognize. To study this problem, we introduce OODConceptBench, a hierarchical benchmark for open-world concept recognition on image-concept pairs. The full benchmark set (L1) contains 30,752 images, and the harder subset (L2) contains 14,285 images. The benchmark covers 560 concept phrases across seven open-world categories. We evaluate 37 vision-language models across release year, model family, and variant. We obtain three main findings. First, target-concept recognition is lower on L2 than on L1 across all 37 evaluated models. Second, later-released models do not necessarily achieve stronger open-world concept recognition. Third, an increase in overall recognition does not imply consistent improvement across all benchmark categories. Together, these results show that OOD remains a challenge even in the era of large-scale vision-language pretraining.

Figures & tables

Figure 1: From predefined-class OOD to model-specific concept recognition. OODConceptBench fixes image–concept pairs and compares recognition across models using a shared calibration rule with model-specific recognition thresholds.

Figure 2: L2 concept-recognition landscape for the 2025 release-year group. The 91 concepts are positioned by group-mean recognition, with lower recognition toward the outer regions and higher recognition toward the center.

\tikz [overlay,baseline][MainTableHeader,rounded corners=1.5mm] (0pt,-2.25pt) rectangle (.75pt); Filtering rule	Criterion	Result
Rule 1 — Common-category reference	$S_{1K}>0.7175$	34,963
Rule 2 — Rare reference	$S_{21K}>0.2976$	L1: 30,752
Rule 3 — Target-concept alignment	$S_{\mathrm{Txt}}<0.3031$	L2: 14,285

Table 1: Filtering criteria and resulting benchmark sizes.

Figure 3: Benchmark composition and coverage. (a) Category counts in L1 and L2, (b) image-source composition, and (c) concept coverage, measured by the number of target concepts with at least one image in L0, L1, and L2.

Figure 4: Per-model target-concept recognition on L1 and L2. Models are grouped by family, and each row compares recognition on L2 and L1 for the same model. Across all 37 evaluated models, recognition is lower on L2 than on L1. * B/32 local denotes the OpenAI CLIP ViT-B/32 checkpoint used as the fixed construction model.

Figure 5: Recognition across release-year groups and benchmark categories. (a) Mean L1 and L2 recognition rates for models in each release-year group from 2021 to 2025. (b) The same comparison for each of the seven benchmark categories. All panels use the same 0–100 scale and recognition-range background bands.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Filtering rule	Calibration setting	Calibration rule	Final threshold
1	Up to 3,000 ID calibration images; ImageNet-1K validation with Tiny-ImageNet validation fallback; ImageNet-1K reference features.	95th percentile of $S_{\mathrm{1K}}$ calibration scores.	$\bm{T_{1}=0.7175}$
2	Filtering-rule-1 calibration survivors satisfying $S_{\mathrm{1K}}>T_{1}$ , plus up to 1,500 Tiny-ImageNet training images; fixed 200-vector long-tail reference bank.	95th percentile of the mixed calibration-pool $S_{\mathrm{21K}}$ scores.	$\bm{T_{2}=0.2976}$
3	5,000 MNIST test images as negative samples; fixed 200-concept calibration vocabulary with repeated subsampling.	Per-round candidate threshold from the maximum target-prompt similarity plus its standard deviation, followed by the multi-round stability check.	$\bm{T_{3}=0.3031}$

Appendix

Table A1: Calibration settings and final thresholds for the three benchmark-construction rules.

Panel of Figure 8 — Figure A1: Filtering rule 1 and filtering rule 2 calibration-score distributions. The vertical lines mark the 95th-percentile thresholds, $T_{1}=0.7175$ and $T_{2}=0.2976$ ; KDE curves are visual summaries only.

Category	L0 $\rightarrow$ L1	L1 $\rightarrow$ L2	L0 $\rightarrow$ L2
Time-lagged	77.44%	76.87%	59.53%
Synthetic / fictional	66.24%	38.64%	25.59%
Cultural blindspots	61.68%	27.71%	17.09%
Fine-grained / texture	47.96%	50.26%	24.11%
Environmental shift	67.09%	49.66%	33.31%
Viewpoint / scale	52.65%	46.50%	24.48%

Appendix

Table A4: Category-wise retention across benchmark construction stages.

Figure A3: Qualitative coverage across the seven benchmark categories. Within each category block, the upper sample is L1-only and the lower sample is L2; labels show the assigned target concept, benchmark layer, and image source.

Variant	Year	Backbone	$Q_{95,m}$	L1	L2
ConvNeXt-CLIP
Base-W	2022	convnext_base_w	0.3750	0.759	0.516
Base-W AugReg	2022	convnext_base_w	0.3454	0.767	0.536
Large-D AugReg	2022	convnext_large_d	0.2279	0.778	0.557
Large-D-320 FT	2022	convnext_large_d_320	0.2273	0.788	0.575
DFN-CLIP

Appendix

Table C1: Model registry and overall target-concept recognition results.

Variant	Time	Synth.	Culture	Fine/tex.	Environ.	View/scale	Typogr.	Overall
ConvNeXt-CLIP
Base-W	0.499	0.922	0.831	0.632	0.758	0.786	0.865	0.759
Base-W AugReg	0.539	0.941	0.806	0.631	0.765	0.779	0.871	0.767
Large-D AugReg	0.541	0.961	0.826	0.638	0.772	0.796	0.878	0.778
Large-D-320 FT	0.567	0.960	0.842	0.639	0.776	0.812	0.883	0.788
DFN-CLIP

Appendix

Table C2: Per-model target-concept recognition rates by concept category on L1, grouped by model family. Category headers are abbreviated for layout; the full names are given below the table. Short variant labels are used for readability.

Variant	Time	Synth.	Culture	Fine/tex.	Environ.	View/scale	Typogr.	Overall
ConvNeXt-CLIP
Base-W	0.373	0.807	0.516	0.295	0.526	0.582	0.645	0.516
Base-W AugReg	0.416	0.854	0.485	0.299	0.544	0.582	0.653	0.536
Large-D AugReg	0.417	0.903	0.526	0.312	0.555	0.620	0.677	0.557
Large-D-320 FT	0.448	0.906	0.568	0.314	0.561	0.653	0.687	0.575
DFN-CLIP

Appendix

Table C3: Per-model target-concept recognition rates by concept category on L2, grouped by model family. Category headers are abbreviated for layout; the full names are given below the table. Short variant labels are used for readability.

Year	Models	Mean L1	Median L1	Mean L2	Median L2	Mean L1–L2 difference	Median L1–L2 difference
2021	5	61.6	60.7	22.8	15.3	38.8	42.4
2022	9	73.6	75.9	48.4	51.6	25.2	24.3
2023	16	62.9	67.5	41.1	44.4	21.8	22.1
2024	3	74.6	75.7	51.4	52.8	23.2	22.9
2025	4	72.5	73.0	52.4	53.1	20.2	20.2

Appendix

Table C5: Model-weighted concept-recognition trends by release-year group. Rates and L1–L2 differences are percentages; year groups differ in family, data, objective, resolution, and backbone composition (Section 6.3).

Family	Reference variant(s)	Compared variant(s)	Reference	Compared	Difference	Caution
ConvNeXt-CLIP	Base-W variants	Large-D variants	52.6	56.6	4.0	Recipe and resolution differ.
DFN-CLIP	B/16 variants	L/14	49.0	56.1	7.1	Training scale and implementation differ.
EVA-CLIP	B/16	L/14	50.0	59.6	9.6	Single-model comparison.
OpenAI CLIP	B/32 variants	B/16 variants	13.6	36.6	23.0	Patch size, resolution, and variants differ.
OpenCLIP-LAION	B/32 variants	B/16	39.5	45.7	6.1	Training tags and data scale differ.
OpenCLIP-LAION	B/16	H/14	45.7	53.0	7.3	Training tags and data scale differ.

Appendix

Table C6: Selected within-family variant comparisons. L2 target-concept recognition rates and differences are percentages. These comparisons are descriptive because training data, objectives, architectures, input resolutions, and implementations may also differ (Section 6.3).

(a) L1
Category	2021 ( $n=5$ )	2025 ( $n=4$ )	Difference (2025 $-$ 2021; pp)
Time-lagged	30.4	54.2	$+23.8$
Synthetic / fictional	74.3	94.1	$+19.9$
Cultural blindspots	77.8	66.2	$-11.6$
Fine-grained / texture	56.4	59.5	$+3.1$
Environmental shift	58.7	72.9	$+14.2$

Appendix

Table C7: Model-weighted target-concept recognition by category for the 2021 and 2025 release-year groups at both benchmark levels. Rates and differences are percentages. Group means and differences are independently rounded to one decimal, with differences computed from the unrounded group means as 2025 minus 2021. The year groups are not matched causal samples (Section 6.4).

Scope	SigLIP	SigLIP2	Difference
L1	66.6	72.5	6.0
L2	44.9	52.4	7.5
Time-lagged	35.1	43.8	8.7
Synthetic / fictional	79.6	86.0	6.4
Cultural blindspots	29.5	38.5	9.0
Fine-grained / texture	24.3	28.2	3.9

Appendix

Table C8: SigLIP-to-SigLIP2 concept-recognition comparison. Values are family means in percentages; category rows use L2 (Section 6.3).

Figure C1: L1 target-concept recognition by model and concept category. Models are grouped by family and use family-specific short variant labels.

Figure C2: L2 target-concept recognition by model and concept category. Models are grouped by family and use family-specific short variant labels.

Figure C3: Concept-level L2 recognition landscapes for the 2021 and 2025 release-year groups using the same fixed 91-concept subset.

ID baseline	Layer	$N_{ID}$	$N_{benchmark}$	Mean confidence	AUROC	FPR95	AUPR-in	AUPR-out	ECE
Oxford-IIIT Pet	L1	3,669	30,752	0.263	0.830	0.650	0.420	0.974	0.187
Oxford-IIIT Pet	L2	3,669	14,285	0.261	0.831	0.658	0.565	0.949	0.115
CIFAR-100	L1	10,000	30,752	0.263	0.395	0.949	0.217	0.667	0.183
CIFAR-100	L2	10,000	14,285	0.261	0.402	0.951	0.372	0.497	0.242
STL-10	L1	8,000	30,752	0.263	0.738	0.793	0.426	0.912	0.093
STL-10	L2	8,000	14,285	0.261	0.741	0.799	0.595	0.838	0.064

Appendix

Table D1: Fixed-ID results across reference datasets.

Protocol element	Specification
Encoder	CLIP ViT-B/32 with standard preprocessing and unit-normalized image features.
Task	Binary benchmark-sample-versus-pooled-control classification, run separately by L1/L2 layer and category.
Classifier	Logistic regression with the liblinear solver and 200 maximum iterations.
Positive class	Benchmark samples from the target layer/category slice.
Pooled control	Seven common-category control groups; no calibration/reference dataset is included.
Shots per class	1, 3, 5, 10, 15, 20, 25, and 30.

Appendix

Table D2: Limited-supervision benchmark-sample-versus-pooled-control classification protocol.

Layer	Shots	Mean acc.	Cat. min	Cat. max
L1	1	0.627	0.585	0.704
L1	3	0.736	0.667	0.811
L1	5	0.779	0.716	0.879
L1	10	0.839	0.753	0.936
L1	15	0.862	0.794	0.949
L1	20	0.871	0.815	0.957

Appendix

Table D3: Layer-level limited-supervision classification means and category ranges.

Figure D1: Category-wise benchmark-sample-versus-pooled-control classification accuracy within each L1/L2 layer across all eight shot settings.

Figure D2: Qualitative t-SNE view of the four L1/L2 layer–source groups using 300 samples per layer–source group. Color encodes layer, shape encodes source, and open markers show group centroids.

Layer	Crawler	Generated	Centroid cos. dist.	Fréchet	Target-concept cos. gap	Conf. gap
L1	14,617	16,135	0.046	0.198	0.036	0.011
L2	8,903	5,382	0.047	0.206	0.025	0.005

Appendix

Table D4: Metric summary by image source for the fixed L1/L2 layers. “Fréchet” is the CLIP-feature Fréchet distance; the cosine gap is the target-concept cosine gap.

Layer	Category	CLIP-feature Fréchet dist.	Centroid cos. dist.	Target-concept cos. gap	Conf. gap
L1	Fine-grained / texture	0.341	0.056	0.060	0.052
L1	Environmental shift	0.377	0.092	0.067	0.043
L1	Viewpoint / scale	0.341	0.066	0.017	0.026
L1	Typographic conflict	0.360	0.059	0.043	0.020
L2	Fine-grained / texture	0.408	0.072	0.022	0.019
L2	Environmental shift	0.414	0.107	0.041	0.018

Appendix

Table D5: Category-level source analyses for slices with both source types. Signed gaps use generated minus crawler means.

Figure D3: Category-level analyses grouped by image source for the four dual-source categories. Signed confidence gaps use generated minus crawler means.

Perturbation	Range	Settings	L2 count range	Jaccard min	Jaccard max	Mean
Fixed	0%	1	14,285–14,285	1.000	1.000	1.000
$T_{1}$ local perturbation	-5% to +5%	6	4,967–16,786	0.348	0.938	0.760
$T_{2}$ local perturbation	-5% to +5%	6	13,214–15,070	0.925	0.987	0.961
$T_{3}$ local perturbation	-5% to +5%	6	10,396–18,560	0.728	0.944	0.844
Joint local perturbation	-5% to +5%	6	6,026–14,568	0.312	0.876	0.658

Appendix

Table D6: Small local threshold changes and Jaccard summary.

Figure D4: L2 overlap under the specified single-threshold and joint local changes. The

0\%

reference marks fixed L2 membership.

Category	Input	Pre	Post	Add. removed	Pre/input	Post/input	Add. pruning
Time-lagged	4,299	3,747	3,728	19	87.16%	86.72%	0.44 pp
Synthetic / fictional	3,892	3,322	3,307	15	85.35%	84.97%	0.39 pp
Cultural blindspots	3,550	1,919	1,839	80	54.06%	51.80%	2.25 pp
Fine-grained / texture	3,626	1,231	1,156	75	33.95%	31.88%	2.07 pp
Environmental shift	4,106	2,930	2,885	45	71.36%	70.26%	1.10 pp
Viewpoint / scale	4,386	2,447	2,410	37	55.79%	54.95%	0.84 pp

Appendix

Table D7: WordNet reference-vocabulary expansion check. Additional pruning uses the common number of input images in the calculation.

Layer	Categories	Mean entropy	Entropy range	Mean semantic loss
L1	7	3.724	3.494–3.840	0.697
L2	7	3.757	3.459–3.975	0.736

Appendix

Table D8: Entropy and semantic-loss summary using a mean that gives each of the seven categories equal weight.

Scope	$n$	Mean	Median
All design phrases	560	0.706922	0.710037
L1-active phrases	553	0.705152	0.708891
L2-active phrases	519	0.706235	0.710064

Appendix

Table D9: Nearest-ImageNet similarity by fixed concept-activity scope.

Figure D5: Nearest-ImageNet similarity for all 560 target concept phrases in the fixed CLIP ViT-B/32 text space. Boxes summarize each category and light points show individual phrases.

Layer	$n$	Mean nearest-ID	Mean target	Pearson $r$	Spearman $\rho$
L1	30,752	0.254929	0.296289	0.128880	0.115637
L2	14,285	0.252504	0.257976	0.067633	0.039612

Appendix

Table D10: Image-level relation between fixed nearest-ID and target-similarity construction scores.

Figure D6: Conditioned image-level relation between fixed nearest-ID and target-similarity construction scores. Annotations report full-data descriptive associations.

Item	Specification
Intended use	Studying vision–language model recognition boundaries on open-world image–concept pairs.
Excluded uses	The benchmark is not a universal safety test or a basis for ranking models across unrelated protocols.
Source provenance	Acquisition source is recorded for benchmark images; any public release will retain source information where permitted.
Image-level release conditions	Dedicated near-duplicate, sensitive-content, licensing, source, and privacy review, together with generated-content and source-policy documentation.
Scope boundary	Evaluation measures target-concept recognition under the stated fixed protocol.

Appendix

Table E1: Intended-use and release-boundary summary.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

OODBench: Out-of-Distribution Benchmark for Large Vision-Language Models
Feb 20, 2026Ling Lin, Yang Bai, Heng Su +5VLM Evaluation VLM Robustness
School of Artificial Intelligence and Data Science, University of Science and Technology of China, Hefei 230000, China · Suzhou Institute for Advanced Research, USTC, Suzhou 215123, China · Institute of High Performance Computing (IHPC), A*STAR, Singapore +1
ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models
Feb 13, 2025Jonathan Roberts, Mohammad Reza Taesiri, Ansh Sharma +31Visually Grounded Reasoning VLM Reasoning
University of Cambridge
Immuno-VLM: Immunizing Large Vision-Language Models via Generative Semantic Antibodies for Open-World Trustworthiness
May 29, 2026Xiang Fang, Wanlong Fang, Wei JiVLM Robustness Large Vision-Language Models
School of Software Engineering, Huazhong University of Science and Technology · Nanyang Technological University, Singapore · Nanjing University

Feb 20, 2026J/K move · Enter open · S save

OODBench: Out-of-Distribution Benchmark for Large Vision-Language Models

Ling Lin, Yang Bai, Heng Su, Congcong Zhu+4

School of Artificial Intelligence and Data Science, University of Science and Technology of China, Hefei 230000, China · Suzhou Institute for Advanced Research, USTC, Suzhou 215123, China · Institute of High Performance Computing (IHPC), A*STAR, Singapore+1

VLM Evaluation VLM Robustness OOD Generalization Large Vision-Language Models

Open paper AskPDF arXiv

Existing Visual-Language Models (VLMs) have achieved significant progress by being trained on massive-scale datasets, typically under the assumption that data are independent and identically distributed (IID). However, in real-world scenarios, it is often impractical to expect that all data processed by an AI system satisfy this assumption. Furthermore, failure to appropriately handle out-of-distribution (OOD) objects may introduce safety risks in real-world applications (e.g., autonomous driving or medical assistance). Unfortunately, current research has not yet provided valid benchmarks that can comprehensively assess the performance of VLMs in response to OOD data. Therefore, we propose OODBench, a predominantly automated method with minimal human verification, for constructing new benchmarks and evaluating the ability of VLMs to process OOD data. OODBench contains 40K instance-level OOD instance-category pairs, and we show that current VLMs still exhibit notable performance degradation on OODBench, even when the underlying image categories are common. In addition, we propose a reliable automated assessment metric that employs a Basic-to-Advanced Progression of prompted questions to assess the impact of OOD data on questions of varying difficulty more fully. Lastly, we summarize substantial findings and insights to facilitate future research in the acquisition and evaluation of OOD data.

Abstract

VLM Evaluation

Explore similar work

OODBench: Out-of-Distribution Benchmark for Large Vision-Language Models

ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models

Immuno-VLM: Immunizing Large Vision-Language Models via Generative Semantic Antibodies for Open-World Trustworthiness