LoCo: Selective Local Competition for Discriminative Open-Vocabulary Multi-Label Recognition
Authors: Zhuoming Li, Bo Han, Xiaoyu Wang, Yuheng Jia
Organizations: School of Software Engineering, Southeast University, Nanjing, China · School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China
Open-Vocabulary Multi-Label Recognition (OV-MLR) aims to identify all queried semantic concepts present in an image. Existing methods mainly improve global image-text matching or text-side representations, leaving local prediction largely under-explored. In this paper, we revisit local prediction and reveal that it can serve as a strong discriminative branch once two properties are properly handled: spatial consistency of local visual features and compatibility-aware local competition. This perspective is motivated by local semantic sparsity: each image region is related to only a small subset of queried concepts, while compatible concepts of different granularities or types may still coexist locally. We instantiate this perspective with LoCo, a selective Local Competition framework for OV-MLR. LoCo adopts a spatially consistent visual encoder to reduce local feature entanglement, and introduces Selective Softmax to impose competition only among locally incompatible categories while preserving compatible responses. Multi-scale inference further improves local prediction at different scales. LoCo can also be ensembled with an existing global predictor to provide complementary image-level context. Experiments on four multi-label benchmarks show that standalone LoCo improves average F1-score and mAP over TagCLIP by 3.4% and 3.8%, respectively. LoCo + DualPrompt achieves the highest averages among the compared methods, with gains of 4.8% and 5.2% over TagCLIP. Our results suggest that local semantic sparsity offers a useful basis for developing more discriminative open-vocabulary multi-label recognition methods. The code is available at https://github.com/lee-plus-plus/LoCo.
Figures & tables
Figure 1: Motivation and key insight of this paper. (a) Existing OV-MLR methods [ 1 , 8 ] produce multi-label predictions with low discriminability, and our model overcomes this limitation. Blue bars denote true positive predictions, while orange bars denote false positive predictions. (b) The potential of local prediction is under-exploited by existing methods [ 1 , 8 ] . Our model unleashes its potential and achieves outstanding performance. (c) The spatial consistency of the dense visual encoder, measured by its segmentation performance, i.e. , mIoU, enables multi-label recognition in a local prediction framework.
Figure 2: Feature entanglement in multi-label scenarios. Blue bars denote true positive predictions, while orange bars denote false positive predictions. As more semantic concepts co-occur in the same image, CLIP’s prediction score gap between positive and negative classes narrows. We attribute this to feature entanglement caused by mutual dilution across different categories of information.
Figure 3: Spatial consistency of dense visual backbones matters for discriminative local prediction. CLIP [ 1 ] , TagCLIP [ 6 ] , and SC-CLIP [ 19 ] are selected as weak, moderate, and strong visual backbones. Softmax is applied at each spatial position to produce dense predictions. With stronger visual backbones, local features are less fragmented and more spatially consistent, thereby producing more discriminative multi-label predictions.
Relation type
Examples
Locally Compatible?
Scene-level Co-occurrence
room–bed ; street–car
No
Covering Relation
person–shirt ; hand–glove
Usually Yes
Overlap Relation
snow–mountain ; light–lamp
Usually Yes
Taxonomic Inclusion
animal–dog ; vehicle–car
Yes
Part–whole Composition
person–arm ; car–wheel
Yes
Surface Marking
sign–letter ; screen–logo
Yes
Table 1: Major types of semantic relations among categories in the VG dataset.
Figure 4: Multi-label local prediction with different prediction strategies. Softmax is effective for enhancing discriminability, but it introduces unreasonable competition between compatible classes, e.g. , clouds and sky , or sun and sunset , resulting in over-competition. Selective Softmax introduces competition between each category and its locally incompatible competitors and produces better predictions.
Figure 5: Framework of the proposed LoCo. An image is first divided into K crops using grid cropping and encoded by a spatially consistent ViT. Local tokens produce crop-level logits. The multi-scale aggregation module spatially aligns and averages these logits before Selective Softmax and spatial max pooling. The resulting local predictions can optionally be ensembled with global predictions averaged over the same crops.
Figure 6: Selective local competition guided by an LLM compatibility prior, where clouds is selected as the target category.
Table 2: Experimental results on open-vocabulary multi-label recognition. mAP measures test-set ranking quality. Avg. is the unweighted mean over four datasets. Best and second-best results are highlighted.
Table 9Table 10
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Classes
CLIP mean
TaI-DPT mean
VOC
20
81.0
24.0
COCO
80
76.0
14.0
NUS
81
80.0
19.0
VG
150
78.0
17.0
Appendix
Table 7: Mean text-prototype cosine similarities, displayed in units of 10−2 . Multiply displayed similarities by 10−2 to recover the dimensionless cosine values.
Model
Positive only
Negative only
Cohen’s D
Mean ± Std.
Mean ± Std.
CLIP
175.0±32.0
138.0±24.0
1.5
TaI-DPT
17.0±5.0
11.0±3.0
2.3
LoCo
449.0±248.0
55.0±78.0
4.1
Appendix
Table 8: Reported response statistics across four datasets. All score statistics are displayed in units of 10−3 ; multiply them by 10−3 to recover the original response scale. Cohen’s D is dimensionless and unscaled.
Table 9: Per-dataset ablation of the standalone local branch. Top: F1-score and mAP. Bottom: micro PR-AUC and Recall@10. Sel. denotes Selective Softmax and M.S. denotes 14-crop inference; unchecked settings use Softmax and one crop, respectively.
Figure 7: Sensitivity analysis of the hyperparameter τ .
Table 10: Per-dataset matched comparison against TaI-DPT on the same SC-CLIP backbone, with and without M.S. Top: F1-score and mAP. Bottom: Recall@10 and micro PR-AUC. Recall@10 is evaluated against annotated positive labels.
Dataset
Five independent generations
Three prompt variants
Edges
Shared
Jaccard
Overlap
Edges
Shared
Jaccard
Overlap
VOC (20)
25–42
25
77.0%
97.6%
31–45
26
63.4%
88.4%
COCO (80)
213–283
174
73.1%
90.6%
190–221
151
71.8%
88.0%
NUS (81)
277–428
221
69.1%
90.0%
266–367
210
63.9%
87.3%
VG (150)
632–1019
531
66.7%
88.9%
593–822
390
52.3%
77.4%
Appendix
Table 11: Compatibility-graph stability under five independent generations and three prompt variants. Edge ranges and shared-edge counts are integers; Jaccard and overlap are percentages.
Dataset
Edges
Shared
Jaccard (%)
Overlap (%)
VOC (20)
25–38
22
70.3%
93.9%
COCO (80)
98–456
70
35.8%
88.6%
NUS (81)
157–597
125
40.9%
84.5%
VG (150)
438–882
286
48.9%
79.7%
Appendix
Table 12: Cross-LLM compatibility-graph consistency. Shared edges form the four-model LLM consensus. Similarities are reported in percent.
Prior
Avg. edges
Avg. F1-score
Avg. mAP
glm-5.2
490.0
58.1
66.0
gemini-3.6-flash
182.3
58.3
66.1
claude-opus-4.6
290.5
58.5
65.6
gpt-5.5
314.8
58.5
66.0
LLM consensus
125.8
58.6
66.0
Reported std.
–
±0.2
±0.2
Appendix
Table 13: Sensitivity to compatibility priors. Edges and scores are averaged over the four datasets. Scores are percentages; best and second-best values include ties at the reported precision.
Dataset
WordNet edges
LLM-consensus edges
Jaccard (%)
Overlap (%)
VOC (20)
0
22
0.0%
0.0%
COCO (80)
3
70
1.4%
33.3%
NUS (81)
34
125
20.5%
79.4%
VG (150)
323
286
39.7%
60.5%
Appendix
Table 14: WordNet-derived relations and the four-model LLM consensus. Jaccard and overlap compare their edge sets.
Figure 8: More qualitative results of LoCo’s dense predictions. Class activation maps in rows 1, 2, and 3 show logits, Softmax predictions, and Selective Softmax predictions, while columns 1, 2, 3, and 4 indicate flowers , garden , plants , and person classes.
Figure 9: More qualitative results of LoCo’s dense predictions. Class activation maps in rows 1, 2, and 3 show raw logits, softmax predictions, and selective softmax predictions, while columns 1, 2, 3, and 4 indicate garden , grass , plants , and person classes.
Figure 10: Failure cases for LoCo on the NUS dataset. Blue bars denote scores for annotated positive categories; orange bars highlight selected categories absent from the original annotations. The original annotations are shown above each image. Some unannotated concepts are visually present.
School of Computer Science and Engineering, South China University of Technology, Guangzhou, 510006, China. · Department of New Network, Pengcheng Laboratory, Shenzhen, 518000, China.