Organizations: Department of Computer Science, Southern University of Science and Technology, Shenzhen, China · Vision Intelligence Center, Hefei CAS Dihuge Automation Co., LTD, Hefei, China · School of Information Science and Technology, Harbin Institute of Technology (Shenzhen), China · School of Control Science and Engineering, Shandong University, Jinan, China, and also with the Key Laboratory of Machine Intelligence and System Control, Ministry of Education, China · Research Center for Industries of the Future and the School of Engineering, Westlake University, Hangzhou, China, and also with OPPO Research, Seattle, WA, USA
Current vision-language detection and grounding models predominantly focus on prompts with positive semantics and often struggle to accurately interpret and ground complex expressions containing negative semantics. A key reason for this limitation is the lack of high-quality training data that explicitly captures discriminative negative samples and negation-aware language descriptions. To address this challenge, we introduce D-Negation, a new dataset that provides objects annotated with both positive and negative semantic descriptions. Building upon the observation that negation reasoning frequently appears in natural language, we further propose a grouped opposition-based learning framework that learns negation-aware representations from limited samples. Specifically, our method organizes opposing semantic descriptions from D-Negation into structured groups and formulates two complementary loss functions that encourage the model to reason about negation and semantic qualifiers. We integrate the proposed dataset and learning strategy into a state-of-the-art language-based grounding model. By fine-tuning fewer than 10 percent of the model parameters, our approach achieves improvements of up to 4.4 mAP and 5.7 mAP on positive and negative semantic evaluations, respectively. These results demonstrate that explicitly modeling negation semantics can substantially enhance the robustness and localization accuracy of vision-language grounding models.
Figures & tables
Fig. 1: Intuition of positive and negative semantic referring. Besides object names and positive logic descriptions, humans can refer to objects with negative-semantic. For instance, when unsure how to describe the color of the calico cat, one might say, “the cat not in black.” Existing grounding models struggle to understand such referring and may even provide opposite localization results. By further fine-tuning, our method can efficiently enhance the grounding ability to complex referring.
Fig. 2: (a) We use the advanced MLLM, GPT-4V, to generate 6 positive semantic and six negative semantic referrings for every selected image. (b) The proposed D-Negation dataset contains a total of 13,893 images across 80 categories. (c) The proposed D-Negation dataset contains 139,980 text annotations. What sets it apart from existing datasets is the significantly higher frequency of negation words, as well as a greater tendency to use modifiers to describe objects.
Fig. 3: Overview of the proposed fine-tuning framework with explicit semantic-opposition constraints. The framework is designed for the D-Negation dataset and is compatible with visual grounding models that incorporate a vision–language fusion module. Given an image and semantically opposed textual descriptions (e.g., positive vs. negative attributes), text embeddings interact with visual features in the fusion module and are decoded into grounding predictions. Beyond standard grounding losses, we introduce two complementary objectives. The TSO loss imposes a distance-based exclusion constraint in the text embedding space, explicitly pushing semantically opposed descriptions (e.g., “red” vs. “not red”) apart. The PNC loss enforces a semantic exclusion constraint , ensuring that a visual region cannot be simultaneously aligned with both polarities of the same attribute. Together, these constraints address negation-induced grounding ambiguity by enforcing semantic opposition at both the linguistic and cross-modal fusion levels.
Method
Image Consumption
Batch Size
Epochs
Grounding-DINO
∼ 6.8M
64
18
APE
17.28M
16
1
Ours
13K
1
1
TABLE I: Comparison of Training Configurations
Method
Intra-scenario
Inter-scenario
Full
Presence
Absence
Full
Presence
Absence
SPHINX-7B [ 24 ]
10.6
11.4
7.9
-
-
-
GLIP-T [ 23 ]
19.1
18.3
21.5
-
-
-
OFA-DOD [ 49 ]
21.6
23.7
15.4
5.7
6.9
2.3
FIBER-B [ 8 ]
22.7
21.5
26.0
-
-
-
MLLM with a larger parameter scale
TABLE II: Performance comparison on D3 dataset.
Grounding Models
Original
+Flickr30k
+Ours
Grounding-DINO-Base
71.1
72.6 (+1.5)
75.3 (+4.2)
Grounding-DINO-Full
74.2
74.0 (-0.2)
77.0 (+2.8)
APE-A
76.8
75.0 (-1.8)
78.9 (+2.1)
APE-B
80.5
78.9 (-1.6)
83.7 (+3.2)
APE-C
78.6
80.1 (+1.4)
82.8 (+4.2)
APE-D
78.9
80.2 (+1.3)
84.1 (+5.2)
TABLE III: Performance comparison on D-Negation test set.
Method
val
testA
testB
@1
@5
@10
@1
@5
@10
@1
@5
@10
Grounding-DINO-Full
53.1
89.7
95.1
59.2
91.0
95.5
46.7
87.8
93.6
Grounding-DINO-Full(+Ours)
51.6
87.2
93.3
60.3
92.6
96.8
48.4
88.2
95.0
APE-C
79.8
-
-
86.8
-
-
76.2
-
-
APE-C(+Ours)
80.5
-
-
87.8
-
-
77.1
-
-
TABLE IV: Performance comparison of different methods on RefCOCO val, testA, and testB datasets.
Table 8
Fig. 4: Qualitative Comparison of Model Outputs. Visualization of the APE model’s outputs on representative examples after fine-tuning with our proposed method. For clarity, the prompt tokens within bounding boxes are displayed beneath each image.
Method
Full
Presence
Absence
D-Negation
Baseline(APE-C)
27.8
27.9
27.3
78.6
+ D-Negation + GOBL
32.5 (+4.7)
32.3 (+4.4)
33.0 (+5.7)
82.8 (+3.2)
+Flick30k+ D-Negation + GOBL
32.9 (+5.1)
32.7 (+4.8)
33.5 (+6.2)
84.1 (+5.5)
TABLE VII: Performance comparison on Intra-scenario of D3 and D-Negation test datasets. Our method also performs better when using flick30k as an additional training set.
Editing Location
Full
Presence
Absence
Text Encoder
+0.7
+0.5
+1.1
Image Backbone
-0.3
-0.1
-0.7
Decoder
+1.2
+0.6
+1.3
Fusion Module
+4.7
+4.4
+5.7
TABLE VIII: Relative performance gains (%) obtained by applying negation supervision to different modules under identical training budgets.
Setting
Full
Presence
Absence
σ=3
+4.3
+4.0
+5.2
σ=5
+4.7
+4.4
+5.7
σ=7
+4.4
+4.1
+5.4
α=0.3
+4.1
+3.8
+5.0
α=0.5
+4.7
+4.4
+5.7
α=0.7
+4.2
+3.9
+5.1
TABLE IX: Sensitivity analysis of hyperparameters σ , α , and β . All experiments are conducted on APE-C with other parameters fixed to default values.
Fig. 5: Performance gain of single and combined attributes on the D3 dataset. Each attribute enhances the model’s performance, and combining multiple attributes leads to even greater improvements. As more attributes are added, the performance gain gradually decreases and eventually stabilizes.
Fig. 6: Heatmap of gains across different attributes. Each attribute enhances performance in its domain and benefits others, demonstrating transferability. The best results occur when all attributes are combined. Every attribute significantly boosts performance within its own domain, while also positively influencing the performance in other domains, demonstrating some degree of transferability. The best performance is achieved when all attributes are combined.
Tuning Module
Params(M)
Proportion(%)
Full/Presence/Absence
Fusion Only
15.7
9.0
17.8 / 17.4 / 19.0
+Decoder
26.7
15.5
15.4 / 15.9 / 16.9
+Decoder +Neck
36.6
21.1
13.4 / 13.6 / 12.9
TABLE X: Comparison of different tuning modules. Base on Grounding-DINO-Base. Performance on Intra-scenario of D3 dataset.
Model
Full
Presence
Absence
APE-B
30.0
29.9
30.3
+ Standard CL
30.9 (+0.9)
30.2 (+0.3)
31.3 (+0.8)
+ GOBL (Ours)
32.4 (+2.4)
32.2 (+2.3)
32.9 (+2.6)
APE-C
27.8
27.9
27.3
+ Standard CL
28.7 (+0.9)
28.5 (+0.6)
29.1 (+1.8)
+ GOBL (Ours)
32.5 (+4.7)
32.3 (+4.4)
33.0 (+5.7)
TABLE XI: Performance comparison on the intra-scenario split of the D3 dataset. Numbers in parentheses denote absolute improvements over the corresponding APE baseline.
Attribute
Full
Presence
Absence
APE-C
27.8
27.9
27.3
Color
30.8 (+3.0)
31.2 (+3.3)
30.2 (+2.9)
Position
30.2 (+2.4)
30.3 (+2.4)
29.2 (+1.9)
State
29.6 (+1.8)
29.9 (+2.0)
29.1 (+1.8)
Color + Position
32.1 (+4.3)
32.2 (+4.3)
31.9 (+4.6)
Position + State
30.4 (+2.6)
30.8 (+2.9)
30.0 (+2.7)
TABLE XII: Performance improvement with single and combined attributes.
Setting
Full
Presence
Absence
σ=2
+1.9
+1.8
+2.2
σ=3
+2.1
+2.0
+2.4
σ=5
+2.4
+2.3
+2.6
σ=7
+2.2
+2.1
+2.5
σ=9
+2.0
+1.9
+2.3
α=0.2
+1.8
+1.7
+2.1
TABLE XIII: Extended sensitivity analysis of hyperparameters σ , α , and β . All experiments are conducted on APE-B with other parameters fixed to default values. Results are reported as relative improvements over the APE-B baseline.