Recent advances in data-centric artificial intelligence highlight inherent limitations in object recognition datasets. One of the primary issues stems from the semantic gap problem, which results in complex many-to-many mappings between visual data and linguistic descriptions. This bias adversely affects performance in computer vision tasks. This paper proposes an image annotation methodology that integrates knowledge representation, natural language processing, and computer vision techniques, aiming to reduce annotator subjectivity by applying visual property constraints. We introduce an interactive crowdsourcing framework that dynamically asks questions based on a predefined object category hierarchy and annotator feedback, guiding image annotation by visual properties. Experiments demonstrate the effectiveness of this methodology, and annotator feedback is discussed to optimize the crowdsourcing setup.
Figures & tables
Figure 1: Images and their labels in existing image datasets. (a) This image appears in three categories of ImageNet, i.e., “Musical Instrument”, “Guitar”, and “Acoustic Guitar”. (b)-(e) These four images are all labelled as the “Brown Bear” category in the Open Image Dataset.
Categories
I
II
III
IV
V
VI
VII
VIII
IX
X
XI
XII
Unrecognised
Alpha
Annotators group 1
98
101
86
102
97
103
92
95
96
96
90
95
49
0.934
Annotators group 2
91
105
93
98
101
98
91
94
87
104
88
92
58
Annotators group 3
95
100
91
101
99
102
95
92
90
98
93
98
46
Table 1: The number of images for categories we obtained by the proposed image annotation methodology from three groups of annotators, where ”I, II, …, XII” denote twelve categories. The “Unrecognised” indicates instances where annotators did not annotate the image into any of the twelve specific categories but rather assigned it to an upper-level category, e.g., its parent category. ”Alpha” refers to Krippendorff’s alpha measure, a statistical tool for assessing the reliability of these annotations.
Image annotation methodology
50 images per task
100 images per task
Alpha
Time (min)
Payments(£/p)
Alpha
Time (min)
Payments(£/p)
Method A
0.912
4.12
1
0.896
8.27
2
Method B
0.937
5.08
1.5
0.914
9.22
3
Method C (Ours)
0.974
5.47
1.5
0.958
9.47
3
Table 2: Results of implementing three different image annotation methodologies, including inter-class agreement evaluation with Krippendorff’s alpha measure, time cost on single tasks, and payment to one annotator for one task. To evaluate the proper number of images per task, we present a comparison of results for tasks with 50 images and 100 images.
Model
Method A
Method C (Ours)
Improvement
AlexNet [ 19 ]
0.543
0.596
9.76%
ZFNet [ 46 ]
0.612
0.657
7.35%
VGG [ 36 ]
0.655
0.743
13.44%
GoogleNet [ 39 ]
0.734
0.835
13.76%
ResNet [ 12 ]
0.593
0.732
23.44%
DenseNet [ 14 ]
0.724
0.793
9.53%
Table 3: Complementary downstream visual recognition results on an ImageNet-derived benchmark annotated under two different designs: a conventional name-based labeling condition and the full visual-property-guided design. Higher values indicate better classification accuracy.
Method
Hierarchy
Visual Properties
50 Images per Task
100 Images per Task
Alpha
Time (min)
Payment (£/p)
Alpha
Time (min)
Payment (£/p)
Method A
✗
✗
0.912
4.12
1.0
0.896
8.27
2.0
Method B
✓
✗
0.937
5.08
1.5
0.914
9.22
3.0
Method C (Ours)
✓
✓
0.974
5.47
1.5
0.958
9.47
3.0
Δ (B − A)
–
–
+0.025
+0.96
+0.5
+0.018
+0.95
+1.0
Δ (C − B)
–
–
+0.037
+0.39
+0.0
+0.044
+0.25
+0.0
Table 4: Ablation study on annotation design under the crowdsourcing setting. Method A uses category names only; Method B adds an object hierarchy; Method C further adds visual properties (visual genus and visual differentia) within the hierarchy. Higher Alpha indicates better inter-annotator agreement.