A lot of expectations have been put for years on integrating domain expert knowledge in image classification models. Several approaches have been explored, Concept Bottleneck Models (CBMs) opened up a new avenue of research leading to many variants, and more recently to Concept-based Embedding Models (CEMs). CBM consider binary encoding of each concept, while CEM expands this idea by embedding each concept through two vectors. However in real-life scenarii, domain experts' knowledge is usually organized in concepts determined by various attributes, each attribute encoded either with numerical values, or range of values, or binary values, or categorical values. In this work, we first finetune an image feature extractor for classifying attributes representing the downstream object classes, where the class attributes have been specified by experts under various encoding formats. A classification head is then learnt from these various attributes to categorize target objects. We experimentally show that it improves the classification for three datasets: Kaggle fish dataset, AWA2 and a more challenging new wood charcoal dataset. We then propose an automatic selection of potential missclassified data. In this second step, experts are asked for those data to eventually modify the predicted attributes in order to improve the classification.
Figures & tables
Figure 1: The global picture of this paper. The data are single images or set of images. We extract features for the attributes prediction. Then, classes are predicted from attributes.
Name
Dorsal spines
Dorsal soft rays
Anal spines
Anal soft rays
Body shape
Cross section
Bangus
2
13-17
2
8-10
e
ov
Big Head
3
7
1-3
12-14
f
ov
Carp
Fourfinger
9
13-15
3
14-16
e
co
Threadfin
Gourami
16-20
7-10
9-11
8-11
f
co
Table 1: Table of attributes available for each fish specie. Four first attributes are values or range of values. Body shapes are categorical variables: elongated (e), fusiform/normal (f) or cylindrical (c), mapped to [0,1,2]. Cross sections are also categorical: oval(ov), compressed (co), circular (ci) or other (ot), mapped to [0,1,2,3].
Figure 2: Example of the IAWA feature “intervessel pit size” with 4 different ordered values as described in section 2.3 . The unit interval is divided in 4 equal parts. Left (l) and right (r) values from equation 1 depend on the present value(s), represented here by coloured boxes: IAWA #25 and #26 are always present for the class considered, so that left value is 0 and right value is 0.5.
Figure 3: The different interval values functions concerning the ’intervessel pit size’ for the different species. In this case, many classes share the same function for this anatomical attribute because they share the same intervals. The goal of this loss function is to be minimal in the correct interval.
Figure 4: The interval values functions “Dorsal soft rays” for different species. Values for “Dorsal soft rays” have been rescaled between 0 and 1. The goal of this loss function is to be minimal in the correct interval.
Figure 5: The vector of anatomy of fishes: 4 float values (rescaled between 0 and 1) and 2 sets of probabilities for “Body shape” and “Cross section” types.
Figure 6: Confusion matrix for the classification of fish species from their anatomical features predictions.
Method
linear layer
k -NN
LogReg
SVM
Baseline
80.94 ± 0.05
73.80 ± 0.02
82.69 ± 0.00
81.94 ± 0.01
With IAWA
82.05 ± 0.01
75.83 ± 0.02
83.19 ± 0.04
79.39 ± 0.06
Table 2: Specimen-level accuracy (%) on the 16-species benchmark , mean ± std over five stratified folds, with EfficientNet-V2L features at 500×500 , best per feature block in bold. Comparison of different classifiers.
Figure 7: Species-level confusion matrix for the best configuration, IAWA feature, logistic regression; 83.19±0.04% ), pooled over the five test folds. White boxes mark the families Salicaceae , Rosaceae and Ericaceae from top-left to bottom-right. The accuracy at family level accuracy is 96.18% .
Figure 8: Misclassified samples are usually associated with a higher criterion value than correctly classified once, although a few exceptions exist.
Figure 9: Awa2 dataset: minimum number of attributes to be modified for correct classification of the test set.
Figure 10: Minimal loss between each data and all classes. Misclassified specimens score higher, so the quantity can be used to select which specimens to submit to an expert.
Figure 11: Specimen-level accuracy under progressive expert correction of the predicted criteria.
Specimen
#wrong
#read
Identified as, before → after
AIERICALCRE
14
7
Erica → Calluna
✓
GEODE
16
4
Rhododendron → Calluna
✓
AIROSPRUEUR75
4
2
P. avium → P. spinosa
✓
AIROSPRUCREf
8
3
P. avium → P. persica
✓
AIERICALEX62
17
6
Crataegus → Calluna
✓
AIERICALLAGf
12
6
V. myrtillus → Calluna
✓
Table 3: Expert intervention on eight misclassified specimens. The first four are Calluna vulgaris , then Prunus spinosa , P. persica , P. avium and Sorbus aucuparia . #wrong: criteria the model read incorrectly among evaluable entries; #read: values recorded by the anthracologist, a subset of those flagged. Identification is by nearest species prototype under the mean squared error of section 5.2 .
Figure 12: Classes prediction for ’Body shape’ and ’Cross section’
Figure 13: Ranges of predicted values for the test set. The box plot are using the true ranges of values even if some class share the same intervals.
Family
Species
#Trees
Images / section
#Img
R
T
Tr
Ericaceae
Arbutus unedo
16
61
35
38
134
Calluna vulgaris
31
130
66
72
268
Erica cinerea
19
81
40
39
160
Rhodo. ferrugineum
20
67
45
44
156
Vaccinium myrtillus
20
92
49
48
189
Table 4: Dataset composition. R: radial, T: tangential, Tr: transverse. All species are hardwoods and all carry a complete expert IAWA annotation (tables 5 and 6 ).
Species
B1
B2
B3
Arbutus unedo
25, 26
40, 41
50
Calluna vulgaris
24
40, 41
50
Crataegus monogyna
24, 25
40
50
Erica cinerea
25
40, 41
49, 50
Prunus avium
25, 26
41
49, 50
Prunus persica
25, 26
40, 41
49, 50
Table 5: Expert-annotated admissible intervals for the three ordinal blocks. B1 : criteria 24–27 ; B2 : criteria 40–43 ; B3 : criteria 46–50. Full text definitions of each criteria and blocks are given in the IAWA hardwood list [ 25 ] and InsideWood [ 24 ] . Criteria declared variable are included, so each entry is the widest band consistent with the annotation. The bands overlap heavily, and several species are identical across all three blocks which is why the ordinal heads collapse onto a constant.
Table 6: Expert IAWA annotations for the 16 species, (73 binary and 13 used for the three intervals blocks table 5 ). Each entry lists the microscopic feature numbers of the IAWA hardwood list [ 25 ] recorded for that species on carbonised material. Numbers in v are declared variable within the species. Criteria absent from an entry are absent in that species.
Figure 14: Per-criterion AUC on correctly classified specimens against misclassified ones. Every scorable criterion declines; the six largest drops are highlighted and labelled by IAWA number.