A lot of expectations have been put for years on integrating domain expert knowledge in image classification models. Several approaches have been explored, Concept Bottleneck Models (CBMs) opened up a new avenue of research leading to many variants, and more recently to Concept-based Embedding Models (CEMs). CBM consider binary encoding of each concept, while CEM expands this idea by embedding each concept through two vectors. However in real-life scenarii, domain experts' knowledge is usually organized in concepts determined by various attributes, each attribute encoded either with numerical values, or range of values, or binary values, or categorical values. In this work, we first finetune an image feature extractor for classifying attributes representing the downstream object classes, where the class attributes have been specified by experts under various encoding formats. A classification head is then learnt from these various attributes to categorize target objects. We experimentally show that it improves the classification for three datasets: Kaggle fish dataset, AWA2 and a more challenging new wood charcoal dataset. We then propose an automatic selection of potential missclassified data. In this second step, experts are asked for those data to eventually modify the predicted attributes in order to improve the classification.
Figures & tables
Figure 1: The global picture of this paper. The data are single images or set of images. We extract features for the attributes prediction. Then, classes are predicted from attributes.
Name
Dorsal spines
Dorsal soft rays
Anal spines
Anal soft rays
Body shape
Cross section
Bangus
2
13-17
2
8-10
e
ov
Big Head
3
7
1-3
12-14
f
ov
Carp
Fourfinger
9
13-15
3
14-16
e
co
Threadfin
Gourami
16-20
7-10
9-11
8-11
f
co
Table 1: Table of attributes available for each fish specie. Four first attributes are values or range of values. Body shapes are categorical variables: elongated (e), fusiform/normal (f) or cylindrical (c), mapped to [0,1,2]. Cross sections are also categorical: oval(ov), compressed (co), circular (ci) or other (ot), mapped to [0,1,2,3].
Figure 2: Example of the IAWA feature “intervessel pit size” with 4 different ordered values as described in section 2.3 . The unit interval is divided in 4 equal parts. Left (l) and right (r) values from equation 1 depend on the present value(s), represented here by coloured boxes: IAWA #25 and #26 are always present for the class considered, so that left value is 0 and right value is 0.5.
Figure 3: The different interval values functions concerning the ’intervessel pit size’ for the different species. In this case, many classes share the same function for this anatomical attribute because they share the same intervals. The goal of this loss function is to be minimal in the correct interval.
Figure 4: The interval values functions “Dorsal soft rays” for different species. Values for “Dorsal soft rays” have been rescaled between 0 and 1. The goal of this loss function is to be minimal in the correct interval.
Figure 5: The vector of anatomy of fishes: 4 float values (rescaled between 0 and 1) and 2 sets of probabilities for “Body shape” and “Cross section” types.
Figure 6: Confusion matrix for the classification of fish species from their anatomical features predictions.
Method
linear layer
k -NN
LogReg
SVM
Baseline
80.94 ± 0.05
73.80 ± 0.02
82.69 ± 0.00
81.94 ± 0.01
With IAWA
82.05 ± 0.01
75.83 ± 0.02
83.19 ± 0.04
79.39 ± 0.06
Table 2: Specimen-level accuracy (%) on the 16-species benchmark , mean ± std over five stratified folds, with EfficientNet-V2L features at 500×500 , best per feature block in bold. Comparison of different classifiers.
Figure 7: Species-level confusion matrix for the best configuration, IAWA feature, logistic regression; 83.19±0.04% ), pooled over the five test folds. White boxes mark the families Salicaceae , Rosaceae and Ericaceae from top-left to bottom-right. The accuracy at family level accuracy is 96.18% .
Figure 8: Misclassified samples are usually associated with a higher criterion value than correctly classified once, although a few exceptions exist.
Figure 9: Awa2 dataset: minimum number of attributes to be modified for correct classification of the test set.
Figure 10: Minimal loss between each data and all classes. Misclassified specimens score higher, so the quantity can be used to select which specimens to submit to an expert.
Figure 11: Specimen-level accuracy under progressive expert correction of the predicted criteria.
Specimen
#wrong
#read
Identified as, before → after
AIERICALCRE
14
7
Erica → Calluna
✓
GEODE
16
4
Rhododendron → Calluna
✓
AIROSPRUEUR75
4
2
P. avium → P. spinosa
✓
AIROSPRUCREf
8
3
P. avium → P. persica
✓
AIERICALEX62
17
6
Crataegus → Calluna
✓
AIERICALLAGf
12
6
V. myrtillus → Calluna
✓
Table 3: Expert intervention on eight misclassified specimens. The first four are Calluna vulgaris , then Prunus spinosa , P. persica , P. avium and Sorbus aucuparia . #wrong: criteria the model read incorrectly among evaluable entries; #read: values recorded by the anthracologist, a subset of those flagged. Identification is by nearest species prototype under the mean squared error of section 5.2 .
Figure 12: Classes prediction for ’Body shape’ and ’Cross section’
Figure 13: Ranges of predicted values for the test set. The box plot are using the true ranges of values even if some class share the same intervals.
Family
Species
#Trees
Images / section
#Img
R
T
Tr
Ericaceae
Arbutus unedo
16
61
35
38
134
Calluna vulgaris
31
130
66
72
268
Erica cinerea
19
81
40
39
160
Rhodo. ferrugineum
20
67
45
44
156
Vaccinium myrtillus
20
92
49
48
189
Table 4: Dataset composition. R: radial, T: tangential, Tr: transverse. All species are hardwoods and all carry a complete expert IAWA annotation (tables 5 and 6 ).
Species
B1
B2
B3
Arbutus unedo
25, 26
40, 41
50
Calluna vulgaris
24
40, 41
50
Crataegus monogyna
24, 25
40
50
Erica cinerea
25
40, 41
49, 50
Prunus avium
25, 26
41
49, 50
Prunus persica
25, 26
40, 41
49, 50
Table 5: Expert-annotated admissible intervals for the three ordinal blocks. B1 : criteria 24–27 ; B2 : criteria 40–43 ; B3 : criteria 46–50. Full text definitions of each criteria and blocks are given in the IAWA hardwood list [ 25 ] and InsideWood [ 24 ] . Criteria declared variable are included, so each entry is the widest band consistent with the annotation. The bands overlap heavily, and several species are identical across all three blocks which is why the ordinal heads collapse onto a constant.
Table 6: Expert IAWA annotations for the 16 species, (73 binary and 13 used for the three intervals blocks table 5 ). Each entry lists the microscopic feature numbers of the IAWA hardwood list [ 25 ] recorded for that species on carbonised material. Numbers in v are declared variable within the species. Criteria absent from an entry are absent in that species.
Figure 14: Per-criterion AUC on correctly classified specimens against misclassified ones. Every scorable criterion declines; the six largest drops are highlighted and labelled by IAWA number.
A popular route to interpretable zero-shot classification asks a large language model (LLM) to describe each class name and prompts CLIP with the resulting descriptors. We show that these descriptors carry little visual evidence of their own: removing the class name from the prompt collapses ImageNet accuracy from 59.5% to 15.5%. The diagnosis is that the descriptors are conditioned on the label rather than on the images, so they describe the concept in general and mislead exactly when the data shifts; an LLM insists that strawberries are red, but every strawberry in ImageNet-Sketch is a colorless line drawing. We therefore select attributes from the target image collection instead: we score a large attribute pool against the images in CLIP's joint embedding space and keep the top-scoring attributes per class. Selected this way, class-name-free attribute prompts reach 23.8% on ImageNet (against 15.5% for LLM descriptors), the gain holds on four shifted ImageNet variants, and reselecting from the LLM's own pool isolates the selection mechanism as the cause. With one image per class, the selected attributes outperform the prompt-tuning method CoOp by 3 points while fitting in under a minute instead of 14 hours, with no learned soft prompt to obscure the decision. Because the attribute set is chosen by the data, it doubles as a readable summary of a dataset, which we use to describe distribution shift in words. Our code and results are available on our project page: https://ggare-cmu.github.io/AttributeSelect/
Long-tailed classification poses a reliability challenge because models trained on imbalanced data are unevenly reliable across frequent and underrepresented classes. While existing methods address imbalance through re-balancing, adjustment, representation learning, or multi-expert modeling, they rarely estimate which expert should be trusted for each class. This paper proposes CLEAR (Class-wise reLiability-aware Expert Aggregation for long-tailed Recognition), a modular ensemble framework for long-tailed classification. CLEAR generates diverse experts through threshold-based structured sampling while preserving the full label space, then estimates a class-wise trust score for each expert using a smoothed class-wise precision formulation. During inference, expert predictions are combined through class-wise generalized product-of-experts aggregation, allowing different experts to be emphasized for different classes. Experiments on CIFAR-100-LT, ImageNet-LT, and Places-LT across multiple backbones show that CLEAR achieves competitive overall accuracy and particularly strong few-shot performance. These results support class-wise expert reliability as a useful design principle for long-tailed ensemble learning.
Gawon Lim
School of Information Sciences University of Illinois Urbana-Champaign
Ensuring fairness in image classification prevents models from perpetuating and amplifying bias. Concept bottleneck models (CBMs) map images to high-level, human-interpretable concepts before making predictions via a sparse, one-layer classifier. This structure enhances interpretability and, in theory, supports fairness by masking sensitive attribute proxies such as facial features. However, CBM concepts have been known to leak information unrelated to concept semantics and early results reveal only marginal reductions in gender bias on datasets like ImSitu. We propose three bias mitigation techniques to improve fairness in CBMs: 1. Decreasing information leakage using a top-k concept filter, 2. Removing biased concepts, and 3. Adversarial debiasing. Our results outperform prior work in terms of fairness-performance tradeoffs, indicating that our debiased CBM provides a significant step towards fair and interpretable image classification.
Schrasing Tong, Antoine Salaun, Vincent Yuan +2
Massachusetts Institute of Technology, United States