When the tail is earned by its companions: measuring label dependence in long-tailed multi-label classification
Organizations: School of Computer Engineering, Pontificia Universidad Católica de Valparaíso, Av. Brasil 2950, Valparaíso, 2340025, Valparaíso, Chile
Abstract
A rare label seldom occurs alone, and a classifier can recover it from the frequent labels it co-occurs with, its companions, without having learned the label itself. Existing diagnostics of this need a trained model: none says before training whether a corpus admits the failure, or what the standard aggregate hides. Macro-F1, like any measure built from per-label confusion counts, is unchanged by how a label's recall divides between occurrences with and without a companion. We introduce a within-label estimand of that division, a reading rule under which a fall in dependence counts only if companion-free recall rises, and a statistic of training labels that bounds, before training, how far reported recall can exceed companion-free recall. Across 23 corpora, 7 training objectives and five families of label-correlation mechanisms, companion-free recall falls short on all six corpora where the estimand is defined. Up to 18.4% of the tail true positives macro-F1 credits would vanish at the companion-free rate, within the training-label bound on every corpus. No long-tail objective reduces the dependence, but architecture moves it. Models whose tail score is computed from other labels' predictions are strongly dependent.
Figures & tables
| Term | Meaning |
| companion | a frequent (head) label that co-occurs with tail label above a threshold, estimated on training labels (Definition 1 ) |
| supported, independent | an occurrence of that does, or does not, carry a member of (Definition 2 ) |
| support ratio | the share of ’s occurrences that are supported; is its mean over the tail |
| , | recall on supported and on independent occurrences |
| within-label effect | the mean over estimable tail labels of (Equation 11 ) |
| inflation | , how far reported recall exceeds independent recall; bounded by |
| Corpus | Domain; vocabulary | train | val | test | card. | tokens | Example tail label: companions ( ) | ||
| EURLEX57K | EU legislation; EUROVOC thesaurus | 45,000 | 6,000 | 6,000 | 4,271 | 5.07 | 2,136 | 350 | Friuli-Venezia Giulia : State aid (1/4) |
| PubMed MeSH | PubMed abstracts; MeSH | 40,000 | – | 10,000 | 21,918 | 12.69 | 10,959 | 206 | Galactosyltransferases : Animals; RNA, Messenger (5/3) |
| Ohsumed | MEDLINE abstracts; MeSH | 54,709 | – | 293,855 | 14,623 | 10.32 | 7,312 | 172 | knowledge, attitudes, practice : adult; family planning; infant, newborn; sex behavior (85/74) |
| Wiki10-31K | Wikipedia articles; social tags | 14,146 | – | 6,616 | 30,938 | 18.64 | 15,469 | 2052 | exercises : dynamics; e; education; effective; +44 more (4/1) |
| AmazonCat-13K | product descriptions; retail taxonomy | 1,186,239 | – | 306,782 | 13,330 | 5.04 | 6,665 | 210 | ages 9-12 : by age; children’s; magazine subscriptions; entertainment (15/1) |
| LexGLUE EURLEX | EU legislation; 100 EUROVOC concepts | 55,000 | 5,000 | 5,000 | 100 | 4.42 | 50 | 1871 | 2441 budget : 1021 EU finance (96/95) |
| Family | Arm | Input at inference | Tail score | Control |
| Graph (adjusting) | identity c | document | , from a label embedding, | – |
| co-occurrence | document | from a two-layer GCN over | identity | |
| GATv2 | document | as above, second layer attention over ’s edges | identity | |
| dynamic | document, companions’ attention vectors | GCN | identity | |
| Correlation network (adjusting) | diagonal c | own pred. | – | |
| frozen c | all pred. | , random and frozen | – |
| Within-label estimand | Difference of F1 | ||||||
| Corpus | 95% CI | gap | null | ||||
| EURLEX57K | 0.262 | 35 | % | % | |||
| PubMed MeSH | 0.401 | 357 | % | % | |||
| Wiki10-31K | 0.241 | 260 | % | % | |||
| Ohsumed | 0.147 | 1,986 | % | % | |||
| AmazonCat-13K | 0.281 | 910 | % | % | |||
| Ohsumed | EURLEX57K | Wiki10-31K | ||||
| Objective | ||||||
| focal | w † | |||||
| asymmetric | ||||||
| class-balanced | w † | |||||
| distribution-balanced | w | d † | ||||
| logit adjustment | w* † | |||||
| Measured on test | Exact bound | Known before training | ||||
| Corpus | inflation | on | on | |||
| EURLEX57K | 470 | 0.036 | 0.354 | 0.411 | 23% | |
| PubMed MeSH | 3,133 | 0.059 | 0.404 | 0.516 | 44% | |
| Ohsumed | 6,732 | 0.189 | 0.540 | 0.570 | 47% | |
| Wiki10-31K | 8,342 | 0.017 | 0.082 | 0.532 | 53% | |
| AmazonCat-13K | 5,640 | 0.098 | 0.149 | 0.254 | 11% | |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Corpus | Encoder | Length | Epochs | Objectives | Tail F1 (BCE) |
| EURLEX57K | Legal-BERT | 512 | 8 | 7 | |
| PubMed MeSH | BiomedBERT | 256 | 5 | 2 | |
| Ohsumed | BiomedBERT | 256 | 5 | 7 | |
| Wiki10-31K | BERT-base | 512 | 10 (20) | 7 | |
| AmazonCat-13K | BERT-base | 256 | 6 (12) | 1 | |
| LexGLUE EURLEX | Legal-BERT | 512 | 8 | 1 |
| Objective | Ohsumed | EURLEX57K | Wiki10-31K |
| focal | † | ||
| asymmetric | |||
| class-balanced | † | ||
| distribution-balanced | † | † | |
| logit adjustment | † | ||
| LDAM |
| over | outside | over | |||||
| Corpus | absolute | relative | excess | labels | share | absolute | |
| EURLEX57K | % | 146 | 24% | 35 | |||
| PubMed MeSH | % | 1,545 | 33% | 357 | |||
| Ohsumed | % | 420 | 6% | 1,986 | |||
| Wiki10-31K | % | 2,285 | 22% | 260 | |||
| AmazonCat-13K | % | 970 | 15% | 910 | |||
| Corpus | Arm | effect | sd | |
| EURLEX57K — graph | ||||
| identity c | 0.661 | 0.131 | – | |
| co-occurrence | 0.729 | 0.109 | 0.9 | |
| GATv2 | 0.732 | 0.108 | 0.9 | |
| dynamic | 0.960 | 0.175 | 3.0 | |
| EURLEX57K — correlation network | ||||
| Corpus | card. | cluster | ||||
| RCV1-V2 full | 23,149 | 749 | 5.12 | 0.621 | 1.76 | rcv1v2 |
| Ohsumed | 54,709 | 14,623 | 10.32 | 0.570 | 3.01 | mesh |
| Wiki10-31K | 14,146 | 30,938 | 18.64 | 0.532 | 14.11 | wiki10_tags |
| MESINESP2 | 249,474 | 22,658 | 8.32 | 0.531 | 3.28 | mesh |
| PubMed MeSH | 40,000 | 21,918 | 12.69 | 0.516 | 5.99 | mesh |
| EURLEX-4K | 15,449 | 3,956 | 5.32 | 0.420 | 2.06 | eurlex |
| Corpus | |||
| Ohsumed | |||
| 1986 | 1569 | 793 | |
| EURLEX57K | – | – | |
| 35 | – | – | |
| Wiki10-31K | – | ||
| 260 | 7 | – |
| Corpus | reported | min | max | cells |
| Ohsumed | 10/15 | |||
| EURLEX57K | 13/15 | |||
| Wiki10-31K | 11/15 | |||
| PubMed MeSH | 13/15 | |||
| AmazonCat-13K | 15/15 | |||
| LexGLUE EURLEX | 8/15 |
Multi-Label Classification
Multi-label classification assigns several potentially related tags to each example, from medical findings and topics to objects and user interests.
See all 148 papers