Don't Pay Attention, PLANT It: Pretraining Attention via Learning-to-Rank
Organizations: Khoury College of Computer Sciences, Northeastern University
Abstract
State-of-the-art Extreme Multi-Label Text Classification models rely on multi-label attention to focus on key tokens in input text, but learning good attention weights is challenging. We introduce PLANT (Pretrained and Leveraged AtteNTion), a plug-and-play strategy for initializing the label-attention module that operates on top of an LLM's output representations. PLANT works by planting label-specific attention using a Learning-to-Rank model trained on mutual information gain, before end-to-end fine-tuning. This architecture-agnostic approach integrates with LLM backbones (e.g., Mistral-7B, Llama-3.1-8B, DeepSeek-V3, and Phi-3-mini). PLANT outperforms SOTA methods across tasks like ICD coding, legal topic classification, and content recommendation, and improves fine-tuned LLMs in controlled comparisons outside the medical domain. Gains are largest on rare labels, although a training-free LLM extraction baseline remains competitive on rarely seen labels. Ablation studies confirm that attention initialization is a key driver of these gains. We make our code available at https://github.com/deb-official/plant
Figures & tables
| Model | AUC | F1 | P@15 | ||
| Macro | Micro | Macro | Micro | ||
| Mistral-7B | 90.2 | 98.7 | 20.0 | 57.0 | 53.8 |
| Mistral-7B + PLANT | 97.4 (+7.2) | 99.5 (+0.8) | 23.0 (+3.0) | 59.2 (+2.2) | 56.9 (+3.1) |
| Llama-3.1-8B | 90.5 | 98.8 | 20.5 | 57.5 | 54.0 |
| Llama-3.1-8B + PLANT | 97.6 (+7.1) | 99.6 (+0.8) | 23.5 (+3.0) | 59.5 (+2.0) | 57.0 (+3.0) |
| DeepSeek-V3 | 90.0 | 98.6 | 19.8 | 56.8 | 53.5 |
| Model | ||||||
| Macro | Micro | Macro | Micro | |||
| CoRelation ( Luo et al., 2024 ) | ||||||
| PLM-CA ( Edin et al., 2024 ) | ||||||
| GKI-ICD ( Zhang et al., 2025b ) | ||||||
| GPT-4 Zero-Shot ( Yuan et al., 2025 ) | ||||||
| PLANT (Ours) | 97.4 | 99.5 | 23.0 | 59.2 | 72.1 | 56.9 |
| Model | Legal Topic Classification ( EURLEX-4K ) | Content Recommendation ( WIKI10-31K ) | ||||
| PECOS ( Yu et al., 2022 ) | ||||||
| ICXML ( Zhu and Zamani, 2023 ) | ||||||
| XRR ( Xiong et al., 2023 ) | ||||||
| X-Transformer + RDE ( Shi et al., 2024 ) | ||||||
| MatchXML ( Ye et al., 2024 ) | ||||||
| EUR-Lex subset | Wiki10 subset | ||||||
| Backbone | P@25 | R@25 | PSP@25 | P@25 | R@25 | PSP@25 | |
| Mistral | FT | 3.2 | 31.6 | 26.2 | 14.6 | 57.5 | 52.3 |
| PLANT | 4.7 | 48.5 | 42.5 | 19.1 | 77.2 | 67.5 | |
| Llama-3.1 | FT | 3.9 | 38.1 | 31.9 | 15.0 | 62.5 | 46.5 |
| PLANT | 5.4 | 54.0 | 49.5 | 19.4 | 78.6 | 68.6 | |
| Phi-3-small | FT | 3.1 | 29.1 | 24.6 | 3.9 | 12.7 | 16.7 |
| Ablation Config | macro-AUC | macro-F1 | |
| 1 Single-Stage BCE | 91.0 ( -6.4 , [-7.2, -5.4] ) | 18.0 ( -5.0 , [-5.9, -3.8] ) | 52.0 ( -4.9 , [-5.7, -3.9] ) |
| 1 Single-Stage Focal Loss | 92.5 ( -4.9 , [-5.8, -4.0] ) | 19.5 ( -3.5 , [-4.2, -2.7] ) | 53.5 ( -3.4 , [-4.0, -2.7] ) |
| 1 PLANT w/ Vanilla BCE | 95.0 ( -2.4 , [-3.0, -1.7] ) | 21.0 ( -2.0 , [-2.6, -1.4] ) | 54.8 ( -2.1 , [-2.7, -1.5] ) |
| 2 PLANT w/o Label Smoothing | 97.0 ( -0.4 , [-0.7, -0.2] ) | 21.8 ( -1.2 , [-1.8, -0.7] ) | 55.8 ( -1.1 , [-1.7, -0.6] ) |
| 2 PLANT w/o Hard Neg Mining | 96.8 ( -0.6 , [-1.0, -0.3] ) | 21.0 ( -2.0 , [-2.7, -1.3] ) | 54.5 ( -2.4 , [-3.1, -1.7] ) |
| 1 PLANT w/ Term Frequency | 96.5 ( -0.9 , [-1.4, -0.5] ) | 20.5 ( -2.5 , [-3.2, -1.8] ) | 54.0 ( -2.9 , [-3.6, -2.1] ) |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Variant (training data) | Macro-F1 (%) | Rare-F1 (%) | Corr(F1, log-freq) | Sim( , MIG)-Rare |
| 1 Epoch Stage 1 PLANT (full data) | 20.5 0.6 | 10.3 0.9 | 0.55 0.05 | 0.35 0.07 |
| PLANT (full data) | 23.0 0.3 | 16.8 0.5 | 0.30 0.03 | 0.78 0.04 |
| Random Attn Init (common only) | 12.5 1.0 | 0.5 0.4 | 0.78 0.03 | 0.08 0.05 |
| PLANT (common only) | 18.0 0.7 | 8.0 0.6 | 0.45 0.04 | 0.52 0.07 |
| : Congestive heart failure | : Other malignant lymphomas | : Transitory tachypnea of newborn |
| DIAGNOSES: Acute congestive heart failure , Diabetes mellitus , Pulmonary edema | 55 year-old female with non Hodgkin’s lymphoma and C1 esterase inhibitor deficiency | Chest x-ray: transient tachypnea of the newborn with respiratory distress |
| MIMIC-III-full | MIMIC-IV-full | |
| Number of documents | 52,723 | 122,279 |
| Number of patients | 41,126 | 65,659 |
| Number of unique codes | 8,929 | 7,942 |
| Codes per instance: Median (IQR) | ||
| Words per document: Median (IQR) | ||
| Documents: Train/val/test [%] |
| EURLEX-4K | WIKI10-31K | |
| Number of train documents | ||
| Number of test documents | ||
| Number of unique labels | ||
| Average number of labels per instance | ||
| Average number of instances per label |
| MIMIC-III-full | MIMIC-IV-full | |||||||||
| Model | AUC | F1 | P@15 | AUC | F1 | P@15 | ||||
| Macro | Micro | Macro | Micro | Macro | Micro | Macro | Micro | |||
| Mistral-7B | 90.8 | 98.9 | 13.5 | 62.0 | 63.5 | 90.2 | 98.7 | 20.0 | 57.0 | 53.8 |
| Mistral-7B + PLANT | 98.1 (+7.3) | 99.9 (+1.0) | 14.7 (+1.2) | 64.1 (+2.1) | 65.8 (+2.3) | 97.4 (+7.2) | 99.5 (+0.8) | 23.0 (+3.0) | 59.2 (+2.2) | 56.9 (+3.1) |
| Llama-3.1-8B | 91.0 | 99.0 | 13.8 | 62.5 | 64.0 | 90.5 | 98.8 | 20.5 | 57.5 | 54.0 |
| Llama-3.1-8B + PLANT | 98.3 (+7.3) | 99.8 (+0.8) | 15.0 (+1.2) | 64.5 (+2.0) | 66.2 (+2.2) | 97.6 (+7.1) | 99.6 (+0.8) | 23.5 (+3.0) | 59.5 (+2.0) | 57.0 (+3.0) |
| MIMIC-III-full | MIMIC-III-top50 | ||||||||||
| Model | |||||||||||
| Macro | Micro | Macro | Micro | Macro | Micro | Macro | Micro | ||||
| Effective-CAN liu et al. [2021] | 92.1 | 98.9 | 10.6 | 58.9 | 75.8 | 60.6 | 92.0 | 94.5 | 66.8 | 71.7 | 66.4 |
| MSMN Yuan et al. [2022] | 95.0 | 99.2 | 10.3 | 58.4 | 75.2 | 59.9 | 92.8 | 94.7 | 68.3 | 72.5 | 68.0 |
| PLM-ICD [ Huang et al., 2022 ] | 92.6 | 98.9 | 10.4 | 59.8 | 77.1 | 61.3 | 91.0 | 93.4 | 66.3 | 71.9 | 66.0 |
| Contrastive + JointLAAT [ Lu et al., 2023 ] | 94.1 | 98.8 | 11.5 | 58.3 | 73.9 | 59.4 | 91.3 | 93.7 | 67.2 | 72.0 | 67.9 |
| Model | AUC | F1 | P@15 / PSP@15 | |||
| Macro | Micro | Macro | Micro | P | PSP | |
| Mistral-7B | 90.2 | 98.7 | 20.0 | 57.0 | 53.8 | 21.5 |
| Mistral-7B + PLANT | 97.4 (+7.2) | 99.5 (+0.8) | 23.0 (+3.0) | 59.2 (+2.2) | 56.9 (+3.1) | 24.8 (+3.3) |
| Llama-3.1-8B | 90.5 | 98.8 | 20.5 | 57.5 | 54.0 | 21.6 |
| Llama-3.1-8B + PLANT | 97.6 (+7.1) | 99.6 (+0.8) | 23.5 (+3.0) | 59.5 (+2.0) | 57.0 (+3.0) | 24.9 (+3.3) |
| DeepSeek-V3 | 90.0 | 98.6 | 19.8 | 56.8 | 53.5 | 21.4 |
| MIMIC-III-few | MIMIC-III-rare50 | |||||||||
| Model | ||||||||||
| Macro | Micro | Macro | Micro | Macro | Micro | Macro | Micro | Macro | Micro | |
| AGMHT [ Song et al., 2021 ] | 18.7 | 29.2 | 17.6 | 49.4 | 19.9 | 20.7 | – | – | – | – |
| KEPTLongformer [ Yang et al., 2022b ] | – | – | – | – | – | – | 82.7 | 83.3 | 30.4 | 32.6 |
| MSMN [ Yuan et al., 2022 ] | 4.3 | 8.5 | 4.5 | 70.9 | 4.2 | 4.5 | – | – | – | – |
| GP [ Yang et al., 2023b ] | 30.2 | 35.3 | 27.9 | 38.5 | 32.9 | 32.6 | – | – | – | – |
| Ablation Config | macro-AUC | macro-F1 | |
| Dataset: MIMIC-III-full , LLM: Mistral-7B | |||
| 1 Single-Stage BCE | 92.0 ( -6.1 , [-6.9, -5.1] ) | 10.0 ( -4.7 , [-5.6, -3.3] ) | 60.0 ( -5.8 , [-6.6, -4.7] ) |
| 1 Single-Stage Focal Loss | 93.5 ( -4.6 , [-5.5, -3.7] ) | 11.5 ( -3.2 , [-4.1, -2.1] ) | 61.5 ( -4.3 , [-5.2, -3.1] ) |
| 1 PLANT w/ Vanilla BCE | 95.5 ( -2.6 , [-3.3, -1.9] ) | 12.7 ( -2.0 , [-2.6, -1.3] ) | 63.0 ( -2.8 , [-3.4, -2.1] ) |
| 2 PLANT w/o Label Smoothing | 97.8 ( -0.3 , [-0.6, -0.1] ) | 13.8 ( -0.9 , [-1.4, -0.4] ) | 64.5 ( -1.3 , [-1.9, -0.7] ) |
| 2 PLANT w/o Hard Neg Mining | 97.5 ( -0.6 , [-1.0, -0.2] ) | 13.0 ( -1.7 , [-2.3, -1.1] ) | 62.5 ( -3.3 , [-4.0, -2.4] ) |
| Dataset | Metric | Mistral-7B | Llama-3.1-8B | Phi-3-small | |||
| FT | PLANT | FT | PLANT | FT | PLANT | ||
| EURLEX-4K (sample) | P@25 | 0.0323 | 0.0465 | 0.0385 | 0.0540 | 0.0308 | 0.0428 |
| R@25 | 0.3164 | 0.4850 | 0.3805 | 0.5401 | 0.2906 | 0.3900 | |
| PSP@25 | 0.2617 | 0.4254 | 0.3189 | 0.4952 | 0.2463 | 0.3968 | |
| F1 (Micro) | 0.0352 | 0.0623 | 0.0272 | 0.0531 | 0.0348 | 0.1084 | |
| WIKI10-31K (sample) | P@5 | 0.3988 | 0.5988 | 0.4065 | 0.5801 | 0.0591 | 0.4711 |
| Backbone | Method | Zero-shot P@1 | Few-shot P@5 | Head P@8 | Overall P@15 |
| Mistral-7B | Fine-tuned baseline | 0.0055 | 0.0545 | 0.2766 | 0.2221 |
| Training-free, zero-shot prompt | 0.2754 | 0.1485 | 0.2964 | 0.2240 | |
| Training-free, two-shot prompt | 0.2575 | 0.1545 | 0.2927 | 0.2271 | |
| PLANT | 0.3087 | 0.1160 | 0.3083 | 0.2460 | |
| Llama-3.1-8B | Fine-tuned baseline | 0.0075 | 0.0087 | 0.1742 | 0.0985 |
| Training-free, zero-shot prompt | 0.2275 | 0.1389 | 0.2762 | 0.2104 |
| Backbone | Dataset | Training Time (Wall-Clock Hours) | Total | |
| Stage 1 (MIG + L2R, 10 epochs) | Stage 2 (End-to-End, up to 20 epochs) | |||
| Mistral-7B | MIMIC-III | 2.1 | 8.4 | 10.5 |
| MIMIC-IV | 4.2 | 18.7 | 22.9 | |
| EUR-LEX | 1.4 | 3.2 | 4.6 | |
| WikiTen | 1.3 | 2.9 | 4.2 | |
| Llama-3.1-8B | MIMIC-III | 2.2 | 9.1 | 11.3 |
| Backbone | Dataset | Stage 1 (MIG + L2R) | Stage 2 (End-to-End) |
| Mistral-7B | MIMIC-III | 12.4 | 28.7 |
| MIMIC-IV | 12.8 | 29.4 | |
| EUR-LEX | 11.9 | 27.2 | |
| WikiTen | 11.7 | 26.9 | |
| Llama-3.1-8B | MIMIC-III | 12.4 | 30.2 |
| MIMIC-IV | 12.8 | 30.9 |
| Backbone | MIMIC-III (1,375 words) | MIMIC-IV (1,492 words) | EUR-LEX | WikiTen |
| Mistral-7B | 1.8 | 1.9 | 0.7 | 1.0 |
| Llama-3.1-8B | 1.9 | 2.0 | 0.8 | 1.1 |
| DeepSeek-V3 | 4.2 | 4.5 | 1.6 | 2.3 |
| Phi-3-mini | 1.2 | 1.3 | 0.4 | 0.7 |
| Backbone | Training Time (Hours): Vanilla | Training Time (Hours): PLANT (Total) | Inference Time (s/doc): Vanilla vs. PLANT |
| Mistral-7B | 18.7 | 22.9 (+22.5%) | 1.7 vs. 1.9 (+11.8%) |
| Llama-3.1-8B | 20.2 | 24.5 (+21.3%) | 1.8 vs. 2.0 (+11.1%) |
| DeepSeek-V3 | 63.4 | 70.9 (+11.8%) | 4.1 vs. 4.5 (+9.8%) |
| Phi-3-mini | 11.6 | 14.8 (+27.6%) | 1.2 vs. 1.3 (+8.3%) |
| Avg. Overhead | — | +20.8% | +10.3% |
Multi-Label Classification
Multi-label classification assigns several potentially related tags to each example, from medical findings and topics to objects and user interests.
See all 150 papers