cs.CLOct 30, 2024

Don't Pay Attention, PLANT It: Pretraining Attention via Learning-to-Rank

Authors: Debjyoti Saha Roy, Byron C. Wallace, Javed A. Aslam

Organizations: Khoury College of Computer Sciences, Northeastern University

Abstract

State-of-the-art Extreme Multi-Label Text Classification models rely on multi-label attention to focus on key tokens in input text, but learning good attention weights is challenging. We introduce PLANT (Pretrained and Leveraged AtteNTion), a plug-and-play strategy for initializing the label-attention module that operates on top of an LLM's output representations. PLANT works by planting label-specific attention using a Learning-to-Rank model trained on mutual information gain, before end-to-end fine-tuning. This architecture-agnostic approach integrates with LLM backbones (e.g., Mistral-7B, Llama-3.1-8B, DeepSeek-V3, and Phi-3-mini). PLANT outperforms SOTA methods across tasks like ICD coding, legal topic classification, and content recommendation, and improves fine-tuned LLMs in controlled comparisons outside the medical domain. Gains are largest on rare labels, although a training-free LLM extraction baseline remains competitive on rarely seen labels. Ablation studies confirm that attention initialization is a key driver of these gains. We make our code available at https://github.com/deb-official/plant

Figures & tables

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. LLM-Enhanced Dual-Branch Learning for Large-Scale Multi-Label Text Classification

    Sep 11, 2026Hui Ye, Jing Zhang, Xiulong Yang +1Late FusionMulti-Label Classification

  2. Label-Centric Knowledge Retrieval for Zero-Shot Text Classification

    Oct 12, 2026Daoping Wu, Dong Wang, Siyu Liu +5Zero-Shot Text ClassificationLarge Language Model-Based Text Classification