Paper ID: 2301.07006

Which Model Shall I Choose? Cost/Quality Trade-offs for Text Classification Tasks

Shi Zong, Josh Seltzer, Jiahua, Pan, Kathy Cheng, Jimmy Lin

Industry practitioners always face the problem of choosing the appropriate model for deployment under different considerations, such as to maximize a metric that is crucial for production, or to reduce the total cost given financial concerns. In this work, we focus on the text classification task and present a quantitative analysis for this challenge. Using classification accuracy as the main metric, we evaluate the classifiers' performances for a variety of models, including large language models, along with their associated costs, including the annotation cost, training (fine-tuning) cost, and inference cost. We then discuss the model choices for situations like having a large number of samples needed for inference. We hope our work will help people better understand the cost/quality trade-offs for the text classification task.

Submitted: Jan 17, 2023