cs.CLOct 16, 2026

Best of Both Worlds: Fine-Tuned Small Models Complement LLMs for Cost and Performance Benefits

Authors: Shivam Mathur, Sri Harish Popuri, Priya Krishnan Sundararajan, Kevin Horecka, Michael Mugo

Organizations: Walmart Inc.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

BERT-based models require thousands of labeled samples to train. Larger models like GPT-X require only 1-5 examples per class. However, performance, cost and time tradeoffs of such models remain under-explored. In this work, we classify survey text data into 16 classes. We focus on experimental approaches and results on performance, cost, and time taken for this problem. We compare these parameters of a trained RoBERTa-based model, and an (off-the-shelf) GPT-3.5 model for text classification. Baseline experiments show that the time taken by GPT-3.5 for generating predictions on 10,000 records is 6 hours, whereas a RoBERTa-based model takes only 12 mins on a T4 GPU. To identify the best prompting approach for GPT-3.5, we explore batch prompting. We experiment with (1) batch-size, and (2) few-shot sampling. We find the best batch size is 20 with human few-shot curation. Experiments show the F1 score of a trained RoBERTa-based model (96%) is much higher than a GPT-3.5 model (83%). In case of GPT-3.5, batch prompting is beneficial in lowering inference costs by ~70%, without hindering task performance. We find the best approach is an ensemble of a RoBERTa-based and GPT-3.5 model (F1: 97.5%) incurring only 4.4% of inference cost compared to a standalone GPT-3.5.

Figures & tables

ModelF1TimeTotal Cost
  traininf. 
RoBERTa961061211K
GPT–3.5
Zero-shot79N/A34016
Few-shot79N/A37144
Table 1: Baseline models with F1-score, training and prediction (inference) time (minutes), and total cost ($). Training is done on 21,000 samples and prediction is done on 10,000 samples. Cost is an estimate of human annotation effort involved in the annotation task. We use Manthano.ai (a free tool) for human annotation cost estimations (in 2023). Costs for GPT–3.5 are only inference costs (no human annotation needed).
Figure 1: Variations in total time (minutes) and cost ($) observed across experiments with batch sizes ranging from 10 to 50 (fewshot kept constant). X-axis labels are annotated as batch_[batch size]_[few-shot count] . Larger batch sizes are more cost-effective but take longer time (in most cases).
Figure 2: Variations in total cost ($) and failure (%) observed across experiments with batch sizes ranging from 10 to 50 (fewshot kept constant). X-axis labels are annotated as batch_[batch size]_[few-shot count] . Larger batch sizes are more cost-effective (similar to Figure 1 ) but lead to higher failure rate.
Batch Size1020304050
F1 (%)8282828283
Time (min)138151181171168
Cost ($)74.84.13.73.4
Failure (%)2.93.86.3841.5
Table 2: Batch size vs. F1, time, cost and failure %. Optimal batch size is 20 which keeps time, cost and failures rate low while maintaining the F1 score. It is also 70% more cost-effective compared to the GPT–3.5 zero-shot baseline ($16 in Table 1 ).
Few-shot #13579
Random-selected
F1 (%)8182838383
Time (min)159172190205201
Cost ($)3.74.86.17.18.3
Failure %4.55.04.14.05.3
Human-curated
Table 3: Few-shot size vs F1, time, cost & failure %. Human-curated few-shot examples are more cost-effective (due to shorter token counts (Table 4 )).
 Mean token count 
SelectionPromptGenerationReasoningCost
Few-shot 1
Random1059.41043.218.43.7
Human910.41033.718.23.5
Few-shot 3
Random1794.41053.919.04.9
Table 4: Cost ($) and token count comparison of random-selected and human-curated few-shot experiments in various stages (prompt, generation and reasoning). Human-curated experiments are more cost efficient. This table complements Table 3 .
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Error (%)ExampleGPT–3.5RoBERTa/Gold
Compound (23%)Train on the equipmentEquipmentDevelopment
Domain / Short Text (35%)SnackNo ComplaintBenefits
Granularity (42%)Info on hours in the assigned deptCommunicateShifts
Appendix
Table 5: Analysis of GPT–3.5 errors compared to correct RoBERTa-based and Gold on a random sample of 100 instances. Examples are paraphrased and hence synthetic. This table complements Appendix A .
More in this topic

Text Classification

Text classification assigns text to categories such as topic, sentiment, intent, stance, or risk, supporting search, moderation, and decisions.

See all 217 papers

Explore similar work

CardsList

  1. The Embedder's Dilemma: LLMs Are Better, but at What Cost?

    Aug 13, 2026Adnan El Assadi, Niklas Muennighoff, Jinhyuk LeeLLM Inference EfficiencyLLM Evaluation

    Harvard University · Stanford University · Independent Researcher

  2. Poodle: Seamlessly Scaling Down Large Language Models with Just-in-Time Model Replacement

    Dec 5, 2025Nils Strassenburg, Boris Glavic, Tilmann RablCost-Aware InferenceSmall Language Models

    Hasso Plattner Institute, Uni Potsdam · University of Illinois Chicago

  3. TextReasoningBench: Does Reasoning Really Improve Text Classification in Large Language Models?

    Mar 20, 2026Xinyu Guo, Yazhou Zhang, Jing QinLLM EvaluationEfficient Language Model Reasoning

    School of Computer Science and Technology, Tianjin University, China · School of Nursing, The Hong Kong Polytechnic University, HongKong