Task-Specific Efficiency Analysis: When Small Language Models Outperform Large Language Models

Jinghan Cao; Yu Ma; Xinjin Li; Qingyang Ren; Xiangyun Chen

cs.CLMar 22, 2026

Task-Specific Efficiency Analysis: When Small Language Models Outperform Large Language Models

Authors: Jinghan Cao, Yu Ma, Xinjin Li, Qingyang Ren, Xiangyun Chen

Organizations: San Francisco State University - Department of Computer Science 1600 Holloway Ave, San Francisco - USA · Carnegie Mellon University - Department of Computer Science 5000 Forbes Ave, Pittsburgh - USA · Columbia University - Department of Computer Science 116th and Broadway, New York - USA · Cornell University - Department of Computer Science 616 Thurston Ave, Ithaca - USA ·

Abstract

Large Language Models achieve remarkable performance but incur substantial computational costs unsuitable for resource-constrained deployments. This paper presents the first comprehensive task-specific efficiency analysis comparing 16 language models across five diverse NLP tasks. We introduce the Performance-Efficiency Ratio (PER), a novel metric integrating accuracy, throughput, memory, and latency through geometric mean normalization. Our systematic evaluation reveals that small models (0.5--3B parameters) achieve superior PER scores across all given tasks. These findings establish quantitative foundations for deploying small models in production environments prioritizing inference efficiency over marginal accuracy gains.

Figures & tables

Figure 1: Model Accuracy Across All Tasks

Model	GSM8K					SQuAD 2.0
Model	Acc	Thr (tokens/s)	Lat (ms/token)	Mem(MB)	PER	Acc	Thr (tokens/s)	Lat (ms/token)	Mem(MB)	PER
Qwen2.5-0.5B	0.3768	7927	0.15	1024	0.4546	0.30	854	1.17	1024	0.50
Qwen2.5-1.5B	0.58	4417	0.25	3072	0.6617	0.66	353	2.83	3072	0.67
Gemma-2-2B	0.555	1761	0.66	4096	0.4887	0.68	433	2.31	4096	0.72
Qwen2.5-3B	0.699	2657	0.42	6144	0.6056	0.72	219	4.58	6144	0.53
Phi-3.5	0.7354	979	1.16	7782	0.4427	0.73	172	5.83	7782	0.45

Table 1: Generation Tasks PER Results Sorted by Model Size

Model	HellaSwag					IMDB					ARC-Easy
Model	Acc	Thr (samples/s)	Lat (ms/sample)	Mem(MB)	PER	Acc	Thr (samples/s)	Lat (ms/sample)	Mem(MB)	PER	Acc	Thr (samples/s)	Lat (ms/sample)	Mem(MB)	PER
Qwen2.5-0.5B	0.39	65	15	1024	0	0.92	193	5	1024	1.00	0.63	574	2	1024	0.80
Qwen2.5-1.5B	0.50	36	28	3072	0.63	0.89	66	15	3072	0.71	0.85	178	6	3072	0.68
Gemma-2-2B	0.52	21	48	4096	0.53	0.84	20	50	4096	0.46	0.40	87	11	4096	0
Qwen2.5-3B	0.55	22	45	6144	0.57	0.89	22	45	6144	0.48	0.88	60	17	6144	0.47
Phi-3.5	0.59	20	51	7782	0.56	0.48	6	154	7782	0	0.91	42	24	7782	0.40

Table 2: Classification Task PER Results Sorted by Model Size

Small Language Models

Small language models are compact neural language systems designed to deliver useful text and reasoning capabilities with limited compute, memory, and energy.

See all 269 papers

Profiling Lightweight Large Language Models
Jul 23, 2026Tomohiro Harada, Enrique Alba, Gabriel LuqueLLM Inference Efficiency LLM Evaluation
Graduate School of Science and Engineering, Saitama University, Saitama, Japan
AE-LLM: Adaptive Efficiency Optimization for Large Language Models
Mar 20, 2026Kaito Tanaka, Masato Ito, Yuji Nishimura +2LLM Inference Efficiency Energy-Efficient ML
SANNO University
Task-Centric Acceleration of Small-Language Models
Feb 27, 2026Dor Tsur, Sharon Adar, Ran LevyLanguage Model Vocabulary Expansion Efficient Language Model Inference
Amazon

Jul 23, 2026J/K move · Enter open · S save

Profiling Lightweight Large Language Models

Tomohiro Harada, Enrique Alba, Gabriel Luque

Graduate School of Science and Engineering, Saitama University, Saitama, Japan

LLM Inference Efficiency LLM Evaluation On-Device Language Model Inference LLM Inference Small Language Models Energy-Aware LLM Inference

Open paper AskPDF arXiv

Lightweight large language models (LLMs) are increasingly being deployed locally on personal computers and are expected to play a growing role in resource-constrained edge and mobile environments. In such settings, energy consumption, execution time, and memory usage directly affect practical usability, yet existing evaluations of LLM efficiency largely rely on proxy descriptors such as parameter count or FLOPs, often decoupled from task precision. This paper introduces a PTME-based experimental framework for the precision-aware profiling of lightweight LLM inference, jointly measuring Precision, execution Time, peak Memory usage, and Energy consumption through direct hardware-level measurements. The methodology is applied to a representative set of lightweight LLMs executed locally under edge-class resource envelopes on a controlled desktop platform, using benchmarks spanning code generation, mathematical reasoning, and multi-task understanding. We find that static proxy descriptors approximate inference cost well but fail to predict precision. Tightening the resource envelope increases cost without affecting precision, amplifying execution time more strongly than energy and penalizing larger models the most. Moreover, no single model dominates across all PTME dimensions, and a Pareto analysis reveals non-dominated configurations that would be hidden by accuracy-only or efficiency-only assessments, providing practical guidance for selecting models under different resource envelopes. These results show that selecting lightweight LLMs by size, FLOPs, latency, or accuracy alone can select the wrong deployment candidate; PTME profiling exposes configurations that preserve useful accuracy at lower physical cost.

Abstract

Small Language Models

Explore similar work

Profiling Lightweight Large Language Models

AE-LLM: Adaptive Efficiency Optimization for Large Language Models

Task-Centric Acceleration of Small-Language Models