Task-Centric Acceleration of Small-Language Models

Dor Tsur; Sharon Adar; Ran Levy

cs.CLFeb 27, 2026

Task-Centric Acceleration of Small-Language Models

Authors: Dor Tsur, Sharon Adar, Ran Levy

Organizations: Amazon

Abstract

Small language models (SLMs) have emerged as efficient alternatives to large language models for task-specific applications. However, they are often employed in high-volume, low-latency settings, where efficiency is crucial. We propose TASC, Task-Adaptive Sequence Compression, a framework for SLM acceleration comprising two use-cases: When performing SLM fine-tuning, we propose TASC-ft, which iteratively enriches the tokenizer vocabulary with high-frequency output n-grams and then fine-tunes the model to utilize the expanded vocabulary. Next, we propose an inference-time method, termed TASC-spec. TASC-spec is a lightweight, training-free speculative decoding method that constructs an n-gram draft model from the task's output corpus, mixing task and context n-gram information.TASC-spec avoids any additional training, while bypassing draft-target vocabulary alignment constraints. We demonstrate the effectiveness of both methods across multiple low output-variability generation tasks. Our methods show consistent improvements in inference efficiency while maintaining task performance.

Figures & tables

Figure 1: Proposed method for a given task corpus. TASC-ft (upper) augments the model’s tokenizer with $n$ -gram tokens and trains their embeddings during the model’s fine-tune. TASC-spec (lower) accelerates the model in inference-time using an $n$ -gram model drafter, providing significant speedup with no additional training.

	Bigram Entropy			# Unique Bigrams For $80\%$ CDF
Task	Input	Output	$\Delta H$	Input	Output	Ratio
Sentiment Analysis (Finance)	$14.5$	$2.21$	$\downarrow-84.7\%$	$33.53K$	$4$	$8383$
Yes/No Questions (Medical)	$15.34$	$3$	$\downarrow-80.48\%$	$17.01K$	$3$	$5698$
POS Tagging	$15.53$	$6.57$	$\downarrow-57.7\%$	$65.3K$	$63$	$1036.3$
Intent Classification & Slot Fill	$13.4$	$9.74$	$\downarrow-37.45\%$	$15.943K$	$774$	$20.6$
Massive Classification (Legal)	$14.43$	$12.76$	$\downarrow-11.6\%$	$36.8K$	$8.34K$	$4.41$

Table 1: Word bigram distribution analysis across several generation tasks. We compare the Shannon entropy and the number of unique word bigrams that cover

80\%

of the bigram empirical distribution on an empirical dataset. The specific dataset names for each task is given in Appendix B .

Figure 2: Effect of added tokens on the tokenized output texts in the EUR-LEX dataset, $n_{\mathsf{max}}=4$ .

# Added Tokens	$0$	$150$	$300$	$500$	$750$	$1000$
Judge Score	$3.53\pm 0.73$	$3.21\pm 0.79$	$3.15\pm 0.82$	$3.23\pm 0.77$	$3.25\pm 0.76$	$3.18\pm 0.79$

Table 2: LLM judge results vs. Number of added tokens, TASC-ft , Asclepius dataset.

Panel of Figure 3 — Figure 3: TASC-ft : Task performance metric vs. average runtime vs. Number of Added tokens. Each dataset has a different metric according to its task: F1 for EUR-LEX , Bert-score for Asclepius and join accurcay for MASSIVE .

Figure 4: Acceptance rate vs. speculated token position.

Dataset	Method	Speedup
Asclepius	$p_{\mathsf{corpus}}$	$\times 1.63$
	TASC-spec (ours)	$\mathbf{\times 3.15}$
	N-Grammys (bigram)	$\times 1.03$
	N-Grammys (mixed)	$\times 3.08$
MedQuaD	$p_{\mathsf{corpus}}$	$\times 1.52$
	TASC-spec (ours)	$\mathbf{\times 1.78}$

Table 3: Target speedup results in several datasets. TASC-spec is employed with

\lambda=0.75

Task	Dataset Name
Sentiment Analysis (Finance)	Malo et al. (2014)
Yes/No Questions (Medical)	Kweon et al. (2024)
POS Tagging	Silveira et al. (2014)
Intent Classification & Slot Fill	FitzGerald et al. (2023)
Massive Classification (Legal)	Chalkidis et al. (2019)
Abbreviation Expansion (Medical)	Kweon et al. (2024)

Table 4: Information on the datasets used to generate Table 1 .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Precision-recall trade off for the ICSF task. Each marker denotes a different number of newly added token $n$ -grams to the tokenizer vocabulary prior to fine-tuning.

Panel of Figure 6 — Figure 6: Precision-recall trade off for the EUR-LEX task. Each marker denotes a different number of newly added token $n$ -grams to the tokenizer vocabulary prior to fine-tuning.

Number of tokens	Intent accuracy ( $\uparrow$ )	Slot F1 ( $\uparrow$ )	Joint accuracy ( $\uparrow$ )	Average sample runtime [ms] ( $\downarrow$ )
0	90.4	76.45	63.0	302
250	91.5	77.0	64.6	260
500	91.3	77.0	64.3	258.9
1000	91.1	76.2	63.4	256

Appendix

Table 5: TASC-ft results on the intent classification and slot fill MASSIVE FitzGerald et al. (2023) dataset.

Method	TASC-ft	TASC-spec
Speedup	$\times 2.1$	$\times 2.17$

Appendix

Table 6: Comparison of TASC-ft and TASC-spec .

EUR-LEX
# Train Epochs	F1	Avg. runtime
$1$	$0.624$	$0.174$
$3$	$0.647$	$0.174$
$5$	$\mathbf{0.655}$	$\mathbf{0.173}$
$10$	$0.637$	$0.181$
MASSIVE

Appendix

Table 7: Number of training Epochs ablation, number of added tokens is

M=1000

across all runs.

# Added Tokens	F1 score	Average Runtime [ms]
0	$69.1\pm 8$	$1280\pm 70.1$
150	$68.9\pm 0.09$	$1220\pm 81.3$
500	$68.3\pm 0.5$	$1190\pm 79.4$
1000	$68.8\pm 0.5$	$1130\pm 90.1$

Appendix

Table 8: TASC-ft Results on EUR-LEX dataset with model MistralV02-7B.

Panel of Figure 7 — Figure 7: Speedup vs. mixing parameter $\gamma$ .

Panel of Figure 8 — Figure 8: Token distribution comparison before and after augmenting the tokenizer vocabulary via TASC-ft (Algorithm 1 ). I cane be seen that applying TASC-ft results in a more uniform token distribution.

-	Output		Input
Dataset	Entropy (before)	Entropy (after)	Entropy (before)	Entropy (after)
EUR-LEX	0.423	0.592	0.4502	0.455
Asclepius	0.402	0.537	0.479	0.562

Appendix

Table 9: Effect of TASC-ft on the token entropy in the output dataset.

Panel of Figure 9 — Figure 9: Evaluation Judge LLM prompt for Asclepius dataset responses (Figure 2 )

Language Model Vocabulary Expansion

See all 8 papers

SlimSpec: Low-Rank Draft LM-Head for Accelerated Speculative Decoding
May 11, 2026Anton Plaksin, Sergei Krutikov, Sergei Skvortsov +1LLM Compression Speculative Decoding
Nebius, Amsterdam, Netherlands.
Task-Specific Efficiency Analysis: When Small Language Models Outperform Large Language Models
Mar 22, 2026Jinghan Cao, Yu Ma, Xinjin Li +2Small Language Models Efficient Language Model Inference
San Francisco State University - Department of Computer Science 1600 Holloway Ave, San Francisco - USA · Carnegie Mellon University - Department of Computer Science 5000 Forbes Ave, Pittsburgh - USA · Columbia University - Department of Computer Science 116th and Broadway, New York - USA +2
Beyond Uniform Tokens: Adaptive Compression for Time Series Language Models
Jun 11, 2026Jialin Gan, Xin Qiu, Guangzhe Chen +1LLM Inference Acceleration Multimodal Token Compression
Zhejiang University · Harbin Institute of Technology · Shandong University

May 11, 2026J/K move · Enter open · S save

SlimSpec: Low-Rank Draft LM-Head for Accelerated Speculative Decoding

Anton Plaksin, Sergei Krutikov, Sergei Skvortsov, Alexander Samarin

Nebius, Amsterdam, Netherlands.

LLM Compression Speculative Decoding LLM Inference Acceleration

Open paper AskPDF arXiv

Speculative decoding speeds up autoregressive generation in Large Language Models (LLMs) through a two-step procedure, where a lightweight draft model proposes tokens which the target model then verifies in a single forward pass. Although the drafter network is small in modern architectures, its LM-head still performs projection to a large vocabulary, becoming one of the major computational bottlenecks. In prior work this issue has been predominantly addressed via static or dynamic vocabulary truncation. Yet mitigating the bottleneck, these methods bring in extra complexity, such as special vocabulary curation, sophisticated inference-time logic or modifications of the training setup. In this paper, we propose SlimSpec, a low-rank parameterization of the drafter's LM-head that compresses the inner representation rather than the output, preserving full vocabulary support. We evaluate our method with EAGLE-3 drafter across three target models and diverse benchmarks in both latency- and throughput-bound inference regimes. SlimSpec achieves

4\text{-}5\times

acceleration over the standard LM-head architecture while maintaining a competitive acceptance length, surpassing existing methods by up to

8\text{-}9\%

of the end-to-end speedup. Our method requires minimal adjustments of training and inference pipelines. Combined with the aforementioned speedup improvements, it makes SlimSpec a strong alternative across wide variety of draft LM-head architectures.

Abstract

Language Model Vocabulary Expansion

Explore similar work

SlimSpec: Low-Rank Draft LM-Head for Accelerated Speculative Decoding

Task-Specific Efficiency Analysis: When Small Language Models Outperform Large Language Models

Beyond Uniform Tokens: Adaptive Compression for Time Series Language Models