Small Language Models

Also known as SLM

Researchers build small language models by training compact models from scratch, distilling larger teachers, or adapting open models with supervised fine-tuning, LoRA/QLoRA, preference optimization, and quantization. To extend their capabilities, systems combine them with retrieval, tools, structured reasoning, verification, or selective handoff to larger models, particularly for edge, private, and latency-sensitive use. Evaluations span reasoning and coding benchmarks such as GSM8K and SWE-bench, structured tasks such as Spider and BIRD text-to-SQL, and domains including medicine, law, and low-resource languages. Specialized small models can be efficient and competitive, but results depend on the task: grounding retrieved context, reliable multi-step reasoning, safety, and generalization remain challenges, alongside trade-offs among accuracy, latency, memory, energy, and cost.

Latest papers

277 papers42 papers in the last four weeks, up 100% on the four weeks before. 0.2% of all new papers.

Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
Nov 11, 2025Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin +12LLM Inference Efficiency AI Accelerator Inference
Stanford University · Together AI
Toward Cybersecurity-Expert Small Language Models
Oct 15, 2025Matan Levi, Daniel Ohayon, Ariel Blobstein +3Cyber Threat Intelligence Cybersecurity
1IBM Research.
Instruction Retrieval at Inference Time for Small Language Models
Oct 15, 2025Kenan Alkiek, David Jurgens, Vinod VydiswaranKnowledge Augmentation for Language Models Small Language Models
School of Information University of Michigan Ann Arbor, MI 48109, USA
Small Language Models are the Future of Agentic AI
Jun 2, 2025Peter Belcak, Greg Heinrich, Shizhe Diao +5Small Language Models LLM Agents
NVIDIA Research · Georgia Institute of Technology
A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone
May 19, 2025Jitai Hao, Qiang Huang, Hao Liu +3Language Model Pretraining LLM Pruning
School of Intelligence Science and Engineering, Harbin Institute of Technology, Shenzhen · Baidu Inc. · Leiden University +1
Even Small Reasoners Should Quote Their Sources: Introducing the Pleias-RAG Model Family
Apr 25, 2025Pierre-Carl Langlais, Pavel Chizhov, Mattia Nee +6Retrieval-Augmented Generation LLM Grounding
PleIAs, Paris, France
LearNAT: Learning NL2SQL with AST-guided Task Decomposition for Large Language Models
Apr 3, 2025Weibin Liao, Xin Gao, Tianyu Jia +6LLM Planning RL for Language Model Reasoning
School of Computer Science, Peking University · 2Key Laboratory of High Confidence Software Technologies, Ministry of Education · School of Computer Sciences, Beijing University of Posts and Telecommunications +5

Nov 11, 2025J/K move · Enter open · S save

Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin, J. Wes Griffin+11

Stanford University · Together AI

LLM Inference Efficiency AI Accelerator Inference On-Device Language Model Inference Small Language Models

Open paper AskPDF arXiv

Large language model (LLM) queries are predominantly processed by frontier models in centralized cloud infrastructure. Demand growth strains this paradigm faster than providers can scale. Two advances create an opportunity to rethink it: small, local LMs (<=20B active parameters) now achieve competitive performance to frontier models on many tasks, and local accelerators (e.g., Apple M4 Max) can host these models at interactive latencies. This raises the question: can local inference viably redistribute demand from centralized infrastructure? This requires measuring both whether local LMs can accurately answer real-world queries and whether they can do so efficiently on power-constrained devices (e.g., laptops). We propose intelligence per watt (IPW), task accuracy per unit of power, as a unified metric for the capability and efficiency of local inference across model-accelerator configurations. We evaluate 20+ state-of-the-art local LMs, 8 hardware accelerators (local and cloud), and 1M real-world single-turn chat and reasoning queries. For each query, we measure accuracy (local LM win rate against frontier models), energy, latency, and power. We find three key results. First, local LMs successfully answer 88.7% of these queries, with accuracy varying by domain. Second, longitudinal analysis from 2023-2025 shows IPW improved 5.3x, driven by both algorithmic and accelerator advances, with locally-serviceable query coverage rising from 23.2% to 71.3%. Third, local accelerators achieve at least 1.4x lower IPW than cloud accelerators running identical models, revealing significant headroom for local accelerator optimization. These findings demonstrate that local inference can meaningfully redistribute demand from centralized infrastructure for a substantial subset of queries, with IPW serving as the critical metric for tracking this transition.