Smaller Language Model
Smaller language models (SLMs) aim to achieve comparable performance to their larger counterparts while significantly reducing computational costs and resource demands. Current research focuses on enhancing SLMs through techniques like knowledge distillation from larger models, dataset augmentation using LLMs, and innovative training methods such as contrastive fine-tuning and instruction tuning. These advancements are crucial for making advanced natural language processing capabilities accessible to a wider range of applications and researchers with limited resources, particularly in domains with constrained computational power or data availability.
Papers
Efficiency at Scale: Investigating the Performance of Diminutive Language Models in Clinical Tasks
Niall Taylor, Upamanyu Ghose, Omid Rohanian, Mohammadmahdi Nouriborji, Andrey Kormilitzin, David Clifton, Alejo Nevado-Holgado
Smaller Language Models are capable of selecting Instruction-Tuning Training Data for Larger Language Models
Dheeraj Mekala, Alex Nguyen, Jingbo Shang