Scratch Project

"Scratch" projects in machine learning encompass the development of models trained entirely from raw data, without leveraging pre-trained weights. Current research focuses on achieving this "from-scratch" training for various model types, including large language models (LLMs), diffusion models, and vision transformers, often employing techniques like autoregressive distillation, sparse regularization, and novel optimization algorithms to improve efficiency and performance. This approach aims to reduce reliance on massive pre-trained models, democratizing access to powerful AI and enabling the creation of specialized models tailored to specific tasks or datasets with limited computational resources. The resulting advancements have implications for various fields, including biomedical research, retail analytics, and autonomous systems.