Sparse to Sparse Training

Sparse-to-sparse training focuses on developing and improving machine learning models that maintain sparsity throughout the entire training process, rather than starting with a dense model and then pruning it. Current research emphasizes efficient algorithms and model architectures, such as those incorporating attention mechanisms and structured sparsity, to reduce computational costs and improve training efficiency in various applications, including federated learning and natural language processing. This approach addresses the limitations of data scarcity and high computational demands in many machine learning tasks, leading to more efficient and effective models for diverse applications ranging from disaster response to personalized healthcare.

Papers

May 10, 2024

ATSumm: Auxiliary information enhanced approach for abstractive disaster Tweet Summarization with sparse training data
Piyush Kumar Garg, Roshni Chakraborty, Sourav Kumar Dandapat
Constructive Approach Structured Summary Human Written Summary Abstractive Summarization Model Auxiliary Information Sparse to Sparse Training Disaster Tweet Summarization

April 24, 2024

Decentralized Personalized Federated Learning based on a Conditional Sparse-to-Sparser Scheme
Qianyu Long, Qiyuan Wang, Christos Anagnostopoulos, Daning Bi
Personalized Federated Learning Decentralized Federated Learning Hop Neighbor Sparse to Sparse Training

January 25, 2024

Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?
Huy Nguyen, Pedram Akbarian, Nhat Ho
Expert Knowledge Sparse Mixture of Expert Softmax Gating Sparse to Sparse Training Temperature Sampling Sparse Gate

November 9, 2023

Mental Health Diagnosis in the Digital Age: Harnessing Sentiment Analysis on Social Media Platforms upon Ultra-Sparse Feature Content
Haijian Shao, Ming Zhu, Shengjie Zhai
Mental Health Social Medium Platform Sparse Feature Digital Age Sparse to Sparse Training

February 6, 2023

Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researchers
Shiwei Liu, Zhangyang Wang
Critical Lesson Sparse Neural Network Sparse Training Dynamic Sparsity Sparse to Sparse Training Sparse Network Training Post Training Sparsity

October 2, 2022

Exploiting More Information in Sparse Point Cloud for 3D Single Object Tracking
Yubo Cui, Jiayao Shan, Zuoxu Gu, Zhiheng Li, Zheng Fang
3D Tracking Single Object Tracking Sparse Point Cloud 3D Single Object Tracking 3D Sparse Sparse to Sparse Training

April 1, 2022

Monarch: Expressive Structured Matrices for Efficient and Accurate Training
Tri Dao, Beidi Chen, Nimit Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, Christopher Ré
Fine Tuning High Efficiency Weight Matrix Structured Matrix Accurate Training Sparse to Sparse Training Butterfly Matrix

January 27, 2022

Achieving Personalized Federated Learning with Sparse Local Models
Tiansheng Huang, Shiwei Liu, Li Shen, Fengxiang He, Weiwei Lin, Dacheng Tao
Sparse Mask Sparse Local Model Sparse to Sparse Training