Model Training

Model training focuses on developing efficient and effective methods for creating accurate and robust machine learning models. Current research emphasizes improving training efficiency through techniques like low-precision computation, optimized memory management (e.g., using recomputation and memory-aware scheduling), and efficient communication strategies in distributed and federated learning settings. These advancements are crucial for scaling model training to larger datasets and more complex architectures, impacting various fields from computer vision and natural language processing to healthcare and industrial applications.

Papers

November 20, 2024

Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training
Jared Fernandez, Luca Wehrstedt, Leonid Shamis, Mostafa Elhoushi, Kalyan Saladi, Yonatan Bisk, Emma Strubell, Jacob Kahn
Large Language Model Neural Network Model Model Training Hardware Accelerator Customer Service LLM Training Positive Scaling

November 15, 2024

Framework for Co-distillation Driven Federated Learning to Address Class Imbalance in Healthcare
Suraj Racha, Shubh Gupta, Humaira Firdowse, Aastik Solanki, Ganesh Ramakrishnan, Kshitij S. Jadhav
New Framework Class Imbalance Healthcare System Model Training Mutual Distillation Group Imbalance Knowledge Sharing

November 5, 2024

Precise Drive with VLM: First Prize Solution for PRCV 2024 Drive LM challenge
Bin Huang, Siyu Wang, Yuanpeng Chen, Yidan Wu, Hui Song, Zifan Ding, Jing Leng, Chengpeng Liang, Peng Xue, Junliang Zhang, Tiankun Zhao
Model Training Place Solution Multi View Image

October 29, 2024

Retrieval-Augmented Approach for Unsupervised Anomalous Sound Detection and Captioning without Model Training
Ryoya Ogura, Tomoya Nishida, Yohei Kawaguchi
Model Training Retrieval Augmented Generated Caption Anomalous Sound Detection Anomalous Sound

October 25, 2024

October 23, 2024

October 20, 2024

Hallucination Detox: Sensitive Neuron Dropout (SeND) for Large Language Model Training
Shahrad Mohammadzadeh, Juan David Guerra, Marco Bonizzato, Reihaneh Rabbany, Golnoosh Farnadi
Large Language Model Model Training Content Hallucination Large Language Reducing Hallucination

October 19, 2024

Pipeline Gradient-based Model Training on Analog In-memory Accelerators
Zhaoxian Wu, Quan Xiao, Tayfun Gokmen, Hsinyu Tsai, Kaoutar El Maghraoui, Tianyi Chen
Model Training Pipeline System Parallel Training Large Deep Learning Model Analog in Memory Computing Analog Compute in Memory

October 17, 2024

Router-Tuning: A Simple and Effective Approach for Enabling Dynamic-Depth in Transformers
Shwai He, Tao Ge, Guoheng Sun, Bowei Tian, Xiaoyang Wang, Ang Li, Dong Yu
Transformer Megatron Decepticons Model Training Large Depth Computation Method Effective Approach Adaptive Depth Computational Overhead

October 10, 2024

Unsupervised Data Validation Methods for Efficient Model Training
Yurii Paniv
Natural Language Processing Vision Language Model Unsupervised Learning Low Resource Language Model Training Speech to Text Cross Lingual Transfer Learning

October 9, 2024

Benchmarking Data Heterogeneity Evaluation Approaches for Personalized Federated Learning
Zhilong Li, Xiaohu Wu, Xiaoli Tang, Tiantian He, Yew-Soon Ong, Mengmeng Chen, Qiqi Liu, Qicheng Lao, Han Yu
Model Training Data Heterogeneity Personalized Federated Learning Evaluation Benchmark Statistical Heterogeneity Benchmark Framework

October 5, 2024

ConDa: Fast Federated Unlearning with Contribution Dampening
Vikram S Chundawat, Pushkar Niroula, Prasanna Dhungana, Stefan Schoepf, Murari Mandal, Alexandra Brintrup
Federated Learning Membership Inference Attack Model Training Unlearning Framework Federated Unlearning

September 28, 2024

Efficient Semantic Diffusion Architectures for Model Training on Synthetic Echocardiograms
David Stojanovski, Mariana da Silva, Pablo Lamata, Arian Beqiri, Alberto Gomez
Diffusion Model Generative Model Model Training High Fidelity Ultrasound Video Synthesis

September 24, 2024

Adversarial Federated Consensus Learning for Surface Defect Classification Under Data Heterogeneity in IIoT
Jixuan Cui, Jun Li, Zhen Mei, Yiyang Ni, Wen Chen, Zengxiang Li
Model Training Data Heterogeneity Surface Defect Detection Fusion Weight Industrial Internet of Thing Federated Adversarial

September 23, 2024

PipeFill: Using GPUs During Bubbles in Pipeline-parallel LLM Training
Daiyaan Arfeen, Zhen Zhang, Xinwei Fu, Gregory R. Ganger, Yida Wang
Model Training Single GPU Training Deep Asset Price Bubble Instruction Hierarchy GPU Energy

September 19, 2024

Performance and Power: Systematic Evaluation of AI Workloads on Accelerators with CARAML
Chelsea Maria John, Stepan Nassyr, Carolin Penke, Andreas Herten
System Performance Model Training Hardware Accelerator Accelerator Tuning

September 17, 2024

FedNE: Surrogate-Assisted Federated Neighbor Embedding for Dimensionality Reduction
Ziwei Li, Xiaoqi Wang, Hong-You Chen, Han-Wei Shen, Wei-Lun Chao
Graph Learning Dimensionality Reduction Model Training Federated Edge Learning Stochastic Neighbor Embedding

September 9, 2024

CoBo: Collaborative Learning via Bilevel Optimization
Diba Hashemi, Lie He, Martin Jaggi
Bilevel Optimization Optimization Problem Model Training Collaborative Learning Client Selection

Model Training

Papers

Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training

Framework for Co-distillation Driven Federated Learning to Address Class Imbalance in Healthcare

Precise Drive with VLM: First Prize Solution for PRCV 2024 Drive LM challenge

Retrieval-Augmented Approach for Unsupervised Anomalous Sound Detection and Captioning without Model Training

BitPipe: Bidirectional Interleaved Pipeline Parallelism for Accelerating Large Models Training

COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training

Towards Active Participant-Centric Vertical Federated Learning: Some Representations May Be All You Need

Securing Federated Learning Against Novel and Classic Backdoor Threats During Foundation Model Integration

Hallucination Detox: Sensitive Neuron Dropout (SeND) for Large Language Model Training

Pipeline Gradient-based Model Training on Analog In-memory Accelerators

Router-Tuning: A Simple and Effective Approach for Enabling Dynamic-Depth in Transformers

Unsupervised Data Validation Methods for Efficient Model Training

Benchmarking Data Heterogeneity Evaluation Approaches for Personalized Federated Learning

ConDa: Fast Federated Unlearning with Contribution Dampening

Efficient Semantic Diffusion Architectures for Model Training on Synthetic Echocardiograms

Adversarial Federated Consensus Learning for Surface Defect Classification Under Data Heterogeneity in IIoT

PipeFill: Using GPUs During Bubbles in Pipeline-parallel LLM Training

Performance and Power: Systematic Evaluation of AI Workloads on Accelerators with CARAML

FedNE: Surrogate-Assisted Federated Neighbor Embedding for Dimensionality Reduction

CoBo: Collaborative Learning via Bilevel Optimization