Training Datasets

Training datasets are crucial for developing effective machine learning models, particularly large language and vision models, but their size and quality significantly impact model performance, cost, and security. Current research focuses on optimizing dataset size and composition through techniques like dataset distillation, pruning, and automated data generation, as well as mitigating issues arising from memorization of biased or sensitive information within existing datasets via methods such as machine unlearning. These advancements are vital for improving model efficiency, robustness, and ethical considerations across diverse applications, from medical image analysis to natural language processing.

Papers

March 28, 2022

Killing Two Birds with One Stone:Efficient and Robust Training of Face Recognition CNNs by Partial FC
Xiang An, Jiankang Deng, Jia Guo, Ziyong Feng, Xuhan Zhu, Jing Yang, Tongliang Liu
Face Recognition Robust Training Deep Face Recognition Training Datasets Fully Connected Layer STONE Deterioration Pattern Discriminative Embeddings Margin Softmax

March 17, 2022

On the Importance of Data Size in Probing Fine-tuned Models
Houman Mehrafarin, Sara Rajaee, Mohammad Taher Pilehvar
Fine Tuning Importance Aware Dataset Size Training Datasets Linguistic Knowledge

March 16, 2022

Learning to Generate Synthetic Training Data using Gradient Matching and Implicit Differentiation
Dmitry Medvedev, Alexander D'yakonov
LeArning Abstract Deep Network Synthetic Training Data Implicit Differentiation Data Distillation Training Datasets Gradient Matching Generative Ai

March 2, 2022

Nemo: Guiding and Contextualizing Weak Supervision for Interactive Data Programming
Cheng-Yu Hsieh, Jieyu Zhang, Alexander Ratner
Weak Supervision Specific Heuristic Training Datasets NeMo Team Heuristic Label

January 13, 2022

Neuron-Specific Dropout: A Deterministic Regularization Technique to Prevent Neural Networks from Overfitting & Reduce Dependence on Large Training Samples
Joshua Shunk
Deep Neural Network Structured Dropout Single Neuron Level Model Overfitting Training Datasets Deterministic Gradient

Training Datasets

Papers

Killing Two Birds with One Stone:Efficient and Robust Training of Face Recognition CNNs by Partial FC

On the Importance of Data Size in Probing Fine-tuned Models

Learning to Generate Synthetic Training Data using Gradient Matching and Implicit Differentiation

Nemo: Guiding and Contextualizing Weak Supervision for Interactive Data Programming

Neuron-Specific Dropout: A Deterministic Regularization Technique to Prevent Neural Networks from Overfitting & Reduce Dependence on Large Training Samples