Training Datasets

Training datasets are crucial for developing effective machine learning models, particularly large language and vision models, but their size and quality significantly impact model performance, cost, and security. Current research focuses on optimizing dataset size and composition through techniques like dataset distillation, pruning, and automated data generation, as well as mitigating issues arising from memorization of biased or sensitive information within existing datasets via methods such as machine unlearning. These advancements are vital for improving model efficiency, robustness, and ethical considerations across diverse applications, from medical image analysis to natural language processing.

Papers

July 30, 2024

Machine Unlearning in Generative AI: A Survey
Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, Meng Jiang
Timely Survey Generative Model Generative AI Machine Unlearning Training Datasets Emergent Reasoning

July 25, 2024

Cost-effective Instruction Learning for Pathology Vision and Language Analysis
Kaitao Chen, Mianxin Liu, Fang Yan, Lei Ma, Xiaoming Shi, Lilong Wang, Xiaosong Wang, Lifeng Zhu, Zhe Wang, Mu Zhou, Shaoting Zhang
Large Language Model Vision Language Model Pathology Datasets Language Disorder Training Datasets Instruction Learning

July 15, 2024

Accessing Vision Foundation Models at ImageNet-level Costs
Yitian Zhang, Xu Ma, Yue Bai, Huan Wang, Yun Fu
Foundation Model Vision Foundation Model Training Datasets

July 8, 2024

Hybrid X-Linker: Automated Data Generation and Extreme Multi-label Ranking for Biomedical Entity Linking
Pedro Ruas, Fernando Gallego, Francisco J. Veredas, Francisco M. Couto
Data Generation Biomedical Entity Human Labeled Biomedical Datasets Training Datasets Biomedical Entity Linking Multi Label Ranking

April 30, 2024

More Compute Is What You Need
Zhen Guo
Transformer Based Model Model Performance Inference Efficiency Training Datasets

April 8, 2024

Automated discovery of symbolic laws governing skill acquisition from naturally occurring data
Sannyuya Liu, Qing Li, Xiaoxuan Shen, Jianwen Sun, Zongkai Yang
Raw Data Scientific Discovery Neural Network Model Skill Learning Training Datasets Unobservable State

January 26, 2024

Unlearning Traces the Influential Training Data of Language Models
Masaru Isonuma, Ivan Titov
Language Model Training Data Harmful Unlearning Unlearned Model Training Datasets Training Data Influence

January 3, 2024

Investigating Semi-Supervised Learning Algorithms in Text Datasets
Himmet Toprak Kesgin, Mehmet Fatih Amasyali
Data Augmentation Text Datasets Training Datasets Augmentation Consistency

October 24, 2023

SoK: Memorization in General-Purpose Large Language Models
Valentin Hartmann, Anshuman Suri, Vincent Bindschaedler, David Evans, Shruti Tople, Robert West
Large Language Model Question Answering Limited Memorization LLM Hallucination Training Datasets

October 4, 2023

How Much Training Data is Memorized in Overparameterized Autoencoders? An Inverse Problem Perspective on Memorization Evaluation
Koren Abitbul, Yehuda Dar
Neural Network Supervised Autoencoder Inverse Problem Training Image Autoencoder Architecture Autoencoder Model Training Datasets Task Specific Memorization Measure

July 16, 2023

Dataset Distillation Meets Provable Subset Selection
Murad Tukan, Alaa Maalouf, Margarita Osadchy
Deep Learning Loss Function Dataset Distillation Training Datasets

May 25, 2023

CENSUS-HWR: a large training dataset for offline handwriting recognition
Chetan Joshi, Lawry Sorenson, Ammon Wolfert, Dr. Mark Clement, Dr. Joseph Price, Dr. Kasey Buckles
Handwriting Recognition Handwritten Document Training Datasets Offline Handwritten Text Census Data Handwriting Dataset Dynamic Handwriting Analysis

April 27, 2023

Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalization
Hamza Kheddar, Yassine Himeur, Somaya Al-Maadeed, Abbes Amira, Faycal Bensaali
Deep Learning Automatic Speech Recognition Transfer Learning Better Generalization Deep Transfer Learning Deep Transfer Training Datasets

March 30, 2023

The impact of training dataset size and ensemble inference strategies on head and neck auto-segmentation
Edward G. A. Henderson, Marcel van Herk, Eliana M. Vasquez Osorio
Convolutional Neural Network Global Impact Human Head Ensemble Method Dataset Size Training Datasets Neck Tumor Segmentation

November 19, 2022

Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training
Zhenglun Kong, Haoyu Ma, Geng Yuan, Mengshu Sun, Yanyue Xie, Peiyan Dong, Xin Meng, Xuan Shen, Hao Tang, Minghai Qin, Tianlong Chen, Xiaolong Ma, Xiaohui Xie, Zhangyang Wang, Yanzhi Wang
Vision Transformer End to End Sparse View Information Redundancy Model Reduction Training Datasets

September 30, 2022

On the Impossible Safety of Large AI Models
El-Mahdi El-Mhamdi, Sadegh Farhadkhani, Rachid Guerraoui, Nirupam Gupta, Lê-Nguyên Hoang, Rafael Pinot, Sébastien Rouault, John Stephan
Large Language Model Large AI Model Training Datasets Secure Machine Learning

May 31, 2022

Neural Retriever and Go Beyond: A Thesis Proposal
Man Luo
Information Retrieval Training Datasets Neural Retriever Electronic Thesis Multi Modal Query

May 19, 2022

Dataset Pruning: Reducing Training Data by Examining Generalization Influence
Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, Ping Li
Deep Learning Training Data Generalization Gap Training Datasets Generalization Behavior Dataset Pruning

May 7, 2022

Training from Zero: Radio Frequency Machine Learning Data Quantity Forecasting
William H. Clark, Alan J. Michaels
Machine Learning Training Data Better Zero Training Datasets Modulation Classification

May 4, 2022

Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning
Antonio Emanuele Cinà, Kathrin Grosse, Ambra Demontis, Sebastiano Vascon, Werner Zellinger, Bernhard A. Moser, Alina Oprea, Battista Biggio, Marcello Pelillo, Fabio Roli
Machine Learning Timely Survey Poisoning Attack Threat Model Training Datasets Distinct Pattern