Data Similarity

Data similarity research focuses on understanding and leveraging the relationships between datasets to improve the efficiency and performance of machine learning algorithms, particularly in distributed and federated learning settings. Current research explores various methods for quantifying and exploiting data similarity, including approaches based on gradient and Hessian similarity, as well as techniques like soft deduplication to manage redundant data in large datasets. These advancements aim to reduce communication costs, improve convergence speed, and enhance the overall efficiency of training large models, impacting fields like natural language processing and other areas requiring large-scale data analysis. Furthermore, research is actively addressing the limitations of relying solely on data similarity assumptions, developing algorithms that perform well even with heterogeneous data.

Papers

November 4, 2024

Enhancing Table Representations with LLM-powered Synthetic Data Generation
Dayu Yang, Natawut Monaikul, Amanda Ding, Bozhao Tan, Kishore Mosaliganti, Giri Iyengar
Faithful Generation Synthetic Data Generation Recommendation Task Table Representation Data Similarity

September 22, 2024

Accelerated Stochastic ExtraGradient: Mixing Hessian and Gradient Similarity to Reduce Communication in Distributed and Federated Learning
Dmitry Bylinkin, Kirill Degtyarev, Aleksandr Beznosikov
Early Stage Convergence High Similarity Hessian Matrix Clean Distribution Data Similarity Gradient Similarity

July 9, 2024

SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training
Nan He, Weichen Xiong, Hanwen Liu, Yi Liao, Lei Ding, Kai Zhang, Guohua Tang, Xiao Han, Wei Yang
Large Language Model Language Model N Gram Large Scale Pretraining Data Deduplication Data Reweighting Data Similarity

May 30, 2024

SPAM: Stochastic Proximal Point Method with Momentum Variance Reduction for Non-convex Cross-Device Federated Learning
Avetik Karagulyan, Egor Shulgin, Abdurakhmon Sadiev, Peter Richtárik
LeArning Abstract Convex Loss Cross Device Federated Learning Stochastic Proximal Point Data Similarity Cross Device

February 29, 2024

On the Convergence of Federated Learning Algorithms without Data Similarity
Ali Beikmohammadi, Sarit Khirirat, Sindri Magnússon
Federated Learning Early Stage Convergence Slow Convergence Federated Learning Algorithm Convergent Algorithm Data Similarity

December 5, 2023

Privacy-Aware Data Acquisition under Data Similarity in Regression Markets
Shashi Raj Pandey, Pierre Pinson, Petar Popovski
Local Differential Privacy Data Market Privacy Aware Data Similarity Regression Market

November 15, 2023

Data Similarity is Not Enough to Explain Language Model Performance
Gregory Yauney, Emily Reif, David Mimno
Large Language Model Multilingual Dataset Similarity Learning Data Similarity Large Scale Comparison

October 20, 2023

What you see is what you get: Experience ranking with deep neural dataset-to-dataset similarity for topological localisation
Matthew Gadd, Benjamin Ramtoula, Daniele De Martini, Paul Newman
Place Recognition Object Localization eXperience Report Personal Memory Localization Performance Visual Genome Visual Memory Data Similarity

September 4, 2023

Composite federated learning with heterogeneous data
Jiaojiao Zhang, Jiang Hu, Mikael Johansson
LeArning Abstract Heterogeneous Data Novel Algorithm Proximal Operator Nonsmooth Regularization Data Similarity Composite Active Learning

June 19, 2022

Compression and Data Similarity: Combination of Two Techniques for Communication-Efficient Solving of Distributed Variational Inequalities
Aleksandr Beznosikov, Alexander Gasnikov
Linear Compression Barzilai Borwein Technique Communication Efficient Variational Inequality Fixed Point Data Similarity

Data Similarity

Papers

Enhancing Table Representations with LLM-powered Synthetic Data Generation

Accelerated Stochastic ExtraGradient: Mixing Hessian and Gradient Similarity to Reduce Communication in Distributed and Federated Learning

SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training

SPAM: Stochastic Proximal Point Method with Momentum Variance Reduction for Non-convex Cross-Device Federated Learning

On the Convergence of Federated Learning Algorithms without Data Similarity

Privacy-Aware Data Acquisition under Data Similarity in Regression Markets

Data Similarity is Not Enough to Explain Language Model Performance

What you see is what you get: Experience ranking with deep neural dataset-to-dataset similarity for topological localisation

Composite federated learning with heterogeneous data

Compression and Data Similarity: Combination of Two Techniques for Communication-Efficient Solving of Distributed Variational Inequalities