Parity Learning

Parity learning, a problem focusing on identifying hidden linear relationships in data, is a benchmark for evaluating learning algorithms' efficiency and effectiveness. Current research investigates the fundamental computational limits of parity learning, exploring trade-offs between memory, sample size, and computational resources using various approaches, including multi-pass streaming algorithms and neural networks with sparse initializations. These studies reveal surprising insights into the dynamics of stochastic gradient descent and highlight the limitations of gradient-based methods in certain scenarios, motivating the exploration of alternative optimization techniques like SAT solvers. The findings contribute to a deeper understanding of both theoretical and practical aspects of machine learning, impacting algorithm design and the development of more efficient learning systems.

Papers

October 12, 2023

Tight Time-Space Lower Bounds for Constant-Pass Learning
Xin Lyu, Avishay Tal, Hongxun Wu, Junzhao Yang
Lower Bound General Bound Quadratic Neural Network Time Constraint O$ Memory Parity Learning

September 7, 2023

Pareto Frontiers in Neural Feature Learning: Data, Compute, Width, and Luck
Benjamin L. Edelman, Surbhi Goel, Sham Kakade, Eran Malach, Cyril Zhang
Raw Data Pareto Frontier Parity Learning

March 14, 2023

Practically Solving LPN in High Noise Regimes Faster Using Neural Networks
Haozhe Jiang, Kaiyue Wen, Yilei Chen
Neural Network Neural Network Model Two Layer Neural Network Neural Network Algorithm Parity Learning

July 18, 2022

Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit
Boaz Barak, Benjamin L. Edelman, Surbhi Goel, Sham Kakade, Eran Malach, Cyril Zhang
Deep Learning Many Sparse Much Progress Sparse Solution Computational Limit Sparse Parity Parity Learning

June 10, 2022

Training Neural Networks using SAT solvers
Subham S. Sahoo
Neural Network Global Optimization Gradient Based Optimisation Parity Learning

Parity Learning

Papers

Tight Time-Space Lower Bounds for Constant-Pass Learning

Pareto Frontiers in Neural Feature Learning: Data, Compute, Width, and Luck

Practically Solving LPN in High Noise Regimes Faster Using Neural Networks

Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit

Training Neural Networks using SAT solvers