Random Projection Quantizer

Random projection quantization is a technique that uses randomized projections and codebooks to represent data in a quantized, lower-dimensional space. Current research focuses on its application in various machine learning domains, including self-supervised speech pre-training (e.g., using models like BEST-RQ and NEST-RQ), differentially private machine learning, and federated learning, often improving efficiency and privacy while maintaining accuracy. This approach offers advantages in reducing computational costs and memory requirements for large models, particularly relevant for resource-constrained environments like edge computing and IoT devices. The simplicity and effectiveness of random projection quantization make it a promising tool for enhancing the scalability and privacy of diverse machine learning applications.

Papers

September 13, 2024

NEST-RQ: Next Token Prediction for Speech Self-Supervised Pre-Training
Minglun Han, Ye Bai, Chen Shen, Youjia Huang, Mingkun Huang, Zehua Lin, Linhao Dong, Lu Lu, Yuxuan Wang
Next Token Prediction Speech Supervised Learning Model Non Streaming Automatic Speech Recognition Non Causal Nest Agent Random Projection Quantizer

May 7, 2024

Open Implementation and Study of BEST-RQ for Speech Processing
Ryan Whetten, Titouan Parcollet, Marco Dinarelli, Yannick Estève
Self Supervised Learning Automatic Speech Recognition Study Feature Open Source Speech Processing Random Projection Quantizer

February 9, 2024

RQP-SGD: Differential Private Machine Learning through Noisy SGD and Randomized Quantization
Ce Feng, Parv Venkitasubramaniam
Stochastic Gradient Descent Private Stochastic Differentially Private Dp SGD Edge Machine Learning Random Projection Quantizer

October 24, 2023

Random Entity Quantization for Parameter-Efficient Compositional Knowledge Graph Representation
Jiaang Li, Quan Wang, Yi Liu, Licheng Zhang, Zhendong Mao
Knowledge Graph Representation Random Projection Quantizer

June 20, 2023

Randomized Quantization is All You Need for Differential Privacy in Federated Learning
Yeojoon Youn, Zihao Hu, Juba Ziani, Jacob Abernethy
Differential Privacy Federated Learning Random Projection Quantizer

June 13, 2023

GQFedWAvg: Optimization-Based Quantized Federated Learning in General Edge Computing Systems
Yangchen Li, Ying Cui, Vincent Lau
Gradient Descent Edge Computing Random Projection Quantizer

December 19, 2022

Randomized Quantization: A Generic Augmentation for Data Agnostic Self-supervised Learning
Huimin Wu, Chenyang Lei, Xiao Sun, Peng-Shuai Wang, Qifeng Chen, Kwang-Ting Cheng, Stephen Lin, Zhirong Wu
Data Augmentation Self Supervised Representation Learning Non Uniform Quantization Channel Masking Random Projection Quantizer

February 3, 2022

Self-supervised Learning with Random-projection Quantizer for Speech Recognition
Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, Yonghui Wu
Self Supervised Learning Speech Recognition Universal Speech Model Random Projection Quantizer

Random Projection Quantizer

Papers

NEST-RQ: Next Token Prediction for Speech Self-Supervised Pre-Training

Open Implementation and Study of BEST-RQ for Speech Processing

RQP-SGD: Differential Private Machine Learning through Noisy SGD and Randomized Quantization

Random Entity Quantization for Parameter-Efficient Compositional Knowledge Graph Representation

Randomized Quantization is All You Need for Differential Privacy in Federated Learning

GQFedWAvg: Optimization-Based Quantized Federated Learning in General Edge Computing Systems

Randomized Quantization: A Generic Augmentation for Data Agnostic Self-supervised Learning

Self-supervised Learning with Random-projection Quantizer for Speech Recognition