Straight Through Estimator

The straight-through estimator (STE) is a technique used to train neural networks with discrete or quantized components, addressing the challenge of non-differentiable activation functions. Current research focuses on improving STE's performance in various applications, including extreme model compression (e.g., for large language models), vector quantization, and incorporating logical constraints into neural networks. This work aims to enhance training stability, reduce error, and improve the accuracy of models employing STE, impacting fields like efficient deep learning deployment and neuro-symbolic AI. Recent advancements explore adaptive STE variants and alternative gradient estimators to overcome limitations of the basic STE approach.