Overestimation Bias

Overestimation bias, a systematic tendency for models to overestimate values or probabilities, is a significant challenge across various machine learning domains. Current research focuses on mitigating this bias in reinforcement learning algorithms like Q-learning and DDPG, often employing techniques such as double Q-learning, ensemble methods, or novel loss functions to improve value estimation accuracy. Addressing overestimation bias is crucial for improving the reliability and performance of machine learning models in applications ranging from robotics and game playing to biomarker discovery and keyphrase generation, where accurate estimations are vital for optimal decision-making. The development of more robust and computationally efficient bias-correction methods remains a key area of active investigation.

Papers

June 12, 2024

Reconciling Kaplan and Chinchilla Scaling Laws
Tim Pearce, Jinyeop Song
Next Token Prediction Scaling Behavior Et Al Overestimation Bias Chinchilla Scaling

March 9, 2024

Dissecting Deep RL with High Update Ratios: Combatting Value Overestimation and Divergence
Marcel Hussing, Claas Voelcker, Igor Gilitschenski, Amir-massoud Farahmand, Eric Eaton
Deep Reinforcement Learning Deep RL Inverse Divergence Momentum Based Overestimation Bias Primacy Bias Dynamic Update to Data Ratio

March 8, 2024

Conservative DDPG -- Pessimistic RL without Ensemble
Nitsan Soffair, Shie Mannor
Diverse Ensemble Behavior Cloning Q Function Overestimation Bias Conservative Prediction

December 27, 2023

Dual-stage optimizer for systematic overestimation adjustment applied to multi-objective genetic algorithms for biomarker selection
Luca Cattelani, Vittorio Fortino
Genetic Algorithm Multi Objective Model Selection Biomarker Selection Dual Stage Overestimation Bias

October 10, 2023

Suppressing Overestimation in Q-Learning through Adversarial Behaviors
HyeAnn Lee, Donghwan Lee
Q Learning Adversarial Behavior Adversarial Game Minimax Q Learning Overestimation Bias

September 25, 2023

Adapting Double Q-Learning for Continuous Reinforcement Learning
Arsenii Kuznetsov
Reinforcement Learning Policy Reinforcement Learning Bias Correction Double Q Learning Overestimation Bias

November 13, 2022

WR-ONE2SET: Towards Well-Calibrated Keyphrase Generation
Binbin Xie, Xiangpeng Wei, Baosong Yang, Huan Lin, Jun Xie, Xiaoli Wang, Min Zhang, Jinsong Su
Keyphrase Generation Author Assigned Keyphrases Set to Set Overestimation Bias

October 11, 2022

Factors of Influence of the Overestimation Bias of Q-Learning
Julius Wagenbach, Matthia Sabatelli
Q Learning Learning Rate External Influence Key Factor Overestimation Bias

January 20, 2022

Addressing Maximization Bias in Reinforcement Learning with Two-Sample Testing
Martin Waltz, Ostap Okhrin
Reinforcement Learning Mean Estimation Q$ Learning Estimation Bias Two Sample Value Based Reinforcement Learning Overestimation Bias Maximization Bias