Paper ID: 2412.10381

Supervised Learning-enhanced Multi-Group Actor Critic for Live-stream Recommendation

Jingxin Liu, Xiang Gao, Yisha Li, Xin Li, Haiyang Lu, Ben Wang

Reinforcement Learning (RL) has been widely applied in recommendation systems to capture users' long-term engagement, thereby improving dwelling time and enhancing user retention. In the context of a short video & live-stream mixed recommendation scenario, the live-stream recommendation system (RS) decides whether to inject at most one live-stream into the video feed for each user request. To maximize long-term user engagement, it is crucial to determine an optimal live-stream injection policy for accurate live-stream allocation. However, traditional RL algorithms often face divergence and instability problems, and these issues are even more pronounced in our scenario. To address these challenges, we propose a novel Supervised Learning-enhanced Multi-Group Actor Critic algorithm (SL-MGAC). Specifically, we introduce a supervised learning-enhanced actor-critic framework that incorporates variance reduction techniques, where multi-task reward learning helps restrict bootstrapping error accumulation during critic learning. Additionally, we design a multi-group state decomposition module for both actor and critic networks to reduce prediction variance and improve model stability. Empirically, we evaluate the SL-MGAC algorithm using offline policy evaluation (OPE) and online A/B testing. Experimental results demonstrate that the proposed method not only outperforms baseline methods but also exhibits enhanced stability in online recommendation scenarios.

Submitted: Nov 28, 2024