Organizations: Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences Chongqing School, University of Chinese Academy of Sciences Chongqing, China · The First Affiliated Hospital, University of Science and Technology of China Hefei, China · City University of Hong Kong Hong Kong, Hong Kong
Interactive recommender systems (IRS) are increasingly optimized with Reinforcement Learning (RL) to capture the sequential nature of user-system dynamics. However, existing fairness-aware methods often suffer from a fundamental oversight: they assume the observed user state is a faithful representation of true preferences. In reality, implicit feedback is contaminated by popularity-driven noise and exposure bias, creating a distorted state that misleads the RL agent. We argue that the persistent conflict between accuracy and fairness is not merely a reward-shaping issue, but a state estimation failure. In this work, we propose DSRM-HRL, a framework that reformulates fairness-aware recommendation as a latent state purification problem followed by decoupled hierarchical decision-making. We introduce a Denoising State Representation Module (DSRM) based on diffusion models to recover the low-entropy latent preference manifold from high-entropy, noisy interaction histories. Built upon this purified state, a Hierarchical Reinforcement Learning (HRL) agent is employed to decouple conflicting objectives: a high-level policy regulates long-term fairness trajectories, while a low-level policy optimizes short-term engagement under these dynamic constraints. Extensive experiments on high-fidelity simulators (KuaiRec, KuaiRand) demonstrate that DSRM-HRL effectively breaks the "rich-get-richer" feedback loop, achieving a superior Pareto frontier between recommendation utility and exposure equity.
Figures & tables
Figure 1. The motivation of DSRM-HRL. (Left) In interactive environments, the observed user state is heavily contaminated by popularity bias (red noise), obscuring true user preferences. (Middle) Existing RL methods typically fail under such noise: general RL agents succumb to the "rich-get-richer" loop (high AD), while naive fairness-aware agents sacrifice accuracy for diversity. (Right) Our proposed DSRM-HRL first purifies the state via diffusion-based denoising and then employs a hierarchical policy to dynamically balance long-term fairness and short-term utility.
Figure 2. The Spurious Reward Trap. Observed rewards are heavily dominated by exposure frequency rather than intrinsic relevance, creating a biased input for policy learning.
Figure 3. State Purification Gain. Denoising the input alone expands the Pareto frontier of accuracy and fairness without complex reward shaping.
Figure 4. Visualization of State Purification. DSRM transforms the popularity-dominated collapsed manifold into a disentangled, semantic preference space.
Figure 5. Overall framework.
Dataset
Usage
Users
Items
Interactions
KuaiRec
Train
7,176
10,728
12,530.8k
Test
1,411
3,327
4,676.5k
KuaiRand
Train
26,285
7,551
1,436.6k
Test
27,285
7,583
1,186.1k
Table 1. Statistics of the training and testing environments.
Dataset
Method
Max Len = 30
Max Len = 50
Len
Reach
Rcum
AD
Len
Reach
Rcum
AD
KuaiRec
A2C
13.433 ± 0.236
0.620 ± 0.056
8.339 ± 0.903
0.498 ± 0.085
22.633 ± 0.386
0.680 ± 0.057
15.422 ± 1.727
0.577 ± 0.137
TD3
15.567 ± 1.733
0.801 ± 0.029
12.007 ± 1.260
0.428 ± 0.073
32.267 ± 6.712
0.850 ± 0.022
26.627 ± 6.444
0.293 ± 0.081
BCQ
20.800 ± 0.616
0.757 ± 0.004
15.036 ± 0.314
0.021 ± 0.003
34.067 ± 0.759
0.825 ± 0.003
28.043 ± 0.343
0.037 ± 0.007
MOFIR
13.733 ± 1.406
0.577 ± 0.092
7.852 ± 1.816
0.668 ± 0.130
22.933 ± 1.901
0.616 ± 0.093
14.596 ± 3.295
0.668 ± 0.120
DORL
14.700 ± 2.376
0.579 ± 0.061
7.490 ± 0.889
0.466 ± 0.236
22.133 ± 0.939
0.667 ± 0.100
14.564 ± 3.407
0.548 ± 0.041
Table 2. Overall Experimental Results
Dataset
Method
Max Len = 30
Max Len = 50
Len
Reach
Rcum
AD
Len
Reach
Rcum
AD
KuaiRec
FLAT
24.333 ± 0.741
0.911 ± 0.003
22.081 ± 1.008
0.013 ± 0.005
31.267 ± 4.658
0.870 ± 0.045
26.691 ± 2.693
0.222 ± 0.291
HRL
23.167 ± 0.411
0.895 ± 0.016
19.942 ± 0.723
0.013 ± 0.001
33.400 ± 0.566
0.863 ± 0.005
28.055 ± 0.396
0.016 ± 0.002
+RCE
15.167 ± 0.386
0.653 ± 0.004
9.588 ± 0.184
0.032 ± 0.001
25.500 ± 0.852
0.737 ± 0.002
17.528 ± 0.104
0.029 ± 0.004
+TCE
15.867 ± 0.403
0.746 ± 0.007
11.491 ± 0.353
0.030 ± 0.002
25.800 ± 0.497
0.793 ± 0.012
20.293 ± 0.444
0.040 ± 0.005
+BOD
15.867 ± 1.793
0.761 ± 0.033
11.740 ± 2.156
0.479 ± 0.303
28.800 ± 3.910
0.827 ± 0.049
23.114 ± 3.816
0.209 ± 0.252
Table 3. Ablation Study
Figure 6. Comparison of different diffusion step.
Method
Time (seconds)
A2C
7512.144
SAC4IF
6754.545
DNaIR
5934.623
HRL
8462.525
RCE
29919.296
DSRM-HRL (Ours)
15909.175
Table 4. Training time comparison on KuaiRec (Max Len=30, Episode=20000).
Figure 7. Training dynamics on KuaiRec and KuaiRand-Pure (Max Len = 30). We report learning curves of Interaction Length (left) and AD (right) over 20,000 training steps averaged across three random seeds. DSRM-HRL exhibits smoother convergence and lower variance than baselines, indicating improved training stability in non-stationary environments.
Item-side fairness is crucial for ensuring the fair exposure of long-tail items in interactive recommender systems. Existing approaches promote the exposure of long-tail items by directly incorporating them into recommended results. This causes misalignment between user preferences and the recommended long-tail items, which hinders long-term user engagement and reduces the effectiveness of recommendations. We aim for a proactive fairness-guiding strategy, which actively guides user preferences toward long-tail items while preserving user satisfaction during the interactive recommendation process. To this end, we propose HRL4PFG, an interactive recommendation framework that leverages hierarchical reinforcement learning to guide user preferences toward long-tail items progressively. HRL4PFG operates through a macro-level process that generates fairness-guided targets based on multi-step feedback, and a micro-level process that fine-tunes recommendations in real time according to both these targets and evolving user preferences. Extensive experiments show that HRL4PFG improves cumulative interaction rewards and maximum user interaction length by a larger margin when compared with state-of-the-art methods in interactive recommendation environments.
Chongjun Xia, Xiaoyu Shi, Hong Xie +3
Chinese Academy of Sciences · University of Technology Sydney · University of Science and Technology of China
Recommender systems often induce filter bubbles and semantic homogenization by monolithically optimizing for immediate user engagement. Standard single-objective models, including traditional Deep Q-Networks, are ill-equipped to navigate the trade-offs between platform retention and critical societal values like information diversity and provider fairness. To address these limitations, we introduce a multi-objective reinforcement learning framework that formalizes recommendation as a semantic multi-objective Markov decision process. By integrating high-fidelity semantic embeddings with a Pareto-DQN agent, our architecture treats engagement, diversity, and fairness as distinct, non-aggregable reward signals, avoiding the pitfalls of static reward scalarization. Empirical evaluations on the MovieLens small dataset shows that our hypervolume based action selection disrupts the feedback loops responsible for semantic collapse. By sustaining high state-trajectory variance, the Pareto-DQN effectively maps the Pareto frontier, achieving gains in auxiliary societal objectives with only marginal impacts on engagement. This work provides a path toward intrinsically aligned, responsible recommender systems.
Cláudio Lúcio Do Val Lopes, Lucca Machado da Silva, André de Oliveira Brandão
Large Language Models (LLMs) have shown strong potential for recommendation by leveraging their semantic understanding and contextual modeling capabilities. Recent studies further introduce reasoning mechanisms to improve user preference modeling. However, explicit natural-language reasoning incurs substantial inference overhead, whereas existing latent reasoning methods mainly focus on generating or verifying intermediate states, leaving their layer-wise preference roles and contributions insufficiently characterized. We propose HiLaR, a Hierarchical Latent Reasoning framework with layer-aware reinforcement optimization for LLM-based recommendation. HiLaR constructs temporal-guided hierarchical user preference representations, aligns them with multiple LLM latent reasoning states, and organizes the reasoning process from broad preferences to fine-grained current intents. To further optimize the reasoning trajectory, HiLaR combines final recommendation feedback with layer-aware process rewards derived from the marginal target-likelihood gain of each state. Experiments on four Amazon benchmark datasets show that HiLaR generally outperforms strong sequential, generative, and LLM-based recommendation baselines. Ablation and sensitivity analyses further verify the contribution of hierarchical representation learning, latent alignment, and process-level optimization. Our code is available in https://github.com/hupeiyu21/HiLaR.
Peiyu Hu, Siying Gu, Weihai Lu +8
1Xi’an Jiaotong-Liverpool University · 2Xiaohongshu · 3Peking University +1