Fairness Begins with State: Purifying Latent Preferences for Hierarchical Reinforcement Learning in Interactive Recommendation

Yun Lu; Xiaoyu Shi; Hong Xie; Xiangyu Zhao; Mingsheng Shang

cs.LGMar 4, 2026

Fairness Begins with State: Purifying Latent Preferences for Hierarchical Reinforcement Learning in Interactive Recommendation

Authors: Yun Lu, Xiaoyu Shi, Hong Xie, Xiangyu Zhao, Mingsheng Shang

Organizations: Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences Chongqing School, University of Chinese Academy of Sciences Chongqing, China · The First Affiliated Hospital, University of Science and Technology of China Hefei, China · City University of Hong Kong Hong Kong, Hong Kong

Abstract

Interactive recommender systems (IRS) are increasingly optimized with Reinforcement Learning (RL) to capture the sequential nature of user-system dynamics. However, existing fairness-aware methods often suffer from a fundamental oversight: they assume the observed user state is a faithful representation of true preferences. In reality, implicit feedback is contaminated by popularity-driven noise and exposure bias, creating a distorted state that misleads the RL agent. We argue that the persistent conflict between accuracy and fairness is not merely a reward-shaping issue, but a state estimation failure. In this work, we propose DSRM-HRL, a framework that reformulates fairness-aware recommendation as a latent state purification problem followed by decoupled hierarchical decision-making. We introduce a Denoising State Representation Module (DSRM) based on diffusion models to recover the low-entropy latent preference manifold from high-entropy, noisy interaction histories. Built upon this purified state, a Hierarchical Reinforcement Learning (HRL) agent is employed to decouple conflicting objectives: a high-level policy regulates long-term fairness trajectories, while a low-level policy optimizes short-term engagement under these dynamic constraints. Extensive experiments on high-fidelity simulators (KuaiRec, KuaiRand) demonstrate that DSRM-HRL effectively breaks the "rich-get-richer" feedback loop, achieving a superior Pareto frontier between recommendation utility and exposure equity.

Figures & tables

Figure 1. The motivation of DSRM-HRL. (Left) In interactive environments, the observed user state is heavily contaminated by popularity bias (red noise), obscuring true user preferences. (Middle) Existing RL methods typically fail under such noise: general RL agents succumb to the "rich-get-richer" loop (high AD), while naive fairness-aware agents sacrifice accuracy for diversity. (Right) Our proposed DSRM-HRL first purifies the state via diffusion-based denoising and then employs a hierarchical policy to dynamically balance long-term fairness and short-term utility.

Panel of Figure 2 — Figure 2. The Spurious Reward Trap. Observed rewards are heavily dominated by exposure frequency rather than intrinsic relevance, creating a biased input for policy learning.

Panel of Figure 3 — Figure 3. State Purification Gain. Denoising the input alone expands the Pareto frontier of accuracy and fairness without complex reward shaping.

Panel of Figure 4 — Figure 4. Visualization of State Purification. DSRM transforms the popularity-dominated collapsed manifold into a disentangled, semantic preference space.

Dataset	Usage	Users	Items	Interactions
KuaiRec	Train	7,176	10,728	12,530.8k
KuaiRec	Test	1,411	3,327	4,676.5k
KuaiRand	Train	26,285	7,551	1,436.6k
KuaiRand	Test	27,285	7,583	1,186.1k

Table 1. Statistics of the training and testing environments.

Dataset	Method	Max Len = 30				Max Len = 50
Dataset	Method	Len	$R_{each}$	$R_{cum}$	AD	Len	$R_{each}$	$R_{cum}$	AD
KuaiRec	A2C	13.433 $\pm$ 0.236	0.620 $\pm$ 0.056	8.339 $\pm$ 0.903	0.498 $\pm$ 0.085	22.633 $\pm$ 0.386	0.680 $\pm$ 0.057	15.422 $\pm$ 1.727	0.577 $\pm$ 0.137
	TD3	15.567 $\pm$ 1.733	0.801 $\pm$ 0.029	12.007 $\pm$ 1.260	0.428 $\pm$ 0.073	32.267 $\pm$ 6.712	0.850 $\pm$ 0.022	26.627 $\pm$ 6.444	0.293 $\pm$ 0.081
	BCQ	20.800 $\pm$ 0.616	0.757 $\pm$ 0.004	15.036 $\pm$ 0.314	0.021 $\pm$ 0.003	34.067 $\pm$ 0.759	0.825 $\pm$ 0.003	28.043 $\pm$ 0.343	0.037 $\pm$ 0.007
	MOFIR	13.733 $\pm$ 1.406	0.577 $\pm$ 0.092	7.852 $\pm$ 1.816	0.668 $\pm$ 0.130	22.933 $\pm$ 1.901	0.616 $\pm$ 0.093	14.596 $\pm$ 3.295	0.668 $\pm$ 0.120
	DORL	14.700 $\pm$ 2.376	0.579 $\pm$ 0.061	7.490 $\pm$ 0.889	0.466 $\pm$ 0.236	22.133 $\pm$ 0.939	0.667 $\pm$ 0.100	14.564 $\pm$ 3.407	0.548 $\pm$ 0.041

Table 2. Overall Experimental Results

Dataset	Method	Max Len = 30				Max Len = 50
Dataset	Method	Len	$R_{each}$	$R_{cum}$	AD	Len	$R_{each}$	$R_{cum}$	AD
KuaiRec	FLAT	24.333 $\pm$ 0.741	0.911 $\pm$ 0.003	22.081 $\pm$ 1.008	0.013 $\pm$ 0.005	31.267 $\pm$ 4.658	0.870 $\pm$ 0.045	26.691 $\pm$ 2.693	0.222 $\pm$ 0.291
	HRL	23.167 $\pm$ 0.411	0.895 $\pm$ 0.016	19.942 $\pm$ 0.723	0.013 $\pm$ 0.001	33.400 $\pm$ 0.566	0.863 $\pm$ 0.005	28.055 $\pm$ 0.396	0.016 $\pm$ 0.002
	+RCE	15.167 $\pm$ 0.386	0.653 $\pm$ 0.004	9.588 $\pm$ 0.184	0.032 $\pm$ 0.001	25.500 $\pm$ 0.852	0.737 $\pm$ 0.002	17.528 $\pm$ 0.104	0.029 $\pm$ 0.004
	+TCE	15.867 $\pm$ 0.403	0.746 $\pm$ 0.007	11.491 $\pm$ 0.353	0.030 $\pm$ 0.002	25.800 $\pm$ 0.497	0.793 $\pm$ 0.012	20.293 $\pm$ 0.444	0.040 $\pm$ 0.005
	+BOD	15.867 $\pm$ 1.793	0.761 $\pm$ 0.033	11.740 $\pm$ 2.156	0.479 $\pm$ 0.303	28.800 $\pm$ 3.910	0.827 $\pm$ 0.049	23.114 $\pm$ 3.816	0.209 $\pm$ 0.252

Table 3. Ablation Study

Panel of Figure 6 — Figure 6. Comparison of different diffusion step.

Method	Time (seconds)
A2C	7512.144
SAC4IF	6754.545
DNaIR	5934.623
HRL	8462.525
RCE	29919.296
DSRM-HRL (Ours)	15909.175

Table 4. Training time comparison on KuaiRec (Max Len=30, Episode=20000).

Panel of Figure 7 — Figure 7. Training dynamics on KuaiRec and KuaiRand-Pure (Max Len = 30). We report learning curves of Interaction Length (left) and AD (right) over 20,000 training steps averaged across three random seeds. DSRM-HRL exhibits smoother convergence and lower variance than baselines, indicating improved training stability in non-stationary environments.

Fairness in Recommender Systems

See all 15 papers

Proactive Guiding Strategy for Item-side Fairness in Interactive Recommendation
Mar 3, 2026Chongjun Xia, Xiaoyu Shi, Hong Xie +3Long-Tail Recommendation Fairness in Recommender Systems
Chinese Academy of Sciences · University of Technology Sydney · University of Science and Technology of China
Breaking the Filter Bubble: A Semantic Pareto-DQN Framework for Multi-Objective Recommendation
Jun 23, 2026Cláudio Lúcio Do Val Lopes, Lucca Machado da Silva, André de Oliveira BrandãoMulti-Objective Reinforcement Learning Algorithmic Fairness
A3Data - MG, Brazil
Hierarchical Latent Reasoning for LLM-based Recommendation
Jul 30, 2026Peiyu Hu, Siying Gu, Weihai Lu +8Sequential Recommendation RL for Language Model Reasoning
1Xi’an Jiaotong-Liverpool University · 2Xiaohongshu · 3Peking University +1

Mar 3, 2026J/K move · Enter open · S save

Proactive Guiding Strategy for Item-side Fairness in Interactive Recommendation

Chongjun Xia, Xiaoyu Shi, Hong Xie, Xianzhi Wang+2

Chinese Academy of Sciences · University of Technology Sydney · University of Science and Technology of China

Long-Tail Recommendation Fairness in Recommender Systems Recommender Systems Algorithmic Fairness

Open paper AskPDF arXiv

Item-side fairness is crucial for ensuring the fair exposure of long-tail items in interactive recommender systems. Existing approaches promote the exposure of long-tail items by directly incorporating them into recommended results. This causes misalignment between user preferences and the recommended long-tail items, which hinders long-term user engagement and reduces the effectiveness of recommendations. We aim for a proactive fairness-guiding strategy, which actively guides user preferences toward long-tail items while preserving user satisfaction during the interactive recommendation process. To this end, we propose HRL4PFG, an interactive recommendation framework that leverages hierarchical reinforcement learning to guide user preferences toward long-tail items progressively. HRL4PFG operates through a macro-level process that generates fairness-guided targets based on multi-step feedback, and a micro-level process that fine-tunes recommendations in real time according to both these targets and evolving user preferences. Extensive experiments show that HRL4PFG improves cumulative interaction rewards and maximum user interaction length by a larger margin when compared with state-of-the-art methods in interactive recommendation environments.

Abstract

Fairness in Recommender Systems

Explore similar work

Proactive Guiding Strategy for Item-side Fairness in Interactive Recommendation

Breaking the Filter Bubble: A Semantic Pareto-DQN Framework for Multi-Objective Recommendation

Hierarchical Latent Reasoning for LLM-based Recommendation