Aligning Multimodal Sequential Recommendations via Robust Direct Preference Optimization with Sparse MoE

Hejin Huang; Jusheng Zhang; Kaitong Cai; Jian Wang; Rong Pan

cs.IRMar 31, 2026

Aligning Multimodal Sequential Recommendations via Robust Direct Preference Optimization with Sparse MoE

Authors: Hejin Huang, Jusheng Zhang, Kaitong Cai, Jian Wang, Rong Pan

Organizations: Sun Yat-sen University · Snap Inc

Abstract

Preference-based alignment objectives have been widely adopted, from RLHF-style pairwise learning in large language models to emerging applications in recommender systems. Yet, existing work rarely examines how Direct Preference Optimization (DPO) behaves under implicit feedback, where unobserved items are not reliable negatives. We conduct systematic experiments on multimodal sequential recommendation to compare common negative-selection strategies and their interaction with DPO training. Our central finding is that a simple modification, replacing deterministic hard negatives with stochastic sampling from a dynamic top-K candidate pool, consistently improves ranking performance. We attribute its effectiveness to two factors: (1) reducing erroneous suppressive gradients caused by false negatives, and (2) retaining informative hard signals while smoothing optimization via controlled stochasticity. With an optional sparse Mixture-of-Experts encoder for efficient capacity scaling, RoDPO achieves up to 5.25% NDCG@5 on three Amazon benchmarks, with nearly unchanged inference cost.

Figures & tables

Figure 1: The “False Negative” Dilemma in RecSys DPO. Left: In NLP, negatives are strictly incorrect, making DPO effective. Middle: In RecSys, hard negatives may include unobserved positives, causing mis-penalization and embedding drift. Right: Top- $K$ sampling adds randomness among candidates, balancing positives.

Figure 2: Overall framework of RoDPO . (a) Multimodal Encoder : Item IDs and text/image features are embedded into a shared space, refined by a sparse MoE, and modeled by modality-specific Transformers with a temporal MoE for next-item prediction. (b) Robust DPO : Preference pairs are formed by the observed item $y_{w}$ and a sampled negative $y_{l}$ from a dynamic top- $K$ pool (excluding $y_{w}$ ), optimized against a frozen reference policy.

Method	Toys and Games				Beauty				Home and Kitchen
	NDCG		MRR		NDCG		MRR		NDCG		MRR
	@5	@10	@5	@10	@5	@10	@5	@10	@5	@10	@5	@10
Traditional Non-time-aware Methods
GRU4Rec	0.0236	0.0289	0.0201	0.0222	0.0271	0.0337	0.0223	0.0260	0.0067	0.0087	0.0056	0.0064
SASRec	0.0348	0.0411	0.0257	0.0295	0.0325	0.0415	0.0246	0.0283	0.0118	0.0148	0.0089	0.0100
LRURec	0.0362	0.0446	0.0278	0.0312	0.0323	0.0412	0.0250	0.0286	0.0112	0.0141	0.0084	0.0096

Table 1: Overall performance comparison. The best results are bolded, and the second best are underlined. Improvements are calculated against the best baseline.

\dagger

denotes that the corresponding method is reproduced by ourselves.

Sampling Strategy	NDCG@5	MRR@5
Random Sampling	0.0491	0.0437
Hard Sampling (Argmax)	0.0497	0.0438
Top-K Sampling (Ours)	0.0521	0.0463

Table 2: Ablation study on negative sampling strategies.

Variant	NDCG@5	MRR@5
Full Model (RoDPO)	0.0521	0.0463
w/o Sparse MoE	0.0516	0.0459
w/o DPO (Baseline)	0.0495	0.0430

Table 3: Ablation on model components.

Panel of Figure 3 — Figure 3: Effect of Top- $k$ Negative Sampling and DPO Coefficient $\beta$ on NDCG@5 and MRR@5.

Model	Trainable Params	Training	Inference	NDCG@5
	(M)	(s/epoch)	(s/batch)
HM4SR	1.59	9.81	0.42	0.0495
RoDPO	1.60	9.84	0.45	0.0521

Table 4: Efficiency comparison. “Inference Latency” denotes the time to process a batch of size 2048.

Figure 4: Kernel Density Estimation (KDE) of predicted logit distributions for positive items (

y_{w}

) and hard negative items (

y_{l}

) on the Amazon Toys and Games dataset.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Datasets	Toys and Games	Beauty	Home and Kitchen
# Users	19,412	22,363	66,520
# Items	11,924	12,101	28,238
# Actions	167,597	198,502	551,682
Avg. Actions/User	8.63	8.88	8.29
Avg. Actions/Item	14.06	16.40	19.54
# Sparsity	99.93%	99.93%	99.97%

Appendix

Table 5: Statistics of processed datasets.

Recommender Systems

Recommender systems predict and rank items for people using interaction histories, content, context, and feedback.

See all 508 papers

DynamicPO: Dynamic Preference Optimization for Recommendation
May 1, 2026Xingyu Hu, Kai Zhang, Jiancan Wu +7Hard Negative Mining Preference Optimization
University of Science and Technology of China, Hefei, China · Shanghai Innovation Institute, Shanghai, China · Meituan, Chengdu, China
MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization
May 11, 2026Rohan Surana, Xintong Li, Sheldon Yu +7Direct Preference Optimization Active Learning
1UC San Diego · 2Adobe Research
Beyond Uniform Forgetting: A Study of Sequential Direct Preference Optimization Across Preference Settings
Jun 18, 2026Pranav Bhandari, Nicolas Fay, Amitava Datta +2Continual Learning for LLMs Direct Preference Optimization
1Network Analysis and Social Influence Modelling (NASIM) Lab · School of Physics Maths and Computing, The University of Western Australia · School of Psychological Science, The University of Western Australia +1

May 1, 2026J/K move · Enter open · S save

DynamicPO: Dynamic Preference Optimization for Recommendation

Xingyu Hu, Kai Zhang, Jiancan Wu, Shuli Wang+6

University of Science and Technology of China, Hefei, China · Shanghai Innovation Institute, Shanghai, China · Meituan, Chengdu, China

Hard Negative Mining Preference Optimization Large Language Model-Based Recommendation User Preference Modeling

Open paper AskPDF arXiv

In large language model (LLM)-based recommendation systems, direct preference optimization (DPO) effectively aligns recommendations with user preferences, requiring multi-negative objective functions to leverage abundant implicit-feedback negatives and sharpen preference boundaries. However, our empirical analyses reveal a counterintuitive phenomenon, preference optimization collapse, where increasing the number of negative samples can lead to performance degradation despite a continuously decreasing training loss. We further theoretically demonstrate that this collapse arises from gradient suppression, caused by the dominance of easily discriminable negatives over boundary-critical negatives that truly define user preference boundaries. As a result, boundary-relevant signals are under-optimized, weakening the model's decision boundary. Motivated by these observations, we propose DynamicPO (Dynamic Preference Optimization), a lightweight and plug-and-play framework comprising two adaptive mechanisms: Dynamic Boundary Negative Selection, which identifies and prioritizes informative negatives near the model's decision boundary, and Dual-Margin Dynamic beta Adjustment, which calibrates optimization strength per sample according to boundary ambiguity. Extensive experiments on three public datasets show that DynamicPO effectively prevents optimization collapse and improves recommendation accuracy on multi-negative preference optimization methods, with negligible computational overhead. Our code and datasets are available at https://github.com/xingyuHuxingyu/DynamicPO.

Abstract

Recommender Systems

Explore similar work

DynamicPO: Dynamic Preference Optimization for Recommendation

MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization

Beyond Uniform Forgetting: A Study of Sequential Direct Preference Optimization Across Preference Settings