VLM2Rec: Resolving Modality Collapse in Vision-Language Model Embedders for Multimodal Sequential Recommendation

Junyoung Kim; Woojoo Kim; Wonbin Kweon; Jaehyung Lim; Dongha Kim; Hwanjo Yu

cs.IRMar 18, 2026

VLM2Rec: Resolving Modality Collapse in Vision-Language Model Embedders for Multimodal Sequential Recommendation

Authors: Junyoung Kim, Woojoo Kim, Wonbin Kweon, Jaehyung Lim, Dongha Kim, Hwanjo Yu

Organizations: Pohang University of Science and Technology · University of Illinois Urbana-Champaign

Abstract

Sequential Recommendation (SR) in multimodal settings typically relies on small frozen pretrained encoders, which limits semantic capacity and prevents Collaborative Filtering (CF) signals from being fully integrated into item representations. Inspired by the recent success of Large Language Models (LLMs) as high-capacity embedders, we investigate the use of Vision-Language Models (VLMs) as CF-aware multimodal embedders for SR. However, we find that standard contrastive Supervised Fine-Tuning (SFT), used to adapt VLMs for embedding generation and inject CF signals, can amplify inherent modality imbalance: optimization becomes dominated by one modality while the other degrades, ultimately undermining recommendation accuracy. To address this, we propose VLM2Rec, a VLM embedder-based framework for multimodal sequential recommendation designed to promote balanced modality utilization. Specifically, we introduce Weak-modality Penalized Contrastive Learning to mitigate gradient imbalance during optimization and cross-modal Relational Topology Regularization to preserve geometric consistency between modalities. Experiments show that VLM2Rec consistently improves over strong baselines in both accuracy and robustness across diverse scenarios.

Figures & tables

Figure 1: Analysis of modality collapse via dropout test (Left, a) and gradient dynamics (Right, b) . SFT amplifies the performance gap between text-only and vision-only inputs and steers the fused update toward the text gradient, while VLM2Rec reduces this gap by rebalancing modality gradients.

Metric	Modality	Beauty			MovieLens
Metric	Modality	$\text{VLM}_{\text{Vanilla}}$	$\text{VLM}_{\text{SFT}}$	VLM2Rec (Ours)	$\text{VLM}_{\text{Vanilla}}$	$\text{VLM}_{\text{SFT}}$	VLM2Rec (Ours)
$U$ ( $\downarrow$ )	Fused	-0.3984	-1.7969	-1.9921	-0.2275	-0.8203	-4.6562
	Vision	-0.0237	-0.0398	-0.2031	-0.0236	-0.0078	-1.7187
	Text	-0.3691	-1.7422	-1.6718	-0.2177	-0.7891	-1.1788
$S$ ( $\uparrow$ )	Fused	1.0340	1.1910	1.1870	1.0117	1.2471	1.2263
	Vision	1.0071	1.0000	1.2134	1.0035	0.9998	1.1134

Table 1: VLM representation geometry (Uniformity

U

, Separability

S

) across three states.

Figure 2: Overview of VLM2Rec. Left : VLM2Rec pretraining and the initialization of SR backbone item embeddings. Right : Proposed training objectives. $\mathcal{L}_{\text{WPCL}}$ adaptively penalizes the weak modality to improve positive-negative discrimination, while $\mathcal{L}_{\text{CRTR}}$ aligns cross-modal relational topology to promote modality-consistent neighborhoods.

Baselines	FM	Seq.CF	MM	Emb.tune	Bal.
ID-based [ 13 , 18 , 68 , 27 ]		✓		✓
BERT [ 8 ]				✓
CLIP [ 41 ]			✓	✓
SLIM [ 56 ]		✓
SLIM + [ 56 ]	✓	✓
LLMEmb [ 31 ]	✓	$\triangle$		✓

Table 2: Positioning against baselines. ✓: yes,

\triangle

: partial/limited.

Data	Metric	ID	Text-based Models							Multimodal-based Models					Ours
Data	Metric	Base	BERT	SLIM	$\text{SLIM}^{+}$	LLMEmb	LLM2Rec	$\text{LLM}_{\text{Van}}$	$\text{LLM}_{\text{SFT}}$	CLIP	$\text{VLM}_{\text{Prm}}$	Note-2	$\text{VLM}_{\text{Van}}$	$\text{VLM}_{\text{SFT}}$	VLM2Rec	Imp.(%)
Backbone: SASRec
Toys	H@10	0.0904	0.0702	0.0733	0.0957	0.0982	0.1005	0.0808	0.1012	0.0728	0.0796	0.1049	0.0848	0.1063	0.1183*	11.29
	H@20	0.1226	0.1039	0.1082	0.1340	0.1374	0.1403	0.1144	0.1412	0.1054	0.1130	0.1436	0.1182	0.1459	0.1589*	8.91
	N@10	0.0524	0.0391	0.0411	0.0559	0.0574	0.0594	0.0460	0.0619	0.0408	0.0450	0.0598	0.0493	0.0603	0.0683*	10.34
	N@20	0.0604	0.0473	0.0495	0.0657	0.0671	0.0687	0.0537	0.0710	0.0483	0.0529	0.0687	0.0574	0.0692	0.0785*	10.56

Table 3: Performance on downstream SR backbone. The best and second-best results are highlighted by boldface and underline . * indicates statistical significance (

p<0.05

, paired t-test over 5 seeds).

Figure 3: Ablation study of VLM2Rec.

Figure 4: Intrinsic CF signal analysis.

Model	Input	Beauty			Toys
Model	Input	N@20	Train	Emb	N@20	Train	Emb
LLMEmb	Item	0.0563	22	58	0.0671	15	38
LLM2Rec	Seq./Item	0.0588	33	58	0.0687	26	39
$\text{LLM}_{\text{SFT}}$	Seq.	0.0604	35	56	0.0710	26	39
NoteLLM-2	Item	0.0621	26	132	0.0687	20	91
$\text{VLM}_{\text{SFT}}$	Seq.	0.0627	117	113	0.0692	92	79

Table 6: Few-shot performance and efficiency analysis.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	#Users	#Items	#Interactions	Avg. Length	Sparsity (%)
Toys	15,921	8,383	108,336	6.8	99.92
Beauty	19,757	9,311	137,300	6.9	99.93
Clothing	30,757	17,087	196,614	6.4	99.96
Sports	32,127	14,820	222,591	6.9	99.95
MovieLens	6,033	2,177	58,234	9.65	99.56

Appendix

Table 7: Statistics of datasets used in the experiments.

Figure 7: Additional analysis of modality collapse on Toys and Clothing via dropout test (a) and gradient dynamics (b) . Clothing is included as a domain where visual information is known to be highly informative in multimodal recommendation.

Metric	Modality	Toys			Clothing
Metric	Modality	$\text{VLM}_{\text{Vanilla}}$	$\text{VLM}_{\text{SFT}}$	VLM2Rec (Ours)	$\text{VLM}_{\text{Vanilla}}$	$\text{VLM}_{\text{SFT}}$	VLM2Rec (Ours)
$U$ ( $\downarrow$ )	Fused	-0.4707	-2.2500	-1.7969	-0.4336	-1.8359	-1.9951
	Vision	-0.0237	-0.0039	-0.5703	-0.2368	-0.0237	-1.2136
	Text	-0.4453	-2.2188	-0.9335	-0.4102	-1.7187	-1.3759
$S$ ( $\uparrow$ )	Fused	1.0609	1.2418	1.1609	1.0444	1.0474	1.1274
	Vision	1.0039	0.9911	1.1551	1.0110	0.9997	1.0706

Appendix

Table 8: Comparison of VLM representation geometry across three states. Metrics include uniformity (

U

), and separability (

S=A_{\text{neg}}/A_{\text{pos}}

VLM Backbone	$\text{VLM}_{\text{SFT}}$				VLM2Rec
VLM Backbone	t2f	v2f	f2f	Gap (%)	t2f	v2f	f2f	Gap (%)
InternVL3-2B	0.0614	0.0558	0.0608	10.04	0.0703	0.0686	0.0702	2.48
Qwen2.5-VL-7B	0.0636	0.0588	0.0632	8.16	0.0746	0.0733	0.0754	1.77
LLaVA-1.5-7B	0.0625	0.0572	0.0620	9.27	0.0733	0.0725	0.0742	1.10
InternVL3-8B	0.0645	0.0597	0.0636	8.04	0.0744	0.0727	0.0750	2.34
Qwen2.5-VL-32B	0.0653	0.0612	0.0647	6.70	0.0761	0.0744	0.0776	2.28

Appendix

Table 9: Modality dropout test across VLM backbones (Beauty, SASRec, N@20).

Data	Metric	ID	Text-based Models							Multimodal-based Models					Ours
Data	Metric	Base	BERT	SLIM	$\text{SLIM}^{+}$	LLMEmb	LLM2Rec	$\text{LLM}_{\text{Van}}$	$\text{LLM}_{\text{SFT}}$	CLIP	$\text{VLM}_{\text{Prm}}$	Note-2	$\text{VLM}_{\text{Van}}$	$\text{VLM}_{\text{SFT}}$	VLM2Rec	Imp(%)
Backbone: FMLP
Toys	H@10	0.0840	0.0653	0.0680	0.0889	0.0912	0.0933	0.0738	0.0945	0.0676	0.0752	0.0974	0.0788	0.0988	0.1097*	11.03
	H@20	0.1138	0.0965	0.1005	0.1245	0.1276	0.1303	0.1049	0.1310	0.0979	0.1063	0.1335	0.1098	0.1356	0.1513*	11.58
	N@10	0.0487	0.0363	0.0381	0.0520	0.0533	0.0551	0.0418	0.0575	0.0378	0.0427	0.0558	0.0458	0.0564	0.0626*	8.87
	N@20	0.0561	0.0440	0.0460	0.0610	0.0623	0.0638	0.0491	0.0660	0.0448	0.0499	0.0638	0.0533	0.0643	0.0731*	10.76

Appendix

Table 10: Performance on additional SR backbones. The best and second-best results are highlighted by boldface and underline . * indicates statistical significance (

p<0.05

, paired t-test over 5 seeds).

Dataset	In-batch	Random	SASRec Top- $k$	Text Sim.	Image Sim.
Clothing	0.0347	0.0342	0.0352	0.0357	0.0360
MovieLens	0.1205	0.1193	0.1218	0.1222	0.1229

Appendix

Table 11: Effect of negative sampling strategies for modality gap estimation (SASRec, N@20).

Embedding	UniSRec		TedRec
Embedding	Clothing	MovieLens	Clothing	MovieLens
BERT	0.0308	0.1086	0.0332	0.1167
$\text{LLM}_{\text{SFT}}$	0.0325	0.1119	0.0348	0.1180
$\text{VLM}_{\text{SFT}}$ (Text)	0.0328	0.1124	0.0355	0.1189
Ours (Text)	0.0336	0.1137	0.0362	0.1208
Embedding	M3SRec		HM4SR

Appendix

Table 12: Modality embedding replacement experiments on state-of-the-art models.

Data	Group	Weak	Ratio	$\Delta_{u,\mathrm{gap}}^{\text{SFT}}$	$\Delta_{u,\mathrm{gap}}^{\text{Ours}}$	User Red.
MovieLens	All	All	1.0000	0.3658	0.0522	0.9362
		Text	0.2728	0.2798	0.0228	0.9064
		Vision	0.7272	0.3981	0.0632	0.9473
	Collapse	All	1.0000	0.6255	0.0684	1.0000
		Text	0.2090	0.5497	0.0243	1.0000
		Vision	0.7910	0.6456	0.0801	1.0000

Appendix

Table 13: Anchor weak-modality split across the All and Collapse groups for each dataset. Ratio reports the fraction of users assigned to each anchor weak modality.

Figure 8: Top-5 prediction lists of each model for the case study.

Figure 9: Model scalability across VLM families and parameter sizes.

Objective	Time Complexity	Space Complexity	In-batch Time	In-batch Space
$\mathcal{L}_{\text{SFT}}$	$O\!\left(2B(N{+}1)d\right)$	$O\!\left(B(N{+}1)\right)$	$O(2B^{2}d)$	$O(B^{2})$
$\mathcal{L}_{\text{WPCL}}$	$O\!\left(6B(N{+}1)d\right)$	$O\!\left(B(N{+}1)\right)$	$O(6B^{2}d)$	$O(B^{2})$
$\mathcal{L}_{\text{CRTR}}$	$O(4BCd)$	$O(BC)$	$O(4B^{2}d)$	$O(B^{2})$

Appendix

Table 14: Time and space complexity comparison of the loss objectives. Here, space complexity denotes additional scratch memory required by the loss computation, excluding backbone activations.

Figure 10: Impact of hyperparameters

\tau_{\text{WPCL}},\tau_{\text{CRTR}}

, and

\lambda

Multimodal Recommendation

Multimodal recommendation combines user behavior with item content such as images, text, audio, and video to personalize rankings.

See all 48 papers

VLM4Rec: Multimodal Semantic Representation for Recommendation with Large Vision-Language Models
Mar 13, 2026Ty Valencia, Burak Barlas, Varun Singhal +2Multimodal Recommendation Multimodal Embedding
University of Southern California Los Angeles, USA
Latent-Aligned Reasoning for Multimodal Recommendation
Sep 4, 2026Jiarui Jin, Anyang JiMultimodal Recommendation VLM Reasoning
Xiaohongshu Inc. · Nanjing University
Multimodal Graph Retrieval-Augmented Sequential Recommendation via Collaborative Filtering Paths
Oct 8, 2026Jason Marcell Setiadi, Xin Cao, Lina YaoRecommender Systems Sequential Recommendation
University of New South Wales, Sydney, Australia

Mar 13, 2026J/K move · Enter open · S save

VLM4Rec: Multimodal Semantic Representation for Recommendation with Large Vision-Language Models

Ty Valencia, Burak Barlas, Varun Singhal, Ruchir Bhatia+1

University of Southern California Los Angeles, USA

Multimodal Recommendation Multimodal Embedding

Open paper AskPDF arXiv

Multimodal recommendation is commonly framed as a feature fusion problem, where textual and visual signals are combined to better model user preference. However, the effectiveness of multimodal recommendation may depend not only on how modalities are fused, but also on whether item content is represented in a semantic space aligned with preference matching. This issue is particularly important because raw visual features often preserve appearance similarity, while user decisions are typically driven by higher-level semantic factors such as style, material, and usage context. Motivated by this observation, we propose LVLM-grounded Multimodal Semantic Representation for Recommendation (VLM4Rec), a lightweight framework that organizes multimodal item content through semantic alignment rather than direct feature fusion. VLM4Rec first uses a large vision-language model to ground each item image into an explicit natural-language description, and then encodes the grounded semantics into dense item representations for preference-oriented retrieval. Recommendation is subsequently performed through a simple profile-based semantic matching mechanism over historical item embeddings, yielding a practical offline-online decomposition. Extensive experiments on multiple multimodal recommendation datasets show that VLM4Rec consistently improves performance over raw visual features and several fusion-based alternatives, suggesting that representation quality may matter more than fusion complexity in this setting. The code is released at https://github.com/tyvalencia/enhancing-mm-rec-sys.

Abstract

Multimodal Recommendation

Explore similar work

VLM4Rec: Multimodal Semantic Representation for Recommendation with Large Vision-Language Models

Latent-Aligned Reasoning for Multimodal Recommendation

Multimodal Graph Retrieval-Augmented Sequential Recommendation via Collaborative Filtering Paths