Multimodal recommendation is commonly framed as a feature fusion problem, where textual and visual signals are combined to better model user preference. However, the effectiveness of multimodal recommendation may depend not only on how modalities are fused, but also on whether item content is represented in a semantic space aligned with preference matching. This issue is particularly important because raw visual features often preserve appearance similarity, while user decisions are typically driven by higher-level semantic factors such as style, material, and usage context. Motivated by this observation, we propose LVLM-grounded Multimodal Semantic Representation for Recommendation (VLM4Rec), a lightweight framework that organizes multimodal item content through semantic alignment rather than direct feature fusion. VLM4Rec first uses a large vision-language model to ground each item image into an explicit natural-language description, and then encodes the grounded semantics into dense item representations for preference-oriented retrieval. Recommendation is subsequently performed through a simple profile-based semantic matching mechanism over historical item embeddings, yielding a practical offline-online decomposition. Extensive experiments on multiple multimodal recommendation datasets show that VLM4Rec consistently improves performance over raw visual features and several fusion-based alternatives, suggesting that representation quality may matter more than fusion complexity in this setting. The code is released at https://github.com/tyvalencia/enhancing-mm-rec-sys.
Figures & tables
Figure 1. Overview of the VLM4Rec framework. Multimodal item content (images, titles, and user–item interactions) is processed through multiple representation paths: the proposed LVLM grounding path (top), which transforms item images into rich semantic descriptions via LLaVA-NeXT 7B and encodes them with Sentence-BERT. Baseline fusion and learned combination methods including concatenation, gating, and attention (middle), and SMORE spectral fusion (top). All embedding variants feed into the same lightweight retrieval stage, where user profiles are constructed via mean pooling and candidates are ranked by cosine similarity.
Model
Recall@5
Recall@10
Recall@20
NDCG@10
NDCG@20
Hit@10
Improv. over BERT
Text-Only Models
LLaVA-NeXT 7B
0.289
0.354
0.394
0.274
0.285
0.370
+54.9%
BERT Text-Only
0.198
0.228
0.269
0.184
0.198
0.320
baseline
LLaVA-based Fusion Variants
LLaVA + Attention
0.256
0.310
0.367
0.250
0.265
0.325
+35.8%
LLaVA + Concatenation
0.173
0.283
0.329
0.176
0.189
0.304
+23.7%
Table 1. Recommendation performance across all twelve model variants. LLaVA-NeXT 7B text-only achieves the best overall performance, reaching 0.354 Recall@10, which is a 54.9% improvement over the BERT text-only baseline. Notably, the text-only LLaVA variant outperforms all multimodal fusion approaches, suggesting that LVLM-generated descriptions already capture rich visual semantics in a highly recommendation-friendly form.
Figure 2. Full test set comparison with uncertainty. Horizontal bars show mean Recall@10 and NDCG@10 performance across all seven models evaluated on the complete dataset (23,318 users); error bars denote user-level standard deviation. BERT text-only remains strongest on the full test set, while attention fusion is the closest multimodal alternative.
Figure 3. LLaVA-covered subset metric heatmap. Rows grouped by model type; columns show Recall@5, @10, @20, NDCG@10, @20, and Hit@10. Color intensity indicates performance (higher is better). LLaVA-based representations consistently outperform BERT-based variants across all metrics.
Figure 4. Modality contribution profile across metrics. Line plot showing performance trajectories for text-only, vision-only, naive fusion, and learned fusion strategies. Across all metrics, text-only remains strongest; learned fusion tracks closely and clearly improves over naive fusion, while vision-only lags throughout.
Multimodal Vision-Language Models (VLMs) have demonstrated remarkable capabilities in cross-modal understanding, yet a fundamental challenge persists when applying them to recommendation: as representations propagate through multi-step reasoning, both visual and textual signals progressively attenuate - a phenomenon we term cross-modal dilution. To address this, we propose LARK (Latent-Aligned Reasoning frameworK), a two-stage latent reasoning framework with complementary alignment mechanisms within a single VLM. In the first stage, learnable latent tokens are interleaved with multi-step chain-of-thought (CoT) reasoning and explicitly aligned with a frozen vision encoder, serving as visual checkpoints that preserve perceptual details throughout the reasoning chain. In the second stage, the latent representations are projected via a bridge MLP and trained with item-to-item contrastive learning; to prevent the reasoning semantics from fading, intermediate features are aligned with the CoT hidden states from the first stage, anchoring the final embeddings to the model's own reasoning output. Experiments on three public benchmarks and one industrial dataset show that LARK achieves state-of-the-art performance across multiple recommendation architectures, with controlled ablations confirming the distinct contribution of each component.
Sequential Recommendation (SR) in multimodal settings typically relies on small frozen pretrained encoders, which limits semantic capacity and prevents Collaborative Filtering (CF) signals from being fully integrated into item representations. Inspired by the recent success of Large Language Models (LLMs) as high-capacity embedders, we investigate the use of Vision-Language Models (VLMs) as CF-aware multimodal embedders for SR. However, we find that standard contrastive Supervised Fine-Tuning (SFT), used to adapt VLMs for embedding generation and inject CF signals, can amplify inherent modality imbalance: optimization becomes dominated by one modality while the other degrades, ultimately undermining recommendation accuracy. To address this, we propose VLM2Rec, a VLM embedder-based framework for multimodal sequential recommendation designed to promote balanced modality utilization. Specifically, we introduce Weak-modality Penalized Contrastive Learning to mitigate gradient imbalance during optimization and cross-modal Relational Topology Regularization to preserve geometric consistency between modalities. Experiments show that VLM2Rec consistently improves over strong baselines in both accuracy and robustness across diverse scenarios.
Junyoung Kim, Woojoo Kim, Wonbin Kweon +3
Pohang University of Science and Technology · University of Illinois Urbana-Champaign
While multi-modal Visual Language Models (VLMs) have demonstrated significant success across various domains, the integration of VLMs into recommendation and retrieval systems remains a challenge, due to issues like training objective discrepancies and serving efficiency bottlenecks. This paper introduces PinCLIP, a large-scale visual representation learning approach developed to enhance retrieval and ranking models at Pinterest by leveraging VLMs to learn image-text alignment. We propose a novel hybrid Vision Transformer architecture that utilizes a VLM backbone and a hybrid fusion mechanism to capture multi-modality content representation at varying granularities. Beyond standard image-to-text alignment objectives, we introduce a neighbor alignment objective to model the cross-fusion of multi-modal representations within the Pinterest Pin-Board graph. Offline evaluations show that PinCLIP outperforms state-of-the-art baselines, such as Qwen, by 20% in multi-modal retrieval tasks. Online A/B testing demonstrates significant business impact, including substantial engagement gains across all major surfaces in Pinterest. Notably, PinCLIP significantly addresses the "cold-start" problem, enhancing fresh content distribution with a 15% Repin increase in organic content and 8.7% higher click for new Ads.