While multi-modal Visual Language Models (VLMs) have demonstrated significant success across various domains, the integration of VLMs into recommendation and retrieval systems remains a challenge, due to issues like training objective discrepancies and serving efficiency bottlenecks. This paper introduces PinCLIP, a large-scale visual representation learning approach developed to enhance retrieval and ranking models at Pinterest by leveraging VLMs to learn image-text alignment. We propose a novel hybrid Vision Transformer architecture that utilizes a VLM backbone and a hybrid fusion mechanism to capture multi-modality content representation at varying granularities. Beyond standard image-to-text alignment objectives, we introduce a neighbor alignment objective to model the cross-fusion of multi-modal representations within the Pinterest Pin-Board graph. Offline evaluations show that PinCLIP outperforms state-of-the-art baselines, such as Qwen, by 20% in multi-modal retrieval tasks. Online A/B testing demonstrates significant business impact, including substantial engagement gains across all major surfaces in Pinterest. Notably, PinCLIP significantly addresses the "cold-start" problem, enhancing fresh content distribution with a 15% Repin increase in organic content and 8.7% higher click for new Ads.
Figures & tables
Figure 1. Overview of the PinCLIP fusion model architecture, highlighting the design of the Hybrid Vision Transformer backbone and the multimodal contrastive objectives for jointly learning image-text alignment and Pin-to-Pin alignment.
Figure 2. Illustration of the text-image dataset. Each image (“Pin”) is associated with multiple text signals. We use longer-form descriptive text signals (title, description, image caption) and shorter-form keyword text signals (navboost search query, annotations).
Figure 4. Retrieval results (Recall@K) for the key evaluation tasks of PinText, Related Pins, and Search.
Image Encoder
Text Encoder
R@1
R@5
R@10
Image-to-Text Retrieval
Hybrid-ViT-B (130M)
Transformer-B (278M)
56.2
71.1
75.7
Hybrid-ViT-B (130M)
Transformer-L (700M)
60.3
73.9
78.3
Hybrid-ViT-g (922M)
Transformer-B (278M)
67.2
79.3
82.7
Hybrid-ViT-g (922M)
Transformer-L (700M)
70.3
81.8
84.9
Table 1. Assessment of the impact of model scaling per modality on PinText image-to-text retrieval performance.
Figure 5. Assessment of the impact of dataset scaling on PinText image-to-text retrieval performance.
Figure 6. Assessment of the impact of freezing image encoder layers on PinText image-to-text retrieval performance.
Locked Layers
GPU Memory (GB)
Throughput (img/s)
0
69.37
1223
12
58.19 (-16.1%)
1418 (+15.9%)
24
48.67 (-29.8%)
1572 (+28.5%)
36
41.38 (-40.3%)
1658 (+35.5%)
Table 2. Assessment of the impact of freezing image encoder layers on 1-node training efficiency (memory/throughput).
Online Metric
Homefeed
Related Pins
Search
Surface Repins
+ 0.91%
+ 1.84%
+ 0.96%
Surface Fresh Repins
+ 5.02%
+ 14.15%
+ 15.35%
Table 3. Online A/B tests by adding PinCLIP as a feature into the ranking models in Homefeed, Related Pins and Search.
Online Metric
Improvement
Click-Through Rate (CTR)
+ 5.02%
Cost Per Click (CPC)
- 0.59%
New Ads (14d) Click Volume
+ 8.67%
Table 4. Online A/B tests by adding PinCLIP as a feature into the Ads ranking model.
Related Pins
Search
Sitewide Repins
Sitewide Fresh Repins
Fulfillment Rate
+ 0.36%
+ 2.3%
+ 0.34%
Table 5. Online A/B tests by using PinCLIP-based candidate generator in Related Pins and Search.
Figure 7. Qualitative comparison of PinCLIP and OmniSearchSage ( Agarwal et al., 2024 ) retrieved candidates on production-scale search corpus. Relevant candidates are highlighted in blue, whereas irrelevant candidates are highlighted in red. PinCLIP produces more semantically similar results compared to competitive baselines for both using text queries or image as inputs.