From Sessions to Serving: Integrating LLM-Based Embeddings into Retrieval for Production Content Recommendation
Organizations: Autodesk, San Francisco, CA, USA
Abstract
Personalized content recommendation on large marketplace websites must reconcile heterogeneous candidate content (e.g., customer stories, product pages, industry pages, and learning content) with sparse, fast-changing user behavior, placing increasing demands on the retrieval stage to represent user interest well, not just the downstream ranking stage. We present LSER (LLM Session Embedding Retrieval), a production content recommendation system for autodesk.com that derives session- and history-level embeddings from a pretrained large language model (LLM) and adds them as auxiliary user-side features in the retrieval-stage user tower alongside existing learned sequence encoders. To keep user-side embeddings fresh under real-time behavior, we build a nearline serving path that asynchronously refreshes sequential and LLM-derived user vectors for our main content recommendation server. We validate the incremental value of the LLM features through an ablation study across our full multi-content-type catalog. Our results indicate that integrating LLM-derived embeddings as additional retrieval signals, together with production-grade nearline serving, is technically feasible and can improve model performances.
Figures & tables
| Metric | Popular products | Customer success |
|---|---|---|
| HR@1 | ||
| HR@5 | ||
| HR@10 | — | |
| MRR |
| Metric | Relative Lift |
|---|---|
| Engagement | |
| Homepage Prod&Ind. carousel CTR | +10.3% |
| AutoCAD customer story CTR | +36.1% |
| Conversion | |
| Product trial start rate | +17.1% |
| Product trial install rate | +10.5% |
Recommender Systems
Recommender systems predict and rank items for people using interaction histories, content, context, and feedback.
See all 578 papers