Generative recommendation (GR) is moving industrial recommender systems (RecSys) from scoring candidate items to autoregressively generating item identifiers. This shift promises higher recommendation quality, but makes serving substantially harder: each request executes a short sequence of dependent decoding steps that explore a large item space and select promising candidates on the latency-critical path. GR not only creates structural mismatches with conventional RecSys pipelines, but also challenges the architectural and optimization assumptions of LLM serving stacks, making direct adoption of existing systems difficult. We present RecoGEM, our production GR serving system. RecoGEM uses a runtime--engine decoupling architecture to move GR-specific serving responsibilities, such as candidate-state transitions and live artifact updates, to a dedicated runtime. Its execution engine exploits GR's short token sequences, large candidate populations, and predictable step shapes to optimize self-attention, candidate selection, and execution planning. RecoGEM has been deployed for over nine months across our production lines, serving applications with hundreds of millions of users and supporting more than ten GR model families. On our OneModel production workloads, RecoGEM reaches over 1,000 QPS per serving instance with P99 latency below 35 ms and a 2.88× throughput improvement over representative baselines.
Figures & tables
Figure 1. Traditional vs. generative recommendation. A cascaded DLRM workflow that scores a shrinking candidate set, beside a generative workflow in which one model decodes the item's semantic identifier.
Figure 2. LLM serving vs. GR serving. llm vs gr
Figure 3. Execution of a GR request. A two-part request diagram. Phase 1, context encoding, takes user static, short-term, and long-term pathways, passes them through a context processor, and produces a context representation or KV cache. Phase 2, autoregressive decoding, starts from BOS and repeatedly applies decoder blocks conditioned on the encoded context to generate SID tokens s1 through sL for one recommended item.
Figure 4. Heterogeneous transitions between decoding steps and online updates of serving artifacts. Left: a GR serving box in which a compute engine runs decoding step l with vocabulary V_l and emits expanded candidates; a selection stage such as beam search keeps the surviving candidates; their candidate state (prefix, score, and KV) becomes the next-step input to the compute engine at step l+1 with vocabulary V_{l+1}. Selection and candidate state sit in a shaded step-transition region outside the engine. Right: online updates of model weights and the SID vocabulary point to the compute engine, and selection-policy updates point to the selection stage.
Figure 5. Production online weight-refit activity in 24-hours: refit frequency (left) and FP32 refit traffic per server (right). Two bar charts over a 24-hour capture window. The left chart shows model weight refits per minute, with a peak of 107 refits per minute. The right chart shows FP32 traffic per server in gigabytes per hour, with a peak of 346 gigabytes per hour.
Figure 6. Attention dimension inversion in GR decode. A two-panel schematic. The left panel shows LLM decode attention as a long horizontal KV sequence tile. The right panel shows GR decode as many short row-local attention rows and a row-local Top-K selection over a candidate width.
Figure 7. Decoding steps in LLM vs. GR serving. A contrast between homogeneous token generation, which repeats the same decode transition from token to token, and heterogeneous GR decoding, where a decode step produces probabilities, tokens, and prefix indices and expands dimensions for the next step.
Figure 8. Candidate selection design choices. A two-panel figure. Panel (a) shows Top-K dimension inversion: LLM decode selects a small K, typically K<10, from a wide token vocabulary, whereas GR beam update selects a large K, typically K>128, from a comparatively narrow semantic-id vocabulary on every step. Panel (b) shows a throughput comparison between radix-based Top-K and bitonic Top-K kernels across K values from 16 to 1024, illustrating that a single sampling-tail kernel choice does not dominate across the K range exercised by GR beam update.
Figure 9. RecoGEM serving system overview. A simplified system architecture diagram. Dense and sparse embeddings and a configuration contract feed an infer-server binary. The binary orchestrates GPU-resident \sysengine execution, model weights, and custom kernels, with a loop arrow indicating iterative execution.
Figure 10. RecoGEM deployment contract. A diagram showing a model graph and deployment policy feeding a contract generator, which produces a deployment contract. The contract contains weight binding, input-output mapping, request batching, and pipeline flow records. The runtime server fetches the contract and uses it to control execution of the compute graph.
Figure 11. Runtime state management for a GR request. A timeline-style diagram showing a \sysbatch as contract-defined GPU tensor slices. Request inputs are copied into the batch buffer before enqueue. Context, self-KV, candidate ids, probabilities, and cache-indirection tensors are carried across decode steps by output-to-input mappings. Final outputs are copied to request buffers before the batch buffer is returned to the pool.
Figure 12. Sequence tiling versus row tiling. Sequence tiling splits one long attention row; row tiling groups R complete short rows per tile.
Figure 13. Fused attention dataflow and cross-tile pipelining. Top: fused per-row attention keeps intermediates on chip. Bottom: loading tile k+1 overlaps computation on tile k.
Figure 14. Step-granular execution planning lowers a single-step GR graph into step-specialized engines by probing per-step shapes and rebuilding under step-bounded profiles. A left-to-right planning pipeline. On the left, a dashed box labeled Training Graph stacks Item Embedding, N decoder blocks, and Candidate Selection, with a Conditional Boundaries box feeding back into the graph. The first Build stage turns this graph into a Broad Range Compute Graph. The Probe Steps stage, over generation steps t equals 1, 2, and onward, passes it through a Profile Optimizer that records per-step shapes. The second Build stage emits a stacked Step Heterogeneous Compute Graph specialized for each generation step.
Figure 15. Performance trend of OneRec-v2. Four curves over batch sizes 1 to 32 compare GeneralGen, \sys, TorchEager, and the adapted TensorRT-LLM runtime. The panels show P99 latency and QPS on logarithmic vertical axes.
Figure 16. Performance trend of OneModel. Four curves over batch sizes 1 to 32 compare GeneralGen, \sys, TorchEager, and the adapted TensorRT-LLM runtime with dynamic beam widths 128, 256, 512, and 1024. The panels show P99 latency and QPS on logarithmic vertical axes.
Figure 17. Batch-32 step-wise kernel breakdown for three serving systems in OneModel. Four groups of stacked bars compare GeneralGen, HomoGen, and \sysacross OneModel's four SID steps at batch size 32. Each bar separates attention, beam search, and other kernels. Each component is divided by \sys's total kernel latency at the same step, so every \sysbar totals one.
Figure 18. Decode self-attention latency for the GR short-prefix setting across batch and head sweeps. Six log-scale line plots arranged in a two-by-three grid, comparing PyTorch Native, SDPA, the earlier per-row \syspath, and the current \sysrow-tiled path across short key/value lengths. The top row fixes the number of heads at 32 and sweeps effective batch over 4096, 8192, and 16384; the bottom row repeats the sweep at 64 heads. \sysis the fastest backend in every panel.
Figure 19. Beam-search Top-K latency for GR-style candidate selection with vocabulary size 8192. A log-scale grouped bar chart comparing Top-K latency across request batch sizes 4, 8, 16, and 32 and beam widths 512 and 1024. \sys's shape-aware Top-K selection is the lowest bar in each group, while forced WarpBitonic becomes much slower at beam width 1024.
Figure 20. QPS and P99 latency over 24h. A dual-axis line plot showing QPS and P99 latency over a 24h period.
Recommendation system delivers substantial economic benefits by providing personalized predictions. Generative recommendation (GR) integrates LLMs to enhance the understanding of long user-item sequences. Despite employing attention-based architectures, GR's workload differs markedly from that of LLM serving. GR typically processes long prompt while producing short, fixed-length outputs, yet the computational cost of each decode phase is especially high due to the large beam width. Furthermore, since the beam search involves a vast item space, the sorting overhead becomes particularly time-consuming. We propose xGR, a GR-oriented serving system that meets strict low-latency requirements under high-concurrency scenarios. First, xGR unifies the processing of prefill and decode phases through staged computation and separated KV cache. Second, xGR enables early sorting termination and mask-based item filtering with data structure reuse. Third, xGR reconstructs the overall pipeline to exploit multi-level overlap and multi-stream parallelism. The experiments on real-world datasets demonstrate that xGR achieves at least 2.89x throughput compared to the state-of-the-art baseline under strict latency constraints.
Qingxiao Sun, Tongxuan Liu, Shen Zhang +13
Beihang University · JD Company · University of Science and Technology Beijing +1
The remarkable success of large language models (LLMs) has provided important inspiration for the next generation of recommender systems. Structurally, recommendation and language generation share a similarity: both aim to produce an ordered sequence that optimizes the user's experience. However, how to precisely absorb the essence of the LLM paradigm into mature industrial recommender systems remains an open problem. There are two challenges. First, it is unclear how to incorporate the LLM paradigm -- sequence-level generation and optimization -- into recommendation. Second, real-world recommender systems are mature systems that have been iteratively customized for years around specific products, business constraints, serving infrastructure, and organizational ownership. Replacing such systems wholesale is often technically risky and organizationally disruptive. In this paper, we propose LIGE-GR, a listwise generation and evaluation recommendation framework that upgrades from a traditional ranking system (itemwise recommendation) toward a generative recommendation paradigm. Instead of rebuilding the entire recommendation stack from scratch, LIGE-GR generalizes the existing pointwise recommendation system into a listwise generation system. This allows mature recommender systems to benefit from listwise optimization while preserving compatibility with existing models, value functions, and serving infrastructure. We validate LIGE-GR in short-video recommendation on Instagram Reels and Facebook Video. On these recommendation surfaces, LIGE-GR improves time spent by 1.14 percent on Instagram Reels and 0.72 percent on Facebook Video, while requiring only modest additional inference resources.
Generative Retrieval (GR) offers a promising paradigm for recommendation through next-token prediction (NTP). However, scaling it to large-scale industrial systems introduces three challenges: (i) within a single request, the identical model inputs may produce inconsistent outputs due to the pagination request mechanism; (ii) the prohibitive cost of encoding long user behavior sequences with multi-token item representations based on semantic IDs, and (iii) aligning the generative policy with nuanced user preference signals. We present GenRec, a preference-oriented generative framework deployed on the JD App that addresses above challenges within a single decoder-only architecture. For training objective, we propose Page-wise NTP task, which supervises over an entire interaction page rather than each interacted item individually, providing denser gradient signal and resolving the one-to-many ambiguity of point-wise training. On the prefilling side, an asymmetric linear Token Merger compresses multi-token Semantic IDs in the prompt while preserving full-resolution decoding, reducing input length by ~2X with negligible accuracy loss. To further align outputs with user satisfaction, we introduce GRPO-SR, a reinforcement learning method that pairs Group Relative Policy Optimization with NLL regularization for training stability, and employs Hybrid Rewards combining a dense reward model with a relevance gate to mitigate reward hacking. In month-long online A/B tests serving production traffic, GenRec achieves 9.5% improvement in click count and 8.7% in transaction count over the existing pipeline.
Yanyan Zou, Junbo Qi, Lunsong Huang +7
JD.com Beijing, China · Waseda University Tokyo, Japan