Generative recommendation (GR) is moving industrial recommender systems (RecSys) from scoring candidate items to autoregressively generating item identifiers. This shift promises higher recommendation quality, but makes serving substantially harder: each request executes a short sequence of dependent decoding steps that explore a large item space and select promising candidates on the latency-critical path. GR not only creates structural mismatches with conventional RecSys pipelines, but also challenges the architectural and optimization assumptions of LLM serving stacks, making direct adoption of existing systems difficult. We present RecoGEM, our production GR serving system. RecoGEM uses a runtime--engine decoupling architecture to move GR-specific serving responsibilities, such as candidate-state transitions and live artifact updates, to a dedicated runtime. Its execution engine exploits GR's short token sequences, large candidate populations, and predictable step shapes to optimize self-attention, candidate selection, and execution planning. RecoGEM has been deployed for over nine months across our production lines, serving applications with hundreds of millions of users and supporting more than ten GR model families. On our OneModel production workloads, RecoGEM reaches over 1,000 QPS per serving instance with P99 latency below 35 ms and a 2.88× throughput improvement over representative baselines.
Figures & tables
Figure 1. Traditional vs. generative recommendation. A cascaded DLRM workflow that scores a shrinking candidate set, beside a generative workflow in which one model decodes the item's semantic identifier.
Figure 2. LLM serving vs. GR serving. llm vs gr
Figure 3. Execution of a GR request. A two-part request diagram. Phase 1, context encoding, takes user static, short-term, and long-term pathways, passes them through a context processor, and produces a context representation or KV cache. Phase 2, autoregressive decoding, starts from BOS and repeatedly applies decoder blocks conditioned on the encoded context to generate SID tokens s1 through sL for one recommended item.
Figure 4. Heterogeneous transitions between decoding steps and online updates of serving artifacts. Left: a GR serving box in which a compute engine runs decoding step l with vocabulary V_l and emits expanded candidates; a selection stage such as beam search keeps the surviving candidates; their candidate state (prefix, score, and KV) becomes the next-step input to the compute engine at step l+1 with vocabulary V_{l+1}. Selection and candidate state sit in a shaded step-transition region outside the engine. Right: online updates of model weights and the SID vocabulary point to the compute engine, and selection-policy updates point to the selection stage.
Figure 5. Production online weight-refit activity in 24-hours: refit frequency (left) and FP32 refit traffic per server (right). Two bar charts over a 24-hour capture window. The left chart shows model weight refits per minute, with a peak of 107 refits per minute. The right chart shows FP32 traffic per server in gigabytes per hour, with a peak of 346 gigabytes per hour.
Figure 6. Attention dimension inversion in GR decode. A two-panel schematic. The left panel shows LLM decode attention as a long horizontal KV sequence tile. The right panel shows GR decode as many short row-local attention rows and a row-local Top-K selection over a candidate width.
Figure 7. Decoding steps in LLM vs. GR serving. A contrast between homogeneous token generation, which repeats the same decode transition from token to token, and heterogeneous GR decoding, where a decode step produces probabilities, tokens, and prefix indices and expands dimensions for the next step.
Figure 8. Candidate selection design choices. A two-panel figure. Panel (a) shows Top-K dimension inversion: LLM decode selects a small K, typically K<10, from a wide token vocabulary, whereas GR beam update selects a large K, typically K>128, from a comparatively narrow semantic-id vocabulary on every step. Panel (b) shows a throughput comparison between radix-based Top-K and bitonic Top-K kernels across K values from 16 to 1024, illustrating that a single sampling-tail kernel choice does not dominate across the K range exercised by GR beam update.
Figure 9. RecoGEM serving system overview. A simplified system architecture diagram. Dense and sparse embeddings and a configuration contract feed an infer-server binary. The binary orchestrates GPU-resident \sysengine execution, model weights, and custom kernels, with a loop arrow indicating iterative execution.
Figure 10. RecoGEM deployment contract. A diagram showing a model graph and deployment policy feeding a contract generator, which produces a deployment contract. The contract contains weight binding, input-output mapping, request batching, and pipeline flow records. The runtime server fetches the contract and uses it to control execution of the compute graph.
Figure 11. Runtime state management for a GR request. A timeline-style diagram showing a \sysbatch as contract-defined GPU tensor slices. Request inputs are copied into the batch buffer before enqueue. Context, self-KV, candidate ids, probabilities, and cache-indirection tensors are carried across decode steps by output-to-input mappings. Final outputs are copied to request buffers before the batch buffer is returned to the pool.
Figure 12. Sequence tiling versus row tiling. Sequence tiling splits one long attention row; row tiling groups R complete short rows per tile.
Figure 13. Fused attention dataflow and cross-tile pipelining. Top: fused per-row attention keeps intermediates on chip. Bottom: loading tile k+1 overlaps computation on tile k.
Figure 14. Step-granular execution planning lowers a single-step GR graph into step-specialized engines by probing per-step shapes and rebuilding under step-bounded profiles. A left-to-right planning pipeline. On the left, a dashed box labeled Training Graph stacks Item Embedding, N decoder blocks, and Candidate Selection, with a Conditional Boundaries box feeding back into the graph. The first Build stage turns this graph into a Broad Range Compute Graph. The Probe Steps stage, over generation steps t equals 1, 2, and onward, passes it through a Profile Optimizer that records per-step shapes. The second Build stage emits a stacked Step Heterogeneous Compute Graph specialized for each generation step.
Figure 15. Performance trend of OneRec-v2. Four curves over batch sizes 1 to 32 compare GeneralGen, \sys, TorchEager, and the adapted TensorRT-LLM runtime. The panels show P99 latency and QPS on logarithmic vertical axes.
Figure 16. Performance trend of OneModel. Four curves over batch sizes 1 to 32 compare GeneralGen, \sys, TorchEager, and the adapted TensorRT-LLM runtime with dynamic beam widths 128, 256, 512, and 1024. The panels show P99 latency and QPS on logarithmic vertical axes.
Figure 17. Batch-32 step-wise kernel breakdown for three serving systems in OneModel. Four groups of stacked bars compare GeneralGen, HomoGen, and \sysacross OneModel's four SID steps at batch size 32. Each bar separates attention, beam search, and other kernels. Each component is divided by \sys's total kernel latency at the same step, so every \sysbar totals one.
Figure 18. Decode self-attention latency for the GR short-prefix setting across batch and head sweeps. Six log-scale line plots arranged in a two-by-three grid, comparing PyTorch Native, SDPA, the earlier per-row \syspath, and the current \sysrow-tiled path across short key/value lengths. The top row fixes the number of heads at 32 and sweeps effective batch over 4096, 8192, and 16384; the bottom row repeats the sweep at 64 heads. \sysis the fastest backend in every panel.
Figure 19. Beam-search Top-K latency for GR-style candidate selection with vocabulary size 8192. A log-scale grouped bar chart comparing Top-K latency across request batch sizes 4, 8, 16, and 32 and beam widths 512 and 1024. \sys's shape-aware Top-K selection is the lowest bar in each group, while forced WarpBitonic becomes much slower at beam width 1024.
Figure 20. QPS and P99 latency over 24h. A dual-axis line plot showing QPS and P99 latency over a 24h period.