cs.LGNov 17, 2026

Optimize the Cob, Then the Kernel: Optimizing Beam Search for Generative Recommendation Inference

Authors: Shing Wai Pun, Kai Wu, Pengfei Xia, Hui Zang, Wei Guo, Yong Liu

Organizations: Huawei Technologies Co., Ltd.

Abstract

Generative recommendation (GR) inference differs substantially from conventional LLM serving, featuring long input sequences, wide beam search, and only a few fixed output tokens. Existing LLM serving engines such as vLLM process beams as batched independent sequences, causing redundant KV cache accesses over the shared prompt prefix. Furthermore, vLLM executes each beam search generation step as a separate engine invocation, incurring host-side beam processing overhead between decoding steps and substantial runtime KV block management in PagedAttention. Using Tree Attention, we concatenate beams along the sequence dimension to eliminate redundant KV accesses, and further optimize execution through pre-generated partial-attention masks, static KV cache allocation, loop-unrolled generation dispatch, TorchAir graph compilation, automatic operator fusion, and vocabulary-constrained decoding. Without the use of custom kernels, our approach reduces end-to-end latency by up to 25x over vLLM on Ascend NPUs, and achieves generation latency comparable to a SOTA solution using custom kernels, showing that the generation pipeline should be optimized holistically before resorting to custom kernel development.

Figures & tables

Explore similar work

CardsList
  1. OneLA: Scaling Linear-Attention Decoding to Large Beams in Generative Recommendation

    Sep 11, 2026Xiangrui Yang, Cheng Peng, Yunfeng Zhao +9Linear AttentionAutoregressive Decoding

  2. xGR: Efficient Generative Recommendation Serving at Scale

    Dec 12, 2025Qingxiao Sun, Tongxuan Liu, Shen Zhang +13LLM ServingLLM Inference Acceleration

  3. Provable Test-Time Scaling for Beam Search in LLM Reasoning

    Sep 29, 2026Qijia He, Yu Huang, Yuan Cheng +2Long-Horizon LLM ReasoningInference-Time Search