Beyond Interleaving: Causal Attention Reformulations for Generative Recommender Systems

Hailing Cheng

cs.IRMar 11, 2026

Beyond Interleaving: Causal Attention Reformulations for Generative Recommender Systems

Authors: Hailing Cheng

Organizations: Linkedin Inc Mountain View, California, USA

Abstract

Generative Recommender Systems (GR) increasingly model user behavior as a sequence generation task by interleaving item and action tokens. While effective, this formulation introduces significant structural and computational inefficiencies: it doubles sequence length, incurs quadratic overhead, and relies on implicit attention to recover the causal relationship between an item and its associated action. Furthermore, interleaving heterogeneous tokens forces the Transformer to disentangle semantically incompatible signals, leading to increased attention noise and reduced representation efficiency.In this work, we propose a principled reformulation of generative recommendation that aligns sequence modeling with underlying causal structures and attention theory. We demonstrate that current interleaving mechanisms act as inefficient proxies for similarity-weighted action pooling. To address this, we introduce two novel architectures that eliminate interleaved dependencies to reduce sequence complexity by 50%: Attention-based Late Fusion for Actions (AttnLFA) and Attention-based Mixed Value Pooling (AttnMVP). These models explicitly encode the $i_n \rightarrow a_n$ causal dependency while preserving the expressive power of Transformer-based sequence modeling.We evaluate our framework on large-scale product recommendation data from a major social network. Experimental results show that AttnLFA and AttnMVP consistently outperform interleaved baselines, achieving evaluation loss improvements of 0.29% and 0.80%, and significant gains in Normalized Entropy (NE). Crucially, these performance gains are accompanied by training time reductions of 23% and 12%, respectively. Our findings suggest that explicitly modeling item-action causality provides a superior design paradigm for scalable and efficient generative ranking.

Figures & tables

Panel of Figure 1 — Figure 1. Interleaved generative recommenders treat items and actions as a single token stream. Action $a_{2}$ attends to all prior tokens, obscuring the direct causal dependency $i_{2}\rightarrow a_{2}$ and introducing attention noise. Interleaved generative recommenders

Panel of Figure 2 — Figure 2. True causal structure of user interactions. Each action $a_{n}$ is a response to the corresponding item $i_{n}$ , conditioned on prior history. This structure is not explicitly represented by interleaved self-attention. True causal structure of user interactions

Figure 3. Traditional Generative Recommender (Interleaving Item and Action Tokens) architecture: the item and action tokens are interleaved before the transformer layers Input is a sequence of Feed post and action embedding, interleaved.

Figure 4. Illustrative toy sequences for Users A and B. The sequences demonstrate contrasting behavioral patterns: User A consistently exhibits positive interactions (e.g., "Like") with dog-related items and negative interactions with cat-related items, while User B exhibits the inverse preference profile. This highlights the model’s task of capturing item-action dependencies for future state prediction. User A and User B input sequence

Figure 5. Attention-based Late Fusion for Action (AttnLFA). Item embeddings are transformed through a series of Transformer blocks (labeled as "Transformers" for clarity) to generate latent sequence representations. These representations serve as both Queries and Keys for the subsequent attention mechanism. In the final stage, action embeddings are integrated as Values via a causally-constrained attention pooling operation, conditioned on the sequence context. The resulting aggregated action representation is then passed to the prediction head for the final output. User A and User B input sequence

Figure 6. The query-shifting mechanism to enforce a strict causal constraint.

Model	Eval Loss	LongDwell NE	Contribution NE	Like NE	Time
Baseline	-	-	-	-	-
AttnLFA	-0.29%	-0.06%	-0.49%	-0.47%	-22.8%

Table 1. Performance comparison between AttnLFA and Baseline. We report the relative improvement across four key dimensions: (a) Multi-task Binary Cross Entropy (BCE) Loss, (b) evaluation Normalized Entropy (NE) for Long Dwell, Contribution, and Like actions, and (c) total training latency. AttnLFA demonstrates superior predictive accuracy (lower NE/Loss) while maintaining competitive computational efficiency.

Figure 7. Attention-based Mixed Value Pooling (AttnMVP) architecture. Item embeddings serve as Queries and Keys in each Transformer layer, while item and action embeddings are additively fused as mixed Values. Across stacked layers, action signals are progressively injected into item representations under strict causal constraints. In the final stage, action embeddings are pooled via causally masked attention conditioned on the sequence-level item representations, and the pooled action representation is fused with the final item embedding to produce action predictions. Attention-based Mixed Value Pooling

Model	Eval Loss	LongDwell NE	Contribution NE	Like NE	Time
Baseline	-	-	-	-	-
AttnMVP	-0.80%	-0.41%	-1.1%	-1.1%	-12.3%
AttnMVP - LFA	-0.78%	-0.40%	-1.0%	-1.0%	-13.02%

Table 2. Relative performance improvement of Eval Loss, major tasks’ Eval Normalized Entropies (NEs) and Training Time for Baseline, AttnMVP and AttnMVP without LFA. (AttnMVP-LFA)

Figure 8. Attention-based Dual-Helix Network (AttnDHN) architecture. Action embedding and Item embeddings are updated in pair-wise sequence in individual Transformer layer. Both transformer layer use either action and item embedding as query and key, and use a combination of item + action embedding as value. In the final stage, action embeddings are pooled via causally masked attention conditioned on the sequence-level item representations, and the pooled action representation is fused with the final item embedding to produce action predictions. Attention-based Mixed Value Pooling

Explore similar work

Feb 25, 2026

Trie-Aware Transformers for Generative Recommendation

Generative recommendation (GR) aligns with advances in generative AI by casting next-item prediction as token-level generation rather than score-based ranking. Most GR methods adopt a two-stage pipeline: (i) item tokenization, which maps each item to a sequence of discrete, hierarchically organized tokens; and (ii) autoregressive generation, which predicts the next item's tokens conditioned on the tokens of user's interaction history. Although hierarchical tokenization induces a prefix tree (trie) over items, standard autoregressive modeling with conventional Transformers often flattens item tokens into a linear stream and overlooks the underlying topology. To address this, we propose TrieRec, a trie-aware generative recommendation method that augments Transformers with structural inductive biases via two positional encodings. First, a trie-aware absolute positional encoding aggregates a token's (node's) local structural context (\eg depth, ancestors, and descendants) into the token representation. Second, a topology-aware relative positional encoding injects pairwise structural relations into self-attention to capture topology-induced semantic relatedness. TrieRec is also model-agnostic, efficient, and hyperparameter-free. In our experiments, we implement TrieRec within three representative GR backbones, achieving notably improvements of 8.83% on average across four real-world datasets.

Zhenxiang Xu, Jiawei Chen, Sirui Chen +5

State Key Laboratory of Blockchain and Data Security, Zhejiang University. · College of Computer Science and Technology, Zhejiang University. · Zhejiang University Hangzhou, China +2

Generative Recommendation Sequential Recommendation Transformer Positional Encoding

Ask PDF

Dec 12, 2025

xGR: Efficient Generative Recommendation Serving at Scale

Recommendation system delivers substantial economic benefits by providing personalized predictions. Generative recommendation (GR) integrates LLMs to enhance the understanding of long user-item sequences. Despite employing attention-based architectures, GR's workload differs markedly from that of LLM serving. GR typically processes long prompt while producing short, fixed-length outputs, yet the computational cost of each decode phase is especially high due to the large beam width. Furthermore, since the beam search involves a vast item space, the sorting overhead becomes particularly time-consuming. We propose xGR, a GR-oriented serving system that meets strict low-latency requirements under high-concurrency scenarios. First, xGR unifies the processing of prefill and decode phases through staged computation and separated KV cache. Second, xGR enables early sorting termination and mask-based item filtering with data structure reuse. Third, xGR reconstructs the overall pipeline to exploit multi-level overlap and multi-stream parallelism. The experiments on real-world datasets demonstrate that xGR achieves at least 2.89x throughput compared to the state-of-the-art baseline under strict latency constraints.

Qingxiao Sun, Tongxuan Liu, Shen Zhang +13

Beihang University · JD Company · University of Science and Technology Beijing +1

LLM Serving LLM Inference Acceleration Recommender Systems Generative Recommendation

Ask PDF

Apr 16, 2026

GenRec: A Preference-Oriented Generative Framework for Large-Scale Recommendation

Generative Retrieval (GR) offers a promising paradigm for recommendation through next-token prediction (NTP). However, scaling it to large-scale industrial systems introduces three challenges: (i) within a single request, the identical model inputs may produce inconsistent outputs due to the pagination request mechanism; (ii) the prohibitive cost of encoding long user behavior sequences with multi-token item representations based on semantic IDs, and (iii) aligning the generative policy with nuanced user preference signals. We present GenRec, a preference-oriented generative framework deployed on the JD App that addresses above challenges within a single decoder-only architecture. For training objective, we propose Page-wise NTP task, which supervises over an entire interaction page rather than each interacted item individually, providing denser gradient signal and resolving the one-to-many ambiguity of point-wise training. On the prefilling side, an asymmetric linear Token Merger compresses multi-token Semantic IDs in the prompt while preserving full-resolution decoding, reducing input length by ~2X with negligible accuracy loss. To further align outputs with user satisfaction, we introduce GRPO-SR, a reinforcement learning method that pairs Group Relative Policy Optimization with NLL regularization for training stability, and employs Hybrid Rewards combining a dense reward model with a relevance gate to mitigate reward hacking. In month-long online A/B tests serving production traffic, GenRec achieves 9.5% improvement in click count and 8.7% in transaction count over the existing pipeline.

Yanyan Zou, Junbo Qi, Lunsong Huang +7

JD.com Beijing, China · Waseda University Tokyo, Japan

Generative Retrieval Recommender Systems Large Language Model-Based Recommendation User Preference Modeling Generative Recommendation

Ask PDF