Beyond Interleaving: Causal Attention Reformulations for Generative Recommender Systems

Hailing Cheng

cs.IRMar 11, 2026

Beyond Interleaving: Causal Attention Reformulations for Generative Recommender Systems

Authors: Hailing Cheng

Organizations: Linkedin Inc Mountain View, California, USA

Abstract

Generative Recommender Systems (GR) increasingly model user behavior as a sequence generation task by interleaving item and action tokens. While effective, this formulation introduces significant structural and computational inefficiencies: it doubles sequence length, incurs quadratic overhead, and relies on implicit attention to recover the causal relationship between an item and its associated action. Furthermore, interleaving heterogeneous tokens forces the Transformer to disentangle semantically incompatible signals, leading to increased attention noise and reduced representation efficiency.In this work, we propose a principled reformulation of generative recommendation that aligns sequence modeling with underlying causal structures and attention theory. We demonstrate that current interleaving mechanisms act as inefficient proxies for similarity-weighted action pooling. To address this, we introduce two novel architectures that eliminate interleaved dependencies to reduce sequence complexity by 50%: Attention-based Late Fusion for Actions (AttnLFA) and Attention-based Mixed Value Pooling (AttnMVP). These models explicitly encode the $i_n \rightarrow a_n$ causal dependency while preserving the expressive power of Transformer-based sequence modeling.We evaluate our framework on large-scale product recommendation data from a major social network. Experimental results show that AttnLFA and AttnMVP consistently outperform interleaved baselines, achieving evaluation loss improvements of 0.29% and 0.80%, and significant gains in Normalized Entropy (NE). Crucially, these performance gains are accompanied by training time reductions of 23% and 12%, respectively. Our findings suggest that explicitly modeling item-action causality provides a superior design paradigm for scalable and efficient generative ranking.

Figures & tables

Panel of Figure 1 — Figure 1. Interleaved generative recommenders treat items and actions as a single token stream. Action $a_{2}$ attends to all prior tokens, obscuring the direct causal dependency $i_{2}\rightarrow a_{2}$ and introducing attention noise. Interleaved generative recommenders

Panel of Figure 2 — Figure 2. True causal structure of user interactions. Each action $a_{n}$ is a response to the corresponding item $i_{n}$ , conditioned on prior history. This structure is not explicitly represented by interleaved self-attention. True causal structure of user interactions

Figure 3. Traditional Generative Recommender (Interleaving Item and Action Tokens) architecture: the item and action tokens are interleaved before the transformer layers Input is a sequence of Feed post and action embedding, interleaved.

Figure 4. Illustrative toy sequences for Users A and B. The sequences demonstrate contrasting behavioral patterns: User A consistently exhibits positive interactions (e.g., "Like") with dog-related items and negative interactions with cat-related items, while User B exhibits the inverse preference profile. This highlights the model’s task of capturing item-action dependencies for future state prediction. User A and User B input sequence

Figure 5. Attention-based Late Fusion for Action (AttnLFA). Item embeddings are transformed through a series of Transformer blocks (labeled as "Transformers" for clarity) to generate latent sequence representations. These representations serve as both Queries and Keys for the subsequent attention mechanism. In the final stage, action embeddings are integrated as Values via a causally-constrained attention pooling operation, conditioned on the sequence context. The resulting aggregated action representation is then passed to the prediction head for the final output. User A and User B input sequence

Figure 6. The query-shifting mechanism to enforce a strict causal constraint.

Model	Eval Loss	LongDwell NE	Contribution NE	Like NE	Time
Baseline	-	-	-	-	-
AttnLFA	-0.29%	-0.06%	-0.49%	-0.47%	-22.8%

Table 1. Performance comparison between AttnLFA and Baseline. We report the relative improvement across four key dimensions: (a) Multi-task Binary Cross Entropy (BCE) Loss, (b) evaluation Normalized Entropy (NE) for Long Dwell, Contribution, and Like actions, and (c) total training latency. AttnLFA demonstrates superior predictive accuracy (lower NE/Loss) while maintaining competitive computational efficiency.

Figure 7. Attention-based Mixed Value Pooling (AttnMVP) architecture. Item embeddings serve as Queries and Keys in each Transformer layer, while item and action embeddings are additively fused as mixed Values. Across stacked layers, action signals are progressively injected into item representations under strict causal constraints. In the final stage, action embeddings are pooled via causally masked attention conditioned on the sequence-level item representations, and the pooled action representation is fused with the final item embedding to produce action predictions. Attention-based Mixed Value Pooling

Model	Eval Loss	LongDwell NE	Contribution NE	Like NE	Time
Baseline	-	-	-	-	-
AttnMVP	-0.80%	-0.41%	-1.1%	-1.1%	-12.3%
AttnMVP - LFA	-0.78%	-0.40%	-1.0%	-1.0%	-13.02%

Table 2. Relative performance improvement of Eval Loss, major tasks’ Eval Normalized Entropies (NEs) and Training Time for Baseline, AttnMVP and AttnMVP without LFA. (AttnMVP-LFA)

Figure 8. Attention-based Dual-Helix Network (AttnDHN) architecture. Action embedding and Item embeddings are updated in pair-wise sequence in individual Transformer layer. Both transformer layer use either action and item embedding as query and key, and use a combination of item + action embedding as value. In the final stage, action embeddings are pooled via causally masked attention conditioned on the sequence-level item representations, and the pooled action representation is fused with the final item embedding to produce action predictions. Attention-based Mixed Value Pooling

Explore similar work

Trie-Aware Transformers for Generative Recommendation
Feb 25, 2026Zhenxiang Xu, Jiawei Chen, Sirui Chen +5Generative Recommendation Sequential Recommendation
State Key Laboratory of Blockchain and Data Security, Zhejiang University. · College of Computer Science and Technology, Zhejiang University. · Zhejiang University Hangzhou, China +2
xGR: Efficient Generative Recommendation Serving at Scale
Dec 12, 2025Qingxiao Sun, Tongxuan Liu, Shen Zhang +13LLM Serving LLM Inference Acceleration
Beihang University · JD Company · University of Science and Technology Beijing +1
GenRec: A Preference-Oriented Generative Framework for Large-Scale Recommendation
Apr 16, 2026Yanyan Zou, Junbo Qi, Lunsong Huang +7Generative Retrieval Recommender Systems
JD.com Beijing, China · Waseda University Tokyo, Japan

Feb 25, 2026J/K move · Enter open · S save

Trie-Aware Transformers for Generative Recommendation

Zhenxiang Xu, Jiawei Chen, Sirui Chen, Yong He+4

State Key Laboratory of Blockchain and Data Security, Zhejiang University. · College of Computer Science and Technology, Zhejiang University. · Zhejiang University Hangzhou, China+2

Generative Recommendation Sequential Recommendation Transformer Positional Encoding

Open paper AskPDF arXiv

Generative recommendation (GR) aligns with advances in generative AI by casting next-item prediction as token-level generation rather than score-based ranking. Most GR methods adopt a two-stage pipeline: (i) item tokenization, which maps each item to a sequence of discrete, hierarchically organized tokens; and (ii) autoregressive generation, which predicts the next item's tokens conditioned on the tokens of user's interaction history. Although hierarchical tokenization induces a prefix tree (trie) over items, standard autoregressive modeling with conventional Transformers often flattens item tokens into a linear stream and overlooks the underlying topology. To address this, we propose TrieRec, a trie-aware generative recommendation method that augments Transformers with structural inductive biases via two positional encodings. First, a trie-aware absolute positional encoding aggregates a token's (node's) local structural context (\eg depth, ancestors, and descendants) into the token representation. Second, a topology-aware relative positional encoding injects pairwise structural relations into self-attention to capture topology-induced semantic relatedness. TrieRec is also model-agnostic, efficient, and hyperparameter-free. In our experiments, we implement TrieRec within three representative GR backbones, achieving notably improvements of 8.83% on average across four real-world datasets.

Abstract

Explore similar work

Trie-Aware Transformers for Generative Recommendation

xGR: Efficient Generative Recommendation Serving at Scale

GenRec: A Preference-Oriented Generative Framework for Large-Scale Recommendation